Add JSONField model field

[paltman]

I have been using this for over a year now and find it quite useful. I posted it about it the other day (​http://paltman.com/2010/feb/25/how-to-store-arbitrary-data-in-a-django-model/) and then thought it might be generally useful as an out of the box field type so I thought I'd submit a patch in case it is deemed "generally useful" enough to be considered for inclusion in django proper.

[paltman]

JSON Field Patch

[Alex Gaynor]

I'm -1 on this, I think it's out of scope for Django core, and easily accomplished outside. Regardless I'm removing the 1.2 milestone, 1.2 is in feature freeze.

[floguy]

+1 on this. There are many implementations of this floating around, and most of them are wrong. It's a common-enough pattern that Django should provide the canonical implementation.

[Marcelo Ramos]

+1 If XMLField is in the core then JSONField deserves to be in too. Also I'm tempted to propose the removal of XMLField, you all know that
the use of XML as a format to store arbitrary data has dramatically decreased in favour of JSON.

[Alex Gaynor]

Replying to floguy:

+1 on this. There are many implementations of this floating around, and most of them are wrong. It's a common-enough pattern that Django should provide the canonical implementation.

How are most of them wrong? It's pretty trivial (there's actually an implementation in the tests).

Replying to marcelor:

+1 If XMLField is in the core then JSONField deserves to be in too. Also I'm tempted to propose the removal of XMLField, you all know that
the use of XML as a format to store arbitrary data has dramatically decreased in favour of JSON.

XMLField at this point does almost nothing (doesn't even do validation at this point).

[floguy]

Most of the time the problems are with dumpdata/loaddata. Anyway, the sheer proliferation of implementations of this should make it clear why a canonical implementation should be in core.

• ​http://www.davidcramer.net/code/448/cleaning-up-with-json-and-sql.html
• ​http://www.djangosnippets.org/snippets/377/
• ​http://www.djangosnippets.org/snippets/1478/
• ​http://github.com/djblets/djblets/blob/master/djblets/util/fields.py#L136
• ​http://github.com/django-extensions/django-extensions/blob/master/django_extensions/db/fields/json.py
• ​http://github.com/bradjasper/django-jsonfield
• ​http://bitbucket.org/offline/django-annoying/src/tip/annoying/fields.py#cl-34
• ​http://bitbucket.org/lorien/django-common/src/0896d0968ea5/common/fields.py#cl-29

[Alex Gaynor]

The fact that people don't know how to organize code externally cannot act as an argument in favor of including everything in django's core.

[floguy]

The proliferation of implementations proves that there's significant interest in the feature. Since each of the implementations are slightly different, it makes it difficult to rely on the behavior of any one implementation. Also, some of the implementations behave incorrectly when using dumpdata/loaddata. Including it in core would solve all of these problems, as well as provide much lower friction for a new user who does not know which implementation to choose.

[Peter Bengtsson]

+1 on this. I'm not sure what it means in terms of exports/dumps/fixtures etc.

@Alex: the reason we do Django is very much because it's a convenient frameworks that helps getting things done quickly.

[Jeff Bauer]

+1 me too, vote. I'd really like to see this added to Django. A single implementation of JSONField would be an improvement over the current state.

[Łukasz Rekucki]

The patch has some tests, but their coverage is very limited. Some things that need testing:

• get_*_json/set_*_json hooks
• does this work properly with dumpdata/loaddata in all formats ?

Having an option to override the encoder/decoder without having to subclass the field would be nice (like in some other implementations).

A major argument for me against having this in core, is that you can't include South support (at least atm, we'll see what happens in 1.4), so I would still prefer a 3rd party implementation that does.

[joshg]

I just wanted to say, I would like this to be built into the core. Thanks!

[Mark Jones]

+1 Considering all the versions of this floating out there and all the patches due to them not being quite right. Putting this is to Django seems to be the right way to do it.

[Russell Keith-Magee]

For the record - I'm -0 on this. For my money, if you're putting JSON in a database field (and expecting it to be searchable etc), you're doing it wrong. I won't stand in the way if someone else in the core team feels particularly enthused, but I'm not rushing myself.

Also, the previous argument about XMLField has been rendered invalid -- we removed XMLField in 1.3.

[Jannis Leidel]

Agreed, I think projects like ​https://github.com/jordanm/django-hstore show that there are db specific fields that are better suited for storing JSON.

[sorl]

I think the usecase for something like this should be simply serialized data in a field, not searchable and not just a mapping to a dict. Given that, we might as well use pickle, but then we might loose data portability. If you want to build an efficient relational db centric application you probably want to use something like this and I think many people realize that but the many implementations floating around makes it hard for them to decide which one to use. If Django wants to advocate for that kind of database design then perhaps it is a good idea to include a serialization field. I have a very simple lazier approach, it can't serialize strings but it only deserializes on attr access, ​https://github.com/aino/django-cerial/blob/master/cerial/fields.py

[ShawnMilo]

I'd like to see this as well, because we use a JSON field frequently. I think the argument against that searching JSON in a queryset is poor design is a straw-man argument. I haven't heard anyone ask for a JSON field with this in mind. It's just convenient to store serialized data to avoid unnecessary auxiliary storage and proliferation of extra fields and models.

Having a canonical JSON field is better than people implementing their own, in my opinion, because it makes it provides consistency across projects without needing to copy/paste code into each new project.

[Fredde]

Replying to Alex:

How are most of them wrong? It's pretty trivial (there's actually an implementation in the tests).

The JSONField included in the in the django tests ([source:django/trunk/tests/modeltests/field_subclassing/fields.py#L56]) is actually not working properly eather. For example, when trying to store a empty list [] or dict {} the JSONField.to_python will convert it to None:

#python
>>> class A(models.Model):
	json_field = JSONField()

>>> a = A()
>>> a.json_field = []
>>> print a.json_fied
None

I don't know how to fix this. From the documentation the following values should be handled by the to_python method:

• An instance of the correct type (e.g., Hand in our ongoing example).
• A string (e.g., from a deserializer).
• Whatever the database returns for the column type you're using.

Hovever, there is no way to know if the value is a "correct type" a serialized representation or the value from the database. For example value = '[1, 2, 3]'. It could eather come from an attribute assignment, or from the database, but it should only be deserialized if it's from the database.

[anonymous]

Replying to russellm:

For the record - I'm -0 on this. For my money, if you're putting JSON in a database field (and expecting it to be searchable etc), you're doing it wrong.

You cannot say that generally. The answer is as always, it depends. There are perfectly valid uses of this, for example, storing a little (dynamic) metadata without setting up a NoSQL database.

[Russell Keith-Magee]

If you're arguing for JSONField as a way of avoiding setting up a NoSQL store, then you've just proved my point. If you need a non-relational store, *use a non-relational store*. Yes, there may be reasons why this isn't possible for political or business reasons, but that doesn't change the fact that:

• There is a better way to solve the problem
• Django shouldn't be encouraging people to hack around the root cause of problems
• The absence of a JSONField in Django core doesn't prevent the community from developing a canonical implementation that can be used as a third-party library.
• Putting JSONField in core wouldn't affect the stability or reliability of that third party implementation -- it just means the core team are the only people who can maintain it.

[Thomas Güttler]

By setting up a NoSQL store you loose the transactional atomic behavior: commit all or nothing. AFAIK two-phase commit is not supported by django. I think a JSONField inside django would be good. I don't want to query the JSON value. I just use JSONField like cache that never expires.

[Lee S]

I would love to see a canonical version of JSON fields included.

I routinely find storing non-structured, non-searchable data in the database to be extremely useful in many use cases. Setting up a NoSQL database is often overkill. The real need is to store the odd JSON field in an otherwise traditional SQL model. The example cited above, storing flexible metadata is a common one -- for example, key/value pairs indicating user preferences.

I'm at the point where I have to try one of the many implementations listed above and cross my fingers that it's not broken.

[Aymeric Augustin]

I've recently rolled my own implementation of this.

I use it in a logging table where I want to store an event_kind (ex : "USER_LOGIN") + a datetime + a bunch of parameters (ex : username, remote IP) that depend on the kind of event. I want to be able to add arbitrary parameters at any point. Normalizing parameters in a separate table (e.g. storing event_id, key, value in another table for each parameter) seems overkill — I don't want to add a model just for this.

So I think there are some real-life use cases for storing a JSON-serialized dict in a database field. Since it's moderately difficult to get right, +0 for adding it in Django.

[radbrad182+djangoproject@…]

JSON fields are also a great way to store basic, dynamic, typed data. Being able to store Key Value data in the DB would be useful:

class Key(models.Model):
   user = models.ForeignKey(User)
   name = models.CharField()
   value = models.JSONField()

user.key_set.create(name="an_integer", value=4)
user.key_set.create(name="a_list", value=['a','b','c'])

Normally to get this kind of basic data typing you would need to store another field for value_type, and then parse manually.

Arguing that this should be stored in a K/V store is not ideal as it adds dependency on another datastore type, and you lose out on user.key_set.get(name='something').

[Riccardo Di Virgilio]

this is my proposition to for the field.

class JsonField(models.Field): 
    __metaclass__ = models.SubfieldBase 
    serialize_to_string = True 
    def get_internal_type(self): 
        return "TextField" 
    def value_to_string(self, obj): 
        return self.get_prep_value(self._get_val_from_obj(obj)) 
    def get_prep_value(self, value): 
        if value: 
            stream = StringIO.StringIO() 
            simplejson.dump(value, stream, cls=DjangoJSONEncoder) 
            value = stream.getvalue() 
            stream.close() 
            return value 
        return None 
    def to_python(self, value): 
        if isinstance(value, (str, unicode)): 
            value = StringIO.StringIO(value) 
            return simplejson.load(value) 
        return value 

maybe there is no need to to create a class
JSONDateEncoder(json.JSONEncoder) because there is DjangoJSONEncoder
already.

then it would be great to implement a lazy translation object for
DjangoJSONEncoder to store ugettext_lazy objects.

for me there is no need to create set_%s_json and get_%s_json method
because the field should handle it directly, and we should use the
to_python method, like all other fields in django.

for example the datetime field directly push a datetime object to the
model (using to_python) and transform it to a string when save
method is called

we should use the same logic here and encode a json object to string
only when we call the save method, no need for a get_%s_json in the
field api.

[anonymous]

What's the best third-party alternative out there? It seems like this is going to drag on for a while in a religious discussion of whether it should be included or not. Just add it and let people decide whether and how to use it.

[Aymeric Augustin]

django-extensions is a fairly popular project that provides an ​implementation of JSONField.

[Aymeric Augustin]

This idea was discussed and vetoed by one of our BDFLs on #django-social. Here are the relevant logs, edited for readability.

2012-01-20 21:18:10 <jacobkm> (...) we've explicitly decided *not* to include (...) JSONField

2012-01-20 21:21:58 <jacobkm> (...) I'm not saying there's no good reason for it, I'm saying the bad reasons outweigh the good and that's why it shouldn't ship with Django.

2012-01-20 21:22:11 <jacobkm> Under the Don't Give Users Loaded Guns Aimed At Feet principle.

[gabewb@…]

It is pretty frustrating to google around for this, finally land here, and see that it's been vetoed by the principle "Don't Give Users Loaded Guns Aimed At Feet." That is weak principle. When it prevents people from using guns in dangerous ways, great, fine. But sometimes people really need to use a gun pointed in the downward direction, and if you don't give it to them, they do exactly what they did --- they make their own guns, and those guns all have their own sights, triggers, and safety mechanisms, and many of them don't work quite right or are liable to jam or backfire, and a newbie has neither the time nor the ability to pick the right one.

In this case, it's a lot better to include JSONField with a warning label than to keep it out entirely.

Incidentally, I found my way here because JSONField was the default custom field listed in ​https://docs.djangoproject.com/en/dev/ref/models/fields/

[Luke Plant]

The WONTFIX will remain unless you bring this up on Django-devs.

However, one further reason why it is a bad idea is the existence of hstore in Postgres.

This is a database specific solution, and is a much better solution than storing JSON in a text field for many applications. Providing a built-in JSONField discourages people from finding better solutions.

Also Googling "django JSONField" gives you several solutions that are easily installable using PyPI, and work fine for most people, so I don't see how it is an issue. It's extremely easy to get hold of a working JSONField if you want one.

[Anssi Kääriäinen]

For the record I think there are many valid use cases for storing JSON in the database. Also, PostgreSQL 9.2 already has JSON typed field, 9.3 might make it nicely searchable. If the JSON improvements get into 9.3 then it would actually be a better solution than HStore.

But, we likely don't want this field in core. If we are restricted to the functionality offered by all databases, then it basically boils down to "convert to text on save, convert to python on load". Which isn't all that great.

My vote is on improving core support for custom fields. If one can write a json field that just works outside of core, then "pip install django-jsonfield" seems like a good way to add this feature...

[anonymous]

How disappointing. In my case, I want to store user-specific settings provided by the UI guys which are used purely for layout, styles, etc... I don't want to have to maintain normalised tables for a constantly-updating data structure which is only ever stored/retrieved as a whole.

I keep finding this in Django (and Python in general) - great ideas get vetoed because "we don't dare trust a developer to use tools properly".

[tom.gruner@…]

If you "Don't Give Users Loaded Guns Aimed At Feet ", how do you ever expect them to dance ( ;

[Zach Borboa]

JSONField for Postgres is now available in Django 1.9

​https://docs.djangoproject.com/en/1.9/ref/contrib/postgres/fields/#jsonfield

[Tim Graham]

Reopening as per ticket:29821#comment:1.

[Simon Charette]

I believe the next step is to come up with a compatibility table of all supported database types and which lookups we want to support.

[Sage Abdullah]

I will be working on this ticket during Google Summer of Code 2019.

[Sage Abdullah]

​PR.

Please note that it's still a draft and I haven't implemented custom lookups and transforms. More info on the PR's description.

[Sage Abdullah]

The implementation is pretty much complete now, and docs have also been added.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 6f82df6:

Refs #12990 -- Moved CheckFieldDefaultMixin to the django.db.models.fields.mixins.

[Sage Abdullah]

​PR.

Moved to a fresh one.

[GitHub <noreply@…>]

In 5c24c16e:

Refs #12990 -- Moved PostgresSimpleLookup to the django.db.models.lookups.PostgresOperatorLookup.

[Mariusz Felisiak <felisiak.mariusz@…>]

In f97f71f5:

Refs #12990 -- Bumped mysqlclient requirement to >= 1.4.0.

MySQLdb.constants.FIELD_TYPE.JSON was added in mysqlclient 1.4.0rc2.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 6789ded0:

Fixed #12990, Refs #27694 -- Added JSONField model field.

Thanks to Adam Johnson, Carlton Gibson, Mariusz Felisiak, and Raphael
Michel for mentoring this Google Summer of Code 2019 project and
everyone else who helped with the patch.

Special thanks to Mads Jensen, Nick Pope, and Simon Charette for
extensive reviews.

Co-authored-by: Mariusz Felisiak <felisiak.mariusz@…>

[GitHub <noreply@…>]

In f59a2b73:

Refs #12990 -- Added DatabaseFeatures.has_json_operators.

CockroachDB also has them.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 5480fab2:

[3.1.x] Refs #12990 -- Added DatabaseFeatures.has_json_operators.

CockroachDB also has them.
Backport of f59a2b730685fc62c5cb44101f54faf8921d9bc7 from master

[Mariusz Felisiak <felisiak.mariusz@…>]

In 5d4b9c1c:

Refs #12990 -- Added example to JSONField release notes.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 7598ba01:

[3.1.x] Refs #12990 -- Added example to JSONField release notes.

Backport of 5d4b9c1cab03f0d057f0c7751862df0302c65cf9 from master

[Mariusz Felisiak <felisiak.mariusz@…>]

In 7cb5712:

Refs #12990 -- Removed django.contrib.postgres.fields.JSONField per deprecation timeline.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 8fdb5a65:

Refs #12990 -- Removed django.contrib.postgres.fields.jsonb.KeyTransform/KeyTextTransform.

Per deprecation timeline.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 2dd6a83:

Refs #12990 -- Removed django.contrib.postgres.forms.JSONField per deprecation timeline.