Add more sophisticated serialization support to JSONField

[Jim Graham]

It would be great if the jsonb field could support serialization of objects to the JSON field. The DjangoJSONEncoder would be helpful here for datetimes, Decimals etc that are in JSON blobs.

I've worked on a similar serializer if it would help to come up with a pull request.

[Tim Graham]

I think the problem here is that we couldn't also deserialize that data since we have no way to know if the user originally passed a string or some other Python type. For example, if we see '1' in the JSONField, do we deserialize it to 1 (integer) or leave it as a string? Using DjangoJSONEncoder for model serialization is different because we have the model field type hint to use for deserialization.

[Jim Graham]

I think serializing would be be optional at the model level, and the user could decide how they wanted objects serialized/deserialize. The default would continue to be no serialization.

Is it worthwhile coming up with a PR for comments, or is this a non-starter?

Thanks.

[Tim Graham]

Could you be a bit more explicit about what the behavior would look like, e.g. with some code examples?

[Marc Tamlyn]

Such a field would actually be quite different in implementation to the current field. We currently lever psycopg2's serialization and while this can be customised[1], it is most easily customisable at the field level. There is a possibility of providing an alternative to Json() for adaption (into the db), but coming back out can only be done at the connection level.

A JSON field which does all of its de-/serialization in the field would be possible of course, one such implementation is ​https://github.com/bradjasper/django-jsonfield/

[1] ​http://initd.org/psycopg/docs/extras.html#json-adaptation

[Tim Graham]

Do you think there are any action items for this ticket then? If not, perhaps we could document the restriction and note that alternative fields exist.

[Jim Graham]

Based on what you've pointed out, I think pointing to the external projects is the best thing to do.

Thanks for your time in reviewing this.

Regards.

[Tim Graham]

​Doc patch. I avoided recommending a particular third-party field since we avoid that as it's difficult to curate a list of packages that doesn't go stale.

[Aymeric Augustin]

I hit the same problem: ​https://groups.google.com/d/msg/django-developers/upg9pgGvaUs/LMN8Xfq8EQAJ

If Django doesn't provide an escape hatch, I suspect I won't be the only one to resort to monkey-patching.

Right now I don't have time to dig into this issue but I don't think we can leave it there altogether.

[Tim Graham <timograham@…>]

In c432dd40:

Refs #25995 -- Documented that JSONField doesn't handle sophisticated serialization.

[Tim Graham <timograham@…>]

In 3503b81:

[1.9.x] Refs #25995 -- Documented that JSONField doesn't handle sophisticated serialization.

Backport of c432dd40bde37667bfe6bb59eaff0a14c50cd27b from master

[Debanshu Kundu]

We could use a custom data type representation scheme to represent dates, decimals etc in JSON.

The scheme could be similar to what MongoDB uses for it's extended JSON (​https://docs.mongodb.com/manual/reference/mongodb-extended-json/#bson-data-types-and-associated-representations).
Eg: a date object would be stored as {"$date": <ISO-8601 date string>}

Or,

It could be something more sophisticated and generic like

{
    "$djExtJson": {
        "$type": <type string>, 
        "$args": <args array>, 
        "$kwargs": <kwargs object> 
    }
}

Where '$type' would be the full path of a class (eg, datetime.date or decimal.Decimal ) and '$args' and '$kwargs' would be the arguments for constructor of the class.

So a date object would be stored as:

{"$djExtJson": {"$type": "datetime.date", "$args": ["2016", "7", "6"]}} 

and a decimal object as:

{"$djExtJson": {"$type": "decimal.Decimal", "$args": ["21.5"]}} 

Using this format, one would be able to store any class's object in JSON. Eg:

# my_utils.py
class ABC():
    def __init__(self, dt: date):
        self.date = dt

An object of above class could be stored as:

{"$djExtJson": {
    "$type": "my_utils.ABC", 
    "$args": [{"$djExtJson": {"$type": "datetime.date", "$args": ["2016", "7", "6"]}}]
}} 

Deserialisation of objects using above format would be very simple, as we would have type information stored withing the object with all the arguments required to call constructor of that type, so we would be able to call the constructor and get the object.

For serialisation, it won't be possible to get arguments which need to be passed to a class's constructor just by looking at a object of that class. So for that, we can have custom code in Django for supporting serialisation of common standard library objects and builtins. And for user defined classes, we can created a mixin (we may call it DjExtJsonSerialisable ) which would simply stores arguments (and keyword arguments) passed to the constructor of a user defined class in some attribute, when a object of a class inheriting from at mixin is created (attribute can __dj_ext_json_data__ ).

[Claude Paroz]

This ​PR adds custom encoding support.

The decoding part is another story. The patch currently recommends using from_db/from_db_value hooks. If anyone can come with a concrete proposal with another method, please propose it.

[Berker Peksag]

​PR #7071 looks good to me. I left a comment about the type of the encoding parameter in the pull request.

[Claude Paroz <claude@…>]

In 13c3e5d:

Fixed #25995 -- Added an encoder option to JSONField

Thanks Berker Peksag and Tim Graham for the reviews.