DjangoJSONEncoder no longer supports simplejson

[Keryn Knight <django@…>]

For better or worse, DjangoJSONEncoder, despite living in the core.serializers package (and to my knowledge not being public API), is useful to support the few extra datatypes in standard json usage, and doesn't have any negative side affects. So it gets used.

Previously, in Django 1.4, core.serializers.json relied upon simplejson, but starting with 1.5 only used json.

Since then, simplejson has moved forward and has a slightly different API that prevents using DjangoJSONEncoder, because it subclasses json.JSONEncoder rather than simplejson.JSONEncoder, thus this worked in 1.4:

import simplejson as json
from django.core.serializers.json import DjangoJSONEncoder
json.dumps("{}", cls=DjangoJSONEncoder)

but in 1.5+, it yields a TypeError, with simplejson==3.5.2:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/path/to/python2.7/site-packages/simplejson/__init__.py", line 382, in dumps
    **kw).encode(obj)
TypeError: __init__() got an unexpected keyword argument 'namedtuple_as_object'

I think it ought to be possible to decouple the implementation such that people wanting to continue using simplejson could still benefit from the implementation of default, by doing something like (untested, pseudo-code):

class RichEncoder(object):
    def default(self, o):
        ...
        return (RichEncoder, self).default(o)

class DjangoJSONEncoder(RichEncoder, json.JSONEncoder): pass
DateTimeAwareJSONEncoder = DjangoJSONEncoder

meanwhile, userland implementations would replace the json.JSONEncoder part with simplejson.JSONEncoder, I guess.

This came up and bit someone in the IRC channel, and I'm reporting it because I think it's supportable, though it'd benefit me nought.

[Aymeric Augustin]

See #18023 for why this is hard.

If I remember correctly, the short version is that simplejson broke both backwards and forwards compatibility with itself, making it impossible to be compatible with arbitrary versions of simplejson.

[Keryn Knight <django@…>]

Perhaps I'm not understanding the nuances of the problem (it wouldn't be the first time), but I'm not convinced that solving this particular problem is difficult (periphery around it, perhaps)
The following code-bomb works for me using simplejson versions (the last PATCH releases of every MINOR version):

• 2.1.6
• 2.6.2
• 3.0.9
• 3.3.3
• 3.4.1
• 3.5.2

and the json module in Python 2.7.6 (I'm guessing/hoping the least Py3K compatible part of this is using xrange, and print as a keyword)

import simplejson
import json
import decimal
import datetime


# So this is taken straight out of the DjangoJSONEncoder implementation
# in master as of 7548aa8ffd46eb6e0f73730d1b2eb515ba581f95
class RichEncoder(object):
    def default(self, o):
        # See "Date Time String Format" in the ECMA-262 specification.
        if isinstance(o, datetime.datetime):
            r = o.isoformat()
            if o.microsecond:
                r = r[:23] + r[26:]
            if r.endswith('+00:00'):
                r = r[:-6] + 'Z'
            return r
        elif isinstance(o, datetime.date):
            return o.isoformat()
        elif isinstance(o, datetime.time):
            if is_aware(o):
                raise ValueError("JSON can't represent timezone-aware times.")
            r = o.isoformat()
            if o.microsecond:
                r = r[:12]
            return r
        elif isinstance(o, decimal.Decimal):
            return str(o)
        else:
            return super(RichEncoder, self).default(o)

# this line is what would end up as DjangoJSONEncoder
# and also aliased as DateTimeAwareJSONEncoder
class stdlibJSONEncoder(RichEncoder, json.JSONEncoder): pass

# this would by necessity be a userland implementation, but is obviously less
# code to copy-paste & maintain.
class simpleJSONEncoder(RichEncoder, simplejson.JSONEncoder): pass


data = {'decimal': decimal.Decimal('4.011'), 'datetime': datetime.datetime.today()}

print "Naive:"
# naive, stdlib
try:
    json.dumps(data)
except TypeError as e:
    print e
# naive, simplejson
try:
    simplejson.dumps(data)
except TypeError as e:
    print e

print "\n\nEncoding from Django:"
print json.dumps(data, cls=stdlibJSONEncoder)
print simplejson.dumps(data, cls=simpleJSONEncoder)
# allow decimals to be strings
print simplejson.dumps(data, cls=simpleJSONEncoder, use_decimal=False)


class CustomJSONEncoder(object):
    def default(self, o):
        # contrived example ...
        if hasattr(o, '__iter__'):
            return tuple(o)
        return super(CustomJSONEncoder, self).default(o)

# so you need to encode *other* things, but you want the date/decimal handling
# both of these would obviously be userland implementations
class stdlibCustomJSONEncoder(CustomJSONEncoder, stdlibJSONEncoder): pass
class simpleCustomJSONEncoder(CustomJSONEncoder, simpleJSONEncoder): pass
# long-winded version.
class simpleCustomJSONEncoder2(CustomJSONEncoder, RichEncoder, simplejson.JSONEncoder): pass

more_data = {'xrange': xrange(0, 10), 'datetime': datetime.datetime.today()}

print "\n\nEncoding from Django, + extra stuff:"
print json.dumps(more_data, cls=stdlibCustomJSONEncoder)
print simplejson.dumps(more_data, cls=simpleCustomJSONEncoder)
# long-winded version, as above
print simplejson.dumps(more_data, cls=simpleCustomJSONEncoder2)

print "\n\nWithout our custom encoder:"
last_data = {'xrange': xrange(0, 10)}
try:
    json.dumps(last_data)
except TypeError as e:
    print e
try:
    simplejson.dumps(last_data)
except TypeError as e:
    print e

None of these exhibit the namedtuple_as_object issue presented in the ticket itself, because their encoders match their expectations.

If simplejson is still buggy, 30-40 releases later, as the mentioned ticket indicated, that's fine; that's for an end-user to deal with, or for simplejson to handle in future releases, but as long as the contract for a JSONEncoder instance's default is just accept o and return a serializable version of it, I'd think the above would continue to work.

By separating the custom default object introspection out of the encoder itself, it at least grants the possibility of users getting the best of both Django's code and simplejson's, if that is their desire. The danger of passing a json.JSONEncoder to simplejson.dumps() as that ticket describes, is then circumventable, though left as an exercise to the user.

[Tim Graham]

If this broke in 1.5 and hasn't been raised since then, I am thinking it may not be worth it. I'm also not sure Django should get further into the serializer business.

Consider the the maintenance burden of adding simplejson to the test dependencies. Also the latest version doesn't support Python 3.2 so we'd have to skip the tests there.