Django postgres JSONField encoding

[Hisham Waleed Karam]

i have JSONField in a model this field store json in 'utf8' but when i am trying to get the value of this field it escaped

model field:

details = JSONField(blank=True, default={})

database row:

code :

from stores.models import Item
i=Item.objects.all()[0]
i.details

output in console and django admin: i want django to show details without escaping

{u'\u0633\u064a\u0628': u'\u0634\u0633\u064a'}

[Tim Graham]

Can you provide a test case that demonstrates the issue?

[Tim Graham]

I can't reproduce the issue given the information provided.

[ErikW]

I have the same thing happening to me.

I have a model using a JSONField (from django.contrib.postgres.fields). When I examine the field from an ipython shell (also happening when interacting with the field from views), it is returning the contents as a string instead of a dict (ie '{}'). I checked table in Postgres, and it is in fact a jsonb column. Not sure why it's not deserializing the field.

If I save the model, it then escapes the field again which adds the additional backslashes mentioned by Hisham above.

This behavior only started after upgrading this codebase from Django 1.9 to 1.10.5 -- the field worked correctly in 1.9.x.

    from django.contrib.postgres.fields import JSONField
    
    class CRMOnboarding(models.Model):
        data = JSONField(default=dict)

IPython Shell:

    In 13:58:31 [6]: x.crmonboarding_set.all().last()
    Out 13:58:31 [6]: <CRMOnboarding: CRM - bb69b886-25d5-4d59-a4d6-23bebc91af70>
    
    In 13:58:34 [7]: crm=_
    
    In 13:58:36 [8]: crm.data
    Out 13:58:36 [8]: '{"fields": {"dob": "null", "vin": "null", "city": "city", "email": "email", "phone": "phone", "state": "state", "Domain": "null", "Handle": "null", "Status": "null", "zip_code": "zip_code", "address_1": "address_1", "address_2": "address_2", "auto_fuel": "null", "auto_make": "null", "auto_size": "null", "auto_type": "null", "auto_year": "null", "last_name": "last_name", "auto_based": "null", "auto_class": "null", "auto_model": "null", "auto_style": "null", "first_name": "first_name", "Syntax good": "null", "ahh_num_cars": "null", "Invalid Reason": "null", "auto_make_model": "null", "confidence_code": "null", "Catch All Domain": "null", "auto_manufacturer": "null", "auto_purchase_date": "null", "auto_purchase_type": "null"}}'
    
    In 13:58:54 [9]: data=_
    
    In 13:58:57 [10]: crm.save()
    
    In 13:58:59 [11]: x.crmonboarding_set.all().last()
    Out 13:58:59 [11]: <CRMOnboarding: CRM - bb69b886-25d5-4d59-a4d6-23bebc91af70>
    
    In 13:59:01 [12]: crm=_
    
    In 13:59:02 [13]: crm.data
    Out 13:59:02 [13]: '"{\\"fields\\": {\\"dob\\": \\"null\\", \\"vin\\": \\"null\\", \\"city\\": \\"city\\", \\"email\\": \\"email\\", \\"phone\\": \\"phone\\", \\"state\\": \\"state\\", \\"Domain\\": \\"null\\", \\"Handle\\": \\"null\\", \\"Status\\": \\"null\\", \\"zip_code\\": \\"zip_code\\", \\"address_1\\": \\"address_1\\", \\"address_2\\": \\"address_2\\", \\"auto_fuel\\": \\"null\\", \\"auto_make\\": \\"null\\", \\"auto_size\\": \\"null\\", \\"auto_type\\": \\"null\\", \\"auto_year\\": \\"null\\", \\"last_name\\": \\"last_name\\", \\"auto_based\\": \\"null\\", \\"auto_class\\": \\"null\\", \\"auto_model\\": \\"null\\", \\"auto_style\\": \\"null\\", \\"first_name\\": \\"first_name\\", \\"Syntax good\\": \\"null\\", \\"ahh_num_cars\\": \\"null\\", \\"Invalid Reason\\": \\"null\\", \\"auto_make_model\\": \\"null\\", \\"confidence_code\\": \\"null\\", \\"Catch All Domain\\": \\"null\\", \\"auto_manufacturer\\": \\"null\\", \\"auto_purchase_date\\": \\"null\\", \\"auto_purchase_type\\": \\"null\\"}}"'
    
    In 13:59:26 [14]: import json
    
    In 13:59:29 [15]: json.loads(data)
    Out 13:59:29 [15]:
    {u'fields': {u'Catch All Domain': u'null',
      u'Domain': u'null',
      u'Handle': u'null',
      u'Invalid Reason': u'null',
      u'Status': u'null',
      u'Syntax good': u'null',
      u'address_1': u'address_1',
      u'address_2': u'address_2',
      u'ahh_num_cars': u'null',
      u'auto_based': u'null',
      u'auto_class': u'null',
      u'auto_fuel': u'null',
      u'auto_make': u'null',
      u'auto_make_model': u'null',
      u'auto_manufacturer': u'null',
      u'auto_model': u'null',
      u'auto_purchase_date': u'null',
      u'auto_purchase_type': u'null',
      u'auto_size': u'null',
      u'auto_style': u'null',
      u'auto_type': u'null',
      u'auto_year': u'null',
      u'city': u'city',
      u'confidence_code': u'null',
      u'dob': u'null',
      u'email': u'email',
      u'first_name': u'first_name',
      u'last_name': u'last_name',
      u'phone': u'phone',
      u'state': u'state',
      u'vin': u'null',
      u'zip_code': u'zip_code'}}
    
    In 13:59:33 [16]: crm.data
    Out 13:59:33 [16]: '"{\\"fields\\": {\\"dob\\": \\"null\\", \\"vin\\": \\"null\\", \\"city\\": \\"city\\", \\"email\\": \\"email\\", \\"phone\\": \\"phone\\", \\"state\\": \\"state\\", \\"Domain\\": \\"null\\", \\"Handle\\": \\"null\\", \\"Status\\": \\"null\\", \\"zip_code\\": \\"zip_code\\", \\"address_1\\": \\"address_1\\", \\"address_2\\": \\"address_2\\", \\"auto_fuel\\": \\"null\\", \\"auto_make\\": \\"null\\", \\"auto_size\\": \\"null\\", \\"auto_type\\": \\"null\\", \\"auto_year\\": \\"null\\", \\"last_name\\": \\"last_name\\", \\"auto_based\\": \\"null\\", \\"auto_class\\": \\"null\\", \\"auto_model\\": \\"null\\", \\"auto_style\\": \\"null\\", \\"first_name\\": \\"first_name\\", \\"Syntax good\\": \\"null\\", \\"ahh_num_cars\\": \\"null\\", \\"Invalid Reason\\": \\"null\\", \\"auto_make_model\\": \\"null\\", \\"confidence_code\\": \\"null\\", \\"Catch All Domain\\": \\"null\\", \\"auto_manufacturer\\": \\"null\\", \\"auto_purchase_date\\": \\"null\\", \\"auto_purchase_type\\": \\"null\\"}}"'
    
    In 13:59:38 [17]: crm.data = json.loads(data)
    
    In 13:59:42 [19]: crm.data
    Out 13:59:42 [19]:
    {u'fields': {u'Catch All Domain': u'null',
      u'Domain': u'null',
      u'Handle': u'null',
      u'Invalid Reason': u'null',
      u'Status': u'null',
      u'Syntax good': u'null',
      u'address_1': u'address_1',
      u'address_2': u'address_2',
      u'ahh_num_cars': u'null',
      u'auto_based': u'null',
      u'auto_class': u'null',
      u'auto_fuel': u'null',
      u'auto_make': u'null',
      u'auto_make_model': u'null',
      u'auto_manufacturer': u'null',
      u'auto_model': u'null',
      u'auto_purchase_date': u'null',
      u'auto_purchase_type': u'null',
      u'auto_size': u'null',
      u'auto_style': u'null',
      u'auto_type': u'null',
      u'auto_year': u'null',
      u'city': u'city',
      u'confidence_code': u'null',
      u'dob': u'null',
      u'email': u'email',
      u'first_name': u'first_name',
      u'last_name': u'last_name',
      u'phone': u'phone',
      u'state': u'state',
      u'vin': u'null',
      u'zip_code': u'zip_code'}}
    
    In 13:59:46 [20]: crm.save()
    
    In 13:59:48 [21]: x.crmonboarding_set.all().last()
    Out 13:59:48 [21]: <CRMOnboarding: CRM - bb69b886-25d5-4d59-a4d6-23bebc91af70>
    
    In 13:59:50 [22]: crm=_
    
    In 13:59:52 [23]: crm.data
    Out 13:59:52 [23]: '{"fields": {"dob": "null", "vin": "null", "city": "city", "email": "email", "phone": "phone", "state": "state", "Domain": "null", "Handle": "null", "Status": "null", "zip_code": "zip_code", "address_1": "address_1", "address_2": "address_2", "auto_fuel": "null", "auto_make": "null", "auto_size": "null", "auto_type": "null", "auto_year": "null", "last_name": "last_name", "auto_based": "null", "auto_class": "null", "auto_model": "null", "auto_style": "null", "first_name": "first_name", "Syntax good": "null", "ahh_num_cars": "null", "Invalid Reason": "null", "auto_make_model": "null", "confidence_code": "null", "Catch All Domain": "null", "auto_manufacturer": "null", "auto_purchase_date": "null", "auto_purchase_type": "null"}}'

[Tim Graham]

Could you please provide a test case for ​Django's test suite?

[Tim Graham]

In particular, it's unclear why crm.data is coming back as a string rather than as a dict. Is it possible your data is corrupted?

[grapemix]

I got the same problem too. I am very sure the bug is not caused by data corruption. I double check with the db table type and the raw data in postgres db. The interesting thing is the data return dict type in django-admin.py shell environment, but the data will return string in the runserver and testcase environment.

Another experiment I did is that converting the string data to dict by json.loads() and save to db in the runserver environment. The data type of the JSONField is dict, but if I fetch the model again, the data type of the JSONField become string.

So please reopen the ticket. This bug gives us lots of trouble.

Thanks.

[Tim Graham]

Steps to reproduce the issue are missing. You can reopen the ticket if you provide a test case to reproduce the issue.

[grapemix]

After debugging like crazy for a few days, we got some good news.

Short story:
Django's JSONField cannot be mixed with django-jsonfield's JSONField to use. See: ​https://bitbucket.org/schinckel/django-jsonfield/issues/57/cannot-use-in-the-same-project-as-djangos, not even just in migration files. After removing all migrations file and run the make migrations cmd, everything is working.

Long story:
We had tried running Django's std postgres testcase and our testcase seperately and mix and matched to try to reproduce the problem. Since we used django-jsonfield's JSONField before and we would like to switch to Django's JSONField at once, we only keep django-jsonfield's JSONField at migrations file to 'transit'. That's why the test result is so in-consistence. After removing all migrations file and run the make migrations cmd, everything is working.

Since we only tested on our dev environment, we are not sure the transition in the prod. If we have not screamed in the following days, that means the transition in the prod is good.
Thanks everyone for your time.