Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#33284 closed Bug (needsinfo)

JSONFIeld decode error with Python 3.9

Reported by: Alan D. Snow Owned by: nobody
Component: Database layer (models, ORM) Version: 3.2
Severity: Normal Keywords: python, json
Cc: Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

https://code.djangoproject.com/ticket/31973

Traceback:

venv/lib/python3.9/site-packages/django/db/models/fields/json.py:83: in from_db_value
    return json.loads(value, cls=self.decoder)

    def loads(s, *, cls=None, object_hook=None, parse_float=None,
            parse_int=None, parse_constant=None, object_pairs_hook=None, **kw):
        """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance
        containing a JSON document) to a Python object.
    
        ``object_hook`` is an optional function that will be called with the
        result of any object literal decode (a ``dict``). The return value of
        ``object_hook`` will be used instead of the ``dict``. This feature
        can be used to implement custom decoders (e.g. JSON-RPC class hinting).
    
        ``object_pairs_hook`` is an optional function that will be called with the
        result of any object literal decoded with an ordered list of pairs.  The
        return value of ``object_pairs_hook`` will be used instead of the ``dict``.
        This feature can be used to implement custom decoders.  If ``object_hook``
        is also defined, the ``object_pairs_hook`` takes priority.
    
        ``parse_float``, if specified, will be called with the string
        of every JSON float to be decoded. By default this is equivalent to
        float(num_str). This can be used to use another datatype or parser
        for JSON floats (e.g. decimal.Decimal).
    
        ``parse_int``, if specified, will be called with the string
        of every JSON int to be decoded. By default this is equivalent to
        int(num_str). This can be used to use another datatype or parser
        for JSON integers (e.g. float).
    
        ``parse_constant``, if specified, will be called with one of the
        following strings: -Infinity, Infinity, NaN.
        This can be used to raise an exception if invalid JSON numbers
        are encountered.
    
        To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
        kwarg; otherwise ``JSONDecoder`` is used.
        """
        if isinstance(s, str):
            if s.startswith('\ufeff'):
                raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
                                      s, 0)
        else:
            if not isinstance(s, (bytes, bytearray)):
>               raise TypeError(f'the JSON object must be str, bytes or bytearray, '
                                f'not {s.__class__.__name__}')
E               TypeError: the JSON object must be str, bytes or bytearray, not list

../../miniconda/envs/midas/lib/python3.9/json/__init__.py:339: TypeError

Suggested fix is to catch TypeError as well as JSONDecodeError here: https://github.com/django/django/blob/adb4100e58d9ea073ee8caa454bb7c885b6a83ed/django/db/models/fields/json.py#L82-L85

        try:
            return json.loads(value, cls=self.decoder)
        except (json.JSONDecodeError, TypeError):
            return value

Change History (11)

comment:1 by Mariusz Felisiak, 3 years ago

Easy pickings: unset
Resolution: duplicate
Status: newclosed

Duplicate of #31973.

comment:2 by Alan D. Snow, 3 years ago

Resolution: duplicate
Status: closednew

This is not a duplicate. It is related to #31973, but a fix has not been applied yet.

comment:3 by Mariusz Felisiak, 3 years ago

Resolution: duplicate
Status: newclosed

#31973 is closed as invalid, so "fix" will not be applied.

comment:4 by Alan D. Snow, 3 years ago

Has patch: set
Resolution: duplicate
Status: closednew

comment:5 by Mariusz Felisiak, 3 years ago

Resolution: duplicate
Status: newclosed

Alan, please stop and don't reopen closed tickets without providing any new details.

comment:6 by Alan D. Snow, 3 years ago

I am confused. What is the resolution?

comment:7 by Alan D. Snow, 3 years ago

Is there something wrong with this?

                                          Table "public.table"
      Column      |           Type           | Collation | Nullable |                           Default                            
------------------+--------------------------+-----------+----------+--------------------------------------------------------------
 id               | integer                  |           | not null | nextval('table_id_seq'::regclass)
 metadata         | jsonb                    |           | not null | 

comment:8 by Alan D. Snow, 3 years ago

When I do:

class Table(models.Model):
    metadata = JSONField(
        blank=True,
        default=dict,
        help_text="Any additional metadata.",
    )

And create a model and query for it:

table = Table.objects.create(metadata={"A": "B"})
found = table.objects.first()

I get an error.
If I set found.metadata = "", and query fort it, the error goes away.

comment:9 by Alan D. Snow, 3 years ago

There is no issue with Python 3.8.

comment:10 by Alan D. Snow, 3 years ago

Current workaround:

class JSONFieldPatch(models.JSONField):
    """
    Patch for Python 3.9 since json returns TypeError
    """

    def from_db_value(self, value, expression, connection):
        try:
            return super().from_db_value(value, expression, connection)
        except TypeError:
            return value


in reply to:  7 comment:11 by Mariusz Felisiak, 3 years ago

Resolution: duplicateneedsinfo

The provided model works for me with Python 3.8 and Python 3.9. I don't think you've explained the issue in enough detail to confirm a bug in Django. Please reopen the ticket if you can debug your issue and provide details about why and where Django is at fault.

Note: See TracTickets for help on using tickets.
Back to Top