Opened 5 years ago

Closed 4 years ago

Last modified 2 years ago

#31221 closed Bug (invalid)

HStoreField returns str instead of dict during tests.

Reported by: Michael Mulholland Owned by:
Component: contrib.postgres Version: dev
Severity: Normal Keywords: hstore
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

It appears that while running tests on my app, the behavior of HStoreField is not consistent. The TestCase I am trying to run is to test a script (which normally runs fine as a management command) that crashes upon trying to use metadata in an HStoreField. Model instances for testing are loaded from a fixture.

While debugging the test, I've found that accessing an HStoreField on a model instance will return a str type instead of the expected dict. What's stranger is that this doesn't occur when the test database already exists and --keepdb is used (the first run with --keepdb will fail, but the second doesn't). This leads me to believe that the way fixtures are loaded during testing is breaking HStoreField's to_python conversion or something.

I found a similar issue #25454, where the same problem was solved, but it looks like the problem is either occurring under different circumstances or has been reintroduced.

My test runner is from this gist, where the test db is created with hstore extension installed: https://gist.github.com/smcoll/bb2533e4b53ae570e11eb2ab011b887b

Currently my knowledge of the django code base isn't extensive enough to suggest a fix, but I have been able to reproduce this problem with the following models/tests:

# models.py
class MyModel(models.Model):
    name = models.CharField(max_length=66)
    metadata = HStoreField(blank=True)
# tests.py
class TestMyApp(TestCase):
    fixtures = ['i_myapp']
    def test_mymodel(self):
        for mymodel_instance in MyModel.objects.all():
            self.assertEqual(type(mymodel_instance.metadata), dict)
# i_myapp.json
[
    {
        "model": "myapp.mymodel",
        "pk": 4,
        "fields": {
            "name": "First",
            "metadata": "{}"
        }
    },
    {
        "model": "myapp.mymodel",
        "pk": 5,
        "fields": {
            "name": "Second",
            "metadata": "{\"details\": \"this one isn't blank\"}"
        }
    }
]

The test test_mymodel fails with:

AssertionError: <class 'str'> != <class 'dict'>

Change History (8)

comment:1 by Simon Charette, 5 years ago

Did you make sure to add django.contrib.postgres to your INSTALLED_APPS as documented?

The latter is in charge of connecting a signal receiver that registers the necessary psycopg2 adapters on connection creation that turns the string provided by PostgreSQL into a Python dict.

If that's the origin of your issue I suggest we re-purpose this ticket to add system checks to all django.contrib.postgres fields that errors when a field is used but self.model._meta.apps.get('postgres') is missing as I've been bitten by that in the past and remember helping a few folks with it since these fields were introduced.

Last edited 5 years ago by Simon Charette (previous) (diff)

in reply to:  1 comment:2 by Michael Mulholland, 5 years ago

Replying to Simon Charette:

Did you make sure to add django.contrib.postgres to your INSTALLED_APPS as documented?

Yes, that was already in the installed apps when I encountered the problem.

When I debug the test program, it does appear to be running register_type_handlers as expected during database setup. It also seems to be performing the deserialization to a python dict when the fixtures are loaded (using to_python) so at least it's aware of hstore. I'm not sure what it's doing with the models after that though, since there's nothing in the test db during testing, and accessing model instance values from the queryset gives the hstore formatted strings.
So the test data I gave above turns into: [(4, 'First', ''), (5, 'Second', '"details"=>"this one isn\'t blank"')]

Last edited 5 years ago by Michael Mulholland (previous) (diff)

comment:3 by Mariusz Felisiak, 5 years ago

Resolution: needsinfo
Status: newclosed
Summary: HStoreField returns str instead of dict during testsHStoreField returns str instead of dict during tests.
Type: UncategorizedBug
Version: 2.2master

Thanks for this ticket, however it works for me. This can be an issue in your custom TestRunner, is there any reason to not use HStoreExtension() in migrations instead of a custom TestRunner.

in reply to:  3 comment:4 by Michael Mulholland, 5 years ago

Replying to felixxm:

is there any reason to not use HStoreExtension() in migrations instead of a custom TestRunner.

Looks like that was my problem. The migrations for the app I work on were created with makemigrations and the original db user wasn't a superuser so they couldn't run HStoreExtension (the extension was being created externally). Upon closer inspection, the only other operation being run by HStoreExtension is clearing cached hstore oids. Adding get_hstore_oids.cache_clear() to my testrunner fixes the issue, though I'll probably just scrap the testrunner and add HStoreExtension.

comment:5 by Mariusz Felisiak, 5 years ago

Resolution: needsinfoinvalid

Thanks for sharing the results of your investigation.

comment:6 by Ryan C. Schwiebert, 4 years ago

Keywords: hstore added
Resolution: invalid
Status: closednew

This has been happening in a web server of mine. It is not clear at all what causes it to happen, but when starts happening it persists.

Django==2.2.13
python==3.5.2
psycopg2-binary==2.8.3
django-tastypie==0.14.3

The model in question has this field:
entities = HStoreField(null=True, default=None)

The input in question is {"1": None}

The traceback looks like this, triggered by an assertion that checks the type of the field to make sure it is a dict:

returned a value for entities that was not a dict. It returned this: "1"=>NULL.
Jun 22 14:09:22 uwsgi[2563]: Traceback (most recent call last):
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/tastypie/resources.py", line 227, in wrapper
Jun 22 14:09:22 uwsgi[2563]:     response = callback(request, *args, **kwargs)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/tastypie/resources.py", line 476, in dispatch_detail
Jun 22 14:09:22 uwsgi[2563]:     return self.dispatch('detail', request, **kwargs)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/tastypie/resources.py", line 499, in dispatch
Jun 22 14:09:22 uwsgi[2563]:     response = method(request, **kwargs)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/tastypie/resources.py", line 1383, in get_detail
Jun 22 14:09:22 uwsgi[2563]:     obj = self.cached_obj_get(bundle=basic_bundle, **self.remove_api_resource_names(kwargs))
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/tastypie/resources.py", line 1202, in cached_obj_get
Jun 22 14:09:22 uwsgi[2563]:     cached_bundle = self.obj_get(bundle=bundle, **kwargs)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/testpackage/frontend/api.py", line 2997, in obj_get
Jun 22 14:09:22 uwsgi[2563]:     ruleData = self.getData(pk)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/testpackage/frontend/api.py", line 2941, in getData
Jun 22 14:09:22 uwsgi[2563]:     update = buildUpdate(entry, Display)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/eventLogger.py", line 133, in buildUpdate
Jun 22 14:09:22 uwsgi[2563]:     raise TypeError('Wrong entities data type.')

I tested using the ORM in a manage.py shell session that recovering this object, I was able to access the entities attribute and indeed get {"1": None}

Restarting the wsgi server sometimes fixes it.

comment:7 by Simon Charette, 4 years ago

Resolution: invalid
Status: newclosed

Unless you can provide a test project to reproduce the issue there isn't much that can be done on our side as nothing proves that Django is at fault.

Every time a similar issue was reported it was due to misconfiguration issue.

If you can provide a test project that follows the aforementioned documented guidelines and reproduces the issue we'll certainly look into it.

comment:8 by Dylan Young, 2 years ago

FYI for those still encountering this issue, this seems to be connected to the caching in django.contrib.postgres.signals.get_hstore_oids. Basically it's caching the oids from the production DB before it starts running tests, so the registration call when the testing DB is created is registering the wrong OIDs as hstore fields.

Still probably a configuration issue (haven't tracked down the cause yet), but hopefully this helps narrow it down for those coming to this issue.

Looks like it's probably still an issue on latest Django as well: https://github.com/django/django/blob/main/django/contrib/postgres/signals.py

A simple upstream fix might be to cache on more than just the DB alias.

Last edited 2 years ago by Dylan Young (previous) (diff)
Note: See TracTickets for help on using tickets.
Back to Top