HStoreField returns str instead of dict during tests.

[Michael Mulholland]

It appears that while running tests on my app, the behavior of HStoreField is not consistent. The TestCase I am trying to run is to test a script (which normally runs fine as a management command) that crashes upon trying to use metadata in an HStoreField. Model instances for testing are loaded from a fixture.

While debugging the test, I've found that accessing an HStoreField on a model instance will return a str type instead of the expected dict. What's stranger is that this doesn't occur when the test database already exists and --keepdb is used (the first run with --keepdb will fail, but the second doesn't). This leads me to believe that the way fixtures are loaded during testing is breaking HStoreField's to_python conversion or something.

I found a similar issue #25454, where the same problem was solved, but it looks like the problem is either occurring under different circumstances or has been reintroduced.

My test runner is from this gist, where the test db is created with hstore extension installed: ​https://gist.github.com/smcoll/bb2533e4b53ae570e11eb2ab011b887b

Currently my knowledge of the django code base isn't extensive enough to suggest a fix, but I have been able to reproduce this problem with the following models/tests:

# models.py
class MyModel(models.Model):
    name = models.CharField(max_length=66)
    metadata = HStoreField(blank=True)

# tests.py
class TestMyApp(TestCase):
    fixtures = ['i_myapp']
    def test_mymodel(self):
        for mymodel_instance in MyModel.objects.all():
            self.assertEqual(type(mymodel_instance.metadata), dict)

# i_myapp.json
[
    {
        "model": "myapp.mymodel",
        "pk": 4,
        "fields": {
            "name": "First",
            "metadata": "{}"
        }
    },
    {
        "model": "myapp.mymodel",
        "pk": 5,
        "fields": {
            "name": "Second",
            "metadata": "{\"details\": \"this one isn't blank\"}"
        }
    }
]

The test test_mymodel fails with:

AssertionError: <class 'str'> != <class 'dict'>

[Simon Charette]

Did you make sure to add django.contrib.postgres to your INSTALLED_APPS ​as documented?

The latter is in charge of ​connecting a signal receiver that ​registers the necessary psycopg2 adapters on connection creation that ​turns the string provided by PostgreSQL into a Python dict.

If that's the origin of your issue I suggest we re-purpose this ticket to add system checks to all django.contrib.postgres fields that errors when a field is used but self.model._meta.apps.get('postgres') is missing as I've been bitten by that in the best and remember helping a few forks with it since these fields were introduced.

[Michael Mulholland]

Replying to Simon Charette:

Did you make sure to add django.contrib.postgres to your INSTALLED_APPS ​as documented?

Yes, that was already in the installed apps when I encountered the problem.

When I debug the test program, it does appear to be running register_type_handlers as expected during database setup. It also seems to be performing the deserialization to a python dict when the fixtures are loaded (using to_python) so at least it's aware of hstore. I'm not sure what it's doing with the models after that though, since there's nothing in the test db during testing, and accessing model instance values from the queryset gives the hstore formatted strings.
So the test data I gave above turns into: [(4, 'First', ''), (5, 'Second', '"details"=>"this one isn\'t blank"')]

[Mariusz Felisiak]

Thanks for this ticket, however it works for me. This can be an issue in your custom TestRunner, is there any reason to not use HStoreExtension() in migrations instead of a custom TestRunner.

[Michael Mulholland]

Replying to felixxm:

is there any reason to not use HStoreExtension() in migrations instead of a custom TestRunner.

Looks like that was my problem. The migrations for the app I work on were created with makemigrations and the original db user wasn't a superuser so they couldn't run HStoreExtension (the extension was being created externally). Upon closer inspection, the only other operation being run by HStoreExtension is clearing cached hstore oids. Adding get_hstore_oids.cache_clear() to my testrunner fixes the issue, though I'll probably just scrap the testrunner and add HStoreExtension.

[Mariusz Felisiak]

Thanks for sharing the results of your investigation.

[Ryan C. Schwiebert]

This has been happening in a web server of mine. It is not clear at all what causes it to happen, but when starts happening it persists.

Django==2.2.13
python==3.5.2
psycopg2-binary==2.8.3
django-tastypie==0.14.3

The model in question has this field:
entities = HStoreField(null=True, default=None)

The input in question is {"1": None}

The traceback looks like this, triggered by an assertion that checks the type of the field to make sure it is a dict:

returned a value for entities that was not a dict. It returned this: "1"=>NULL.
Jun 22 14:09:22 uwsgi[2563]: Traceback (most recent call last):
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/tastypie/resources.py", line 227, in wrapper
Jun 22 14:09:22 uwsgi[2563]:     response = callback(request, *args, **kwargs)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/tastypie/resources.py", line 476, in dispatch_detail
Jun 22 14:09:22 uwsgi[2563]:     return self.dispatch('detail', request, **kwargs)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/tastypie/resources.py", line 499, in dispatch
Jun 22 14:09:22 uwsgi[2563]:     response = method(request, **kwargs)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/tastypie/resources.py", line 1383, in get_detail
Jun 22 14:09:22 uwsgi[2563]:     obj = self.cached_obj_get(bundle=basic_bundle, **self.remove_api_resource_names(kwargs))
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/tastypie/resources.py", line 1202, in cached_obj_get
Jun 22 14:09:22 uwsgi[2563]:     cached_bundle = self.obj_get(bundle=bundle, **kwargs)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/testpackage/frontend/api.py", line 2997, in obj_get
Jun 22 14:09:22 uwsgi[2563]:     ruleData = self.getData(pk)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/testpackage/frontend/api.py", line 2941, in getData
Jun 22 14:09:22 uwsgi[2563]:     update = buildUpdate(entry, Display)
Jun 22 14:09:22 uwsgi[2563]:   File "/opt/venvs/test/lib/python3.5/site-packages/eventLogger.py", line 133, in buildUpdate
Jun 22 14:09:22 uwsgi[2563]:     raise TypeError('Wrong entities data type.')

I tested using the ORM in a manage.py shell session that recovering this object, I was able to access the entities attribute and indeed get {"1": None}

Restarting the wsgi server sometimes fixes it.

[Simon Charette]

Unless you can provide a test project to reproduce the issue there isn't much that can be done on our side as nothing proves that Django is at fault.

Every time a similar issue was reported it was due to misconfiguration issue.

If you can provide a test project that follows the aforementioned documented guidelines and reproduces the issue we'll certainly look into it.

[Dylan Young]

FYI for those still encountering this issue, this seems to be connected to the caching in django.contrib.postgres.signals.get_hstore_oids. Basically it's caching the oids from the production DB before it starts running tests, so the registration call when the testing DB is created is registering the wrong OIDs as hstore fields.

Still probably a configuration issue (haven't tracked down the cause yet), but hopefully this helps narrow it down for those coming to this issue.

Looks like it's probably still an issue on latest Django as well: ​https://github.com/django/django/blob/main/django/contrib/postgres/signals.py

A simple upstream fix might be to cache on more than just the DB alias.