#34849 closed Bug (fixed)
Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.
| Reported by: | Mariusz Felisiak | Owned by: | David Sanders |
|---|---|---|---|
| Component: | Database layer (models, ORM) | Version: | 5.0 |
| Severity: | Release blocker | Keywords: | |
| Cc: | Adam Johnson, Florian Zimmermann | Triage Stage: | Ready for checkin |
| Has patch: | yes | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description
Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes with:
RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
Check out:
./runtests.py postgres_tests
...
======================================================================
ERROR: test_register_serializer_for_migrations (postgres_tests.test_apps.PostgresConfigTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/local/lib/python3.10/unittest/case.py", line 591, in run
self._callTestMethod(testMethod)
File "/usr/local/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
method()
File "/django/tests/postgres_tests/test_apps.py", line 64, in test_register_serializer_for_migrations
with self.modify_settings(INSTALLED_APPS={"append": "django.contrib.postgres"}):
File "/django/django/test/utils.py", line 405, in __enter__
return self.enable()
File "/django/django/test/utils.py", line 587, in enable
super().enable()
File "/django/django/test/utils.py", line 474, in enable
apps.set_installed_apps(self.options["INSTALLED_APPS"])
File "/django/django/apps/registry.py", line 362, in set_installed_apps
self.populate(installed)
File "/django/django/apps/registry.py", line 124, in populate
app_config.ready()
File "/django/django/contrib/postgres/apps.py", line 71, in ready
register_type_handlers(conn)
File "/django/django/contrib/postgres/signals.py", line 40, in register_type_handlers
oids, array_oids = get_hstore_oids(connection.alias)
File "/django/django/contrib/postgres/signals.py", line 24, in get_hstore_oids
return get_type_oids(connection_alias, "hstore")
File "/django/django/contrib/postgres/signals.py", line 10, in get_type_oids
cursor.execute(
File "/django/django/db/backends/utils.py", line 77, in execute
return self._execute_with_wrappers(
File "/django/django/db/backends/utils.py", line 90, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/django/django/db/backends/utils.py", line 94, in _execute
warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
======================================================================
ERROR: test_register_type_handlers_connection (postgres_tests.test_apps.PostgresConfigTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/local/lib/python3.10/unittest/case.py", line 591, in run
self._callTestMethod(testMethod)
File "/usr/local/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
method()
File "/django/tests/postgres_tests/test_apps.py", line 36, in test_register_type_handlers_connection
with modify_settings(INSTALLED_APPS={"append": "django.contrib.postgres"}):
File "/django/django/test/utils.py", line 405, in __enter__
return self.enable()
File "/django/django/test/utils.py", line 587, in enable
super().enable()
File "/django/django/test/utils.py", line 474, in enable
apps.set_installed_apps(self.options["INSTALLED_APPS"])
File "/django/django/apps/registry.py", line 362, in set_installed_apps
self.populate(installed)
File "/django/django/apps/registry.py", line 124, in populate
app_config.ready()
File "/django/django/contrib/postgres/apps.py", line 71, in ready
register_type_handlers(conn)
File "/django/django/contrib/postgres/signals.py", line 40, in register_type_handlers
oids, array_oids = get_hstore_oids(connection.alias)
File "/django/django/contrib/postgres/signals.py", line 24, in get_hstore_oids
return get_type_oids(connection_alias, "hstore")
File "/django/django/contrib/postgres/signals.py", line 10, in get_type_oids
cursor.execute(
File "/django/django/db/backends/utils.py", line 77, in execute
return self._execute_with_wrappers(
File "/django/django/db/backends/utils.py", line 90, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/django/django/db/backends/utils.py", line 94, in _execute
warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
----------------------------------------------------------------------
Ran 590 tests in 1.840s
FAILED (errors=2, skipped=8, expected failures=3)
@modify_settings repopulates apps that register type handlers and hits the database to select OIDs (if not cached).
Regression in fbd16438f46bc2128926958ad24331da5d1b406f (#33143).
Change History (12)
comment:1 by , 14 months ago
| Triage Stage: | Unreviewed → Accepted |
|---|
comment:2 by , 14 months ago
comment:3 by , 14 months ago
Shouldn't Jenkins have picked this up earlier, ...
No, it shouldn't. First of all we run the entire test suite not random subsets of test classes. Secondly, this is quite tricky and you have to debug how types are registered for django.contrib.postgresql and how @modify_settings works:
- types are registered when connection is created by
connection_createdhandler, so it's not when app is initialized, so far so good, - when
@modify_settingsmanipulatesINSTALLED_APPSthen we undo effects ofPostgresConfig.ready(), @modify_settingscallsPostgresConfig.ready()again, tries to registered handlers for existing connections and hits the database to select OIDs.
comment:4 by , 14 months ago
Maybe worth noting: for me this depends on the --parallel option with --parallel 1 producing even more errors:
--parallel 2 no errors:
> python runtests.py --settings test_postgres --parallel 2 postgres_tests Found 590 test(s). ..........................................................................................................................x......x............x............................................................................................................................................................................................................................................................................................................................................................................................................................................................... ---------------------------------------------------------------------- Ran 590 tests in 7.974s OK (expected failures=3)
--parallel 4 two errors:
> python runtests.py --settings test_postgres --parallel 4 postgres_tests
Found 590 test(s).
EE........................................................................................................................x......x............x...............................................................................................................................................................................................................................................................................................................................................................................................................................................................
======================================================================
ERROR: test_register_serializer_for_migrations (postgres_tests.test_apps.PostgresConfigTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\unittest\case.py", line 59, in testPartExecutor
yield
File "C:\Program Files\Python310\lib\unittest\case.py", line 591, in run
self._callTestMethod(testMethod)
File "C:\Program Files\Python310\lib\unittest\case.py", line 549, in _callTestMethod
method()
File "D:\django\tests\postgres_tests\test_apps.py", line 64, in test_register_serializer_for_migrations
with self.modify_settings(INSTALLED_APPS={"append": "django.contrib.postgres"}):
File "D:\django\django\test\utils.py", line 405, in __enter__
return self.enable()
File "D:\django\django\test\utils.py", line 587, in enable
super().enable()
File "D:\django\django\test\utils.py", line 474, in enable
apps.set_installed_apps(self.options["INSTALLED_APPS"])
File "D:\django\django\apps\registry.py", line 362, in set_installed_apps
self.populate(installed)
File "D:\django\django\apps\registry.py", line 124, in populate
app_config.ready()
File "D:\django\django\contrib\postgres\apps.py", line 71, in ready
register_type_handlers(conn)
File "D:\django\django\contrib\postgres\signals.py", line 40, in register_type_handlers
oids, array_oids = get_hstore_oids(connection.alias)
File "D:\django\django\contrib\postgres\signals.py", line 24, in get_hstore_oids
return get_type_oids(connection_alias, "hstore")
File "D:\django\django\contrib\postgres\signals.py", line 10, in get_type_oids
cursor.execute(
File "D:\django\django\db\backends\utils.py", line 77, in execute
return self._execute_with_wrappers(
File "D:\django\django\db\backends\utils.py", line 90, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\django\django\db\backends\utils.py", line 94, in _execute
warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
======================================================================
ERROR: test_register_type_handlers_connection (postgres_tests.test_apps.PostgresConfigTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\unittest\case.py", line 59, in testPartExecutor
yield
File "C:\Program Files\Python310\lib\unittest\case.py", line 591, in run
self._callTestMethod(testMethod)
File "C:\Program Files\Python310\lib\unittest\case.py", line 549, in _callTestMethod
method()
File "D:\django\tests\postgres_tests\test_apps.py", line 36, in test_register_type_handlers_connection
with modify_settings(INSTALLED_APPS={"append": "django.contrib.postgres"}):
File "D:\django\django\test\utils.py", line 405, in __enter__
return self.enable()
File "D:\django\django\test\utils.py", line 587, in enable
super().enable()
File "D:\django\django\test\utils.py", line 474, in enable
apps.set_installed_apps(self.options["INSTALLED_APPS"])
File "D:\django\django\apps\registry.py", line 362, in set_installed_apps
self.populate(installed)
File "D:\django\django\apps\registry.py", line 124, in populate
app_config.ready()
File "D:\django\django\contrib\postgres\apps.py", line 71, in ready
register_type_handlers(conn)
File "D:\django\django\contrib\postgres\signals.py", line 40, in register_type_handlers
oids, array_oids = get_hstore_oids(connection.alias)
File "D:\django\django\contrib\postgres\signals.py", line 24, in get_hstore_oids
return get_type_oids(connection_alias, "hstore")
File "D:\django\django\contrib\postgres\signals.py", line 10, in get_type_oids
cursor.execute(
File "D:\django\django\db\backends\utils.py", line 77, in execute
return self._execute_with_wrappers(
File "D:\django\django\db\backends\utils.py", line 90, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\django\django\db\backends\utils.py", line 94, in _execute
warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
----------------------------------------------------------------------
Ran 590 tests in 4.268s
FAILED (errors=2, expected failures=3)
--parallel 1 five errors:
> python runtests.py --settings test_postgres --parallel 1 postgres_tests
Found 590 test(s).
EEEEE...................................x......x............x............................................................................................................................................................................................................................................................................................................................................................................................................................................................................
======================================================================
ERROR: setUpClass (postgres_tests.test_aggregates.TestAggregateDistinct)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\django\django\test\testcases.py", line 1250, in setUpClass
super().setUpClass()
File "D:\django\django\test\testcases.py", line 197, in setUpClass
cls._cls_modified_context.enable()
File "D:\django\django\test\utils.py", line 587, in enable
super().enable()
File "D:\django\django\test\utils.py", line 474, in enable
apps.set_installed_apps(self.options["INSTALLED_APPS"])
File "D:\django\django\apps\registry.py", line 362, in set_installed_apps
self.populate(installed)
File "D:\django\django\apps\registry.py", line 124, in populate
app_config.ready()
File "D:\django\django\contrib\postgres\apps.py", line 71, in ready
register_type_handlers(conn)
File "D:\django\django\contrib\postgres\signals.py", line 40, in register_type_handlers
oids, array_oids = get_hstore_oids(connection.alias)
File "D:\django\django\contrib\postgres\signals.py", line 24, in get_hstore_oids
return get_type_oids(connection_alias, "hstore")
File "D:\django\django\contrib\postgres\signals.py", line 10, in get_type_oids
cursor.execute(
File "D:\django\django\db\backends\utils.py", line 77, in execute
return self._execute_with_wrappers(
File "D:\django\django\db\backends\utils.py", line 90, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\django\django\db\backends\utils.py", line 94, in _execute
warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
======================================================================
ERROR: setUpClass (postgres_tests.test_aggregates.TestGeneralAggregate)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\django\django\test\testcases.py", line 1250, in setUpClass
super().setUpClass()
File "D:\django\django\test\testcases.py", line 197, in setUpClass
cls._cls_modified_context.enable()
File "D:\django\django\test\utils.py", line 587, in enable
super().enable()
File "D:\django\django\test\utils.py", line 474, in enable
apps.set_installed_apps(self.options["INSTALLED_APPS"])
File "D:\django\django\apps\registry.py", line 362, in set_installed_apps
self.populate(installed)
File "D:\django\django\apps\registry.py", line 124, in populate
app_config.ready()
File "D:\django\django\contrib\postgres\apps.py", line 71, in ready
register_type_handlers(conn)
File "D:\django\django\contrib\postgres\signals.py", line 40, in register_type_handlers
oids, array_oids = get_hstore_oids(connection.alias)
File "D:\django\django\contrib\postgres\signals.py", line 24, in get_hstore_oids
return get_type_oids(connection_alias, "hstore")
File "D:\django\django\contrib\postgres\signals.py", line 10, in get_type_oids
cursor.execute(
File "D:\django\django\db\backends\utils.py", line 77, in execute
return self._execute_with_wrappers(
File "D:\django\django\db\backends\utils.py", line 90, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\django\django\db\backends\utils.py", line 94, in _execute
warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
======================================================================
ERROR: setUpClass (postgres_tests.test_aggregates.TestStatisticsAggregate)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\django\django\test\testcases.py", line 1250, in setUpClass
super().setUpClass()
File "D:\django\django\test\testcases.py", line 197, in setUpClass
cls._cls_modified_context.enable()
File "D:\django\django\test\utils.py", line 587, in enable
super().enable()
File "D:\django\django\test\utils.py", line 474, in enable
apps.set_installed_apps(self.options["INSTALLED_APPS"])
File "D:\django\django\apps\registry.py", line 362, in set_installed_apps
self.populate(installed)
File "D:\django\django\apps\registry.py", line 124, in populate
app_config.ready()
File "D:\django\django\contrib\postgres\apps.py", line 71, in ready
register_type_handlers(conn)
File "D:\django\django\contrib\postgres\signals.py", line 40, in register_type_handlers
oids, array_oids = get_hstore_oids(connection.alias)
File "D:\django\django\contrib\postgres\signals.py", line 24, in get_hstore_oids
return get_type_oids(connection_alias, "hstore")
File "D:\django\django\contrib\postgres\signals.py", line 10, in get_type_oids
cursor.execute(
File "D:\django\django\db\backends\utils.py", line 77, in execute
return self._execute_with_wrappers(
File "D:\django\django\db\backends\utils.py", line 90, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\django\django\db\backends\utils.py", line 94, in _execute
warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
======================================================================
ERROR: test_register_serializer_for_migrations (postgres_tests.test_apps.PostgresConfigTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\django\tests\postgres_tests\test_apps.py", line 64, in test_register_serializer_for_migrations
with self.modify_settings(INSTALLED_APPS={"append": "django.contrib.postgres"}):
File "D:\django\django\test\utils.py", line 405, in __enter__
return self.enable()
File "D:\django\django\test\utils.py", line 587, in enable
super().enable()
File "D:\django\django\test\utils.py", line 474, in enable
apps.set_installed_apps(self.options["INSTALLED_APPS"])
File "D:\django\django\apps\registry.py", line 362, in set_installed_apps
self.populate(installed)
File "D:\django\django\apps\registry.py", line 124, in populate
app_config.ready()
File "D:\django\django\contrib\postgres\apps.py", line 71, in ready
register_type_handlers(conn)
File "D:\django\django\contrib\postgres\signals.py", line 40, in register_type_handlers
oids, array_oids = get_hstore_oids(connection.alias)
File "D:\django\django\contrib\postgres\signals.py", line 24, in get_hstore_oids
return get_type_oids(connection_alias, "hstore")
File "D:\django\django\contrib\postgres\signals.py", line 10, in get_type_oids
cursor.execute(
File "D:\django\django\db\backends\utils.py", line 77, in execute
return self._execute_with_wrappers(
File "D:\django\django\db\backends\utils.py", line 90, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\django\django\db\backends\utils.py", line 94, in _execute
warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
======================================================================
ERROR: test_register_type_handlers_connection (postgres_tests.test_apps.PostgresConfigTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\django\tests\postgres_tests\test_apps.py", line 36, in test_register_type_handlers_connection
with modify_settings(INSTALLED_APPS={"append": "django.contrib.postgres"}):
File "D:\django\django\test\utils.py", line 405, in __enter__
return self.enable()
File "D:\django\django\test\utils.py", line 587, in enable
super().enable()
File "D:\django\django\test\utils.py", line 474, in enable
apps.set_installed_apps(self.options["INSTALLED_APPS"])
File "D:\django\django\apps\registry.py", line 362, in set_installed_apps
self.populate(installed)
File "D:\django\django\apps\registry.py", line 124, in populate
app_config.ready()
File "D:\django\django\contrib\postgres\apps.py", line 71, in ready
register_type_handlers(conn)
File "D:\django\django\contrib\postgres\signals.py", line 40, in register_type_handlers
oids, array_oids = get_hstore_oids(connection.alias)
File "D:\django\django\contrib\postgres\signals.py", line 24, in get_hstore_oids
return get_type_oids(connection_alias, "hstore")
File "D:\django\django\contrib\postgres\signals.py", line 10, in get_type_oids
cursor.execute(
File "D:\django\django\db\backends\utils.py", line 77, in execute
return self._execute_with_wrappers(
File "D:\django\django\db\backends\utils.py", line 90, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\django\django\db\backends\utils.py", line 94, in _execute
warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
----------------------------------------------------------------------
Ran 518 tests in 14.649s
FAILED (errors=5, expected failures=3)
follow-up: 6 comment:5 by , 14 months ago
First of all we run the entire test suite not random subsets of test classes.
So it's expected that the tests behave differently when I only run a subset of them? I can't even reproduce this if I just run the failing tests by themselves...
> python runtests.py --settings test_postgres postgres_tests.test_apps # works > python runtests.py --settings test_postgres postgres_tests.test_aggregates # works > python runtests.py --settings test_postgres --parallel 1 postgres_tests.test_apps postgres_tests.test_aggregates # works > python runtests.py --settings test_postgres --parallel 2 postgres_tests.test_apps postgres_tests.test_aggregates # fails
comment:6 by , 14 months ago
Replying to Florian Zimmermann:
First of all we run the entire test suite not random subsets of test classes.
So it's expected that the tests behave differently when I only run a subset of them?
Of course it's not, that's the issue. The way how @modify_settings and initializing django.contrib.postgres work are non trivial. We have here a long standing isolation issue, revealed by fbd16438f46bc2128926958ad24331da5d1b406f.
comment:7 by , 14 months ago
@Florian to demonstrate the issue, you can make the postgres tests fail with --parallel=1 if you replace the lines getting type oids with their underlying non-cached versions (get_hstore_oids() and get_citext_oids() are cached so when it comes time to reload the module there's no db interaction for a single process)
Eg this diff shows what I mean (I'm using psycopg2 so I've replaced it in the psycopg2 block)
--- a/django/contrib/postgres/signals.py
+++ b/django/contrib/postgres/signals.py
@@ -55,7 +55,7 @@ else:
if connection.vendor != "postgresql" or connection.alias == NO_DB_ALIAS:
return
- oids, array_oids = get_hstore_oids(connection.alias)
+ oids, array_oids = get_type_oids(connection.alias, "hstore")
# Don't register handlers when hstore is not available on the database.
#
# If someone tries to create an hstore field it will error there. This is
If you apply that change then --parallel=1 will fail.
comment:8 by , 14 months ago
Here's a failing test that I came up with:
--- a/tests/postgres_tests/test_apps.py
+++ b/tests/postgres_tests/test_apps.py
@@ -5,7 +5,8 @@ from django.db import connection
from django.db.backends.signals import connection_created
from django.db.migrations.writer import MigrationWriter
from django.test import TestCase
-from django.test.utils import modify_settings
+from django.test.utils import modify_settings, override_settings
+from django.contrib.postgres.signals import get_hstore_oids
try:
from django.contrib.postgres.fields import (
@@ -27,6 +28,11 @@ except ImportError:
@unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL specific tests")
class PostgresConfigTests(TestCase):
+ def test_install_app(self):
+ get_hstore_oids.cache_clear()
+ with override_settings(INSTALLED_APPS=["django.contrib.postgres"]):
+ pass
+
def test_register_type_handlers_connection(self):
from django.contrib.postgres.signals import register_type_handlers
comment:9 by , 14 months ago
| Has patch: | set |
|---|
comment:10 by , 14 months ago
| Owner: | changed from to |
|---|---|
| Status: | new → assigned |
| Triage Stage: | Accepted → Ready for checkin |
Note:
See TracTickets
for help on using tickets.
I'm super confused. Shouldn't Jenkins have picked this up earlier, i.e. when fbd16438f46bc2128926958ad24331da5d1b406f (#33143) was merged?