django.auth create_permissions must clear the content type cache before creating permissions

[Sean Legassick]

I hit a problem which took some time to track down, where at the DB flush stage in a sequence of tests (using TransactionTestCase) the recreation of permissions was failing with a FK constraint error.

This was caused by inserting a permission referring to a content type that didn't exist in the DB.

This happened because the content type was still in the cache, even thought the django_content_type table had been truncated.

The cache hadn't been cleared because post_syncdb signal dispatch had called create_permissions before calling update_contenttypes (which does clear the cache and recreate the content types correctly).

The real problem here is that create_permissions and update_contenttypes are both connected to the post_syncdb signal, with the former depending on the latter having been run first, but the dispatcher doesn't guarantee order of dispatch (or rather it dispatches in the order the signal handlers are connected, but that order depends on the order in which modules are loading which is not well-defined). Unfortunately that's a hard problem to solve, and I don't have any ideas about that short of substantive changes to the signal dispatcher.

An easier solution to this particular problem is for create_permissions to clear the content types cache before it recreates permissions, that way the necesarry content types will be created as needed (see attached patch).

[Jason Yan <tailofthesun@…>]

I just got hit by this same issue and worked around it in a similar manner. I instead removed the signal that was connected to update_contenttypes and moved the update_contenttypes call to django.core.management.commands.flush.

For anyone that might be possibly searching for this, here's the exception from the tests without the fix:

IntegrityError: insert or update on table "auth_permission" violates foreign key constraint "content_type_id_refs_id_728de91f"
DETAIL:  Key (content_type_id)=(2) is not present in table "django_content_type".

[PeterRussell]

It seems that the reason that this happens is that the syncdb command is firing the post_syncdb signal for each application, as returned from get_apps() [source:/django/trunk/django/core/management/sql.py@14849#L178 here]. Get_apps appears to return apps in app-name order (see [source:/django/trunk/django/db/models/loading.py@14849#L120 here]), and auth comes before contenttypes asciibetically. It seems the right solution is probably to have a separate post_contenttypes_refreshed signal, and have create_permissions listen for that. It would also be great if updating permissions could fire its own signal, so you can update groups etc based on that. (We've been having a somewhat similar issue).

[orb@…]

I'm having this problem too. We worked around it by extending TransactionTestCase so that _pre_setup() invokes ContentType.objects.clear_cache(). I'd really love to see a solution to this.

[anonymous]

A workaround based on comment 7 is available here: ​http://djangosnippets.org/snippets/2752/

[Chris Wilson]

+1. This just bit us as well.

I think if flush is going to invalidate caches (which it appears to do), then it should be sending a signal that allows holders of caches to respond appropriately. So something like this in flush.py:

for app in set(all_models):
    models.signals.post_flush.send(sender=app, app=app,
        created_models=created_models, verbosity=verbosity,
        interactive=interactive, db=db)
emit_post_sync_signal(set(all_models), verbosity, interactive, db)

Assuming a new signal flush which ContentTypeManager listens for, and then flushes its cache.

[Renato Alves]

Test Django project - created with Django 1.4.5

[Renato Alves]

The attached file is a fresh django project created with Django 1.4.5 which allows reproducing the mentioned error.
NOTE: This is not reproducible with the sqlite backend (MySQL was not tested but was reported to fail with InnoDB in other tickets #9207).

Ticket #9207 is closed as fixed but due to lack of reproducibility. From the error it looks like a duplicate of the current.
The workaround described on comment 3 from #9207 also solves the problem (putting contrib.contenttypes before contrib.auth in INSTALLED_APPS).

PS: I don't know if there's any reason why INSTALLED_APPS has the current order but having django-admin.py invert the order could workaround this problem.

[Renato Alves]

Another duplicate: #6259

[anonymous]

I think I have run into the same problem while trying to use Selenium to test Django admin capabilities. When I try to update a user's permissions, I get the FK error. You can see [my code and issue on SO](​http://stackoverflow.com/questions/18281137/selenium-django-gives-foreign-key-error). So I think this is still a present bug in Django 1.5.1 / MySQL.

[Thomas Güttler]

I think this bug is still alive in Django1.7.

I get this traceback:

File ".../django/core/management/__init__.py", line 385, in execute_from_command_line
  utility.execute()
File ".../django/core/management/__init__.py", line 377, in execute
  self.fetch_command(subcommand).run_from_argv(self.argv)
File ".../django/core/management/base.py", line 288, in run_from_argv
  self.execute(*args, **options.__dict__)
File ".../django/core/management/base.py", line 338, in execute
  output = self.handle(*args, **options)
File ".../django/core/management/commands/migrate.py", line 165, in handle
  emit_post_migrate_signal(created_models, self.verbosity, self.interactive, connection.alias)
File ".../django/core/management/sql.py", line 268, in emit_post_migrate_signal
  using=db)
File ".../django/dispatch/dispatcher.py", line 198, in send
  response = receiver(signal=self, sender=sender, **named)
File ".../django/contrib/auth/management/__init__.py", line 117, in create_permissions
  Permission.objects.using(using).bulk_create(perms)
File ".../django/db/models/query.py", line 410, in bulk_create
  self._batched_insert(objs_without_pk, fields, batch_size)
File ".../django/db/transaction.py", line 339, in __exit__
  connection.commit()
File ".../django/db/backends/__init__.py", line 176, in commit
  self._commit()
File ".../django/db/backends/__init__.py", line 145, in _commit
  return self.connection.commit()
File ".../django/db/utils.py", line 94, in __exit__
  six.reraise(dj_exc_type, dj_exc_value, traceback)
File ".../django/db/backends/__init__.py", line 145, in _commit
  return self.connection.commit()

django.db.utils.IntegrityError: insert or update on table "auth_permission" violates foreign key constraint "auth_content_type_id_508cf46651277a81_fk_django_content_type_id"
 DETAIL:  Key (content_type_id)=(1) is not present in table "django_content_type".

 

[Thomas Güttler]

The six years old patch solved the issue for me:

ContentType.objects.clear_cache()

[Thomas Güttler]

A work around is this line in your migration:

from django.contrib.contenttypes.models import ContentType

class Migration(migrations.Migration):
    operations = [
        ...,
        migrations.RunPython(lambda apps, schema_editor: ContentType.objects.clear_cache()),
    ]

[Thomas Güttler]

I created a pull request: ​https://github.com/django/django/pull/5184

It includes a test. Thanks to Tim Graham for the support.

​https://groups.google.com/d/msg/django-developers/CB4cGwBYo8o/7qcekYNeAgAJ

[Peter Bengtsson]

I can confirm that this is still a bug in Django 1.7.
I can't speak of the PR because I haven't studied it but making sure the order of the INSTALLED_APPS is right, as per ​http://stackoverflow.com/a/18292090/205832 did solve it for me.

[Koen Vossen]

Don't forget to use a database that checks foreign key constraints instead of the testrunner's default sqlite when trying to reproduce this bug.

[Sven R. Kunze]

@guettli @timgraham

It just works. Nice. Thanks for that suggestion.

[Lynn Cyrin]

issue is still present in django 1.11.2

[Michał Pasternak]

I am having this problem with Django 1.11.5 and purest with pytest-django

________________________________________________________________ ERROR at teardown of test_foo ________________________________________________________________

self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x10ee8e630>

    def _commit(self):
        if self.connection is not None:
            with self.wrap_database_errors:
>               return self.connection.commit()
E               psycopg2.IntegrityError: B����D:  wstawianie lub modyfikacja na tabeli "auth_permission" narusza klucz obcy "auth_permission_content_type_id_2f476e4b_fk_django_co"
E               DETAIL:  Klucz (content_type_id)=(893) nie wyst��puje w tabeli "django_content_type".

../../envs/django-bpp/lib/python3.6/site-packages/django/db/backends/base/base.py:236: IntegrityError

The above exception was the direct cause of the following exception:

self = <django.test.testcases.TransactionTestCase testMethod=__init__>

    def _post_teardown(self):
        """Performs any post-test things. This includes:

            * Flushing the contents of the database, to leave a clean slate. If
              the class has an 'available_apps' attribute, post_migrate isn't fired.
            * Force-closing the connection, so the next test gets a clean cursor.
            """
        try:
>           self._fixture_teardown()

../../envs/django-bpp/lib/python3.6/site-packages/django/test/testcases.py:925:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../envs/django-bpp/lib/python3.6/site-packages/django/test/testcases.py:960: in _fixture_teardown
    inhibit_post_migrate=inhibit_post_migrate)
../../envs/django-bpp/lib/python3.6/site-packages/django/core/management/__init__.py:131: in call_command
    return command.execute(*args, **defaults)
../../envs/django-bpp/lib/python3.6/site-packages/django/core/management/base.py:330: in execute
    output = self.handle(*args, **options)
../../envs/django-bpp/lib/python3.6/site-packages/django/core/management/commands/flush.py:88: in handle
    emit_post_migrate_signal(verbosity, interactive, database)
../../envs/django-bpp/lib/python3.6/site-packages/django/core/management/sql.py:53: in emit_post_migrate_signal
    **kwargs
../../envs/django-bpp/lib/python3.6/site-packages/django/dispatch/dispatcher.py:193: in send
    for receiver in self._live_receivers(sender)
../../envs/django-bpp/lib/python3.6/site-packages/django/dispatch/dispatcher.py:193: in <listcomp>
    for receiver in self._live_receivers(sender)
../../envs/django-bpp/lib/python3.6/site-packages/django/contrib/auth/management/__init__.py:83: in create_permissions
    Permission.objects.using(using).bulk_create(perms)
../../envs/django-bpp/lib/python3.6/site-packages/django/db/models/query.py:449: in bulk_create
    obj_without_pk._state.db = self.db
../../envs/django-bpp/lib/python3.6/site-packages/django/db/transaction.py:223: in __exit__
    connection.commit()
../../envs/django-bpp/lib/python3.6/site-packages/django/db/backends/base/base.py:262: in commit
    self._commit()
../../envs/django-bpp/lib/python3.6/site-packages/django/db/backends/base/base.py:236: in _commit
    return self.connection.commit()
../../envs/django-bpp/lib/python3.6/site-packages/django/db/utils.py:94: in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
../../envs/django-bpp/lib/python3.6/site-packages/django/utils/six.py:685: in reraise
    raise value.with_traceback(tb)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x10ee8e630>

    def _commit(self):
        if self.connection is not None:
            with self.wrap_database_errors:
>               return self.connection.commit()
E               django.db.utils.IntegrityError: B����D:  wstawianie lub modyfikacja na tabeli "auth_permission" narusza klucz obcy "auth_permission_content_type_id_2f476e4b_fk_django_co"
E               DETAIL:  Klucz (content_type_id)=(893) nie wyst��puje w tabeli "django_content_type".

../../envs/django-bpp/lib/python3.6/site-packages/django/db/backends/base/base.py:236: IntegrityError

===================================================================== 29 tests deselected =====================================================================
====================================================== 1 passed, 29 deselected, 1 error in 3.18 seconds =======================================================

My package versions

pytest-base-url (1.4.1)
pytest-cov (2.5.1)
pytest-django (3.1.2)
pytest-html (1.16.0)
pytest-instafail (0.3.0)
pytest-localserver (0.3.6)
pytest-metadata (1.5.0)
pytest-splinter (1.8.5)
pytest-variables (1.7.0)

Django (1.11.5)

[Michał Pasternak]

Moving contenttypes apps before auth in INSTALLED_APPS fixed it.

[Jared Proffitt]

Michał's fix works for me as well. If I move 'django.contrib.contenttypes' before 'django.contrib.auth' in INSTALLED_APPS, it works.

Would be nice to either have a fix that doesn't require a specific order of apps, or some documentation mentioning you need the specific app order.

[Claude Paroz]

​PR

[Tim Graham <timograham@…>]

In bec651a:

Fixed #10827 -- Ensured ContentTypes are created before permission creation.