IntegrityError with TransactionTestCase and serialized_rollback=True

[Tim Court]

Using a TransactionTestCase with a sqlite database raises an IntegrityError in Django 1.7. This is the error I'm seeing:

Traceback (most recent call last):
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/test/testcases.py", line 182, in __call__
    self._pre_setup()
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/test/testcases.py", line 754, in _pre_setup
    self._fixture_setup()
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/test/testcases.py", line 797, in _fixture_setup
    connections[db_name]._test_serialized_contents
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/backends/creation.py", line 428, in deserialize_db_from_string
    obj.save()
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/core/serializers/base.py", line 173, in save
    models.Model.save_base(self.object, using=using, raw=True)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/base.py", line 618, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/base.py", line 699, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/base.py", line 732, in _do_insert
    using=using, raw=raw)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 921, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 920, in execute_sql
    cursor.execute(sql, params)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 485, in execute
    return Database.Cursor.execute(self, query, params)
IntegrityError: columns app_label, model are not unique

I can reproduce this behaviour using the default project template, created using startproject and startapp, and nothing more than a single TransactionTestCase with two tests and the serialized_rollback option.

Here is a minimal sample project that demonstrates the problem: ​https://github.com/tctimmeh/djangomigrate. Running manage.py test mtestapp results in the error above.

It looks like the django.contrib.contenttypes app has a post-migrate script that populates the content type table. However, using the serialized_rollback option to refresh any migrated data also inserts saved duplicates into the content type table, which results in this exception.

[khoobks]

I realise that this is not a good workaround but as a temporary measure I found that the following works in my situation.

I subclassed the TransactionTestCase and added the following code.

def _fixture_setup(self):
    super(MyTransactionTestCase, self)._fixture_setup()
    for app_config in apps.get_app_configs():
        update_contenttypes(app_config)
        create_permissions(app_config)
        create_default_site(app_config)

def _fixture_teardown(self):
    signals.post_migrate.disconnect(create_default_site, sender=apps.get_app_config('sites'))
    signals.post_migrate.disconnect(update_contenttypes)
    signals.post_migrate.disconnect(create_permissions, dispatch_uid="django.contrib.auth.management.create_permissions")
    signals.post_migrate.disconnect(update_group_permissions)

    super(MyTransactionTestCase, self)._fixture_teardown()

    signals.post_migrate.connect(update_contenttypes)
    signals.post_migrate.connect(create_permissions, dispatch_uid="django.contrib.auth.management.create_permissions")
    signals.post_migrate.connect(create_default_site, sender=apps.get_app_config('sites'))

The the overridden _fixture_teardown disables the post_migrate signal handlers from executing and creating the problematic rows in the database. This allows the TransactionTestCase to successfully reload the original database state. The handlers are then manually invoked in the overriden _fixture_setup in order to ensure that for subsequent tests, the ContentTypes, Sites and Permissions are available.

[Dan Poirier]

FYI, I see this with Postgres as well, so it doesn't appear to be SQLite specific.

[Tim Graham]

I had a look at the ​PR, but I'm not confident enough to commit it. Also, some documentation might be needed (the loss of the post_migrate signal in this scenario might be backwards incompatible for some users).

[Tim Graham]

Added a comment with an idea about a test on the pull request.

[Tim Graham]

​PR with test added.

[Tim Graham <timograham@…>]

In d3fdaf90:

Fixed #23727 -- Inhibited the post_migrate signal when using serialized_rollback.

When using a TransactionTestCase with serialized_rollback=True,
after creating the database and running its migrations (along with
emitting the post_migrate signal), the contents of the database
are serialized to _test_serialized_contents.

After the first test case, _fixture_teardown() would flush the
tables but then the post_migrate signal would be emitted and new
rows (with new PKs) would be created in the django_content_type
table. Then in any subsequent test cases in a suite,
_fixture_setup() attempts to deserialize the content of

_test_serialized_contents, but these rows are identical to the

rows already in the database except for their PKs. This causes an
IntegrityError due to the unique constraint in the
django_content_type table.

This change made it so that in the above scenario the post_migrate
signal is not emitted after flushing the tables, since it will be
repopulated during fixture_setup().

[Romain Garrigues]

I have seen that this patch has been applied to Django 1.9.x.

For previous Django versions, I guess we can resolve this problem by setting:

TEST_NON_SERIALIZED_APPS = ['django.contrib.contenttypes']

as proposed in StackOverflow issue ​http://stackoverflow.com/questions/29226869/django-transactiontestcase-with-rollback-emulation/35359897 by @se7entyse7en. Is it right ?