'manage.py test' failing for apps that access read-only databases

[kthhrv]

​http://groups.google.com/group/django-users/browse_thread/thread/466ff5d2ab57c5a1/8131fd4a7d7a7cb6?lnk=gst&q=testing#8131fd4a7d7a7cb6

searching 'Testing framework' component i can't find a ticket for this issue so here it is.

I have an app that accesses two databases one of which is read-only, when i run 'manage.py test <app>' i get an insufficient privileges error while its trying to create/destroy the tables on the read-only database. This causes the test run to fail then and there.

all the models in the read-only app are marked managed=False.

we can't always have RW access to legacy databases.

is there a workaround for this issue? i'd love to start running tests on this app.

thx
Keith

[Luke Plant]

What kind of workaround do you envisage?

If you are looking for read-write tests, the normal work-around for a read-only database is to create a separate, writeable, blank database and use that for your tests. You will need a different settings file, or a settings file that has some way of knowing to use a different DB when you are in test mode.

If you cannot create a new database, and cannot create any tables in your existing one, you would have to run on the live data. Assuming this database is being used for something, you are presumably not going to want to do any writes to the DB at all, which limits you to read-only tests on your actual live data. Is that what you are wanting? It is already possible - just use the TestCase and test runners provided by unittest rather than using the Django subclasses and manage.py, and your tests will connect to the normal database. You will miss out on some goodness from Django's TestCase, but not much, because most of it is to do with writing to databases. You can also instantiate your own Client for testing views, so you don't miss out on that.

[jonathan.ca@…]

A simple hack that seems to work for this is to modify the action of setup_databases in django/test/simple.py

 def setup_databases(self, **kwargs):
        from django.db import connections
        old_names = []
        mirrors = []
        for alias in connections:
            connection = connections[alias]
+           # If the alias name is contained within this setting, 
+           # just skip round the loop.
+           # This causes all tests to occur against the aliased
+           # connection, and not create a test db.
+           try:
+               if alias in settings.RUN_TESTS_ON_LIVE_DB:
+                   continue
+           except:
+               pass

            # If the database is a test mirror, redirect it's connection
            # instead of creating a test database.
            if connection.settings_dict['TEST_MIRROR']:
                mirrors.append((alias, connection))
                mirror_alias = connection.settings_dict['TEST_MIRROR']
                connections._connections[alias] = connections[mirror_alias]
            else:

                old_names.append((connection, connection.settings_dict['NAME']))
                connection.creation.create_test_db(self.verbosity, autoclobber=not self.interactive)
        return old_names, mirrors

and then place the names of the connections inside the variable in your settings.py:

    RUN_TESTS_ON_LIVE_DB = ['dbname',]

This seems to force the tests to run against the live database, write operations will simply fail due to permissions and one no longer needs complex alternative db setups.

[Russell Keith-Magee]

This sort of thing makes my teeth itch, because it's *really* bad testing practice -- tests should be repeatable, and anything with live data, by definition, isn't guaranteed repeatable.

A more interesting, but related use case is when you don't have permission to create or destroy databases, but you do have access to a database that you can use for testing purposes. Since solving this use case essentially implies solving the use case for this ticket, I'll mark this accepted.

[Russell Keith-Magee]

#14339 was a ticket requesting the alternate use case I described.

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

Change Easy pickings from NULL to False.

[RK]

What about if it were possible to specify in the database parameters that the test runner should create the test databases using a different database server config entirely? Eg:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'myproject',
        'HOST': 'maindb',
         # ... plus some other settings
    },
    'readonly': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'myproject',
        'HOST': 'readonlydb',
        'TEST_DATABASE': 'default'
        # ... plus some other settings
    }
}

This would trigger the test runner to create another test database on the "default" server, for use with anything during the test which tries to access the "readonly" database.

[Shai Berger]

The Oracle backend supports a test parameter CREATE_DB that tells it not to create the (tablespace, in this context equivalent to database) for testing but rather use an existing one. It still tries to create tables, but I guess the managed=False setting would take care of that in this context.

I would like to make that parameter apply to all database backends.

[Greg Chapple]

As of b7aa7c4ab4372d2b7994d252c8bc87f77dd217ae you can use the --keepdb parameter to use an existing database rather than creating a new one for testing. This sounds more or less like what shai is describing above. It will run any outstanding migrations, and flush the data, but ultimately the database will be preserved at the end of the run.

[Shai Berger]

6 years after a general workaround was suggested, and almost two years since the last comment, I think that anybody who wants improvement on this should specify exactly what they need and why.

In particular -- the combination of --keepdb and database routers disallowing migrations on the read-only database will stop any automatic schema changes, and using Luke's suggestions for test-cases should stop data changes as well.

[Carl Marshall]

I'd like to add some more to this again as I'm having this problem with Django 2.1.

I have some large Oracle DBs which are read only, and my app is providing an API to. All my models are set to be managed=False, and I've set the databases to have:

        'TEST': {
            'CREATE_DB': False,
            'CREATE_USER': False,
            'USER': env('DATABASE_USER_DEV'),
            'PASSWORD': env('DATABASE_PASSWORD_DEV'),
        }

However, running python manage.py test results in django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create the django_migrations table (ORA-02000: missing ALWAYS keyword) which suggests to me that we need a "read-only" database option that will stop all attempts to generate data on these connections.

For reference, the app uses a local sqlite or postgress DB for the data it can store (and the standard Django apps) as the default database, but there is no scope to be able to get write access to these external databases, or have them duplicated for testing in any meaningful way (and I understand Russell Keith-Mageen concern about live data in tests).

I'm not sure what the best solution here would be, for example:

• Using Create DB as False doesn't always mean you can't test the database
• Tests need to look for any table on the database that is managed, if none found, treat as read-only (seems like an expensive option)
• Have a READ_ONLY test flag that would cause the standard tests to skip some of the write based tests

The main output I want is to be able to run my tests (including the standard ones from Django and other packages) without the early termination of the test script because of the migrations exception.

[Jacob Walls]

It's likely that the use case from comment:14 is solved through the combination of setting 'TEST': {'MIGRATE': False} (new in 3.1) and the fix for #23273 (merged for 4.1) to skip creating the django_migrations table.

Setting back to needsinfo in case someone has details to add around the interaction of read-only databases with the test runner.