Error with new pyscopg2 2.4.2 release and tests

[anonymous]

The test command doesn't work with psycopg2==2.4.2 but does with 2.4.1:

Creating test database for alias 'default'...
Traceback (most recent call last):
  File "manage.py", line 14, in <module>
    execute_manager(settings)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/commands/test.py", line 37, in handle
    failures = test_runner.run_tests(test_labels)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/test/simple.py", line 359, in run_tests
    old_config = self.setup_databases()
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/test/simple.py", line 296, in setup_databases
    test_db_name = connection.creation.create_test_db(self.verbosity, autoclobber=not self.interactive)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/db/backends/creation.py", line 351, in create_test_db
    self._create_test_db(verbosity, autoclobber)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/db/backends/creation.py", line 416, in _create_test_db
    self.set_autocommit()
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/db/backends/creation.py", line 472, in set_autocommit
    self.connection.connection.autocommit = True
psycopg2.ProgrammingError: autocommit cannot be used inside a transaction

[ampledata]

Going out on a limb and guessing this is due to the transaction control 'overhaul' in Psycopg 2.4.2:

Transaction control has been overhauled: a new connection method set_session() allows setting all the session properties affecting the transactions behaviour: the isolation level but it can also be used to have auto-commit, read-only, and deferrable transactions.

Para: ​http://www.initd.org/psycopg/articles/2011/06/12/psycopg-242-released/

[piro]

The quick fix seems to reverse the test and check for the existence of set_isolation_level before autocommit.

The proper fix is to add a set_autocommit method to the specific database driver.

[piro]

Opened a discussion about the issue in the psycopg ML.

​http://archives.postgresql.org/psycopg/2011-06/msg00028.php

[piro]

Also note that if the exception is raised it means that you have already started a transaction somehow. Asking for autocommit with a transaction already in progress is asking the drivers to be generous and give you a rollback at best. A correct behaviour for a more robust set_autocommit() could be to call connection.rollback() as well.

[Ramiro Morales]

I can reproduce this issue only when running application tests. It doesn't show itself when running the Django test suite.

[Ramiro Morales]

Simple fix

[Ramiro Morales]

More invasive fix

[Nicolas Lara]

I would rise NotImplementedError in the base django/db/backends/creation.py instead of just 'passing'

[Ramiro Morales]

Drop .set_autocommit(), replace it with a method that explicitly states what we use it for.

[Ramiro Morales]

Restoring the summary -- I can see this with the django test suite, most surely I didn't use the right psycopg2 installation when tested things before.

[anonymous]

I'm starting with django and wanted to try PostgreSQL database. After seing that ./manage.py runserver does not work, I tried with 'testserver' and found about this problem, but wonder is it just problem in the testsuite or django-1.3 does not work with psycopg2-2.4.2 in general?

[gour]

Excuse me that my comment went through as 'anonymous' although it belongs to me. :-(

[Aymeric Augustin]

Django 1.3 does not work with psycopg2 2.4.2 in general; use 2.4.1.

[Ramiro Morales]

In [16520]:

Fixed #16250 -- Made test database creation support code in the PostgreSQL DB backend compatible with psycopg2 2.4.2.

Implemented this by adding an internal hook for work that should be performed
before that point.

Also, regarding the DatabaseCreation.set_autocommit() method:

• Stop using it for such tasks
• Stop providing an implementation that tries to cover all the possible idioms a third party database backend DB-API 2 driver could need to activate autocommit. It is now left for third party backends to implement.

This can be backwards incompatible in the case of user applications that:

• Had started using this method
• Use a third a party database backend

[Shai Berger]

Just for the record, the fix was not included in Django 1.3.1; we still need 2.4.1.

[anonymous]

As a Django 1.3 user, how am I supposed to install psycopg2 2.4.1? It doesn't appear on PyPI, nor did I find it anywhere else. I need Windows binaries for it. Are you trying to keep it a secret?...

[Ramiro Morales]

Replying to anonymous:

As a Django 1.3 user, how am I supposed to install psycopg2 2.4.1? It doesn't appear on PyPI, nor did I find it anywhere else. I need Windows binaries for it. Are you trying to keep it a secret?...

Why don't you ask psycopg2 questions on the psycopg2 user support channels and stop trolling (even worse, trolling as an anonymous user) here?.

[anonymous]

Nonetheless, anonynmous's point stands that the fix should be included in the next Django 1.3

[anonymous]

And anonymous, you can find 2.4.1 on pypi:

pip install psycopg2==2.4.1

[Luke Plant]

Bug fixes like this are no longer backported to the stable branch - it will not be in the next 1.3.X release. See ​https://docs.djangoproject.com/en/dev/internals/release-process/#supported-versions

This bug is not 'critical', because it can be avoided by using a different version of psycopg2.

[Donald Stufft]

Replying to aaugustin:

Django 1.3 does not work with psycopg2 2.4.2 in general; use 2.4.1.

I just talked to aaugustin about this statement. He said that his initial impression of the extent of Django and psycopg2 incompatibility. was incorrect. The only place in Django that is affected by the changes made in psycopg2 2.4.2 is the test runner, so psycopg2 is fine to use in cases where you do not need to run the tests (such as when deploying to production servers).

[Aymeric Augustin]

I confirm — I can't figure out why I wrote this. Maybe my subconscious thinks that Django is unusable without the test runner :)

Thanks dstufft for correcting me.

[yodiaditya]

When it will included in django 1.3.1 ?

[jakewan]

Replying to yodiaditya:

When it will included in django 1.3.1 ?

See comment 26

[dpifke]

Ubuntu users running the latest release (Oneiric) will encounter this issue, since Ubuntu ships Django 1.3.1 and Psycopg2 2.4.2.

This has been reported to Ubuntu as issue 905837:

​https://bugs.launchpad.net/ubuntu/+source/python-django/+bug/905837

[Antti Kaihola]

A quick copy-paste script for replacing the psycopg2 Debian package with 2.4.1 from PyPI on Ubuntu 11.10 and distributions based on it (like Linux Mint 12):

sudo apt-get remove python-psycopg2
sudo apt-get install -y python-dev libpq-dev
sudo apt-get install -y python-pip  # if not installed manually
sudo pip install psycopg2==2.4.1

[ncikic]

May I ask a clarification as new djang/python developer about this issue?
Is the recommended solution to apply the fixes as in Changeset 16520 by Mr. ramiro or should i revert to psycopg2 2.4.1?
What I need is the clarification about the following comment in Changeset 16520
"Also, regarding the DatabaseCreation.set_autocommit() method:

Stop using it for such tasks
Stop providing an implementation that tries to cover all the possible idioms a third party database backend DB-API 2 driver could need to activate autocommit. It is now left for third party backends to implement. "

Does this mean the psycopg2 2.4.2 introduced the bug or it is django bug?

So the point is: Will the Changeset 16520 be part of the next django release or it is just a quick fix and for the next release it is up to psycopg2 developers to fix the problem definitely (note that at present the last psycopg2 is 2.4.4)?
Thanks

[Aymeric Augustin]

The easiest solution is to use psycopg2 2.4.1. Note that the issue only appears when your run tests.

[anonymous]

Replying to aaugustin:

The easiest solution is to use psycopg2 2.4.1. Note that the issue only appears when your run tests.

When I use psycopg2 2.4.1, I get:

django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: cannot import name _connect

In fact I seem to get that with anything less than the latest psycopg2. I'm using pgsql-9.1 and the latest version of django.

[Aymeric Augustin]

Replying to anonymous:

When I use psycopg2 2.4.1, I get:
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: cannot import name _connect

I just encountered this issue after downgrading psycopg2 to 2.4.1 on Django's own CI server.

I had to wipe previous installations:

rm -r /usr/lib/python2.5/site-packages/psycopg2*
rm -r /usr/local/lib/python2.6/dist-packages/psycopg2*
rm -r /usr/local/lib/python2.7/dist-packages/psycopg2*

and then reinstall psycopg2.

[zap.aibulatov@…]

I have installed psycopg2==2.4.1 via pip on my virtualenv, and now i have this error:

Traceback (most recent call last):
  File "./manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 429, in execute_from_command_line
    utility.execute()
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 261, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 67, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/south/management/commands/__init__.py", line 10, in <module>
    import django.template.loaders.app_directories
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/template/loaders/app_directories.py", line 21, in <module>
    mod = import_module(app)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/contrib/admin/__init__.py", line 3, in <module>
    from django.contrib.admin.helpers import ACTION_CHECKBOX_NAME
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/contrib/admin/helpers.py", line 3, in <module>
    from django.contrib.admin.util import (flatten_fieldsets, lookup_field,
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/contrib/admin/util.py", line 1, in <module>
    from django.db import models
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/__init__.py", line 78, in <module>
    connection = connections[DEFAULT_DB_ALIAS]
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/utils.py", line 93, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/utils.py", line 33, in load_backend
    return import_module('.base', backend_name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 24, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: can't import mx.DateTime module

[anonymous]

This is a psycopg2 bug: ​http://psycopg.lighthouseapp.com/projects/62710/tickets/53

[dave@…]

I have django 1.3.1 installed as well as psycopg-2.4.1 and I still get the exact same error above.

Is there any way to work around this to get my unit tests to work?

[Aymeric Augustin]

Upgrade to 1.3.5.

[dave@…]

I've upgraded to 1.3.5. Same problem.

[Aymeric Augustin]

Sorry, I confused this ticket with another one.

Either the version of psycopg you have isn't 2.4.1, or you're seeing an different problem - even if the traceback is similar.