Apache setup using sqlite3 breaks when performing a field__regex filter

[eternicode]

Using a simple apache mod_wsgi setup, setting the project to use sqlite3 as the backend, and running MyModel.objects.filter(field__regex=r'') (and later evaluating that query with a .count() ) causes the backend to throw an "OperationalError: user-defined function raised exception".

I've managed to trace this to the 'import re' statement inside _sqlite_regexp in db/backends/sqlite3/base.py. After moving that import statement to outside that function (making it a module-level import), the exception is no longer raised and everything works fine.

My specific regex that I initially tried was r'\bfeatured\b' on a django-tagging TagField, but it happened with an empty string (r'') as well.

I noticed this while using 1.1.2. A quick test with 1.2 confirmed the error shows up there (as a DatabaseError, same message), too. I haven't tested with SVN.

Strangely, the regex filter works perfectly when run from ./manage.py shell, or from the same page viewed on ./manage.py runserver. It's just when it's setup with apache that it has issues for some reason.

[Russell Keith-Magee]

I can't reproduce this problem. It sounds like something is very badly messed up with your server environment, and the problem lies in the regex library.

[Sashan <anedvedicky@…>]

I've hit same problem after upgrading lenny to debian:

django-1.1 -> django-1.2.4;
python2.4 -> python2.6

The workaround proposed by eternicode really works, which is big mystery to me.

I've noticed the statement 'import re' is used at the beginning of
db.backends.sqlite3.base module. So I've just removed 'import re' line in
_sqlite_regex() function.

Once workaround has been applied, it started to work, until I've hit another
problem with url reverse. All templates, which are using url tag fail with
'Caught ImportError while rendering: No module named urls' exception.
Everything points towards apache misconfiguration or upgrade path.

I'm using wsgi module 3.3. My apache config follows further below. Any
help very appreciated. I suspect I need to tune environment a bit, but
I have no idea where to start. Thanks in advance for any hint.

--- cut here to get /etc/apache2/envvars ----

# envvars - default environment variables for apache2ctl

# this won't be correct after changing uid
unset HOME

# for supporting multiple apache2 instances
if [ "${APACHE_CONFDIR##/etc/apache2-}" != "${APACHE_CONFDIR}" ] ; then
        SUFFIX="-${APACHE_CONFDIR##/etc/apache2-}"
else
        SUFFIX=
fi

# Since there is no sane way to get the parsed apache2 config in scripts, some
# settings are defined via environment variables and then used in apache2ctl,
# /etc/init.d/apache2, /etc/logrotate.d/apache2, etc.
export APACHE_RUN_USER=www-data
export APACHE_RUN_GROUP=www-data
export APACHE_PID_FILE=/var/run/apache2$SUFFIX.pid
export APACHE_RUN_DIR=/var/run/apache2$SUFFIX
export APACHE_LOCK_DIR=/var/lock/apache2$SUFFIX
# Only /var/log/apache2 is handled by /etc/logrotate.d/apache2.
export APACHE_LOG_DIR=/var/log/apache2$SUFFIX

## The locale used by some modules like mod_dav
export LANG=C
## Uncomment the following line to use the system default locale instead:
#. /etc/default/locale

export LANG

## The command to get the status for 'apache2ctl status'.
## Some packages providing 'www-browser' need '--dump' instead of '-dump'.
#export APACHE_LYNX='www-browser -dump'

--- cut here to get /etc/sites-enabled/www-site ----

<VirtualHost *:80>
        ServerName www.example.com 
        WSGIScriptAlias /       /srv/example/django.wsgi
</VirtualHost>

--- cut here to get /etc/sites-enabled/media ----

<VirtualHost *:80>
        ServerName media.example.com
        DocumentRoot /srv/media
        <Directory *>
                Options -Indexes
        </Directory>
</VirtualHost>

--- cut here to get /srv/example/django.wsgi ----

import os
import sys

sys.stdout = sys.stderr
#sys.path.append('/srv/')
sys.path.append('/srv/parliament-poll-stats/')
#sys.path.append('/srv/parliament-poll-stats/json')
#sys.path.append('/srv/parliament-poll-stats/utils')
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

[Paul McMillan]

I think I may have found your problem. You're talking about using sqlite3, but the ​sqlite module listed on the Debian page as related is actually the python bindings for sqlite2.

"For an interface to SQLite 3, see the package python-pysqlite1.1, python-pysqlite2 or python-apsw."

-​http://packages.debian.org/wheezy/python-sqlite

Try installing one of those packages and see if it fixes your problem. Don't forget to put the re module back the way it is supposed to go.

[Sashan <anedvedicky@…>]

Replying to PaulM:

"For an interface to SQLite 3, see the package python-pysqlite1.1, python-pysqlite2 or python-apsw."

-​http://packages.debian.org/wheezy/python-sqlite

sure you were right, thanks for your help.

[Sashan <anedvedicky@…>]

Replying to Sashan <anedvedicky@…>:

Replying to PaulM:

"For an interface to SQLite 3, see the package python-pysqlite1.1, python-pysqlite2 or python-apsw."

-​http://packages.debian.org/wheezy/python-sqlite

sure you were right, thanks for your help.

sorry - still no success, once I've cleaned thy .pyc files I got same error, even though python-pysqlite2 package
is installed.

the apache error log says:
[Thu Feb 10 22:53:18 2011] [error] /usr/lib/pymodules/python2.6/django/db/backends/sqlite3/base.py:243: RuntimeWarning: Parent module 'dango.db.backends.sqlite3' not found while handling absolute import
[Thu Feb 10 22:53:18 2011] [error] import re

looks like environment is somehow screwed:
the python path is content is as follows:
[ '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/local/lib/python2.6/dist-packages', '/usr/lib/python2.6/dist-packages', '/usr/lib/python2.6/dist-packages/PIL', '/usr/lib/pymodules/python2.6', '/srv/parliament-poll-stats/' ]
the last path is location of my django-site. django is installed in /usr/lib/pymodules/python2.6/

even if I workaround the problem with 'import re' I will get stuck in {% url %}, which complains on ImportError if some urls module, which is part of my project.
looks like it is a problem of Debian, not django.

[Sashan <anedvedicky@…>]

So here we have a final summary. There were actually two problems with my set up.
The ImportError was caused by some deficiencies in wscgi module. I've fixed by
using excellent blog entry ​http://blog.dscpl.com.au/2010/03/improved-wsgi-script-for-use-with.html by Graham Dumpleton.

The problem with import of re module in _sqlite_regexp() has been fixed by following
steps discussed here: ​http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Multiple_Python_Sub_Interpreters

adding a directive:

    WSGIApplicationGroup %{GLOBAL}

saved my day.

so I think it's not a problem of Django, not a problem of debian, it's just changes in wscgi module.

also the python bindings for sqlite are not needed, since sqlite3 comes by default since python2.5.
I'm running 2.6.

[Sashan <anedvedicky@…>]

Replying to Sashan <anedvedicky@…>:

FYI the error message in apache error log when importing re in _sqlite_regex()
looked as follows:

[error] cannot unmarshal code objects in restricted execution mode

[Marvin]

This problem occurs in version 1.3 too.
Removing unnecessary "import re" string from _sqlite_regex() function solves it.

[Ramiro Morales]

In [16799]:

Fixed #14138 -- Removed a superfluous import in the sqlite3 DB backend.

This could be of help with some issues people were seeing when deploying
Django with sqlite and Apache.