Sqlite segmentation fault regression when running tests (since 1.7.2)

[Steve Jalim]

A project has a reasonably large number of tests (~1600). As part of our dev workflow, we run tests against sqlite first, then against postgres before checkin.

On 1.7.1 all tests pass on sqlite, bar some deliberate DB-specific exclusions for tests that need specific postgres features. They also all pass on postgres.

On 1.7.2, all tests still pass on postgres (though the server needed a restart) but the sqlite test run fails at varying points with "segmentation fault" (and an exit code of 139, accordingly).

The segmentation fault occurs regardless of whether an in-memory or file-based database is used. When the file-based run crashes, the DB is around 495k in size.

There is very little output, even at max verbosity, because this is coming from something lower down than Django itself. However, because it only started happening with 1.7.2 and goes away when we revert to 1.7.1, I am assuming a change somewhere caused the regression.

Setup spec:

• Headless Virtualbox + Vagrant with 1GB allocated RAM
• Ubuntu 14.04 LTS, fully up to date
• libsqlite3-0 version: 3.8.2-1ubuntu2 amd64
• Python: 2.7.5 (up-to-date version for 14.04 - unclear which parts, if any of 2.7.8 have been backported)

Any steers here would be much appreciated. apollo13 on #django pointed out that I could git bisect the commits between 1.7.1 and 1.7.2, which I am open to doing if I can make the time.

Cheers
Steve

[Tim Graham]

I think there is little chance we can help without additional information, especially the git bisect results.

[Shai Berger]

gdb can be used to debug Python programs at both Python and C levels; this may help you.

See ​https://docs.python.org/devguide/gdb.html and ​https://wiki.python.org/moin/DebuggingWithGdb

Another info which may be even more useful is to pinpoint a test which causes the segfault. If you can share code (I assume, smaller than your entire project) which causes the failure it will be very helpful.

If possible, try running your code on other versions of Python -- non-Ubuntu, or even Python3. You may be able to get a consistent segfault that way.

[Steve Jalim]

Thanks shaib - I'll see what gdb can turn up.

Yes, I'd already considered providing a sample project, but that's not practical as the segfault appears to happen in different places/times in the suite run and a small run does not trigger it. I shall keep digging.

[Steve Jalim]

Just as an update for the future: I 'solved' this by throwing more RAM at it. Haven't seen it happen since.

[Tomáš Ehrlich]

This regression is caused by #20392. In the ticket there's a mention about sqlite:

I also ran into some weird threading issues with sqlite that was causing a segmentation fault.

For me it happens in OS X 10.10.2, Python 2.7.6 and latest master branch.

[Tim Graham]

The original report is against Django 1.7.2 -- the change for the linked ticket is only present on 1.8+.

[Tomáš Ehrlich]

I see. It works for stable/1.7.x and 1.7.2 for me though...

[Tomáš Ehrlich]

It's probably related only to Python 2.7.6. In Python 3.4.1 the test case runs fine.

Unfortunately I haven't found the cause of segmentation fault. Here is the traceback (begins with the last test case):

SAVEPOINT "s140735267844864_x61"
test_use_explicit_o2o_to_parent_from_abstract_model (model_inheritance_regress.tests.ModelInheritanceTest) ... ok
ROLLBACK TO SAVEPOINT "s140735267844864_x61"
Fatal Python error: Segmentation fault

Current thread 0x00007fff7ba5b300 (most recent call first):
  File "/Users/tricoder/Projects/django/django/db/backends/sqlite3/base.py", line 317 in execute
  File "/Users/tricoder/Projects/django/django/db/backends/utils.py", line 63 in execute
  File "/Users/tricoder/Projects/django/django/db/backends/base/base.py", line 215 in _savepoint_rollback
  File "/Users/tricoder/Projects/django/django/db/backends/base/base.py", line 255 in savepoint_rollback
  File "/Users/tricoder/Projects/django/django/db/transaction.py", line 252 in __exit__
  File "/Users/tricoder/Projects/django/django/test/testcases.py", line 932 in _rollback_atomics
  File "/Users/tricoder/Projects/django/django/test/testcases.py", line 988 in _fixture_teardown
  File "/Users/tricoder/Projects/django/django/test/testcases.py", line 844 in _post_teardown
  File "/Users/tricoder/Projects/django/django/test/testcases.py", line 184 in __call__
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/suite.py", line 108 in run
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/suite.py", line 70 in __call__
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/runner.py", line 151 in run
  File "/Users/tricoder/Projects/django/django/test/runner.py", line 178 in run_suite
  File "/Users/tricoder/Projects/django/django/test/runner.py", line 211 in run_tests
  File "./tests/runtests.py", line 236 in django_tests
  File "./tests/runtests.py", line 414 in <module>
[1]    38431 segmentation fault  ./tests/runtests.py -v3 model_inheritance_regress

Some clues:

• Good library for catching tracebacks on segfault is faulthandler ​http://faulthandler.readthedocs.org/
• Some failing test suites are model_inheritance_regress (with traceback above) and admin_views
• It always crashes on "ROLLBACK TO SAVEPOINT", even in different tests (admin_views for example).
• It doesn't depend on how many savepoints is created and released (admin_views tests create about 350 savepoints, while model_inheritance_regress creates about 60 savepoints)

[Chris Foresman]

I can confirm that updating from default 2.7.6 on OS X 10.10 to 2.7.9 gets rid of the error.

I did notice that Homebrew also installed an updated version of sqlite3; so I can't say for 100% sure if it was just the Python update or the updated sqlite3 that Homebrew binds to Python 2.7.9.

[Tim Graham]

As Django only officially supports the latest micro release, I think we can close this.

[David Szotten]

i can reproduce on 2.7.9

$ python --version
Python 2.7.9

$ python ./tests/runtests.py -v2  admin_views.tests.AdminViewBasicTest
# [snip]
...
test_sort_indicators_admin_order (admin_views.tests.AdminViewBasicTest) ... ok
Segmentation fault: 11

with faulthandler:

$  python ./tests/runtests.py -v2  admin_views.tests.AdminViewBasicTest
[snip]

Current thread 0x00007fff791a1300 (most recent call first):
  File "/Users/david/dev/django/django/db/backends/sqlite3/base.py", line 316 in execute
  File "/Users/david/dev/django/django/db/backends/utils.py", line 62 in execute
  File "/Users/david/dev/django/django/db/backends/base/base.py", line 214 in _savepoint_rollback
  File "/Users/david/dev/django/django/db/backends/base/base.py", line 254 in savepoint_rollback
  File "/Users/david/dev/django/django/db/transaction.py", line 252 in __exit__
  File "/Users/david/dev/django/django/test/testcases.py", line 950 in _rollback_atomics
  File "/Users/david/dev/django/django/test/testcases.py", line 1003 in _fixture_teardown
  File "/Users/david/dev/django/django/test/testcases.py", line 862 in _post_teardown
  File "/Users/david/dev/django/django/test/testcases.py", line 190 in __call__
  File "/Users/david/.pythonz/pythons/CPython-2.7.9/lib/python2.7/unittest/suite.py", line 108 in run
  File "/Users/david/.pythonz/pythons/CPython-2.7.9/lib/python2.7/unittest/suite.py", line 70 in __call__
  File "/Users/david/.pythonz/pythons/CPython-2.7.9/lib/python2.7/unittest/runner.py", line 151 in run
  File "/Users/david/dev/django/django/test/runner.py", line 178 in run_suite
  File "/Users/david/dev/django/django/test/runner.py", line 211 in run_tests
  File "./tests/runtests.py", line 256 in django_tests
  File "./tests/runtests.py", line 434 in <module>

[David Szotten]

a few more data points

can reproduce with this subset of tests (but removing any more make the run pass)

admin_views.tests.AdminViewBasicTest.test_allows_attributeerror_to_bubble_up admin_views.tests.AdminViewBasicTest.test_change_list_null_boolean_display admin_views.tests.AdminViewBasicTest.test_change_list_sorting_model_meta admin_views.tests.AdminViewBasicTest.test_change_view_with_show_delete_extra_context admin_views.tests.AdminViewBasicTest.test_changelist_with_no_change_url admin_views.tests.AdminViewBasicTest.test_disallowed_filtering admin_views.tests.AdminViewBasicTest.test_multiple_sort_same_field admin_views.tests.AdminViewBasicTest.test_popup_dismiss_related admin_views.tests.AdminViewBasicTest.test_proxy_model_content_type_is_used_for_log_entries admin_views.tests.AdminViewBasicTest.test_sort_indicators_admin_order

separately, setting TEST in DATABASES to use a sqlite db on disk instead of in ram seems to fix the issue

[Steve Jalim]

FWIW when I encountered this, I found that:

1) It took a non-small number of tests as part of the run to trigger it (I didn't get as far as a firm number)

2) my project hit the segfault regardless of in-memory or disk-based database

Hope that helps,

S

[Michael Kelly]

I ran into this as well and did a bunch of testing:

• I'm running on OS X Yosemite 10.10.2. I can consistently replicate by running admin_views.tests.AdminViewBasicTest on the latest master branch of Django.
• I was able to replicate with Python 2.7.9 and 3.4.1, both built with pyenv.
• I was able to replicate with the system python 2.7.6.
• I was unable to replicate with an older version of Python installed via homebrew, but that was because it didn't have sqllite built into Python.
• I was unable to replicate with a fresh install of Python 2.7.9 from homebrew. When I upgraded it also upgraded my homebrew-installed version of sqlite (or whatever it's called) to the latest version as well.

I suspect the issue is related to the sqlite version. I can't figure out which sqlite version pyenv is using when it builds python, but it might be useful for anyone affected to try re-building python with a recent version of sqlite to see if they still get the issue.

[Swapneel Patnekar]

If this can be of any help, I ran these tests on Mac OSX and Ubuntu Linux,

On Mac OSX with Python 3.4.3 inside a virtualenv,

$ python
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Ran 9273 tests in 260.431s

OK (skipped=603, expected failures=6)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

On Ubuntu Linux 14.04,

$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:38)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Ran 9275 tests in 231.808s

OK (skipped=548, expected failures=7)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

However, on Mac OSX using the built-in python 2.7.6,

............................................................................................<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.<model:person>:13: (WARNING/2) "raw" directive disabled.
<model:person>:16: (WARNING/2) "include" directive disabled.
.....................................................................................................................................................................................................................................[1] 52476 segmentation fault

Results :
Tests are successful on Mac OSX python 3.4.3 and Ubuntu Linux (14.04) 2.7.6.
Test fails on Mac OSX 2.7.6.

[Simon Charette]

From what I learned from the sprint it looks like homebrew builds Python with the currently installed version of SQLite hence the failures on different versions here.

What should be really useful to isolate the bug would be to try rebuilding Python with different versions of SQLite, create a virtualenv with this newly installed python interpreter and run something along those lines:

/path/to/intepreter/python -c"
import sqlite3
connection = sqlite3.connect(':memory:')
cursor = connection.cursor()
cursor.execute('SELECT sqlite_version()')
print(cursor.fetchone())
"

To try to figure out which version of SQLite provided by Homebrew we should advise not to build Python with.

[Matthew Somerville]

On Mac OS 10.9.5, using MacPorts, python 3.3.6, I get a segfault with SQLite 3.8.8.2, but not with SQLite 3.8.7.4. I had a segfault (during messages in my case, but it doesn't appear to be "precise test related"), switched with "port activate sqlite3 @3.8.7.4_0", and no longer had a segfault.

[Greg Chapple]

Did some git bisecting, and found the commit which introduced this issue: ​https://github.com/django/django/commit/da9fe5c717c179a9e881a40efd3bfe36a21bf4a6

Investigating a fix now, but not 100% sure on what's going on. Commenting out the calls to these two methods prevents the segfault, but triggers a whole ton of other issues: ​https://github.com/gchp/django/blob/master/django/test/testcases.py#L967-L981

[Tomáš Ehrlich]

I've found that commit too https://code.djangoproject.com/ticket/24080#comment:7, but don't know how to fix it. It seems it's related to python version on OS X

[Josep Cugat]

When building python with pyenv, it uses the system version of sqlite3 (3.8.5 in Yosemite), and the segmentation fault occurs.

To avoid this, install the latest version of sqlite from homebrew (3.8.10.2 at the time of this writing):

$ brew install sqlite
==> Downloading https://homebrew.bintray.com/bottles/sqlite-3.8.10.2.yosemite.bottle.tar.gz
######################################################################## 100.0%
==> Pouring sqlite-3.8.10.2.yosemite.bottle.tar.gz
==> Caveats
This formula is keg-only, which means it was not symlinked into /usr/local.

Mac OS X already provides this software and installing another version in
parallel can cause all kinds of trouble.

OS X provides an older sqlite3.

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

    LDFLAGS:  -L/usr/local/opt/sqlite/lib
    CPPFLAGS: -I/usr/local/opt/sqlite/include

==> Summary
🍺  /usr/local/Cellar/sqlite/3.8.10.2: 9 files, 2.8M

And before installing a new python version with pyenv, execute this (the paths are taken from the output above):

$ export LDFLAGS="-L/usr/local/opt/sqlite/lib"
$ export CPPFLAGS="-I/usr/local/opt/sqlite/include"
$ pyenv install 2.7.10

The compiled python interpreter will use the newer sqlite version from homebrew, and no more segfaults!

[Matt W]

So this seems to be hitting us too on our Ubuntu 14.04 build hosts. If we run the tests on our Ubuntu 12.04 hosts, we don't seem to have the problem. I've tried upgrading Sqlite to the latest available release (3.8.2) with no luck. I also tried to force the test DB runner to use disk-backed sqlite databases, but ran into issue ​https://github.com/django-nose/django-nose/issues/97.

Any news on this issue?

[CJ Avilla]

We're running into the same issue. Ubuntu 14.04. Python 2.7.6

Even after upgrading the sqlite package to 3.8.10.2 still having the same segmentation fault.

In [1]: import sqlite3
In [2]: sqlite3.version
Out[2]: '2.6.0'
In [3]: sqlite3.sqlite_version
Out[3]: '3.8.10.2'

[Denis Stebunov]

We're running into the same issue with our project tests suite. 100% reproducible. Django 1.8.3, Debian Jessie + Python 2.7.10, using Docker.

I tried:

• using latest pysqlite2 version 2.6.3 instead of sqlite3 version 2.6.0 from Python standard library;
• doubling RAM size for the virtual machine which is running tests;
• upgrading sqlite3 version from 3.8.7.1 to 3.8.11.1.

None of above worked for me. Finally, I tried running it from gdb and it shows Segmentation fault in sqlite3.c, but I have no idea what can I do with it:

(gdb) run manage.py test api.tests.test_member --settings api.settings
Starting program: /usr/local/bin/python manage.py test api.tests.test_member --settings api.settings
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20-gdb.py", line 59, in <module>
    from libstdcxx.v6.printers import register_libstdcxx_printers
  File "/usr/lib/x86_64-linux-gnu/../../share/gcc-4.9/python/libstdcxx/v6/printers.py", line 19, in <module>
    import itertools
ImportError: No module named itertools
Creating test database for alias 'default'...
/usr/local/lib/python2.7/site-packages/redis_cache/utils.py:4: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
  from django.utils.importlib import import_module

..
Program received signal SIGSEGV, Segmentation fault.
memjrnlRead (pJfd=0x158a058, zBuf=<optimized out>, iAmt=4, iOfst=98552)
    at sqlite3.c:82326
82326	    memcpy(zOut, &pChunk->zChunk[iChunkOffset], nCopy);

[Mark Beacom]

I encountered this error on an Ubuntu 12.04 server running Python 2.7.3 with Django 1.8.3 during jenkins testing.
I attempted to resolve the issue by updating Python to 2.7.10. Due to my server running 12.04, I was unable to upgrade to the latest SQLite3 (sqlite3.sqlite_version: '3.8.11.1') via APT due to the package restrictions.

Ultimately, I resolved the issue by upgrading manually via SQLite 3.8.11.1 source (​https://www.sqlite.org/2015/sqlite-autoconf-3081101.tar.gz) and apt-get install --reinstall'ing python 2.7.10!

You can install SQLite 3.8.11.1 by building from the previously mentioned source .tar.gz:

$ wget https://www.sqlite.org/2015/sqlite-autoconf-3081101.tar.gz
$ tar xvfz sqlite-autoconf-3081101.tar.gz
$ cd sqlite-autoconf-3081101
$ ./configure --prefix=/usr/local
$ make
$ make install
$ sudo apt-get install --reinstall python2.7 python2.7-minimal python2.7-dev python2.7-dbg

[Matt Davis]

Our software organization is also having the same issues outlined in this bug report, and I wanted to expand on my own findings.

The issue: Segfault in running unit tests which haven't changed in months to years when upgrading from Django 1.6.x to Django 1.8.x

Running faulthandler prior to py.test reveals the same kind of stack trace everyone is posting which is essentially that in the new _rollback_atomics code that happens it causes the seg-fault. I break-pointed some of the code and generally the error happens for us when we catch an IntegrityError and handle it outside of an atomic() block. The thing is running the same test in isolation allows the test it segfaulted on to pass, but running in the group it causes the failure.

I have experienced the failure originally on:
Ubuntu 14.04 LTS with Python 2.7 and sqlite3 3.8.7.1.
Upgraded sqlite3 3.8.7.1 to 3.8.11.1 as outlined by stebunovd, but issue persisted.
Installed Ubuntu 15.04 w/ Python 2.7 and sqlite3 3.8.7.1 (default) to a VM and the issue persisted.
Upgraded Ubuntu 15.04 sqlite3 3.8.7.1 to 3.8.11.1 as outlined by stebunovd, but issue persisted.

In both cases I ensured that the old version of sqlite3 was removed and python was rebuilt after the CLI returned the updated version info for sqlite3.

One interesting point to note is that upgrading sqlite3 did postpone the segmentation fault until later in the test suite, but it too happened in the same manner suggesting that the new version doesn't actually fix the problem here.

[Matt Davis]

Just wanted to include also my stack trace from faulthandler running on the latest everything 15.04:

Current thread 0x00007f6d07308700 (most recent call first):

File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 316 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 62 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 214 in _savepoint_rollback
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254 in savepoint_rollback
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/db/transaction.py", line 252 in exit
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/test/testcases.py", line 941 in _rollback_atomics
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/test/testcases.py", line 998 in _fixture_teardown
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/test/testcases.py", line 853 in _post_teardown
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/django/test/testcases.py", line 189 in call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/unittest.py", line 137 in runtest
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 90 in pytest_runtest_call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 394 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 123 in init
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 107 in wrapped_call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 393 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 528 in _docall
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 521 in call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 137 in <lambda>
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 149 in init
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 137 in call_runtest_hook
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 119 in call_and_report
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 75 in runtestprotocol
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/runner.py", line 65 in pytest_runtest_protocol
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 394 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 123 in init
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 107 in wrapped_call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 393 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 528 in _docall
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 521 in call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/main.py", line 142 in pytest_runtestloop
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 394 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 528 in _docall
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 521 in call
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/main.py", line 122 in _main
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/main.py", line 84 in wrap_session
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/main.py", line 116 in pytest_cmdline_main
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 394 in execute
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 528 in _docall
File "/home/matt/.virtualenvs/order/local/lib/python2.7/site-packages/_pytest/core.py", line 521 in call
File "/home/matt/.pycharm_helpers/pycharm/pytestrunner.py", line 35 in main
File "/home/matt/.pycharm_helpers/pycharm/pytestrunner.py", line 60 in <module>

bash: line 1: 4383 Segmentation fault (core dumped) env

[Ned Batchelder]

I am also encountering this error. In my case, there are three specific tests, which if I skip, avoid the problem. It is entirely reproducible, the same test will segfault every time if I don't skip it.

I'm using Python 2.7.3 on ubuntu 12.04, with sqlite.version == "2.6.0" and sqlite.sqlite_version == "3.7.9"

[Josh Smeaton]

Hi Ned,

Are you able to show us an entire test that is causing problems? It might help if we had examples to look at.

[Lee Semel]

This is affecting us also, randomly dying during our automated tests. It is entirely repeatable, but unpredictable where it will break. It happens both locally on our Macs (sqlite v3.7.13) and on Linux. Upgrading to sqlite 3.8 just makes it worse. It happens between tests, where Django tries to rollback the test transaction in _rollback_atomics. The only reliable solution is to stop using Django's TestCase class and instead use the one from unittest, manually deleting in tearDown() any models that the test creates.

[Ned Batchelder]

@jarshwah sorry, didn't see your earlier comment. The test suite is on Open edX, which is one of those unwieldy projects that are hard to hand over as reproducible test cases. I can walk you through how to get it set up in a Vagrant instance if you want, it isn't *that* hard.

[Alex Morozov]

Just to confirm the @jcugat solution: I was having runtests.py crashing with cryptic errors on OS X 10.11.1. After installing a new sqlite version via brew, errors are gone.

[Alex Riina]

I ran into this after an update to django1.8.6. I was able to trace it down to a test that wasn't crashing but was missing a super(MyTest, cls).tearDownClass(). The test that actually crashed used override_settings and adding either the super tearDownClass or removing the override_settings resolved the issue for me.

Unfortunately, coming up with an example is proving hard because I'm only seeing the segfault much later (I don't see the error when running the test with a missing tearDownClass and then the test with override_settings even though I see the error when I run the full suite).

class MyTest(TestCase):
    @classmethod
    def tearDownClass(cls):
        pass

...

@override_settings(...)
class MyOtherTest(TestCase):
    pass

I'll try to distill this to a full example.

[Tim Graham]

#25860 seems to be a duplicate and also points to a case where django.test.TestCase fails to be called (if skipTest is raised in setUpClass(). The original report is about a regression in 1.7.2 but the subsequent reports seem to be for 1.8 and related to setUpTestData() so it seems we should retitle this issue, create a new one, or use #25860 (closing this one as needsinfo since we still don't have anything to reproduce a problem on 1.7.2). It's not immediately to me if/how we'd fix the problem since it's somewhat related to user code, but maybe we can at least do better to detect the situation or document it.

[Martijn Jacobs]

Just a small note as I also ran into this issue (an exception in backends/sqlite3/base.py when running tests with nose) It's difficult to exactly pinpoint versions of python/sqlite/django as it's behaving differently on different platforms.

However, I managed to fix the segmentation fault by changing the TestCase as following:

from django.test import TransactionTestCase as TestCase

class MyTestCase(TestCase):
    """My segfaulting testcase"""
    cleans_up_after_itself = True

So I suspect something with cleaning up after a testcase is finished. (See also ​http://django-nose.readthedocs.org/en/latest/usage.html#escaping-the-grime)

[Antti Ruotsalainen]

Were able to fix this issue by calling 'flush' command in tearDown. This behaviour was taken from TransactionTestCase._fixture_teardown()

call_command('flush', verbosity=0, interactive=False,
                         reset_sequences=False,
                         allow_cascade=False,
                         inhibit_post_migrate=False)

[Pieter De Decker]

As far as I can tell, all the fixes in this thread are workarounds that - at least temporarily - keep the segmentation fault from manifesting itself while not addressing the root cause.

Now, I don't have a solution myself. But I just want to point out that creating a dummy boolean value to trigger a subtle change in the way memory is allocated does not guarantee that the bug won't be triggered again and that it does nothing to address the underlying issue.

[Aymeric Augustin]

Since Django doesn't contain any C code, a segfault must be a bug in Python or in another third-party software. The bug can be caused by the software itself or by modifications performed by distributors.

The purpose of this ticket is to identify the issue more precisely, so it can be reported upstream, and to identify workarounds in the meantime. It's highly unlikely to result in changes to Django.

[Raphaël Hertzog]

Since the switch to SQLite 3.12 I can reproduce a similar segfault (same stacktrace output by faulthandler) with Django 1.9.5 and Python 2.7.11 or 3.5.1.

$ cd tests
$ PYTHONPATH=.. python3 -X faulthandler ./runtests.py --verbosity=2 --parallel=1 admin_views.tests.AdminViewBasicTest
[...]
test_popup_dismiss_related (admin_views.tests.AdminViewBasicTest) ... ok
Fatal Python error: Segmentation fault

Current thread 0x00007fab40575700 (most recent call first):
  File "/home/rhertzog/deb/pkg/python-django/django/db/backends/sqlite3/base.py", line 321 in execute
  File "/home/rhertzog/deb/pkg/python-django/django/db/backends/utils.py", line 62 in execute
  File "/home/rhertzog/deb/pkg/python-django/django/db/backends/base/base.py", line 288 in _savepoint_rollback
  File "/home/rhertzog/deb/pkg/python-django/django/db/backends/base/base.py", line 328 in savepoint_rollback
  File "/home/rhertzog/deb/pkg/python-django/django/db/transaction.py", line 243 in __exit__
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 1016 in _rollback_atomics
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 1073 in _fixture_teardown
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 919 in _post_teardown
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 217 in __call__
  File "/usr/lib/python3.5/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.5/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.5/unittest/runner.py", line 176 in run
  File "/home/rhertzog/deb/pkg/python-django/django/test/runner.py", line 494 in run_suite
  File "/home/rhertzog/deb/pkg/python-django/django/test/runner.py", line 533 in run_tests
  File "./runtests.py", line 275 in django_tests
  File "./runtests.py", line 458 in <module>

Maybe SQLite 3.12 makes it more likely to trigger to the problem or something like that...

[Pieter De Decker]

Related? ​https://www.sqlite.org/src/info/7f7f8026eda38

[Aymeric Augustin]

Quite likely, but that isn't confirmed yet.

[Denis Stebunov]

Colleague of mine discovered a workaround - try renaming test methods to change their execution order. For example, rename the method on which Segmentation fault occurs to be the first in its test class (in alphabetical order). Sounds weird, but it worked both for him and for me.

[Iacopo Spalletti]

My experience with this issue is mostly related to ManyToMany fields.
I even traced all the SQL queries, and apparently sqlite does not handle the cleanup phase well and segfaults when truncating the data when destroying database. I can try reproduce this and post the log, if this can be useful.
I managed to solve that by clearing many to many fields in tearDown:

    def tearDown(self):
        self.user.sites.clear()
        for post in Post.objects.all():
            post.sites.clear()
            post.tags.clear()
        super(BaseTest, self).tearDown()

[Jason Morrison]

I also see this issue (Django 1.9.9, OSX, Python 2.7.11) and confirm that updating from OSX built-in sqlite3 3.8.5 to homebrew sqlite3 3.14.0 fixes this issue, as well as fixes the sqlite repro case detailed in ​https://www.sqlite.org/src/info/7f7f8026eda38

My shell notes: ​https://gist.github.com/jasonm/5aa9406c88e29e3d550c1b42f9435cf2

[Tim Graham]

I think we can close this issue as nothing indicates a problem in Django itself and SQLite has made a fix that seems to help.

[Zia Fazal]

On a linux distribution(ubunutu) this gist can help installing latest(3.14) sqlite.​https://gist.github.com/ziafazal/cf2b10b38b39e1795d5d8eaa993760ce