Django wsgi.py segfaults Apache

[harel@…]

After a very standard and minor release, our Apache/Nginx/Django/wsgi setup ceased to function with the only error "Premature end of script headers: django.wsgi" in the apache log.
There is no expat problem (checked with sample wsgi app with and without that module include). The server was working fine. The codebase was updated slightly and apache restarted when the horrors began. The codebase was rolled back and still the problem persists.
Below are my wsgi file, apache config and a dump from running wsgi under the debuger.

I will have to rebuilt the server very soon, as the problem just won't go away. I am holding off in case anyone here finds this interesting and needs more information.

The system is Debian Squeeze, Apache/2.2.16 (Debian), Python 2.6.4, Django 1.3.1. modwsgi 3.3-2
All packages are installed via apt-get/pip. Another server with similar setup doesn't exhibit the problem.

wsgi file:
import sys
import os

sys.path.append('/home/app/')
sys.path.append('/home/app/topmodule/')

os.environDJANGO_SETTINGS_MODULE = 'app.settings'
os.environCELERY_LOADER = "django"

import app.settings
import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

apache config:
WSGIScriptAlias / /home/app/topmodule/django.wsgi
WSGIDaemonProcess app user=app group=app processes=6 threads=1
WSGIProcessGroup app
Have also tried with :
WSGIApplicationGroup %{GLOBAL}

Django Middleware:
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'glowconsole.middleware.ViewNameMiddleware',
'django.middleware.gzip.GZipMiddleware'

Debugger dump when running apache2 -X with the pdb debugger wrapper (also here: ​http://paste.pocoo.org/show/546803/)
batman:/home/harel# /usr/sbin/apache2 -X
['/usr/local/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg', '/usr/local/lib/python26.zip', '/usr/local/lib/python2.6', '/usr/local/lib/python2.6/plat-linux2', '/usr/local/lib/python2.6/lib-tk', '/usr/local/lib/python2.6/lib-old', '/usr/local/lib/python2.6/lib-dynload', '/usr/local/lib/python2.6/site-packages', '/usr/local/lib/python2.6/site-packages/PIL', '/home/glowconsole/', '/home/glowconsole/glowconsole/', '/usr/local/lib/python2.6/site-packages/django']

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(240)call()

-> from django.conf import settings
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(244)call()

-> if self._request_middleware is None:
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(245)call()

-> self.initLock.acquire()
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(246)call()

-> try:
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(247)call()

-> try:
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(249)call()

-> if self._request_middleware is None:
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(250)call()

-> self.load_middleware()
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(256)call()

-> self.initLock.release()
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(258)call()

-> set_script_prefix(base.get_script_name(environ))
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(259)call()

-> signals.request_started.send(sender=self.class)
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(260)call()

-> try:
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(261)call()

-> try:
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(262)call()

-> request = self.request_class(environ)
(Pdb) n

/usr/local/lib/python2.6/site-packages/django/core/handlers/wsgi.py(272)call()

-> response = self.get_response(request)
(Pdb) l
267 'status_code': 400,
268 }
269 )
270 response = http.HttpResponseBadRequest()
271 else:
272 -> response = self.get_response(request)
273 finally:
274 signals.request_finished.send(sender=self.class)
275
276 try:
277 status_text = STATUS_CODE_TEXT[response.status_code]
(Pdb) n
Segmentation fault

Some var dumps from that debugger session (also here: ​http://paste.pocoo.org/show/546846/):
(Pdb) request
<WSGIRequest
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{'csrftoken': '6f41c55c934085b286fccb834e910643',

'sessionid': '58a6595013bf69a456a532b318446315',
'ys-account_id': 'n%3A585',
'ys-ad_preview_window-1332956': 'o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272',
'ys-client_id': 'n%3A202',
'ys-title': 's%3ASlots%20Farm%20-CA%205'},

META:{'DOCUMENT_ROOT': '/etc/apache2/htdocs',

'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
'HTTP_ACCEPT_LANGUAGE': 'en-GB,en-US;q=0.8,en;q=0.6',
'HTTP_CACHE_CONTROL': 'max-age=0',
'HTTP_CONNECTION': 'close',
'HTTP_COOKIE': 'ys-ad_preview_window-1332956=o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272; ys-account_id=n%3A585; ys-client_id=n%3A202; ys-title=s%3ASlots%20Farm%20-CA%205; csrftoken=6f41c55c934085b286fccb834e910643; sessionid=58a6595013bf69a456a532b318446315',
'HTTP_HOST': 'batman:8100',
'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11',
'HTTP_X_FORWARDED_FOR': '217.138.9.130',
'PATH_INFO': u'/',
'PATH_TRANSLATED': '/home/glowconsole/glowconsole/django.wsgi/',
'QUERY_STRING': ,
'REMOTE_ADDR': '81.94.195.18',
'REMOTE_PORT': '54800',
'REQUEST_METHOD': 'GET',
'REQUEST_URI': '/',
'SCRIPT_FILENAME': '/home/glowconsole/glowconsole/django.wsgi',
'SCRIPT_NAME': u,
'SERVER_ADDR': '81.94.195.18',
'SERVER_ADMIN': 'harel@…',
'SERVER_NAME': 'batman',
'SERVER_PORT': '8100',
'SERVER_PROTOCOL': 'HTTP/1.0',
'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at batman Port 8100</address>\n',
'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)',
'mod_wsgi.application_group': 'theglowmachine.com:8100|',
'mod_wsgi.callable_object': 'application',
'mod_wsgi.handler_script': ,
'mod_wsgi.input_chunked': '0',
'mod_wsgi.listener_host': 'batman',
'mod_wsgi.listener_port': '8100',
'mod_wsgi.process_group': ,
'mod_wsgi.request_handler': 'wsgi-script',
'mod_wsgi.script_reloading': '1',
'mod_wsgi.version': (3, 3),
'wsgi.errors': <mod_wsgi.Log object at 0x7f4f886a7df0>,
'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f4f87277990>,
'wsgi.input': <mod_wsgi.Input object at 0x7f4f886a7db0>,
'wsgi.multiprocess': True,
'wsgi.multithread': True,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 1)}>

(Pdb) locals()
{'start_response': <built-in method start_response of mod_wsgi.Adapter object at 0x7f4f87277990>, 'request': <WSGIRequest
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{'csrftoken': '6f41c55c934085b286fccb834e910643',

'sessionid': '58a6595013bf69a456a532b318446315',
'ys-account_id': 'n%3A585',
'ys-ad_preview_window-1332956': 'o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272',
'ys-client_id': 'n%3A202',
'ys-title': 's%3ASlots%20Farm%20-CA%205'},

META:{'DOCUMENT_ROOT': '/etc/apache2/htdocs',

'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
'HTTP_ACCEPT_LANGUAGE': 'en-GB,en-US;q=0.8,en;q=0.6',
'HTTP_CACHE_CONTROL': 'max-age=0',
'HTTP_CONNECTION': 'close',
'HTTP_COOKIE': 'ys-ad_preview_window-1332956=o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272; ys-account_id=n%3A585; ys-client_id=n%3A202; ys-title=s%3ASlots%20Farm%20-CA%205; csrftoken=6f41c55c934085b286fccb834e910643; sessionid=58a6595013bf69a456a532b318446315',
'HTTP_HOST': 'batman:8100',
'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11',
'HTTP_X_FORWARDED_FOR': '217.138.9.130',
'PATH_INFO': u'/',
'PATH_TRANSLATED': '/home/glowconsole/glowconsole/django.wsgi/',
'QUERY_STRING': ,
'REMOTE_ADDR': '81.94.195.18',
'REMOTE_PORT': '54800',
'REQUEST_METHOD': 'GET',
'REQUEST_URI': '/',
'SCRIPT_FILENAME': '/home/glowconsole/glowconsole/django.wsgi',
'SCRIPT_NAME': u,
'SERVER_ADDR': '81.94.195.18',
'SERVER_ADMIN': 'harel@…',
'SERVER_NAME': 'batman',
'SERVER_PORT': '8100',
'SERVER_PROTOCOL': 'HTTP/1.0',
'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at batman Port 8100</address>\n',
'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)',
'mod_wsgi.application_group': 'theglowmachine.com:8100|',
'mod_wsgi.callable_object': 'application',
'mod_wsgi.handler_script': ,
'mod_wsgi.input_chunked': '0',
'mod_wsgi.listener_host': 'batman',
'mod_wsgi.listener_port': '8100',
'mod_wsgi.process_group': ,
'mod_wsgi.request_handler': 'wsgi-script',
'mod_wsgi.script_reloading': '1',
'mod_wsgi.version': (3, 3),
'wsgi.errors': <mod_wsgi.Log object at 0x7f4f886a7df0>,
'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f4f87277990>,
'wsgi.input': <mod_wsgi.Input object at 0x7f4f886a7db0>,
'wsgi.multiprocess': True,
'wsgi.multithread': True,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 1)}>, 'self': <django.core.handlers.wsgi.WSGIHandler object at 0x7f4f886a7d90>, 'environ': {'mod_wsgi.listener_port': '8100', 'HTTP_COOKIE': 'ys-ad_preview_window-1332956=o%3Awidth%3Dn%253A532%5Eheight%3Dn%253A472%5Ex%3Dn%253A455%5Ey%3Dn%253A272; ys-account_id=n%3A585; ys-client_id=n%3A202; ys-title=s%3ASlots%20Farm%20-CA%205; csrftoken=6f41c55c934085b286fccb834e910643; sessionid=58a6595013bf69a456a532b318446315', 'mod_wsgi.listener_host': 'batman', 'SERVER_SOFTWARE': 'Apache/2.2.16 (Debian)', 'SCRIPT_NAME': u, 'mod_wsgi.handler_script': , 'SERVER_SIGNATURE': '<address>Apache/2.2.16 (Debian) Server at batman Port 8100</address>\n', 'REQUEST_METHOD': 'GET', 'PATH_INFO': u'/', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': , 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3', 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11', 'HTTP_CONNECTION': 'close', 'SERVER_NAME': 'batman', 'REMOTE_ADDR': '81.94.195.18', 'mod_wsgi.request_handler': 'wsgi-script', 'wsgi.url_scheme': 'http', 'PATH_TRANSLATED': '/home/glowconsole/glowconsole/django.wsgi/', 'SERVER_PORT': '8100', 'wsgi.multiprocess': True, 'mod_wsgi.input_chunked': '0', 'SERVER_ADDR': '81.94.195.18', 'DOCUMENT_ROOT': '/etc/apache2/htdocs', 'mod_wsgi.process_group': , 'SCRIPT_FILENAME': '/home/glowconsole/glowconsole/django.wsgi', 'SERVER_ADMIN': 'harel@…', 'wsgi.input': <mod_wsgi.Input object at 0x7f4f886a7db0>, 'HTTP_HOST': 'batman:8100', 'wsgi.multithread': True, 'mod_wsgi.callable_object': 'application', 'HTTP_CACHE_CONTROL': 'max-age=0', 'REQUEST_URI': '/', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'wsgi.version': (1, 1), 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_X_FORWARDED_FOR': '217.138.9.130', 'wsgi.errors': <mod_wsgi.Log object at 0x7f4f886a7df0>, 'REMOTE_PORT': '54800', 'HTTP_ACCEPT_LANGUAGE': 'en-GB,en-US;q=0.8,en;q=0.6', 'mod_wsgi.version': (3, 3), 'wsgi.run_once': False, 'mod_wsgi.application_group': 'theglowmachine.com:8100|', 'mod_wsgi.script_reloading': '1', 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f4f87277990>, 'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch'}, 'settings': <django.conf.LazySettings object at 0x7f4f721f6a50>}