AttributeError: '_Frame' object has no attribute 'f_back' / Django and celery error logging gone wrong

[Simon Chenard]

When an error is raised inside a celery task, django crashes with the error described in the summary. This error is not raised anywhere else in the application, only in a celery task. Came across this situation trying to understand why I was not getting error emails for celery tasks. Triggered this exception by adding :

CELERY_WORKER_HIJACK_ROOT_LOGGER = False

Implying otherwise the error is not redirected to the AdminEmailHandler.

Django version : 1.10.2
Celery version : 4.0.0

(stacktrace is below)

The method "get_traceback_frame_variables" is the cause of this problem. By wrapping a try except around the code of the method and returning an empty array, I roughly get what I expect : an email describing the error with the adequate traceback.

My logging setup :

LOGGING = {

'version': 1,
'disable_existing_loggers': False,
'filters': {

'task_id': {

'()': 'lib.logging.task_filter.TaskIDFilter'

},
'require_debug_false': {

'()': 'django.utils.log.RequireDebugFalse'

}

},
'formatters' : {

'task': {

'format': LOGGING_TASK_FORMAT

},

},
'handlers': {

'mail_admins': {

'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler',
'include_html': True,
'filters': require_debug_false, # commented or not, the error happens with DEBUG = True or False

},

},
'loggers': {

: {

'handlers': ['console', 'mail_admins'],
'level': 'INFO',
'propagate': True

},

}

}

Here is the stacktrace :

Traceback (most recent call last):

File "/home/simon/code/caravan/fibre/env-aws/lib/python3.4/site-packages/celery/app/trace.py", line 382, in trace_task

I, R, state, retval = on_error(task_request, exc, uuid)

File "/home/simon/code/caravan/fibre/env-aws/lib/python3.4/site-packages/celery/app/trace.py", line 324, in on_error

task, request, eager=eager, call_errbacks=call_errbacks,

File "/home/simon/code/caravan/fibre/env-aws/lib/python3.4/site-packages/celery/app/trace.py", line 158, in handle_error_state

call_errbacks=call_errbacks)

File "/home/simon/code/caravan/fibre/env-aws/lib/python3.4/site-packages/celery/app/trace.py", line 212, in handle_failure

self._log_error(task, req, einfo)

File "/home/simon/code/caravan/fibre/env-aws/lib/python3.4/site-packages/celery/app/trace.py", line 242, in _log_error

extra={'data': context})

File "/usr/lib/python3.4/logging/init.py", line 1341, in log

self._log(level, msg, args, kwargs)

File "/usr/lib/python3.4/logging/init.py", line 1409, in _log

self.handle(record)

File "/usr/lib/python3.4/logging/init.py", line 1419, in handle

self.callHandlers(record)

File "/usr/lib/python3.4/logging/init.py", line 1481, in callHandlers

hdlr.handle(record)

File "/usr/lib/python3.4/logging/init.py", line 853, in handle

self.emit(record)

File "/home/simon/code/caravan/fibre/env-aws/lib/python3.4/site-packages/django/utils/log.py", line 119, in emit

message = "%s\n\n%s" % (self.format(no_exc_record), reporter.get_traceback_text())

File "/home/simon/code/caravan/fibre/env-aws/lib/python3.4/site-packages/django/views/debug.py", line 325, in get_traceback_text

c = Context(self.get_traceback_data(), autoescape=False, use_l10n=False)

File "/home/simon/code/caravan/fibre/env-aws/lib/python3.4/site-packages/django/views/debug.py", line 264, in get_traceback_data

frames = self.get_traceback_frames()

File "/home/simon/code/caravan/fibre/env-aws/lib/python3.4/site-packages/django/views/debug.py", line 420, in get_traceback_frames

'vars': self.filter.get_traceback_frame_variables(self.request, tb.tb_frame),

File "/home/simon/code/caravan/fibre/env-aws/lib/python3.4/site-packages/django/views/debug.py", line 191, in get_traceback_frame_variables

current_frame = tb_frame.f_back

AttributeError: '_Frame' object has no attribute 'f_back'