running tests in parallel doesn't show exception chain, even with tblib

[Chris Jerdonek]

Running tests in parallel doesn't show more than the first link of an exception chain. For example, with the following test (with tblib installed):

def test(self):
    time.sleep(2)
    try:
        raise ValueError('foo')
    except Exception:
        raise KeyError('bar')

this is what the failure looks like when running tests in parallel:

Traceback (most recent call last):
  File "/Users/.../versions/3.6.4/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/Users/.../versions/3.6.4/lib/python3.6/unittest/case.py", line 605, in run
    testMethod()
  File "/Users/.../myproject/myproject/tests/test_a.py", line 13, in test2
    raise KeyError('bar')
KeyError: 'bar'

In contrast, this is what it looks like when running them not in parallel:

Traceback (most recent call last):
  File "/Users/.../myproject/myproject/tests/test_a.py", line 11, in test2
    raise ValueError('foo')
ValueError: foo

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/.../myproject/myproject/tests/test_a.py", line 13, in test2
    raise KeyError('bar')
KeyError: 'bar'

Side question: why must pickling be used at all? Why can't we simply render the traceback to a string using Python's traceback module, and then send the string back? It seems like that would result in simpler, more understandable code and sidestep many of the difficulties.

(This was reported using Python 3.6.4. and master as of Jan. 15, 2018: 02365d3f38a64a5c2f3e932f23925a381d5bb151 .)

[Chris Jerdonek]

One more thing to add: in the case where the full chain isn't being shown, there's also no way for the user to know that anything is missing (without also running in non-parallel mode and comparing the test outputs).

[Simon Charette]

Side question: why must pickling be used at all? Why can't we simply render the traceback to a string using Python's traceback module, and then send the string back? It seems like that would result in simpler, more understandable code and sidestep many of the difficulties.

From what I know It's just the default behavior of the multiprocessing module for inter process exception bubbling. I feel like it's a bit harder than that as you still have to construct a traceback object from the string on the parent process side to raise the appropriate exception.

[Chris Jerdonek]

For future reference, here is one possible work-around that I think would be better than the status quo. If an exception raised during a test either (1) fails the "pickleable test," or (2) has a chain of length bigger than one (e.g. non-None __cause__ attribute), then the traceback could be formatted to a string in the child process and a pickleable exception of the form UnpickleableTestException(traceback) could be sent back to the parent process. The end result won't look as clean as the pickled case, but at least the information of the full exception chain should display. Also, in the case of unpickleable exceptions, it should prevent the test run from aborting completely. So it would address two issues.

[Chris Jerdonek]

FYI, I implemented a proof-of-concept for the idea I described in my previous comment and posted it here: ​https://github.com/cjerdonek/django/tree/ticket_29023

In particular, it handles both chained exceptions and non-picklable exceptions, and without aborting the test run in the latter case.

I included example "before" and "after" outputs for both cases of chained exceptions and non-picklable exceptions in the commit message of this commit: ​https://github.com/cjerdonek/django/commit/0040040ca4fec08a9980a24f08bcad900e2b0f22