running servers.tests may hang in parallel mode on Mac OS X

[Chris Jerdonek]

From master (97513269d73520d10722bbd10404be6ac4d48d07), running the following command from the tests directory hangs for me after a couple tests:

$ ./runtests.py servers.tests

This is with Python 3.5.2 on Mac OSX.

There is no error message. It just hangs. It works okay when passing --parallel 1.

[Tim Graham]

I can't reproduce on Ubuntu 14.04, Python 3.5.2. Just in case, how many parallel process do you have? I have 4 here.

[Chris Jerdonek]

8, but it also happens when I pass --parallel 2. To narrow it down, it happens when I run the following (I believe the test setup executes, but it never gets to running the tests):

$ ./runtests.py servers.tests.LiveServerDatabase servers.tests.LiveServerViews

[Tim Graham]

You could check if the issue is present at the time that parallel test running was added in Django and if not, bisect to find the commit that introduced the problem.

[Bas Peschier]

Just ran the tests at the mentioned commit on my macOS Sierra public beta with a fresh 3.5.2 python environment. No problems there.

[Tim Graham]

Feel free to reopen if you find that Django is at fault.

[Chris Jerdonek]

I narrowed this down further to the following:

The hang happens when calling urllib.request.urlopen() inside LiveServerBase. And digging further, inside CPython's request.py, the hang happens when calling _scproxy._get_proxies() (see ​here for the code).

There seem to be known issues around _scproxy in CPython for Mac OS X users. For example, see the following two issues in CPython's bug tracker:

(1) Apple-supplied libsqlite3 on OS X is not fork safe; can cause crashes:

​http://bugs.python.org/issue27126

(See this comment, in particular: ​http://bugs.python.org/issue27126#msg266405 )

(2) exception error in _scproxy.so when called after fork

​http://bugs.python.org/issue13829

There is a chance that there are easy workarounds on the Django side that can be made to accommodate Mac OS X users.

Also, for the record I'm using 10.11.6.

[Chris Jerdonek]

I was able to get things working by causing Mac OS X's auto-detection of proxy information to be skipped (so that _scproxy._get_proxies() is not called). This possibility is mentioned ​here in CPython's urllib.request.ProxyHandler documentation.

More detailed instructions appear in urllib.request's module docstring (see ​here for that part of the docstring).

To disable the auto-detection, I followed the instructions in the above-mentioned module docstring and modified LiveServerBase.urlopen() to look like the following (this is just a proof of concept):

def urlopen(self, url):
    import urllib.request
    # Pass an empty dictionary to disable auto-detecting the proxy.
    proxy_support = urllib.request.ProxyHandler({})
    opener = urllib.request.build_opener(proxy_support)
    urllib.request.install_opener(opener)

    return urlopen(self.live_server_url + url)

After doing this, the following command worked fine:

$ ./runtests.py servers.tests

Since this seems like a fundamental bug / limitation on Mac OS X (in at least some versions), it seems worth considering including Mac-specific code to enable the parallel running of tests on this platform.

[Tim Graham]

I wonder what's special about your system compared to other Mac OS contributors who didn't encounter the problem.

[Chris Jerdonek]

I don't know yet why I'm the first person to notice / report this. In case it would help, I started working on a reduced way to reproduce this (independent of the Django test suite).

​http://www.evanjones.ca/fork-is-dangerous.html

[Chris Jerdonek]

Oh, the person that wrote the blog post already created a "reduced test case" that I started to look for: ​https://github.com/evanj/fork-thread-bugs/blob/master/osx_python_crash_v1.py

[Kees Hink]

Here is a documentation improvement showing a workaround: ​https://github.com/django/django/pull/11203

[Tim Bell]

I have encountered the same problem (on MacOS 10.14.4 Mojave) and confirm that the solution works for me. This would be a useful addition to the documentation.

[Carlton Gibson]

Closing as per discussion on PR: lack of a decent reproduce, plenty of Google results for the error, not our responsibility, and likely a problem that's been resolved in recent releases (but c.f. lack of reproduce...)

[Carlton Gibson]

I'm at the sprints at DjangoCon. Many new contributors have hit this, so I think the doesn't reproduce worry is passed. I'm going to re-open this accordingly. We should doc so new contributors are able to get going.

Related is #30806 (which is targeting running tox specifically.)

[Carlton Gibson]

I just want to adjust this, so moving to Accepted and assigning to myself.

[Carlton Gibson <carlton@…>]

In 5d9cf79b:

Fixed #27086 -- Doc'd fix for test suite hang on macOS.

[Carlton Gibson <carlton.gibson@…>]

In fffce1a:

[3.0.x] Fixed #27086 -- Doc'd fix for test suite hang on macOS.

Backport of 5d9cf79baf07fc4aed7ad1b06990532a65378155 from master

[Carlton Gibson <carlton.gibson@…>]

In 4f7ba25e:

[2.2.x] Fixed #27086 -- Doc'd fix for test suite hang on macOS.

Backport of 5d9cf79baf07fc4aed7ad1b06990532a65378155 from master