LiveServerTestCase prematurely closing HttpResponse with Django 2.0

[Alexander Todorov]

I'm trying to migrate an existing project to Django 2.0 and I'm hitting a problem with some of our tests, all of the problematic tests are very similar in nature:

1) They are all LiveServerTestCase based
2) All of them have a XML-RPC client which issues requests to the Django app
3) Using django-modern-rpc the application under test is able to execute a function and return a result
3) All error out on the first request (most of the times) with traceback from the xmlrpc client module in Python telling me that that connection has been closed without a response.

Here are the few data points I was able to collect:

• Locally happens on both MySQL and SQLite but on SQLite less frequently, looks like a race condition. In Travis CI happens on SQLite but not on MySQL so go figure!
• with a local MySQL instance I'm able to reproduce all the times
• the root-cause seems to be in wsgiref.handlers.py::BaseHandler.run() which calls finish_response() which executes HttpResponse.close(). In run() there's also a comment about async servers and that they should not call .close() inside .finish_response()

• if I modify Django's basehttp.py::ServerHandler.http_version to '1.0' all tests seem to pass regardless if we use a multi-threaded or single-threaded LiveServer!

• This is on Python 3.5, RHEL 7 system.

So it looks like the problem is with ServerHandler (presumably tests) wanting to use HTTP 1.1 but I can't narrow it down firther ATM.

To reproduce you may checkout the code at ​https://github.com/kiwitcms/Kiwi;
pip install -r requirements/devel.txt; pip install --upgrade Django and

./manage.py test --noinput --settings=tcms.settings.test.mysql tcms.xmlrpc.tests.test_logging.TestXMLRPCLogging.test_logging_with_authenticated_user

[Tim Graham]

It sounds similar to ac756f16c5bbbe544ad82a8f3ab2eac6cccdb62e. I'm not sure if anyone will be interested in debugging your project to find if Django is at fault.

[Tim Graham]

Please reopen if you can explain why Django is at fault.

[Alexander Todorov]

@Tom Forbes,
thanks for the hint, I've also been looking around the sme parts of the code. Reverting the commit you listed above and testing with ServerHandler.http_version = '1.1' makes all of my tests to PASS but './manage.py test' doesn't exit!.

All of my responses also include the Content-Length header. I will let you know when I find more.

[Alexander Todorov]

I think there's a bug inside Python's xmlrpc.client.Transport class which isn't playing nicely when Django closes the connection when HTTP 1.1 is in use. In particular this piece of code in xmlrpc/client.py

   1129     def request(self, host, handler, request_body, verbose=False):
   1130         #retry request once if cached connection has gone cold
   1131         for i in (0, 1):
   1132             try:
   1133                 return self.single_request(host, handler, request_body, verbose)
   1134             except OSError as e:
   1135                 if i or e.errno not in (errno.ECONNRESET, errno.ECONNABORTED,
   1136                                         errno.EPIPE):
   1137                     raise
   1138             except http.client.RemoteDisconnected:
   1139                 if i:
   1140                     raise

if I swap the except blocks then everything seems to work for me (ac756f16c5bbbe544ad82a8f3ab2eac6cccdb62e is applied).

About the statement above that './manage.py test' doesn't terminate: I have the feeling that the server is still running while there's no client to issue more requests (test has finished). I'm not sure if Django can do anything about this.

I'd love to hear your comments on my findings since I don't know this area of Django or Python's http/xmlrpc libraries very well but feel free to close.

[Tim Graham]

I don't have any insight to offer.

[Tom Forbes]

I don't have much to add either, except that it would be good to open a ticket with Python if it is indeed an issue in xmlrpc. If this is causing issues with a core standard library, which is hopefully we'll tested, then this could be triggering other libraries and code that is not reported.

I have some time this weekend to review this ticket and perhaps dig into it a bit, but it would be nice to just be able to enable keep alive in all but streaming requests.

[Alexander Todorov]

FYI the Python bug is:
​https://bugs.python.org/issue26402