Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#28968 closed Bug (invalid)

LiveServerTestCase prematurely closing HttpResponse with Django 2.0

Reported by: Alexander Todorov Owned by: nobody
Component: HTTP handling Version: 2.0
Severity: Normal Keywords:
Cc: Tom Forbes Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I'm trying to migrate an existing project to Django 2.0 and I'm hitting a problem with some of our tests, all of the problematic tests are very similar in nature:

1) They are all LiveServerTestCase based
2) All of them have a XML-RPC client which issues requests to the Django app
3) Using django-modern-rpc the application under test is able to execute a function and return a result
3) All error out on the first request (most of the times) with traceback from the xmlrpc client module in Python telling me that that connection has been closed without a response.

Here are the few data points I was able to collect:

  • Locally happens on both MySQL and SQLite but on SQLite less frequently, looks like a race condition. In Travis CI happens on SQLite but not on MySQL so go figure!
  • with a local MySQL instance I'm able to reproduce all the times
  • the root-cause seems to be in wsgiref.handlers.py::BaseHandler.run() which calls finish_response() which executes HttpResponse.close(). In run() there's also a comment about async servers and that they should not call .close() inside .finish_response()
  • if I modify Django's basehttp.py::ServerHandler.http_version to '1.0' all tests seem to pass regardless if we use a multi-threaded or single-threaded LiveServer!
  • This is on Python 3.5, RHEL 7 system.

So it looks like the problem is with ServerHandler (presumably tests) wanting to use HTTP 1.1 but I can't narrow it down firther ATM.

To reproduce you may checkout the code at https://github.com/kiwitcms/Kiwi;
pip install -r requirements/devel.txt; pip install --upgrade Django and

./manage.py test --noinput --settings=tcms.settings.test.mysql tcms.xmlrpc.tests.test_logging.TestXMLRPCLogging.test_logging_with_authenticated_user

Change History (7)

comment:1 by Tim Graham, 7 years ago

Cc: Tom Forbes added

It sounds similar to ac756f16c5bbbe544ad82a8f3ab2eac6cccdb62e. I'm not sure if anyone will be interested in debugging your project to find if Django is at fault.

comment:2 by Tim Graham, 7 years ago

Resolution: needsinfo
Status: newclosed

Please reopen if you can explain why Django is at fault.

comment:3 by Alexander Todorov, 7 years ago

Resolution: needsinfo
Status: closednew

@Tom Forbes,
thanks for the hint, I've also been looking around the sme parts of the code. Reverting the commit you listed above and testing with ServerHandler.http_version = '1.1' makes all of my tests to PASS but './manage.py test' doesn't exit!.

All of my responses also include the Content-Length header. I will let you know when I find more.

comment:4 by Alexander Todorov, 7 years ago

I think there's a bug inside Python's xmlrpc.client.Transport class which isn't playing nicely when Django closes the connection when HTTP 1.1 is in use. In particular this piece of code in xmlrpc/client.py

   1129     def request(self, host, handler, request_body, verbose=False):
   1130         #retry request once if cached connection has gone cold
   1131         for i in (0, 1):
   1132             try:
   1133                 return self.single_request(host, handler, request_body, verbose)
   1134             except OSError as e:
   1135                 if i or e.errno not in (errno.ECONNRESET, errno.ECONNABORTED,
   1136                                         errno.EPIPE):
   1137                     raise
   1138             except http.client.RemoteDisconnected:
   1139                 if i:
   1140                     raise

if I swap the except blocks then everything seems to work for me (ac756f16c5bbbe544ad82a8f3ab2eac6cccdb62e is applied).

About the statement above that './manage.py test' doesn't terminate: I have the feeling that the server is still running while there's no client to issue more requests (test has finished). I'm not sure if Django can do anything about this.

I'd love to hear your comments on my findings since I don't know this area of Django or Python's http/xmlrpc libraries very well but feel free to close.

comment:5 by Tim Graham, 7 years ago

Resolution: invalid
Status: newclosed

I don't have any insight to offer.

comment:6 by Tom Forbes, 7 years ago

I don't have much to add either, except that it would be good to open a ticket with Python if it is indeed an issue in xmlrpc. If this is causing issues with a core standard library, which is hopefully we'll tested, then this could be triggering other libraries and code that is not reported.

I have some time this weekend to review this ticket and perhaps dig into it a bit, but it would be nice to just be able to enable keep alive in all but streaming requests.

comment:7 by Alexander Todorov, 7 years ago

FYI the Python bug is:
https://bugs.python.org/issue26402

Note: See TracTickets for help on using tickets.
Back to Top