Support multiple open connections in LiveServerTestCase

[jwxiao@…]

Run a selenium test with LiveServerTestCase. Django starts StoppableWSGIServer. The StoppableWSGIServer is not multithread support. When the codes to be tested call urllib.urlopen to the same server (ie: StoppableWSGIServer) the selenium test will hang and fail because of time-out.

[Julien Phalip]

Thank you for the report, however there is too little information here to identify if or where there is an issue. Could you please provide a detailed explanation of how to reproduce the issue you're facing, or even better, could you provide some code or a test case? Thanks!

[Julien Phalip]

Just a clarification: please feel free to reopen this ticket if you can provide more detailed information.

[Joeri Bekker]

Although this is not my ticket, I just ran into the same issue.

The problem occurs typically when, within a request/response cycle, another request is done to the same StoppableWSGIServer. Now, the first request can never finish because the second request is waiting for the StoppableWSGIServer to respond (but it never will because it's waiting for the first request...)

# view
def index(self, request):
    if request.GET.get('inner_request') == 'true':
        # Within a request/response, perform another request to the same live server.
        # This one get's stuck.
        urllib.urlopen(request.build_absolute_uri())
    return HttpResponse()

# test
class LiveServerTests(LiveServerTestCase):
    def test_request_within_request(self):
        response = requests.get(self.live_server_url, params={'inner_request': 'true'})

        # This code is never reached (or the live server throws a timeout)
        self.assertEqual(response.status_code, 200)

The use case is somewhat fictional.

[Aymeric Augustin]

Considering that:

• "the use case is somewhat fictional",
• the ticket was reopened without any additional information,
• more threading will certainly create more race conditions on shutdown, especially when it comes to the database connections — it's taken months to eliminate most from LiveServerTestCase, and I'm sure there are still some left,

I don't think we should add threading without more compelling arguments.

[Aymeric Augustin]

I'm hitting this problem, which means I have a concrete use case to offer.

I have a view that generates a PDF document by converting a HTML document served by the same server (with phantomjs). It works as follows:

• user makes a HTTP request for a PDF document
• pdf_view() starts a phantomjs subprocess
• phantomjs makes a HTTP request for the equivalent HTML document
• html_view() renders a HTML template
• phantomjs gets the HTML, converts it to PDF, and returns that
• pdf_view() returns the PDF

I cannot use that view with the LiveServerTestCase because it's single threaded. The thread runs pdf_view(), waits for phantomjs, but there's no thread available to run html_view(), so the test blocks there.

[Aymeric Augustin]

The following monkey-patch does the job for me.

import socketserver

from django.test import testcases


class ThreadedWSGIServer(socketserver.ThreadingMixIn, testcases.WSGIServer):
    """
    Make Django's live server test case multi-threaded.

    See https://code.djangoproject.com/ticket/20238#comment:6

    """


testcases.WSGIServer = ThreadedWSGIServer

I'm using the PostgreSQL database backend.

I don't think this proposal is sufficient for in-memory SQLite. A WSGIHandler subclass that replaces the database connection for each new thread by the database connection from the main thread is likely needed.

---

By the way, this piece of code in django.core.servers.basehttp:

    if threading:
        httpd_cls = type(str('WSGIServer'), (socketserver.ThreadingMixIn, WSGIServer), {})
    else:
        httpd_cls = WSGIServer

could be replaced with:

    httpd_cls = ThreadedWSGIServer if threading else WSGIServer

assuming the definition of ThreadedWSGIServer above.

There's no reason (that I can think of) to define a class dynamically like that here. Saving a couple lines isn't worth the obfuscation.

[Aymeric Augustin]

jwxiao, I'm sorry for not understanding your report originally.

[Tim Graham]

This was also requested in #25970.

[Tim Graham]

Support for multiple open connections is also needed if you want to write a test that uses multiple browser instances as demonstrated in #27665 (closed as duplicate).

[Claude Paroz]

I simply put Aymeric suggestion to a patch here: ​https://github.com/django/django/pull/7768
If someone has an idea for a test, please help.

[Aymeric Augustin]

Nadege also prepared a patch but hasn't submitted a PR yet. I was planning to take a look at her work after the holidays.

[Claude Paroz]

Oh, sorry, I didn't intend to duplicate her work.

[Nadege]

We actually made the sames changes, I added some tests as well.
They don't fail but hang whithout the patch.
I tried to do a test that could show potential database issue because aymeric sugested that might be a problem
but things seems good.

Patch is here ​https://github.com/django/django/pull/7832

[Tim Graham]

Unless you're motivated to solve the Python 2 failures, we can target this patch for Django 2.0 which drops support for Python 2. The 1.11 feature freeze is Friday.

[Nadege]

I'm good with targeting Django 2.0

[Tim Graham <timograham@…>]

In bece8378:

Fixed #20238 -- Added threading support to LiveServerTestCase.

[Chris Jerdonek]

FYI, I just filed #32416, which lists a possible regression caused by this fix.