runserver crashes with "Remote end closed connection without response"

[rvernica]

I created an app like in the tutorial and started Django. When I issue successive requests, the session gets disconnected. Here is an example:

>>> s = requests.Session()
>>> s.get('http://localhost:8000/'); s.get('http://localhost:8000')
<Response [404]>
Traceback (most recent call last):
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "...local/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "...local/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "...local/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...local/lib/python3.6/site-packages/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "...local/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "...local/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "...local/lib/python3.6/site-packages/requests/adapters.py", line 495, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

It I have a small timeout between the requests it works fine:

>>> s = requests.Session()
>>> s.get('http://localhost:8000/'); time.sleep(1); s.get('http://localhost:8000')
<Response [404]>
<Response [404]>

It also works fine if I don't use sessions:

>>> requests.get('http://localhost:8000/'); requests.get('http://localhost:8000')
<Response [404]>
<Response [404]>

I initially reported this as a bug in the requests library but it does not appear to be the case. See discussion here ​https://github.com/requests/requests/issues/4784

[Herbert Fortes]

Hi,

I do not have this error here.

I have 15 unapplied migration(s), the teste/urls.py has the include, the core/views.py app
has only the HttpResponse.

Only 'python manage.py runserver'. I got two 404.

Maybe there is a step that I am missing?

[Herbert Fortes]

certifi==2018.10.15
chardet==3.0.4
Django==2.1.2
idna==2.7
pytz==2018.5
requests==2.19.1
urllib3==1.23

python 3.6.6

[Tim Graham]

I also can't reproduce. I searched past bugs for "Remote end closed connection without response" and found #28440 but this was fixed in Django 2.0. I also failed to reproduce this with Django 1.11.

[Jaap Roes]

I was also hit by this and found a reliable way to reproduce this:

First create a Dockerfile with the following content:

FROM python:3

RUN mkdir /test
WORKDIR /test
RUN pip install requests django pyinotify
RUN django-admin startproject tmp .
RUN echo '\
import requests\n\
with requests.Session() as s:\n\
  for _ in range(5):\n\
    print(s.get("http://localhost:8000/").status_code)\n'\
 > test.py
CMD /bin/bash -c './manage.py runserver 0:8000 & sleep 5 && python test.py'

Now run:

docker build -t connection-error .
docker run -it --rm connection-error

This reliably fails with:

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

When I remove don't pyinotify the test script doesn't fail.

So there seems to be some weird interaction with pyinotify, runserver and a requests.Session.

[Jaap Roes]

After some further testing I can also reproduce the issue (less reliably) without pyinotify by increasing the amount of requests (e.g. for _ in range(50)). So I guess it's a more general issue caused by the reloader?

EDIT: Running the devserver with the --noreload flag (and without installing pyinotify) still causes the error to show. Only when I disable threading (--nothreading) the issue goes away, unless pyinotify is installed then both --nothreading and --noreload need to be used.

[Tim Graham]

I guess the question remains whether Django is at fault or if there's some bug in Python or elsewhere.

[rvernica]

Just FYI, there is some discussion related to this in ​Remote end closed connection without response on successive Session requests and possibly in ​requests.Session doesn't properly handle closed keep-alive sessions

[Jaap Roes]

I did some further testing, and the test script I created works with Django 1.11 *and* the current master. So I ran a git bisect and saw that ​this commit has fixed the issue. Is it possible to backport this fix?

[Tim Graham]

Yes, I'm open to it. Is the regression due to ac756f16c5bbbe544ad82a8f3ab2eac6cccdb62e or e6065c7b8363202c5eb13ba10c97a8c24d014b45?

[Jaap Roes]

I can confirm the test script works fine on e6065c7b8363202c5eb13ba10c97a8c24d014b45 and fails on ac756f16c5bbbe544ad82a8f3ab2eac6cccdb62e, so the latter commit is the root cause of this issue.

EDIT: pasted the correct commit hashes

[Tim Graham <timograham@…>]

In e1721ece:

[2.1.x] Fixed #29849 -- Fixed keep-alive support in runserver.

Ticket #25619 changed the default protocol to HTTP/1.1 but did not
properly implement keep-alive. As a "fix" keep-alive was disabled in
ticket #28440 to prevent clients from hanging (they expect the server to
send more data if the connection is not closed and there is no content
length set).

The combination of those two fixes resulted in yet another problem:
HTTP/1.1 by default allows a client to assume that keep-alive is
supported unless the server disables it via 'Connection: close' -- see
RFC2616 8.1.2.1 for details on persistent connection negotiation. Now if
the client receives a response from Django without 'Connection: close'
and immediately sends a new request (on the same tcp connection) before
our server closes the tcp connection, it will error out at some point
because the connection does get closed a few milli seconds later.

This patch fixes the mentioned issues by always sending 'Connection:
close' if we cannot determine a content length. The code is inefficient
in the sense that it does not allow for persistent connections when
chunked responses are used, but that should not really cause any
problems (Django does not generate those) and it only affects the
development server anyways.

Refs #25619, #28440.

Regression in ac756f16c5bbbe544ad82a8f3ab2eac6cccdb62e.
Backport of 934acf1126995f6e6ccba5947ec8f7561633c27f from master.

[Tim Graham <timograham@…>]

In 5d327a63:

Refs #29849 -- Forwardported 2.1.4 release note.

[Jaap Roes]

Thanks Tim!

I'm still wondering why it's so easy to trigger when pyinotify is installed. If anyone can explain that to me, I'd love to know :)