Add ASGI support to Django.

[Andrew Godwin]

The ASGI specification (​http://asgi.readthedocs.io) is now stable and Django should add support for it in addition to WSGI. This doesn't require too many deep changes, unlike a proposed "async-ification" of Django to allow async views and similar, which will require a DEP.

This will allow both seamless hosting of Django on ASGI servers, and will also significantly improve performance for sites that have slow file uploads as a blocking operation, as this will run asynchronously rather than holding open a thread.

[Andrew Godwin]

The pull request for this is ​https://github.com/django/django/pull/11209

[Carlton Gibson]

...and will also significantly improve performance for sites that have slow file uploads as a blocking operation, as this will run asynchronously rather than holding open a thread.

From the conversation on the PR, it looks as if file uploads aren't yet enabled:

Florian: This probably cannot use the upload handlers for similar reasons?
Andrew: I haven't looked into that yet, but probably not. File input in general has been a pain in Channels and I expect the same here.
​https://github.com/django/django/pull/11209#pullrequestreview-232567459

Is that right? If so do we need to pull this aspect into a separate ticket?

[Ran Benita]

It looks like the patch is going to be merged. I feel like I should at least point out some downsides before this step is taken. (Note, I am not a Django dev, just a user).

The patch adds an unconditional asyncio import. It is quite heavy for something most users won't use at least initially. On my aging laptop:

• Import time: 50ms
• Memory usage: ~5MB

Command for import time: python3.7 -X importtime -c 'import asyncio' |& tail -1

Script for memory usage:

from resource import getrusage, RUSAGE_SELF

before = getrusage(RUSAGE_SELF)
import asyncio
after = getrusage(RUSAGE_SELF)

print(f'{after.ru_maxrss - before.ru_maxrss}kb')

The patch adds 2 dependencies, one direct asgiref and one transitive async_timeout.

The patch implements the ASGI specification, but that specification has not been adopted widely yet and has not been proposed for official status (that I know of) like WSGI has.

ASGI depends on asyncio, but there are other alternatives worth considering (like trio and curio). And of course there is gevent which has been used successfully by many companies (including mine) for async Djagno for some time.

The patch on its own will run all views in a thread pool (asgiref.sync_to_async -> loop.run_in_executor -> default ThreadPoolExecutor). The default number of workers used is the number of cores * 5, so for example an 8 core will use 40 workers. If postgresql is used with persistent connections, this is a hazard since postgresql doesn't like this many connections. So users will need to use a connection pool, or disable persistent connections.

In my opinion, it would be hasty be merge ASGI support before the overall plan is decided. If a DEP is later proposed, and rejected (I hope not!), will it still make sense to have merged this?

[Andrew Godwin]

Regarding ASGI not being "official": Python core has made it reasonably clear to me that they don't want to have a PEP for ASGI, and giving one to WSGI was a bad idea in the first place. Thus, adoption is down to us just doing it. Flask have also said they are going to attempt to support it, as have Twisted.

The threads observation is an astute one, and I think worth calling out in the deployment documentation explicitly.

The async_timeout dependency is only for the async test harness in asgiref, and thus should never be imported under normal conditions.

The reason I have done this separately from the DEP is that it's not, in my eyes, strictly a new "feature" - instead, it is adding compatibility for a server-application protocol, and some last-resort async safety. Even if we didn't implement ASGI, I would still argue that we definitely need to make the ORM understand and protect itself from coroutine-based connection corruption, and that would involve importing asyncio (and probably asgiref for its helpful features in this area) anyway!

[Andrew Godwin <andrew@…>]

In 415e899d:

Refs #30451 -- Added HttpRequest._set_content_type_params() hook.

[Mariusz Felisiak <felisiak.mariusz@…>]

In a415ce70:

Fixed #30451 -- Added ASGI handler and coroutine-safety.

This adds an ASGI handler, asgi.py file for the default project layout,
a few async utilities and adds async-safety to many parts of Django.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 7f19e37:

Refs #30451 -- Added more tests for ASGIRequest and ASGIHandler.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 19895e89:

Refs #30451 -- Added asgiref to the tests requirements.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 45de0c29:

[3.0.x] Refs #30451 -- Doc'd asynchronous support and async-safety.

Backport of 635a3f8e6e0303e8a87586ef6be4ab0cc01d7c0d from master

[Mariusz Felisiak <felisiak.mariusz@…>]

In 635a3f8e:

Refs #30451 -- Doc'd asynchronous support and async-safety.

[GitHub <noreply@…>]

In 441103a:

Refs #33173, Refs #30451 -- Fixed ResourceWarning from unclosed body files in ASGI handler on Python 3.11+.