Support for WSGI applications within Django

[Gustavo Narea]

With the attached patch, it will be possible to run WSGI applications from Django and even use such applications as Django views.

It depends on the patch I supplied in #8927.

Please let me know what you think about it.

[James Bennett]

I think it'd be nice if this explained how it differs from the WSGI-oriented SoC branch, and indicated whether it builds on that work.

[Gustavo Narea]

Replying to ubernostrum:

I think it'd be nice if this explained how it differs from the WSGI-oriented SoC branch, and indicated whether it builds on that work.

Hi.

I did it all from scratch and the result is quite different from what I found at: ​http://github.com/rfk/django/blob/f96eb6068552e228f705beda9c8181db2079a797/django/wsgi/wsgi_to_django.py

Most importantly, it's much simpler because I am not trying to reconstruct the WSGI environ if it isn't already present. I assume it's already available (see patch in #8927). Therefore it also passes the wsgi.input and wsgi.error file-like objects on to the WSGI app, without reconstructing them.

Cheers.

[Gustavo Narea]

Support for WSGI apps within Django

[Gustavo Narea]

I've updated the patch because I found a couple of minor bugs in the previous one and the test suite didn't have a 100% of code coverage.

[Gustavo Narea]

Since it's not going to make it by v1.2 and we don't want to have a patchy Django for so long, I reimplemented this patch and others in an independent package:
​http://bitbucket.org/2degrees/twod.wsgi/

If you'd like to implement this in Django, I'd recommend copying the relevant modules from twod.wsgi (which is licensed under the BSD license too) because I fixed a few tiny bugs with my original patch.

[Russell Keith-Magee]

Marking as accepted (since, like #8297, it's a reasonable idea). However, I'm still unclear to the extent this patch overlaps with existing WSGI efforts.

[Gustavo Narea]

Replying to russellm:

Marking as accepted (since, like #8297, it's a reasonable idea).

I'm glad to hear that! Here's the documentation for that functionality: ​http://packages.python.org/twod.wsgi/manual/embedded-apps.html

Please let me know if you like the way it's implemented, so that I can port it from twod.wsgi into Django.

However, I'm still unclear to the extent this patch overlaps with existing WSGI efforts.

Here's my (biased) response: ​http://packages.python.org/twod.wsgi/about.html#twod-wsgi-versus

[patchhammer]

wsgi-apps.diff fails to apply cleanly on to trunk

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

I'm in favor of keeping this feature outside of Django, for the following reasons.

1) Embedding WSGI applications in Django isn't the most robust design. It's much better to dispatch at the WSGI level between Django and other applications.

2) While it's easy to just make this feature work for simple use cases, it's very difficult to make the gateway 100% WSGI-compliant (even if the recently-added StreamingHttpResponse could help abide by the buffering rules). I'm afraid this would add a significant maintainance load, while benefitting only a few users with non-trivial architectures.

3) There isn't any reason why this needs to live in core, and there are a few implementations floating in the wild. I have my own implementation of this too -- I use it in development as a replacement for uwsgi.applications (​http://projects.unbit.it/uwsgi/wiki/ApplicationsDict).

[Gustavo Narea]

1) "Dispatch at the WSGI level" is ambiguous. I assume you mean dispatch before Django is reached, which wouldn't be better or even accomplish the same. Picture this example: You want to use the ​WSGI X-Sendfile application from a Django view because you want to (1) check that the user can download the file, (2) log who's downloading the file and (3) log when the file was downloaded. Try to do that outside of Django and you'll end up with an unmaintainable piece of code.
2) I'm not sure I understand this point well enough. What does the gateway have to do with all this? Anyway, I can assure you that this is a piece of cake and it's 100% reliable in 100% of the cases you can imagine. I've reimplemented a variant of that patch in ​twod.wsgi and presented demos of how you can proxy applications like ​Trac and ​the PHP-powered Wordpress via Django. What complicated situations can you imagine that wouldn't be supported?
3) I think you're confusing "embedding WSGI applications inside Django" with "routing WSGI applications". Both can be quite useful, depending on your needs, but there's a big difference. This ticket is about embedding, but you're talking about routing.

I also want to add that all the other mainstream WSGI framework have supported this outside-the-box for years.

I'll dare reopening this ticket because I think there's been a misunderstanding, and would like to prevent this ticket from going unnoticed for years again.

[Aymeric Augustin]

Replying to Gustavo:

1) "Dispatch at the WSGI level" is ambiguous. I assume you mean dispatch before Django is reached, which wouldn't be better or even accomplish the same. Picture this example: You want to use the ​WSGI X-Sendfile application from a Django view because you want to (1) check that the user can download the file, (2) log who's downloading the file and (3) log when the file was downloaded. Try to do that outside of Django and you'll end up with an unmaintainable piece of code.

For such cases, I suggest to use an external Django view to WSGI mapper — twod.wsgi or one of the many other packages providing the same feature.

2) I'm not sure I understand this point well enough. What does the gateway have to do with all this? Anyway, I can assure you that this is a piece of cake and it's 100% reliable in 100% of the cases you can imagine. I've reimplemented a variant of that patch in ​twod.wsgi and presented demos of how you can proxy applications like ​Trac and ​the PHP-powered Wordpress via Django. What complicated situations can you imagine that wouldn't be supported?

Django is a WSGI application. If it acts as a WSGI server too, that makes it a WSGI gateway. As such, it'll be expected to honor the spec for WSGI gateways. People can be unbelievably annoying with spec-compliance bugs.

Let's take this sentence from PEP 3333:

WSGI servers, gateways, and middleware must not delay the transmission of any block; they must either fully transmit the block to the client, or guarantee that they will continue transmission even while the application is producing its next block.

twod.wsgi doesn't respect this when the write() callable is used. (To you credit, you handled the iterator case correctly.)

Describing something as a piece of cake and 100% correct isn't good for your credibility when I can find a bug through code inspection!

3) I think you're confusing "embedding WSGI applications inside Django" with "routing WSGI applications". Both can be quite useful, depending on your needs, but there's a big difference. This ticket is about embedding, but you're talking about routing.

Well, the very first example in twod.wgsi's docs is called embedding but it actually shows routing:
​http://pythonhosted.org/twod.wsgi/manual/embedded-apps.html#embedding-non-django-applications

The second example is about embedding:
​http://pythonhosted.org/twod.wsgi/manual/embedded-apps.html#calling-them-from-a-django-view

The concepts aren't as neatly delimited as you make it sound, and both can be implemented as external apps.

I also want to add that all the other mainstream WSGI framework have supported this outside-the-box for years.

Django isn't a "WSGI framework" per se — it uses WSGI as a deployment platform, that's all. Besides, this isn't a competition.

I'll dare reopening this ticket because I think there's been a misunderstanding, and would like to prevent this ticket from going unnoticed for years again.

Sure, we can leave it open for now, I'll let someone else make the decision.

[Gustavo Narea]

Replying to aaugustin:

Replying to Gustavo:

1) "Dispatch at the WSGI level" is ambiguous. I assume you mean dispatch before Django is reached, which wouldn't be better or even accomplish the same. Picture this example: You want to use the ​WSGI X-Sendfile application from a Django view because you want to (1) check that the user can download the file, (2) log who's downloading the file and (3) log when the file was downloaded. Try to do that outside of Django and you'll end up with an unmaintainable piece of code.

For such cases, I suggest to use an external Django view to WSGI mapper — twod.wsgi or one of the many other packages providing the same feature.

Then why not provide it in Django and prevent the proliferation of implementations to accomplish this? That's indicative that there's a demand for this.

2) I'm not sure I understand this point well enough. What does the gateway have to do with all this? Anyway, I can assure you that this is a piece of cake and it's 100% reliable in 100% of the cases you can imagine. I've reimplemented a variant of that patch in ​twod.wsgi and presented demos of how you can proxy applications like ​Trac and ​the PHP-powered Wordpress via Django. What complicated situations can you imagine that wouldn't be supported?

Django is a WSGI application. If it acts as a WSGI server too, that makes it a WSGI gateway. As such, it'll be expected to honor the spec for WSGI gateways. People can be unbelievably annoying with spec-compliance bugs.

I see. Agreed.

Let's take this sentence from PEP 3333:

WSGI servers, gateways, and middleware must not delay the transmission of any block; they must either fully transmit the block to the client, or guarantee that they will continue transmission even while the application is producing its next block.

twod.wsgi doesn't respect this when the write() callable is used. (To you credit, you handled the iterator case correctly.)

Actually, the write() callable is supported just fine. twod.wsgi doesn't do anything with it because it delegates it to WebOb's Request.call_application(), which supports write() the way it's supposed to: ​https://github.com/Pylons/webob/blob/master/webob/request.py#L1238

Describing something as a piece of cake and 100% correct isn't good for your credibility when I can find a bug through code inspection!

The approach is a piece of cake, even if there are bugs in my implementation, which may well be the case because I'm a human! "100% correct" are your own words, isn't that right? I said "100% reliable in 100% of the cases you can imagine", and I stand by it.

And what you've found isn't a bug, unless I've misinterpreted your reply.

3) I think you're confusing "embedding WSGI applications inside Django" with "routing WSGI applications". Both can be quite useful, depending on your needs, but there's a big difference. This ticket is about embedding, but you're talking about routing.

Well, the very first example in twod.wgsi's docs is called embedding but it actually shows routing:
​http://pythonhosted.org/twod.wsgi/manual/embedded-apps.html#embedding-non-django-applications
The second example is about embedding:
​http://pythonhosted.org/twod.wsgi/manual/embedded-apps.html#calling-them-from-a-django-view

No, it's embedding. make_wsgi_app() returns a Django view, which makes a huge difference.

Consider this example: If the embedded WSGI app relies on REMOTE_USER for authentication and Django previously authenticated the current user, then the embedded app will reuse Django's authentication. By contrast, if you route requests to the same WSGI app outside of Django, as you suggest, the embedded WSGI app won't be able to reuse Django's authentication.

The concepts aren't as neatly delimited as you make it sound,

Well, they are and they've been for ages with other Python frameworks.

and both can be implemented as external apps.

No.

I also want to add that all the other mainstream WSGI framework have supported this outside-the-box for years.

Django isn't a "WSGI framework" per se — it uses WSGI as a deployment platform, that's all. Besides, this isn't a competition.

I didn't mean that as a call for competition or anything. I just wanted to point out that everywhere else this functionality is taken for granted and yet here we are discussing whether it should be part of Django itself, while other Django users continue creating their own implementations for the missing functionality.

It may not be its primary focus, but Django is a WSGI framework, just like TurboGears, Pylons, etc. ​WSGI is more just a deployment platform.

I'll dare reopening this ticket because I think there's been a misunderstanding, and would like to prevent this ticket from going unnoticed for years again.

Sure, we can leave it open for now, I'll let someone else make the decision.

Thanks! I wonder whether this should be taken to the mailing list.

[Aymeric Augustin]

Discussion on django-developers: ​https://groups.google.com/d/msg/django-developers/137ZyyBNq5c/C6v7LUdZYNUJ

[Mariusz Felisiak]

After reconsideration I think we shouldn't move it forward. It's quite niche, can live in a 3rd party package, and not worth the additional complexity. Moreover there was no consensus on the mailing list. Thanks for you efforts!