improve lazy loading and resource usage

[Waldemar Kornewald]

This is a general problem, but it's particularly noticeable on App Engine. Django loads a lot of modules up-front even if they're not needed. This makes the first request on a new app server instance very slow because the user has to wait for Django to do its imports. This can take five seconds if you also enable the admin UI. The attached patch is the first step towards more lazy loading and it can shave off 0.5-1s if you have a simple app. The following components are loaded on-demand:

• forms
• messages
• CSRF
• most of the file upload handling code (that can still be improved, though)
• management

Overall, I think it's 42 modules less for very simple sites.

Personally, I'd like to see Django as a more serious alternative to small frameworks and REST solutions because I love the ORM, but there's too much overhead for simple sites that only use a small fraction of Django's features. Independent of that, I think Django's resource usage should be improved, anyway.

[Waldemar Kornewald]

patch against r12411

[Alex Gaynor]

I'm am violently -1 on this patch. a) it complicates code, b) it slows down execution, c) it caters to the fact that GAE has a model that almost no one else has (seriously, we chucked CGI a while ago ;)), d) it's far better solved by using a custom import hook in your wsgi file (the same way mercurial does) in order to achieve lazy loading.

[Waldemar Kornewald]

Well, I agree that GAE is very stupid in this case. :)

Still, (a) and (b) are hardly a huge problem. Those imports should be very fast once the respective modules have been loaded for the first time.

From your (c) and (d) I get the impression that you might have misunderstood the GAE issue. This is only about the very first request to a fresh app server instance (cold start). After the cold start all subsequent requests are fast because they reuse the main() import hook. So, the problem is that the very first request has to wait for Django to load and that takes a lot of time.

[Alex Gaynor]

Yes, the purpose of the mercurial import hook is to automate lazy loading of modules (when you import something it basically returns a "fake" module, and only looking up the attribute on the module actually triggers the import): ​http://sayspy.blogspot.com/2009/06/lazy-imports-in-25-lines-of-code.html

[Waldemar Kornewald]

Ah, sorry. Then it seems I misunderstood (GAE provides some other import hook for the wsgi handler module). Thanks for the link. I'll take a look and see how much it can help. The biggest problem will probably be django.forms because django.db.models.fields not only imports that module, but also accesses its attributes because it uses a field class as a default parameter. We could at least move that module access into the formfield function.

[Aymeric Augustin]

Closing for the reasons given by Alex.

Moving imports in functions and methods is messy, it goes against established conventions, and it can create subtle import loops. This isn't an acceptable solution.