Allow loading apps dynamically at runtime

[Christian González]

I stumbled upon a problem I am not able to solve, and I think this is a feature that should be in Django core AFAIKT.
Until Django 1.x it was possible to add apps dynamically to INSTALLED_APPS during runtime, and reload the cache, see here:
​https://stackoverflow.com/questions/19833393/django-dynamically-add-apps-as-plugin-building-urls-and-other-settings-automat

Even if I think this is a bit of a hack and it was never really meant that way (?), nevertheless there should be an official possibility to do so.

I have made a Django application with dynamically apps at server startup time - this is no problem, just search for a plugins dir during settings.py and add the dotted paths to INSTALLED_APPS. Works fine, but one problem remains:

If I want to enable dynamic app installation - like an "app store" in my application - users can download apps from my "store", and enable them using my web interface - there currently is no way to achieve that besides restarting the Django server, right?

During setup, Apps.populate() is called - and its docstring says that the method is idempotent. Is it possible/allowed/recommended to add some paths to django.conf.settings.INSTALLED_APPS and call that populate() again later during runtime?

[Tim Graham]

As the author of the app loading refactor, Aymeric may have some input.

[Aymeric Augustin]

This proposal removes the guarantees of deterministic behavior that the app-loading refactor brought. One example among many: if two apps override the same template, which app wins? The order of INSTALLED_APPS defines this. What happens in the scenario you're proposing?

I'd like to see a thorough discussion of the pros and cons before making this decision as well as an analysis of which caches need to be invalidated and how this could happen.

I wrote extensively about this topic on the django-developers mailing-list during the app-loading refactor. As far as I can tell, all this material is still relevant, so you should be able to get the list of topics to discuss from there. You can also check out my conference talks about app-loading.

[Christian González]

Should I discuss this on the mailing list then, or here?

Just to answer your questions:

• regarding the INSTALLED_APPS order: Sure, if you have a stati application, this order makes sense. If you have an application that uses plugins, the application itself is responsible for the order the plugins are loaded. To make this deterministic, would be nonsense IMHO, as different applications use different plugin methods (Django provides None). So It could be that there is a "plugin resolution order" where dependendant plugins are loaded after their dependencies - which makes sense, or the app could decide that plugins define an integer "weight" (which is no good practice IMHO), etc. So, that's the sense of "dynamic" apps.

• Regarding templates, and which app wins - I always fount the templating system a bit difficult - IMHO app templates should have implicit namespaces. But that would break virtually any existing Django install - so forget it. But: Same as before: The loading order decides, like in INSTALLED_APPS. Let the application decide in which order the plugins should be loaded.

In my simple mind, I call apps.populate([myfirstapp_plugin, other_plugin, last_plugin_app]) like setup.py calls it with settings.INSTALLED_APPS. And populate works through all he apps in this order and merges them into the system.

I wrote a Django application that merely consists of the plugin system. The "real" main application is already a "Core" plugin (with a version etc) which is listed in INSTALLED_APPS as app, because it is always loaded and part of the base system. And all other plugins are loaded dynamically. So they can have dependencies like "core>=1.5.2" and it won't load it when the core version is below that. It's pre-alpha, just look here for the extension system ​https://gitlab.com/nerdocs/medux/MedUX/tree/develop/medux/extensionsystem (e.g. the PluginManager, is in development. I'm no professional software engineer, so please look away if you see horrible coding style ;-) - or better tell me)

I tried really many ways to accomplish that, and some of the first ones were plugins that are no django apps. But I came across some issues, and had to rewrite much of the stuff Django does already perfectly, like loading models, migrations, etc. So in the sense of DRY, I wanted to have plugins being Django apps. But there the problem I describe here starts: I have to restart the server, rewriting the settings.py - which is nonsense in my eyes.

[Christian González]

Just wanted to state that IMHO this is a real *regression* in the Django evolvement, as in earlier Django versions this was possible.
How can I help here? Should I start a thread on the mailing list?

[Christian González]

Aymeric Augustin: Please just tell me where I should discuss this topic: here or in the devel mailing list. IMHO this is a real gap in the Django environment.
And I'd like to contribute to help close it. But tell me where - I don't want to pollute the devel mailing list with things that don't belong there.

But I can't understand why this issue is closed that fast.

[Aymeric Augustin]

django-developers is the right place for discussing app loading. See ​https://groups.google.com/forum/#!searchin/django-developers/app-loading%7Csort:date/django-developers/_iggZzrYtJQ/FWFPgCflSnkJ for the last episode.

Unfortunately this is a poor example of open-source development. App-loading is such a complex issue that I felt unable to handle it fully in public. Instead I had lots private discussions with a small group of active core devs, I merged patches once I had a rough agreement from that group, and I handled complaints after the fact. Apparently enough people felt the result was worth it that I didn't get kicked out but I decided I wouldn't do that again.

Anyway, that mailing list discussion provides some context. You should make sure you understand why the app registry population is designed the way it is before attempting to change it. You should also demonstrate that you have reviewed past issues related to app loading order and especially unstable import ordering.

[Christian González]

Thanks. I'll do my best there.

[Tim Graham]

​django-developers discussion