Provide an API to configure arbitrary file storage backends

[Aymeric Augustin]

Currently Django has two hardcoded file storage backends, "default" (for media files) and "static" (for static files). Idioms for configuring file storage backends in pluggable apps are cumbersome.

It would be nice to be able to configure arbitrary file storage backends, like caches and databases e.g.:

FILE_STORAGES = {
   'media': {
       'BACKEND': settings.DEFAULT_FILE_STORAGE,
       'OPTIONS': {
            'location': settings.MEDIA_ROOT,
            'base_url': settings.MEDIA_URL,
            # possible override of settings.FILE_CHARSET
       },
   },
   'static': {
       'BACKEND': settings.STATICFILES_STORAGE,
       'OPTIONS': {
            'location': settings.STATIC_ROOT,
            'base_url': settings.STATIC_URL,
            # replacement for STATICFILES_FINDERS and STATICFILES_DIRS that would look a lot like template loaders
            # possible override of settings.FILE_CHARSET
       },
   }

This was discussed on django-developers: ​https://groups.google.com/d/msg/django-developers/gEbFApLLuzg/IW1LDUwmEgAJ

There were some concerns about introducing another large dict in settings. The general ideas was uncontroversial.

[Keryn Knight]

For what it's worth, bmispelon and I discussed something akin to this recently, out of which I started implementing something similar ​entirely as an experiment. One thing I liked about where I went with that was that a named storage key in the dictionary was a dotted path, and behaved somewhat like a logging config in that the nearest 'parent' match such that each model field which uses a storage could be addressed separately, like so:

class Test(Model):
    a = FileField(storage='myapp.test.a')
    b = ImageField(storage='myapp.test.b')

and then if myapp.test.a or myapp.test.b were in the configuration dictionary, they would be used, but if not, and myapp.test was, that would be used, and so on walking backwards up the dotted path (falling back to ostensibly default [''] in the end)

The notion there was third-party apps could namespacing storages per-field, but at a project level one only has to opt-in as much as one cares (ie: myapp, or myapp.test, or default etc.) to have finegrained control.

[Sasha Gaevsky]

Configurable file storage backends was already proposed to introduce in terms of task #23251.

[Sasha Gaevsky]

Replying to aaugustin:

FILE_STORAGES = {
   'media': {
       'BACKEND': settings.DEFAULT_FILE_STORAGE,
       'OPTIONS': {
            'location': settings.MEDIA_ROOT,
            'base_url': settings.MEDIA_URL,
            # possible override of settings.FILE_CHARSET
       },
   ...
}

Media backend will have media key in settings, not default, to keep it consistent with the current settings?

[Sasha Gaevsky]

As proposed Aymeric Augustin in the original PR, I started to compose DEP, so now it's in progress and I hope to finish it sometime soon.

[Jarosław Wygoda]

I'd like to introduce a file storage registry similar to BaseConnectionHandler (django/utils/connection.py) and EngineHandler (django/template/utils.py).

Example settings.py snippet:

STORAGES = {  # rename to FILE_STORAGES to make it more explictit?
    'example': {
        'BACKEND': 'django.core.files.storage.FileSystemStorage',
        'OPTIONS': {
            'location': '/example',
            'base_url': '/example/',
        },
    },
}

Changes introduced by this pr are backward compatible. Users can still use existing settings to configure static and media storages.

Currently storages can be retrieved from the following objects:

django/core/files/storage.py:

• get_storage_class
• DefaultStorage
• default_storage

django/contrib/staticfiles/storage.py:

• ConfiguredStorage
• staticfiles_storage

What do you think about deprecating them?

I'll write tests and docs if this approach is acceptable.

​https://github.com/django/django/pull/15610

[Mariusz Felisiak <felisiak.mariusz@…>]

In 9e7cb27:

Refs #26029 -- Doc'd django.core.files.storage.default_storage.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 04ec8bf9:

[4.1.x] Refs #26029 -- Doc'd django.core.files.storage.default_storage.

Backport of 9e7cb27a5b7363239d1db02d29fe12efdf25b899 from main

[Carlton Gibson]

Unchecked flags, for a new review.

[Carlton Gibson]

PR looks promising, but have comments outstanding ref adding the new STORAGES defaults.

[Carlton Gibson]

I think the PR looks close. (I suggested a few docs tweaks)

Main remaining point (for me) is being sure about the signal handling with override_settings, and the usual Settings/UserSettingsHolder complexities.

[Mariusz Felisiak <felisiak.mariusz@…>]

In d16079d:

Refs #26029 -- Added LazySettings._show_deprecation_warning() hook.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 1ec3f09:

Fixed #26029 -- Allowed configuring custom file storage backends.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 32940d3:

Refs #26029 -- Deprecated DEFAULT_FILE_STORAGE and STATICFILES_STORAGE settings.

[Mariusz Felisiak <felisiak.mariusz@…>]

In f72f420:

Refs #26029 -- Removed DEFAULT_FILE_STORAGE and STATICFILES_STORAGE settings.

This also removes django.core.files.storage.get_storage_class().

Per deprecation timeline.

[GitHub <noreply@…>]

In fa6e6f31:

[5.0.x] Refs #26029 -- Improved get_storage_class() deprecation warning with stacklevel=2.

Addition of the stacklevel argument shows the source of the
deprecated call, making updating the client code simpler.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 0cbc92bc:

[4.2.x] Refs #26029 -- Improved get_storage_class() deprecation warning with stacklevel=2.

Addition of the stacklevel argument shows the source of the
deprecated call, making updating the client code simpler.

Backport of fa6e6f31137ee4025adfb27f88c07eebd488d446 from stable/5.0.x.

[GitHub <noreply@…>]

In 6dcab75d:

Refs #26029 -- Extended docs for the StorageHandler default instance.

Third-party packages that provide storages need to rely on the
StorageHandler API in order to allow users to use the storages module
instance to override defaults. Minimally documenting these methods
allows package authors to rely on them.

Co-authored-by: Sarah Boyce <42296566+sarahboyce@…>

[Natalia <124304+nessita@…>]

In 56ad530a:

[5.1.x] Refs #26029 -- Extended docs for the StorageHandler default instance.

Third-party packages that provide storages need to rely on the
StorageHandler API in order to allow users to use the storages module
instance to override defaults. Minimally documenting these methods
allows package authors to rely on them.

Co-authored-by: Sarah Boyce <42296566+sarahboyce@…>

Backport of 6dcab75d5d8c2ef18de15323930057e6fa9ec00f from main.