Dictionary based EMAIL_PROVIDERS settings

[Jacob Rief]

As discussed in ​https://groups.google.com/g/django-developers/c/R8ebGynQjK0/m/Tu-o4mGeAQAJ and during the sprints at Django Con Europe 2024 (Carton Gibson, Natalia Bidart, Jacob Rief), we now have consensus that we want to add this feature, even though this proposal has been rejected in https://code.djangoproject.com/ticket/22734 10 years ago.

Reason for this change of opinion is that nowadays developers want to use different email backends and that the number of configuration settings for email providers has been steadily growing over the years.

So we want to replace all the settings starting with EMAIL_...and replace them against a dictionary based approach such as:

EMAIL_PROVIDERS = {
    "default": {
        "BACKEND": "…",
        "HOST":  "…",
        ...        
    },
}

[Adam Johnson]

Great to see this get a ticket. Good luck Jacob, if you are planning on taking it up.

To bikeshed on the setting name, I think EMAIL_PROVIDERS is a little unclear. One might configure several backends using the same actual provider, such as for different domains. I think it would work to use EMAILS (like DATABASES, CACHES, STORAGES). Yes, it’s a little confusing that the setting configures email backends, not actual emails, but the same could be said for the other settings.

When this ticket is done, it would also be possible to add a fixer to django-upgrade to rewrite the settings, like the existing ​STORAGES fixer. If you feel brave, please give it a try!

[Mike Edmunds]

[​django-anymail maintainer here]

I'm excited to see this getting traction. There are several common, real-world use cases for it.

For the setting name, I'd suggest EMAIL_BACKENDS, since it configures email backends, and a lot of existing docs and tutorials talk about "backends." Also, EMAIL_PROVIDERS and EMAILS could imply configuration for both sending and receiving email. (But I don't feel that strongly.)

Handful of questions:

1) Are these settings provided as kwargs to the backend's constructor (with the names lowercased)? E.g., with:

EMAIL_PROVIDERS = {
    "default": {
        "BACKEND": "django.core.mail.backends.smtp.EmailBackend",
        "HOST": "smtp-relay.gmail.com",
        "USER": "app@example.com",
        "PASSWORD": env["GMAIL_APP_PASSWORD"],
    },
    "transactional": {
        "BACKEND": "anymail.backends.mailgun.EmailBackend",
        "API_KEY": env["MAILGUN_API_KEY"],
        "SENDER_DOMAIN": "app.example.com",
    },
}

am I assuming correctly that a request for the "transactional" provider would end up creating:

connection = anymail.backends.mailgun.EmailBackend(
    api_key="...", 
    sender_domain="app.example.com",
    # (and no host, user, or password args)
)

2) How does a caller request a particular provider? Is there a new argument to send_mail and friends? (name="..."? provider="..."?)

3) It would be helpful if the logging AdminEmailHandler and mail_managers could be easily configured to use a different provider than django.contrib.auth. E.g., maybe admin/manager email uses EMAIL_PROVIDERS["admin"] if present. (It's fairly common to want internal SMTP for admin notifications, but a transactional email service provider for password resets. And usually you want the transactional ESP to be the default.)

In general, for third-party libraries that send email, do you envision them adding a new setting to select an email provider (ALLAUTH_EMAIL_PROVIDER = "transactional")? Or maybe django-allauth would want to try provider="allauth" first and fall back to provider="default"? Or…?

4) Is DEFAULT_FROM_EMAIL affected by this at all? There's an argument the default from_email will often need to vary by provider, but that might require changes to all email backends. The simplest answer is no, it's a global default across all providers. (Ditto SERVER_EMAIL for admin/manager notifications.)

Again, I'm glad to see this proposal moving forward, and happy to test with django-anymail when the time comes. (Anymail backends ​already support constructor kwargs for their settings, so if I'm understanding the first item correctly, it should "just work.")

[Mike Edmunds]

[Withdrawing my earlier comment about the settings name: I've warmed to EMAIL_PROVIDERS. It's descriptive and reflects common technical jargon: "Email Service Provider" is how most providers of email sending APIs describe themselves. And it naturally leads to a terse, meaningful param name: provider="<key>"—vs. something like name=, backend_id=, connection_name=, backend= (already used for something else), or email= (which would practically guarantee confusion).]

[Mike Edmunds]

Related to my question (1) above, is there going to be a certain set of distinguished keys that can be used at the root of a provider definition, like HOST and USER, while other parameters have to be put inside OPTIONS (like with the current DATABASES setting)? I'm, uh, "asking for a friend" who maintains a bunch of non-SMTP email backends that tend to require params like API_KEY or SERVER_TOKEN but couldn't care less about USER or PASSWORD.

I'm bringing this up because—just like DATABASES—I suspect we'll eventually want to allow email provider configuration that isn't a backend param. For example, a hypothetical MESSAGE_DEFAULTS feature could address both my question (4) above about provider-specific DEFAULT_FROM_EMAIL/SERVER_EMAIL as well as a suggestion in #36365 to allow setting header options:

EMAIL_PROVIDERS = {
    "admin": {
        "BACKEND": "django.core.mail.backends.smtp.EmailBackend",
        "OPTIONS": {
            "host": "smtp-relay.gmail.com",
            "user": "app@corp.example.com",
            "password": env["GMAIL_APP_PASSWORD"],
        },
        "MESSAGE_DEFAULTS": {
            "from_email": "app-operations@corp.example.com",
            "reply_to": "it@corp.example.com",
            "headers": {"Auto-Submitted": "auto-generated"},
        },
    },
    "default": {
        "BACKEND": "anymail.backends.mailersend.EmailBackend",
        "OPTIONS": {
            "api_token": env["MAILERSEND_API_TOKEN"],
        },
        "MESSAGE_DEFAULTS": {
            "from_email": "noreply@app.example.com",
            # TODO: upgrade our MailerSend account to allow headers
            # "headers": {"Auto-Submitted": "auto-generated"},
        },
    },
}

(Here I've just pushed all backend params into OPTIONS, to avoid relegating non-SMTP backends to second class. Also, to be clear, I'm not proposing MESSAGE_DEFAULTS become part of this work; just asking how we leave room for configuring a feature like it.)

[Jacob Rief]

Created a draft pull request for this ticket: ​https://github.com/django/django/pull/18421

[Jacob Rief]

@Mike Edmunds

addressing your questions from June 10th, 2024

ad 1) yes, the provider name usually will be lowercased, jsut as with databases, caches, etc.

ad 2) I would prefer send_mail(provider="…", …)`.

ad 3) good point, should maybe be implemented in a separate issue.

ad 4) currently I did neither address DEFAULT_FROM_EMAIL nor EMAIL_SUBJECT_PREFIX.

addressing your questions from June 25th, 2024

first) On the sprints at DjangoCon Europe we (Natalia, Carlton, Jacob) brainstormed for meaningful names. The conclusion was that EMAIL_PROVIDERS was the best choice.

second) hmm, didn't think about this yet. Currently it would behave just like using the setting EMAIL_… but moved into a dictionary based setting.

[Jacob Rief]

@Mike Edmunds

in your second post on 2023-06-25 you made a good point about using a sub-dict OPTIONS to maintain non-SMTP email backends. I now changed the code to support this.

We probably should consider use-cases, where users want to be informed through messaging apps rather than email. Signal, WhatApp, Telegram, etc. all offer an API for this purpose and in Django we should allow easy integration for them. Therefore I fully back your proposal.

[Mike Edmunds]

@Jacob Rief Appreciate the responses.

You raise a really interesting idea about non-email messaging apps—I actually hadn't considered that case. (It probably merits its own discussion at some point.)

I know of three cases where existing third-party libraries deal with Django email and are likely to be impacted by EMAIL_PROVIDERS. I'd hope we can consider all of these in the design:

1. Email backends that call ESPs' HTTP APIs to send email. (This is what I meant by "non-SMTP email backends.")
   • Examples: ​django-ses, ​postmarker, ​django-anymail (disclosure: I maintain django-anymail)
   • These backends typically have their own settings (such as POSTMARK_API_KEY) that are different from Django's existing EMAIL_* settings
   • I think your latest PR update to support OPTIONS effectively handles their needs. (And in many cases, these libraries won't even need to be updated. That's great!)

2. "Wrapper" email backends which add functionality (such as an asynchronous send queue) and then connect to another email backend for the actual sending
   • Examples: ​django-mailer, ​django-celery-email, ​django-post-office
   • These packages typically have their own setting to specify the "wrapped" backend and then call Django's get_connection(backend=setting_value). E.g.: with django-mailer you set EMAIL_BACKEND = "mailer.backend.DbBackend" and then MAILER_EMAIL_BACKEND = "django.core.mail.backends.stmp.EmailBackend".
   • These libraries should continue to work for now with deprecated EMAIL_* settings, but will start to break as the deprecated settings are removed. We should probably deprecate the backend param to get_connection() now, and add a new get_connection(provider=...) option. (Or deprecate get_connection() and add a new get_provider() to replace it.)
   • Wrapper email backends will need updates to properly use EMAIL_PROVIDERS. For example, django-mailer could replace its MAILER_EMAIL_BACKEND setting with a new provider option:

     EMAIL_PROVIDERS = {
         "default:" {
             "BACKEND": "mailer.backend.DbBackend",
             "OPTIONS": {
                 # django-mailer needs a NEW option for its wrapped provider:
                 "provider": "smtp",
             },
         },
         "smtp": {
             "BACKEND": "django.core.mail.backends.stmp.EmailBackend",
             "OPTIONS": { ... },
         }
     }
     

   • Or, could we think of a way django-mailer could generically wrap all defined EMAIL_PROVIDERS (except itself), without the user needing to specify a wrapped version of each?

3. Third-party libraries that send email
   • Examples: ​django-newsletter; email confirmation in ​django-allauth and ​django-user-accounts; password reset email in django.contrib.auth
   • As I mentioned earlier, a common use case is wanting a transactional ESP for password resets, internal SMTP for admin messages, and a bulk ESP for newsletters
   • How would we expect these libraries to allow specifying EMAIL_PROVIDER aliases for the different types of mail they send?
   • (django.utils.log.AdminEmailHandler is another case of this, and already supports an ​email_backend option. I'd suggest it should also have an email_provider option, and as you say that could be a separate ticket.)

For B & C, I'm just suggesting that we think through what we might expect those packages to do, and then make sure the EMAIL_PROVIDERS feature will be compatible with those expectations. (Does that make sense?)

[btw, I agree the names EMAIL_PROVIDERS and provider="<alias>" are the best choice. My earlier comments on naming were in response to comment:1 from Adam Johnson, and I came to the same conclusion as you and Natalia and Carlton.]

[Jacob Rief]

Now, that I touched Django's mail sending code, I noticed that the current implementation has a serious flaw: get_connection(backend=…) returns a configured email backend. This means that whenever one calls this function, the connection parameters must be provided together. The consequence is that typical configuration settings, such as hostname, user, password, etc. must be provided from inside the calling code, rather that from a configuration file, aka settings.py. I therefore left the backend parameter in get_connection but made it mutually exclusive with the new parameter provider. In my opinion we should add a deprecation warning whenever someone uses backend. What do you think?

Anyway, the current implementation from my pull request seems to work now. I still have to add/change the documentation. Does anybody want to review this pull request?

[Mike Edmunds]

In my opinion we should add a deprecation warning whenever someone uses backend. What do you think?

Agreed. (It's also going to cause problems for wrapper email backends after the deprecated EMAIL_* settings are removed—see item B in comment:12.)