Add a mechanism for page cache invalidation.

[Laurent Tramoy]

I've recently started to use caching for some of my pages, through the cache_page decorator, everything works fine.

However, as soon as any dev implements a caching mechanism, they also need to think about cache invalidation, and that's my problem: there is no easy way to invalidate a specific page in Django, since the key relies on the hash cached request's full URL

I think it would be nice for Django to provide helpers to invalidate:

• a specific url with specific query params, no matter the headers
• a specific url with any query params, for one or more specific headers (among others)
• a group of urls

and the same list with a path instead of a full URL, in case the app uses a single scheme, host, and port.

Problem is, we would need to have several distinct cache keys to be able to do all this, which implies more keys, more calls to the cache and / or bigger keys. Since some people may choose performance over cache control, a good solution might be to make _generate_cache_key and get_cache_key customizable.

Here is the code I wrote to implement some of these features, after looking at django.utils.cache (I use a Redis cache with django-redis):

import re
from typing import Dict, Optional

from django.conf import settings
from django.core.cache import cache
from django.core.handlers.wsgi import WSGIRequest
from django.utils.cache import get_cache_key


def invalidate_view_cache(
    path: str,
    *,
    vary_headers: Optional[Dict[str, str]] = None,
    key_prefix: Optional[str] = None,
    invalidate_whole_prefix: bool = False,
) -> int:
    """
    This function first creates a fake WSGIRequest to compute the cache key.
    The key looks like:
    views.decorators.cache.cache_page.key_prefix.GET.0fcb3cd9d5b34c8fe83f615913d8509b.c4ca4238a0b923820dcc509a6f75849b.en-us.UTC
    The first hash corresponds to the full url (including query params),
    the second to the header values

    if invalid_whole_prefix is True, we will invalidate all the keys matching the prefix
    that also match the header hash and prefix.
    This mean we will deletes all the keys returned by
    views.decorators.cache.cache_page.key_prefix.GET.????????????????????????????????.c4ca4238a0b923820dcc509a6f75849b.en-us.UTC
    To be safe, we won't do it if key_prefix is None or ''

    vary_headers should be a dict of every header used for this particular view
    In local environment, we have two defined renderers (default of DRF), thus DRF adds
    'Accept` to the Vary headers

    Note: If LocaleMiddleware is used, 
    we'll need to use the same language code as the one in the cached request
    """
    request = WSGIRequest(
        {
            "PATH_INFO": path,
            "REQUEST_METHOD": "GET",
            "HTTP_HOST": settings.HOST,
            "wsgi.input": None,
            "wsgi.url_scheme": f"http{'s'*settings.IS_DEPLOYED_ENVIRONMENT}",
        }
    )

    if vary_headers:
        request.META.update(vary_headers)

    cache_key = get_cache_key(request, key_prefix=key_prefix)

    if cache_key is None:
        return 0

    if invalidate_whole_prefix and key_prefix:
        # specific to the redis implementation
        keys = cache.keys(re.sub("[0-9a-f]{32}", "[0-9a-f]" * 32, cache_key, 1))
        return cache.delete_many(keys)

    return cache.delete(cache_key)

I can try to work on a more generic implementation, if you think this idea is worth it.

[Carlton Gibson]

Hi Laurent.

Thanks for the report. I was going to Accept this as a new feature — I've spent a good amount of time rifling through cache keys to work out which one to delete by hand. I think this would be a good addition.

But it looks to me like #5815 is the same issue, so let's close as a duplicate.

Can I ask you take your idea to the DevelopersMailingList and ask for some input of your thoughts for an implementation? I would be good if you want to take this on to push it forwards! 👍

[Laurent Tramoy]

Hi Carlton,

I'm sorry I never answered, I thought I would receive an alert by email, but I did not. I'll send my suggestions to the mailing list soon !