Adds per-view cache refreshing (clearing)

[k0001]

This patch adds the possibility to clear(refresh) cached data per view.

SmileyChris sugested it would be good to add another key to the cached keys where we could keep track of all the different cache_page_key hashes, so we that can delete those then.

It is possible to clear the cache for an specific view, to achieve that, you
should tell your view's url path (and optionally, a key_prefix) to
clear_cache_for_path(path, key_prefix=None).

    from django.utils.cache import clear_cache_for_path
    
    # clear the all cached data for path '/blog/posts/2/'
    clear_cache_for_path('/blog/posts/2/')

Note that it will delete every page cached from matching this path, Vary headers
doesn't matter.

NOTE: Thanks SmileyChris for your help and supervision with this.

[k0001]

adds per-view clear cache support

[Jacob]

I like the idea. However, I'm -1 on the implementation -- adding another key to the cache seems unnecessary.

[Chris Beaven]

I don't see how we can get around adding a key to index the view's cache keys. Since they are hashed, there's no way of retrieving them with the current cache API (you'd need some type of .get_starts_with() method which I'm not sure would be even possible)

[Julian Bez]

One could argue that you are doing something wrong if you need per-site or per-view cache clearing.

If you have a page that changes more than once in a month, you are better off using the cache template tag. This allows you to share cached content between views and the keys can be deleted more easily when something got updated. With per-view caching you always cache the whole page, for every page. Imagine how often you cache your header or footer that way if you have a larger site.

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

Change Easy pickings from NULL to False.

[Aymeric Augustin]

Setting to DDN since there's some disagreement between Jacob and Chris.

Explicit cache invalidation can be useful in some scenarios, it'd be nice to support this.

[Jacob]

If there's really not a better implementation (perhaps there isn't) then I'm OK with this going in as-is.

[Carlton Gibson]

#31938 was a duplicate request similar functionality.

[Mike Lissner]

Just to add a data points here, I spent a bunch of time today trying to figure out which key to delete to nuke a cached page and eventually gave up and just nuked a larger part of the cache in hopes of getting lucky (I did).

Having a helper util just to figure out what the cache key value is would be really nice.

Here's a summary of my failed research:

• "Expire a view-cache in Django?" (​https://stackoverflow.com/q/2268417/64911) — Says it works with < django 1.7. I didn't try it, but I don't think it'd work.
• "How to invalidate cache_page in Django?" (​https://stackoverflow.com/q/33749369/64911) — Accepted answer just says to cache the queryset instead of using cache_page.
• "How to manually clear/update a cached view in django" (​https://stackoverflow.com/q/47040438/64911) — Has a solution, but it's quite old and doesn't work. This is when I gave up.

The reason I need this (and couldn't wait for the cache to just expire) is because I use the db cache to cache really big sitemaps that don't change very often. The cache is usually on the order of a couple of weeks.

[Carlton Gibson]

#34271 was a duplicate with a suggestion to make the cache key just depend on the request path:

from pathlib import Path

from django.conf import settings
from django.core.cache import caches

default_cache = caches[settings.CACHE_MIDDLEWARE_ALIAS]
cache_key = ".".join(Path(request.path[1:]).parts)

if not (response := default_cache.get(cache_key)):
    response = view_func(request, *args, **kwargs)
    response.add_post_render_callback(
        lambda r: default_cache.set(cache_key, r, timeout)
    )

That would be easy to invalidate outside the request context.

Allowing/documenting easier make_key customisation might allow us to move this forward? 🤔

[Ahter Sönmez]

I haven't had a chance to look at this since the sprint in DjangoCon Europe (nearly 6 months ago). As far as I remember, I already had "something" working.

In the next few days, let me check what was done, and share some progress here.

[Ahter Sönmez]

Here's a quick update on this:

I think this can be done. However, there are a few things to take into consideration.

• Django's core caching code is quite old, 18+ years. I believe big part of it should be redesigned and refactored so it's easy to work on it and extend it in non-hacky ways. While this refactor is out of scope for this PR, I think it might be beneficial. I think this might be why this ticket is quite old.
• Even when we solve this problem, due to the way cache keys are generated, this will be only supported by some cache backends (ie. Redis) which allow filtering by keys or part of keys.
• There are better design patterns that would allow more intelligent caching in the projects/apps that use Django. Especially projects that use layered architecture. However, this is not applicable to Django as a library as it cannot be opinionated about how devs would like to structure their code in their apps.

I finally now have a chance to have a look at this again in DjangoCon Europe 2023 sprints. This time we've also paired with Marco Silva and we're tackling the issues together. I'm hoping to work on this more (given we're now two) and get something out as a PR soon.

[Petr Dlouhý]

I would like to add my use case to widen the perspective.
I am running a website with ~4k RPM and clearing cache for some view could result in timeouts due to cache hammering.
Multiple requests would bump into the cleared cache at the same time and it would result in too many DB requests which would slow down the single database significantly (the requests would take tens of seconds).

So clearing the cache even for single page/view is not an option for me.
The solution for me would be to directly inject new keys.