Changes between Initial Version and Version 3 of Ticket #20584


Ignore:
Timestamp:
Jul 6, 2018, 4:29:54 AM (6 years ago)
Author:
Christian Barcenas
Comment:

Re-opening this ticket, and updating the description with more details. The Memcache backend's get_many() function definitely fails when the key inputs are iterables which can be consumed only once, such as generators.

Legend:

Unmodified
Added
Removed
Modified

  • Ticket #20584

    • Property Has patch set
    • Property Triage Stage Unreviewed → Ready for checkin
    • Property Summary Django's Memcached backend get_many() doesn't handle generators → Django's Memcached backend get_many() doesn't handle iterators
    • Property Version 1.5 → master

  • Ticket #20584 – Description

    initial v3  
    1 When the "keys" parameter to get_many() is a generator, the values will be lost in the zip function. 
     1When the `keys` parameter to `get_many()` is an iterator, its values will be consumed in a list comprehension, but then later the already-consumed iterator is passed to `zip`.
    22
    3 https://github.com/django/django/blob/master/django/core/cache/backends/memcached.py#L93
     3https://github.com/django/django/blob/master/django/core/cache/backends/memcached.py#L90
    44
    5 Here's a simplified code example:
    6 
     5This causes a very confusing `ValueError` which is raised when the cache backend attempts to map lower-level Memcache-backend cache keys back to higher-level Djanco cache keys.
    76
    87{{{
    9 def make_key(k):
    10         return k
    11 
    12 user_ids = (11387, 1304318)
    13 
    14 keys = ('user_%d' % x for x in user_ids)
    15 
    16 new_keys = map(lambda x: make_key(x), keys)
    17 
    18 m = dict(zip(new_keys, keys))
    19 
    20 assert( m == {} )
     8Traceback (most recent call last):
     9  File "[snip]/django/tests/cache/tests.py", line 1345, in test_get_many_accepts_iterator
     10    values = cache.get_many(iter(['fizz']))
     11  File "[snip]/django/lib/python3.6/site-packages/Django-2.2.dev20180706090656-py3.6.egg/django/core/cache/backends/memcached.py", line 91, in get_many
     12    return {m[k]: v for k, v in ret.items()}
     13  File "[snip]/django/lib/python3.6/site-packages/Django-2.2.dev20180706090656-py3.6.egg/django/core/cache/backends/memcached.py", line 91, in <dictcomp>
     14    return {m[k]: v for k, v in ret.items()}
     15KeyError: ':1:fizz'
    2116}}}
    22 
    23 
    24 I believe this is related to this zip() behaviour: http://stackoverflow.com/questions/11210300/why-does-zip-drop-the-values-of-my-generator
    25 
    26 I encountered this bug when upgrading from django 1.3 to django 1.5.1
Back to Top