map says a queryset is not iterable

[ihucos]

So I got a funny bug with iterators map and Django that I can not localize or reproduce.
In our production server we get this sometimes, mainly for only one customer and it only fails on a very small percentage of all requests.

Sorry for only chunks of information, not sure if it is enough to process this bug request:

Th error message:

TypeError: argument 2 to map() must support iteration

Chunk of the code where this happens:

            if filter(
                    lambda obj:
                    self.pk in verbose_map(attrgetter('pk'),
                                           obj.resources.all()) and
                    obj.begins < end and obj.ends > dt, resource_override_qs):
                return 0

Trying to reconstruct resource_override_qs I come up with:

facility = Facility.objects.\
                prefetch_related('resources',
                                 'opening_hours',
                                 'openinghour_overrides',
                                 'slow_hours').\
                get(pk=facility.pk)
 resource_override_qs = facility.facilityresourceoverride_set.prefetch_related('resources')

What I did was to replace map with verbose_map in our production environment.

def verbose_map(function, iterable):
    try:
        return map(function, iterable)
    except TypeError, exc:
        raise TypeError('map failed: "{}". iterable: {}, type: {}, attrs: {}'.format(  # NOQA
            exc, iterable, type(iterable), dir(iterable)))

After two weeks or so the error occurred again:

map failed: "argument 2 to map() must support iteration". iterable: [<FacilityResource: 4>, <FacilityResource: 5>, <FacilityResource: 6>, <FacilityResource: 7>], type: <class 'django.db.models.query.QuerySet'>, attrs: ['__and__', '__bool__', '__class__', '__deepcopy__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__getstate__', '__hash__', '__init__', '__iter__', '__len__', '__module__', '__new__', '__nonzero__', '__or__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_add_hints', '_as_sql', '_batched_insert', '_clone', '_create_object_from_params', '_db', '_earliest_or_latest', '_extract_model_params', '_fetch_all', '_filter_or_exclude', '_for_write', '_has_filters', '_hints', '_insert', '_known_related_objects', '_merge_known_related_objects', '_merge_sanity_check', '_next_is_sticky', '_populate_pk_values', '_prefetch_done', '_prefetch_related_lookups', '_prefetch_related_objects', '_prepare', '_raw_delete', '_result_cache', '_setup_aggregate_query', '_sticky_filter', '_update', 'aggregate', 'all', 'annotate', 'as_manager', 'bulk_create', 'complex_filter', 'count', 'create', 'dates', 'datetimes', 'db', 'defer', 'delete', 'distinct', 'earliest', 'exclude', 'exists', 'extra', 'filter', 'first', 'get', 'get_or_create', 'in_bulk', 'is_compatible_query_object_type', 'iterator', 'last', 'latest', 'model', 'none', 'only', 'order_by', 'ordered', 'prefetch_related', 'query', 'raw', 'reverse', 'select_for_update', 'select_related', 'update', 'update_or_create', 'using', 'value_annotation', 'values', 'values_list']

As we can see map really gets an QuerySet object, that queryset object does reveal its content in str but python's built in map says its not iterable nevertheless
Alone for the customer where this happens the most that part of the code should be called at least a couple of times per day.

I am going with a workaround like calling list() on the queryset before mapping it or so.
But my company should be ok, to try this again with another verbose_map that may provide more debugging information. My efforts to reproduce this in our code base failed.

[Tim Graham]

No, I can't figure out if or where a bug might in Django based on the report.

[Shai Berger]

FWIW, the original traceback (which your verbose_map() hides) may be much more informative. Guessing that you're on Python 2, you may want to rewrite it as something like:

def verbose_map(function, iterable):
    try:
        return map(function, iterable)
    except TypeError:
        e_cls, e_inst, traceback = sys.exc_info()
        raise TypeError, (
                       'map failed: "{}". iterable: {}, type: {}, attrs: {}'.
                       format(e_inst, iterable, type(iterable), dir(iterable))
                  ), traceback

[ihucos]

We got where it comes from.
This exception occurs after Heroku times out the request, but apparently it does not kill it: ​https://devcenter.heroku.com/articles/request-timeout

So this happens in some funny Heroku environment that I really can't reproduce.

[Brian Hahn]

I am also seeing this issue, and have to force the queryset into a list to workaround this error. Any plausible idea why this might be happening?

[Vitaliy Yelnik]

I found a way to reproduce the error.

Class A is a simplified version of QuerySet

class A(object):

    def __init__(self):
        self._result_cache = None

    def _fetch_all(self):
        if self._result_cache is None:
            self._result_cache = list(self.iterator())

    def iterator(self):
        for x in range(10):
            if x == 5:
                raise MemoryError  # the type of exception doesn't matter
            yield x

    def __iter__(self):
        self._fetch_all()
        return iter(self._result_cache)

In [2]: map(str, A())
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/home/user/workspace/project/service/models.pyc in <module>()
----> 1 map(str, A())

TypeError: argument 2 to map() must support iteration

When using python 3:

In [6]: map(str, A())
---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-6-aad9944017be> in <module>()
----> 1 map(str, A())

<ipython-input-5-4a8af6429c4d> in __iter__(self)
     15 
     16         def __iter__(self):
---> 17                 self._fetch_all()
     18                 return iter(self._result_cache)
     19 

<ipython-input-5-4a8af6429c4d> in _fetch_all(self)
      6         def _fetch_all(self):
      7                 if self._result_cache is None:
----> 8                         self._result_cache = list(self.iterator())
      9 
     10         def iterator(self):

<ipython-input-5-4a8af6429c4d> in iterator(self)
     11                 for x in range(10):
     12                         if x == 5:
---> 13                                 raise MemoryError
     14                         yield x
     15 

MemoryError:

[Carlton Gibson]

I'm going to reopen so we review the reproduce.

[Simon Charette]

I think we can close as wontfix given map doesn't turn __iter__() exceptions into TypeError on Py3k and we don't support Py2k anymore.

Thanks for the investigation Vitaliy.

[Vitaliy Yelnik]

There are at least two ways to workaround the error on Py2k:

• force the queryset into a list when using map
• use imap (from itertools) instead of map

[Philipp Kuznetsov]

The problem is not only with the map. We faced the same issue when using code like this:

if model_instance in model.objects.all():
    ...

Replying to Simon Charette:

I think we can close as wontfix given map doesn't turn __iter__() exceptions into TypeError on Py3k and we don't support Py2k anymore.

Thanks for the investigation Vitaliy.

[Simon Charette]

Philipp, if you hit a similar exception that means map must be involved somehow.

If you didn't call it yourself then it's likely caused by a usage of map internally in the queryset iteration code.

In this case the ticket resolution still stands, Python 2 map's shadowing of the underlying exception makes it impossible to determine its true nature and thus we cannot conclude Django is at fault. As mentioned previously this should all be solved on Python 3 where map doesn't exhibit this behaviour.

By the way an in operation on a queryset you are disposing off is likely to trigger a MemoryError just like in the original report. I'd suggest you opt for something along queryset.filter(pk=instance.pk).exists() instead.

[Philipp Kuznetsov]

I'm really sorry, of course in case of in exception message is argument of type 'QuerySet' is not iterable. This has nothing to do with map and ofc there's an obvious workaround with exists(). What I am afraid of is if there are more of this hidden cases where queryset's __iter__ raises an exception and it's shadowed by the outer code exception.

Replying to Simon Charette:

Philipp, if you hit a similar exception that means map must be involved somehow.

If you didn't call it yourself then it's likely caused by a usage of map internally in the queryset iteration code.

In this case the ticket resolution still stands, Python 2 map's shadowing of the underlying exception makes it impossible to determine its true nature and thus we cannot conclude Django is at fault. As mentioned previously this should all be solved on Python 3 where map doesn't exhibit this behaviour.

By the way an in operation on a queryset you are disposing off is likely to trigger a MemoryError just like in the original report. I'd suggest you opt for something along queryset.filter(pk=instance.pk).exists() instead.

[Abhijeet Viswa]

I'm unsure if bumping this up is against forum rules. However, I get this Error in Py3k in production. I'm not using map in my code, though I'm unsure if Django is internally using it.

This is the code block that produces the error. The save method is overriding the save method of a Mode called Transaction.

def save(self, *args, **kwargs):
        obj = self
        if not obj.ref_number:  # Set the reference number before saving
            transactions = Transaction.objects.values_list("ref_number", flat=True)     # cache the list
            while True:
                ref_number = random.randint(1000000001, 9999999999)
                if ref_number not in transactions:
                    obj.ref_number = ref_number
                    super(Transaction, obj).save(*args, **kwargs)
                    return
        else:
            super(Transaction, obj).save(*args, **kwargs)

The erroring line of code is: if ref_number not in transactions:
This piece of code was working fine until we had modified another unrelated part of the source code.

items = OrderItem.objects.prefetch_related('toppings').select_related('item').select_related('item__category')
        order = Order.objects.filter(uuid=order_uuid).prefetch_related(
            Prefetch(
                lookup='items',
                queryset=items
            )
        ).select_related('outlet').select_related('user').select_related('outlet__group') \
         .select_for_update(of=('self', 'outlet'))

What we did was add the call to select_for_update, followed by a transaction which modifies the Order and Outlet model. Here, the Order model is a child model of the Transaction model.

My guess as to what the problem is is that select_for_update locks the relevant rows/table which results in the line of code in the save method to fail internally, ultimately being caught as a cryptic type error. I'll be modifying
transactions = Transaction.objects.values_list("ref_number", flat=True)
to
transactions = Transaction.objects.values_list("ref_number", flat=True).list() to see if that'll fix the problem.

PS: I'm sorry if my formatting/post/bump is against norms. I'm new here, with this being my first comment.

Replying to Simon Charette:

I think we can close as wontfix given map doesn't turn __iter__() exceptions into TypeError on Py3k and we don't support Py2k anymore.

Thanks for the investigation Vitaliy.