Opened 9 years ago

Last modified 8 years ago

#25862 closed Bug

LazyObject doesn't support multiprocessing on Python 2 — at Version 5

Reported by: Johannes Maron Owned by: nobody
Component: Core (Other) Version: dev
Severity: Normal Keywords: py2
Cc: info@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Claude Paroz)

Initial summary: AttributeError: 'FileSystemStorage' object has no attribute 'location'

I don't really want to bug anyone with python 2.7, for I couldn't care less. But, I stumbled upon something rather odd.

Easiest is to just check out the error on travis including the stack trace:
https://travis-ci.org/codingjoe/django-stdimage/builds/94713726

Everything works fine in Python 3, the attribute is available, as it should. It it's an instance attribute and properly set in the __init__ method:
https://github.com/django/django/blob/master/django/core/files/storage.py#L169-L172

I guess the error is here:
https://github.com/django/django/blob/master/django/utils/_os.py#L24-L32

I'm not sure tho, because the method must return at least None, therefore I don't see how the attribute error is being raised.

Change History (5)

comment:1 by Claude Paroz, 9 years ago

Resolution: worksforme
Status: newclosed

Creating a storage instance as a function parameter ((DefaultStorage() for render_variations) looks a bit suspicious to me. I'm not convinced the problem is on Django's side.

comment:2 by Johannes Maron, 9 years ago

Let me try to convince you then. I just changed it to the documented default_storage helper, which is nothing other than DefaultStorage().

I also just added a little test, that tests only storage.location. That works, that doesn't mean there isn't a problem tho.
See the tests don't even execute the case with a default storage.
It uses the FieldFile's storage.

Besides all that, I don't do any wizardry with the storages, nor in the settings. It must strike you, that there is a different behavior on python 2.7 and 3.*

Last edited 8 years ago by Tim Graham (previous) (diff)

comment:3 by Johannes Maron, 9 years ago

Resolution: worksformefixed

I found the bug. It's the lazy object implementation of the default storage. It doesn't work in python 2 using multiprocessing.

This little snippet allows to produce the error.

def has_location(storage):
        return hasattr(storage, 'location')


def test_multiprocessing_lazy_objects(image_upload_file):
    SimpleModel.objects.create(image=image_upload_file)
    obj = SimpleModel.objects.last()
    storage = obj.image.storage

    p = Pool(2)
    assert all(p.map(has_location, [default_storage, storage]))

comment:4 by Johannes Maron, 9 years ago

Resolution: fixed
Status: closednew

comment:5 by Claude Paroz, 9 years ago

Description: modified (diff)
Summary: AttributeError: 'FileSystemStorage' object has no attribute 'location'LazyObject doesn't support multiprocessing on Python 2
Triage Stage: UnreviewedAccepted
Note: See TracTickets for help on using tickets.
Back to Top