Python 3 error in runserver when a directory is missing __init__.py

[Jessamyn Smith]

Using Python 3.4.2
One app directory is missing a number of files, including init.py
This results in an error when trying to run any manage.py command.
It seems odd that the app directory is showing up multiple times in the path, and that duplication seems to cause issues for Django.

Here is an example traceback:

(testenv)~/Development/testenv-dev/testenv$ python manage.py runserver
Traceback (most recent call last):
  File "/Users/jessamyn/.virtualenvs/testenv/lib/python3.4/site-packages/django/apps/config.py", line 100, in create
    entry = module.default_app_config
AttributeError: 'module' object has no attribute 'default_app_config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/jessamyn/.virtualenvs/testenv/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Users/jessamyn/.virtualenvs/testenv/lib/python3.4/site-packages/django/core/management/__init__.py", line 354, in execute
    django.setup()
  File "/Users/jessamyn/.virtualenvs/testenv/lib/python3.4/site-packages/django/__init__.py", line 21, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/jessamyn/.virtualenvs/testenv/lib/python3.4/site-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/Users/jessamyn/.virtualenvs/testenv/lib/python3.4/site-packages/django/apps/config.py", line 103, in create
    return cls(entry, module)
  File "/Users/jessamyn/.virtualenvs/testenv/lib/python3.4/site-packages/django/apps/config.py", line 41, in __init__
    self.path = self._path_from_module(app_module)
  File "/Users/jessamyn/.virtualenvs/testenv/lib/python3.4/site-packages/django/apps/config.py", line 70, in _path_from_module
    "with a 'path' class attribute." % (module, paths))
django.core.exceptions.ImproperlyConfigured: The app module <module 'emptyapp' (namespace)> has multiple filesystem locations (['/Users/jessamyn/Development/testenv-dev/testenv/emptyapp', '/Users/jessamyn/Development/testenv-dev/testenv/emptyapp', '/Users/jessamyn/Development/testenv-dev/testenv/emptyapp']); you must configure this app with an AppConfig subclass with a 'path' class attribute.

[Carl Meyer]

I don't know how or why a Python 3 namespace package would end up with a __path__ containing the same directory multiple times (seems like a bug in Python, if someone is able to track down the cause), but regardless Django may as well be robust against it. We should just de-dupe __path__, probably by converting to a set, in AppConfig._path_to_module.

[Caio Ariede]

I couldn't reproduce this issue.

jessamynsmith - could you please provide more information on how to reproduce it?

[Jessamyn Smith]

Replying to caioariede:

I couldn't reproduce this issue.

jessamynsmith - could you please provide more information on how to reproduce it?

Here is a repository which replicates the issue:

​https://github.com/jessamynsmith/django-error-python3

[Caio Ariede]

This is odd.

Seems to be related to this bug in Python: ​http://bugs.python.org/issue19469

Could you please give the output for the following commands?

python --version
python -c 'import no_init; print(no_init.__path__)'

Thanks!

[Jessamyn Smith]

$ python --version
Python 3.4.2
$ python -c 'import no_init; print(no_init.path)'
_NamespacePath(['/Users/jessamyn/Development/django_error_py3/django_error_py3/no_init', '/Users/jessamyn/Development/django_error_py3/django_error_py3/no_init'])

[Carl Meyer]

@caioariede I don't think we need or want an end-to-end test for this that tries to recreate the actual Python bug using a real package (it seems it's platform-specific). It would be sufficient to just add a unit test using a fake module object with a __path__ attribute containing dupes, and add the de-duping logic in _path_from_module as mentioned above, and reference the Python bug url in a comment.

[Caio Ariede]

Replying to carljm:

@caioariede I don't think we need or want an end-to-end test for this that tries to recreate the actual Python bug using a real package (it seems it's platform-specific). It would be sufficient to just add a unit test using a fake module object with a __path__ attribute containing dupes, and add the de-duping logic in _path_from_module as mentioned above, and reference the Python bug url in a comment.

Yes, I will do. Seems to be related to Python 3.4.2 (I want to check), I couldn't reproduce it with 3.4.3.

[Tim Graham]

If it's fixed in 3.4.3, then I suggest to close as "won't fix" as we only officially support the latest micro release of each Python series.

[Carl Meyer]

Yes, I agree with Tim -- if it's fixed in 3.4.3 then we don't need to do anything.

[Caio Ariede]

Cannot reproduce it with Python 3.4.2 too. I'm assuming this bug is still alive in Python 3.x

Pull request: ​https://github.com/django/django/pull/5182

[Tim Graham]

Jessamyn, it looks like you are on Mac. How did you install Python on your machine? I also haven't been able to reproduce the problem on Python 3.4.0 and Ubuntu 14.04. Can you test the proposed patch?

[Jessamyn Smith]

Replying to timgraham:

Jessamyn, it looks like you are on Mac. How did you install Python on your machine? I also haven't been able to reproduce the problem on Python 3.4.0 and Ubuntu 14.04. Can you test the proposed patch?

Sorry, I tried to reply to this previously and it looks as if my reply was lost. I installed Python using homebrew. I have not had a chance to try to get 3.4.0. I had the issue on 3.4.2 and not on 3.4.3. I will try to get a chance to test the patch later today.

[Jessamyn Smith]

I can confirm that the proposed patch fixes the issue.

[Tim Graham]

Are you saying that the issue is fixed on Python 3.4.3 or that you didn't have a chance to test there?

[Jessamyn Smith]

Replying to timgraham:

Are you saying that the issue is fixed on Python 3.4.3 or that you didn't have a chance to test there?

I tested using python 3.4.3

[Caio Ariede]

I'm on Mac and I couldn't reproduce this issue with 3.4.2 or 3.4.3.

The Python bug report says it even affects Python 3.5.

[Tim Graham]

Jessamyn, it'd be great if you could provide a test case for the ​Python bug or at least nail down the reason why you can reproduce the problem but no one else who has tried can. I'm a bit reluctant to commit the patch until we figure that out.

[Jessamyn Smith]

Replying to timgraham:

Jessamyn, it'd be great if you could provide a test case for the ​Python bug or at least nail down the reason why you can reproduce the problem but no one else who has tried can. I'm a bit reluctant to commit the patch until we figure that out.

So manage.py in that repo runs fine for the rest of you, using python 3.x and django 1.8 on OS X?

[Caio Ariede]

Yes, it runs fine for me. Using either 3.4.2 and 3.4.3 on OSX.

[Carl Meyer]

Personally I think we should just commit the patch. It's harmless, correct, improves robustness, and it has zero downside.

[Tim Graham]

I'm not entirely opposed, but I think it would nice not to commit a patch that references a Python bug which is closed as "cannot reproduce" which seems to be where that bug is headed unless it gets some help. Of course, if we cannot understand the issue any better, we can just remove the ticket reference in our patch.

[Jessamyn Smith]

Replying to timgraham:

I'm not entirely opposed, but I think it would nice not to commit a patch that references a Python bug which is closed as "cannot reproduce" which seems to be where that bug is headed unless it gets some help. Of course, if we cannot understand the issue any better, we can just remove the ticket reference in our patch.

I'm not sure what else to investigate. My system:
OS X 10.10.5
Using Terminal.app with bash
Python 3.4.3 installed using homebrew
virtualenv==13.0.3
virtualenvwrapper==4.3.3.dev34

I just tested and the old code also fails without using virtualenv. I wonder why it doesn't fail on others' machines.

[Tim Graham]

All right, I guess we just merge the patch and move on to more important things.

[Tim Graham <timograham@…>]

In 6a98396:

Fixed #25246 -- Guarded against duplicate paths in AppConfig.