Management can't load custom commands when separately packaged apps share a common base module

[nOw2]

django.core.management.find_management_module() loads custom commands for manage.py by finding the path of the module and examining file directly.

This fails when apps are within packages that share a common base name, but where the files are NOT in the same directories, example:

• app 1: company.division.project_a.app1 stored in path packages/company.subdivision.project_a.app1
• app 2: company.division.project_b.app2 stored in path packages/company.subdivision.project_b.app2

Custom commands in app 2 will not be found.

A code trace from pdb follows. Excuse the rather complicated example, but this is from a genuine problem and only the modules names have been changed.

> site-packages/django/core/management/__init__.py(43)find_management_module()
-> parts.append('management')
(Pdb) parts
['company', 'subdivision', 'project_b', 'app2']

. . .

> site-packages/django/core/management/__init__.py(62)find_management_module()
-> f, path, descr = imp.find_module(part, path and [path] or None)
(Pdb) 
ImportError: 'No module named project_b'
> site-packages/django/core/management/__init__.py(62)find_management_module()
-> f, path, descr = imp.find_module(part, path and [path] or None)
(Pdb) l
 57  	        if os.path.basename(os.getcwd()) != part:
 58  	            raise e
 59  	
 60  	    while parts:
 61  	        part = parts.pop()
 62  ->	        f, path, descr = imp.find_module(part, path and [path] or None)
 63  	    return path
 64  	
 65  	def load_command_class(app_name, name):
 66  	    """
 67  	    Given a command name and an application name, returns the Command
(Pdb) part
'project_b'
(Pdb) path
'packages/company.subdivision.project_a.app1/company/subdivision'

Issue was found in 1.3.1 but the code appears the same in the current trunk version:
https://code.djangoproject.com/browser/django/trunk/django/core/management/__init__.py?rev=17842#L60

[nOw2]

The following replacement function fixes the issue, but introduces a performance problem - there's a small but noticeable delay on starting manage.py when the project contains many applications (testing with ~30 here).

def find_management_module(app_name):
    """
    Determines the path to the management module for the given app_name,
    without actually importing the application or the management module.

    Raises ImportError if the management module cannot be found for any reason.
    """

    path = os.path.join(__import__(app_name, fromlist="mangement").__path__[0], "management")
    if not os.path.exists(path):
        raise ImportError("No management commands for %s" % app_name)
    
    return path

[Andrew Godwin]

It looks like you've got packages with "." in their directory names - that's not a valid Python module name, and while "import" may import it, it does a lot of things it shouldn't do. Module names (and thus directory names) should consist only of letters, numbers and the underscore character.

[nOw2]

The packages don't have a period in their names, but the directory above the python path does though I don't believe that has an impact.

i.e. for:

packages/company.subdivision.project_a.app1/company/subdivision/project_a/app1/

PYTHONPATH="packages/company.subdivision.project_a.app1/"

Python searches this: "company/subdivision/project_a/app1/"

Package name is "company.subdivision.project_a.app1"

---

The issue comes from developing each package separately but in the same Python namespace, which is the method we've come up with to handle this particular large project.

The issue comes in because there are many such packages like this:

packages/company.subdivision.project_a.app1/company/subdivision/project_a/app1/
packages/company.subdivision.project_a.app2/company/subdivision/project_a/app2/
packages/company.subdivision.project_b.app1/company/subdivision/project_b/app1/

The way the current find_management_module() works is to find the first file/directory on disk that corresponds to the Python module "company", even though there may be many such directories. i.e. the package name "company.subdivision.project_a.app1" has a unique init.py, but the package name "company" is not.

(nb, the _ _ init _ _.py in the "shared" name components looks like this:

    import pkg_resources

    pkg_resources.declare_namespace(__name__)

)

In the above example, "import company.subdivision.project_b.app1" causes find_management_module() to fail when it can't find project_b in the directory "packages/company.subdivision.project_a.app1/company/subdivision/". That directory is first on the path to match company.subdivision but is not the correct one for company.subdivision.project_b.app1!

I recognise that this is something of an edge case.
I have yet to test if the issue continues when the packages are installed via setuptools/distribute, I'm unsure if the issue would persist once installed to site-packages or into eggs. My workaround patch is in our current production release.