django.db.models get_apps() return different results according to the server used

[kilian <kilian.cavalotti@…>]

Using apache2 + mod_python, the admin interface do not display all of the applications listed in INSTALLED_APPS, while all those applications correctly appear in the main admin page using the integrated web server.

Weirder is the fact that, for the missing app, the 'should-be' admin URLs for adding or editing objects (admin/app/object/add/) can be successfully accessed. It seems that the missing application is correctly seen and working (objects actually can be managed with direct URLs), but it's not shown on the main admin page.

I narrowed the problem to the fact that get_apps() from django.db.models does not return the same list when called from mod_python or from the integrated server. More precisely, it seems that for one of the applications, in db/models/loading.py, the load_app() function checking for 'models' attribute behaves differently according to the server used.

The hasattr(mod, 'models') call returns False for the hostManage application using mod_python, and True for the same application using the integrated server. Thus, it's not listed in _app_list, and is not displayed on the admin page.

Adding

h = hasattr(mod, 'models')
assert False, h

in the load_app() function from db/models/loading.py gives the following with mod_python:

Variable	Value
app_name	'dNA.hostManage'
h	False
mod	<module 'dNA.hostManage' from '/var/www/dNA/hostManage/__init__.pyc'>

I removed *.pyc, and checked permissions on my folders and directories, but they seems to be ok since I the hasattr(mod, 'models') works from the shell:

>>> mod = __import__('hostManage', '', '', ['models'])
>>> mod
<module 'hostManage' from '/var/www/dNA/hostManage/__init__.pyc'>
>>> hasattr(mod, 'models')
True

So, I can't figure out why, but the hasattr(mod, 'models') function returns different values according to the server used. And I guess it's not thewanted behavior. ;)

---

for reference, this issue has been discussed ​there and ​there

[mattimustang@…]

I hit this exact same issue yesterday too when cutting over from django's dev server to apache2/mod_python 3.2.8. I have an app that calls get_model() and it wasn't working under mod_python.

The snippet below demonstrates it:

>>> from django.db.models.loading import get_model
>>> model = get_model('asset', 'Asset')
>>> repr(model)
'None'
>>> from nong.asset.models import Asset
>>> model = get_model('asset', 'Asset')
>>> repr(model)
"<class 'nong.asset.models.Asset'>"

[Malcolm Tredinnick]

The mattimustang comment has nothing to do with the initial bug report, as far as I can see. However, it is addressed/fixed in [3490].

[Malcolm Tredinnick]

A status update:

After a lot of back and forth in email with Kilian, we now understand why this is happening, although a fix still remains to be found. Kilian had a model that was using a queryset on a model as a constructor to another field. Something like this:

class ModelA(models.Model):
   # ...

class ModelB(models.Model):
   some_field = models.CharField(choices = ModelA.objects.all(), ...)

and this was causing some problems with the app/model cache in django.db.models.loading.

The fix for now is "don't to do that", but I would like to get a better solution in the end.

[oliver@…]

Interesting. I encountered a very similar (or the same) issue this morning, and tracked down this bug report.

One other twist that may or may not be related to the original case. In my case everything works (or fails to) exactly as described by killian, but removing certain lines from my models makes them reappear on the django admin interface:

class Item( models.Model ):

objects = ItemManager()

#The limit_choices_to in this line will cause this model to disaapear from django admin
type_code = models.ForeignKey( DbCode, null=False, blank=False, limit_choices_to = { 'parent_codeexact' : DbCode.objects.filter( label='ItemType')[0].id } )

DbCode in this code is just another Model used to store constants in the database (a code table). Eliminating limit_choices_to causes this model to appear. Perhaps that will help someone else find a workaround.

In another instance one of my Models which loads a queryset at global scope won't be visible on the admin page unless I remove the global query:

# A bunch of model and manager definitions preceed this slightly hackish code
try:

foo = codes
print "codes defined"

except:

try:

codes = DbCode.objects.build_code_objects()

except:

print "Warning codes object not built."

There do appear to be differences in import using the test server and mod_python/fastcgi. Tracing another bug in my code recently I found that:

from django.contrib import auth.models.User

would fail under mod_python/fastcgi but worked fine with the test server. (syntax is from memory, ymmv)

Hope this helps!

[anonymous]

Yuck. Here are those snippets wikified, my appologies:

class Item( models.Model ):
    objects = ItemManager()
    
    #The limit_choices_to in this line will cause this model to disaapear from django admin
    type_code     = models.ForeignKey( DbCode, null=False, blank=False, limit_choices_to = { 'parent_code__exact' : DbCode.objects.filter( label='ItemType')[0].id } )

and

#Global scope
try:
    foo = codes
    print "codes defined"
except:
    try:
        codes = DbCode.objects.build_code_objects()
    except:
        print "Warning codes object not built."

[portland@…]

This problem seems to have popped up again in a different context. First, some version numbers (joy):

Django SVN 5609
Python 2.5.1
mod_python 3.2.10
Apache 2.3.3

I'm trying to implement ​django-comment-utils, the package James Bennett cooked up for comment moderation. I'm using revision 72 of the SVN codebase, and I'm encountering the same issues with the admin site (app doesn't show up in admin list, though its add/edit/list pages are still available and work fine). The culprit appears to be the exact same problem&mdash;get_apps() returns a different set of applications depending on whether I'm calling it from mod_python, the development server, or shell (the latter two work, the former doesn't). Specifically, if I turn on moderation on a model, the application it's associated with disappears. (See ​http://code.google.com/p/django-comment-utils/issues/detail?id=4 for more detail)

The tricky part is none of the affected models uses a QuerySet in constructing a model field, and a cursory glance at the comment-utils code reveals nothing similar. Here's the model itself:

class Story(models.Model):
    head = models.CharField(maxlength=255)
    deck = models.CharField(maxlength=255, blank=True, null=True)
    slug = models.SlugField(prepopulate_from=("head",), help_text='Automatically written based on the headline. If nothing shows up here, try typing in the headline instead of copying and pasting.')
    section = models.ForeignKey(Section)
    issue = models.ForeignKey(Issue)
    label = models.CharField(maxlength=255, blank=True, help_text='ex. "Signed Editorial," "Feature." Optional.')
    content = models.TextField()
    summary = models.TextField(help_text='Sum up the story in a single paragraph.')
    section_order = models.PositiveSmallIntegerField("Order in section", help_text="Determines the order of all stories in the section, with lower numbers at the top (so a story with order priority 1 would be at the top).")
    enable_comments = models.BooleanField(default=True)

    class Admin:
        fields = (
            ('Top matter', {'fields': ('head', 'deck', 'summary', 'label')}),
            ('Content', {'fields': ('content',), 'classes': 'richedit'}),
            ('Metadata', {'fields': ('issue', 'section', 'section_order', 'slug')}),
            ('Comments', {'fields': ('enable_comments',)}),
        )
        list_display    = ('head', 'list_authors', 'issue', 'section',)
        list_filter     = ('issue', 'section',)
        search_fields   = ['head', 'deck']
        ordering        = ('-issue',)

        js = ('js/journal/formatting-controls.js',)

    class Meta:
        verbose_name_plural     = "Stories"
        unique_together         = (("head","issue"),("issue","section","section_order"))
        ordering                = ('-issue', 'section_order')
    
    def __unicode__(self):
        return self.head

    def get_absolute_url(self):
        return ('journal.stories.views.detail_story', (), {
            'datestring': self.issue.pub_date.strftime("%Y-%m-%d"),
            'section': self.section.slug,
            'slug': self.slug})
    get_absolute_url = models.permalink(get_absolute_url)

[portland@…]

Some further digging into the issue: I think the problem is Django isn't handling some class references very well. The culprit in my case turned out to be that in order to turn on comment moderation, I had to insert the following code somewhere:

class StoryModerator(AlwaysModerate):
    akismet = True
    akismet_auto_delete = True
    enable_field = 'enable_comments'

moderator.register(Story, StoryModerator)

where Story is the name of the model I'm trying to moderate. If I put that code in the same models.py where Story lives, the trouble with the admin app and get_apps() not working properly surfaces. If I pull the code out of models.py and call it from a separate file (in this case a file imported by views.py) then everything works fine. Similarly, the problem with using QuerySets in the field constructor above may not be the use of a QuerySet in and of itself, but specifically the call to another model in the same models.py file.

This might not be news to anyone except me, but it wasn't obvious from the comments above and I thought it was worth mentioning just in case.

[Malcolm Tredinnick]

(In [5919]) Rewrote portions of the app- and model-cache initialisation to handle some corner cases. It is now possible to use m2m relations before everything is imported and still get the right results later when importing is complete. Also, get_apps() should always return the same results, so apps won't randomly disappear in the admin interface.

Also reorganised the structure of loading.py, since the number of global variables was exploding. The public API is still backwards compatible.

Fixed #1796 and #2438 (he claims, optimistically).

[aaraines@…]

I've been hitting a problem with loading.py the last week or so, in the same vein as this thread. One app doesn't return in get_apps(), and it's because of the hasattr(mod, "models") check in load_app(). It started happening after I added a particular model the file, but I've tried removing it and the admin still leaves out the app (it's a clean deploy, so no pyc is left around). I've added a getattr() in load_app() to see what exactly is failing, but it only gives me an ImportError, which is what I already knew.

Works fine with runserver, but not with mod_python 3.3.1/Python 2.4.4 or 2.5.1. I'm at version 6023 of django/trunk.
The models file is at ​http://draines.com/tmp/foo-models.py.txt.

[Aidas Bendoraitis aka Archatas]

I experienced this bug/feature as well. After analyzing the core code of django I found out a workaround for the problem.

If you need to assign some limit_choices_to or default value which is related to the instances of other models, you can create a function which returns the necessary value and assign the function (not called) to the limit_choices_to or default attributes. It won't be called at the load time, so the modules will be loaded correctly, but the function will be called at run time, for instance, when the administration is loaded.

[semmons99@…]

I had similar problems with my apps not all showing up when using apache2.2 vs. runserver. I added the directory to the project to my python path in httpd.conf instead of just the parent folder:

Before:

<Location "/">
  SetHandler python-program
  PythonPath "['c:/DjangoSites'] + sys.path"
  PythonHandler django.core.handlers.modpython
  SetEnv DJANGO_SETTINGS_MODULE yourdata.settings
  PythonDebug On
</Location>

After:

<Location "/">
  SetHandler python-program
  PythonPath "['c:/DjangoSites', 'c:/DjangoSites/yourdata'] + sys.path"
  PythonHandler django.core.handlers.modpython
  SetEnv DJANGO_SETTINGS_MODULE yourdata.settings
  PythonDebug On
</Location>

[kilian]

I'm posting the following comment on behalf of Dennis Parker:

I just experienced a slightly different form of this problem.

I have several apps in my site, and one of them stopped showing up on the admin home page, but was accessible by direct url.  The usual "works in dev server, doesn't work in mod_python" stuff applies. Python2.5, Apache 2.0.x and 2.2.x latest mod_python, django 7716. I read the comments on this ticket, and was able to change my code to eliminate the problem.

In my case, the app (call it "App A") that was creating the problem was NOT the one that failed to appear (call it "App B"). The offending app (App A) only had a couple of admin enabled model classes, so I don't know how much of that module was being processed successfully. What I do know is that App A had some (non-model) classes in it that did some object creation of shared instances as class attributes of other classes in the same module.

Based on a hunch generated by the comments on this page, I moved the class attribute bearing, non-model classes in App A to a different module, and App B showed up in the admin page again.

The only additional piece of data that might be relevant is that the non-model classes in question use SQLAlchemy to access a different database schema (not my model schema), and I am using the SQLAlchemy connection pooling trick for my django database (MySQL) backend, so there may be some database connection startup issues involved.

I don't know if this will help debug the ultimate problem, but I hope that anyone else who has this problem will see that they should look for initialization issues in all their app models, not just the app that disappears.

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

Change Easy pickings from NULL to False.

[Claude Paroz]

No new info since 5 years, and mod_python is now gone. Reopen if reproducible with recent code.