raw() sql bug when using a model with many fields with long names

[Kevin Renskers]

I love the new raw() function to write custom sql and get back ORM objects. However, when using joins, things get weird.

As long as you use "select *", everything works fine:

for r in Model.objects.raw('SELECT * FROM model INNER JOIN othermodel ON othermodel.model_id = model.id'): 
    print r 

However, this can result in a LOT of columns that you don't need, and possible duplicate column names.

Solution would seem to be this:

for r in Model.objects.raw('SELECT model.id, model.name FROM model INNER JOIN othermodel ON othermodel.model_id = model.id'): 
    print r 

Sadly, this doesn't work. I get an "type() argument 1 must be string, not unicode" error from django/db/models/query_utils.py in deferred_class_factory, line 274.

Weirdly enough, this does work:

for r in Model.objects.raw('SELECT model.* FROM model INNER JOIN othermodel ON othermodel.model_id = model.id'): 
    print r 

[Kevin Renskers]

I found the source of the problem. My model has a lot of fields, and most of them have long, verbose names.

The error is caused in django/db/models/query_utils.py, on line 267:
name = "%s_Deferred_%s" % (model.name, '_'.join(sorted(list(attrs))))

This results in a *very* long string, which cannot be handled, or so it seems.

When I change this line to something like this, it does work:
name = "%s_Deferred_%s" % (model.name, '_lala')

It's weird that this problem only presents itself when you don't use model.*. When using model.* in the raw query, everything works fine.

This also fixes bug #13712.

[anonymous]

This problem also presents itself in the admin. When I try to load the admin page for my model, I get this error:

Exception Type: TypeError at /admin/biobench/measurement/
Exception Value: __init__() keywords must be strings

Traceback:

File "/opt/APPS/biobench/acc/Biobench/packages/django/core/handlers/base.py" in get_response
  100.                     response = callback(request, *callback_args, **callback_kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/contrib/admin/options.py" in wrapper
  239.                 return self.admin_site.admin_view(view)(*args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in _wrapped_view
  76.                     response = view_func(request, *args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/views/decorators/cache.py" in _wrapped_view_func
  69.         response = view_func(request, *args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/contrib/admin/sites.py" in inner
  190.             return view(request, *args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in _wrapper
  21.             return decorator(bound_func)(*args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in _wrapped_view
  76.                     response = view_func(request, *args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in bound_func
  17.                 return func(self, *args2, **kwargs2)
File "/opt/APPS/biobench/acc/Biobench/packages/django/contrib/admin/options.py" in changelist_view
  1,071.             'selection_note': _('0 of %(cnt)s selected') % {'cnt': len(cl.result_list)},
File "/opt/APPS/biobench/acc/Biobench/packages/django/db/models/query.py" in __len__
  81.                 self._result_cache = list(self.iterator())
File "/opt/APPS/biobench/acc/Biobench/packages/django/db/models/query.py" in iterator
  274.                             only_load=only_load)
File "/opt/APPS/biobench/acc/Biobench/packages/django/db/models/query.py" in get_cached_row
  1,208.             obj = klass(**dict(zip(field_names, fields)))

My model has 175 fields (yeah...), which on average are 40 characters long.

[ondrej.kohout@…]

We have the same problem even with a small number of fields.

Using some_queryset.only('somefield') results in new model/contenttype creation with
diferred fields. We've used this qs method in some south migration. South
sends post_syncdb after model creations. Then django contrib.auth wants to add
permission after receiving this signal and there is a problem with
Permission.name (i.e. 'Can add page_ deferred_layout_migrated_site_id_skin_template')
field size limited to max length=50 (or Permission.codename limited to 100)

This is maybe more a problem of South application (sending signals of undeclared model
creations), but this is also a cause of uncontrolled size of generated model name in deferred_class_factory()

[Tobias McNulty]

attempting to reproduce...

[Tobias McNulty]

test showing the specified query works (and this is not a bug)

[Tobias McNulty]

same patch, named for this bug (not the duplicate)

[Tobias McNulty]

I was NOT able to reproduce this bug. I attached a test showing that the query you've given works in the test environment. If you're still having issues please try to update this test to generate the error you're experiencing, or provide more information about your environment such that we can reproduce the bug.

[Kevin Renskers]

It only generates an error when your model has a lot of fields, which have long names. The code in django/db/models/query_utils.py, line 267, generates a string that is too long to handle. With "normal" models (i.e. not a ton of fields with long names), the query does indeed work fine.

This is explained in my first comment on this ticket. I could create a small Django project to prove this error, if that would help.

[Tobias McNulty]

Yes, some code (as little as possible) demonstrating the issue would be great, thanks.

[Kevin Renskers]

OK, did some extensive testing. Turns out, my problem has nothing to do with the number of fields on a model. Rather, I had some fields with unicode names, done like this:

MyModel.add_to_class(u'field_name', models.CharField(max_length=30))

I used this add_to_class way of adding fields to a model because the fields were already defined somewhere else. But, the field names are unicode strings (because they are also used in other languages with weird characters). This causes the "type() argument 1 must be string, not unicode" error in deferred_class_factory.

Now that I know this, I can cast my field names to the proper string type before adding them to my model. But maybe it's also a good idea to cast the name variable to a string in the deferred_class_factory function, just to be safe (see django/db/models/query_utils.py, line 267)

Still, the size of the name variable can really get out of hand when you have a lot of fields. In my case, it was a string with over 8000 characters...

[anonymous]

this is a probleblem with mentioned uncontrolled size of generated model name. Collecting field names to make a model name.. nono

[Tobias McNulty]

As far as I can tell, add_to_class is an undocumented method and hence unsupported. Probably just best if you ensure that you pass in non-unicode strings.