AssertionError at `Query.change_aliases`

[Ram Rachum]

(Django version 1.7.7)

I'm getting an assertion error on the first line of Query.change_aliases:

        assert set(change_map.keys()).intersection(set(change_map.values())) == set()

Problem is, I can't post more data, because the queryset has confidential client information :(

I can post the value of change_map, with identifying details renamed:

    {'T5': 'T10',
     'T6': 'T11',
     'T8': 'T13',
     'my_app_follow': 'my_app_follow',
     'my_app_stack__readers': 'T7',
     'my_app_stack__writers': 'T9',
     'my_app_user': 'T8'}

This happened when doing & on two querysets, one of which having .distinct() applied on it, among other things.

[Tim Graham]

Couldn't you change the names of the models and values of the data and offer a test case to reproduce the error?

[Ram Rachum]

These querysets are passed through several functions in different files so it'll be hard to construct them in straight code. It'll be about an hour of work for me which is too much.

[Tim Graham]

Well, trying to reproduce this from the provided details will likely take much longer than an hour.

[Tim Graham]

Reporter provided the SQL from the queries that are being ANDed. Maybe this is enough to be helpful to an ORM expert; reopening on that basis.

SELECT "soufle_app_stack"."id", "soufle_app_stack"."datetime_created", "soufle_app_stack"."datetime_updated", "soufle_app_stack"."_name", "soufle_app_stack"."owner_id", "soufle_app_stack"."is_public_reading", "soufle_app_stack"."is_discussion_stack", "soufle_app_stack"."datetime_canceled" FROM "soufle_app_stack" LEFT OUTER JOIN "soufle_app_stack__readers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__readers"."stack_id" ) LEFT OUTER JOIN "soufle_app_stack__writers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__writers"."stack_id" ) WHERE ("soufle_app_stack"."datetime_canceled" IS NULL AND ("soufle_app_stack"."is_public_reading" = True OR "soufle_app_stack__readers"."user_id" = 147 OR "soufle_app_stack__writers"."user_id" = 147 OR "soufle_app_stack"."owner_id" = 147)) ORDER BY "soufle_app_stack"."datetime_created" DESC

SELECT "soufle_app_stack"."id", "soufle_app_stack"."datetime_created", "soufle_app_stack"."datetime_updated", "soufle_app_stack"."_name", "soufle_app_stack"."owner_id", "soufle_app_stack"."is_public_reading", "soufle_app_stack"."is_discussion_stack", "soufle_app_stack"."datetime_canceled" FROM "soufle_app_stack" LEFT OUTER JOIN "soufle_app_stack__readers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__readers"."stack_id" ) LEFT OUTER JOIN "soufle_app_stack__writers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__writers"."stack_id" ) INNER JOIN "soufle_app_follow" ON ( "soufle_app_stack"."id" = "soufle_app_follow"."stack_id" ) WHERE ("soufle_app_stack"."datetime_canceled" IS NULL AND ("soufle_app_stack"."is_public_reading" = True OR "soufle_app_stack__readers"."user_id" = 147 OR "soufle_app_stack__writers"."user_id" = 147 OR "soufle_app_stack"."owner_id" = 147) AND NOT ("soufle_app_stack"."id" IN (SELECT U1."private_stack_id" AS "private_stack_id" FROM "soufle_app_user" U1 WHERE (U1."id" = 147 AND U1."private_stack_id" IS NOT NULL))) AND NOT ("soufle_app_stack"."id" IN (SELECT U1."public_stack_id" AS "public_stack_id" FROM "soufle_app_user" U1 WHERE (U1."id" = 147 AND U1."public_stack_id" IS NOT NULL))) AND "soufle_app_follow"."user_id" = 147) ORDER BY "soufle_app_stack"."datetime_created" DESC

[Ram Rachum]

Can anyone possibly help with this? I'm stuck on this problem and I don't know how to solve it.

[Josh Smeaton]

Can you try your queryset out with django 1.8 please? I've tried to emulate the idea by combining two querysets, one that has distinct applied, and that behaviour is not allowed (in 1.8):

In [9]: Company.objects.filter(name='HI').distinct() & Company.objects.exclude(name='HI')
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-9-b0c9de44f4df> in <module>()
----> 1 Company.objects.filter(name='HI').distinct() & Company.objects.exclude(name='HI')

/Users/smeatonj/Development/django/django/db/models/query.py in __and__(self, other)
    305         combined = self._clone()
    306         combined._merge_known_related_objects(other)
--> 307         combined.query.combine(other.query, sql.AND)
    308         return combined
    309

/Users/smeatonj/Development/django/django/db/models/sql/query.py in combine(self, rhs, connector)
    494             "Cannot combine queries once a slice has been taken."
    495         assert self.distinct == rhs.distinct, \
--> 496             "Cannot combine a unique query with a non-unique query."
    497         assert self.distinct_fields == rhs.distinct_fields, \
    498             "Cannot combine queries with different distinct fields."

AssertionError: Cannot combine a unique query with a non-unique query.

So if you hit the same problem as I did in 1.8, then there's little motivation to try to investigate the cause of your original problem in 1.7 since it's explicitly disallowed in 1.8. I would venture that the reason it's disallowed in 1.8 (assuming it was "valid" in 1.7) is for similar reasons that you're seeing now.

The SQL dump provided above doesn't appear to show any distinct clauses, but your original problem description did. We don't have enough information here to figure out the problem.

If you could at least provide the stacktrace when the assert is hit, that might help narrow it down. But, honestly, if this is important enough to your application then you should spend the time needed to create a reproducible series of steps. We help where we can, but I think you'd be hard pressed to find someone willing to try combination after combination trying to arrive at a similar situation that you're in because it'd take you "too much time".

[Ram Rachum]

Hi Josh,

Thanks for your help.

• Turns out I was wrong about the .distinct thing, it's to be applied after the failing code and not before it, so it's not relevant. That was my mistake. In other words .distinct is definitely not used in the failing query, so we can't expect to get the "Cannot combine a unique query with a non-unique query" error.

• I tried to run my code on Django 1.8, and got the exact same error.

• Here is a traceback of the error: ​http://i.imgur.com/nM748ne.jpg The frame just above the one visible is the frame in which I do & between two querysets. (Didn't include these frames because they have client code.) (This traceback was done on Django 1.8, if you want the same on Django 1.7.7 I can do that too.) If you'll want me to expand any of the "Local vars" sections, let me know.

Also, if you're interested enough and have the time to do a VNC or TeamViewer session to my dev machine, I can show you the whole thing. Of course, I'll totally understand if you're too busy. If so I'll try to do a sample app. (Trying to avoid billing my client for the hours that this will take.)

Thanks,
Ram.

[Josh Smeaton]

It's not likely that just seeing the problem is going to be enough to reproduce on my own. I sympathise with your situation, but you're trying to avoid billing your client by palming the work off for free to open source contributors. We're not tier 3 support. You should (IMO) be billing for bug fixes in situations like this.

I'm happy to meet you half way. If you can put a test case together then we can work on fixing the issue.

Cheers

[kevmitch]

minimal example project exhibiting the error

[kevmitch]

In Django 1.8, with the models

from django.db import models

class Year(models.Model):
    date = models.DateField(auto_now_add=True)

class Student(models.Model):
    year    = models.ForeignKey('Year')
    number  = models.PositiveIntegerField()
    courses = models.ManyToManyField('Course')

class Course(models.Model):
    year       = models.ForeignKey('Year')
    department = models.CharField(max_length=4)
    number     = models.PositiveIntegerField()

the following

course_instance.student_set.all() & year_instance.student_set.exclude(courses__in=[other_course_instance])

will raise

Traceback (most recent call last):
  File "/home/kevmitch/projects/apprentice/and_assertion_error/enrollment/tests.py", line 24, in test_and
    result = math101.student_set.all() & year.student_set.exclude(courses__in=[anth100])
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 211, in __and__
    combined.query.combine(other.query, sql.AND)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 556, in combine
    w.relabel_aliases(change_map)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/where.py", line 285, in relabel_aliases
    child.relabel_aliases(change_map)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/where.py", line 287, in relabel_aliases
    self.children[pos] = child.relabeled_clone(change_map)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/lookups.py", line 185, in relabeled_clone
    new.rhs = new.rhs.relabeled_clone(relabels)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 329, in relabeled_clone
    clone.change_aliases(change_map)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 787, in change_aliases
    assert set(change_map.keys()).intersection(set(change_map.values())) == set()
AssertionError

The tests of attached project demonstrates it in action.

Essential elements for reproduction:

• One of the querysets generated from related manager of the ManyToMany target
• The other queryset is generated by exclude()ing on the ManyToManyField from the related manager of a "versioning" ForeignKey model

[Tim Graham]

Bisected to 01f2cf2aecc932d43b20b55fc19a8fa440457b5f

[Tim Graham]

​Removing the assertion seems to work. Is there a better solution?

[Tim Graham <timograham@…>]

In 2dc9ec56:

Fixed #24525 -- Fixed AssertionError in some complex queries.

Thanks Anssi Kääriäinen for providing the solution.

[Tim Graham <timograham@…>]

In 62347208:

[1.8.x] Fixed #24525 -- Fixed AssertionError in some complex queries.

Thanks Anssi Kääriäinen for providing the solution.

Backport of 2dc9ec5616a942de3a0886a707f93988f56dd594 from master

[Christian Karrié]

Hi all, unfortunately I have to reopen this issue.

After migrating from Django 1.10.6 to 1.11.9 the error reoccurs:

Code:

qs_out = qs.filter(*out_filters)
qs_in = qs.filter(*in_filters)
qs_merged = qs_in | qs_out

Vars:

in_filters = [<Q: (NOT (AND: ('contract__customer_id', 1)))>, <Q: (NOT (AND: ('contract__category__deletion', True)))>, <Q: (NOT (AND: ('contract__date_from__lte', datetime.date(2018, 1, 10)), ('contract__category__blocked', True), ('contract__date_until__gte', datetime.date(2018, 1, 10))))>, <Q: (AND: ('id', 0))>]
out_filters = [<Q: (OR: (AND: (NOT (AND: ('contract__source_customer_id', 1))), ('contract__customer_id', 1)), (AND: (NOT (AND: ('contract__source_customer_id', 1))), (NOT (AND: ('contract__customer_id', 1)))))>, <Q: (NOT (AND: ('contract__category__incoming', True)))>]

It seems to fail, if qs_in or qs_out is empty. But I'm not sure.

Printing change_map gives me:

{'T7': u'invoice_customer', u'invoice_customer': 'T7'}

Removing the assert works.

[Tim Graham]

As the fix for this ticket is released, please open a new ticket with complete details to reproduce. Ideally, you could also ​bisect to find the commit where the behavior changed.