Django admin count is run on different query than listed rows

[craig.kerstiens@…]

Within django admin, when count is run it includes the filters but omits the joins. In some cases this can cause the count to return an incorrect number from what is displayed in the list. The count should be run across the same query that returns the results to the admin list.

[Luke Plant]

This report is rather vauge. There are filters and joins that can be added in various different ways - and in fact filters often require joins to be added. There are also several 'counts' that appear on an admin change list, depending on paging and filters!

Please give a fuller report, preferably with an example, so we can track down what you mean.

It is possible, given that the admin uses django.core.paginator.Paginator, that it is Paginator at fault, or the ORMs QuerySet.count() method, but it is very difficult to tell with this report.

[Craig Kerstiens <craig.kerstiens@…>]

The simplest way to reproduce this is to include a foreign key within your list display. When you have the foreign key there, select related is called, if you have another foreign key (not displayed in the list), that is not set to Null=True, it runs an inner join, something similar to:

SELECT table1.col1, table2.col2
FROM table1
INNER JOIN table2 on table1.col1 = table2.col1
INNER JOIN table3 on table1.col1 = table3.col1

The resulting query produces less results than a query similar to:

SELECT COUNT(*) FROM table_foo WHERE column_name = "bar"

One would expect for the count to be run on top of the initial query that produces the results (with all filters or joins on it).

[Craig Kerstiens <craig.kerstiens@…>]

Upon further investigation the count on the query is correct, it is specifically with how paginator runs count that it returns incorrectly.

[Luke Plant]

It is difficult to work out how Paginator could be at fault, since to get the count it simply does QuerySet.count(). So that must be the method at fault, which takes us to django/db/models/sql/query/Query.get_count(), which is not removing any filters or joins as far as I can see.

And we still don't have enough information to reproduce this - an example set of models, and some example code using Paginator or QuerySet.count() is what we need. If it is the latter at fault, we simply need an example where len(queryset) != queryset.count().

Please re-open if you can provide that. Thanks!

[Craig Kerstiens <craig.kerstiens@…>]

Example repro app

[Craig Kerstiens <craig.kerstiens@…>]

I've attached an example app that reproduces the case. Included is a db with data that does reproduce it. If you navigate within the admin to models1, you'll see the count lists 101 items, where the list shows none, due to the inner join canceling out items.

Username/pass for admin is: demo/demo,

You should be able to boot strap the entire example app with:

virtualenv --no-site-packages .
source bin/activate
bin/pip install -r requirements.txt

[Luke Plant]

Many thanks for taking the time to do this, I can reproduce it now.

We've got at least 2 potential bugs:

1) The fact that appending select_related() to Model1.objects.all() causes the query to return no objects. Perhaps we need to special foreign keys to self somehow, not sure.

2) The fact that Model1.objects.all().select_related().count() != len(Model1.objects.all().select_related())

[Luke Plant]

OK, cancel that. I've found your table contains invalid data - the 'self_ref_id' column of the Model1 is full of NULLs, but the DB model definition doesn't allow that. If I put null=True i.e. self_ref = models.ForeignKey('self', null=True), then everything is fine again.

Please re-open if this doesn't fix your issue.

[Julien Phalip]

This might be a similar "issue" as in #11460.