Model.relatedmanager.count() always 0 for custom ManyToMany relationships with through model with to_fields

[mfontana-elem]

While upgrading my codebase to Django 5.1 I started noticing some tests were failing. At the end of the day, the reason was that, given an object's manager for a ManyToMany relation I was getting object.relatedmanager.count() != len(object.relatedmanager.all()). The LHS was always zero.

To illustrate my setup, I have the following models:

from django.db import models


class Course(models.Model):
    uuid = models.UUIDField(unique=True, editable=False, verbose_name="UUID")
    title = models.CharField(max_length=200)

class Student(models.Model):
    uuid = models.UUIDField(unique=True, editable=False, verbose_name="UUID")
    name = models.CharField(max_length=100)
    courses = models.ManyToManyField(Course, through="CourseStudent", related_name='students')

class CourseStudent(models.Model):
    course = models.ForeignKey(Course, to_field="uuid", on_delete=models.RESTRICT, null=False)
    student = models.ForeignKey(Student, to_field="uuid", on_delete=models.RESTRICT, null=False)

And the problem is that it course.students.count() generates the following SQL (both for SQLite and Postgres):
SELECT COUNT(*) AS "__count" FROM "courses_coursestudent" WHERE "courses_coursestudent"."course_id" = '00000000000000000000000000000001',
which is of course wrong.

Under the exact same setup, Django 5.0 generates:
SELECT COUNT(*) AS "__count" FROM "courses_student" INNER JOIN "courses_coursestudent" ON ("courses_student"."uuid" = "courses_coursestudent"."student_id") WHERE "courses_coursestudent"."course_id" = '0817704cdf2d478dba5d629b8997f73b'.

I attach a .zip containing a Dockerfile and a src directory to easily reproduce this behavior.

Best regards,

[Sarah Boyce]

Thank you! Replicated and bisected to 66e47ac69a7e71cf32eee312d05668d8f1ba24bb (#29725)

Possible regression test/asserts

tests/m2m_through/tests.py

@@ -521 +521 @@
 
     def test_retrieval(self):
         # Forward retrieval
+        self.assertEqual(self.curry.ingredients.exists(), True)
+        self.assertEqual(self.curry.ingredients.count(), 3)
         self.assertSequenceEqual(
             self.curry.ingredients.all(), [self.pea, self.potato, self.tomato]
         )

[Simon Charette]

I can work on that one.

We can't be systematically using self.instance.pk ​here

[Sarah Boyce <42296566+sarahboyce@…>]

In c3a23aa:

Fixed #36197 -- Fixed improper many-to-many count() and exists() for non-pk to_field.

Regression in 66e47ac69a7e71cf32eee312d05668d8f1ba24bb.

Thanks mfontana-elem for the report and Sarah for the tests.

[Sarah Boyce <42296566+sarahboyce@…>]

In 6b8a6e1:

[5.2.x] Fixed #36197 -- Fixed improper many-to-many count() and exists() for non-pk to_field.

Regression in 66e47ac69a7e71cf32eee312d05668d8f1ba24bb.

Thanks mfontana-elem for the report and Sarah for the tests.

Backport of c3a23aa02faa1cf1d32e43d66858e793cd9ecac4 from main.

[Sarah Boyce <42296566+sarahboyce@…>]

In 8488074f:

[5.1.x] Fixed #36197 -- Fixed improper many-to-many count() and exists() for non-pk to_field.

Regression in 66e47ac69a7e71cf32eee312d05668d8f1ba24bb.

Thanks mfontana-elem for the report and Sarah for the tests.

Backport of c3a23aa02faa1cf1d32e43d66858e793cd9ecac4 from main.