Opened 6 years ago

Last modified 6 years ago

#29577 closed Cleanup/optimization

Generated SQL for exists filter does Exists(...)=true. Having just Exists(...) without "true" is faster by nearly 50% in many of my queries — at Initial Version

Reported by: Michael Chiciak Owned by: nobody
Component: Database layer (models, ORM) Version: 2.0
Severity: Normal Keywords: slow query
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I am trying to optimize my exists filters, since on large record sets I noticed some slowness. I have narrowed down part of it to the "= true" portion of the "Exists(...)" subquery. Removing the "= true" gives a noticeable performance boost

Also, it would be really nice to be able to just filter and not have a field inside the select portion of the query. When I use the Paginator class for pagination, it turns the count query into one big subquery and adds all my Exists filters as select fields and thus performs a Group BY on each of them, drastically slowing down my count query. Only having the Exists in the where clause would be very nice. Is there a way to do this already?

Sample django code

fits_vehicle = ProductFitment.objects.filter_vehicle(2008, 2013, 'Chevrolet', 'Corvette').filter(product=OuterRef('pk')).only("pk")
queryset = queryset.annotate(fits_vehicle=Exists(fits_vehicle)).filter(fits_vehicle=True)

# With exists = true 500ms
SELECT COUNT(*)
FROM "aces_pies_data_product" 
WHERE EXISTS(SELECT 1 FROM "aces_pies_data_productfitment" U0 INNER JOIN "django_vehiclefitment_vehicle" U1 ON (U0."vehicle_id" = U1."id") INNER JOIN "django_vehiclefitment_vehiclemake" U2 ON (U1."make_id" = U2."id") INNER JOIN "django_vehiclefitment_vehiclemodel" U3 ON (U1."model_id" = U3."id") WHERE (U2."name" = 'Chevrolet' AND U3."name" = 'Corvette' AND (2008 BETWEEN U0."start_year" AND U0."end_year" OR 2009 BETWEEN U0."start_year" AND U0."end_year" OR 2010 BETWEEN U0."start_year" AND U0."end_year" OR 2011 BETWEEN U0."start_year" AND U0."end_year" OR 2012 BETWEEN U0."start_year" AND U0."end_year" OR 2013 BETWEEN U0."start_year" AND U0."end_year") AND U0."product_id" = ("aces_pies_data_product"."id"))) = true

#With no "true" 350MS
SELECT COUNT(*)
FROM "aces_pies_data_product" 
WHERE EXISTS(SELECT 1 FROM "aces_pies_data_productfitment" U0 INNER JOIN "django_vehiclefitment_vehicle" U1 ON (U0."vehicle_id" = U1."id") INNER JOIN "django_vehiclefitment_vehiclemake" U2 ON (U1."make_id" = U2."id") INNER JOIN "django_vehiclefitment_vehiclemodel" U3 ON (U1."model_id" = U3."id") WHERE (U2."name" = 'Chevrolet' AND U3."name" = 'Corvette' AND (2008 BETWEEN U0."start_year" AND U0."end_year" OR 2009 BETWEEN U0."start_year" AND U0."end_year" OR 2010 BETWEEN U0."start_year" AND U0."end_year" OR 2011 BETWEEN U0."start_year" AND U0."end_year" OR 2012 BETWEEN U0."start_year" AND U0."end_year" OR 2013 BETWEEN U0."start_year" AND U0."end_year") AND U0."product_id" = ("aces_pies_data_product"."id")))

Change History (0)

Note: See TracTickets for help on using tickets.
Back to Top