Repository navigation
Query optimization - #670
Merged
Merged
Conversation
fstachura
force-pushed
the
query-optimization
branch
3 times, most recently
from
August 13, 2026 12:39
1a7374b to
55bdbad
Compare
stephenfin
force-pushed
the
query-optimization
branch
from
October 15, 2026 20:10
55bdbad to
c6cb3e8
Compare
This property is not used anymore, the last reference was removed in 17726d7 (views: Move and refactor patch-forms). Computing it may execute an unnecessary count query.
Patch list is pretty slow partially because it filters on a query with a lot of extra fields. For example, I suspect that the tag count fields are computed even for rows that did not make it into the final result, same with joins. This patch changes this by separating the filtering stage, in which only patch IDs are queried, and the aggregation stage.
stephenfin
force-pushed
the
query-optimization
branch
from
October 15, 2026 21:15
c6cb3e8 to
67dbad4
Compare
The new index uses project_id as the first key. This is important, as it seems that almost all web queries filter over project_id. The index also includes the submission date sorted descending, as this is the default sorting key for the patches list. Co-authored-by: Stephen Finucane <stephen@that.guru>
Allowing the users to set an arbitrary number of items per page may cause excessive load on the database. For example, a request to the patch list has to compute the number of tags for each listed patch.
stephenfin
force-pushed
the
query-optimization
branch
from
October 15, 2026 21:29
67dbad4 to
d2e197c
Compare
stephenfin
enabled auto-merge
October 15, 2026 21:34
Member
|
I added a missing migration and placed |
Contributor
Author
|
Sorry about the migration. I ran my tests again and the results are inconclusive, Postgres did slightly better and Mysql slightly worse. Differences were less than 200ms and the results are still much better than baseline, so this is not meant as an argument for one or the other option, just LGTM and FYI. I would leave it as it is merged now. Thanks! |
Member
|
Thanks for the additional validation 🙏 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I noticed that queries made by Patchwork can be pretty slow on larger databases.
This PR changes two larger things that seemed to cause performance issues.
I split the query in generic_list into a filtering part and an aggregation part. In the first part, only patch ids are queried. Then, another query is made to retrieve more detailed patch data. I'm not 100% sure, but it seems that MySQL and Postgres may compute all the fields (including JOINs) even for rows that don't make it into the final result due to LIMIT/OFFSET which negatively impacts performance.
I added a new index. The first three fields of the index are fields that the default patch view uses to filter patches - project_id, archived and state. The fourth field is the field used for the default order - date, sorted descending (unlike the current covering index). I'm 100% sure that project_id should be the first field in the index, and that date should be after all the fields that are most commonly used in filters.
In my tests, I have noticed substantial performance improvements on large databases (>1 million patches), on a low-end machine (~2018-laptop tier, 16GB RAM, 2 cores, 2 threads, SSD) on both Postgres and MySQL. I can share more details about my tests, but to keep it short: on main, some queries would take 5+ seconds (average on MySQL was 2 seconds, on Postgres some would cause a timeout), with these patches the average is <300 ms. (I tested with default date descending ordering, on 5 scenarios: default filters, filtering by submitter, delegate, submitter and delegate, state. I did not do stress-tests, just "one-shot"). Query plans also became shorter, in some cases were replaced by straightforward index lookup.
https://dev.mysql.com/doc/refman/8.4/en/order-by-optimization.html