Skip to content

perf: Optimize N+1 query in ProcessFlakesTask#1163

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/perf/process-flakes-n1-query
Open

perf: Optimize N+1 query in ProcessFlakesTask#1163
sentry[bot] wants to merge 1 commit into
mainfrom
seer/perf/process-flakes-n1-query

Conversation

@sentry

@sentry sentry Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

The root cause analysis identified an N+1 query pattern within the app.tasks.flakes.ProcessFlakesTask in apps/worker/services/test_analytics/ta_process_flakes.py.

Specifically:

  1. The process_flakes_for_commit function fetched uploads for a commit.
  2. It then iterated over these uploads, and for each upload, process_single_upload would call get_testruns(upload), resulting in a separate database query for testruns per upload. This led to N+1 queries (1 for uploads + N for testruns).
  3. Additionally, handle_pass was calling .save() individually for each expiring flake, leading to further N database writes.

This PR addresses these N+1 issues by:

  • Introducing get_all_testruns to fetch all relevant testruns for all uploads in a single batched query, grouping them by upload_id.
  • Modifying process_flakes_for_commit to use this new batched fetch, passing the pre-fetched testruns to process_single_upload.
  • Modifying handle_pass to append expiring flakes to a list, which are then bulk-updated in process_flakes_for_commit after the loop completes.

These changes reduce the number of database queries from an N+1 pattern to a fixed number of queries, significantly improving performance for tasks processing multiple uploads and testruns.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. In 2022 this entity acquired Codecov and as result Sentry is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

Fixes WORKER-YS4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants