Combine Query Store's flushed and in-memory interval slices at collection (#1907) - #1919
Conversation
…tion sys.query_store_runtime_stats returns the flushed slice and the still-in-memory slice of one runtime_stats_interval_id as two separate rows, and they are ADDITIVE members of one interval rather than competing snapshots of it. The collector selected straight from the view, so both were stored -- and they then shared every column of the read-side dedup key (#1841/#1845/#1853) AND collection_time, so the viewer's ROW_NUMBER and the rollups' last(execution_count, collection_time) were both ordering by a value identical for both rows. The survivor was whichever the engine emitted first: 8 executions shown where 94 was true, differently on each run. Reproduces on box SQL Server 2022 (16.0.4255.1), not just Azure: 100 flushed + 25 in memory came back as two rows while dm_exec_procedure_stats reported 125 at the same instant. BuildPayloadBody -- the one body both execution shapes run -- now groups on the natural key of the view (plan_id, runtime_stats_interval_id, execution_type, replica group). execution_count SUMs; every avg_* takes the count-weighted mean, because Query Store stores an average and a count but never a total; min_*/max_* take the extreme; first/last execution time the interval's span. The emitted row shape is unchanged -- same 55 columns, same order, no migration -- only the row count per interval. The cutoff moves from a per-slice WHERE to HAVING MAX(...) at interval grain, which is load-bearing: the flushed slice is static, so a per-slice predicate stops matching it once the growing in-memory slice advances the watermark, and the sum degrades to the sliver alone. The IN (...) pre-filter is a prune only, and the fixed query is faster than the one it replaces (375ms vs 453ms on a real 212k-row Query Store) -- half the rows means half the plan XML to materialize. The deprecated Dashboard's collect.query_store_collector had the same defect against the same view and takes the same fix. Rows already collected cannot be repaired: all 19 read-side dedup sites in both apps gain a documented execution_count tie-break so a pre-fix tie resolves to the flushed slice deterministically instead of flapping. Closest-available, not correct -- residual tracked in #1912. Verified live on SQL Server 2022 (emitted SQL returns 125/70 against raw slices of {100,25}/{60,10}, matching dm_exec_procedure_stats) and on PostgreSQL 18 + TimescaleDB 2.28.1 (corrected rollups report the hand-computed 155 and the exact weighted mean; the same store fed split slices can never reach the interval's true 125). Lite 1854 passed / 0 failed, Darling 3997 passed / 0 failed, full solution rebuild 0 warnings / 0 errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two additive conflicts, both resolved by keeping BOTH sides: - QueryStoreCorrectedRollupLiveTests.cs: #1877's re-hold test and #1907's slice-aggregation test were added at the same point in the file. Both kept, along with #1907's SeedTiedSlicesAsync helper. - CHANGELOG.md: adjacent Fixed entries and link-ref blocks. Verified after resolving: 19 of 19 read-side dedup sites still carry the tie-break, full-solution build 0 warnings / 0 errors, Darling 4020 passed / 0 failed against live PG 18 + TimescaleDB 2.28.1, Lite 1900 passed / 0 failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ision # Conflicts: # CHANGELOG.md
| @@ -1,4 +1,4 @@ | |||
| /* | |||
| /* | |||
There was a problem hiding this comment.
Nit: this edit introduced a UTF-8 BOM (EF BB BF) before the leading /* comment (confirmed with od -c). No other file touched in this PR has one (e.g. the new QueryStoreSliceTieBreakSourceTests.cs starts with a plain /*), so this looks like an accidental artifact of whatever tool made the edit rather than an intentional change. Harmless to the build, but worth stripping for consistency.
ReviewWhat this does: Correctness: Traced through Lite/Darling parity: Verified directly — Testing: Notably thorough — live SQL Server 2022 verification of the actual emitted SQL (not a hand-written approximation) across all three version shapes, a live Postgres/TimescaleDB test that asserts the corrected rollup and the pre-fix split-slice behavior in the same store/refresh so it can't pass by accident, and a test that discovers Minor nit (left inline): Observation, not a blocker: The Dashboard proc fix ( No SQL injection, secrets, or unsafe file/network/process concerns — all the new SQL fragments are built from internal boolean version-gate flags, not user input, consistent with the rest of the collector's existing pattern. |
…egation Both from the PR review on #1919. - QueryStoreCorrectedRollupLiveTests.cs picked up a UTF-8 BOM when the merge resolution was scripted. Removed; the file matches the rest of the codebase again. - The Dashboard proc's half of the fix had no automated regression test, and nothing else could have caught one: the aggregation lives inside a dynamically assembled @SQL string so the compiler sees nothing, the sql-validation workflow only proves the proc COMPILES, and the tests that would execute it are the DB-touching classes CI filters out. A source guard in Installer.Tests (which CI does run, and which needs no database) pins the grouping key, the weighted mean on every avg_ column discovered FROM THE FILE, the interval-grain HAVING, the absence of the per-slice WHERE that would reintroduce the bug, and the version gate on replica_group_id. Watched red: changing one weighted mean to AVG() fails EveryAveragedColumnUsesTheCountWeightedMean. Installer.Tests 194 passed / 0 failed under CI's own non-DB filter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review: #1919 — combine Query Store's flushed/in-memory slices at collectionWent through the diff in detail (collector SQL, all 12 read-side dedup sites in both apps, the deprecated Dashboard proc, and the new test coverage). This is an unusually well-documented and well-tested fix — the reasoning in the source comments matches the code exactly everywhere I checked it. What I verified:
One pre-existing (not introduced by this PR) inconsistency worth a note: the shared collector's cutoff comparisons use Nothing else stood out as a correctness, security, or Lite/Darling parity issue. (Per review scope, not flagging missing-index recommendations.) |
…ision # Conflicts: # CHANGELOG.md
…ision # Conflicts: # CHANGELOG.md
| -- the store, which cannot be rewritten: it deterministically picks the FLUSHED slice, the one holding the | ||
| -- bulk of the interval's work, instead of flapping. Closest-available, not correct — the correct value is | ||
| -- the SUM of the slices, which no read-side rule can express (#1912). This applies to EVERY dedup site in | ||
| -- both apps; a source-containment test pins all of them, so deleting one here fails loudly. |
There was a problem hiding this comment.
This comment says "a source-containment test pins all of them," but that's only true on the Darling side. Lite.Tests/QueryStoreDedupReadTests.cs's EveryQueryStoreAggregateInTheFile_CarriesADedupCte guard only scans LocalDataService.QueryStore.cs (5 sites). It does not cover the tie-break added in this same PR to Lite/Analysis/DrillDownCollector.Queries.cs or Lite/Analysis/DuckDbFactCollector.QueryPerf.cs — I confirmed neither file is referenced anywhere under Lite.Tests/.
Darling's counterpart, Darling.Tests/QueryStoreSliceTieBreakSourceTests.cs, explicitly enumerates all 8 files / 12 dedup sites across the app and fails loudly if one is dropped or moved. Lite has no equivalent cross-file guard, so a future edit that silently drops the execution_count DESC tie-break from either of those two files (exactly the failure mode #1841/#1845/#1853/#1907 keep recurring on) would not be caught by any test today, unlike on the Darling side.
Worth either widening the Lite guard to a DedupSites-style list covering all 7 sites (mirroring the Darling test), or softening this comment so it doesn't claim coverage that doesn't exist.
Review summaryReviewed the full diff (collector SQL, all 12/7 read-side dedup sites in Darling/Lite, the deprecated Dashboard proc, and the new/updated tests). What the PR does: Correctness: I traced the SQL carefully —
Lite/Darling parity: Left one inline comment — Darling has a comprehensive cross-file source-containment guard ( Minor/non-blocking observation (not flagged inline, no action needed): the deprecated Dashboard script already used Security/perf: No injection surface introduced (identifiers are Overall this is a well-reasoned, thoroughly tested fix for a real correctness bug. The one actionable item is the Lite test-coverage gap noted inline. |
From the review bot's inline comment on #1919, taking its first option rather than the comment-softening one: softening would have been a Lite/Darling parity scope-down. QueryStoreDedupReadTests.EveryQueryStoreAggregateInTheFile_CarriesADedupCte reads exactly one file, LocalDataService.QueryStore.cs, which is right for what it checks but left 2 of Lite's 7 dedup sites -- the ones in Analysis/DrillDownCollector.Queries.cs and Analysis/DuckDbFactCollector.QueryPerf.cs, both of which this same PR added tie-breaks to -- covered by no test at all. Dropping execution_count DESC from either would have gone uncaught, which is the recurrence mode #1841/#1845/#1853/#1907 keep demonstrating, and Darling was already guarded against it. Lite.Tests/QueryStoreSliceTieBreakSourceTests mirrors Darling's counterpart exactly: an enumerated DedupSites list (3 files, 7 sites) rather than a glob, so MOVING a read has to be re-declared instead of silently shrinking coverage, and a missing tie-break fails naming the file. It also carries a parity check that reads Darling's guard and asserts its declared total, so neither app can quietly fall behind the other. The source comments in both apps claimed "a source-containment test pins all of them" -- true only on the Darling side. Both now name the actual tests and their real counts. Watched red: dropping the tie-break from DuckDbFactCollector.QueryPerf.cs leaves the OLD guard green (proving the gap was real) while the new guard fails with "Lite\Analysis\DuckDbFactCollector.QueryPerf.cs @ char 9829". Lite 1903 passed / 0 failed, build 0 warnings / 0 errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
GUARD WIDENED — pushed
The overclaiming comment is fixed in both apps — each now names the actual test and its real count instead of "a source-containment test pins all of them". Watched red, and it demonstrates the gap was real: dropping the tie-break from Lite 1903 passed / 0 failed; full-solution build 0 warnings / 0 errors. |
|
Reviewed. This is an unusually well-documented and well-tested fix — the live repros on both SQL Server 2022 and Azure SQL DB, the weighted-mean test that discovers One gap, not touched by this PR: the deprecated Dashboard's own analysis reads against
Both dedup with Given No other correctness, security, or Lite/Darling parity issues found in the diff itself. Not flagging missing-index recommendations per your standing note. |
…ision # Conflicts: # CHANGELOG.md
Review: Combine Query Store's flushed and in-memory interval slices at collection (#1907)OverviewThis is a genuine correctness fix. What's solid
Issue: the deprecated Dashboard's write-side fix isn't matched by its own read sideThe PR explicitly applies "the same fix" to the deprecated Dashboard's collector (
These are the Dashboard's counterpart to the exact class of dedup site fixed everywhere else (note they even use Left inline comments on both spots. If the deprecated Dashboard is still receiving bug fixes for this defect class (the install-script change shows it is), it seems worth completing — either add the same tie-break there, or note explicitly why the read side is being left out of scope. Minor / non-blocking
No SQL injection, secrets, or file/network/process concerns — this is pure T-SQL query construction over internal collector context, consistent with the existing |
|
(Follow-up: GitHub won't let me attach inline comments to |
Fixes #1907.
The bug
sys.query_store_runtime_statsreturns the flushed slice and the still-in-memory slice of oneruntime_stats_interval_idas two separate rows, and they are ADDITIVE members of one interval, notcompeting snapshots of it. The collector selected straight from the view, so both were stored — and they
then shared every column of the read-side dedup key (#1841/#1845/#1853) and
collection_time. So theviewer's
ROW_NUMBER() ... ORDER BY collection_time DESCand the rollups'last(execution_count, collection_time)were both ordering by a value identical for both rows, and the survivor was whicheverthe engine happened to emit first. Live: 8 executions reported where 94 was true, with a different
subset wrong on each run.
The dedup itself is correct and stays — it exists to collapse re-collections of one interval across
cycles. It just cannot also be asked to add two slices within one cycle, and no read-side rule can
express both. A TimescaleDB continuous aggregate cannot express it at all.
It is not an Azure peculiarity. #1907's evidence was Azure SQL Database only and flagged box SQL
Server as untested. It reproduces there — SQL Server 2022 (16.0.4255.1),
DATA_FLUSH_INTERVAL_SECONDS = 60, 100 executions then a forced flush then 25 more:sys.dm_exec_procedure_stats, a wholly separate source read at the same instant, said 125 and 70.SUM matches; the larger slice alone does not. Both slices carry an identical
first_execution_time, sothe tier-1 proxy key had exactly the same exposure as the real interval id — which is why pre-tier-2 rows
are affected too.
With Query Store's default 900s flush against its default 3600s interval, one interval can hold several
flushed slices, so the split is not bounded at two.
The fix
BuildPayloadBody— the ONE body both the on-prem and Azure execution shapes run (#1844's single-bodyguarantee, drift guard still green) — groups on exactly the natural key of the view and combines:
count_executionsSUMavg_*(11 of them)SUM(avg * count) / NULLIF(SUM(count), 0)min_*/max_*MIN/MAX(includesmin_dop/max_dop, which have no avg)first_execution_time/last_execution_timeMIN/MAX— the interval's own spanThe weighted mean is the whole game: Query Store stores an average and a count but never a total, so
avg * countis what recovers a slice's total. A plainAVG()of the slice averages would weight a25-execution sliver the same as a 100-execution flush. Verified live — slices of (1778.42 over 100) and
(2245.60 over 25) combine to 1871.856, which is
(1778.42*100 + 2245.60*25) / 125, and is not the2012.01 an unweighted average gives.
Every
avg_*column goes through oneWeightedAveragehelper rather than being written out by hand,because the wrong form is not a compile error and does not look wrong. A test discovers the
avg_columnsfrom the emitted SQL and requires the weighted shape on each, so a newly added one is covered the
moment it appears.
The cutoff had to move from WHERE to HAVING
This is load-bearing, not tidiness. The flushed slice is static, so once the growing in-memory slice
pushes the watermark past its
last_execution_timethe flushed slice stops qualifying — and a sum overthe survivors is the sliver alone, the original defect with an aggregate bolted on top.
HAVING MAX(last_execution_time) > @cutoff_timeasks the question at interval grain: did this interval see newactivity, and if so give me all of it. It is strictly more permissive than the predicate it replaces, so
nothing that used to be collected stops being.
It is faster than what it replaces
Measured on a real 212,000-row Query Store (SQL 2025), full 55-column payload, warm, three runs:
Half the rows means half the
nvarchar(max)query text and plan XML to materialize and ship, which morethan pays for the aggregate. The
IN (...)pre-filter is a prune, not a semantic — its interval listis by construction a superset of what the
HAVINGkeeps, so it can never subtract a row — and the thirdrow is why it is there.
Nothing about the stored row moved
Same 55 columns in the same order, same positional writers,
GoldenCollectorSchemauntouched, nomigration and no storage-version bump. Only the number of rows per interval (now at most one per interval
per collection). The
TOPbackstop now caps intervals rather than slices, and sits outside theaggregate for cause: a cap falling mid-interval would emit a partial sum, which is worse than omitting the
interval.
replica_group_idjoins the grouping key under the same 2022+/Azure gate the attribution column carries,for the same bind-safety reason — naming a column that does not exist in a
GROUP BYfails the wholeSELECT, and on Azure's per-database path that means every database. It has to be in the key where it does
exist: two replicas' rows for one interval are different work, and grouping without it would sum a
secondary's executions into the primary's — the exact bug replica attribution was added to prevent.
Deprecated Dashboard: same defect, same fix
install/09_collect_query_store.sqlreads the same view the same way and had the same bug. Sameaggregation, same
WHEREtoHAVINGmove, same gatedreplica_group_id. It is a proc body change, so anupgrade re-applies it with no schema step and no
upgrades/entry. Verified live against SQL 2022 byexecuting the proc's own generated SQL in all three version shapes — it returns 125 and 70 where the raw
view holds the split slices.
Legacy rows: made deterministic, and the residual is filed
Rows already collected cannot be rewritten. All 19 read-side dedup sites across both apps now order by
collection_time DESC, execution_count DESC, so a pre-fix tie resolves to the flushed slice — the oneholding the bulk of the interval's work — deterministically instead of flapping. On new rows the clause
can never fire, because there is one row per partition per collection.
That is closest-available, not correct. The correct value is the sum, and the materialized CAGGs
cannot be tie-broken at all (
last()has no tie-break, and the tied rows are gone once materialized).Filed as #1912, including the part that does not age out: the indefinitely-kept daily tier keeps
understated counts for the pre-fix period.
Testing
Watched red per mutation, each reverted after:
AVG(qsrs.avg_x)Payload_EveryAverageColumn_IsTheCountWeightedMeanreplica_group_idungated in the GROUP BYBuildPerItemQuery_ReplicaGroupIdEntersTheGroupingKey_OnlyWhereItBindsHAVINGback to a per-sliceWHERETiedSlicesOfOneInterval_...(returned the 25 sliver) + the source guardThe behavioural DuckDB test seeds both insertion orders as two independent queries, so it cannot pass
by accident on an engine that happens to favour the earlier row — without the tie-break one arm returns
the sliver.
Live SQL Server 2022 — the emitted collector SQL (not a hand-composed approximation) executed in all
three version shapes; all 55 columns bind on the 2016 floor, on 2017, and on 2022. Against the repro
database it returns 125 and 70 where the raw view holds
{100, 25}and{60, 10}, matchingdm_exec_procedure_statsexactly.Live PostgreSQL 18 + TimescaleDB 2.28.1 — a new
QueryStoreCorrectedRollupLiveTestscase asserts bothhalves in one store on one refresh: the corrected rollups report the hand-computed 155 for an hour of
two combined intervals plus the exact execution-weighted mean off the same rows, while the same store fed
the pre-fix split slices returns a single slice and can never reach the 125 they add up to. Gating
verified honestly — it SKIPS without
DARLING_TEST_PGand PASSES with it.Suites: Lite 1854 passed / 0 failed; Darling 3997 passed / 0 failed (10 skipped = gated tests
needing a live SQL Server or the bundled runtime). Full-solution
-t:Rebuild: 0 warnings, 0 errors.Installer.Tests deliberately not run.
Deferred, filed before this PR opened
with the reason none of them repairs the indefinitely-kept daily tier.
🤖 Generated with Claude Code