Extract scheme-selection policy into a pluggable CostModel#8698
Draft
connortsui20 wants to merge 5 commits into
Draft
Extract scheme-selection policy into a pluggable CostModel#8698connortsui20 wants to merge 5 commits into
connortsui20 wants to merge 5 commits into
Conversation
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | encode_varbin[(1000, 4)] |
142.3 µs | 159.1 µs | -10.55% |
| ❌ | Simulation | encode_varbin[(1000, 8)] |
143.8 µs | 160.7 µs | -10.55% |
| ❌ | Simulation | encode_varbin[(1000, 32)] |
147.6 µs | 164.7 µs | -10.33% |
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[128] |
273.6 ns | 244.4 ns | +11.93% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing cost-model-r2-costmodel (9e00bc1) with develop (357a4ba)
Footnotes
-
42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
21 tasks
connortsui20
force-pushed
the
cost-model-r2-costmodel
branch
from
e39c549 to
e17a967
Compare
connortsui20
force-pushed
the
cost-model-r0-golden
branch
3 times, most recently
from
00ccec3 to
1c1ad1b
Compare
connortsui20
force-pushed
the
cost-model-r2-costmodel
branch
from
e17a967 to
83988a0
Compare
connortsui20
force-pushed
the
cost-model-r0-golden
branch
from
1c1ad1b to
29b6106
Compare
connortsui20
changed the title
connortsui20
force-pushed
the
cost-model-r2-costmodel
branch
from
7672d77 to
cf51915
Compare
connortsui20
added
changelog/break
connortsui20
force-pushed
the
cost-model-r2-costmodel
branch
from
0228bc7 to
56e29f9
Compare
Pure refactor of choose_best_scheme and the sampling estimator: - estimate_compression_ratio_with_sampling now returns a SampledEstimate carrying the compressed sample array alongside its score, instead of measuring nbytes and dropping the array. - Selection bookkeeping switches from (scheme, EstimateScore) tuples to a crate-private Candidate carrying the mechanical facts about each option (scheme, score, input bytes, value count, sampled array, cascade ancestry). Candidates are still compared by ratio, equal ratios retain the existing evaluation-order winner, and the sampled array is dropped at the end of selection, so behavior is unchanged. The not-yet-read Candidate fields are the inputs a pluggable cost model will use in the follow-up commits without requiring a new Scheme API. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Introduce the first cost-model slice tracked by #8701: - Cost is an opaque finite f64 newtype where lower is better. Units belong to each model because costs are compared only within one model. - CostModel computes a Candidate's cost (None rejects it) and the cost of the canonical baseline that every selected candidate must strictly beat. - SizeCost assigns Cost(-ratio) to candidates and Cost(-1.0) to canonical, reproducing the historical ratio-argmax selection. SizeCost uses the exact ratio signal rather than recomputing input/ratio. That avoids IEEE division collapsing distinct ratios to the same byte estimate and changing the winner through evaluation-order tie-breaking. Candidate becomes public because it is the trait's argument and is re-exported through the cost module. Unit tests pin the validity gate, pairwise ordering, and tie behavior against the historical rules. The selection loop is wired to the new model in the next commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
choose_best_scheme now computes every scored candidate's cost through the compressor's CostModel. The winner is the minimum-cost candidate whose cost is strictly below CostModel::canonical_cost. Strict comparisons preserve the existing evaluation-order behavior for equal costs. The legacy comparison helpers whose policy moved into SizeCost and the selection loop are removed. CascadingCompressor gains with_cost_model(Arc<dyn CostModel>) and uses SizeCost by default, so the default output remains unchanged. AlwaysUse short-circuiting and the final byte-acceptance gate remain outside the model boundary. The winner_compress trace span gains estimated_cost alongside estimated_ratio, and WinnerEstimate retains the winning cost for tracing. The golden suite has no snapshot changes under the default SizeCost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
cost-model-r2-costmodel
branch
from
56e29f9 to
318345f
Compare
connortsui20
commented
Jul 10, 2026
Comment on lines
+25
to
+26
| //! is kept only if it is byte-wise smaller than its input. This is an axiom for **all** | ||
| //! models — compression never grows bytes. A model that prefers canonical (e.g. for speed) |
Member
Author
There was a problem hiding this comment.
I can't think of a case where we would have an encoding that is larger than the canonical encoding (even to speed up computation). The closest I can think of is byte bool as an encoding, but really that is like a new type.
connortsui20
added
changelog/feature
connortsui20
force-pushed
the
cost-model-r2-costmodel
branch
from
318345f to
adb05ec
Compare
connortsui20
added
changelog/break
connortsui20
force-pushed
the
cost-model-r2-costmodel
branch
2 times, most recently
from
01a6f06 to
d7bb7fe
Compare
Replace the ratio-named scheme contract with an explicit candidate pipeline: - Scheme::evaluate returns SchemeEvaluation. - Immediate and deferred evaluations produce an opaque CandidateEstimate. - The compressor adds the scheme, input, sample, and cascade context to construct a borrowed Candidate for CostModel::cost. - Selection retains and compares only the model-defined Cost. Remove the current-winner ratio feedback path, CompressionRatioThreshold, and estimated-ratio winner tracing. Deferred callbacks no longer observe selection state, so Delta and Sequence always finish their evaluation before the active model decides whether their candidates are competitive. Any future pruning must be expressed as a cost-model capability rather than a SizeCost special case in scheme evaluation. SizeCost continues to consume the candidate's transient compression-ratio evidence and preserves the historical winners, canonical acceptance rule, and golden outputs. Tests cover custom ordering, canonical rejection, immediate and sampled candidate evidence, and a lower-ratio deferred candidate selected by a custom model. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Place the model-facing Candidate, Cost, CostModel, and SizeCost under the cost module. Split Cost and CostModel into cost/model.rs, keep the size submodule private, and re-export every public cost-model type through cost::. Rename the public estimate module to evaluation because it now defines scheme evaluation states, deferred resolution, and candidate evidence rather than selector scoring. Update every scheme implementation and internal link to use the new path. Each concept now has one public path, and the source layout reflects the candidate-to-cost-model boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
cost-model-r2-costmodel
branch
from
d7bb7fe to
9e00bc1
Compare
connortsui20
added
the
action/benchmark
github-actions
Bot
removed
the
action/benchmark
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals ProfilingVortex (geomean): 1.016x ➖ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.016x ➖, 1↑ 2↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.990x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.978x ➖, 0↑ 0↓)
datafusion / parquet (0.973x ➖, 0↑ 0↓)
datafusion / arrow (0.996x ➖, 2↑ 0↓)
duckdb / vortex-file-compressed (0.985x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.988x ➖, 0↑ 0↓)
duckdb / parquet (0.992x ➖, 1↑ 1↓)
duckdb / duckdb (0.997x ➖, 0↑ 0↓)
File Size Changes (9 files changed, +0.3% overall, 5↑ 4↓)
Totals:
|
Contributor
Benchmarks: FineWeb NVMeVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.903x ➖, 4↑ 0↓)
datafusion / vortex-compact (0.911x ➖, 3↑ 0↓)
datafusion / parquet (0.916x ➖, 4↑ 0↓)
duckdb / vortex-file-compressed (0.923x ➖, 3↑ 0↓)
duckdb / vortex-compact (0.942x ➖, 0↑ 0↓)
duckdb / parquet (0.919x ➖, 2↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: TPC-DS SF=1 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.014x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.004x ➖, 0↑ 0↓)
datafusion / parquet (1.000x ➖, 2↑ 0↓)
duckdb / vortex-file-compressed (0.990x ➖, 1↑ 0↓)
duckdb / vortex-compact (0.981x ➖, 7↑ 2↓)
duckdb / parquet (0.999x ➖, 0↑ 1↓)
duckdb / duckdb (0.949x ➖, 8↑ 1↓)
File Size Changes (6 files changed, -0.1% overall, 1↑ 5↓)
Totals:
|
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
Benchmarks: Statistical and Population GeneticsVerdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (1.001x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.006x ➖, 0↑ 0↓)
duckdb / parquet (0.994x ➖, 0↑ 0↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
Contributor
Benchmarks: FineWeb S3Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.946x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.930x ➖, 0↑ 0↓)
datafusion / parquet (1.089x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.026x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.012x ➖, 0↑ 0↓)
duckdb / parquet (1.095x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Clickbench Sorted on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.961x ➖, 2↑ 1↓)
datafusion / parquet (1.011x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.963x ➖, 1↑ 0↓)
duckdb / parquet (0.971x ➖, 1↑ 0↓)
duckdb / duckdb (1.001x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -0.0% overall, 98↑ 103↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.037x ➖, 2↑ 5↓)
datafusion / vortex-compact (1.028x ➖, 1↑ 4↓)
datafusion / parquet (0.983x ➖, 3↑ 2↓)
datafusion / arrow (1.043x ➖, 2↑ 5↓)
duckdb / vortex-file-compressed (0.896x ✅, 12↑ 3↓)
duckdb / vortex-compact (1.028x ➖, 0↑ 4↓)
duckdb / parquet (1.015x ➖, 0↑ 4↓)
duckdb / duckdb (0.997x ➖, 0↑ 0↓)
File Size Changes (27 files changed, +0.0% overall, 13↑ 14↓)
Totals:
|
Contributor
Benchmarks: Clickbench on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.009x ➖, 0↑ 3↓)
datafusion / parquet (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.011x ➖, 1↑ 2↓)
duckdb / parquet (1.002x ➖, 1↑ 0↓)
duckdb / duckdb (1.013x ➖, 0↑ 0↓)
File Size Changes (105 files changed, +0.0% overall, 48↑ 57↓)
Totals:
|
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
Benchmarks: Random AccessVortex (geomean): 0.908x ➖ How to read Verdict and Engines
unknown / unknown (0.939x ➖, 9↑ 0↓)
|
Contributor
Benchmarks: Appian on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.090x ➖, 0↑ 4↓)
datafusion / parquet (1.091x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed (1.073x ➖, 0↑ 2↓)
duckdb / parquet (1.064x ➖, 0↑ 0↓)
duckdb / duckdb (1.063x ➖, 0↑ 0↓)
File Size Changes (4 files changed, -0.0% overall, 2↑ 2↓)
Totals:
|
Contributor
Benchmarks: CompressionVortex (geomean): 0.994x ➖ How to read Verdict and Engines
unknown / unknown (1.011x ➖, 1↑ 4↓)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking Issue: #8701
Rationale for this change
The compressor's selection objective — maximize estimated compression ratio — is hardcoded into the selection loop, so it cannot be varied per workload (#7697). This PR separates candidate evaluation from selection policy: schemes produce model-independent candidate evidence, the configured
CostModelconverts each completeCandidateintoCost, and the compressor selects the lowest-cost candidate.The default
SizeCostpreserves today's ratio ordering and canonical-acceptance threshold. It computes cost directly from the exact ratio evidence (Cost(-ratio), canonical baselineCost(-1.0)) rather than recomputing bytes, so IEEE rounding cannot turn a strict ratio ordering into a cost tie.What changes are included in this PR?
Details
Scheme::evaluatereplaces the ratio-namedexpected_compression_ratiocontract. It returnsSchemeEvaluation: skip, forceAlwaysUse, produce an opaqueCandidateEstimate, or request aDeferredEvaluation.CandidateEstimatewith compressor-owned context (scheme, input, optional compressed sample, and cascade ancestry), and constructs a borrowedCandidateforCostModel::cost.vortex_compressor::costcontains a finite, totally orderedCost, theCostModeltrait, the defaultSizeCost, and the model-facingCandidate.Costis the only candidate-ranking value retained or compared by the selector; compression ratio is transient candidate evidence consumed bySizeCost, not selector state.AlwaysUse, the final byte-acceptance gate, and extension-versus-storage selection — are documented explicitly.winner_compresstrace span gains model-definedestimated_cost; the old estimated-ratio winner field is removed. The achieved ratio remains output observability after compression, not selection policy.canonical_cost, and sampled arrays/evidence are exposed correctly. The golden encoding-tree and size snapshots remain unchanged.Removing the old ratio-specific deferred early exits may increase compression work for Delta and Sequence, but it does not change default winners or serialized output.
This PR does not add model-driven pruning, scheme priors, Delta-specific policy relocation, builder/session plumbing, or a production time-based model.
What APIs are changed? Are there any user-facing changes?
This adds
cost::{Cost, CostModel, SizeCost, Candidate},evaluation::CandidateEstimate, andCascadingCompressor::with_cost_model.This deliberately changes the scheme-facing API:
Scheme::expected_compression_ratiobecomesScheme::evaluate;CompressionEstimate,EstimateVerdict,DeferredEstimate, andEstimateFnare replaced bySchemeEvaluation,ResolvedEvaluation,DeferredEvaluation, andDeferredEvaluationFn. Deferred callbacks no longer receive a best-so-far argument. Downstream scheme implementations must migrate to the candidate-evaluation types.The default
SizeCostpreserves existing selection ordering, canonical acceptance, encoding trees, and array sizes.