Skip to content

Add verified 1.0.0 benchmark history and advisory same-run comparisons - #51

Merged
dombrovsky merged 10 commits into
mainfrom
benchmarks-history-comparison
Oct 18, 2026
Merged

dombrovsky merged 10 commits into
mainfrom
benchmarks-history-comparison

Conversation

@dombrovsky

@dombrovsky dombrovsky commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Implements #45 first, then #44, starting from PR #50's exact 7e3410545c4cdf38602f2cdd01b2e9e67d9bad51. This stacked PR targets benchmarks-1.0.0; merge #50 first, then retarget this PR to main. It does not use PR #47. Production trees, root build configuration, and all common C# harness/project/configuration files remain unchanged from #50.

Behavior

  • Benchmark history explicitly selects the release-based checkout for TaskFlow 1.0.0, verifies library/annotations/build/signing identities against 0800af84229d7beb3a6fa507d236d0ff533b0c92, and measures all 16 stable cases with SDK 10.0.100 / runtime 10.0.0 / BDN 0.15.8, three warmups and eight actual iterations. Main and exact release selections, weekly scheduling, provenance, raw JSON and readable summaries are supported. Dry/incomplete results are rejected.
  • A separate trusted publisher appends immutable records and an index to benchmark-data. Non-force push plus fetch/reapply retry preserves concurrent writers and reruns. Release records have no automatic expiry. Generated results never enter source branches. Read-only PR measurement jobs cannot publish.
  • The existing Documentation workflow includes the durable data in its complete Jekyll build and remains the sole Pages deployment. A trusted successful history workflow refreshes it through workflow_run; data-branch pushes do not loop. The chart/table page separates incompatible configurations and exposes time, throughput, valid allocations, variability, release labels, provenance and raw evidence links.
  • Benchmark comparison supports release and PR merge-base selection. It executes identical compatible harnesses sequentially on the same runner with independent build/raw directories and matched settings. Summaries include both metrics, absolute/percentage changes, uncertainty, unsupported/missing data and infrastructure failures. Performance changes are advisory; failures of infrastructure remain visible. The representative subset and full common suite are available.

Measured evidence and validation

  • Real release measurement: https://github.com/dombrovsky/TaskFlow/actions/runs/39103783011 — all 16 measured cases, not Dry; verified release checkout and runtime 10.0.0; raw measured-history artifact and workflow summary. Its publication job correctly skipped the untrusted PR context.
  • Real release and resolved main measurements: https://github.com/dombrovsky/TaskFlow/actions/runs/39104432556 — both measurement jobs succeeded and retained summaries/raw artifacts; publication correctly skipped.
  • Same-run comparison experiments: https://github.com/dombrovsky/TaskFlow/actions/runs/39104432591 — unchanged release-compatible candidate, intentionally slower disposable synthetic candidate, and old release 1.0.0-rc2 exclusion jobs all succeeded. Summaries and separate downloadable artifacts contain the reports. The actual PR merge-base mode also passed in this run.
  • Local measured release run: all 16 cases with real warmups/repeated samples, allocations except enqueue-only. Browser verification using these actual records rendered the explicit TaskFlow 1.0.0 table/chart point and allocation chart; enqueue-only allocations correctly produced no point. All 16 case choices remain available. This was a local preview, not a deployed Pages baseline.
  • Local unchanged comparison: timing changes from -22.1% to +6.8% all reported insufficient evidence because of variability/uncertainty. Synthetic SpinWait(5000) comparison: approximately 110–328x slower means; three material slowdown labels and the noisiest case insufficient evidence. The exact uncommitted synthetic diff is in provenance; it is removed with its disposable worktree and rejected by history publication. No production change is committed.
  • Historical pre-middleware snapshot 0e597547a98ec96b5e027ea7079e3876cb149432 vs release: 11 paired measured cases, identical harness/environment identity; all five middleware cases explicitly omitted from both sides. 1.0.0-rc2 has an incompatible Enqueue API and is explicitly unsupported rather than silently adapted.
  • Nine reporting/storage tests pass; actionlint passes all three changed workflows; JavaScript syntax and actual browser rendering pass. Immutable retries/reruns and a forced stale-writer push rejection followed by fetch/reapply preserved both raw record trees/index entries in temporary Git repositories.
  • Existing full Jekyll build passes: https://github.com/dombrovsky/TaskFlow/actions/runs/39104432544. No deployment was attempted from the PR.
  • Required solution restore/format commands were exercised. Release-source formatting is not clean: formatter touched 89 source/test files with only whitespace/BOM changes, which were backed up and reverted; style verification also reports pre-existing IMPORTS errors in TaskSchedulerBaseFixture.Scheduling.cs and LoggingTaskSchedulerExtensions.cs. No production formatting changes are included. Existing .NET 10 tests: 268 passed / one timing-sensitive core failure; its focused retry passed all three applicable fixtures. Existing .NET 8 core tests: 238 passed / 11 cancellation/timing failures. These occur in unchanged production/test source; no production fixes are included in this benchmark PR.

Required activation — live publication is not complete

The real 1.0.0 baseline is captured in Actions evidence, not yet durably published or deployed. main does not contain these new workflows. Do not close #45's live-publication acceptance on the strength of a PR artifact or local preview.

  1. Merge Add 1.0.0-based core and middleware benchmark baseline #50. Retarget this PR to main, review and merge it. No merge is performed by this session.
  2. On main, dispatch Benchmark history, source=1.0.0. This explicitly measures the release checkout again; running merged main is never substituted for the release.
  3. Confirm successful publication creates benchmark-data/index.json and the immutable release raw tree. Confirm the triggered Documentation deployment succeeds and https://dombrovsky.github.io/TaskFlow/benchmarks/ shows the TaskFlow 1.0.0 series, table and chart with provenance/run/raw links. That URL is the intended live view; it does not yet show this baseline.
  4. Dispatch source=main for the trusted current-main history point. Subsequent relevant merges, schedules and release events collect automatically. The manual comparison workflow also activates on the default branch.

Use the existing Pages Actions environment and permit the publisher's requested contents: write if repository policy restricts it; measurement jobs must remain read-only. Full reproduction, variance, compatibility, storage/retention, retries and activation guidance is in TaskFlow.Benchmarks/README.md.

Refs #45 and #44; depends on #50.

@dombrovsky dombrovsky changed the title Add verified release benchmark history and advisory comparisons Add verified 1.0.0 benchmark history and advisory same-run comparisons Oct 18, 2026
@dombrovsky
dombrovsky marked this pull request as ready for review October 18, 2026 06:36
@dombrovsky
dombrovsky changed the base branch from benchmarks-1.0.0 to main October 18, 2026 06:48
@dombrovsky
dombrovsky merged commit d3d9a9e into main Oct 18, 2026
13 of 14 checks passed
@dombrovsky
dombrovsky deleted the benchmarks-history-comparison branch October 18, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Track benchmark history by commit and publish trend charts

1 participant