Skip to content

docs: auto-update for merged PRs (2026-09-02) - #4109

Open
aheritier wants to merge 4 commits into
mainfrom
docs/auto-update
Open

docs: auto-update for merged PRs (2026-09-02)#4109
aheritier wants to merge 4 commits into
mainfrom
docs/auto-update

Conversation

@aheritier

Copy link
Copy Markdown
Collaborator

Documentation changes

Brings docs/features/evaluation/index.md up to date with the evaluation
features merged into main over the last 36 hours. The assertions
(code-based grading) and --repeat pass@k/pass^k features landed across
several PRs but were never documented; the skills-related PRs and the
StartableToolSet backoff PR already carried their own doc updates and needed
no further changes.

  • Eval Criteria: document the new assertions field and the supported
    assertion types (contains, not_contains, equals, starts_with,
    ends_with, regex, cost_threshold, tool_called).
  • Scoring Metrics: add the Assertions row now that assertions are
    actually run and scored (not just accepted by the schema).
  • New "Repeat Metrics (pass@k / pass^k)" section: explain what --repeat
    now prints and what the two metrics measure.
  • Regression gate: note that assertion rate is now one of the aggregate
    metrics compared against --baseline, and that pass@k/pass^k are reported
    as informational (non-gating) deltas.

One commit per source PR:

PRs reviewed with no doc changes needed

Document the new `assertions` field on eval criteria (code-based
grading checks) added in #4088: the field in the
Eval Criteria table and the supported assertion types.

Source PR: #4088
Assertions are now actually run and scored as part of the eval
pipeline (#4100), not just accepted in the schema.
Add the Assertions row to the Scoring Metrics table and note there are
now four scoring dimensions.

Source PR: #4100
--repeat now computes and prints pass@k and pass^k consistency metrics
(#4092). Add a Repeat Metrics subsection explaining
what they measure and a sample of the console output.

Source PR: #4092
…ression gate

The baseline regression comparison now includes assertion rate as a
gating aggregate metric, and reports pass@k/pass^k as informational
deltas when both runs used --repeat (#4103).
Document this in the Regression gate section.

Source PR: #4103
@aheritier
aheritier requested a review from a team as a code owner September 2, 2026 04:05
@aheritier aheritier added kind/docs Documentation-only changes area/docs Documentation changes labels Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation changes kind/docs Documentation-only changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant