Gate the release on the metric one run can measure - #130
Merged
Conversation
The v0.8.5 release was blocked by a false-finding count of 7 against a ceiling of 6, and by gate-verdict correctness of 71.4% against a floor of 74.8%. Neither was a regression. Six runs of a single unchanged binary against this corpus, four on managed routing and two pinned to the qualified upstream provider, produced 4 to 7 false findings and gate-verdict correctness between 71.4% and 84.3%. Every request goes out at temperature 0, so that spread is the provider's own nondeterminism, not sampling. Pinning the provider does not remove it: the widest false-finding count came from a pinned run. A threshold of 2 findings and 2 percentage points sits well inside a spread of 3 findings and 12.9 points, so it stops two releases in six at random. A gate that fails at random is worse than no gate, because it teaches everyone to bypass it. Detection rate spanned 3.5 points over the same six runs and is the metric that most directly answers whether the reviewer found the defects, so it keeps its threshold and keeps blocking, alongside cost and latency. False findings and gate-verdict correctness are still computed, still compared, and now reported rather than enforced, with the measured spread printed beside them so a single unusual run is legible as one. Narrowing them again means reducing the noise, not tightening the number. Comparing a median across repeated runs is the direct fix, at proportionally more cost per release.
Contributor
Author
|
Confirming the escalation deliberately: these two metrics stop blocking, and the gate computation is updated in the same diff ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v0.8.5 release was blocked by a false-finding count of 7 against a ceiling of 6, and by gate-verdict correctness of 71.4% against a floor of 74.8%. Neither was a regression. Six runs of a single unchanged binary against this corpus, four on managed routing and two pinned to the qualified upstream provider, produced 4 to 7 false findings and gate-verdict correctness ranging from 71.4% to 84.3%. Every request goes out at temperature 0, so that spread is the provider's own nondeterminism rather than sampling, and pinning the provider does not remove it: the widest false-finding count came from a pinned run. Thresholds of 2 findings and 2 percentage points sit well inside a spread of 3 findings and 12.9 points, which stops roughly two releases in six at random, and a gate that fails at random is worse than no gate because it teaches everyone to bypass it. Detection rate spanned 3.5 points over the same six runs and most directly answers whether the reviewer found the defects, so it keeps its threshold and keeps blocking alongside cost and latency; the other two remain computed and compared but are reported rather than enforced, with the measured spread printed beside them so an unusual run stays legible as one. Narrowing them again means reducing the noise rather than tightening the number, and comparing a median across repeated runs is the direct fix at proportionally more cost per release.