Skip to content

Regenerate the retrieval baseline on demand, in CI - #680

Merged
mkreyman merged 1 commit into
masterfrom
ci/regenerate-retrieval-baseline
Aug 12, 2026
Merged

Regenerate the retrieval baseline on demand, in CI#680
mkreyman merged 1 commit into
masterfrom
ci/regenerate-retrieval-baseline

Conversation

@mkreyman

Copy link
Copy Markdown
Owner

Unblocks the next two items in the agreed order.

Why it's needed

Adding a golden question — including one grown from real logged queries — makes the eval run
:incomparable. question_set_changed?/1 compares the question-id set and refuses,
rather than letting an unmatched question read as a non-regression. That guard is right; it
just means the baseline has to be regenerable, and it wasn't.

Why in CI — and a correction

I wrote in #670's body, and repeated it, that a local run "degrades to keyword_only and
would overwrite the embeddings-arm baseline." That was wrong. The eval's vectors are
deterministic functions of the committed text — no provider is involved — and the
no_embedding_key warnings in the log are the keyword_only arm passing
embedding: {:error, :no_api_key} deliberately, exactly as its moduledoc says.

The real reason is the corpus. This job seeds a fresh database; a developer's dev DB
carries whatever else it has accumulated. Same commit, same command:

mrr answered@5
CI 0.746 22 / 26
local 0.192 5 / 26

A baseline captured locally would bake one machine's private database into a deploy gate for
everyone.

Shape

  • workflow_dispatch input, off by default.
  • Uploads the file as an artifact; CI never commits it. A gate that can rewrite its own
    threshold unattended is not a gate — download, read the diff, commit deliberately.
  • The gate step is skipped on a regeneration run. Leaving it in would compare the run
    against a baseline that same run had just written — a comparison that passes whatever the
    numbers are, which is the most expensive kind of green.
gh workflow run CI --ref <branch> -f regenerate_retrieval_baseline=true
gh run download <run-id> -n retrieval-baseline

Procedure documented in the runbook, including the measured local-vs-CI gap so the next
person doesn't re-derive it from a confusing local result the way I did.

mix precommit green (7,514). Reviewed inline (this session's system prompt forbids
dispatching review agents; per CLAUDE.md the gate is satisfied by the best available reviewer
with that stated).

Unblocks the next two items: adding a golden question - including one grown
from real logged queries - makes the eval run incomparable, because
question_set_changed?/1 compares the question-id SET and refuses rather than
letting an unmatched question read as a non-regression. That guard is right. It
just means the baseline has to be regenerable, and it was not.

IT HAS TO HAPPEN IN CI, and not for the reason I claimed twice today. I wrote in
#670's body that a local run degrades to keyword_only and would overwrite the
embeddings-arm baseline. That was wrong. The eval's vectors are deterministic
functions of the committed text, so no provider is involved at all, and the
no_embedding_key warnings in the log are the keyword_only ARM passing
{:error, :no_api_key} deliberately, exactly as its moduledoc says.

The real reason is the corpus. This job seeds a fresh database; a developer's dev
DB carries whatever else it has accumulated. Measured on the same commit and the
same command: CI scores mrr 0.746 with 22 of 26 answered, a local run scores
0.192 with 5 of 26. A baseline captured locally would bake one machine's private
database into a deploy gate for everyone.

The regeneration is a workflow_dispatch input, off by default, and uploads the
file as an ARTIFACT. CI never commits it: a gate that can rewrite its own
threshold unattended is not a gate. Download it, read the diff, commit it
deliberately.

The gate step is SKIPPED on a regeneration run. Leaving it in would compare the
run against a baseline that run had just written - a comparison that passes
whatever the numbers are, which is the most expensive kind of green.

Procedure documented in docs/runbooks/search-events-analysis.md, including the
measured local-vs-CI gap so the next person does not re-derive it from a
confusing local result the way I did.
@mkreyman
mkreyman enabled auto-merge (squash) August 12, 2026 19:34
@mkreyman
mkreyman merged commit 07e77c9 into master Aug 12, 2026
15 checks passed
@mkreyman
mkreyman deleted the ci/regenerate-retrieval-baseline branch August 12, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant