Skip to content

fix(recall): importance filters, it does not rank - #35

Merged
DottytheHomeless merged 1 commit into
mainfrom
fix/drop-importance-from-ranking
Aug 20, 2026
Merged

fix(recall): importance filters, it does not rank#35
DottytheHomeless merged 1 commit into
mainfrom
fix/drop-importance-from-ranking

Conversation

@MXAntian

Copy link
Copy Markdown
Contributor

The measurement

A self-rated field that does not predict use was moving results 2-4 places.

RRF separates adjacent ranks by ~0.0026. The importance term at weight 0.05 buys:

gap score delta ≈ rank positions
imp 8 vs 7 (the common case) 0.0050 2
imp 9 vs 7 0.0100 4
imp 10 vs 3 0.0350 17

A top-8 injection is decided inside that range — four places is the difference between a row reaching the model and not.

What it was spending that on

On a live 8.6k-row store:

rows ever recalled    importance avg 7.70
rows never recalled   importance avg 7.36     Δ 0.34 on a 1-10 scale
90.1% of the corpus at >= 7

Self-rated, saturated, uncorrelated with use.

It had already been demoted once, from a heavy weight to 0.05, and the comment beside it already read "don't let self-rated importance drive ranking". 0.05 was still enough to drive it.

What changes

Removed from both scoring paths. FTS weights renormalized across the three surviving signals (0.61 / 0.22 / 0.17) so the score keeps its 0-1 shape.

importance keeps the jobs it is good at — min_importance filtering, surface-cold thresholds, display. Filtering on it is a caller stating a floor; ranking on it is the store guessing.

On the test

Calibrated to the failure regime rather than an obvious case. My first version gave the low-importance row a large relevance edge — which survived the tilt, so it passed with and without the fix and proved nothing.

It now ties relevance and gives the low-importance row a single prior access, worth ~0.0455, deliberately less than the ~0.07 an importance gap of 10-vs-3 was buying:

before fix:  ✗ evidence of use outranks a higher self-rating
             used(imp3,acc1)@1  rated(imp10,acc0)@0
after fix:   ✓

level-rank-offset.test.mjs pins the composite as source text, so its regex moved with the formula; its actual assertion (no level-weight multiplier) is unchanged.

325 passed / 0 failed.

🤖 Generated with Claude Code

A self-rated field that does not predict use was moving results 2-4 places.

It had already been demoted once — from a heavy weight down to 0.05 in the
hybrid path and 0.1 in the FTS one — on the reasoning that a weak prior is
harmless. Measured against RRF's actual spacing, it is not. Adjacent ranks are
separated by ~0.0026, so:

  imp 8 vs 7  ->  0.0050  ~2 rank positions
  imp 9 vs 7  ->  0.0100  ~4 rank positions
  imp 10 vs 3 ->  0.0350  ~17 rank positions

A top-8 injection is decided inside that range: four places is the difference
between a row reaching the model and not.

And the leverage was being spent on noise. On a live 8.6k-row store, rows that
have ever been recalled average importance 7.70; rows never recalled average
7.36. A 0.34 gap on a 1-10 scale, with 90% of the corpus at >=7 — self-rated,
saturated, uncorrelated with use. The comment sitting next to the term already
said "don't let self-rated importance drive ranking"; 0.05 was still enough to
drive it.

importance keeps the jobs it is actually good at: the min_importance filter,
surface-cold thresholds, and display. Filtering on it is a caller stating a
floor. Ranking on it is the store guessing.

FTS weights renormalized across the three surviving signals (0.61/0.22/0.17) so
the score keeps its 0-1 shape.

The test is calibrated to the failure regime, not to an obvious case. A first
version gave the low-importance row a large relevance edge, which survived the
tilt — so it passed with and without the fix and proved nothing. It now ties
relevance and gives the low-importance row a single prior access, worth ~0.0455,
deliberately less than the ~0.07 an importance gap of 10-vs-3 was buying.
Verified red before the change and green after.

level-rank-offset.test.mjs pins the composite as source text, so its regex moved
with the formula; its actual assertion (no level-weight multiplier) is unchanged.

325 passed / 0 failed.

Co-authored-by: 千夏 <qianxia@clawgamers.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@MXAntian
MXAntian marked this pull request as ready for review August 20, 2026 06:21
@DottytheHomeless
DottytheHomeless merged commit 47fa402 into main Aug 20, 2026
2 checks passed
@DottytheHomeless
DottytheHomeless deleted the fix/drop-importance-from-ranking branch August 20, 2026 06:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants