fix(recall): importance filters, it does not rank - #35
Merged
Conversation
A self-rated field that does not predict use was moving results 2-4 places. It had already been demoted once — from a heavy weight down to 0.05 in the hybrid path and 0.1 in the FTS one — on the reasoning that a weak prior is harmless. Measured against RRF's actual spacing, it is not. Adjacent ranks are separated by ~0.0026, so: imp 8 vs 7 -> 0.0050 ~2 rank positions imp 9 vs 7 -> 0.0100 ~4 rank positions imp 10 vs 3 -> 0.0350 ~17 rank positions A top-8 injection is decided inside that range: four places is the difference between a row reaching the model and not. And the leverage was being spent on noise. On a live 8.6k-row store, rows that have ever been recalled average importance 7.70; rows never recalled average 7.36. A 0.34 gap on a 1-10 scale, with 90% of the corpus at >=7 — self-rated, saturated, uncorrelated with use. The comment sitting next to the term already said "don't let self-rated importance drive ranking"; 0.05 was still enough to drive it. importance keeps the jobs it is actually good at: the min_importance filter, surface-cold thresholds, and display. Filtering on it is a caller stating a floor. Ranking on it is the store guessing. FTS weights renormalized across the three surviving signals (0.61/0.22/0.17) so the score keeps its 0-1 shape. The test is calibrated to the failure regime, not to an obvious case. A first version gave the low-importance row a large relevance edge, which survived the tilt — so it passed with and without the fix and proved nothing. It now ties relevance and gives the low-importance row a single prior access, worth ~0.0455, deliberately less than the ~0.07 an importance gap of 10-vs-3 was buying. Verified red before the change and green after. level-rank-offset.test.mjs pins the composite as source text, so its regex moved with the formula; its actual assertion (no level-weight multiplier) is unchanged. 325 passed / 0 failed. Co-authored-by: 千夏 <qianxia@clawgamers.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The measurement
A self-rated field that does not predict use was moving results 2-4 places.
RRF separates adjacent ranks by ~0.0026. The importance term at weight 0.05 buys:
A top-8 injection is decided inside that range — four places is the difference between a row reaching the model and not.
What it was spending that on
On a live 8.6k-row store:
Self-rated, saturated, uncorrelated with use.
It had already been demoted once, from a heavy weight to 0.05, and the comment beside it already read "don't let self-rated importance drive ranking". 0.05 was still enough to drive it.
What changes
Removed from both scoring paths. FTS weights renormalized across the three surviving signals (0.61 / 0.22 / 0.17) so the score keeps its 0-1 shape.
importancekeeps the jobs it is good at —min_importancefiltering, surface-cold thresholds, display. Filtering on it is a caller stating a floor; ranking on it is the store guessing.On the test
Calibrated to the failure regime rather than an obvious case. My first version gave the low-importance row a large relevance edge — which survived the tilt, so it passed with and without the fix and proved nothing.
It now ties relevance and gives the low-importance row a single prior access, worth ~0.0455, deliberately less than the ~0.07 an importance gap of 10-vs-3 was buying:
level-rank-offset.test.mjspins the composite as source text, so its regex moved with the formula; its actual assertion (no level-weight multiplier) is unchanged.325 passed / 0 failed.
🤖 Generated with Claude Code