forked from lance-format/lance-graph
-
Notifications
You must be signed in to change notification settings - Fork 0
plan: deepnsm-v2-lexical-address-v1 — harvest of closed #1303 #1307
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
75934e3
plan: deepnsm-v2-lexical-address-v1 — harvest of closed #1303
claude 575d76f
plan lexical-address: address Codex review — ordinal 0, ranks, f by s…
claude f572882
plan lexical-address: canonical u8 encoding; per-reference resolution…
claude e503098
Merge remote-tracking branch 'origin/main' into claude/deepnsm-v2-lex…
claude 59401ff
plan lexical-address: no sentinel in the u8 — unknown is a missing row
claude 1883766
plan lexical-address: known/unknown is an alpha bit, not a missing row
claude 727a66d
plan lexical-address: ratified v3 (5+3 council); escalate the alpha fit
claude 914316c
plan lexical-address: lemma_evidence lookup for ambiguous (surface-ke…
claude a33047c
Merge origin/main (#1308) into claude/deepnsm-v2-lexical-address
claude 6e89355
Merge origin/main (#1306) into claude/deepnsm-v2-lexical-address
claude File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
24 changes: 24 additions & 0 deletions
24
.claude/board/entries/2026-09-30-three-reference-sets-are-not-ordinal-aligned.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| # 2026-09-30 — The three COCA reference sets are neither nested nor ordinal-aligned | ||
|
|
||
| **Status:** MEASURED · OPEN — plan `.claude/plans/deepnsm-v2-lexical-address-v1.md` §2 (D-LXA-2, G-REF). Measured on PR #1303 (closed unmerged); carried here. | ||
|
|
||
| ## Measured (python over the committed CSVs and the Tigris artifact) | ||
|
|
||
| | reference | rows | distinct keys | key | | ||
|
AdaWorldAPI marked this conversation as resolved.
|
||
| |---|---|---|---| | ||
| | COCA4096 — `crates/deepnsm/word_frequency/word_rank_lookup.csv`, ranks ≤ 4096 | 4,096 ranks | 3,559 words | rank: one row per (word, PoS), each rank unique; a homograph spans several ranks | | ||
| | COCA5K_LEMMA — `lemmas_5k.csv` | 5,050 | 4,380 lemmas / 5,050 (lemma, PoS) | (lemma, PoS) | | ||
| | COCA20K_ACAD — `academic_20k.csv`; Tigris `lance-graph/codebooks/deepnsm-v2-academic-coca-v1/` | 20,845 | 18,559 words / 20,842 (word, Pos) | `word_id` = admission order | | ||
|
|
||
| - 5k ∩ 20k: 4,895 of 5,050 (lemma, PoS); 4,264 words; **116 of the 5k lemmas are absent from the 20k**. | ||
| - 4096 ∩ 20k: 3,462 of 3,559 words. | ||
| - **Ordinal alignment: 4 of 4,264 shared words carry the same ordinal in the 5k and in the 20k** (`the`, `there`, `care`, `wage`; 0-based first-occurrence order, exact match). ⊘ First recorded as 3: that count dropped ordinal 0, a falsy-zero error (Codex, #1307). | ||
| - Ambiguity differs per set: the 20k carve drops 2,286 rows to one id per word: 2,283 extra `(word, Pos)` keys plus 3 exact duplicate source rows (`wastewater/n`, `disproportionately/r`, `instill/v`) (MANIFEST: 18,559 of 20,480 reserved slots; basins 73..79 empty); the 4096 gives each (word, PoS) its own rank; the 5k keeps (lemma, PoS) distinct. | ||
| - The Tigris "academic codebook" TSV is a `PaletteVocab::from_frequency_ranked` carve (`word_id, basin, slot, …`), not a trained centroid codebook. Its `academic_20k.csv` sha256 equals the committed file's. | ||
|
|
||
| ## Consequence | ||
| A reading contract names (reference id, version, digest); an ordinal is stable only within its reference; correspondence is an explicit (lemma, PoS) map with `Ambiguous{n}` / `Missing` as values; switching references must preserve the declared map and never reuse an ordinal. Gate G-REF re-derives this table and diffs it. | ||
|
|
||
| ## Open | ||
| - Whether the academic reference gets its own codebook or shares COCA4096's through the correspondence map (plan §5). | ||
| - `range`/`disp` are loaded by no `src/` code; `Vocabulary::load` never opens `lemmas_5k.csv`. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.