feat(explore): markdown index (#361) with a section-first doc tier

[bompus]

Follow-up to #361 and #1439, with the measurement from this comment on #361.

What this carries. #361's markdown extractor as it stands (headings, sections, tables and links as nodes; name-matcher and resolution hooks; its five test files), rebased onto main, plus a doc tier in codegraph_explore and three fixes the index needs to leave code answers alone.

The doc tier. A doc-shaped query that names a markdown file renders that file's best sections first and whole: the top three by idf-weighted line hits, where a term that also appears in the file's own path weighs zero (it located the file, not the line) and a heading the query covers word for word counts as a named section; 8k characters per file, spent in score order and rendered in file order. The blast-radius, relationships and "additional files" blocks stay off unless a code file rendered too. Code queries take the same path they did before.

Three fixes for a shared index.

• detectGeneratedFile skips the header check for markdown: a README that quotes "generated by" was dropped from the index.
• Markdown nodes leave the subgraph unless the doc tier seeded them: section bodies share the FTS table with code, so an English word in a README matched into code queries.
• The explore budget tiers count code files only. Markdown took a 466-file repo to 575 indexed files, across the 500-file breakpoint, and every code answer grew a Relationships block.

The server instructions now say markdown is indexed; #361's text still listed docs under what codegraph does not index, and measured as merged the model never called explore for a doc question (24 cells, 0 calls).

---

The retrieval fixes (bc3dfc9, 625e2f8)

The tier as first pushed could find a file and then render nothing of it. Pointing it at a real 84-file repo instead of a fixture turned up five queries that returned no result at all — CONTRIBUTING.md, readme, and three phrasings that name a heading containing a common word. One cause runs under all of them: the tier honoured "the query names this file" at the file gate and then forgot it at the section gate. Named got a file past the two-hit rule and past DOC_LOW_PATH, and then if (sections.length === 0) continue dropped it for want of a scoring line.

Nothing could score such a line, because three rules combine to guarantee zero:

• A term appearing in the file's own path weighs 0 — and for a bare filename query that is the only term, so every weight is 0.
• lineScore needs two distinct terms on one line; a one-term query cannot reach two.
• coveredHeading requires a term of non-zero weight, which by the first rule does not exist.

bc3dfc9 fixes the last two. coveredHeading falls back to presence when no term carries weight — the rarity signal is meaningless with one term, but "the query covers this heading" still means something — and a named file with no scoring section falls back to its first three non-wrapper headings. A user who types a filename has already said which file they want.

625e2f8 adds the two smaller ones:

• readme names README.md as surely as README.md does, but carries no .md for MD_PATH to see, so it never became named at all. A term equal to the file's own stem now unlocks the two-hit gate and the section fallback — but deliberately not the DOC_LOW_PATH bypass, which stays the privilege of an explicitly spelled path. readme should not surface a fixture copy, and a test asserts it does not.
• DOC_QUERY_NOISE words were stripped from the query and then demanded of the heading. Filtered on one side only, so a heading containing such a word could not be covered by any query that exists: "Cloning the repo on Windows" can never have repo covered, because repo is never a term. Both sides are filtered now.

Measured. All five dead queries return their file and the right section. Against the change, 11 control queries — 3 doc, 8 code — are byte-identical across two builds on the same index; the tier is retrieval-only, so the same index serves both arms and only dist/ is rebuilt. __tests__/explore-doc-tier-named.test.ts is new: 5 assertions driven through ToolHandler, not against the helper, and it ablates to exactly 3 failures against the pre-fix tools.ts.

---

Markdown path references under the native kernel (fixed in 56577ad)

An earlier revision of this PR disclosed this as a known limitation. It is now fixed, and the history is left here because the way it was caught is the useful part.

The gap. extractMarkdownPathReferences* — the code→document edge, a code file's string literal resolving to the heading it names — was implemented only on the wasm TreeSitterExtractor. tryKernelExtract returns before that extractor is constructed, so for any language in DEFAULT_ROUTED the capture never ran, and two of this PR's own tests failed whenever a kernel prebuild was present. It was invisible on CI because the prebuilds are not tracked in git, so an ordinary checkout runs the wasm path, and the one workflow step that runs with prebuilds downloaded runs __tests__/kernel-*.test.ts only — never the full suite with a kernel loaded.

The fix. The candidate finder and normalizer are ported to codegraph-kernel/src/markdown.rs and called from all 15 routed languages, not only the JS family. The two extraction methods are one macro (markdown_refs_impl!) because the wasm arm they must match is one implementation. The string half runs at both walker sites in every language; the subtree half is needed only where a walker stops before a declaration's value (tsjs, python, java, csharp, ruby, lua) and is unused elsewhere by design, which is stated at the macro rather than left as a puzzle.

One wire-format detail: markdown refs reach the store through addReference, which denormalizes both filePath and language, unlike the ordinary ref path. REF_FLAG_LANGUAGE joins REF_FLAG_FILE_PATH so decode re-attaches both, and only on these refs — parity compares ref objects whole, so an extra field on an ordinary ref fails the gate.

How the fixtures earned it. Every language's torture fixture gains the same eight shapes, including both rejection cases. On the first run after wiring, 8 of 12 parity files passed and java, csharp, ruby and lua each came up short by exactly 7 refs — the seven variable initializers. An identical shortfall across four independent walkers is a structural fact rather than four separate bugs, and it pointed straight at the cause: those walkers stop before a declaration's value. That diagnosis was only available because the number was the same in all four.

One shape is worth calling out because it is a deliberate non-rejection. https://example.com/remote.md is not rejected: the candidate regex matches starting at the // inside the scheme, so the :// look-back never fires and both arms emit example.com/remote.md. My first unit test asserted the opposite and failed; the test was wrong, not the port. It is now pinned by the parity fixture rather than by an assertion I reasoned my way to.

Scope. Markdown files were never affected: markdown is not in DEFAULT_ROUTED, so .md files are indexed by their own extractor and the doc tier, the sections, and the retrieval fixes always worked normally.

---

Measured (doc tier). Headless Claude Code, Opus, a repo with 109 markdown files, six doc-question tasks, three rounds of 12 fresh cells against the shipped build with the same repo rules:

	Round 1	Round 2	Round 3
Correct	12 / 12	12 / 12	12 / 12
Tool calls, median (shipped: 4)	1	1	1
Explore adopted	10 / 12	11 / 12	10 / 12
No Grep or Read after explore	8 / 12	9 / 12	9 / 12

Every explore call chose the right file and section; the misses are the model grepping a file the prompt already names, and one prompt it reads as being about its own scheduling tools. Cost per cell $0.30 against $0.54.

Code answers on the same repo, one fresh session per query, 25 code queries against main: 11 byte-identical, 3 the same lines reordered, 10 swap a fourth- or fifth-ranked padding file (markdown documents shift FTS tie ranks), and one prose prompt with no identifier answers from two markdown files instead of five unrelated code files.

One column reads as a precision regression and is not one. Edges with provenance = 'heuristic' multiply on this branch — on vitest, 219 → 5,089. Splitting them by whether either endpoint is a .md file:

vitest	edges	heuristic	code-side	markdown
b9ca4b7	73,264	219	219	0
this branch	86,996	5,089	219	4,870 (all contains)

Code-side heuristic edges are identical to the edge; every added one is a markdown section-contains edge, which is structural rather than inferred. The same split on vite reads 36 → 3,082 with the code side unchanged — different repo, different totals, same invariant. Found jointly with the session reviewing this branch, who spotted the column and knew how it would read.

A code-side change this branch does cause, and did not claim. Indexing markdown alters code resolution as a side effect. On vite, against b9ca4b7: 71 edges removed, 0 added.

target	edges
playground/ssr-html/test-stacktrace.js::vite	60
packages/vite/src/node/preview.ts::cors	6
docs/.vitepress/theme/composables/sponsor.ts::Sponsors	2
…/server/middlewares/hostCheck.ts::host-validation-middleware	2
…/server/index.ts::ViteDevServer	1

An earlier revision gave this as "71 rows, 69 distinct" and explained the gap as two valid units. That was wrong in a way worth stating, because the smaller number was the defective one. My dedupe key was (kind, source, line, target) with no column, and an import line carries two distinct references — the module specifier at column 0 and the binding it introduces a few columns later. The key merged those two into one entry, on this diff and everywhere else I had used it. Adding col makes distinct and rows agree exactly, here (71) and on the whole baseline graph (27,778 = 27,778). There is one unit; the second number was a key that could not see a column.

The 60 are import … from 'vite' — the npm package — previously resolving onto that playground file's own vite node. They carried resolvedBy: exact-match, so this is candidate dilution at matchByExactName, not the fuzzy path: nodes named vite go 159 → 161, the two additions being README.md and docs/guide/cli.md, and the candidates.length === 1 branch that stamps exact-match no longer sees a unique survivor. It declines, correctly — the true target is external and has never been in the graph.

So a reviewer diffing edge counts meets a 71-edge drop on a PR that presents itself as retrieval-only. It is a precision gain rather than a regression, it is unrelated to the heuristic column above, and it is the largest single code-side effect measured on vite for any change in this area. Identified by the session reviewing this branch; the mechanism and the candidate-pool counts are mine.

Tests. cargo build --release exits 0 with no dead-code warnings, 25 Rust unit tests pass, and tsc --noEmit exits 0. All 13 kernel parity files pass, now including markdown shapes in every language's fixture. Full suite on Node v24.16.0 (the version scripts/build-bundle.sh vendors, inside engines.node), Windows, with a kernel prebuild present: 4,189 passed, 24 failed, 0 assertion failures.

Two corrections to how this paragraph read in an earlier revision.

The first is mine. It previously reported 4,185 passed / 26 failed with 2 assertion failures, and described those two as pre-existing before correcting that. They were caused by this PR — the markdown path-reference tests — and the ablation that first told me otherwise was not a control at all: I reverted to a commit on this branch, which still contained the whole feature. Both now pass, which is the difference between the two runs.

The second is about the remaining 24, and it is a wrong cause rather than an imprecise one. I attributed them to Windows Defender on-access scanning of binaries outside C:\Program Files. They are Windows teardown defects — an unclosed db handle, an un-awaited child, waitForMarker returning on existence — and #1717 fixes them:

arm	failed	passed	assertion failures
this branch	24	4,189	0
this branch + #1717	1	4,210	0

The single survivor is __tests__/sync.test.ts → "persists an oversized skipped file so later syncs do not retry it (#1557)", a 5 s timeout. That one is scanner-sensitive: it writes 'const value = 1;\n' ×70,000, close to a worst case for on-access scanning, which a separate investigation measured at 12.6 s to first-read against ~1 ms unscanned. So the Defender finding is real and explains exactly one of the 24; generalising it to the whole class was the error. Neither version changed a number in this PR — every claim here is a count, not a latency — but the ablation that would have caught it was merging #1717, which I had reviewed without ever running against this branch.

Re-index after upgrading: the markdown nodes are written while indexing.

[bompus]

Real-repo run: the index half holds up, the retrieval half is brittle

Ran this against vitejs/vite (84 markdown files) alongside upstream main, same checkout, fresh index per arm.

Indexing works

1,635 → 1,719 files, exactly the 84 markdown files. 3,130 markdown nodes, and the sections carry real heading names:

file    CONTRIBUTING.md
module  Vite Contributing Guide      :1
module  Repo Setup                   :5
module  Cloning the repo on Windows  :19
module  Documentation                :32

Code queries are unaffected on three of four probes — byte-identical rankings and graph mass against main. On the fourth (import.meta.glob) plugins/importMetaGlob.ts drops out of the top 5 in favour of types/importMeta.d.ts; worth a look, but it's a shuffle within the same file set, not a loss.

Retrieval works when it fires, and it fires well

query	top markdown result
the contributing documentation for this repo	#1 CONTRIBUTING.md
the vite documentation on the plugin api	#1 docs/guide/api-plugin.md
docs on the release process	#1 CONTRIBUTING.md, #3 docs/releases.md

That second one is a good answer out of 1,719 files.

But several natural doc queries return no markdown at all

query	result
CONTRIBUTING.md	nothing — 0 files grouped
readme	nothing
the contributing guide	nothing
readme installation documentation	nothing
the contributing guide for windows	16 files grouped, 0 markdown

The bare filename is the sharpest case: the query is the file name, and the user gets an empty response.

It is not the seed lookup. I called it directly against the built index, and CONTRIBUTING.md comes back as the #1 hit for every relevant combination:

searchNodes("contributing", kinds=file)                  -> 7   #1 CONTRIBUTING.md
searchNodes("contributing", kinds=file|module|constant)  -> 30  #1 CONTRIBUTING.md
searchNodes("guide",        kinds=file|module|constant)  -> 30  CONTRIBUTING.md present

So the candidate is found and then lost downstream, in section selection. Two rules in collectDocSeeds are consistent with it: a line scores only when two distinct query terms meet on it (hit.length >= 2), and a term that appears in the file path is weighted 0 — so for CONTRIBUTING.md the only term is the one the weighting discards, no section can score, and a seed with no sections contributes nothing.

I want to be straight about how far I took that: I confirmed the seed layer directly and I can show the working/failing boundary, but I did not step through section scoring for each failing query, and adding a third term (the contributing guide for windows) revived the code search without reviving the doc tier — so the rule I described above is consistent with the evidence, not proven to be the whole story.

Suggested shape, and I'm happy to write it: when the query names a .md path, or when the doc word is the only significant term, seed the file's first N sections rather than requiring per-line term co-occurrence. A user who types a filename has already told us which file they want.

The DOC_WORD gate itself is fine, and my first three probe queries deserved to miss — "how do I submit a pull request to this project" carries no doc word, and that's the design working, not a bug.

[bompus]

Root cause of the retrieval misses, and a correction to my last comment

I said above that two rules in collectDocSeeds were "consistent with" the misses and that I had not stepped through section scoring. I have now. My description was incomplete in a way that matters: the failing queries die at two different places, and the two rules I named are not two contributors — they are one cause disabling two independent recoveries.

readme and the contributing guide never reach section scoring at all

They are dropped at the file gate, tools.ts:918:

const hits = termScore(lp + ' ' + headings.map((h) => h.name).join(' '));
if (!named && hits < 2) continue;

termScore counts distinct query terms, so hits can never exceed terms.length. For readme the term list is exactly ['readme'] — md is dropped by the length >= 3 filter — so hits maxes out at 1 and the file is structurally incapable of clearing a < 2 bar. No amount of matching helps; the query has only one term to match with.

This is a different failure from the one I described, and it covers the readme and bare-doc-word cases.

CONTRIBUTING.md reaches section scoring and dies at tools.ts:996

Here named is true, so line 918 correctly lets it through — this is why my direct searchNodes probes found the file every time. It then produces zero sections and is discarded whole:

if (sections.length === 0) continue;

Every section scores 0, and the reason is single. Line 938 zeroes the weight of any term appearing in the file path:

if (lp.includes(t)) return [t, 0];

For a bare filename query the only term is the path, so the entire term set is weightless. That then disables both scoring paths at once:

• lineScore (:945) requires hit.length >= 2 — two distinct terms meeting on one line. With one term this is unsatisfiable regardless of weight.
• coveredHeading (:965) — the escape hatch built precisely for one-word headings that the two-term rule cannot see — requires a term with weight > 0:

  const t = terms.find((q) => (weight.get(q) ?? 0) > 0 && w.includes(q));
  if (!t) return 0;

  The only term has weight 0, so it returns 0 on every heading.

So the path-zeroing rule does not merely discount the filename token; it takes the fallback down with it. That is why adding a third term (the contributing guide for windows) revived the code search without reviving the doc tier — the observation I flagged as unexplained.

The asymmetry is the bug in one line: named is honoured at 918 and ignored at 996. A user who types a filename has told us which file they want, and the file layer agrees — then the section layer, which has no notion of named at all, throws it away.

A third problem, which I had not spotted

coveredHeading requires every significant word of a heading to be covered — a strict AND over heading words. Its word filter is length plus STOP_WORDS (:961), but query terms are filtered by length, STOP_WORDS, and DOC_QUERY_NOISE (:883):

const DOC_QUERY_NOISE = new Set(['repo', 'repos', 'one', 'line', 'must', 'next', 'per', 'point', 'name', 'names', 'two', 'each']);

A DOC_QUERY_NOISE word is therefore stripped from the query but still demanded of the heading — so any heading containing one of those words can never be covered, no matter what the user types. Cloning the repo on Windows in vite's CONTRIBUTING.md is exactly this: repo must be covered and cannot be, so the section is unreachable through coveredHeading even for the query the contributing guide for windows, which names two of its three words.

That is a small set of words, but repo, name and section are common in headings.

Suggested fix

Three changes at the three points, keeping the DOC_WORD gate exactly as it is — the gate is fine and my first three probe queries deserved to miss:

1. :996 — honour named. When a seed is named and scores no sections, seed the file's first N sections rather than dropping it.
2. :965 — fall back when every term is path-zeroed. If no term has weight > 0, match on presence instead of weight, so coveredHeading still functions for a single-term filename query.
3. :961 — apply DOC_QUERY_NOISE to heading words too, so a word that cannot appear in the term list is not required of the heading.

I'll open this as a follow-up commit on this branch unless you'd rather have it separately. Happy to be told the shape is wrong — (1) is the one I am confident about, and (3) is a two-line change with a clear argument; (2) is the one where I would most welcome a second opinion, since "no term has weight > 0" may deserve to fail loudly rather than degrade.

Measurement caveat

The vite numbers in my previous comment were produced with the index built under Bun. That is not this project's supported runtime — engines.node is >=20.0.0 <25.0.0 — and I have since set up the vendored v24.16.0 from build-bundle.sh to re-confirm my other measurements. The root-cause analysis above is read from source and does not depend on those counts, but the retrieval tables in the earlier comment should be treated as measured-under-Bun until I re-run them.