fix(chunking): stable content-defined chunking with hash-based reuse planning

[MauryaQbit]

Fixes #1649.

Problem

On the self-hosted server, updating an existing document via the /v3/documents\ customId upsert path re-embeds nearly all chunks when the edit is anywhere but the tail: a one-paragraph mid-file edit to a 105 KB doc re-embedded 1007 of 1102 chunks (~21 min CPU on 2 vCPU with local embeddings). Append-only documents diff perfectly, which points at position-sensitive chunk boundaries defeating the per-chunk reuse check for mid-file edits: an early insertion/deletion shifts every downstream boundary, so every downstream chunk hash changes and reuse never hits.

Fix

New zero-dependency shared package @repo/chunking\ (\packages/chunking) implementing exactly the two mitigations suggested in the issue:

1. Content-defined chunking (\stableChunkText) — markdown headings and fenced code blocks are hard resync points (never merged across), and within a section boundaries fall on sentence/code-line units with FastCDC-style content-hash anchored cuts, so boundaries re-synchronize a few units after an edit instead of shifting to end-of-document. Boundaries always land on sentence/line edges; documented 2-sentence overlap preserved (restricted to the same section so sections stay independent).
2. Anchor-based resync (\planChunkReuse) — matches the new chunk list against stored chunks by content hash (FIFO per hash, O(n)) instead of by position, so unchanged chunks hit regardless of where they moved. Returns { reused, embedIndices }: carry over old embeddings, embed only \embedIndices.

Intended adoption on the upsert path: \chunks = stableChunkText(content)\ then \plan = planChunkReuse(storedChunks, chunks)\ and embed only \plan.embedIndices. Pure TS, no
ode:crypto, runs in Node/Bun/Workers.

Verification

14 new vitest tests (\packages/chunking/src/stable-chunking.test.ts, \�un run test\ in the package after install; verified here with vitest 3.2.4 — 14/14 pass). On a generated 45 KB / 90-chunk markdown doc:

scenario	stable chunker	naive fixed-size chunker
mid-file sentence replace	89 reused, 1 embeds	44 reused, 48 embed
append new section	90 reused, 1 embeds	—
delete a section	0 new embeds	—

Also covered: determinism, sequential positions + hash identity, sentence-boundary cuts, intact code-fence lines, duplicate-chunk FIFO pairing, position-independent matching, empty-input edge cases. \ sc --noEmit --strict\ (incl.
oUncheckedIndexedAccess\ + \exactOptionalPropertyTypes) and \�iome check\ are clean.

Note: the ingestion/embedding pipeline itself lives outside this monorepo, so this PR provides the algorithm + tested reference implementation for the pipeline to adopt; no existing behavior in this repo changes.

[MauryaQbit]

Update: pushed lockfile + CI wiring so this is merge-ready pending review.

Second commit (\9d22f5e)

• \�un.lock: registers the new \packages/chunking\ workspace (keeps \�un install --frozen-lockfile\ green).
• .github/workflows/ci.yml: runs chunking unit tests + type checking when \packages/chunking\ changes (same conditional pattern as tools/ai-sdk).

Verified locally with the exact CI toolchain (bun 1.3.6)

• \�un install --frozen-lockfile\ — no changes, passes
• \�un run --cwd packages/chunking test\ — 14/14 pass
• \�un run --cwd packages/chunking check-types\ — clean
• \�unx biome ci --changed --since=origin/main --no-errors-on-unmatched\ — clean

Note: the Actions runs for this fork PR show \�ction_required, so CI needs a maintainer to approve the workflow runs. Ready for review.

[yesprasad]

🔎 TracePull — evidence-guided merge review

@MauryaQbit @Dhravya @MaheshtheDev @Prasanna721

Go / No-Go: GO WITH CHECKS

The new stable chunking pipeline is well structured: it preserves section boundaries, splits oversized individual units, derives stable content-defined cuts, and uses content hashes for reuse.

One behavior warrants validation before relying on maxChars as a hard output bound.

---

Area to check — final chunk may exceed maxChars

Location: packages/chunking/src/stable-chunking.ts:346

run.push(unit)
runChars += unit.text.length

if (
  !isLast &&
  run.length >= minSentences &&
  (run.length >= maxSentences || runChars >= maxChars || anchor)
) {
  runs.push(run)
  run = []
  runChars = 0
}

runChars is evaluated only after the current unit has been appended. On the final unit, !isLast prevents a cut altogether.
That means a final run can exceed maxChars even though the option is documented as a hard character cap. The same question applies to final rendered content when a heading and overlap are included.

flowchart TD
    A[stableChunkText markdown options] --> B[splitSections]
    B --> C[unitsForSection\nsplit oversized units]
    C --> D[Append current unit to run]
    D --> E{Non-final unit\nand cut condition met?}
    E -->|Yes| F[Push run and reset]
    E -->|No| G[Keep accumulating]
    F --> H[renderChunk heading overlap run]
    G --> I{Final unit?}
    I -->|Yes| J[Push final run]
    J --> H
    H --> K[Emit StableChunk]
    D -. current unit added before size check .-> E
    I -. final unit bypasses cut .-> J

Loading

Suggested validation

Please add a regression that creates a section where the final unit pushes the accumulated run beyond maxChars, then assert the intended contract explicitly:

const chunks = stableChunkText(
  `First sentence. ${"word ".repeat(80)}Final sentence.`,
  { maxChars: 256, minSentences: 1 },
)

expect(chunks.every((chunk) => chunk.content.length <= 256)).toBe(true)

If maxChars is intended to bound only the pre-render run—not final StableChunk.content including heading and overlap—document that distinction instead.

Why this matters

The implementation's stability/reuse guarantees are strongest when chunk size behavior is deterministic as well. A trailing oversized chunk can affect downstream embedding limits, storage assumptions, and consistency of chunk sizing.

---

_{TracePull reviewed the changed algorithm and source-cited execution path. This observation has not been runtime-confirmed in this review.}