fix(clones): bound body bytes before tokenizing large literals - #1784
Merged
Merged
Conversation
The 4,096-token exclusion still admitted a body containing one multi-megabyte literal because the literal is a single token. That record again exceeded the 4 MiB text page and parked graph seating. Enforce the original 64 KiB source-byte guard before tokenization alongside the token guard; carry both limits through contracts and UI, and retain no token streams for either exclusion. Move every extractor revision in the same cut so old poisoned artifacts are re-extracted.
|
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
ScriptedAlchemy
added a commit
that referenced
this pull request
Sep 18, 2026
The packaged query-fallback receipts drifted once every extractor revision moved for the clone-body bound: c7eb62e (PR #1741) took Rust v8 to v9, TypeScript/protobuf/SQL v4 to v5 and the rest v3 to v4, and 542d28c (PR #1784) took Rust to v10. The extractor revision is part of the extraction batch identity, so the sealed generation every candidate binding names moves with it, and the fallback subpayload digest hashes those bindings. Bisecting the packaged comparison over the range since the last re-pin (3320ad4) lands on c7eb62e, whose only production change is those three revision integers. The ranking did not change. Comparing the packaged report at 3320ad4 against the current tip, every per-query row is byte identical: same first useful rank, returned candidates, wrong-scope and forbidden hits, and quality. Both partitions keep their exact conceptual-miss sets (10 train, 7 validation) and their mean reciprocal rank (476881 ppm train, 587222 ppm validation). Only the provenance identity inside the receipt moved. Re-pin both partition receipts, packaged::WORKLOAD_SHA256, and the byte-pinned workload digest both search-eval bins assert. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
cursor Bot
pushed a commit
that referenced
this pull request
Sep 18, 2026
The packaged query-fallback receipts drifted once every extractor revision moved for the clone-body bound: c7eb62e (PR #1741) took Rust v8 to v9, TypeScript/protobuf/SQL v4 to v5 and the rest v3 to v4, and 542d28c (PR #1784) took Rust to v10. The extractor revision is part of the extraction batch identity, so the sealed generation every candidate binding names moves with it, and the fallback subpayload digest hashes those bindings. Bisecting the packaged comparison over the range since the last re-pin (3320ad4) lands on c7eb62e, whose only production change is those three revision integers. The ranking did not change. Comparing the packaged report at 3320ad4 against the current tip, every per-query row is byte identical: same first useful rank, returned candidates, wrong-scope and forbidden hits, and quality. Both partitions keep their exact conceptual-miss sets (10 train, 7 validation) and their mean reciprocal rank (476881 ppm train, 587222 ppm validation). Only the provenance identity inside the receipt moved. Re-pin both partition receipts, packaged::WORKLOAD_SHA256, and the byte-pinned workload digest both search-eval bins assert. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Live regression after the token-only cut
With the cgroup and flexible builder fixes installed, the fresh generation finally reached the real deterministic blocker:
The 4,096-token guard does not bound serialized bytes: one multi-megabyte string/raw literal is one syntax token. #1577 is the only open PR touching
clone_body.rs; it optimizes token deserialization and predates even the token bound, so it does not solve this.Fix
ExcludedTooLarge, drops conservative/rename streams, and carries bothmaximum_tokensandmaximum_bytesthrough MCP/dashboard contracts.BodyExceedsSizeBoundand count 0; no giant literal is copied merely to count it.Behavioral evidence