Repository navigation
fix: leave out a document whose converted text carries a NUL byte - #1
Merged
Merged
Conversation
The store refuses source text that carries a NUL, and every other source
obeys that contract before it is sent: a text file with a NUL is left out
as `unsupported_binary` (local.rs), and a connector document with one is
refused (connectors.rs). Converted document text was the one path that
did not check: `bounded()` checked only length and emptiness, so a
converter that emits a NUL for a glyph with no Unicode mapping
(pdf-extract does, on a math font) had its upload refused, and the failed
step aborted the whole sync with "Invalid source text", naming no file.
`bounded()` now refuses a NUL the way it refuses text past the limit: the
document is listed in the sync result ("Document not converted: Extracted
text contains a NUL character") and the rest of the folder syncs. No
single file fails a sync.
Signed-off-by: Ryan Kuhl <rkk09c@gmail.com>
Contributor
Author
|
Woohoo, first merge into main!! 🙌 |
VikramLex
added a commit
that referenced
this pull request
Oct 15, 2026
The changes since 0.1.0 and PR #1, as the 0.1.1 entry of CHANGELOG.md lists them: the engine follows Graphify 0.9.91; the engine holds its working data in compact records and needs less memory, with the same graph; Claude Code gets five tools, lean replies and test files last; rustls 0.23.46 fixes RUSTSEC-2026-0346 to RUSTSEC-2026-0349; and a Windows preview (install.ps1 and Windows release archives). Every crate is at 0.1.1. Signed-off-by: VikramLex <vikramlex@karlexai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
graphicle sync/graphicle setupabort withInvalid source text(naming no file) when a converted document's text carries a NUL byte.What happened
In the repository this was found in (a knowledge-graph project over US tax law, with a folder of research PDFs),
graphicle syncfailed like this:Bisecting the tree isolated two PDFs, then a single page of one of them: a math-heavy table, where
pdf-extractwrites U+0000 into the extracted text for a font glyph with no Unicode mapping. The other 11 papers in that folder sync fine, and nine other PDFs in the repository are already listed as skipped when conversion fails.Root cause
Store::sync_uploadrefuses source text carrying a NUL, and every other source obeys that contract before it is sent: a text file with a NUL is left out asunsupported_binary(local.rs), and a connector document with one is refused ("must be UTF-8 text without NUL characters",connectors.rs). Converted document text was the one path that did not check:bounded()indocuments.rs, which every conversion path ends in (in-process, OCR helper, worker child), checked only length and emptiness. The NUL reached the upload, the store refused it, and the failed step aborted the whole sync.Change
bounded()refuses text with a NUL the way it refuses text past the limit, so one document cannot fail a sync (the ruletests/it/core_limits.rsstates): the document is listed in the sync result and the rest of the folder syncs.Test
documents::tests::extracted_text_with_a_nul_is_refused: extracted text with a NUL is refused. Removing the check makes it fail (unwrap_err()onOk("before\0after")), and with it the text never reachessync.upload.Verification
In
crates/graphicle, with Rust 1.95.0:cargo fmt --check,cargo clippy --locked --all-targets -- -D warnings(with and without--features local-inference): cleancargo test --locked --all-targets: all green (214 unit, 19 extension, 425 integration, 3 ignored)cargo deny check: licences, bans and sources pass. Advisories report RUSTSEC-2026-0346 through RUSTSEC-2026-0349 (TLS stack) and a yankedyoke-derive;Cargo.lockis untouched by this change, so that is the state ofmaintoographicleprogram built from this branch syncs a folder holding the two PDFs that reported the failure, both listed as skipped, and the sync completes; same for a full copy of the reporting repository with both PDFs includedNotes
Document not converted: Document conversion failed or exceeded its resource limitsas any other failed conversion, because the worker's own message does not cross the process boundary; in-process (library callers, the test) the refusal's own message is reported. Unifying that reporting is a separate change.CHANGELOG.mdto you.crates/graphicle-graph) also depends onpdf-extract; I did not touch its document path, since the failure reported here is in the program's sync.