Skip to content

fix: leave out a document whose converted text carries a NUL byte - #1

Merged
Vikram-Lex merged 1 commit into
Graphicle-AI:mainfrom
lame:fix/converted-text-nul
Oct 15, 2026
Merged

Vikram-Lex merged 1 commit into
Graphicle-AI:mainfrom
lame:fix/converted-text-nul

Conversation

@lame

@lame lame commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

graphicle sync / graphicle setup abort with Invalid source text (naming no file) when a converted document's text carries a NUL byte.

What happened

In the repository this was found in (a knowledge-graph project over US tax law, with a folder of research PDFs), graphicle sync failed like this:

graphicle: syncing <project>...
graphicle: Invalid source text

Bisecting the tree isolated two PDFs, then a single page of one of them: a math-heavy table, where pdf-extract writes U+0000 into the extracted text for a font glyph with no Unicode mapping. The other 11 papers in that folder sync fine, and nine other PDFs in the repository are already listed as skipped when conversion fails.

Root cause

Store::sync_upload refuses source text carrying a NUL, and every other source obeys that contract before it is sent: a text file with a NUL is left out as unsupported_binary (local.rs), and a connector document with one is refused ("must be UTF-8 text without NUL characters", connectors.rs). Converted document text was the one path that did not check: bounded() in documents.rs, which every conversion path ends in (in-process, OCR helper, worker child), checked only length and emptiness. The NUL reached the upload, the store refused it, and the failed step aborted the whole sync.

Change

bounded() refuses text with a NUL the way it refuses text past the limit, so one document cannot fail a sync (the rule tests/it/core_limits.rs states): the document is listed in the sync result and the rest of the folder syncs.

"skipped": [{"path": "…", "reason": "Document not converted: Extracted text contains a NUL character"}]

Test

documents::tests::extracted_text_with_a_nul_is_refused: extracted text with a NUL is refused. Removing the check makes it fail (unwrap_err() on Ok("before\0after")), and with it the text never reaches sync.upload.

Verification

In crates/graphicle, with Rust 1.95.0:

  • cargo fmt --check, cargo clippy --locked --all-targets -- -D warnings (with and without --features local-inference): clean
  • cargo test --locked --all-targets: all green (214 unit, 19 extension, 425 integration, 3 ignored)
  • cargo deny check: licences, bans and sources pass. Advisories report RUSTSEC-2026-0346 through RUSTSEC-2026-0349 (TLS stack) and a yanked yoke-derive; Cargo.lock is untouched by this change, so that is the state of main too
  • Folder-level reproduction: the graphicle program built from this branch syncs a folder holding the two PDFs that reported the failure, both listed as skipped, and the sync completes; same for a full copy of the reporting repository with both PDFs included

Notes

  • Through the conversion worker (the path the CLI takes), a refusal is reported with the same generic Document not converted: Document conversion failed or exceeded its resource limits as any other failed conversion, because the worker's own message does not cross the process boundary; in-process (library callers, the test) the refusal's own message is reported. Unifying that reporting is a separate change.
  • I left CHANGELOG.md to you.
  • The graph engine (crates/graphicle-graph) also depends on pdf-extract; I did not touch its document path, since the failure reported here is in the program's sync.

The store refuses source text that carries a NUL, and every other source
obeys that contract before it is sent: a text file with a NUL is left out
as `unsupported_binary` (local.rs), and a connector document with one is
refused (connectors.rs). Converted document text was the one path that
did not check: `bounded()` checked only length and emptiness, so a
converter that emits a NUL for a glyph with no Unicode mapping
(pdf-extract does, on a math font) had its upload refused, and the failed
step aborted the whole sync with "Invalid source text", naming no file.

`bounded()` now refuses a NUL the way it refuses text past the limit: the
document is listed in the sync result ("Document not converted: Extracted
text contains a NUL character") and the rest of the folder syncs. No
single file fails a sync.

Signed-off-by: Ryan Kuhl <rkk09c@gmail.com>
@Vikram-Lex
Vikram-Lex merged commit 30e3111 into Graphicle-AI:main Oct 15, 2026
7 of 8 checks passed
@lame
lame deleted the fix/converted-text-nul branch October 15, 2026 01:19
@lame

lame commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Woohoo, first merge into main!! 🙌

VikramLex added a commit that referenced this pull request Oct 15, 2026
The changes since 0.1.0 and PR #1, as the 0.1.1 entry of CHANGELOG.md
lists them: the engine follows Graphify 0.9.91; the engine holds its
working data in compact records and needs less memory, with the same
graph; Claude Code gets five tools, lean replies and test files last;
rustls 0.23.46 fixes RUSTSEC-2026-0346 to RUSTSEC-2026-0349; and a
Windows preview (install.ps1 and Windows release archives). Every crate
is at 0.1.1.

Signed-off-by: VikramLex <vikramlex@karlexai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants