Char semantics: follow C on NUL and non-UTF-8 bytes (fixes five fuzz-found divergences) - #2
Merged
Merged
Conversation
parse_var ended the variable-name run with a Rust `!var_chars.contains()` loop, and var_chars does not list NUL. C ends it with strlencspn, whose `strchr(reject, byte)` finds a NUL in the reject string's terminator, so a NUL ends the name even though it is not in the set. Without this the port folded a NUL into the variable and tokenized `@` runs differently. This was a false positive: differential fuzzing flagged `\0"@\0"/@\0\xef` as an injection here (fingerprint `sov`, blacklisted) while C sees it as clean. With the fix the port agrees with C, and the corpus differential stays at zero. parse_word already lists NUL in its own set, so it was unaffected; the number scans (strlenspn) are a separate NUL case, tracked separately.
The sp_password force-true searched the input by decoding it to a UTF-8
str first (from_utf8(...).unwrap_or("")), which collapses the whole
string to "" on any non-UTF-8 byte, so the needle was never found. It
also lowercased, making the match case-insensitive.
C's my_memmem is a case-sensitive search over the raw input bytes: it
finds sp_password regardless of surrounding high bytes. Match it with a
byte-window search.
Found by the differential fuzzer: a comment-terminated fingerprint with
sp_password embedded among high bytes was a false negative (Rust: false,
C: true). The text corpus reaches this construct only in ASCII, so a
dedicated test in the differential suite pins the non-UTF-8 case against
C. Full corpus differential stays at 0.
The collate + bareword fold retypes the bareword as an SQL type when it contains '_'. C uses strchr on the raw token value; the port searched value_as_str(), which returns "<binary>" on any non-UTF-8 byte, so a '_' sitting next to a high byte was lost and the bareword kept its type. Search the token value bytes directly, as C does. Same class as the sp_password fix. A dedicated differential test pins the non-UTF-8 case (fingerprint `At`, the `t` being TYPE_SQLTYPE). Full corpus stays at 0.
The 0x/0b prefix scans and the B'..'/X'..' string forms hand-rolled digit loops that stop at a NUL byte. C scans them with strlenspn, whose strchr-based membership test treats a NUL as a digit, so a NUL inside the literal is consumed as part of the number rather than ending it. Route all four through the existing strlenspn helper, matching C. A dedicated differential test pins the NUL-in-literal cases, including a UNION injection whose hex literal contains a NUL. Full corpus stays at 0.
ndreno
changed the title
C's h5_is_white is strchr(" \t\n\v\f\r", ch), and strchr matches the
string's own NUL terminator, so a NUL byte counts as whitespace. The
port's h5_is_white and is_whitespace omitted it, so a NUL inside an
attribute name or an unquoted attribute value did not end the token as C
does. The tokenizer then ran on and could reach a later `</`+backtick as
a comment, flagging XSS where C stays in a tag context and does not.
Add NUL to both predicates. state_tag_name keeps its own explicit NUL
branch, so it is unaffected. Found by the differential fuzzer; a dedicated
test pins the minimized input. Full corpus differential stays at 0.
ndreno
changed the title
The per-pull-request fuzz job now passes its two-minute-per-detector budget, so its comment no longer described what it does. Reword it as the short smoke test it is. Add fuzz-campaign.yml: a nightly and on-demand differential fuzzing job with a tunable per-detector budget (default 30 minutes), which drives toward an hours-clean surface and fails visibly on a divergence, uploading the crashing input for triage. Local fuzzing is unreliable here, so the campaign runs in CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The differential fuzzer finds divergences from the C library within
seconds, nearly all in one family: the port decodes raw input to a UTF-8
string, or stops a scan at a NUL, where C works on bytes and treats a NUL
as an ordinary set member (its
strchr(set, ch)matches the set string'sown NUL terminator). This PR fixes five such classes, each following the
C control flow rather than a single input, and adds a differential test
for each (the text corpus reaches none of them: it has no NUL and no
non-UTF-8 bytes).
Fixes
Variable name stops at a NUL (
parse_var). C'sstrlencspnendsthe name at the first NUL. Was a fuzz false positive.
sp_passwordsearched over raw bytes (is_not_whitelist). C'smy_memmemis a case-sensitive byte search; the port decoded to astring first, which collapses to
""on any non-UTF-8 byte. Was afuzz false negative.
Collate bareword
_searched over raw bytes (fold). C'sstrchrsearches the raw token value; the port used a lossy string.
Number literals scanned with
strlenspn(0x/0bprefixes andB'..'/X'..'forms). C counts an embedded NUL as a digit, so a NULinside the literal is consumed rather than ending it.
NUL is whitespace in the HTML5 tokenizer (
h5_is_white). C'sstrchr(" \t\n\v\f\r", ch)matches the terminator, so a NUL ends anattribute name or unquoted value; the port ran on and could reach a
later
</+backtick as a comment, flagging XSS where C does not. Was afuzz false positive.
The whole
strchr-over-a-literal family is now audited across bothdetectors and consistent:
h5_is_white,char_is_white(NUL and 0xA0),every
strlenspn/strlencspncaller, and the collate_search.Gate
The full-corpus differential (162,963 inputs) stays at 0 divergences on
both verdicts and fingerprints. The reporting-only fuzz job is the source
of these; each class it finds is triaged and fixed here.