Skip to content

fix(tokenizer): raise Liquid::SyntaxError for invalid UTF-8 whatever the first token is - #2148

Draft
EvilGenius13 wants to merge 1 commit into
mainfrom
fix-tokenizer-invalid-utf8-syntax-error
Draft

EvilGenius13 wants to merge 1 commit into
mainfrom
fix-tokenizer-invalid-utf8-syntax-error

Conversation

@EvilGenius13

@EvilGenius13 EvilGenius13 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What and why

A template with an invalid byte raises a plain ArgumentError instead of Liquid::SyntaxError when the first token that isn't a {{ }} variable is a tag. The first skip_until validates the whole string, so whichever scanner method runs first decides which error comes out. Only next_text_token turns it into a SyntaxError, which #1905 added. The tag scanners came from the StringScanner rewrite in #1844 and never did. The split in {% liquid %} mode raises the plain ArgumentError too.

Template.parse checks valid_encoding? before it tokenizes, so it never hits this. Callers that build the tokenizer through ParseContext#new_tokenizer do. test_invalid_utf8 already builds a Tokenizer directly and expects SyntaxError, so that is the tokenizer's contract. In Shopify core the ArgumentError gets past rescue Liquid::Error in theme asset validation, so a bad byte fails the theme import or background job instead of reporting a syntax error.

This checks valid_encoding? once in Tokenizer#initialize, raises the same message as before, and removes the rescue in next_text_token that can no longer fire. Ruby caches the result of the encoding check on the string, so checking up front costs next to nothing.

#2056 added the same check but never fully merged.

…the first token is

A template with invalid bytes raised a plain ArgumentError when its first
non-variable token was a tag, or when it was a {% liquid %} body. Callers that
rescue Liquid::Error, such as validators that build the tokenizer through
ParseContext#new_tokenizer, crashed instead of reporting a syntax error.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant