Repository navigation
fix(llm): use UTF-8 for model CLI text pipes - #577
Merged
codelion merged 1 commit intoNov 1, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Claude Code and Copilot CLI backends decode process output with the system locale because
subprocess.run(text=True)has no encoding. The Codex app-server backend does the same withPopen. On a non-UTF-8 locale, a successful process emitting UTF-8 non-ASCII text either raisesUnicodeDecodeErroror produces corrupted text. Claude's stdin prompt can also raiseUnicodeEncodeError; the Codex JSONL reader cannot deliver the Unicode message.Reproduced with real local Python processes under
LC_ALL=C,PYTHONUTF8=0,PYTHONCOERCECLOCALE=0(US-ASCII default). No model CLI or authenticated service is needed.Change
Set
encoding="utf-8"on the three model CLI subprocess boundaries. This covers text stdin/stdout/stderr without changing arguments, retries, timeout handling, exit-status handling, or the JSON-RPC protocol.Add four real-process regressions covering Claude output, Claude stdin echo, Copilot output, and a Codex JSONL notification containing Chinese text and an emoji. The tests launch an isolated interpreter with the non-UTF-8 environment; they skip on platforms that retain a UTF-8 default. They never change the user's global locale or run a real model CLI.
Validation
Python 3.12.13, macOS arm64, main
cb49662150aa794149d477379c605a3bff0c08a2:OPENAI_API_KEY=test-key-for-unit-tests python -m unittest discover tests: 1,007 tests run, 2 skipped; all remaining tests pass.git diff --check: pass.The initial test harness itself incorrectly re-encoded its Unicode literal; that was corrected before recording the baseline failures above. The regressions now reach the production subprocess code. Native Windows was not run, and no live Claude, Copilot or Codex invocation was tested.
Related open changes #515, #558 and #576 affect default budgets, JSON-RPC handling and exit status respectively; their diffs do not set the pipe encoding. This patch is limited to the shared CLI text-I/O contract.
Prepared with AI assistance from independent source review, with actual local child-process validation.