Repository navigation
fix(llm): reject failed Claude CLI responses before returning stdout - #576
Merged
codelion merged 1 commit intoNov 2, 2026
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ClaudeCodeLLMtreats a non-zero Claude CLI exit as a warning when stdout is non-empty, returns that text as a successful generation and skips its retry loop. A failed process that writes a partial response or error text can consequently be passed to evolution as a valid model response. The sibling Copilot CLI backend already raises on non-zero exit.Change
Raise
RuntimeErroron a non-zero exit before accepting stdout. Include the exit code and the existing 500-character stderr limit. The existing retry loop now retries this failure and raises after its configured attempt budget. Successful stdout, stdin prompt delivery, empty-response validation and subprocess timeout handling remain unchanged.Add regressions for a real local Python process emitting stdout/stderr and exiting 7, a failed response followed by a successful retry through
generate(), and retry exhaustion when every failed attempt emits stdout.This is separate from #515, which changes the default budget in
__init__; its production diff does not touch exit-status handling.Validation
Python 3.12.13, macOS arm64, main
cb49662150aa794149d477379c605a3bff0c08a2:generate()check with a real offline CLI stub: before, returns the first failed process's partial stdout after one attempt; after, retries and returns only the successful response after two attempts.OPENAI_API_KEY=test-key-for-unit-tests python -m unittest discover tests: 1,006 tests run, 2 skipped; all remaining tests pass.git diff --check: pass.The process regression executes only the running Python interpreter, and the offline public-path check replaces
claudewith a temporary script. No actual Claude CLI, authentication flow, or live model/provider was invoked. Public retry regressions mock the subprocess transport while exercising the production retry logic.Prepared with AI assistance from an independent source review; subprocess status, returned output and attempt counts were verified directly.