Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I fixed unnecessary model requests when an LLM judge has zero weight.
The evaluator already ignores zero-weight feedback, but
generate_all_with_contextstill calls every configured model before that filtering happens. A disabled judge therefore consumes a request, and an exception from it can discard the active judges' feedback. In an offline reproduction with weights[0.0, 1.0], both clients are called once.Skip non-positive-weight models at dispatch. Keep a
Noneresponse slot for each skipped model so the evaluator's response indices still match the normalized weights and prompt logs. Update the response/logging type annotations to describe those slots. Positive-weight model ordering, forwarded options, weighted aggregation, client initialization, and active-judge error propagation are unchanged. This does not add all-zero-weight configuration validation.Five new async regression tests use the real ensemble and, for feedback checks, the real evaluator with a temporary evaluation file. They cover disabled judges in the first/middle/last positions, disabled-client errors before and after an active judge, unequal-weight aggregation and missing metrics, aligned prompt logging, ordinary response ordering/options, and active-client error propagation. On the original dispatch implementation, three test methods fail across six failing subtests/assertions; the two compatibility controls pass.
Validation on macOS / Python 3.12:
OPENAI_API_KEY=test-for-offline-tests PYTHONPATH="$PWD" python -m unittest discover -s tests -p 'test_*.py' -v: 959 tests passed.git diff --checkpassed. Existing formatting in the production modules is preserved.I used AI assistance for implementation, regression tests, and local validation. No paid model calls or model-backed integration runs were performed.