Skip to content

fix(llm): skip zero-weight judge requests - #567

Merged
codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
Yushu-Li:fix/skip-disabled-llm-judges
Oct 31, 2026
Merged

codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
Yushu-Li:fix/skip-disabled-llm-judges

Conversation

@Yushu-Li

Copy link
Copy Markdown
Contributor

I fixed unnecessary model requests when an LLM judge has zero weight.

The evaluator already ignores zero-weight feedback, but generate_all_with_context still calls every configured model before that filtering happens. A disabled judge therefore consumes a request, and an exception from it can discard the active judges' feedback. In an offline reproduction with weights [0.0, 1.0], both clients are called once.

Skip non-positive-weight models at dispatch. Keep a None response slot for each skipped model so the evaluator's response indices still match the normalized weights and prompt logs. Update the response/logging type annotations to describe those slots. Positive-weight model ordering, forwarded options, weighted aggregation, client initialization, and active-judge error propagation are unchanged. This does not add all-zero-weight configuration validation.

Five new async regression tests use the real ensemble and, for feedback checks, the real evaluator with a temporary evaluation file. They cover disabled judges in the first/middle/last positions, disabled-client errors before and after an active judge, unequal-weight aggregation and missing metrics, aligned prompt logging, ordinary response ordering/options, and active-client error propagation. On the original dispatch implementation, three test methods fail across six failing subtests/assertions; the two compatibility controls pass.

Validation on macOS / Python 3.12:

  • All five new regression tests passed.
  • All eight existing ensemble/LLM-feedback tests passed.
  • OPENAI_API_KEY=test-for-offline-tests PYTHONPATH="$PWD" python -m unittest discover -s tests -p 'test_*.py' -v: 959 tests passed.
  • Black and isort checks on the new test module, Python 3.10 syntax checks on all changed files, and git diff --check passed. Existing formatting in the production modules is preserved.

I used AI assistance for implementation, regression tests, and local validation. No paid model calls or model-backed integration runs were performed.

@codelion
codelion merged commit da3f4d5 into algorithmicsuperintelligence:main Oct 31, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants