Skip to content

feat(evaluations): add LD judge event support - #63

Draft
donei003 wants to merge 5 commits into
mainfrom
feature/ld-judges-phase3
Draft

feat(evaluations): add LD judge event support#63
donei003 wants to merge 5 commits into
mainfrom
feature/ld-judges-phase3

Conversation

@donei003

@donei003 donei003 commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • add Judge and Scorer criterion references for SDK-run evaluations, passed via run(criteria=[...])
  • resolve LD Judge configs (with a valid {kind: "evaluation"} context) before mutating evaluation/run state; resolution failures fail fast and carry the underlying cause
  • write judge/scorer criteria onto evaluation creation
  • execute each criterion per generated row — concurrently, bounded by the same concurrency parameter as generation — and emit one $ld:ai:offline-evals:evaluation event per (row, criterion) result
  • include criterion identity, judge metadata, validated score/reason, usage, timings, and cause-coded error payloads in evaluation events

Design notes

  • Public parameter is criteria= (union type Criterion = Judge | Scorer) to match the wire format (criteria / criterionType); a scorer is not a judge.
  • Scorer.fn receives the public DatasetRow dataclass plus the generated output, so internal result-dict keys are not part of the customer contract.
  • Judge configs are passed to the handler unrendered; the handler owns the single parse_template pass, so {{...}} sequences inside generated output or dataset values are never expanded into judge prompts.
  • Judge scores are validated when the judge responds (numeric, finite, 0–1). Invalid output becomes a per-criterion ERROR event with a cause code (invalid_judge_output, invalid_score, handler_raised, scorer_raised, generation_incomplete) and a top-level errorMessage, mirroring generation events — it never aborts the run after LLM spend. Queued generation events are flushed in a finally.
  • Duplicate criterion identities (judge key / scorer name collisions) are rejected before any records are created, since criterionType is part of the deterministic event identity.
  • The judge response contract (formatting instructions + {score, reasoning} parsing) is shared with the online judge path via judge_scoring.py.
  • Event payloads are frozen dataclasses with explicit wire serialization — no new runtime dependencies (the earlier pydantic dependency was dropped).

Notes

  • No pre-built LD Judges are created or hardcoded by the SDK; callers pass judge keys directly.
  • Evaluation results are emitted as individual LD events per (row, criterionType), not via batch ingest.
  • Still draft pending backend contract review: event/criteria naming and the cause-code taxonomy for error.code (string cause codes here vs. numeric 5001 on generation events).

Validation

  • uv run ruff format . / uv run ruff check .
  • uv run pytest -q — 1151 passed, 11 skipped
  • uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/judge_scoring.py packages/client/src/launchdarkly_ai_server/judges.py packages/client/src/launchdarkly_ai_server/__init__.py
  • Verified against the real ldclient that the judge-resolution context is valid (Context.from_dict({"kind": "evaluation", "key": ...}).valid is True)

Full uv run mypy . currently fails before checking due to duplicate test conftest module names across packages, which appears unrelated to this change.

@donei003
donei003 force-pushed the feature/ld-judges-phase3 branch from e7fffd7 to 614adca Compare September 2, 2026 20:38
@donei003

donei003 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the feedback in the draft:

  1. Added pydantic event DTOs in evaluations/events.py:
    • EvaluationEventPayload base DTO for shared fields (projectKey, evaluationRunId, datasetKey, etc.)
    • LDJudgeEvaluationEventPayload for LD Judge events
    • DeterministicScorerEvaluationEventPayload for local scorer events
      Event emission now serializes those DTOs via model_dump(by_alias=True, exclude_none=True).
  2. Removed user-supplied Judge.version; event version now only comes from the resolved LD variation metadata.
  3. Judge resolution now passes an empty context to extract_variation for this phase.

Validation after changes:

  • uv run ruff format .
  • uv run ruff check .
  • uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/__init__.py
  • uv run pytest -q — 1128 passed, 11 skipped

@donei003
donei003 force-pushed the feature/ld-judges-phase3 branch from 614adca to 6637d48 Compare September 2, 2026 21:14
@donei003

donei003 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Updated for the usage feedback:

  • Added a dedicated pydantic TokenUsage DTO with inputTokens / outputTokens aliases.
  • Moved usage off the shared base event DTO and onto LDJudgeEvaluationEventPayload only.
  • Deterministic scorer events now cannot carry usage through their DTO (extra="forbid") and the test asserts no usage field is emitted for scorers.

Validation:

  • uv run ruff format .
  • uv run ruff check .
  • uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/__init__.py
  • uv run pytest -q — 1128 passed, 11 skipped

@donei003
donei003 force-pushed the feature/ld-judges-phase3 branch from 6637d48 to 6104ae0 Compare September 2, 2026 21:56
@donei003

donei003 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Updated status from a Literal to an EvaluationStatus enum on the base event DTO. The pydantic config uses enum values when serializing the track payload, so emitted events still send "COMPLETE" / "ERROR".

Validation:

  • uv run ruff format .
  • uv run ruff check .
  • uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/__init__.py
  • uv run pytest -q — 1128 passed, 11 skipped

@donei003
donei003 force-pushed the feature/ld-judges-phase3 branch from 6104ae0 to 7ed3d05 Compare September 2, 2026 21:56
@donei003

donei003 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Updated kind to use an enum as well:

  • Added EvaluationEventKind enum with JUDGE / SCORER values.
  • Base event DTO now has kind: EvaluationEventKind.
  • Judge/scorer event DTOs set their default kind from the enum.
  • Serialization still emits "judge" / "scorer" via use_enum_values=True.

Validation:

  • uv run ruff format .
  • uv run ruff check .
  • uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/__init__.py
  • uv run pytest -q — 1128 passed, 11 skipped

@donei003
donei003 force-pushed the feature/ld-judges-phase3 branch from 7ed3d05 to ccfdc0a Compare September 2, 2026 22:15
@donei003

donei003 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Updated based on the latest feedback:

  • Moved extract_variation to a top-level import in evaluations/runner.py.
  • Simplified _resolve_judges now that we pass an empty context; it no longer accepts unused project_key / run_key args.
  • Removed both rowCount and expectedCriteriaCount from the evaluation run creation payload. The run payload is now just source + datasetId.
  • Updated tests accordingly.

Validation:

  • uv run ruff format .
  • uv run ruff check .
  • uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/__init__.py
  • uv run pytest -q — 1128 passed, 11 skipped

donei003 and others added 4 commits September 2, 2026 16:45
@donei003 @claude
refactor(evaluations): criteria naming + shared judge scoring contract
f63cb3c 
Rename the public run() parameter judges= to criteria= (with
JudgeReference -> Criterion and evaluations/judges.py -> criteria.py):
the wire format already calls these criteria, and a Scorer is not a
judge. Scorer callbacks now receive the public DatasetRow instead of
the internal result dict, so internal key renames cannot break
customer scorers; Criterion and DatasetRow are exported.

Extract the judge response contract (formatting instructions, JSON
score parsing, finite-number guard) into judge_scoring.py shared by
the online judge path and the offline evaluations runner. The two
copies had already drifted textually.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@donei003 @claude
fix(evaluations): correct judge resolution, score validation, and ren…
1eb1976 
…dering

- Resolve LD judges with a valid {kind: evaluation, key: project} context.
  The previous empty context is invalid to the real LD SDK, so every
  resolution returned the None default and failed as 'not found'. The
  resolution error now also carries the underlying cause instead of
  always claiming the judge does not exist.
- Validate judge scores when the judge responds: non-JSON output,
  non-numeric, non-finite, and out-of-range scores become per-criterion
  ERROR events with cause codes (invalid_judge_output, invalid_score,
  handler_raised, generation_incomplete, scorer_raised) instead of
  crashing the run at event-build time after all LLM spend.
- Pass judge configs to the handler unrendered. The handler owns the
  single template pass, so {{...}} sequences inside generated output or
  dataset values can no longer be expanded into the judge prompt.
  Absent judge variables render as empty strings rather than leaving
  literal mustache in the prompt.
- Reject duplicate criterion identities (judge keys / scorer names)
  before any records are created; they would share event identity.
- Flush queued generation events in a finally so they reach LaunchDarkly
  even when the criteria phase fails.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@donei003 @claude
refactor(evaluations): drop pydantic, run criteria concurrently
46bfa00 
Rewrite the evaluation event payloads as frozen dataclasses with an
explicit to_track_payload(), matching the Usage/RunSummary wire pattern
used everywhere else in the SDK, and remove the pydantic>=2 dependency.
The models validated the SDK's own dicts, and a validation failure
surfaced as a run-aborting crash at emission time; payload construction
is now also wrapped per result so one bad criterion result is logged
and skipped instead of dropping the whole batch. ERROR events carry a
top-level errorMessage for parity with generation events.

Run (row x criterion) pairs through the same ConcurrencyController and
concurrency parameter the generation phase uses, instead of one
criterion at a time: a 200-row dataset with 3 judges was 600 serial
LLM calls.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@donei003 @claude
polish(evaluations): event logging, docstrings, judge scoring tests
da69d33 
Emit per-event telemetry lines through the module logger instead of
printing to the host application's stdout, refresh the run() docstring
(no longer generation-only), and cover the shared judge response parser
with unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@donei003