feat(project): add project add evaluator llm-as-a-judge

[notgitika]

What

Adds agentcore project add evaluator llm-as-a-judge. this attaches a custom LLM-as-a-Judge evaluator to a project. The judge is another LLM prompted with scoring instructions and a rating scale, written into spec.evaluators in agentcore.json. The existing CDK constructs render it as an AWS::BedrockAgentCore::Evaluator, so project deploy provisions it.

Command

agentcore project add evaluator llm-as-a-judge \
  --name <name> \
  --level <SESSION|TRACE|TOOL_CALL> \
  --model <bedrock-id|inference-profile-arn> \
  --instructions <text | file://path | ->  \
  --rating-scale <1-5-quality|1-3-simple|pass-fail|good-neutral-bad> | --rating-scale-file file://scale.json \
  [--description <text>] [--kms-key-arn <arn>] [--tags '{"k":"v"}']

Details

• New evaluator subrouter under project add with an llm-as-a-judge subcommand (mirrors the credentials subrouter pattern). Wires a new evaluator resource type through AddResourceInput and FsProjectManager.addResource / toProjectSpecKey.
• Rating scales: four presets (1-5-quality, 1-3-simple, pass-fail, good-neutral-bad) that expand into the schema's numerical/categorical shapes with judge-facing definitions, plus --rating-scale-file for a fully custom scale.
• Instructions: accept inline text, file://<path>, or - (stdin) via the shared SourceResolver.
• Placeholder validation: instructions must embed at least one level-appropriate placeholder (e.g. {context} for SESSION) and use no placeholder outside that level's set. This mirrors AgentCoreEvaluationControlPlaneService's InstructionValidator, so invalid instructions are rejected locally at add/build instead of surfacing as an opaque CloudFormation rollback at deploy time.

Testing

• Unit tests for the handler (presets, custom scale file, file:// instructions, description/KMS/tags, duplicate name, invalid spec, and all validation error paths) and for the schema-level placeholder validation.
• Verified end-to-end: project add evaluator llm-as-a-judge → project deploy provisions a real AWS::BedrockAgentCore::Evaluator (CREATE_COMPLETE) in a dev account. Test resources were cleaned up afterward.

Scope

Spec-write + deploy of the evaluator resource. Consistent with sibling add commands.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.36066% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.43%. Comparing base (f1a651c) to head (e14f409).
⚠️ Report is 7 commits behind head on refactor.

Files with missing lines	Patch %	Lines
...lers/project/add/evaluator/llm-as-a-judge/index.ts	96.36%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           refactor    #2124    +/-   ##
==========================================
  Coverage     97.42%   97.43%            
==========================================
  Files           429      432     +3     
  Lines         26314    26550   +236     
==========================================
+ Hits          25637    25869   +232     
- Misses          677      681     +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Looks good

Nice PR — well-structured and consistent with sibling add subcommands. Highlights:

• Handler is a thin flag-parsing layer; EvaluatorSchema.safeParse is the single source of truth for validation.
• Tests use real temp directories and drive the CLI end-to-end via createRootHandler (no fs mocks) — mocking is limited to TestCoreClient / TestGlobalConfigAccessor at true I/O boundaries. test.each covers all validation branches.
• Placeholder validator in src/projectSchemas/evaluator.ts fails fast locally with a clear message instead of surfacing as a CloudFormation rollback at deploy time. structuredClone on presets in resolvePreset guards against downstream mutation of the shared table.
• Wiring through AddResourceInput, FsProjectManager.addResource, and toProjectSpecKey is correct and matches the existing pattern.

A couple of very minor observations, not blockers:

• LEVEL_ALLOWED_PLACEHOLDERS duplicates a list that lives server-side in InstructionValidator; if the service ever expands the allowed set (e.g. lifts the account-feature gate on skill placeholders for non-TOOL_CALL levels), this table will silently reject valid instructions. A short comment pointing at the service source is already there; consider a follow-up to keep them in sync (or accept unknown placeholders with a warning rather than a hard error).
• findInstructionPlaceholders trims whitespace inside { ... }, so { context } passes local validation but is sent to Bedrock verbatim. If the service does exact matching, the local check would be a false positive. Cheap fix if it matters: don't trim, or normalize the stored instructions.

Neither of these needs to be addressed before merging.