feat: support imperative eval recommendation command by nborges-aws · Pull Request #2111 · aws/agentcore-cli

nborges-aws · 2026-08-26T15:05:17Z

Description

Adds imperative command-line support for AgentCore evaluation recommendations:

eval recommendation start
eval recommendation get
eval recommendation list
eval recommendation delete

Summary of Changes

Supports SYSTEM_PROMPT_RECOMMENDATION and TOOL_DESCRIPTION_RECOMMENDATION recommendation types
start accepts recommendation configuration as inline JSON, a file:// path, or stdin (-) through SourceResolver class
Supports the service API fields for description, KMS key ARN, and tags
Adds the recommendation operations to EvalClient, the shared handler types, and TestCoreClient
Adds recorded golden fixtures covering the complete start, get, list, and delete flow

Type of Change

Testing

How have you tested the change?

Added unit tests and golden fixture tests exercising recommendation logic.

bun run test (2052 pass, 0 fail)
I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

jariy17 · 2026-08-26T15:36:43Z

+    description: "get a recommendation by id",
+    flags: [flag("id", "the ID of the recommendation", z.string().optional())],
+    handle: async (ctx, flags) => {
+      if (!flags["id"]) throw new InputValidationError("required option '--id <id>' not specified");


we can definitely abstracted this so we have a common error message for this type of thing. OBO

yeah I think marking flags as required and supporting it directly in the framework would be ideal, but likely OOS here.

jariy17 · 2026-08-26T15:53:35Z

  }

+  async startRecommendation(
+    request: StartRecommendationRequest,


We create our own types here because handlers define the interface for the core client. Even if it's a one-to-one mapping, we should still define our own type.

sure will update

jariy17 · 2026-08-26T15:58:42Z

@@ -0,0 +1,119 @@
+import { describe, expect, test } from "bun:test";


We don't need this. Use Golden tests please

are the handler tests sufficient? I think these are testing the same things at a lower level.

jariy17 · 2026-08-26T16:00:12Z

+      flag("tags", "tags as key=value (repeatable) or JSON object", z.array(z.string()).optional()),
+    ],
+    handle: async (ctx, flags) => {
+      if (!flags["name"]) {


const requiredFlags: Array<[string, string]> = [ ["name", "--name <name>"], ["type", "--type <type>"], ["recommendation-config", "--recommendation-config <recommendation-config>"], ]; for (const [key, usage] of requiredFlags) { if (!flags[key]) { throw new InputValidationError(`required option '${usage}' not specified`); } }

nit: we should use something like this.

Sure will add something like this

jariy17 · 2026-08-26T16:01:45Z

+  return call.args;
+}
+
+describe("eval recommendation command hierarchy", () => {


do we need these tests?

+1, I've seen these tests in other places, but they feel pretty low value. The tests on the handlers would fail if the hierarchy is incorrect. Also, what coverage do we get on the tests below that we don't get on the fixture based tests?

I added these tests to cover behavior the fixtures don't exercise. E.g. malformed json, exercising source resolver, etc. But consensus is against unit test so I've removed this + the other one entirely

Hweinstock · 2026-08-26T18:23:36Z

+    description: "get a recommendation by id",
+    flags: [flag("id", "the ID of the recommendation", z.string().optional())],
+    handle: async (ctx, flags) => {
+      if (!flags["id"]) throw new InputValidationError("required option '--id <id>' not specified");


yeah I think marking flags as required and supporting it directly in the framework would be ideal, but likely OOS here.

Hweinstock · 2026-08-26T18:26:23Z

+  return (error as Error).name === "ResourceNotFoundException";
+}
+
+async function waitForTerminal(


is there a way to leverage

agentcore-cli/src/testing/timing.tsx

Line 16 in 01c9317

export async function waitFor(

for these?

Good call out I'll switch to use this util

Hweinstock · 2026-08-26T18:28:46Z

+  return call.args;
+}
+
+describe("eval recommendation command hierarchy", () => {


+1, I've seen these tests in other places, but they feel pretty low value. The tests on the handlers would fail if the hierarchy is incorrect. Also, what coverage do we get on the tests below that we don't get on the fixture based tests?

Hweinstock · 2026-08-26T18:29:29Z

+      status: "PENDING",
+      createdAt: new Date("2026-08-26T12:00:00.000Z"),
+      updatedAt: new Date("2026-08-26T12:00:00.000Z"),
+    } as StartRecommendationResponse;


why do we need to cast here?

N/A since test file was removed altogether

Hweinstock · 2026-08-26T18:31:32Z

@@ -0,0 +1,119 @@
+import { describe, expect, test } from "bun:test";


are the handler tests sufficient? I think these are testing the same things at a lower level.

…dling

agentcore-devx-automation · 2026-08-27T14:16:38Z

Claude Security Review: no high-confidence findings. (run)

codecov-commenter · 2026-08-27T14:18:09Z

Codecov Report

❌ Patch coverage is 97.70115% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.41%. Comparing base (a1f2f32) to head (2783292).

Files with missing lines	Patch %	Lines
src/handlers/eval/recommendation/start/index.tsx	94.20%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           refactor    #2111    +/-   ##
==========================================
  Coverage     97.41%   97.41%            
==========================================
  Files           453      458     +5     
  Lines         27637    27810   +173     
==========================================
+ Hits          26922    27091   +169     
- Misses          715      719     +4

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jariy17 reviewed Aug 26, 2026

View reviewed changes

Hweinstock reviewed Aug 26, 2026

View reviewed changes

nborges-aws added 3 commits August 27, 2026 10:14

feat: support imperative eval recommendation command

40d9d04

chore: leverage shared utils + delete stale tests

f5dcf4e

fix: create recommendation input interface + update required flag han…

2783292

…dling

nborges-aws force-pushed the eval-recommendation branch from 295482e to 2783292 Compare August 27, 2026 14:14

github-actions Bot added the size/l PR size: L label Aug 27, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 27, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 27, 2026

jariy17 approved these changes Aug 27, 2026

View reviewed changes

jariy17 merged commit c105d4f into refactor Aug 27, 2026
22 checks passed

jariy17 deleted the eval-recommendation branch August 27, 2026 16:29

		@@ -0,0 +1,119 @@
		import { describe, expect, test } from "bun:test";

Uh oh!

Conversation

nborges-aws commented Aug 26, 2026

Description

Type of Change

Testing

Checklist

Uh oh!

jariy17 Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agentcore-devx-automation Bot commented Aug 27, 2026

Uh oh!

codecov-commenter commented Aug 27, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jariy17 Aug 26, 2026 •

edited

Loading