feat(eval): batch simulate — --ingestion-wait-ms + per-example failures/sessions by jariy17 · Pull Request #2098 · aws/agentcore-cli

jariy17 · 2026-08-25T18:11:58Z

What

Batch simulate review follow-ups (base: refactor):

--ingestion-wait-ms flag (default 180000, 0 skips) → InvokeDatasetInput.waitIngestionMs. Drops the SIMULATE_INGESTION_WAIT_MS env var — value flows through the input only.
Per-example failures. runExamples returns failures: {item,error}[]; invokeDataset surfaces failures: [{exampleId, error}] — names which examples dropped + why, not just a count.
Output now renders sessions[] + failures[] (below).
Uses Golden Test Pattern for simulate instead of dedicated unit tests for invokeDataset

Output

{
  "batchEvaluationId": "batch-eval-test", "status": "RUNNING",
  "examplesInvoked": 1, "examplesFailed": 1,
  "sessions": [ { "exampleId": "ok1", "sessionId": "s1" } ],
  "failures": [ { "exampleId": "bad", "error": "HTTP 500" } ]
}

sessions[] — exampleId ↔ sessionId. Join key: a later eval batch-evaluation get returns results[] keyed by sessionId, mapping each score back to its dataset row.
failures[] — always present ([] on a clean run); names which examples dropped + why.

…ailures/sessions Follow-ups from the batch-evaluation simulate review: - Add --ingestion-wait-ms (default 180000, 0 skips); thread via InvokeDatasetInput.waitIngestionMs. Removes the SIMULATE_INGESTION_WAIT_MS env var — tests pass the value through the input. - runExamples now returns per-item failures (item + error), not a bare count + firstError; invokeDataset surfaces failures: [{ exampleId, error }] so a partial failure names which examples dropped and why. - batch simulate output renders sessions[] (exampleId <-> sessionId join key for a later get) and failures[] (omitted when empty).

agentcore-devx-automation · 2026-08-25T18:14:14Z

Claude Security Review: no high-confidence findings. (run)

codecov-commenter · 2026-08-25T18:15:01Z

Codecov Report

❌ Patch coverage is 92.85714% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.23%. Comparing base (343d213) to head (ebc604e).
⚠️ Report is 12 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	86.36%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2098      +/-   ##
============================================
- Coverage     97.33%   97.23%   -0.11%     
============================================
  Files           417      417              
  Lines         25250    25269      +19     
============================================
- Hits          24578    24570       -8     
- Misses          672      699      +27

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-08-25T20:21:11Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-25T20:24:19Z

Claude Security Review: no high-confidence findings. (run)

- inject newSessionId into EvalClient (default randomUUID) so replay fixtures + goldens are deterministic - teach makeRecordingSend to freeze/revive a streaming SDK response (InvokeAgentRuntime), which stringify couldn't serialize - add simulate fixture-golden case; move handler edges to batch-evaluation.test.tsx - split invokeDataset.test.ts into run.test.ts (pool) + load.test.ts (parse + GT-shape); delete it and simulate.test.tsx

agentcore-devx-automation · 2026-08-26T22:11:50Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-26T22:29:16Z

Claude Security Review: no high-confidence findings. (run)

…dant expects)

agentcore-devx-automation · 2026-08-26T22:34:59Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-26T22:38:06Z

Claude Security Review: no high-confidence findings. (run)

Hweinstock · 2026-08-27T01:08:04Z

        );
        if (r.invoked === 0) {
-          const detail = r.firstError ? `; first error: ${r.firstError.message}` : "";
+          const first = r.failures[0];


would it make sense to just join all the errors? Or are these already communicated to the customer somewhere else?

Just wanted to make sure they'd be aware of individual datasets failing even if the overall command doesn't fail.

Yes all the errors would show in the debug logs. I assumed that if all dataset examples have failed, it would be the same error. However, maybe we should convene these errors directly to the tui or cli. I'll look into this in a follow up

Hweinstock · 2026-08-27T01:19:14Z


+const STREAM_TAG = "$stream";
+
+async function freezeStream(response: unknown): Promise<unknown> {


took me a sec, but this makes sense and feels like a simple solution.

correct me if I misunderstood, but for streaming apis the response is not json serializable, so we convert the response via the stream tag. In the process, we resolve the whole stream, so we need to convert the data back to a stream before passing it downstream to consumers who are expecting a stream.

nice!

Yep ur correct. The reviveStream name seems a little weird to me, maybe Ill use mimicStream.

nborges-aws

PR overall LGTM. One concern with a gap in testing. Tests are all either exercise happy path, or mock invokeDataset with a mocked failure object in the response. Nowhere is exercising actual invocation failures assemble that failure object correctly.

The deleted invokeDataset test is actually the only place that exercised this.

notgitika

Left a comment, could be followup

notgitika · 2026-08-27T07:47:05Z

+  if (typeof stream?.transformToString !== "function") return response;
+  return {
+    ...(response as Record<string, unknown>),
+    response: { [STREAM_TAG]: await stream.transformToString() },


Could we serialize transformToByteArray() as base64 instead? transformToString() decodes as UTF-8, so binary or invalid UTF-8 runtime responses are changed during replay.

can do in a follow up.

jariy17 · 2026-08-27T16:58:22Z

I'll address the missing bad paths in a follow up

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Aug 25, 2026

agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress and removed agentcore-harness-reviewing AgentCore Harness review in progress labels Aug 25, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 25, 2026

chore: drop explanatory comments from batch simulate follow-up

2b80a3c

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Aug 25, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 25, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 25, 2026

fix: always render failures[] in batch simulate output

e5976aa

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Aug 25, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 25, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 25, 2026

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Aug 26, 2026

github-actions Bot added size/l PR size: L and removed size/m PR size: M labels Aug 26, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 26, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 26, 2026

test(eval): drop run.test.ts + load.test.ts

c1e0c9f

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 26, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 26, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 26, 2026

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 26, 2026

chore: drop explanatory comments from the simulate test refactor

6a5cc69

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 26, 2026

test(eval): simulate fixture asserts via matchGolden only (drop redun…

ebc604e

…dant expects)

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 26, 2026

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 26, 2026

jariy17 marked this pull request as ready for review August 26, 2026 22:34

agentcore-devx-automation Bot added claude-security-reviewing Claude Code /security-review in progress and removed claude-security-reviewing Claude Code /security-review in progress labels Aug 26, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 26, 2026

Hweinstock approved these changes Aug 27, 2026

View reviewed changes

nborges-aws reviewed Aug 27, 2026

View reviewed changes

notgitika approved these changes Aug 27, 2026

View reviewed changes

jariy17 merged commit b78d396 into refactor Aug 27, 2026
26 checks passed

jariy17 deleted the feat/eval-simulate-followups branch August 27, 2026 16:57


		const STREAM_TAG = "$stream";

		async function freezeStream(response: unknown): Promise<unknown> {

Uh oh!

Conversation

jariy17 commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Output

Uh oh!

agentcore-devx-automation Bot commented Aug 25, 2026

Uh oh!

codecov-commenter commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

agentcore-devx-automation Bot commented Aug 25, 2026

Uh oh!

agentcore-devx-automation Bot commented Aug 25, 2026

Uh oh!

agentcore-devx-automation Bot commented Aug 26, 2026

Uh oh!

agentcore-devx-automation Bot commented Aug 26, 2026

Uh oh!

agentcore-devx-automation Bot commented Aug 26, 2026

Uh oh!

agentcore-devx-automation Bot commented Aug 26, 2026

Uh oh!

Hweinstock Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

jariy17 Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Hweinstock Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

jariy17 Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

nborges-aws left a comment

Choose a reason for hiding this comment

Uh oh!

notgitika left a comment

Choose a reason for hiding this comment

Uh oh!

notgitika Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

jariy17 Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jariy17 commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jariy17 commented Aug 25, 2026 •

edited

Loading

codecov-commenter commented Aug 25, 2026 •

edited

Loading