feat(eval): ab-test config-bundle run (create)

[jariy17]

Command structure

agentcore eval ab-test config-bundle run
  -n, --name <name>              required   A/B test name
  -g, --gateway <id>             required   deployed gateway id
  --control <json>               required   {"config-bundle","bundle-version"}  (inline · file:// · -)
  --treatment <json>             required   {"config-bundle","bundle-version"}  (inline · file:// · -)
  --online-eval <id>             required   online-evaluation config id
  --treatment-weight <1-99>                 control weight = 100 - this (default 50)
  --gateway-filter <json>                   GatewayFilter {"targetPaths":[...]}  (inline · file:// · -)
  --role-arn <arn>                          execution-role override (else auto-provisioned)
  --enable-on-create <true|false>           start immediately (default true; false = create PAUSED)

Starts an A/B test that splits gateway traffic between two config-bundle versions on one runtime. First create command in the ab-test family (get/list/pause/resume/stop/delete already merged in #2102).

Actual input → output

Input (run against the exploratory account, us-west-2):

agentcore eval ab-test config-bundle run \
  --name orders-prompt-v2-rollout \
  --gateway orders-gateway-demo-000 \
  --control '{"config-bundle":"orders-agent-prompt-abc","bundle-version":"1111...1111"}' \
  --treatment '{"config-bundle":"orders-agent-prompt-abc","bundle-version":"2222...2222"}' \
  --online-eval online-eval-abc123 \
  --treatment-weight 20 \
  --gateway-filter '{"targetPaths":["/orders/checkout"]}'

Actual output — the CLI resolves + validates the gateway before any create, so an unreachable gateway fails fast and creates nothing:

Error: User: arn:aws:sts::…:assumed-role/Admin/… is not authorized to perform: bedrock-agentcore:GetGateway

Output on success (shape returned by CreateABTest — a real create was intentionally not run: it provisions IAM + routes live traffic on a shared account):

{
  "abTestId": "orders-prompt-v2-rollout-abc123",
  "abTestArn": "arn:aws:bedrock-agentcore:us-west-2:…:ab-test/orders-prompt-v2-rollout-abc123",
  "name": "orders-prompt-v2-rollout",
  "status": "CREATING",
  "executionStatus": "RUNNING",
  "createdAt": "2026-08-26T15:00:00.000Z"
}

With --enable-on-create false, executionStatus is NOT_STARTED (created PAUSED, no traffic).

How it works

• One SDK call — CreateABTestCommand on the data-plane client. Builds C/T1 variants (weights 100 - w / w), evaluationConfig = { onlineEvaluationConfigArn }, and passes --gateway-filter straight through as the SDK GatewayFilter.
• --control / --treatment / --gateway-filter each accept inline JSON, file://<path>, or - (stdin) via SourceResolver — same pattern as online-eval create's --filters.
• Execution role — required by the API. When --role-arn is omitted, Core provisions one (abTestExecutionRole.ts: confused-deputy-scoped trust + gateway-rule/get inline policy), retries CreateABTest while IAM propagates (data-plane 403 too), and rolls the role back if create fails.
• Validation is server-side — CreateABTest validates gateway READY, bundle versions, and the online-eval (enabled iff starting) by assuming the role. The CLI resolves ids → ARNs and surfaces the service's 4xx; no duplicated client-side state machine.
• --runtime dropped — not a CreateABTest field (deviates from the doc on purpose).

Tests

• ab-test.create.test.tsx — flag→request mapping, --gateway-filter passthrough, --enable-on-create/--role-arn, identical-variant rejection, weight bounds, required-flag matrix, malformed/mis-shaped JSON.
• abTestExecutionRole.test.ts — 64-char name cap, trust + inline policy bodies, create-vs-reuse.

Bug bash — 4 agents + live exploratory (us-west-2)

• Standard / Munger / AI-slop (3/10) / live: all green. Live bash ran 8 non-destructive cases — every path errored before any create; no state mutated.
• Fixed from review (commit 2): identical-variant guard now compares the (bundle, version) pair (was version-only); CreateABTest retries on data-plane AccessDenied; accountId extraction throws instead of a silent "*"; --treatment-weight must be an integer; added the role-module unit tests.
• bun test → 723 pass / 0 fail · typecheck → 0 errors · oxlint → clean.

Follow-ups (separate PRs)

• Structured --json error output (errors print plain text to stderr — cross-cutting, all commands).
• TUI wizard for config-bundle run.
• stop's gateway-rule promotion suggestion (still a TODO).

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Looks good

Nice, tightly-scoped PR. A few observations, none of which I think need to block merging:

• retryWhileRoleUnassumable in src/core/eval.tsx retries on any AccessDeniedException / HTTP 403. Because callers using their own --role-arn bypass this path (the ternary in createConfigBundleABTest), this is only reached for auto-provisioned roles — where the retry is well justified. It does mean a genuinely misconfigured auto-provisioned role (or a caller missing bedrock-agentcore:CreateABTest — though that would be surprising to reach this call at all) will add up to ~15s before surfacing the real error. Fine as-is, worth revisiting if the retry ever gets reused elsewhere.
• provisionAbTestRole unconditionally calls PutRolePolicy even when the role already existed, which overwrites any prior inline policy under ABTestExecutionPolicy. Since we own the role name (deterministic, AgentCoreABTest-…), this is intentional and correct — just noting so the invariant is explicit.
• On failure cleanup, deleteAbTestRole only removes the single inline policy we attached and then deletes the role, and only runs when provisioned.created is true. That is the safe subset. If a future change ever attaches managed policies here, the cleanup will need to grow to match.
• abTestExecutionRoleName uses the raw testName in the base. IAM role names are constrained to [\w+=,.@-]; if AgentCore ever accepts A/B test names outside that set, CreateRole will fail with an IAM-shaped error rather than something the user can act on. Probably a non-issue given the service's own name validation, but worth keeping in mind.

Tests correctly mock only at the AWS SDK client boundary (IAMClient.send) and at the Core interface, which matches the guidance. No handler-level telemetry is needed here — telemetry is centralized in this repo.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.16413% with 85 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.08%. Comparing base (097e1f0) to head (60e8758).
⚠️ Report is 4 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	10.12%	71 Missing ⚠️
src/core/abTestExecutionRole.tsx	88.97%	14 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2113      +/-   ##
============================================
- Coverage     97.38%   97.08%   -0.30%     
============================================
  Files           440      443       +3     
  Lines         26626    26955     +329     
============================================
+ Hits          25929    26170     +241     
- Misses          697      785      +88     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)