Skip to content

fix: make harness PR reviewer evaluate against the PR base branch - #2106

Merged
jariy17 merged 2 commits into
refactorfrom
fix/harness-review-base-branch
Aug 25, 2026
Merged

fix: make harness PR reviewer evaluate against the PR base branch#2106
jariy17 merged 2 commits into
refactorfrom
fix/harness-review-base-branch

Conversation

@notgitika

Copy link
Copy Markdown
Contributor

Problem

The AI PR reviewer (agentcore-devx-automation "AgentCore Harness Review") reviews PRs against the default branch (main) rather than the PR's actual base branch. Its local context clones (/opt/workspace/agentcore-cli) sit on main, so PRs targeting refactor are analyzed against the main source layout.

This produces confidently-wrong findings: e.g. on #2105 it reported a "split-brain read/write" regression citing src/cli/commands/deploy/actions.ts, src/lib/schemas/io/path-resolver.ts, and src/cli/operations/init/files.ts — none of which exist on refactor (which uses src/handlers/ and src/core/).

Fix

The actual checkout logic lives in the reusable workflow in aws/agentcore-devx-devtools (not editable here), so this fixes it at the lever we control — the prompts referenced by pr-automation.yml:

  • review.md: instruct the reviewer to determine the PR's base.ref and git fetch/git checkout it in the local clone before reading files for context, and to not raise findings premised on a file/path being absent without verifying against that base branch.
  • system.md: document that agentcore-cli has a long-lived refactor branch that diverges from main, and that PRs must be analyzed against their own base branch.

Notes

  • pull_request_target reads the workflow + prompts from the base branch, so this must land on refactor to fix refactor-targeted PRs. A parallel change on main keeps parity for main-targeted PRs.
  • A more robust fix (checking out the base branch in the runner) belongs in aws/agentcore-devx-devtools; this prompt-level mitigation is the in-repo stopgap.

The AI PR reviewer's local context clones sit on the default branch
(main), so PRs targeting the refactor branch were reviewed against the
main source layout — producing findings about files and code paths that
do not exist on refactor (e.g. src/cli / src/lib vs src/handlers /
src/core).

Instruct the reviewer, in both the system and review prompts, to sync the
local clone to the PR's base branch before reading files for context, and
to not raise findings premised on a file or path being absent without
verifying against that base branch.
@github-actions github-actions Bot added the size/s PR size: S label Aug 25, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Aug 25, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 25, 2026

@agentcore-devx-automation agentcore-devx-automation Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AgentCore Harness Review

Verdict: Changes requested

Nice, well-scoped fix — the base-branch-vs-main mismatch is a real problem that will bite refactor-targeted PRs, and the layout facts in both prompts (src/handlers/+src/core/ on refactor vs. src/cli/+src/lib/ on main) match what's actually on those branches. One concrete concern before merging:

The example curl uses $CLONE_TOKEN, which isn't present at container runtime

In .github/harness/prompts/review.md (the new block, roughly lines 11–30) the recipe is:

curl -sH "Authorization: Bearer $CLONE_TOKEN" https://api.github.com/repos/aws/agentcore-cli/pulls/<number> | jq -r .base.ref

But per .github/harness/Dockerfile and .github/harness/README.md, CLONE_TOKEN is a build-time ARG used only to configure git config --global url."https://${CLONE_TOKEN}@github.com/".insteadOf .... The README explicitly states: "The token is never sent to the Harness runtime or persisted in this image." So at review time $CLONE_TOKEN will be empty, and curl -H "Authorization: Bearer " against api.github.com returns HTTP 401 (verified). jq -r .base.ref then emits null, and the follow-up git checkout null will fail — dropping the agent back to main, i.e. the exact failure mode this PR is trying to prevent.

Options:

  1. Drop the auth header entirely. aws/agentcore-cli is public, so an unauthenticated call to /repos/aws/agentcore-cli/pulls/<n> succeeds and returns .base.ref. Simplest fix and matches how the review agent already fetches PR metadata today.
  2. Rely on git ls-remote / a plain git fetch --all plus resolving the base ref from the diff/PR context the harness already passes in, rather than a fresh REST call.
  3. If you actually want an authenticated call, arrange for the reusable workflow (aws/agentcore-devx-devtools/.github/workflows/reusable-pr-ai-review.yml) to inject a token env var into the runtime container, and reference that variable here — and update .github/harness/README.md to match.

Everything else (the system.md blurb, the "don't raise findings whose premise is a missing file until you've confirmed the base branch" guardrail, the file paths modified) looks good.

@agentcore-devx-automation agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 25, 2026
@codecov-commenter

codecov-commenter commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.42%. Comparing base (1d96d1f) to head (cf80f92).

Additional details and impacted files
@@            Coverage Diff            @@
##           refactor    #2106   +/-   ##
=========================================
  Coverage     97.42%   97.42%           
=========================================
  Files           429      429           
  Lines         26286    26286           
=========================================
  Hits          25609    25609           
  Misses          677      677           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

CLONE_TOKEN is a build-time ARG (git config only), not present at
container runtime, so the Authorization header would send an empty bearer
and 401 — yielding a null base ref and a failed checkout back to main.
aws/agentcore-cli is public, so drop the header.
@github-actions github-actions Bot added size/s PR size: S and removed size/s PR size: S labels Aug 25, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 25, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@jariy17
jariy17 merged commit 0c79e59 into refactor Aug 25, 2026
20 of 21 checks passed
@jariy17
jariy17 deleted the fix/harness-review-base-branch branch August 25, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/s PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants