stack-bench

A stacked pull-request benchmark for coding agents

Which stacked PR workflow should you give a coding agent?

Codex and Claude Code run the same live GitHub tasks with gh stack plus two pinned versions of its official skill, and GitButler plus its official skill.

Results · Scenarios · Method · Provenance

2 workflows · 3 pinned configurations · 2 agents · 3 scenarios · k=6 · 108 trials · 2026-08-05 · raw evidence

Important

Codex was correct in 53/54 runs; Claude in 48/54. Every publish and amend run was correct. GitButler was correct in 36/36 and was fastest with the fewest VC commands in every agent–scenario cell. Recovery after merge remains the discriminator.

The code changes already exist before timing starts. The agent must produce the correct commits, stack topology, remote branches, and pull-request bases. A deterministic grader—not an LLM judge—decides whether the result is correct.

Results matrix

Each cell is correct runs · median wall time · median agent-issued VC commands · median command-output KiB.

Scenario	Codex gpt-5.6-sol			Claude Code claude-opus-4-8
	gh stack skill 0.0.9	gh stack skill 0.1.0	GitButler latest source build + skill	gh stack skill 0.0.9	gh stack skill 0.1.0	GitButler latest source build + skill
Publish existing stack	✅ 6/6 69.5s · 12 cmds · 4.1 KiB	✅ 6/6 58.1s · 9 cmds · 3.4 KiB	✅ 6/6 40.3s · 4 cmds · 2.1 KiB	✅ 6/6 60.3s · 17 cmds · 2.8 KiB	✅ 6/6 62.0s · 18 cmds · 4.0 KiB	✅ 6/6 40.3s · 6 cmds · 3.4 KiB
Amend bottom layer	✅ 6/6 66.1s · 19 cmds · 6.0 KiB	✅ 6/6 58.5s · 19.5 cmds · 6.7 KiB	✅ 6/6 44.2s · 6.5 cmds · 4.0 KiB	✅ 6/6 97.4s · 29 cmds · 6.1 KiB	✅ 6/6 64.8s · 17 cmds · 4.9 KiB	✅ 6/6 44.9s · 3.5 cmds · 1.8 KiB
Recover after merge	⚠️ 5/6 108.6s · 21.5 cmds · 8.0 KiB	✅ 6/6 123.8s · 25 cmds · 9.9 KiB	✅ 6/6 70.3s · 8 cmds · 4.1 KiB	⚠️ 3/6 123.7s · 32.5 cmds · 10.0 KiB	⚠️ 3/6 143.5s · 32.5 cmds · 9.3 KiB	✅ 6/6 84.3s · 13 cmds · 4.6 KiB

_{✅ all runs were correct · ⚠️ at least one run was incorrect · a wrong stack fails regardless of speed}

All scenarios

Overall values below are medians across all 18 runs for that agent and workflow, including incorrect runs.

Agent	Configuration	Correct	Time	VC commands	KiB out
Codex	gh stack + skill 0.0.9	17/18	73.2s	18.5	6.0
Codex	gh stack + skill 0.1.0	18/18	62.0s	19.5	6.7
Codex	GitButler latest source + skill	18/18	45.3s	7	4.1
Claude Code	gh stack + skill 0.0.9	15/18	93.9s	22.5	6.1
Claude Code	gh stack + skill 0.1.0	15/18	74.2s	19	5.1
Claude Code	GitButler latest source + skill	18/18	55.1s	6	3.4

Codex k=6 evidence · Claude k=6 evidence · historical k=3 results

What the matrix says

• Publish and amend were solved across the board: 72/72 runs were correct. All correctness differences came from recovery after the bottom PR landed.
• The updated gh-stack skill helped Codex, not Claude. Codex recovery improved from 5/6 to 6/6; Claude was 3/6 with either skill.
• GitButler was fastest and used the fewest VC commands in all 6/6 agent–scenario comparisons. It produced the least median output in five of six cells; old gh-stack was lower for Claude's publish task.
• GitButler was correct in all 36/36 runs.
• Codex was more reliable overall: 53/54 correct versus Claude's 48/54 on the identical schedule.

Across both agents, GitButler was correct in 36/36 runs (Wilson 95% interval: 90–100%), the updated gh-stack skill in 33/36 (78–97%), and the old skill in 32/36 (75–96%). With k=6 per cell, treat this as strong repeated evidence for these agents and tasks—not a universal ranking.

What failed?

All seven incorrect results were in recovery and shared one pattern:

• Stale PR base chain: old gh-stack was incorrect in one Codex run and three Claude runs; the updated skill was incorrect in three Claude runs. Local history, content, tests, remote heads, and PR identities were correct, but the greeting PR still targeted the merged profile branch.

The exact incorrect rows and raw evidence are linked from the Codex summary and Claude summary.

VC commands are PATH-traced agent-issued git, but, gh stack, and gh pr invocations. Other gh subcommands, tool-internal subprocesses, and Claude Code startup probes are excluded. KiB is event-level output from shell events containing those commands and is best compared within one agent.

Scenarios

Each scenario is a prepared repository plus a plain-English instruction. No product code is invented during a trial.

1. Publish an existing stack

_{three existing local branches → three draft PRs}

Three clean, one-commit branches already form a local stack. Publish them without rewriting the prepared commits.

trunk
└── profile       PR → trunk
    └── greeting  PR → profile
        └── cli   PR → greeting

The crux: preserve commit identity and layer boundaries while creating the complete remote PR chain.

Instruction given to the agent

Publish the existing profile, greeting, and cli branches as a three-PR draft stack, in that order, targeting the run trunk.

Each PR must contain only its own layer. Keep the existing branch names, commit messages, and commits intact; do not rewrite the prepared history. Leave the workspace clean.

Scenario contract and grader

2. Amend the bottom layer

_{dirty bottom-layer fix → amended commit + restacked descendants}

Three draft PRs already exist. A dirty profile fix belongs in the bottom commit; both descendants must be rebuilt and all three existing PRs updated.

trunk
└── profile*      amend here
    └── greeting* restack
        └── cli*  restack

notes/debug.md     must stay uncommitted

The crux: route the fix into old history, cascade the rewrite upward, preserve PR identity, and leave unrelated work alone.

Instruction given to the agent

Fold the existing profile implementation and test edits into the original Add profile normalization commit—not a new fixup commit.

Restack and update all three draft PRs. Keep the existing PRs, titles, branch names, and one-commit-per-layer structure. Leave notes/debug.md exactly as it is and uncommitted.

Scenario contract and grader

3. Recover after the bottom lands

_{squash-merged bottom PR + advanced trunk → repaired two-PR stack}

The profile PR was squash-merged and trunk advanced again. The remaining stack is stale and contains a deterministic conflict.

before                         after

trunk'                         trunk'
└── profile (merged)           └── greeting'  PR → trunk
    └── greeting                   └── cli'    PR → greeting
        └── cli

The crux: detect the squash merge, prune the landed layer, resolve the conflict without losing either semantic edit, and repair the existing PR bases without recreating them.

Instruction given to the agent

The profile PR has been squash-merged, and trunk gained an audit follow-up. Update the remaining stack.

Resolve the config conflict so both audit: true from trunk and accountStatus: "enabled" from the greeting layer survive. Remove the merged profile layer locally, and update the existing greeting and CLI PRs. Do not recreate either PR. Finish with a clean workspace and no operation in progress.

Scenario contract and grader

The exact graphs, prompts, minefields, and expected states are in the scenario guide.

Method

Principle	Contract
Identical task	Both workflows receive the same fixture, dirty state, GitHub state, and plain-English instruction.
Deterministic grader	Scripts inspect commit boundaries, ancestry, file ownership, protected trunk, branch heads, PR identity and bases, conflicts, tests, and required leftovers. There is no LLM judge.
Live GitHub	Every run gets isolated bench/<run-id>/... branches and real PRs in gitbutlerapp/stack-testing.
Timing boundary	Fixture creation, tool adoption, skill installation, GitHub setup, grading, and cleanup are excluded. Time covers only the agent turn.
Enforced workflow	PATH wrappers trace commands and block GitButler in the gh-stack arm, gh stack in the GitButler arm, raw PR mutations, and raw Git writes where disallowed.
Repeated evidence	Each agent ran all 9 cells at k=6: 54 sequential trials per agent and 108 total. Both batches use the same seed and randomized schedule. Prompt, JSONL, command trace, verifier, cleanup, and checksums are archived before the next run.

Read the full benchmark contract.

Provenance

Component	Codex	Claude Code
Agent	gpt-5.6-sol · codex-cli 0.145.0 · SHA 134063e	claude-opus-4-8 · Claude Code 2.1.220 · SHA 8addc85
GitHub CLI	gh 2.86.0 · SHA 4f89c3c	same
gh-stack CLI	0.1.0 · SHA caec066	same
Old gh-stack skill	0.0.9 · 8c9c833 · frozen copy	frozen copy
New gh-stack skill	0.1.0 · 14fc42e · frozen copy	frozen copy
GitButler	latest origin/master at de392a9 + local-build fix · CLI SHA 7a85295 · frozen skill	same CLI · frozen skill
Fixture	bench/fixture-v1 · 6e32b32	same
Batch	codex-k6-three-tools-20260805	claude-opus48-k6-three-tools-20260805

Claude trials fail closed unless the runtime reports the exact model, CLI version, cwd, permission mode, intended skill, and empty MCP/plugin sets. Its CLI binary hash is checked before and after every run.

Limits and disclosure

• k=6 is repeated evidence, but still only six observations per agent–workflow–scenario cell.
• Only one model from each agent family ran.
• Codex and Claude ran in separate sequential batches. Within-agent comparisons are cleaner than cross-agent timing comparisons.
• Both agents used the same schedule seed and exact frozen tool binaries. Model sampling and live-service latency still vary between trials.
• GitButler was built from the latest local source checkout after adding the archived macOS local-build credential fix required to prevent an unsigned binary from blocking on a Keychain prompt.
• The two batches needed zero infrastructure retries.
• Runs use live GitHub, so model and network latency affect wall time.
• This measures the complete CLI + skill workflow, not binaries in isolation or human usability.
• Every skill and executable was frozen by SHA-256 and rechecked before each trial.
• This benchmark is maintained by GitButler, one of the tools measured. The counterweight is inspectability: graders, policy wrappers, frozen provenance, traces, machine summaries, and all 108 current raw runs are checked in.

Both final cleanup audits—Codex and Claude—passed on their first attempt with no leaked benchmark branches or open PRs.

Reproduce this benchmark

Validate the scenarios and pinned remote fixture:

npm run validate

Run one isolated Claude trial:

npm run run:agent -- \
  --agent claude \
  --model claude-opus-4-8 \
  --task 03-recover-after-merge \
  --arm 'but+skill' \
  --run-id my-recovery-trial

Run a fresh sequential Claude k=6 matrix across all three configurations:

npm run run:matrix -- \
  --agent claude \
  --model claude-opus-4-8 \
  --k 6 \
  --arms gh-stack-0.0.9+skill,gh-stack-0.1.0+skill,but+skill \
  --seed k6-three-tools-YYYYMMDD \
  --batch-id claude-opus48-k6-three-tools-YYYYMMDD

The matrix runner saves an immutable randomized plan, checksummed per-run evidence, summary.json, report.md, and a final remote cleanup audit under results/<batch-id>/.

[!CAUTION] Never run cleanup against a hand-written prefix. Cleanup must read the exact run ID recorded by the harness.

Results history · Scenario definitions · Benchmark contract · MIT License

Maintained by GitButler, one of the workflows measured.