Skip to content

Admit all four binary targets instead of queueing the fourth - #125

Closed
Bencheng21 wants to merge 2 commits into
mainfrom
goreleaser-parallelism-4
Closed

Bencheng21 wants to merge 2 commits into
mainfrom
goreleaser-parallelism-4

Conversation

@Bencheng21

Copy link
Copy Markdown
Contributor

Why

GoReleaser sizes its build semaphore from the CPU count — cmd/release.go:

ctx.Parallelism = runtime.GOMAXPROCS(0)
if options.parallelism > 0 { ctx.Parallelism = options.parallelism }

and internal/pipe/build/build.go passes it straight into the limiter:

g := semerrgroup.New(ctx.Parallelism)

macos-latest has 3 vCPUs and the binaries job builds four targets (linux amd64/arm64, darwin amd64/arm64), so the fourth waits for a free slot.

This is measurable and near-universal. Across 30 sampled connector releases the gap between consecutive build starts is sharply bimodal:

Gap Measurements Range
1→2 and 2→3 60 max 12.9 ms
3→4 29 of 30 106–233 s
3→4 1 of 30 0.5 ms

Nothing falls in between, so no threshold choice affects the result. On baton-datadog the fourth target idled 11m40s, and the build stage was 13m13s of a 16m job.

The single exception — baton-servicedesk-plus — admitted all four within 0.5 ms on an identical runner image, image version, provisioner, workflow ref and caller config. That's consistent with GitHub billing the standard macOS tier as "3-core or 4-core", i.e. a mixed fleet.

What this changes

  • Passes -p 4 to the binaries GoReleaser invocation, overriding the CPU-derived default so all four targets start immediately.
  • Adds a step recording core count, CPU model and RAM. Without it a run can't be read correctly: on a 4-core machine all four targets start regardless, which would mask the flag's effect entirely.

Deliberately unchanged: runs-on stays macos-latest. This is measurable against the existing baseline with no billing change. Larger runners are not free for public repositories and 160 of the 521 baton repos are public, so the runner-size route forfeits a free tier that this change doesn't touch.

What this does not claim

This removes queueing, not CPU shortage. Four builds would share three cores rather than three sharing three; total CPU is unchanged.

Measured locally, four cold cross-compiles of a connector need ~392 CPU-seconds and reach 8.2× parallelism when given 12 cores, at peak RSS under 1 GB. So the work is CPU-bound and genuinely wants more cores than this provides. A modest gain is the honest expectation.

The point is to separate the two effects. If -p 4 recovers most of the time, the fix is a flag. If it doesn't, the shortage is proven rather than assumed — which is the evidence #116 and #117 currently lack.

Expected ceiling

Worth stating up front: binaries runs in parallel with goreleaser-windows and goreleaser-docker, so shrinking it only helps until one of those becomes the floor. Across 55 surveyed connectors the median headroom is 47 seconds, and for 14 of 55 binaries already isn't the longest job. The wins are concentrated in a handful of connectors (baton-github 6:01 of headroom, baton-file 4:02); the median connector has little to gain from any of these three PRs.

Validation

  • Parsed release.yaml with PyYAML; confirmed args, runs-on and step order.
  • Re-checked the GORELEASER_CURRENT_TAG pin on all four GoReleaser steps (the invariant from Pin GoReleaser to requested release tag #112).
  • scripts/test-normalize-release-options.sh, test-derive-iam-role-name.sh, test-release-config-templates.py, go build ./...

Not validated: no release has run with this flag. Hence draft.

How to test

Point one connector at this branch and cut a prerelease tag, then compare the building binaries phase against its baseline — scripts/profile-release.sh reads it off the log. Check the Report runner hardware output first: if ncpu=4, that run can't measure the flag and needs repeating.

Relationship to the other PRs

PR Change Cost
this -p 4 on the existing runner $0
#117 macos-latest-xlarge (5-core M2) $0.102/min, forfeits free tier on 160 public repos
#116 split linux onto ubuntu $0, larger diff (~850 lines)

I'd land this one first — it's the cheapest, and its result tells you whether either of the others is worth the trouble.

🤖 Generated with Claude Code

Bencheng21 and others added 2 commits September 22, 2026 20:09
**Why**

GoReleaser sizes its build semaphore from the CPU count. cmd/release.go does:

    ctx.Parallelism = runtime.GOMAXPROCS(0)
    if options.parallelism > 0 { ctx.Parallelism = options.parallelism }

and internal/pipe/build/build.go passes that straight into the limiter:

    g := semerrgroup.New(ctx.Parallelism)

macos-latest has 3 vCPUs and the binaries job builds four targets (linux
amd64/arm64, darwin amd64/arm64), so the fourth waits for a free slot.

This is measurable and near-universal. Across 30 sampled connector releases the
gap between consecutive build starts is sharply bimodal: gaps 1->2 and 2->3 are
at most 12.9ms (60 measurements), while gap 3->4 is 106-233s in 29 of 30 cases.
Nothing falls in between. On baton-datadog the fourth target idled 11m40s, and
the build stage took 13m13s of a 16m job.

The single exception, baton-servicedesk-plus, admitted all four within 0.5ms on
an identical runner image, image version, provisioner, workflow ref and caller
config -- consistent with GitHub billing the standard macOS tier as "3-core or
4-core", i.e. a mixed fleet.

**What this changes**

- Passes -p 4 to the binaries GoReleaser invocation, overriding the CPU-derived
  default so all four targets start immediately.
- Adds a step recording core count, CPU model and RAM. Without it a run cannot
  be read correctly: on a 4-core machine all four targets start regardless, which
  would mask the flag's effect entirely.

Deliberately left alone: runs-on stays macos-latest, so this is measurable
against the existing baseline with no billing change. Larger runners are not
free for public repositories and 160 of the 521 baton repos are public, so the
runner-size route forfeits a free tier that this change does not touch.

**What this does not claim**

This removes queueing, not CPU shortage. Four builds would share three cores
rather than three sharing three; total CPU is unchanged. Measured locally, four
cold cross-compiles of a connector need ~392 CPU-seconds and reach 8.2x
parallelism when given 12 cores, with peak RSS under 1GB -- so the work is
CPU-bound and genuinely wants more cores than this provides.

The point is to separate the two effects. If -p 4 recovers most of the time, the
fix is a flag. If it does not, the shortage is proven rather than assumed, which
is the evidence the runner-size and job-split proposals currently lack.

**Validation**

- Parsed release.yaml with PyYAML; confirmed args, runs-on and step order.
- Re-checked the GORELEASER_CURRENT_TAG pin on all four GoReleaser steps.
- scripts/test-normalize-release-options.sh, test-derive-iam-role-name.sh,
  test-release-config-templates.py, go build ./...

Not validated: no release has run with this flag.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Forces the queueing condition so its cost can be measured on a runner that
would not otherwise exhibit it.

GoReleaser defaults ctx.Parallelism to runtime.GOMAXPROCS(0), and the standard
macOS fleet is mixed -- GitHub bills it as "3-core or 4-core". Measured across
30 connector releases, 29 admitted three of the four targets and queued the
fourth for 106-233s, but baton-github-test consistently lands on 4-core machines
and starts all four within 1.4ms. On that repo a higher limit is a no-op, so the
queueing cost cannot be observed.

Pinning to 3 reproduces the majority condition on any machine, making the two
cases comparable on the same repo:

    -p 3 (this branch)  vs  default (4 slots on that hardware)

A measured v0.1.141 run on macos-latest-xlarge reported ncpu=5 and completed
building binaries in 124s, against a 99-100s baseline on macos-latest with all
four targets already concurrent -- no improvement, because that connector was
never queueing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Bencheng21 Bencheng21 closed this Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant