Admit all four binary targets instead of queueing the fourth - #125
Closed
Bencheng21 wants to merge 2 commits into
Closed
Bencheng21 wants to merge 2 commits into
Bencheng21 wants to merge 2 commits into
Conversation
**Why**
GoReleaser sizes its build semaphore from the CPU count. cmd/release.go does:
ctx.Parallelism = runtime.GOMAXPROCS(0)
if options.parallelism > 0 { ctx.Parallelism = options.parallelism }
and internal/pipe/build/build.go passes that straight into the limiter:
g := semerrgroup.New(ctx.Parallelism)
macos-latest has 3 vCPUs and the binaries job builds four targets (linux
amd64/arm64, darwin amd64/arm64), so the fourth waits for a free slot.
This is measurable and near-universal. Across 30 sampled connector releases the
gap between consecutive build starts is sharply bimodal: gaps 1->2 and 2->3 are
at most 12.9ms (60 measurements), while gap 3->4 is 106-233s in 29 of 30 cases.
Nothing falls in between. On baton-datadog the fourth target idled 11m40s, and
the build stage took 13m13s of a 16m job.
The single exception, baton-servicedesk-plus, admitted all four within 0.5ms on
an identical runner image, image version, provisioner, workflow ref and caller
config -- consistent with GitHub billing the standard macOS tier as "3-core or
4-core", i.e. a mixed fleet.
**What this changes**
- Passes -p 4 to the binaries GoReleaser invocation, overriding the CPU-derived
default so all four targets start immediately.
- Adds a step recording core count, CPU model and RAM. Without it a run cannot
be read correctly: on a 4-core machine all four targets start regardless, which
would mask the flag's effect entirely.
Deliberately left alone: runs-on stays macos-latest, so this is measurable
against the existing baseline with no billing change. Larger runners are not
free for public repositories and 160 of the 521 baton repos are public, so the
runner-size route forfeits a free tier that this change does not touch.
**What this does not claim**
This removes queueing, not CPU shortage. Four builds would share three cores
rather than three sharing three; total CPU is unchanged. Measured locally, four
cold cross-compiles of a connector need ~392 CPU-seconds and reach 8.2x
parallelism when given 12 cores, with peak RSS under 1GB -- so the work is
CPU-bound and genuinely wants more cores than this provides.
The point is to separate the two effects. If -p 4 recovers most of the time, the
fix is a flag. If it does not, the shortage is proven rather than assumed, which
is the evidence the runner-size and job-split proposals currently lack.
**Validation**
- Parsed release.yaml with PyYAML; confirmed args, runs-on and step order.
- Re-checked the GORELEASER_CURRENT_TAG pin on all four GoReleaser steps.
- scripts/test-normalize-release-options.sh, test-derive-iam-role-name.sh,
test-release-config-templates.py, go build ./...
Not validated: no release has run with this flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Forces the queueing condition so its cost can be measured on a runner that
would not otherwise exhibit it.
GoReleaser defaults ctx.Parallelism to runtime.GOMAXPROCS(0), and the standard
macOS fleet is mixed -- GitHub bills it as "3-core or 4-core". Measured across
30 connector releases, 29 admitted three of the four targets and queued the
fourth for 106-233s, but baton-github-test consistently lands on 4-core machines
and starts all four within 1.4ms. On that repo a higher limit is a no-op, so the
queueing cost cannot be observed.
Pinning to 3 reproduces the majority condition on any machine, making the two
cases comparable on the same repo:
-p 3 (this branch) vs default (4 slots on that hardware)
A measured v0.1.141 run on macos-latest-xlarge reported ncpu=5 and completed
building binaries in 124s, against a 99-100s baseline on macos-latest with all
four targets already concurrent -- no improvement, because that connector was
never queueing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
GoReleaser sizes its build semaphore from the CPU count —
cmd/release.go:and
internal/pipe/build/build.gopasses it straight into the limiter:macos-latesthas 3 vCPUs and the binaries job builds four targets (linux amd64/arm64, darwin amd64/arm64), so the fourth waits for a free slot.This is measurable and near-universal. Across 30 sampled connector releases the gap between consecutive build starts is sharply bimodal:
Nothing falls in between, so no threshold choice affects the result. On
baton-datadogthe fourth target idled 11m40s, and the build stage was 13m13s of a 16m job.The single exception —
baton-servicedesk-plus— admitted all four within 0.5 ms on an identical runner image, image version, provisioner, workflow ref and caller config. That's consistent with GitHub billing the standard macOS tier as "3-core or 4-core", i.e. a mixed fleet.What this changes
-p 4to the binaries GoReleaser invocation, overriding the CPU-derived default so all four targets start immediately.Deliberately unchanged:
runs-onstaysmacos-latest. This is measurable against the existing baseline with no billing change. Larger runners are not free for public repositories and 160 of the 521 baton repos are public, so the runner-size route forfeits a free tier that this change doesn't touch.What this does not claim
This removes queueing, not CPU shortage. Four builds would share three cores rather than three sharing three; total CPU is unchanged.
Measured locally, four cold cross-compiles of a connector need ~392 CPU-seconds and reach 8.2× parallelism when given 12 cores, at peak RSS under 1 GB. So the work is CPU-bound and genuinely wants more cores than this provides. A modest gain is the honest expectation.
The point is to separate the two effects. If
-p 4recovers most of the time, the fix is a flag. If it doesn't, the shortage is proven rather than assumed — which is the evidence #116 and #117 currently lack.Expected ceiling
Worth stating up front: binaries runs in parallel with
goreleaser-windowsandgoreleaser-docker, so shrinking it only helps until one of those becomes the floor. Across 55 surveyed connectors the median headroom is 47 seconds, and for 14 of 55 binaries already isn't the longest job. The wins are concentrated in a handful of connectors (baton-github6:01 of headroom,baton-file4:02); the median connector has little to gain from any of these three PRs.Validation
release.yamlwith PyYAML; confirmedargs,runs-onand step order.GORELEASER_CURRENT_TAGpin on all four GoReleaser steps (the invariant from Pin GoReleaser to requested release tag #112).scripts/test-normalize-release-options.sh,test-derive-iam-role-name.sh,test-release-config-templates.py,go build ./...Not validated: no release has run with this flag. Hence draft.
How to test
Point one connector at this branch and cut a prerelease tag, then compare the
building binariesphase against its baseline —scripts/profile-release.shreads it off the log. Check theReport runner hardwareoutput first: ifncpu=4, that run can't measure the flag and needs repeating.Relationship to the other PRs
-p 4on the existing runnermacos-latest-xlarge(5-core M2)I'd land this one first — it's the cheapest, and its result tells you whether either of the others is worth the trouble.
🤖 Generated with Claude Code