Skip to content

Commit d3e97f4

Browse files
authored
ci: scope pull request checks to what the diff can affect (#8946)
* ci: scope pull request checks to what the diff can affect A `changes` job classifies the pull request's files against explicit allowlists (.github/scripts/ci-changes.sh, replacing desktop-live-changes.sh): - content posts, author profiles and their public assets: lint, test, build - apps/docs: lint, test - apps/desktop: lint, test, desktop-live - agent guidance (.agents, .claude, .cursor, AGENTS.md, CLAUDE.md, root README): lint Any other file, any .github change, an empty diff, or a diff that cannot be worked out runs everything; pushes and dispatches always run everything. The live desktop suite keeps its own skip list, now including the content asset folders it previously missed. `ci` accepts a skip only when `changes` set that job's flag to exactly `false`, so a failed or skipped `changes`, or a lost output, fails it. * ci: move the ci gate into a tested script and reject non-boolean change flags The ci aggregator's skip rules now live in .github/scripts/ci-gate.sh, read from NEEDS, with table-driven tests covering every guard. The change-classifier test helper fails on any flag value other than exactly true or false.
1 parent bc3673a commit d3e97f4

8 files changed

Lines changed: 546 additions & 146 deletions

File tree

‎.github/scripts/ci-changes.sh‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
#!/usr/bin/env bash
2+
# Prints which checks a pull request's changes can affect, as `<flag>=true|false` lines for
3+
# $GITHUB_OUTPUT. A flag is `false` only when every changed file is on an allowlist whose jobs
4+
# fully cover it. A file on no allowlist sets every flag but `desktop_live` (which keeps its own
5+
# skip list) to `true`; a change under .github/, an empty diff, or any failure to work out the
6+
# diff sets all of them.
7+
#
8+
# full integration and e2e: any file not allowlisted below
9+
# test the Vitest shards: content, docs app, and desktop changes, plus `full`
10+
# build the app's `next build`: content changes, plus `full`
11+
# desktop_live the live Electron suite: any file outside its own skip list
12+
#
13+
# lint always runs and is not gated.
14+
#
15+
# Usage: ci-changes.sh <base-sha>
16+
set -u
17+
18+
# Posts, author profiles, and the public assets only posts reference. The test shards read them
19+
# (lib/content/og-image.test.ts) and `next build` prerenders every post with its MDX components,
20+
# which check:library-content compiles but never renders. Components under content/ are code.
21+
content='^apps/sim/content/([^/]+/)*[^/]+\.mdx?$|^apps/sim/content/authors/[^/]+\.json$|^apps/sim/public/(authors|blog|changelog|library)/'
22+
# Its own workspace tests, its type-check and docs-manifest audit in lint, and apps/sim tests
23+
# that read its pages (tools/netsuite/netsuite.test.ts).
24+
docs='^apps/docs/'
25+
# Nothing outside the Electron app imports it; the desktop bridge and protocol packages are
26+
# consumed by apps/sim and stay unlisted.
27+
desktop='^apps/desktop/'
28+
# Agent guidance, read only by the lint job's audits.
29+
guidance='^(\.agents|\.claude|\.cursor)/|^([^/]+/)*(AGENTS|CLAUDE)\.md$|^README\.md$'
30+
# The live desktop suite drives apps/desktop against the app, so it skips more than the rest.
31+
desktop_live_skip="$guidance"'|^(apps/docs/|apps/pii/|apps/sim/content/|apps/sim/public/(authors|blog|changelog|library)/|packages/(python-sdk|ts-sdk)/)|\.mdx?$|(^|/)LICENSE$'
32+
33+
emit() {
34+
printf 'full=%s\ntest=%s\nbuild=%s\ndesktop_live=%s\n' "$1" "$2" "$3" "$4"
35+
}
36+
run_all() {
37+
emit true true true true
38+
echo "Running every check: $1" >&2
39+
exit 0
40+
}
41+
matches() { printf '%s\n' "$names" | grep -qE "$1"; }
42+
outside() { printf '%s\n' "$names" | grep -qvE "$1"; }
43+
44+
base=${1:-}
45+
[ -n "$base" ] || run_all 'no base commit'
46+
git fetch --quiet --depth=1 origin "$base" || run_all "could not fetch $base"
47+
names=$(git diff --no-renames --name-only "$base" HEAD) || run_all "could not diff against $base"
48+
[ -n "$names" ] || run_all 'no changed files listed'
49+
if matches '^\.github/'; then run_all 'a CI file changed'; fi
50+
51+
desktop_live=false
52+
if outside "$desktop_live_skip"; then desktop_live=true; fi
53+
if outside "$content|$docs|$desktop|$guidance"; then
54+
emit true true true "$desktop_live"
55+
unlisted=$(printf '%s\n' "$names" | grep -vE "$content|$docs|$desktop|$guidance" | head -n 1)
56+
echo "Running every check (desktop_live=$desktop_live): $unlisted is not on an allowlist" >&2
57+
exit 0
58+
fi
59+
60+
test=false build=false
61+
if matches "$content|$docs|$desktop"; then test=true; fi
62+
if matches "$content"; then build=true; fi
63+
emit false "$test" "$build" "$desktop_live"
64+
echo "Scoped checks: test=$test build=$build desktop_live=$desktop_live" >&2

‎.github/scripts/ci-gate.sh‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#!/usr/bin/env bash
2+
# The `ci` job's verdict: the single required check, so it must never pass a run whose checks did
3+
# not all pass. Reads the job's `needs` context (`toJSON(needs)`) from $NEEDS and exits non-zero,
4+
# naming each offending job, unless every job succeeded or was skipped as `changes` intended.
5+
#
6+
# A skip passes only for a gated job whose `changes` flag is exactly `false`:
7+
#
8+
# integration, e2e full
9+
# test test
10+
# build build
11+
# desktop-live desktop_live
12+
#
13+
# Every other skip fails: an ungated job (lint, `changes` itself), a gated job whose flag is
14+
# `true`, missing, or anything else (a failed or skipped `changes` leaves no outputs). A failure
15+
# or a cancellation always fails.
16+
#
17+
# Usage: NEEDS='<toJSON(needs)>' ci-gate.sh
18+
set -euo pipefail
19+
20+
needs=${NEEDS:-}
21+
if ! jq -e 'type == "object"' <<< "$needs" > /dev/null 2>&1; then
22+
echo "::error::The needs context is not a JSON object: '$needs'"
23+
exit 1
24+
fi
25+
26+
failed="$(jq -r '
27+
{"integration": "full", "e2e": "full", "test": "test", "build": "build", "desktop-live": "desktop_live"} as $gate
28+
| (.changes.outputs // {}) as $flags
29+
| to_entries[]
30+
| select(.value.result != "success")
31+
| select(.value.result != "skipped" or $gate[.key] == null or $flags[$gate[.key]] != "false")
32+
| "\(.key): \(.value.result)"
33+
' <<< "$needs")"
34+
if [ -n "$failed" ]; then
35+
echo "::error::Checks did not pass:"
36+
echo "$failed"
37+
exit 1
38+
fi
39+
echo "All checks passed."

‎.github/scripts/desktop-live-changes.sh‎

Lines changed: 0 additions & 24 deletions
This file was deleted.

‎.github/workflows/checks.yml‎

Lines changed: 50 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,37 @@ permissions:
88
contents: read
99

1010
jobs:
11+
# Which checks a pull request's changes can affect (.github/scripts/ci-changes.sh). A job is
12+
# skipped only when the script prints `false` for its flag; an unset flag, and every push or
13+
# dispatch (the deploy chain's checks), runs it. Every gated job needs this one, so its failure
14+
# skips them and fails `ci` below.
15+
changes:
16+
name: changes
17+
runs-on: &runner-2vcpu ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
18+
timeout-minutes: 5
19+
outputs:
20+
full: ${{ github.event_name != 'pull_request' || steps.classify.outputs.full != 'false' }}
21+
test: ${{ github.event_name != 'pull_request' || steps.classify.outputs.test != 'false' }}
22+
build: ${{ github.event_name != 'pull_request' || steps.classify.outputs.build != 'false' }}
23+
desktop_live: ${{ github.event_name != 'pull_request' || steps.classify.outputs.desktop_live != 'false' }}
24+
steps:
25+
- name: Checkout code (mirror)
26+
if: github.event_name == 'pull_request' && !github.event.pull_request.head.repo.fork
27+
uses: useblacksmith/checkout@25227e61ff9dafe400e22fa487b673eac4e4409a # v1.8.1
28+
with:
29+
fetch-depth: 2
30+
- name: Checkout code
31+
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
32+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6
33+
with:
34+
fetch-depth: 2
35+
- name: Classify the pull request's changes
36+
id: classify
37+
if: github.event_name == 'pull_request'
38+
env:
39+
BASE: ${{ github.event.pull_request.base.sha }}
40+
run: bash .github/scripts/ci-changes.sh "$BASE" >> "$GITHUB_OUTPUT"
41+
1142
integration:
1243
# Runs the real-infrastructure test layer: every `*.integration.ts` in packages/db and
1344
# apps/sim, discovered by glob (`vitest run --mode integration`), against the database each
@@ -19,6 +50,8 @@ jobs:
1950
# shard runs its files one at a time against its own database. Files run serially, so a shard
2051
# barely uses more than one core: 4 vCPU is enough.
2152
name: integration (${{ matrix.provision }}, ${{ matrix.shard }}/4)
53+
needs: changes
54+
if: needs.changes.outputs.full == 'true'
2255
runs-on: &runner-4vcpu ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
2356
timeout-minutes: 20
2457
strategy:
@@ -143,6 +176,8 @@ jobs:
143176
# self-hosted suites fit on 4.
144177
e2e:
145178
name: e2e (${{ matrix.group }})
179+
needs: changes
180+
if: needs.changes.outputs.full == 'true'
146181
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && matrix.runner || 'ubuntu-latest' }}
147182
timeout-minutes: ${{ matrix.group == 'mobile' && 40 || 20 }}
148183
strategy:
@@ -217,39 +252,12 @@ jobs:
217252
if-no-files-found: ignore
218253
retention-days: 7
219254

220-
# Pull requests skip the live desktop suite only when every change is clearly unrelated to the
221-
# app it drives (docs, other apps, published content). Anything else, and any failure to work
222-
# out the diff, runs it: a pull request that skipped it wrongly would first fail on staging.
223-
desktop-changes:
224-
name: desktop-changes
225-
runs-on: &runner-2vcpu ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
226-
timeout-minutes: 5
227-
outputs:
228-
changed: ${{ github.event_name != 'pull_request' || steps.diff.outputs.changed != 'false' }}
229-
steps:
230-
- name: Checkout code (mirror)
231-
if: github.event_name == 'pull_request' && !github.event.pull_request.head.repo.fork
232-
uses: useblacksmith/checkout@25227e61ff9dafe400e22fa487b673eac4e4409a # v1.8.1
233-
with:
234-
fetch-depth: 2
235-
- name: Checkout code
236-
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
237-
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6
238-
with:
239-
fetch-depth: 2
240-
- name: Diff against the pull request's base
241-
id: diff
242-
if: github.event_name == 'pull_request'
243-
env:
244-
BASE: ${{ github.event.pull_request.base.sha }}
245-
run: bash .github/scripts/desktop-live-changes.sh "$BASE" >> "$GITHUB_OUTPUT"
246-
247255
# Desktop tools in the real Electron app against a local app, on its own runner: the
248256
# Electron app, the dev app and its realtime server together outgrow the e2e runner.
249257
desktop-live:
250258
name: desktop-live
251-
needs: desktop-changes
252-
if: needs.desktop-changes.outputs.changed == 'true'
259+
needs: changes
260+
if: needs.changes.outputs.desktop_live == 'true'
253261
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-16vcpu-ubuntu-2404' || 'ubuntu-latest' }}
254262
timeout-minutes: 30
255263
services:
@@ -592,6 +600,8 @@ jobs:
592600
# task hash, so a shard only ever replays its own result.
593601
test:
594602
name: test (${{ matrix.shard }}/2)
603+
needs: changes
604+
if: needs.changes.outputs.test == 'true'
595605
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}
596606
timeout-minutes: 15
597607
strategy:
@@ -660,6 +670,8 @@ jobs:
660670
# not the native Turbopack workers that dominate, so it cannot prevent this.
661671
build:
662672
name: build
673+
needs: changes
674+
if: needs.changes.outputs.build == 'true'
663675
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-16vcpu-ubuntu-2404' || 'linux-x64-8-core' }}
664676
# Build durations crossed 15 minutes as the app grew (10m02 on Jul 29 AM,
665677
# 14m44 after the folders/desktop/library merges, then two straight
@@ -713,25 +725,22 @@ jobs:
713725
run: bunx turbo run build --filter=@sim/app
714726

715727
# One status for every check above: the single check to require on a branch ruleset, so adding,
716-
# sharding or renaming a job never means editing the ruleset. Skipped is a pass (the desktop live
717-
# suite skips on pull requests that cannot affect it); a failure or a cancellation is not.
728+
# sharding or renaming a job never means editing the ruleset. .github/scripts/ci-gate.sh decides:
729+
# a skip passes only for a gated job whose `changes` flag is exactly `false`; any other skip (a
730+
# failed or skipped `changes`, a lost output, an ungated job) fails, as does a failure or a
731+
# cancellation. A failed checkout fails the gate too.
718732
# `always()`, not `!cancelled()`: a skipped job satisfies a required check, so a gate that skips
719733
# on a cancelled run would report a cancelled head commit as passing.
720734
ci:
721735
name: ci
722-
needs: [integration, e2e, desktop-changes, desktop-live, lint, test, build]
736+
needs: [changes, integration, e2e, desktop-live, lint, test, build]
723737
if: ${{ always() }}
724738
runs-on: *runner-2vcpu
725739
timeout-minutes: 5
726740
steps:
741+
- *checkout-mirror
742+
- *checkout-plain
727743
- name: Require every check to pass
728744
env:
729-
RESULTS: ${{ toJSON(needs) }}
730-
run: |
731-
failed="$(jq -r 'to_entries[] | select(.value.result != "success" and .value.result != "skipped") | "\(.key): \(.value.result)"' <<< "$RESULTS")"
732-
if [ -n "$failed" ]; then
733-
echo "::error::Checks did not pass:"
734-
echo "$failed"
735-
exit 1
736-
fi
737-
echo "All checks passed."
745+
NEEDS: ${{ toJSON(needs) }}
746+
run: bash .github/scripts/ci-gate.sh

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,8 @@ name: ci
1818
on:
1919
push:
2020
branches: [main, staging, dev]
21-
# No path filter: `checks / ci` is the required status check, and a pull request whose run
22-
# never starts never reports it, so it could not merge. Docs-only pull requests are rare enough
23-
# that running CI on them costs nothing measurable.
21+
# No path filter: a pull request whose run never starts never reports `checks / ci`. The
22+
# `changes` job in checks.yml scopes the jobs to what the diff can affect instead.
2423
pull_request:
2524
branches: [main, staging, dev]
2625

0 commit comments

Comments
 (0)