Performance gate: per-PR baseline overlays, so baselines stop conflicting

.github/workflows/linux-build-run.yml

@@ -396,6 +396,10 @@ jobs:
       # results travel with the screenshots.
       - name: ParparVM vs JDK 25 (performance gate)
         if: success() || failure()
+        # Names this pull request's overlay (perf-baseline/pr/<number>.json) in the
+        # comment. A workflow_dispatch run carries no pull request, so the gate asks gh.
+        env:
+          GH_TOKEN: ${{ github.token }}
         run: |
           vm/selfhost/ci-perf-gate.sh run linux-${{ matrix.arch }} artifacts/linux-port/raw/perf \
             --hello-workload "$GITHUB_WORKSPACE/artifacts/linux-port/raw/translation-record.jsonl" \

.github/workflows/parparvm-selfhost.yml

@@ -133,7 +133,7 @@ jobs:
       #  - Keeping it in both places would pay for the expensive -O3 self-host build
       #    twice on any PR carrying the `selfhost` label.
       #
-      # The baselines and tolerance live in vm/selfhost/perf-baseline.json, the
+      # The baselines and tolerance live in vm/selfhost/perf-baseline/, the
       # measurement in vm/selfhost/perf-gate.py.
 
       # Both trees, so a divergence can be inspected rather than guessed at from

.github/workflows/parparvm-tests-windows.yml

@@ -558,6 +558,10 @@ jobs:
       # builds only ask javac for -source/-target 1.8 against an explicit bootclasspath.
       - name: ParparVM vs JDK 25 (performance gate, x64)
         if: success() || failure()
+        # Names this pull request's overlay (perf-baseline/pr/<number>.json) in the
+        # comment. A workflow_dispatch run carries no pull request, so the gate asks gh.
+        env:
+          GH_TOKEN: ${{ github.token }}
         shell: bash
         run: |
           export JDK_8_HOME="${JDK_8_HOME:-$JAVA_HOME}"
@@ -763,6 +767,10 @@ jobs:
       # builds only ask javac for -source/-target 1.8 against an explicit bootclasspath.
       - name: ParparVM vs JDK 25 (performance gate, arm64)
         if: success() || failure()
+        # Names this pull request's overlay (perf-baseline/pr/<number>.json) in the
+        # comment. A workflow_dispatch run carries no pull request, so the gate asks gh.
+        env:
+          GH_TOKEN: ${{ github.token }}
         shell: bash
         run: |
           export JDK_8_HOME="${JDK_8_HOME:-$JAVA_HOME}"

.github/workflows/perf-baseline.yml

@@ -0,0 +1,117 @@
+name: Performance baselines
+
+# The ParparVM performance gate's baselines live in vm/selfhost/perf-baseline/:
+# base/ holds the consolidated rows, and each pull request that changes a baseline
+# writes only its own pr/<number>.json, so baseline changes stay in the pull
+# request's diff without two branches ever editing the same file
+# (vm/selfhost/perf_baseline.py explains the layout).
+#
+#   check  every pull request: the overlays resolve against base/ without
+#          contradicting each other, and the pull request touched only its own
+#          overlay (base/ is the fold's alone). No paths filter: several of this
+#          repository's pull requests exceed GitHub's paths-filter diff limit, and a
+#          filtered check would silently not run on exactly those.
+#   fold   nightly on master: every overlay on master belongs to a merged pull
+#          request, so it is folded into base/ and deleted, in one commit that
+#          provably changes no verdict. Then the website is rebuilt, because the
+#          Port Status page's ParparVM vs JDK 25 table is rendered from these rows.
+#          A push made with GITHUB_TOKEN triggers no workflow, hence the dispatch.
+
+on:
+  pull_request:
+    branches:
+      - master
+  push:
+    branches:
+      - master
+  schedule:
+    - cron: '20 3 * * *'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  check:
+    if: github.event_name == 'pull_request' || github.event_name == 'push'
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          # The pull request's base commit has to be present for the diff.
+          fetch-depth: 0
+
+      - name: The gate's own unit tests
+        working-directory: vm/selfhost
+        run: python3 -m unittest test_perf_gate
+
+      - name: Check the baselines
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: |
+          if [ -n "${BASE_SHA}" ]; then
+            python3 vm/selfhost/perf_baseline.py check --base "${BASE_SHA}" --pr "${PR_NUMBER}"
+          else
+            python3 vm/selfhost/perf_baseline.py check
+          fi
+
+  fold:
+    if: (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master'
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: write
+      actions: write
+    concurrency:
+      group: perf-baseline-fold
+      cancel-in-progress: false
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          ref: master
+
+      - name: Fold merged overlays into base/
+        id: fold
+        run: |
+          set -euo pipefail
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          # A merge landing while this runs makes the push non-fast-forward. Start over
+          # from the new master rather than rebasing: the fold is cheap, and redoing it
+          # folds the overlay that just arrived as well.
+          for attempt in 1 2 3; do
+            git fetch --quiet origin master
+            git reset --quiet --hard origin/master
+            python3 vm/selfhost/perf_baseline.py fold
+            git add -A vm/selfhost/perf-baseline
+            if git diff --staged --quiet; then
+              echo "No overlays to fold."
+              echo "folded=false" >> "$GITHUB_OUTPUT"
+              exit 0
+            fi
+            git diff --staged --stat
+            git commit --quiet -m "ci: fold merged performance baselines into perf-baseline/base"
+            if git push origin HEAD:master; then
+              echo "folded=true" >> "$GITHUB_OUTPUT"
+              exit 0
+            fi
+            echo "master moved during the fold (attempt ${attempt}); retrying."
+          done
+          echo "Could not push the fold after 3 attempts." >&2
+          exit 1
+
+      # One-shot on purpose. If this dispatch fails after the fold was pushed, a re-run
+      # finds nothing to fold and skips it -- and that is acceptable: website-docs.yml
+      # also rebuilds on its own daily schedule (and port-status-nightly dispatches it),
+      # so the JDK 25 table, which shows gate baselines rather than live results, is at
+      # most a day behind. Persisting a "rebuild owed" flag to close that window would
+      # add state for no visible benefit.
+      - name: Rebuild the Port Status page
+        if: steps.fold.outputs.folded == 'true'
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: gh workflow run website-docs.yml --ref master -f deploy_production=true

.github/workflows/scripts-macos.yml

@@ -247,6 +247,10 @@ jobs:
       # got it, and JDK 25 is fetched privately so no later step sees a different JDK.
       - name: ParparVM vs JDK 25 (performance gate)
         if: success() || failure()
+        # Names this pull request's overlay (perf-baseline/pr/<number>.json) in the
+        # comment. A workflow_dispatch run carries no pull request, so the gate asks gh.
+        env:
+          GH_TOKEN: ${{ github.token }}
         run: |
           set -u
           ENV_FILE="${TMPDIR%/}/codenameone-tools/tools/env.sh"

.github/workflows/website-docs.yml

@@ -45,6 +45,11 @@ on:
       - 'vm/backend/src/**'
       - 'vm/backend/impl/**'
       - 'vm/backend/shared-sources.sh'
+      # build.sh renders the Port Status JDK 25 table with this script's `summary`, so a
+      # change to its output shape must go through the Hugo build and validate_port_status.
+      # The baseline DATA (vm/selfhost/perf-baseline/**) is left out on purpose: a row
+      # changes numbers, not shape, and the nightly fold dispatches a rebuild for it.
+      - 'vm/selfhost/perf_baseline.py'
       - '.github/workflows/website-docs.yml'
   push:
     branches: [main, master]
@@ -78,6 +83,8 @@ on:
       - 'vm/backend/src/**'
       - 'vm/backend/impl/**'
       - 'vm/backend/shared-sources.sh'
+      # As above: the JDK 25 table's renderer.
+      - 'vm/selfhost/perf_baseline.py'
       - '.github/workflows/website-docs.yml'
   workflow_dispatch:
     inputs:

.gitignore

@@ -196,3 +196,6 @@ cn1-build-hints.json
 # Concatenated CSS the native-theme build feeds the compiler. Regenerated on
 # every run; the parts under native-themes/<theme>/*.css are the source.
 native-themes/*/target/
+
+# Rendered from vm/selfhost/perf-baseline by scripts/website/build.sh; never committed.
+docs/website/data/port_status_jdk25.json

docs/website/assets/css/extended/cn1-port-status.css

@@ -490,6 +490,21 @@
     min-width: 1320px;
 }
 
+.cn1-port-status__matrix--jdk25 table {
+    min-width: 900px;
+}
+
+.cn1-port-status__ratio {
+    display: block;
+    white-space: nowrap;
+}
+
+.cn1-port-status__ratio small {
+    color: var(--secondary);
+    display: block;
+    font-size: .68rem;
+}
+
 .cn1-port-status__matrix--performance tbody th small {
     color: var(--secondary);
     display: block;

docs/website/layouts/_default/port-status.html

@@ -319,6 +319,63 @@ <h2 id="performance-title">{{ $support.benchmark.title }}</h2>
     </p>
   </section>
 
+  {{- /* Rendered by scripts/website/build.sh from vm/selfhost/perf-baseline: the ratios
+         every platform build's performance gate holds ParparVM to, so this table and the
+         gate cannot disagree. Absent from a bare `hugo` run, hence the guard. */ -}}
+  {{- with site.Data.port_status_jdk25 }}
+  {{- $jdk := . -}}
+  <section class="cn1-port-status__performance" data-jdk25-evidence aria-labelledby="jdk25-title">
+    <header>
+      <p class="cn1-port-status__eyebrow">Gated performance</p>
+      <h2 id="jdk25-title">ParparVM against JDK 25</h2>
+      <p>
+        Each value is ParparVM's time or peak memory divided by HotSpot JDK 25 running the same code on the same CI runner,
+        as the median of interleaved, paired runs; below 1.00x ParparVM is faster or smaller. These are the baselines every
+        platform build is gated on: a pull request that moves one past its tolerance, in either direction, fails until the new value is
+        recorded in the repository. A platform measured on several runner CPU models shows the median across them and, below it, the range.
+      </p>
+    </header>
+    <div class="cn1-port-status__matrix cn1-port-status__matrix--performance cn1-port-status__matrix--jdk25" aria-label="ParparVM to JDK 25 ratio matrix">
+      <table>
+        <thead>
+          <tr>
+            <th scope="col">Workload</th>
+            {{- range $jdk.platforms }}<th scope="col" title="CPU models: {{ delimit .cpus ", " }}"><span>{{ .name }}</span></th>{{- end }}
+          </tr>
+        </thead>
+        <tbody>
+          {{- range $jdk.benchmarks }}
+          {{- $benchmark := . -}}
+          <tr data-jdk25-row="{{ .id }}">
+            <th scope="row"><strong>{{ .name }}</strong><p>{{ .description }}</p><small>Time and RAM, ParparVM / JDK 25</small></th>
+            {{- range $jdk.platforms }}
+              {{- $platform := . -}}
+              {{- $cell := index .benchmarks $benchmark.id -}}
+              {{- $covered := 0 -}}{{- with $cell -}}{{- $covered = len .cpus -}}{{- end -}}
+              <td data-jdk25-cell data-platform="{{ .id }}" title="{{ .name }}: {{ $benchmark.name }}, ParparVM / JDK 25 across {{ $covered }} of {{ len .cpus }} CPU model{{ if ne (len .cpus) 1 }}s{{ end }}">
+                {{- with $cell -}}
+                  {{- range $metric := slice "time" "memory" -}}
+                    {{- $m := index $cell $metric -}}
+                    {{- $lo := printf "%.2f" (float $m.min) -}}
+                    {{- $hi := printf "%.2f" (float $m.max) -}}
+                    <span class="cn1-port-status__ratio">{{ if eq $metric "time" }}Time{{ else }}RAM{{ end }} {{ printf "%.2fx" (float $m.median) }}
+                      {{- if ne $lo $hi }}<small>{{ $lo }}-{{ $hi }}x</small>{{ end -}}
+                    </span>
+                  {{- end -}}
+                {{- else -}}Not measured{{- end -}}
+              </td>
+            {{- end }}
+          </tr>
+          {{- end }}
+        </tbody>
+      </table>
+    </div>
+    <p class="cn1-port-status__source-note">
+      <a href="https://github.com/codenameone/CodenameOne/tree/master/vm/selfhost/perf-baseline" target="_blank" rel="noopener">Performance gate baselines</a>
+    </p>
+  </section>
+  {{- end }}
+
   <footer class="cn1-port-status__method">
     <h2>How to read this page</h2>
     <p>

scripts/website/build.sh

@@ -814,6 +814,13 @@ if [ "${WEBSITE_REFRESH_PORT_STATUS}" = "true" ]; then
   "${REPO_ROOT}/scripts/website/sync_port_status_reports.sh"
 fi
 "${PYTHON_BIN}" "${REPO_ROOT}/scripts/hellocodenameone/conformance/port_status.py" validate
+# The ParparVM vs JDK 25 table is rendered from the performance gate's own baselines, the
+# ratios every platform build is held to. Generated here rather than committed: a copy in
+# the tree would go stale whenever a pull request rebaselines, and requiring each such
+# pull request to refresh it would bring back the merge conflicts the baseline layout
+# exists to avoid (vm/selfhost/perf_baseline.py).
+"${PYTHON_BIN}" "${REPO_ROOT}/vm/selfhost/perf_baseline.py" summary \
+  --out "${WEBSITE_DIR}/data/port_status_jdk25.json"
 
 build_javadocs_for_site
 build_developer_guide_for_site

scripts/website/preview.sh

@@ -6,6 +6,9 @@ REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
 WEBSITE_DIR="${REPO_ROOT}/docs/website"
 
 python3 "${REPO_ROOT}/scripts/hellocodenameone/conformance/port_status.py" validate
+# Generated, never committed; see scripts/website/build.sh.
+python3 "${REPO_ROOT}/vm/selfhost/perf_baseline.py" summary \
+  --out "${WEBSITE_DIR}/data/port_status_jdk25.json"
 
 if [ ! -d "${WEBSITE_DIR}" ]; then
   echo "Website directory not found: ${WEBSITE_DIR}" >&2

scripts/website/validate_port_status.mjs

@@ -159,6 +159,26 @@ function validate() {
             fail(`performance cells for ${name} must identify the platform in their tooltip`);
         }
     }
+    // The ParparVM / JDK 25 table, rendered from the performance gate's baselines by
+    // build.sh. Its size comes from the data it was rendered from, not a number spelled
+    // here, so adding a gated platform or benchmark cannot fail this one CI round at a time.
+    const jdkData = JSON.parse(fs.readFileSync(
+        path.resolve(publicDirectory, "..", "data", "port_status_jdk25.json"), "utf8"));
+    const jdkRows = countMatches(page, /\bdata-jdk25-row(?:=|\s|>)/g);
+    const jdkCellTags = page.match(/<td\b[^>]*data-jdk25-cell[^>]*>/g) || [];
+    if (jdkData.platforms.length < 5 || jdkRows !== jdkData.benchmarks.length ||
+        jdkCellTags.length !== jdkRows * jdkData.platforms.length) {
+        fail("the ParparVM against JDK 25 table is incomplete");
+    }
+    for (const platform of jdkData.platforms) {
+        if (!jdkCellTags.some((tag) => attribute(tag, "data-platform") === platform.id &&
+            attribute(tag, "title").startsWith(`${platform.name}:`))) {
+            fail(`JDK 25 cells for ${platform.name} must identify the platform in their tooltip`);
+        }
+    }
+    if (/Not measured/.test(page.match(/data-jdk25-evidence[\s\S]*?<\/section>/)?.[0] || "")) {
+        fail("a gated platform is missing a benchmark the gate measures everywhere");
+    }
     const primaryCellTags = Array.from(page.matchAll(/<td\b(?=[^>]*\bdata-feature-cell\b)[^>]*>/gi), match => match[0]);
     for (const cell of primaryCellTags) {
         const port = attribute(cell, "data-port");

vm/selfhost/README.md

@@ -118,16 +118,35 @@ In a `--cores` sweep the count is pinned with CPU affinity on Linux and Windows.
 has no affinity API, so there it is logical: both arms are told it (`CN1_GC_MARK_THREADS`,
 `-XX:ActiveProcessorCount`) and neither is confined to it; the table marks those rows.
 
-**The gate compares against `perf-baseline.json`**: a ratio per platform and benchmark,
-and a tolerance per metric. A ratio more than the tolerance above its
-baseline fails the build, and the comment names the benchmark, the metric and the size
-of the change.
-
-**Calibrating.** Baselines must come from the runners that enforce them: a ratio depends
-on the hardware, so one measured on a developer machine is not a baseline for CI. A
-benchmark with no entry is reported as "not gated", and the comment carries the entry to
-add. When a change moves performance on purpose, update the entries in the same pull
-request, from that pull request's own run.
+**The gate compares against `perf-baseline/`**: a ratio per platform, runner CPU model
+and benchmark, and a tolerance per metric. A ratio more than the tolerance away from its
+baseline fails the build, in either direction -- above it is a regression, below it an
+improvement that has to be recorded, or a later change could give it back without
+failing anything. The comment names the benchmark, the metric and the size of the change.
+
+**Calibrating and rebaselining.** Baselines must come from the runners that enforce them:
+a ratio depends on the hardware, so one measured on a developer machine is not a baseline
+for CI. Both kinds of change go into the pull request's own
+`perf-baseline/pr/<number>.json`, written from the failing job's `perf-results.json`:
+
+```bash
+# a runner CPU model with no rows yet
+python3 vm/selfhost/calibrate-perf-baseline.py --pr 5931 perf-results.json
+# a change that moves performance on purpose
+python3 vm/selfhost/calibrate-perf-baseline.py --pr 5931 --reason "why" perf-results.json
+```
+
+No pull request edits `perf-baseline/base/`. The nightly `fold` job in
+`.github/workflows/perf-baseline.yml` moves merged overlays there. The one-file layout
+this replaced made unrelated branches conflict on every merge; `perf_baseline.py`
+explains how overlays combine and when two of them are a real conflict. A branch still
+carrying edits to the old `perf-baseline.json` converts them, from a checkout of this
+layout, with `perf_baseline.py import-legacy --pr N --reason "..." --ref origin/<branch>`.
+It reads the branch's copy and the copy at its merge base, so only the branch's own edits
+are imported.
+
+The Port Status page's ParparVM vs JDK 25 table is rendered from these same rows
+(`perf_baseline.py summary`, run by `scripts/website/build.sh`).
 
 ## Native collection and string implementation