[jwrosewell]

Problem

A pull request into rc runs all 15 configurations in ci/options.json, whatever it changes. Six of them are macOS, and only five macOS jobs run at once across the whole organisation. Every configuration runs the three IP Intelligence tests that load the multi-gigabyte Enterprise data file, every Windows configuration installs WMIC, which is only there for one of those tests, and fourteen configurations run the Selenium contract, the flakiest check in the window.

Cost, measured from run history

• The one build since 2 November where every job passed first time (run 41531885450, #22) waited 109 minutes from the first job queued to the last finished and used 601 runner minutes.
• The change to comments, a message and a link in #32 (run 41708521029) took 143 minutes and 1,006 runner minutes over three attempts, because a Mockito failure on Ubuntu Java 17 and a Selenium timeout on macOS Intel Java 17 each cancelled the rest.
• In Ubuntu Java 17, 25 minutes, IPIntelligenceTests, MetadataOnPremTest and CompareOnPremTest took 12.5 of the 16.7 unit test minutes. In Windows Java 17, 56 minutes, WMIC took 12.5 minutes to install and the same three tests 22.7.
• Four by-hand runs on 2 November found no eligible pull request and built rc in full, using 1,234 runner minutes. Configurations cancelled when another failed threw away 1,148 runner minutes in the window.
• On 4 November the nightly (run 41790027166) built pull request Update submodules #38 on all 15 configurations, starting its six macOS jobs between 07:29 and 08:10, before the publish began at 08:59. The publish's macOS jobs then waited 18 to 54 minutes for a runner.

Change

1. A new ci/select-options.ps1 chooses the configurations for common-ci's selection step, which a companion common-ci change adds.
   • Standard set, for every change. Ubuntu_Java_17 and Ubuntu_Java_8, the main JDK and the oldest, with the Selenium contract on Ubuntu_Java_17 alone.
   • A change to a native module or the cxx submodule, in more than its comments, adds Windows_Java_17 and Ubuntu_ARM_Java_17, where the native library is built differently.
   • A pom.xml change adds Ubuntu_Java_11 and Ubuntu_Java_21.
   • A change outside the package trees, such as ci, adds every configuration but macOS.
   • Only when rc itself or a release is built do the six macOS configurations and the performance tests run.
2. At the standard depth, ci/run-unit-tests.ps1 leaves out the three tests in the new ci/standard-excludes.txt through Surefire's surefire.excludesFile, and ci/setup-environment.ps1 does not install WMIC. A change to IP Intelligence or the cxx submodule runs them.
3. ci/run-integration-tests.ps1 runs the Selenium contract only where the selection leaves SeleniumContract true, which it is by default.
4. The Pull Requests workflow passes fail-fast: false and build-without-pull-requests: false, and gains a full-matrix box for a full build of a pull request by hand.
5. The nightly runs its publish first and builds pull requests after it, as the publish builds main, which the pull requests into rc do not change.
6. A new release-pull-request.yml builds the release pull request from rc into main on every configuration, as a dry run that merges nothing.
7. ci/README.md describes the selection.

Expected saving

From the job times above, by the kind of change.

Change	Configurations	Wait	Runner minutes
Today, any change	15	109	601
Documentation alone	0	0	0
Code or comments in a package tree, outside IP Intelligence	2	about 12	about 20
IP Intelligence	2	about 24	about 45
A pom.xml	4	about 12	about 43
A native module or the cxx submodule	4	about 55, set by Windows	about 120
CI or other files outside the package trees	9	about 65	about 358

The reduced times take the measured minutes of the three tests, WMIC and performance off the median of each configuration. Applied to the changed files of the 26 pull requests opened since 1 November, many of them CI set-up, the median wait falls from 109 to about 40 minutes and the mean runner minutes from 601 to about 151. Over 2 and 3 November the change, the fail-fast setting and the by-hand builds of rc it stops come to about 2,300 runner minutes a day. The release pull request adds one full build, about 601 runner minutes, for each release, and no pull request build takes a macOS runner.

How it was checked

• Every changed PowerShell file parses, and every changed workflow parses as YAML. The jobs moved in nightly-pipeline.yml are identical to the old ones apart from their order, needs, if and the new input.
• The class names and packages in ci/standard-excludes.txt are the ones the Ubuntu Java 17 log reports, and the Device Detection MetadataOnPremTest is not matched. Surefire 2.22.2, which the poms name, reads the file from the surefire.excludesFile property and ignores its # lines (SurefirePlugin.java). A file of excludes replaces Surefire's own **/*$* default (AbstractSurefireMojo.java), so the file lists it too.
• The selection rules were applied, in a copy, to the changed files of every pull request since 1 November, which gives the table above.
• With today's common-ci, which never calls ci/select-options.ps1, nothing in a build changes.
• Not built yet. The workflow changes need the common-ci change on common-ci main, and builds are held while today's package releases use the runners. One temporary build of this branch against the common-ci branch is the next step.

What it gives up

On a pull request, macOS, JDK 11 and 21 unless a pom.xml changes, the Windows and ARM native builds unless native code changes, the three IP Intelligence tests unless IP Intelligence changes, thirteen of the fourteen Selenium runs, and performance. A fault that shows only there is found by the nightly build of rc or by the release pull request build, before anything reaches main.

Merge order

After the common-ci change (branch pull-request-depth) is released to common-ci main. This can merge into rc before that, but rc must not be released into main until then, because the workflows pass inputs that common-ci main does not have yet.

What remains

• The "Cloud 51Did Tests" filter their pull_request trigger on main alone, so no pull request into rc has run them since changes moved to rc. A second commit adds rc, and is pushed once today's package releases are out, because changing that workflow runs it on this pull request.
• The cause of the Mockito failure in DataUpdateServiceTests.DataUpdateService_Register_AutoUpdateConfiguredInterval and of the Selenium WebDriver timeouts, neither established.
• The nightly still starts at 03:00. The Node and Python nightlies move away from that time in their own pull requests.