Skip to content

ci: run a fixed list of modules in the GraalVM nightly - #15088

Open
blakeli0 wants to merge 1 commit into
googleapis:mainfrom
blakeli0:ci/graalvm-nightly-fixed-modules
Open

blakeli0 wants to merge 1 commit into
googleapis:mainfrom
blakeli0:ci/graalvm-nightly-fixed-modules

Conversation

@blakeli0

@blakeli0 blakeli0 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

The nightly GraalVM jobs split 248 modules over 20 Kokoro builds for each of the three GraalVM versions and run their tests as native images. Since 2023, the issues filed about these jobs show no native image defect that was specific to one generated module, apart from the memory use of the largest ones. Each of the 60 builds is also a chance for a Kokoro infrastructure failure to turn the nightly red.

This change tests nine modules instead, in one build per GraalVM version:

Module Why it stays
java-aiplatform (gRPC), java-compute (REST) The two largest generated modules. Their native image builds need the most memory
java-ftp Small REST client with google.longrunning operations, numeric enums and the Locations mixin. java-compute has none of these
java-containeranalysis, java-dns, java-notification, java-resourcemanager, java-samples, java-translate Handwritten code with integration tests, and no GraalVM job of their own

Of the 248 modules, only seven have integration tests (IT*.java), and all seven are in the list. The nightly keeps running every integration test that it runs today.

Changes

  • .kokoro/common.sh: the list, graalvm_nightly_modules, with the reason for each module.
  • .kokoro/build.sh: a graalvm-nightly job type that tests the list.
  • .kokoro/nightly/graalvm-sub-jobs/native-{a,b,c}: use the new job type, drop the sharding variables, and delete the build configs of child builds 2 to 20.
  • .kokoro/common_test.sh: checks that each listed module is in the root pom.xml and has tests that the native profile runs.

After this change, no build uses the old graalvm job type or its sharding code. That code stays for now so that this change is only the switch. A follow-up can remove it.

Why one child build remains

Each nightly job keeps one child build instead of running the tests itself. Kokoro retries a child build that it could not build from the parent job, and that also covers a child build that never started. A job without child builds gets no retry in that case.

Merge order

Merge after cl/1025754480. That change reduces each nightly job to one child build and moves it to machines with 32 GB of memory. Until it lands, the nightly still starts the other 19 child builds per GraalVM version, and this change deletes their build configs.

Testing

  • .kokoro/common_test.sh passes locally. CI does not run it for this change: the workflow that runs it only does so for changes under sdk-platform-java/ or google-auth-library-java/.
  • JOB_TYPE=graalvm-nightly .kokoro/build.sh with mvn, gcloud, terraform and jq replaced by stubs. It sets up Terraform for the five listed modules that have a .cloud directory, and runs mvn test -Pnative on the 31 submodules of the nine modules.
  • Not yet run on Kokoro. The nightly jobs are scheduled on main, and a run of this branch needs cl/1025754480 first.

b/580634620
b/580635246

The nightly GraalVM jobs split 248 modules over 20 Kokoro builds for
each of the three GraalVM versions and run their tests as native images.
Since 2023, the issues filed about these jobs show no native image
defect that was specific to one generated module, apart from the memory
use of the largest ones. Each of the 60 builds is also a chance for a
Kokoro infrastructure failure to turn the nightly red.

Test nine modules instead, in one build per GraalVM version:

- java-aiplatform (gRPC) and java-compute (REST), the two largest
  generated modules. Their native image builds need the most memory.
- java-ftp, a small REST client with google.longrunning operations,
  numeric enums and the Locations mixin. java-compute has none of these.
- java-containeranalysis, java-dns, java-notification,
  java-resourcemanager, java-samples and java-translate. They contain
  handwritten code with integration tests and have no GraalVM job of
  their own.

Of the 248 modules, only seven have integration tests (IT*.java), and
all seven are in the list. The nightly therefore keeps running every
integration test that it runs today.

The list is graalvm_nightly_modules in .kokoro/common.sh, and the new
graalvm-nightly job type tests it. After this change, no build uses the
old graalvm job type or its sharding code. That code stays for now so
that this change is only the switch, and a follow-up can remove it.

Each nightly job keeps one child build instead of running the tests
itself. Kokoro retries a child build that it could not build from the
parent job, and that also covers a child build that never started. A job
without child builds gets no retry in that case.

Do not merge before cl/1025754480. That change reduces each nightly job
to one child build and moves it to machines with 32 GB of memory. Until
it lands, the nightly still starts the other 19 child builds per GraalVM
version, and this change deletes their build configs.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new graalvm-nightly job type to run nightly GraalVM checks on a fixed list of modules, consolidating the nightly GraalVM sub-jobs by removing redundant configuration files and updating the remaining configurations. A test is also added to verify that the specified nightly modules exist in the root POM and contain native tests. Feedback was provided regarding .kokoro/common_test.sh, where using find ... | grep -q . in a pipeline under pipefail can lead to flaky test failures due to SIGPIPE when grep exits early, and the non-portable -quit flag should be avoided.

Comment thread .kokoro/common_test.sh
Comment on lines +439 to +450
local root_modules
root_modules=$(extract_pom_modules pom.xml)
local module
for module in "${graalvm_nightly_modules[@]}"; do
if [[ ! " ${root_modules} " == *" ${module} "* ]]; then
echo "graalvm_nightly_modules has ${module}, which is not a module of the root pom.xml"
exit 1
fi
if ! find "${module}" -path '*/src/test/java/*' \( -name 'IT*.java' -o -name '*ClientTest.java' \) -print -quit 2>/dev/null | grep -q .; then
echo "graalvm_nightly_modules has ${module}, which has no tests that the native profile runs"
exit 1
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using grep -q in a pipeline (find ... | grep -q .) when pipefail is enabled can lead to flaky test failures. As soon as grep -q finds a match, it exits successfully and closes the pipe. If find attempts to write further output, it will receive a SIGPIPE (exit status 141), causing the entire pipeline to fail with a non-zero exit code under pipefail.

Additionally, -quit is a GNU find extension and is not portable (e.g., it will fail on macOS/BSD find unless GNU find is explicitly installed).

To make this check robust, portable, and safe under pipefail, you can use command substitution instead of a pipeline to check if any matching files exist.

Suggested change
local root_modules
root_modules=$(extract_pom_modules pom.xml)
local module
for module in "${graalvm_nightly_modules[@]}"; do
if [[ ! " ${root_modules} " == *" ${module} "* ]]; then
echo "graalvm_nightly_modules has ${module}, which is not a module of the root pom.xml"
exit 1
fi
if ! find "${module}" -path '*/src/test/java/*' \( -name 'IT*.java' -o -name '*ClientTest.java' \) -print -quit 2>/dev/null | grep -q .; then
echo "graalvm_nightly_modules has ${module}, which has no tests that the native profile runs"
exit 1
fi
local root_modules
root_modules=$(extract_pom_modules pom.xml)
local module
local has_tests
for module in "${graalvm_nightly_modules[@]}"; do
if [[ ! " ${root_modules} " == *" ${module} "* ]]; then
echo "graalvm_nightly_modules has ${module}, which is not a module of the root pom.xml"
exit 1
fi
has_tests=$(find "${module}" -path '*/src/test/java/*' \( -name 'IT*.java' -o -name '*ClientTest.java' \) 2>/dev/null)
if [[ -z "${has_tests}" ]]; then
echo "graalvm_nightly_modules has ${module}, which has no tests that the native profile runs"
exit 1
fi

@blakeli0
blakeli0 marked this pull request as ready for review November 19, 2026 23:09
@blakeli0
blakeli0 requested a review from a team as a code owner November 19, 2026 23:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant