Skip to content

[improve][ci] Support smaller runners and restricted fork environments - #26558

Merged
merlimat merged 2 commits into
apache:masterfrom
lhotari:lh-improve-ci-portability
Sep 13, 2026
Merged

[improve][ci] Support smaller runners and restricted fork environments#26558
merlimat merged 2 commits into
apache:masterfrom
lhotari:lh-improve-ci-portability

Conversation

@lhotari

@lhotari lhotari commented Sep 12, 2026

Copy link
Copy Markdown
Member

Motivation

Pulsar CI should also run in forks with smaller runners and restrictive default workflow permissions. A 4 GB Gradle heap plus up to four test JVMs leaves too little headroom on an 8 GB runner. Private repositories also need usable defaults when external build scans or GitHub CodeQL are unavailable.

The permissions case described in #22568 still applies: enterprise policy can enforce read-only default token permissions, while individual workflows can explicitly request their required scopes.

Modifications

  • Select a smaller memory profile automatically on Linux runners with up to 8 GiB RAM: a 2 GB Gradle heap, two workers, up to two test forks, and worker recycling after 50 detected classes. Add an action input to select the profile explicitly; larger runners retain their defaults.
  • Expose test heap, fork count, worker lifetime and heap-dump location as Gradle properties. Restore heap dumps to /tmp. Preserve XML-suite, profiling and task-specific isolation settings.
  • Disable Develocity injection and build-scan publishing unless repository visibility is public. Allow repositories with GitHub Code Security enabled to opt into CodeQL using CI_ENABLE_CODEQL=true.
  • Declare the token permissions needed by each affected CI job, including PR metadata reads, scheduled-run cancellation and artifact cleanup. This adapts the approach in [improve][ci] Add explicit GitHub Actions permissions #22568 to the current workflows and avoids broad workflow-wide write grants.
  • Document the settings in CONTRIBUTING.md.

Verifying this change

  • Make sure that the change passes the CI checks.

Fork CI: lhotari/pulsar#267. The build/license and flaky-test jobs passed. The connector process system-test job failed; other jobs are still running.

Local validation:

  • assemble quickCheck spotlessCheck checkstyleMain checkstyleTest passed with a 2 GB Gradle heap and two workers.
  • 41 tests passed across ConcurrentLongHashMapTest and ConcurrentLongPairSetTest, using a 1100 MB test heap and one class per test worker.
  • Configuration checks passed for standard defaults, the smaller profile, and integration/XML-suite task overrides. Configuration-cache reuse passed.
  • Eight memory-profile cases passed in a Linux container capped at 8 GiB, including threshold boundaries, explicit selection and existing settings without a trailing newline.
  • 24 GitHub expression cases passed for build-scan visibility, access-key combinations and CodeQL opt-in. Actionlint reports the same seven pre-existing diagnostics as upstream, with no new diagnostics.
  • A cold Linux/JDK 21 build of assemble and 21 selected tests passed in a container limited to 8 GiB RAM, no swap, and two CPUs (205 tasks executed). The recorded memory peak was about 5 GiB, with zero OOM events.
  • Local review completed with no remaining findings.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations — CI runner and test JVM settings described above
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Select a smaller Gradle memory profile on 8 GiB Linux runners, expose test JVM settings, and retain heap dumps. Keep external build scans off for non-public repositories and make CodeQL optional where it requires repository enablement. Declare the token permissions used by CI jobs instead of depending on repository defaults.

Assisted-by: Codex
Distinguish candidate class batching from later TestNG group and wildcard filtering.

Assisted-by: Codex

@void-ptr974 void-ptr974 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@merlimat
merlimat merged commit d1ef957 into apache:master Sep 13, 2026
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants