Skip to content

Run tests with ParallelTestRunner instead of JuliaInterpreter - #848

Merged
devmotion merged 7 commits into
mainfrom
dmw/parallel-test-runner
Oct 19, 2026
Merged

devmotion merged 7 commits into
mainfrom
dmw/parallel-test-runner

Conversation

@devmotion

@devmotion devmotion commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

An experiment to replace the JuliaInterpreter-based test setup (from #500) with plain compiled tests run in parallel by ParallelTestRunner.jl.

Commits

  1. Require Julia 1.10 and remove pre-1.10 code paths. ParallelTestRunner needs Julia ≥ 1.10, so this raises julia compat to 1.10 (the current LTS) and bumps the version to 1.74.0. It removes version checks that are now always true or always false, along with the pre-1.7 logabsdet(::UmfpackLU) backport and therefore the SuiteSparse dependency. CI tests lts instead of 1.6.
  2. Run tests with ParallelTestRunner.
    • Each test file runs in its own sandbox module on a worker pool and loads the packages it uses itself.
    • The shared helpers (Multiplier, @gpu, …) move into a ChainRulesTestHelpers module that is loaded once per worker, together with the Adapt piracy for the mock GPU array.
    • fastmath_able.jl no longer needs its own @interpret wrapper.
  3. Apply format suggestions to the new test/testutils.jl and to the lines changed in this PR.
  4. Upload coverage to Codecov in the CI and nightly workflows, as in ChainRulesCore.jl. The README now shows a Codecov badge and a CI badge for main, and no longer has the Travis badge.
  5. Check depwarn rules and wider-γ 3-arg * gradients in tests. The Base.depwarn rule tests now use @test_deprecated, so the deprecation warning no longer shows up in the test output. Three gradient comparisons in arraymath.jl that were never wrapped in @test are now actually checked.
  6. Set workflow concurrency and cache permissions. CI, nightly and integration tests cancel superseded PR runs with concurrency, as in ChainRulesCore.jl, which replaces the Cancel workflow. CI and nightly grant actions: write and otherwise only contents: read, so julia-actions/cache can delete its old caches without relying on the repository's default token permissions.
  7. Skip non-finite power gradients in FastMath tests. FastMath functions assume that no value is NaN or Inf, so non-finite results of the FastMath ^ rules are undefined. Since LLVM 23 (Upgrade to LLVM 23 JuliaLang/julia#63626), they are folded to arbitrary values, which made the now compiled POWERGRADS tests fail on nightly. In the FastMath variant, the tests no longer check expected values that are NaN or Inf.

Notes

  • test/rulesets/Core/core.jl was never included before. It is discovered and run now. Its scalar ifelse rrule test needed check_inferred=false, because the pullback's return type depends on the value of the condition.
  • Tests are now selected by path prefix rather than substring, e.g. Pkg.test("ChainRules"; test_args=["rulesets/LinearAlgebra"]). A leading ! excludes tests, and --list lists them.
  • The @gpu reverse tests in test/rulesets/Base/array.jl fail because GPUArrays 12.1.1 was released before the JLArrays version it needs (reverse(::JLArray) errors with GPUArrays 12.1.1 and JLArrays 0.4.2 (JLArrays release missing) JuliaGPU/GPUArrays.jl#817). main is affected in the same way; Mark JLArray reverse tests broken with JLArrays 0.4.2 and GPUArrays 12.1.1 #849 marks these tests broken until a fixed JLArrays is released.
  • On the previous run, the test jobs took 9–18 min, compared with 30–57 min on main. The default settings fit the runners: 4 workers on 4 vCPUs, and peak worker RSS of about 2 GB, well below the memory limits.

🤖 Generated with Claude Code

@devmotion
devmotion force-pushed the dmw/parallel-test-runner branch from 0d8ef1a to 706847f Compare October 16, 2026 13:00

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

JuliaFormatter

[JuliaFormatter] reported by reviewdog 🐶

_gpu_test(frule, xdots, f, xs...; kw...)


[JuliaFormatter] reported by reviewdog 🐶

_gpu_test(frule, xdots, f, g, xs...; kw...)


[JuliaFormatter] reported by reviewdog 🐶

Multiplier_pullback(dΩ) = (Tangent{typeof(m)}(; x = dΩ * y'), m.x' * dΩ)


[JuliaFormatter] reported by reviewdog 🐶

Multiplier_pullback_3(dΩ) = (
Tangent{typeof(m)}(; x = dΩ * (y * z)'),
m.x' * dΩ * z',
(m.x * y)' * dΩ,
)


[JuliaFormatter] reported by reviewdog 🐶

Divider_pullback(dΩ) = (Tangent{typeof(d)}(; x = -2 * dΩ * y^2 / d.x^3), 2 * dΩ * y / d.x^2)


[JuliaFormatter] reported by reviewdog 🐶

(c::Counter)(x) = x * (c.n += 1)


[JuliaFormatter] reported by reviewdog 🐶

struct NoRules; end

devmotion and others added 6 commits October 16, 2026 23:33
Raise the minimum supported Julia version to 1.10 (the current LTS) and
drop the version checks that are now always true or always false:

- inline the LU/Cholesky pivot aliases now that `RowMaximum`/`NoPivot`
  always exist
- unwrap the fused 3-arg `*` rules and the `\` array workarounds
- remove the pre-1.7 `logabsdet(::UmfpackLU)` backport and with it the
  SuiteSparse dependency
- drop the corresponding version gates and `isdefined` guards in tests

CI now tests `lts` instead of 1.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the JuliaInterpreter-based test setup, which interpreted every
top-level `@testset` via an `include` mapexpr hack and a hand-maintained
`compiled_modules` list, with plain compiled tests orchestrated by
ParallelTestRunner.jl. Each test file now runs in its own sandbox module
on a pool of worker processes.

- Every test file loads the packages it uses itself.
- The shared helpers (`Multiplier`, `@gpu`, ...) move into a
  `ChainRulesTestHelpers` module in `test/testutils.jl`, which is loaded
  once per worker together with the `Adapt` piracy for the mock GPU array.
- `test_method_tables()` moves into `test_helpers.jl`.
- `rulesets/Core/core.jl` is now discovered and run (it was never
  included before); its scalar `ifelse` rrule test does not infer
  because the pullback's return type depends on the condition's value.

Tests are now selected by path prefix, e.g.
`Pkg.test("ChainRules"; test_args=["rulesets/LinearAlgebra"])`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Format the new `test/testutils.jl` and the lines changed in this branch
with JuliaFormatter (BlueStyle).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Process coverage and upload it to Codecov in the CI and nightly
workflows, as in ChainRulesCore.jl. This replaces the commented-out
Coveralls/Codecov steps in CI and the Coveralls upload in the nightly
workflow.

Update the README badges: show Codecov coverage, point the CI badge to
`main`, and drop the obsolete Travis badge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The `Base.depwarn` frule/rrule tests call `depwarn`, which emitted a
  deprecation warning into the test output; check it with
  `@test_deprecated` instead.
- The comparisons of 3-arg `*` gradients for γ of a wider type were
  not wrapped in `@test`, so they were never checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Cancel superseded runs of the CI, nightly and integration test
  workflows on pull requests with `concurrency`, as in ChainRulesCore.jl,
  and remove the `Cancel` workflow that did this with
  `styfle/cancel-workflow-action`.
- Grant `actions: write` (and only `contents: read` otherwise) in the
  workflows that use `julia-actions/cache`, so it can delete its old
  caches without relying on the repository's default token permissions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devmotion
devmotion force-pushed the dmw/parallel-test-runner branch from 04f8576 to 2e4cb28 Compare October 16, 2026 21:33
@codecov

codecov Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@5f30885). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #848   +/-   ##
=======================================
  Coverage        ?   96.57%           
=======================================
  Files           ?       28           
  Lines           ?     3619           
  Branches        ?        0           
=======================================
  Hits            ?     3495           
  Misses          ?      124           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

FastMath functions assume no NaN or Inf, so these results are undefined.
Since LLVM 23 (JuliaLang/julia#63626) they are folded to arbitrary values,
which broke the compiled FastMath POWERGRADS tests on nightly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devmotion
devmotion marked this pull request as ready for review October 17, 2026 00:22

@devmotion devmotion left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked all changes (with whitespace changes hidden). Seems fine to me.

@devmotion
devmotion merged commit 2d84375 into main Oct 19, 2026
10 of 11 checks passed
@devmotion
devmotion deleted the dmw/parallel-test-runner branch October 19, 2026 11:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant