Skip to content

Fix CI on Julia 1.13 and nightly - #847

Merged
devmotion merged 12 commits into
mainfrom
dmw/fix-ci-julia-1.13
Oct 16, 2026
Merged

devmotion merged 12 commits into
mainfrom
dmw/fix-ci-julia-1.13

Conversation

@devmotion

@devmotion devmotion commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

CI has been failing on every PR since early 2026: GitHub's julia: 1 now resolves to Julia 1.13, and nightly to 1.14-DEV. Root causes, each traced to an upstream change:

1. All @gpu tests error on Julia ≥ 1.13.0-DEV.300 (224 errors)

_gpu_macro spliced the function value _gpu_test into the @test call. Since JuliaLang/julia#57839 (d934b032ea5b, 1.13.0-DEV.300), @test f(args...) lowers to Test.eval_test_function(f, args, kwargs, quoted_func::Union{Expr,Symbol}, …), so every @gpu test threw a MethodError (the CI log shows "Test threw exception" with no message). The fix splices the name instead.

2. GPU tests need JLArrays 0.4 (supersedes #846)

Once the @gpu tests actually run, muladd (3×3), Diagonal * Matrix, and tr hit "Scalar indexing is disallowed" on 1.13. JLArrays 0.1 caps GPUArrays at 10.3.1, which predates Julia ≥ 1.12's LinearAlgebra changes: tr(::StridedMatrix) scalar loop, the 2×2/3×3 matmul3x3! path, and Diagonal mul!. GPUArrays 12.1 (JLArrays 0.4.2) overrides these. Compat is now JLArrays = "0.1, 0.4" (Julia 1.6 still resolves 0.1).

The vararg * broadcast @gpu_broken test now passes on 1.13. Checked locally: it is still broken on 1.6, 1.10, 1.11, and 1.12, and passes on 1.13 with both GPUArrays 10.3.1 and 12.1.0. It's now gated on VERSION >= v"1.13".

JLArrays 0.4 also implements reverse without scalar indexing, so the reverse @gpu_broken tests now pass with it. They're gated on parentmodule(reverse, Tuple{JLArray{Float32,1}}) !== Base.

3. Matrix functions of Hermitian{<:Real} (194 failures)

Since JuliaLang/julia#55951 (4d2184159b15, 1.12.0-DEV.1322), real-valued f(::Hermitian{<:Real}) returns Hermitian, not Symmetric. _matfun now matches the primal type under that exact version bound. All 15 matrix-function rules and sincos go through _matfun, so this is the single place to change. The equivalent test gate lives in one helper. This supersedes #840/#833.

Since JuliaLang/LinearAlgebra.jl#1359 (44b6bc177f, 1.13.0-DEV.887), exp and positive semi-definite sqrt are computed as (U * √D) * (U * √D)', not as U * D * U'. The primal then differed from f(A) by roundoff (36 Y_ad == Y failures). _matfun now reproduces Base's computation exactly. Checked locally: the primal is bitwise identical to f(A) for all 15 functions on 1.6, 1.12, 1.13 and 1.14-DEV.

4. Zygote integration (Threads.@spawn)

On 1.13, Threads.@spawn calls Threads._spawn_set_thrpool(::Task, ::Symbol), which wraps a foreigncall that Zygote can't differentiate. It's now marked @non_differentiable (guarded with isdefined, following the existing pattern in nondiff.jl). Verified by the IntegrationTest/Zygote job: the structures.jl foreigncall errors are gone. A guarded test in base.jl checks that the rule is defined.

5. JuliaInterpreter: nightly precompile crash, plus world age

JuliaInterpreter 0.9.46 (capped by the test-only compat 0.8,0.9) fails to precompile on 1.14-DEV. Compat is now "0.9, 0.12": only the minors CI actually resolves (1.6 → 0.9.x, 1.13/nightly → 0.12).

JuliaInterpreter 0.12 respects world age, which exposed two things:

  • Tests that did @eval using … / @eval struct … inside an interpreted @testset couldn't see the new bindings. They now use local bindings or a top-level struct.
  • The sortslices rrule was never inferrable; 0.9 hid this. Its pullback captured an index tuple whose type depends on the value of dims, and it now rebuilds that tuple inside the pullback (via a _sortslices_inds helper shared with the frule).

6. Upcoming LinearAlgebra changes (nightly)

7. Workflows never ran on main

CI, IntegrationTest, and JuliaNightly triggered on push: branches: [master], but the default branch is main.

8. Testsets run compiled on Julia 1.14 (nightly)

JuliaInterpreter 0.12.0 (latest release) doesn't support 1.14-DEV. It fails on a new foreigncall form ("ccall function name tuple can have at most 2 elements"; 1420 errors per job). It also looked up methods with Type{T} signatures, so it missed rules that ChainRulesCore's macros defined with Core.TypeEgal{T} self-types: rrule(Float64, 1.2) returned nothing when interpreted. ChainRulesCore 1.26.2 (JuliaDiff/ChainRulesCore.jl#714) uses Type{T} again, but the foreigncall failure remains. On VERSION >= v"1.14.0-DEV" the testsets now run compiled.

9. lu rules for singular matrices

Since JuliaLang/LinearAlgebra.jl#1529 (Julia 1.14), BLAS triangular solves throw on a zero diagonal. On ≤ 1.13 they returned Inf/NaN. So the lu! frule threw a SingularException for singular A with check=false, because it divided by all of U.

For square and wide A, the factors are differentiable as long as only the last pivot is zero. The rules computed L \ (P * ΔA) / U, which uses U[q, q] even though the derivative doesn't depend on it.

  • Square and wide A: the frule and the lu pullback now share one branch that only inverts the leading (q-1)×(q-1) block of U. This gives the correct derivative when the last pivot is zero.
  • No derivative: the rules return NaN, matching the previous behaviour with check=false. That's an earlier zero pivot, which makes L discontinuous, or a zero last pivot of tall A.
  • Detecting the cases: F.info gives the index of the first zero pivot. It is negated for unpivoted LU since LU factorization: add allowsingular keyword argument JuliaLang/julia#52957, so the rules negate it back under the version bound 1.11.0-DEV.1535.

Compared with master, the new rules have the same allocations and are within ±7% of its speed (n = 10 to 500). They're slightly more accurate against a BigFloat reference.

New tests run test_frule/test_rrule at matrices whose last pivot is exactly zero: square and wide shapes, RowMaximum and NoPivot. The test matrices use dyadic L with |L[i, j]| < 1 and power-of-two pivots, so LAPACK factors them exactly. Further tests check for NaN on tall A with a zero last pivot and on zeros(n, n). The existing check=false rrule test never ran the pullback, because its tangent had no factors.

Known remaining failures (not fixable here)

  • IntegrationTest/Zygote: Zygote's gradcheck_p2.jl hard-codes @test … broken = VERSION >= v"1.12" && MT <: Hermitian{Float64} for matrix functions and sincos. With fix 3 these now pass, so they report 17 "Unexpected Pass". Zygote needs to drop those broken flags.

JuliaNightly is green: the 1.14 ambiguities failure (@non_differentiable Array(::AbstractArray{Bool}) vs rrule(::Type{Array}, ::HermOrSym), JuliaDiff/ChainRulesCore.jl#713) is fixed in ChainRulesCore 1.26.2.

Version bumped to 1.73.1.

🤖 Generated with Claude Code


Closes #840. Closes #833. Closes #842. Closes #845. Closes #846. Closes #828. Closes #817.

devmotion and others added 11 commits October 14, 2026 10:38
- `@gpu` test helper: splice the name `_gpu_test` instead of the function
  value. Since Julia 1.13.0-DEV.300 (JuliaLang/julia#57839) `@test` lowers to
  `eval_test_function(..., quoted_func::Union{Expr,Symbol}, ...)`, so every
  `@gpu` test errored with a `MethodError`.
- `_matfun`: real-valued functions of `Hermitian{<:Real}` return `Hermitian`
  since Julia 1.12.0-DEV.1322 (JuliaLang/julia#55951); match the primal type.
- Mark `Threads._spawn_set_thrpool` non-differentiable (hit by
  `Threads.@spawn` in Zygote on Julia 1.13).
- Test-only compat: `JuliaInterpreter = "0.9, 0.12"`; 0.9 no longer
  precompiles on nightly.
- Workflows: run on pushes to `main` (was `master`, which doesn't exist).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Test-only compat: `JLArrays = "0.1, 0.4"`. With `@gpu` tests running
  again, GPUArrays 10 (JLArrays 0.1) hits scalar indexing in Julia >= 1.12
  LinearAlgebra (`tr(::StridedMatrix)`, 3x3 `matmul3x3!`, Diagonal `mul!`);
  GPUArrays 12.1 (JLArrays 0.4) overrides these.
- Tests: avoid `@eval using`/`@eval struct` inside interpreted testsets.
  JuliaInterpreter 0.12 respects world age, so those bindings were invisible.
- `sortslices` rrule: don't capture the index tuple, whose type depends on
  the value of `dims`; the rrule is now inferrable.
- broadcast.jl: vararg `*` broadcast works on GPU since Julia 1.13
  (still broken on 1.6, 1.10, 1.11, 1.12).
- `normalize` rrules: replace `LinearAlgebra.__normalize!`, removed in
  JuliaLang/LinearAlgebra.jl#1634, by a local copy of the upstream logic.
- structured.jl: skip `Adjoint(::Adjoint)`/`Transpose(::Transpose)` where
  they throw (JuliaLang/LinearAlgebra.jl#1739).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…s 0.4

Since JuliaLang/LinearAlgebra.jl#1359 (Julia 1.13.0-DEV.887), `exp` and
positive semi-definite `sqrt` of `Hermitian`/`Symmetric` matrices are
computed as `(U * √D) * (U * √D)'`, so `_matfun` no longer reproduced
`f(A)` exactly.

JLArrays 0.4 implements `reverse` without scalar indexing, so the
`@gpu_broken` tests passed unexpectedly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JuliaInterpreter 0.12 fails on the new `foreigncall` form and looks up
methods with `Type{T}` instead of `Core.TypeEgal{T}` signatures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With JuliaLang/LinearAlgebra.jl#1529, triangular solves throw for zero
diagonal entries, so the `lu!` frule failed for singular A with
`check=false` on Julia 1.14.

For square and wide A the rules now only invert the leading
(q-1)×(q-1) block of U, so they compute the derivative, which exists
if only the last pivot is zero. Otherwise, the factors are not
differentiable and the rules return NaN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
- `sortslices` rules: factor out the thrice-repeated index tuple.
- `normalize` tests: cover the subnormal-norm branch of `_normalize!`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JuliaDiff JuliaDiff deleted a comment from github-actions Bot Oct 15, 2026
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@devmotion
devmotion merged commit 3483c63 into main Oct 16, 2026
9 of 10 checks passed
@devmotion
devmotion deleted the dmw/fix-ci-julia-1.13 branch October 16, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant