Skip to content

Make the CUDA build compile under nvcc - #254

Merged
zfergus merged 7 commits into
mainfrom
feature/cuda-build
Sep 8, 2026
Merged

Make the CUDA build compile under nvcc#254
zfergus merged 7 commits into
mainfrom
feature/cuda-build

Conversation

@zfergus

@zfergus zfergus commented Sep 8, 2026

Copy link
Copy Markdown
Member

Description

Makes the existing IPC_TOOLKIT_WITH_CUDA=ON configuration compile under nvcc. No CUDA code is added — this is the build layer the GPU work sits on, split out so it can be reviewed on its own.

  • fmt — nvcc's device front-end (EDG) rejects two things in the fmt that spdlog bundles: a compile-time /utf-8 probe that misfires because the front-end never sees the flag, and a char32_t table using hex escapes with the high bit set. Neither is reachable from a compiler flag, so both are patched behind __CUDACC__ via CPM's PATCHES.
  • Flag leaks — MSVC-style flags passed bare are read by nvcc as input file names. /FS and the SIMD /arch flags now carry a COMPILE_LANGUAGE guard. CMake already forwards /FS to the host compiler for CUDA targets via -Xcompiler, and device code has no use for CPU SIMD, so neither needs a CUDA equivalent.
  • xsimd — the include and the batch-type block in ipc/utils/simd.hpp are guarded on __CUDACC__; the batch types and their operations are host-only.
  • --expt-relaxed-constexpr — device code reaches std::numeric_limits and std::array, whose members are constexpr but not __device__. PUBLIC so downstream CUDA consumers, the tests included, inherit it.
  • Dev container.devcontainer/cuda/ mirrors the existing .devcontainer/, based on NVIDIA's CUDA devel image so nvcc is available. build-cuda.sh is the batch counterpart: it builds the same image and compiles the project with CUDA on, so a machine with no NVIDIA GPU can check the CUDA build. The source is mounted read-only and rsynced into a named volume, so a run never writes into the host tree.

Dependency bumps

Both are existing dependencies moving to a newer pin — no new packages, same fetch mechanism (CPM), same licenses.

  • TinyAD — guards /bigobj by compile language so a bare flag no longer leaks into nvcc.
  • Scalable CCD — fixes the MSVC + CUDA 13.3 build and bumps its own dependencies.

Effect on the default (CUDA off) build

Most of this is inert without nvcc, but three pieces are not, and are worth a reviewer's attention:

  • The two dependency pins change for every configuration.
  • The spdlog include moves ahead of the CCD dependencies. Scalable CCD carries its own spdlog recipe guarded on if(TARGET spdlog::spdlog), so whichever recipe runs first wins — and if that is Scalable CCD's, the unpatched spdlog is the one CPM registers and the patch never applies. Reordering changes which recipe registers spdlog in all builds, not just CUDA ones.
  • target_compile_options(ipc_toolkit PUBLIC ${SIMD_CXX_FLAGS}) becomes PUBLIC "$<$<COMPILE_LANGUAGE:CXX>:${SIMD_CXX_FLAGS}>". A C++-only build should receive exactly the same flags, but the list now expands inside a generator expression, so this is the one change that could silently alter the default build rather than fail loudly.

How Has This Been Tested?

  • Full CUDA build with nvcc 13.3, CMAKE_CUDA_ARCHITECTURES=86 — 525/525 targets including the device-link step. This exercises everything this PR adds: the patched fmt is what lets nvcc parse spdlog at all, --expt-relaxed-constexpr is what lets device code reach std::numeric_limits/std::array, and the flag guards are what keep -march=native out of nvcc's argument list. Built at the tip of the stack (Add the host/device shared source mechanism #255, Make the low-level math callable from device code #257), which contains this PR — not standalone.
  • CI CUDA (Debug) and CUDA (Release) both pass, building library, tests, and Python bindings with nvcc on x86 Linux.
  • Full CPU test suite on Linux x86 with clang 22: 4,189,135 assertions in 324 test cases, all passing. This confirms the SIMD_CXX_FLAGS generator expression still delivers the flags — a list expanding inside a genex could have silently dropped them rather than failing loudly, and -march=native is present on the compile line.
  • All other CI checks pass (Linux/macOS/Windows Debug+Release, Python 3.14, Clang-Tidy, formatting).

Not tested:

  • The MSVC-specific guards. /FS and the SIMD /arch flags only exist for MSVC, and nothing in a Linux container or in CI reaches them — there is no MSVC + CUDA job. Those two hunks are reviewed by reading.
  • build-cuda.sh itself. The CUDA build above was run natively on a Linux box with the toolkit installed, not through the container. The Dockerfile and the script remain unexercised.

Note for reviewers: the Scalable CCD bump

Running the GPU tests turned up an order-dependent illegal memory access in Scalable CCD's narrow phase (cuda/narrow_phase/root_finder.cu), reached by the pre-existing test_gpu_ccd tests. It poisons the CUDA context, so every later cudaMalloc in the process returns 700.

This bump does not cause it. I built both pins and ran four fixed seeds against each; the failure counts are identical on c80af01 (before) and 8f9347c (after). The root cause is that MAX_UNIT_SIZE is sized from currently-free GPU memory while root_finder.cu sets m_tail from d_data.size() independently, so when memory is tight the buffer's tail exceeds its capacity. It is pre-existing, upstream, and deserves its own issue. CI has never caught it because cuda.yml builds but never runs the tests — the runners have no GPU.

Test Configuration:

  • OS and Version: Arch Linux x86_64 (AMD Ryzen 9 5900XT, AVX2) and macOS 26.6 (arm64)
  • Compiler and Version: clang 22.1.8, gcc 16.2.1, nvcc 13.3 (V13.3.73), AppleClang 21.0; CMake 4.4.2, CMAKE_BUILD_TYPE=Release
  • GPU: NVIDIA GeForce RTX 3070 (compute capability 8.6)

Checklist

  • I have followed the project style guide
  • My code follows the clang-format style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@codecov

codecov Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.71%. Comparing base (ecba377) to head (415a8bb).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #254   +/-   ##
=======================================
  Coverage   96.70%   96.71%           
=======================================
  Files         191      191           
  Lines       17294    17295    +1     
  Branches      927      928    +1     
=======================================
+ Hits        16724    16726    +2     
+ Misses        570      569    -1     
Flag Coverage Δ
unittests 96.71% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The dev container covers interactive work; build-cuda.sh is the batch
counterpart, building the same image and compiling the project with CUDA
enabled so a machine without an NVIDIA GPU (or CI) can keep the CUDA build
green. The source is mounted read-only and rsynced into a named volume, so a
run never writes into the host working tree and ninja can still build
incrementally.
nvcc's device front-end (EDG) chokes on two things in the fmt that spdlog
bundles: a compile-time /utf-8 probe that misfires because the front-end never
sees the flag, and a char32_t table using hex escapes with the high bit set.
Neither is reachable from a compiler flag, so we patch both behind __CUDACC__.

The spdlog include also moves ahead of the CCD dependencies. Scalable CCD
carries its own spdlog recipe guarded on `if(TARGET spdlog::spdlog)`, so
whichever runs first wins -- and if that is Scalable CCD's, the unpatched
spdlog is the one CPM registers and the patch never applies.
- TinyAD: guards /bigobj by compile language so a bare flag no longer leaks
  into nvcc, which would treat it as an input file.
- Scalable CCD: fixes the MSVC + CUDA 13.3 build and bumps its own
  dependencies.
MSVC-style flags passed bare are interpreted by nvcc as input file names, so
/FS and the SIMD /arch flags now carry a COMPILE_LANGUAGE guard. CMake already
forwards /FS to the host compiler for CUDA targets via -Xcompiler, and device
code has no use for CPU SIMD, so neither needs a CUDA equivalent.

The xsimd include in simd.hpp is guarded on __CUDACC__ for the same reason:
the batch types and their operations are host-only.
Device code reaches std::numeric_limits and std::array, whose members are
constexpr but not marked __device__. --expt-relaxed-constexpr lets nvcc accept
them. PUBLIC so downstream CUDA consumers, the tests included, inherit it.
@zfergus
zfergus merged commit 2b4c21e into main Sep 8, 2026
26 of 27 checks passed
@zfergus
zfergus deleted the feature/cuda-build branch September 8, 2026 20:37
@zfergus zfergus added this to the v2.0.0 milestone Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant