Harden host-side sparse index width and allocation safety - #2822
Conversation
|
After #2816, I think the follow-up direction needs a policy decision before I move #2822 or the resource-cache draft further. I currently have two related follow-ups:
The next steps depend on which CUDA sparse-backend direction the project prefers. I see four practical routes:
My technical preference is route 1, because it gives the cleanest correctness story and the cleanest future GPU-resident Krylov path. If that is too large as the immediate next step, route 2 is a reasonable limited scope for #2822. Could the maintainers confirm which direction they prefer before I mark #2822 ready or move the #2816-based resource-cache draft into the main project? |
|
I think both are going too deep into the software engineering weeds. |
## Proposed Changes This PR adds an end-to-end CUDA linear solve path: the Krylov solvers keep their host control flow, and `CSysVector` operations, the SpMV and the Jacobi preconditioner are dispatched to CUDA kernels when `ENABLE_CUDA=YES`. Transfers are explicit and owned by the object responsible for the data, with no coherency or dirty-flag tracking in `CSysVector` / `CSysMatrix`: - `CSysMatrixVectorProduct` uploads the matrix on construction - the preconditioner uploads its data in `Build()` - `CSysSolve` uploads `b` and `x` and downloads `x` in `HandleTemporariesIn/Out`, which is also where device evaluation is switched on and off - preconditioners without a device implementation (ILU, LU-SGS, Linelet, PaStiX) download the input, apply on the host, and upload the result A solve is therefore fully device resident for Identity and Jacobi preconditioners: **2 uploads and 1 download per linear system**, independent of the Krylov subspace size. Implementation notes: - `cuBLAS` for `dot` / `norm`, custom kernels for the block-LDU SpMV, the Jacobi apply, and `CSysVector` expression assignment - `CSysVector` operands are captured in expressions by value, so an arbitrary expression tree is trivially copyable into the assignment kernel; the required expression shapes are explicitly instantiated in `CSysVectorGPU.cu`, consistent with how `CSysMatrix` is instantiated - device work is issued by a single thread with the OpenMP team synchronized around it, so the GPU path is usable from inside the existing parallel regions; this is internal to the linear algebra layer - the CUDA translation units are only linked into the primal libraries, since they cannot be compiled with the CoDiPack defines; device dispatch is compiled out of the AD builds - adds `LINEAR_SOLVER_PREC= NONE` (identity) ## Related Work Follows the review direction in #2822 (show a working end-to-end GPU linear solve before splitting out infrastructure) and the implementation preferences in #2816. ## Validation - CPU vs GPU on inviscid NACA0012, 25 iterations, RTX 4070 Ti SUPER (sm_89), for FGMRES + `JACOBI`, FGMRES + `NONE`, FGMRES + `ILU` (host preconditioner path) and BCGSTAB. Results agree to the printed precision in double, and to ~6 significant figures in mixed precision. BCGSTAB agrees exactly once the linear system is converged (`LINEAR_SOLVER_ERROR=1e-10`); at loose tolerances the two paths diverge through BCGSTAB's own sensitivity, not a difference in the algebra. - Mixed, normal and single precision builds all compile and run; device dispatch confirmed live in each (kernel launch counts track the subspace size). - `OMP_NUM_THREADS=1` and `4` give bit-identical results on the GPU path. - Builds verified: primal, primal + AD + directdiff, `enable-cuda` + `with-omp`, mixed / normal / single precision. - Transfer counts measured per solve: 2 H2D + 1 D2H + 1 matrix upload for Jacobi and `NONE`, plus one download/upload pair per preconditioner application for ILU. Earlier validation of the original design (6 representative cases, `nsys` / `ncu` profiling) predates the rework of the transfer and dispatch model and should be repeated. ## PR Checklist - [x] I am submitting my contribution to the develop branch. - [x] My contribution generates no new compiler warnings (try with --warnlevel=3 when using meson). - [x] My contribution is commented and consistent with SU2 style (https://su2code.github.io/docs_v7/Style-Guide/). - [x] I used the pre-commit hook to prevent dirty commits and used `pre-commit run --all` to format old commits. - [ ] I have added a test case that demonstrates my contribution, if necessary. - [ ] I have updated appropriate documentation (Tutorials, Docs Page, config_template.cpp), if necessary. --------- Co-authored-by: Pedro Gomes <pcarruscag@gmail.com>
9eafcf7 to
0a86312
Compare
0a86312 to
780448d
Compare
pcarruscag
left a comment
There was a problem hiding this comment.
I think that the su2uint I introduced addresses this, and we have custom kernels for almost everything now, since we switched to the LDU format.
So this work is probably not needed anymore? What do you think?
I checked the current
For example, So the switch to LDU/custom CUDA kernels removes the old cuSPARSE-related concern, but it does not remove this host-side allocation/index-width issue. |
Proposed Changes
Introduce
su2_index_tas the host-side sparse index type:Use it through the host sparse-pattern path, including graph-toolbox aliases, geometry sparse-pattern accessors,
CSysMatrixLDU metadata,CSysVectordimensions, and PaStiX sparse input handling.Add checked arithmetic for sparse storage sizes and checked casts at external boundaries. CUDA device sparse indices remain
su2uint; host indices are checked before upload.This prevents silent overflow or truncation in large host-side sparse matrices. On LLP64 platforms,
unsigned longis 32-bit, so products such asnnz_blocks * nVar * nEqncan wrap before allocation. For example, a5 x 5double block matrix crosses the 32-bit unsigned limit at:The affected paths now fail explicitly instead of allocating with wrapped sizes.
Related Work
Keeps the LDU/custom-kernel direction discussed in #2838. This does not replace the current CUDA sparse backend with cuSPARSE.
Validation:
git diff --checkuvx --python 3.10 pre-commit run -a -vninja -C buildninja -C build-verify-cudaninja -C build-verify-pastixninja -C build-harness-omp-cpuperiodic2dOpenMP regression,SU2_CFD -t 2, passed at iteration 1400 within1e-4fea_topology/quick_start/settings.cfg,SU2_CFD, completed withExit SuccessPR Checklist
pre-commit run --allto format old commits.