Harden the test suite for memory-constrained GPUs#596
Open
michel2323 wants to merge 2 commits into
Open
Conversation
The proactive-GC heuristic in the memory pool computes its pressure thresholds against the device's total memory, but _allocated_bytes only tracks the current process, and GC in one process cannot free buffers held by another. When several processes share one device — the parallel test suite being the prime example — each process happily grows to 80% of the card before feeling any pressure, so their combined footprint exhausts physical memory and allocations start failing with OutOfGPUMemoryError (hundreds of such errors on an 8GB Arc A750 with --jobs=2). Introduce ONEAPI_MEMORY_LIMIT (a byte count, or a percentage of device memory like "50%") as a per-process soft budget: _maybe_gc now scales its 40%/80% thresholds by the limit instead of the full card. This is deliberately soft — oversized single allocations still go through, with the existing retry_reclaim ladder as the reactive backstop.
Three changes, motivated by a full-suite run on an 8GB Arc A750 with --jobs=2 failing 520 tests (366 of them OutOfGPUMemoryError, cascading across ~20 test files): - give each test worker an equal ONEAPI_MEMORY_LIMIT share of device memory, so the pool's proactive GC starts collecting before the workers collectively exhaust the card; - pass recycle_on_failure/retries to ParallelTestRunner (once it supports them, >= 2.7): a failing test file can leave its worker — or the driver — in a state where every subsequent allocation fails, so workers are recycled on failure and failed files get one retry, sequentially, on a fresh worker with an otherwise-idle device; - load BFloat16s.jl on the workers (Core.BFloat16 alone lacks the host-side conversions and rand sampling the testsuite's CPU reference path needs; this alone accounted for ~150 errors), and only include BFloat16 in the generic test element types when the host-side support is actually complete — probed at runtime, so the eltype enables itself once BFloat16s.jl implements the missing div/rem family. Device-side BFloat16 (the bfloat16.jl example) is exercised regardless.
Contributor
|
Your PR requires formatting changes to meet the project's style guidelines. Click here to view the suggested changes.diff --git a/test/runtests.jl b/test/runtests.jl
index 37bf165..7e8c621 100644
--- a/test/runtests.jl
+++ b/test/runtests.jl
@@ -186,5 +186,7 @@ else
(;)
end
-runtests(oneAPI, args; testsuite, init_code, init_worker_code, env = worker_env,
- failure_handling...)
+runtests(
+ oneAPI, args; testsuite, init_code, init_worker_code, env = worker_env,
+ failure_handling...
+) |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #596 +/- ##
=======================================
Coverage 79.06% 79.06%
=======================================
Files 50 50
Lines 3362 3377 +15
=======================================
+ Hits 2658 2670 +12
- Misses 704 707 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running the full suite on an 8GB Arc A750 with
--jobs=2fails 520 tests on current main: 366OutOfGPUMemoryErrors cascading across ~20 test files (all of which pass individually), plus ~150 host-side BFloat16 errors. Three changes:ONEAPI_MEMORY_LIMIT(new, documented): a per-process soft memory budget — a byte count or a percentage like"50%"— that scales the pool's proactive-GC thresholds._allocated_bytesonly tracks the current process and GC in one process cannot free another's buffers, so parallel test workers each budgeting the whole card collectively exhaust it before any of them feels pressure. The test runner now gives each worker an equal share.Failure containment: workers are recycled after a failing file and failed files are retried once, sequentially, on a fresh worker with an otherwise-idle device — a failing test can leave the worker (or driver) in a state where every subsequent allocation fails, and a retry on an idle device distinguishes resource-contention casualties from real failures. This uses the
recycle_on_failure/retriesoptions proposed in Add recycle_on_failure and retries options to runtests JuliaTesting/ParallelTestRunner.jl#148 and is version-gated, so it is inert until a ParallelTestRunner release ships them.BFloat16 host support: load BFloat16s.jl on the workers (Core.BFloat16 alone lacks the conversions and rand sampling the testsuite's CPU reference path needs), and only include BFloat16 in the generic test element types when host-side support is complete — probed at runtime, so the eltype enables itself once BFloat16s.jl implements the missing div/rem family. Device-side BFloat16 (the bfloat16.jl example) is exercised regardless.
On the A750 this takes the suite from 10167 pass / 7 fail / 520 errors to 11751 pass / 1 fail / 4 errors with zero OOM (the residual failures are the atomics bug fixed in #595 and a local-toolchain onemkl version check, also addressed there); one file that failed under concurrent memory pressure was rescued by its idle retry. Independent of #595; the two combine cleanly.