DEMdemo_PlateSinkage: collect forces in the force kernel, and documen… - #80
Open
DanNegrut wants to merge 1 commit into
Open
DEMdemo_PlateSinkage: collect forces in the force kernel, and documen…#80DanNegrut wants to merge 1 commit into
DanNegrut wants to merge 1 commit into
Conversation
…t the measured gain
By default DEME accumulates contact forces onto their owners in a separate pass over
the contact array (forceToAcc) after the force kernel has run. SetCollectAccRightAfterForceCalc
folds that accumulation into the force kernel itself. Its API comment said this "may give
some performance boost if you have only polydisperse spheres, no clumps"; measured on this
demo, which uses 3-sphere clumps, it is the largest single performance lever in the code,
so the demo now turns it on and the comment now states what was measured.
Method: identical source on both machines, fixed step 5e-6 s (about 310,000 steps at
every size), one GPU each, throughput taken over the physics loops only so JIT compilation
is excluded. Three random bed packings per cell, eleven for the two 152,670-grain default
cells. Million grain-steps per second:
grains Blackwell default Blackwell in-kernel MI350X default MI350X in-kernel
152,670 142.1 171.4 (+21%) 142.0 278.8 (+96%)
308,486 134.5 166.3 (+24%) 137.1 335.5 (+145%)
613,998 128.5 155.0 (+21%) 124.6 378.3 (+204%)
Blackwell is an RTX PRO 6000 workstation card under CUDA; MI350X is under ROCm 7.2 via
the HIP build. On the default path the two tie within 2%; with in-kernel collection the
MI350X leads by 1.6x to 2.4x. The reason is visible in the per-phase timers: the separate
forceToAcc pass is 18% of dT time on CUDA and 83% on HIP, and on HIP it also slows the
concurrent kT phases (binning, pair search, history map) by 2.5x to 2.9x, presumably
through memory-system contention, since kT's code is untouched by this option.
The other alternative, UseCubForceCollection, was measured too and is slower than the
default on both backends (-55% to -59% on CUDA at every size; -26% to +69% on HIP), so
in-kernel collection is the right choice here rather than one of three.
Physics is essentially unchanged. Plate pressure at 10 mm sinkage shifts between the two
collection modes by 0.8% to 6.0% across the six machine-and-size cells, against a
bed-to-bed scatter of 4% to 19% from reseeding alone, so the shift is at or below the
test's own repeatability. In-kernel reads slightly higher in five of the six cells, which
is what a different atomic accumulation order rounding differently would look like.
What in-kernel collection gives up: tracker force-pair queries throw while it is on
(DEMTracker::assertThereIsForcePairs). This demo reads the plate load through
ContactAcc, which goes via GetOwnerAcc and is not guarded, and contact-force recording
and output are unaffected in general; the comment now says so, since the previous text
did not mention it.
Whether the library default should change is left to the maintainer; this commit only
changes the demo and the documentation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…t the measured gain
By default DEME accumulates contact forces onto their owners in a separate pass over the contact array (forceToAcc) after the force kernel has run. SetCollectAccRightAfterForceCalc folds that accumulation into the force kernel itself. Its API comment said this "may give some performance boost if you have only polydisperse spheres, no clumps"; measured on this demo, which uses 3-sphere clumps, it is the largest single performance lever in the code, so the demo now turns it on and the comment now states what was measured.
Method: identical source on both machines, fixed step 5e-6 s (about 310,000 steps at every size), one GPU each, throughput taken over the physics loops only so JIT compilation is excluded. Three random bed packings per cell, eleven for the two 152,670-grain default cells. Million grain-steps per second:
Blackwell is an RTX PRO 6000 workstation card under CUDA; MI350X is under ROCm 7.2 via the HIP build. On the default path the two tie within 2%; with in-kernel collection the MI350X leads by 1.6x to 2.4x. The reason is visible in the per-phase timers: the separate forceToAcc pass is 18% of dT time on CUDA and 83% on HIP, and on HIP it also slows the concurrent kT phases (binning, pair search, history map) by 2.5x to 2.9x, presumably through memory-system contention, since kT's code is untouched by this option.
The other alternative, UseCubForceCollection, was measured too and is slower than the default on both backends (-55% to -59% on CUDA at every size; -26% to +69% on HIP), so in-kernel collection is the right choice here rather than one of three.
Physics is essentially unchanged. Plate pressure at 10 mm sinkage shifts between the two collection modes by 0.8% to 6.0% across the six machine-and-size cells, against a bed-to-bed scatter of 4% to 19% from reseeding alone, so the shift is at or below the test's own repeatability. In-kernel reads slightly higher in five of the six cells, which is what a different atomic accumulation order rounding differently would look like.
What in-kernel collection gives up: tracker force-pair queries throw while it is on (DEMTracker::assertThereIsForcePairs). This demo reads the plate load through ContactAcc, which goes via GetOwnerAcc and is not guarded, and contact-force recording and output are unaffected in general; the comment now says so, since the previous text did not mention it.
Whether the library default should change is left to the maintainer; this commit only changes the demo and the documentation.