Use a block-reduction search in TensorPrimitives.IndexOfMin/Max/MinMagnitude/MaxMagnitude - #133969
Open
Mrnikbobjeff wants to merge 3 commits into
Open
Mrnikbobjeff wants to merge 3 commits into
Mrnikbobjeff wants to merge 3 commits into
Conversation
…gnitude/MaxMagnitude IndexOfMinMaxCore tracked a running best value and a vector of indices per lane, which costs a compare and two selects per element and needs per-element-size variants so that indices fit in lanes. On AVX-512 hardware the blends of the running value also go through mask-register conversions, and IndexOfMin<int> over 65536 elements runs at about 5 GElem/s on a Zen 4 box, no faster than the .NET 10 package. Replace it with a two-pass block reduction: - Pass 1 reduces each block of 32 vectors to its best element with a pure vector loop (one load and one lane-wise Reduce per vector, two accumulators), then a horizontal Aggregate, and keeps the first block whose result beats the running result under the operator's strict Compare. - Pass 2 scans only the winning block for the first element the result does not beat, which is the operator's own tie rule (first equal element; -0 before +0 for min; the sign preference for equal magnitudes). IIndexOfMinMaxOperator<T> gains Reduce (scalar and per vector width); each operator forwards it to its aggregation operator (Vector.Min/Max/MinMagnitude/MaxMagnitude), the same operators the previous code used for its final horizontal step and bitwise match, so the tie behavior is unchanged. Because those reductions implement IEEE minimum/maximum and propagate NaN, a block containing a NaN reduces to NaN and the first NaN of that block is returned; the per-vector NaN checks are no longer needed. Indices never live in vector lanes, so the Size4Plus/Size2/Size1 variants and ElementWiseSelect are removed. Measured with BenchmarkDotNet on an AMD Ryzen 7 7800X3D (.NET 10.0.5, N = 65536 ints): IndexOfMin<int> goes from 5.1 GElem/s to 35.6-36.7 GElem/s on increasing, random, decreasing and sparse-hit inputs, which is the L2 bandwidth ceiling for that working set. The hot loop is two vpminsd with memory operands per iteration. Tests: adds multi-block length, NaN and signed-zero cases for all four searches (the existing lengths stop at 256 elements, one block). The IndexOf* tests pass for all element types on the 512-, 256- and 128-bit and scalar paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
Tagging subscribers to this area: @dotnet/area-system-numerics |
Add debug-only checks to IndexOfMinMaxCore: after each block reduction, no element of the block beats the reduction result under the operator's Compare and no NaN survived the reduction (the properties the final scan and the NaN handling rely on), the element size is one the lane extraction supports, and pass 1 selected a block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IndexOfMinMaxCore(shared byIndexOfMin,IndexOfMax,IndexOfMinMagnitude,IndexOfMaxMagnitude) tracks a running best value plus a vector of indices per lane. That costs a compare and two selects per element, needs theSize4Plus/Size2/Size1variants so indices fit in lanes, and on AVX-512 hardware the blends of the running value go through mask-register conversions.IndexOfMin<int>over 65536 elements runs at about 5 GElem/s on a Zen 4 box, no faster than the .NET 10 package.This PR replaces the loop with a two-pass block reduction, written against the operator interface so all four searches use it:
Reduceper vector, two independent accumulators), then a horizontalAggregate, and keeps the first block whose result beats the running result under the operator's strictCompare.IIndexOfMinMaxOperator<T>gainsReduce(scalar and per vector width). Each operator forwards it to its aggregation operator (Vector.Min/Max/MinMagnitude/MaxMagnitude), which the previous code already used for its final horizontal step and bitwise match, so tie behavior is unchanged. Those reductions implement IEEEminimum/maximumand propagate NaN, so a block containing a NaN reduces to NaN and the first NaN of that block is returned; the per-vector NaN checks go away. Indices never live in vector lanes, so the per-element-size variants andElementWiseSelectare removed (536 lines added, 597 removed).Performance
BenchmarkDotNet 0.15.8, AMD Ryzen 7 7800X3D, .NET 10.0.5,
IndexOfMin<int>, N = 65536 (256 KB), GElem/s:main(this core, unmodified)main,DOTNET_PreferredVectorBitWidth=256The block loop compiles to two
vpminsd zmmwith memory operands per iteration; the result is independent of the input pattern and matches a hand-writtenVector256block-min argmin on the same machine (the L2 fill bandwidth for this working set). Inputs shorter than one vector use the scalar fallback.Tests
Helpers.TensorLengthsstops at 256 elements, which is a single block at 256 bits).System.Numerics.Tensors.Testsfiltered to*IndexOf*passes on the live net11.0 testhost, and the same test sources compiled against the net10.0 build pass on the 512-, 256- and 128-bit paths and with hardware intrinsics disabled (DOTNET_PreferredVectorBitWidth=256,DOTNET_EnableAVX2=0,DOTNET_EnableHWIntrinsic=0), for all 17 element types.