Skip to content

RISC-V: register-block the ZVL256B GEMV-N kernel - #6057

Open
6eanut wants to merge 1 commit into
OpenMathLib:developfrom
6eanut:riscv64-sgemv-n-block4
Open

6eanut wants to merge 1 commit into
OpenMathLib:developfrom
6eanut:riscv64-sgemv-n-block4

Conversation

@6eanut

@6eanut 6eanut commented Sep 18, 2026

Copy link
Copy Markdown

Summary

Process four consecutive columns per pass over y in the ZVL256B GEMV-N kernel
for unit y stride.

The kernel previously walked one column of A at a time. For each column it
loaded y, loaded the column, issued one vfmacc, stored y back, and then
repeated the whole sequence for the next column. Every column therefore paid a
y load and a y store round-trip, and because VL was recomputed from the
remaining row count on each pass, the vsetvli and the pointer bumps also sat
inside the hot loop.

GEMV_N_BLOCK (4) consecutive columns are now accumulated into a single y
slice that stays resident in vector registers, so the y round-trip is paid
once per block instead of once per column, and the main loop's VL is pinned to
the hardware maximum, which lets the compiler hoist the vsetvli and
strength-reduce the pointer arithmetic out of the loop.

The columns are still consumed in ascending j and the four contributions are
folded into the accumulator in that same order, so each y[i] sees the same
sequence of addends as the original column-at-a-time loop; the intermediate y
vector merely stays in a register instead of being stored and reloaded. The row
remainder of each block keeps a variable VL, and the columns left over after the
last full block are finished by the two-column and single-column loops. The
result is bit-identical.

The non-unit y stride path is untouched.

This backs both SGEMV_N and DGEMV_N.

Performance

benchmark/sgemv.goto and benchmark/dgemv.goto, one X100 core at 2.2 GHz,
single thread, warm-up once and the median of three runs per size:

Benchmark Size Baseline Patched Change
sgemv 256 5885.21 5817.72 -1.15%
sgemv 512 4848.66 6303.53 +30.01%
sgemv 1024 4584.54 5206.73 +13.57%
dgemv 256 2410.66 3158.90 +31.04%
dgemv 512 3670.95 4483.65 +22.14%
dgemv 1024 1899.48 1945.03 +2.40%

Testing

  • make tests returns 0: 125/125 utests, 1473/1473 extension tests, no new
    CBLAS failure.
  • Differential test under QEMU with guard pages around every buffer: 21375
    single-precision, 21375 double-precision and 21375 non-unit-y-stride cases,
    covering m = 1..129, n = 1..33, several leading dimensions and x
    increments, and alpha values including 0, produce output bit-identical to the
    unpatched kernel, with no out-of-bounds access. Both builds were also checked
    against a double-precision reference.
  • The inc_y != 1 branch was diffed against the baseline and is byte-identical.
  • An end-to-end call through the public sgemv_ entry point on both the
    baseline and patched libraries produces identical results.

The ZVL256B GEMV-N kernel for unit y stride walked one column of A at a
time. For each column it loaded y, loaded the column, issued one
vfmacc, stored y back, and then repeated the whole sequence for the next
column. Every column therefore paid a y load and a y store round-trip,
and because VL was recomputed from the remaining row count on each pass,
the vsetvli and the pointer bumps also sat inside the hot loop.

Process GEMV_N_BLOCK (4) consecutive columns per pass over y. A single y
slice stays resident in vector registers while all four columns are
accumulated into it, so the y round-trip is paid once per block instead
of once per column, and the main loop's VL is pinned to the hardware
maximum, which lets the compiler hoist the vsetvli and strength-reduce
the pointer arithmetic out of the loop.

The columns are still consumed in ascending j and the four contributions
are folded into the accumulator in that same order, so each y[i] sees
the same sequence of addends as the original column-at-a-time loop; the
intermediate y vector merely stays in a register instead of being
stored and reloaded. The row remainder of each block keeps a variable
VL, and the columns left over after the last full block are finished by
the original two-column and single-column loops, unchanged.

The result is bit-identical. A differential run of the baseline and
patched kernels against each other, and of both against a
double-precision reference, over 21375 single-precision and 21375
double-precision cases covering m = 1..129, n = 1..33, several leading
dimensions and x increments, and alpha values including 0, with guard
pages around every buffer, shows no mismatch and no out-of-bounds
access. The same sweep for a non-unit y stride, which the patch leaves
untouched, is likewise bit-identical. The library built from the patched
kernel passes 125/125 utests and 1473/1473 extension tests.

Measured on one X100 core at 2.2 GHz, warm-up once and the median of
three runs per size: sgemv 256 -1.15%, 512 +30.01%, 1024 +13.57%;
dgemv 256 +31.04%, 512 +22.14%, 1024 +2.40%.

Co-authored-by: Yuansheng <yuansheng@isrc.iscas.ac.cn>
Co-authored-by: Ning Tian <tianning24@iscas.ac.cn>
Signed-off-by: jiakai xu <xujiakai2025@iscas.ac.cn>
@martin-frbg martin-frbg added this to the 0.3.35 milestone Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants