Speed up decimal integer parsing with SWAR - #161019
kiana1kaslana wants to merge 8 commits into
Conversation
rustbot
added
S-waiting-on-review
|
Thanks for the pull request, and welcome! The Rust Project is excited to review your changes, and you should hear from @JohnTitor (or someone else) some time within the next two weeks. Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (
Why was this reviewer chosen?The reviewer was selected based on:
|
This comment has been minimized.
This comment has been minimized.
| if radix == 10 { | ||
| while let [a, b, c, d, e, f, g, h, rest @ ..] = digits { | ||
| let chunk = u64::from_le_bytes([*a, *b, *c, *d, *e, *f, *g, *h]); | ||
| if !is_8digits(chunk) { | ||
| return Err(PIE { kind: InvalidDigit }); | ||
| } | ||
| let parsed = parse_8digits(chunk) as $int_ty; | ||
| result = result * (100_000_000u32 as $int_ty); | ||
| if is_positive { | ||
| result = result + parsed; | ||
| } else { | ||
| result = result - parsed; | ||
| } | ||
| digits = rest; | ||
| } | ||
| } |
There was a problem hiding this comment.
There should probably be a 32- and (maybe) 16-bit version so this doesn't wind up slower on those platforms
There was a problem hiding this comment.
Added 32-bit support in 68fd666. Since 8 digits don't fit in a u32 without overflow, I went with a 4-digit SWAR path using the same idea but scaled down — 2 wrapping_mul instead of 3. Anything left after that falls through to the original loop.
|
As per https://forge.rust-lang.org/policies/llm-usage.html, could you let me know did you use/how you used an LLM to create this PR? |
Used an LLM to help me understand the issue and brainstorm the SWAR approach. I wrote and tested the code myself on my machine, including checking the 4-digit variant against a Python reference. Could you help add the llm-assisted label? |
|
Alright, thanks! Anyway I have more PRs than my cap so r? libs |
Can you provide measurements across all of the benchmarks for the integer parsing? Also, can you reformat your PR description to be more readable? Remember that PR descriptions should focus on describing why and potentially a high-level of how, but including details about specific helper functions is probably not useful. |
| #[cfg(not(target_pointer_width = "32"))] | ||
| #[inline] | ||
| const fn is_8digits(v: u64) -> bool { | ||
| let a = v.wrapping_add(0x4646_4646_4646_4646); |
There was a problem hiding this comment.
You can probably make these more readable with the core-internal usize::repeat_u8. If we used that could we unify is_{4,8}digits?
| /// Checks if all 8 bytes in `v` are ASCII decimal digits (`b'0'..=b'9'`). | ||
| /// | ||
| /// Uses a SWAR (SIMD Within A Register) technique to check all 8 bytes | ||
| /// without per-byte branching. |
There was a problem hiding this comment.
I don't think saying that this is "SWAR" is particularly helpful. Maybe we can instead give a brief description of why this works? E.g., similar to what is on the parsing routines? Ideally the constants (0x30/0x46) would also be clearer. It's not obvious to me why we're adding 0x46 (F) rather than 0x39 (9) here, for example.
rustbot
added
S-waiting-on-author
|
Reminder, once the PR becomes ready for a review, use |
Use SIMD-within-a-register to process 8 ASCII digits at once in from_str_radix when radix == 10 and the result is guaranteed not to overflow. Falls back to the existing per-digit loop for the remaining 0-7 digits. The fast path uses two helper functions: - is_8digits: branch-free check that all 8 bytes are b'0'..=b'9' - parse_8digits: 3 multiplications to pack 8 digits into a u64 Benchmark on 16-20 digit decimal strings (5000 iterations, stage 1): bench_u64_from_str_radix_10_long 98818 -> 73194 ns (-25.9%) bench_i64_from_str_radix_10_long 149705 -> 120089 ns (-19.8%) Also add LONG_ASCII_NUMBERS and from_str_radix_long_bench macro to exercise the fast path with strings that trigger 2+ SWAR iterations.
Align trailing `//` comments vertically in the new benchmark data constant. rustfmt in nightly runs during the tidy CI job flags the misaligned comments.
8-digit SWAR uses u64 ops that are emulated on 32-bit and slower than the per-byte loop. Add a 4-digit u32 variant so 32-bit targets get the speedup too. On 64-bit the 4-digit path also picks up the tail after the 8-digit loop. The 8-digit functions are cfg-gated to avoid dead code on 32-bit.
An out-of-line call on the long-input path forced LLVM to set up a stack frame in the hot caller and cost 13% on short u64 inputs. With the helper inlined the short-input regression is down to ~5% and long inputs keep their speedup. Also switch from_ascii_bytes_radix_impl to #[inline(always)] so the larger impl is still inlined into callers.
68fd666 to
a941871
Compare
|
This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed. Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers. |
Cover the 16-digit batch boundary, invalid digits inside the batched window, u64::MAX / i64::MIN, and overflow past them.
|
The short-input regression came from the #[inline(never)] on the SWAR helper. The call is only reachable on the long-input path, but it still forced LLVM to set up a stack frame in the hot caller, which cost about 13% on short u64 inputs. The helper is now #[inline], and from_ascii_bytes_radix_impl is #[inline(always)] so the larger impl still reaches hot callers. New numbers. Alternating A/B runs of two stage-1 builds, bench binaries run back to back within each round, 15 rounds. Ratio = PR / main, geometric mean, lower is better.
Both builds come from the same checkout, one at main and one at this branch, so the only difference is the change under test. The two bench binaries run alternately within each round to cancel machine drift, and the per-round ratio is what gets averaged. A focused 30-round run of bench_u64_from_str_radix_10 has a median ratio of 1.050 (min 0.969, max 1.138). Current shape of the change: radix 10 only, and only for types wider than 4 bytes. For u8..u32 the batch path is dead code (swar_min_len is usize::MAX for them). at most 16 leading digits are batched: 8-digit u64 batches on 64-bit targets, 4-digit u32 batches on 32-bit. The batch arithmetic is unchecked, but batching stops at 16 digits, which can never overflow the type. the tail after the batches goes through the existing checked loop, so overlong or invalid inputs are still rejected there. On the remaining ~5% for short u64 inputs: the short-input loop is now instruction for instruction identical to main. The only differences in the hot closure are one push/pop of %rbx and the larger inlined body (about 480 vs 256 bytes), so I believe this is close to the floor for this approach. @rustbot ready |
rustbot
added
S-waiting-on-review
|
Thanks for providing the more complete summary of how this performs. Can you address the other feedback I provided? (PR description contents and formatting, review comments in the previous review)? Thanks!
I think this implies we're macro-expand a bunch of code that never actually gets run, right? Can we separate out the SWAR paths such that they're not created in the first place, or do we run into problems doing that? It may be that we still want macro generation but could pull the swar code out into a separate module and dispatch to it for |
Mark-Simulacrum
added
S-waiting-on-author
from_str_int_impl! expanded a copy of swar_parse_decimal for every integer type, including u8..u32 where the code is dead. Give the macro a SWAR and a no-SWAR arm and only instantiate the batch parser for u64, i64, u128 and i128. The batch parser now lives in a dedicated decimal_swar module and does its arithmetic in i64, so signed values keep their sign when cast to i128 instead of going through u64 two's complement. Also build the digit-check constant with usize::repeat_u8 and document why it adds 0x46 rather than subtracting 0x30.
This comment has been minimized.
This comment has been minimized.
from_str_int_impl now delegates to from_str_int_impl_inner, so the macro-backtrace note in the expected stderr changed.
The is_digits comment claimed the +0x46 test overflows only past 0xb9,
but wrapping past 0x100 actually leaves the high bit clear again; those
bytes are caught by the -'0' side. Rewrote it to derive why 0x46 is the
right constant ('9' + 0x46 = 0x7f, ':' + 0x46 = 0x80) and which range
each half covers.
parse_8digits/parse_4digits now document what each multiply constant
does instead of leaving 2561/6_553_601/42_949_672_960_001 unexplained.
|
I've rewritten the description to not lean on the name and instead explain the mechanics: The digit check is the same one dec2flt ships as is_8digits: subtracting '0' wraps and sets the high bit for bytes below '0', and for the upper bound 0x46 is the unique constant with '9' + 0x46 = 0x7f and ':' + 0x46 = 0x80, so a single bit separates the last digit from the first non-digit. (Why not 0x39: that would put ':' at 0x73 with the high bit still clear, and the test would never fire.) Bytes that wrap past 0x100 look like digits to the addition but are caught by the subtraction. I've also expanded the code comments in decimal_swar (a82935d): is_digits now derives the constants the same way, and parse_8digits documents what each multiply constant (2561 / 6_553_601 / 42_949_672_960_001) does — each is 10^g * 2^(8g) + 1 for the group size g it merges. |
|
☔ The latest upstream changes (presumably #163178) made this pull request unmergeable. Please resolve the merge conflicts by rebasing. |
This branch has not been deployed
6 participants
Validate and fold 8 digits at a time in decimal integer parsing
from_str_radix— and thereforestr::parse— chews through decimalinput one byte at a time: a branch and a multiply-add per digit, with
the multiply on the critical path. For radix 10 on
u64,i64,u128and
i128this PR adds a fast path that validates and folds 8 digitsper iteration with a handful of whole-word operations.
The interesting part is the validation. A byte is a digit iff it lies in
b'0'..=b'9', and that per-byte range check can be turned into ahigh-bit check that works on all 8 bytes at once. Subtracting
b'0'wraps around and sets the high bit for any byte below
'0'. For theupper bound,
0x46is chosen because it is the unique constant thatputs
'9'at 0x7f and':'at 0x80 — a single bit then separates thelast digit from the first non-digit above it (0x46 = 0x7f - 0x39;
adding 0x39 instead would leave
':'at 0x73 with the high bit clearand the test would never fire). Bytes far above
'9'wrap past 0x100and look like digits again to the addition, but the subtraction catches
those. OR the two results, mask each byte's high bit, one branch for 8
bytes.
dec2fltships this exact check asis_8digits; this PR reusesit for integer parsing.
Once a chunk is known to be all digits, three multiply-shifts fold it
into its numeric value.
& 0x0fstrips the0x3ASCII high nibble,then
* 2561 >> 8merges neighboring bytes into two-digit values(2561 = 10 * 256 + 1),
* 6_553_601 >> 16merges those into four-digitvalues, and
* 42_949_672_960_001 >> 32produces the eight-digitvalue — each multiplier just adds each group to the group above it
times the right power of ten, with masks discarding the overlapping
garbage the multiplies leave in between.
The fast path only kicks in for radix 10 and for types of at least 8
bytes; smaller types keep the old loop, and the per-type macro has a
separate arm for them so no batch code is even instantiated. On
64-bit targets up to 16 leading digits are batched (32-bit targets use
4-digit batches to stay off 64-bit multiplies). 16 decimal digits
always fit in an
i64, so the batch arithmetic is safe to rununchecked even in debug builds. Whatever is left after the batches
goes through the existing checked loop, so overlong and invalid inputs
are rejected exactly as before.
Benchmarks
Alternating A/B runs of two stage-1 builds, bench binaries run back to
back within each round, 15 rounds. Ratio = PR / main, geometric mean,
lower is better.
Long decimal inputs get 1.7x - 3.8x faster. The small signed types
improve because
from_ascii_bytes_radix_implis now#[inline(always)]and reaches their callers.The one regression, 1.07x on short u64 inputs, is dominated by the
first round of the first pairing; the remaining 14 rounds sit at
~1.00x, and the short-input loop is instruction-for-instruction
identical to main. The residual difference is one push/pop of %rbx plus
the larger inlined body (~480 vs 256 bytes) in the hot closure.
Boundary tests for the batch cutoffs (15/16/17 digits, values around
each type's min/max, sign handling, non-digits in every batch position)
are in
library/coretests/tests/num/mod.rs; all existingnum::testspass.