Repository navigation
Conversation
Unless you build with some other SHA-1 implementation, every object we
hash goes through sha1collisiondetection. That's what we want for
safety, but it's not free. On the machine I'm testing with (a 4-vCPU
Xeon VM, which has SHA-NI and AVX2), sha1dc hashes at about 400 MiB/s,
while OpenSSL's SHA-1 (which does no detection) does about 1100 MiB/s.
On an Apple M5 Max, sha1dc manages about 700 MiB/s.
That matters for anything that hashes a lot of data we don't trust,
like index-pack.
Sam Reis recently wrote about closing that gap for gitoxide [1], with a
from-scratch rewrite of the detection code as a Rust crate [2]. There
are really three separate tricks in there:
1. Compress with the CPU's SHA-1 instructions, and spill the expanded
message schedule as we go. That's all the detection needs from the
vast majority of blocks.
2. The "unavoidable bitconditions" (UBC) filter, which rules out ~95%
of blocks before we get to the expensive part, is most of what
detection costs. Its conditions can be rewritten into an
equivalent set that lines up in vector lanes, and the crate has a
solver that does that for each instruction set.
3. Recompress the few blocks the filter can't rule out with the SHA-1
instructions, too.
We could in theory use the crate itself through our Rust support. But
that support is optional, the crate needs a much newer Rust than we
allow, and none of the three is all that much code once you know what
it's doing. So let's do it in C, which helps every build.
We can't really do any of that inside sha1dc/ itself. It's third-party
code (optionally a submodule), and its block loop is built around its
scalar compression, which stores intermediate states as it goes. So this
patch adds sha1dc-accel/, which has its own block loop but works on
sha1dc's SHA1_CTX and uses its table of disturbance vectors (DVs). For
each block it asks a "backend" to compress it (spilling the schedule and
the two states that recompression starts from), runs the UBC filter,
and then recompresses the block for each DV the filter didn't rule out.
For now there's only one backend: a portable C compression, plus
sha1dc's own ubc_check(). So this is just a reshuffling of what sha1dc/
already does, and the speedups come in the following patches. It comes
out a little slower. With 256MB of random data on the Xeon (GCC 13,
pinned to one CPU):
Benchmark 1: test-tool.base sha1
Time (mean ± σ): 727.3 ms ± 60.8 ms [User: 677.2 ms, System: 40.7 ms]
Range (min … max): 607.5 ms … 837.0 ms 30 runs
Benchmark 2: test-tool.new sha1
Time (mean ± σ): 797.2 ms ± 115.5 ms [User: 745.4 ms, System: 41.8 ms]
Range (min … max): 630.5 ms … 1078.0 ms 30 runs
Summary
test-tool.base sha1 ran
1.10 ± 0.18 times faster than test-tool.new sha1
That's within the noise of this machine (which is quite noisy, sadly),
but callgrind counts 2.3% more instructions for the new code, and on the
M5 Max (Apple clang), hashing 1GiB goes from 1.49s to 1.55s (median of
9 runs), about 4% slower. That's a price I'm willing to pay for what
comes next.
A few other notes:
- The results must be identical to sha1dc/'s: the same digest, and
the same verdict on collisions. That includes the modes that Git
itself doesn't use (the "safe hash" mitigation, disabling the UBC
filter, and detecting reduced-round collisions). The new unit test
checks exactly that against sha1dc/ for every backend the machine
can run, on random data. There's only one so far, but there will be
more. It also checks that sha1dc/'s table of DVs is laid out the way
we expect, so that a change there fails loudly.
- Likewise, "test-tool sha1dc-backends" lists the backends, and
GIT_TEST_SHA1DC_BACKEND picks one for "test-tool sha1", which lets
t0013 run its collision test against each of them. While I was
there I added a second collision, the SHA-mbles chosen-prefix one
[3]. It trips the same disturbance vector as SHAttered, II(52,0),
but it's a different kind of attack, and gets detected in the
tenth block rather than the fifth.
- "test-tool sha1dc-compare" hashes a file both ways, in every
combination of those modes and with the input split around each
block, and dies if the two ever disagree. t0013 runs it for each
backend on SHAttered, SHA-mbles and a reduced-round collision (from
the crate's tests), which is the only way to exercise what happens
after a collision is found. The last has no NUL byte, so
t/.gitattributes marks t0013's .bin files as binary.
- The backend is picked once, on first use. Threads can race to pick
it, but they'd all store the same value; with GCC and clang we use
relaxed atomics for that. Other compilers only get the portable
backend, so they have nothing to pick, and no shared state.
- DC_SHA1_EXTERNAL builds are left alone, since we don't know what
their context looks like. For everybody else, DC_SHA1_NO_ACCEL
goes back to using sha1dc/ on its own.
[1] https://sam.dev/blog/faster-sha1-collision-detection
[2] https://github.com/srijs/sha1dc
[3] https://sha-mbles.github.io/
Signed-off-by: Scott Chacon <scott@gitbutler.net>
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
After compressing a block, sha1dc checks its expanded message schedule
against the "unavoidable bitconditions" (UBCs) of each of its 32
disturbance vectors (DVs). Each condition says that one bit of W[i]
XORed with one bit of W[j] must have a particular value if an attack
along that DV is in progress. A failed condition rules out its DVs, and
if all 32 are ruled out, which happens for about 95% of blocks, we can
skip the expensive recompression.
That makes the check a cheap filter, but it's cheap only compared to
recompressing. It's 111 scalar statements, of which 47 run for every
block, and the rest sit behind tests of whether their DVs are still
alive. On this machine it takes about half as long as the compression
itself.
The conditions for any one DV aren't a fixed list, though. They're
equations over the bits of the schedule, and equations chain: if bit A
must equal bit B, and B must equal C, then A must equal C. So each DV
really has a whole space of equivalent condition sets. The generator
that wrote sha1dc's ubc_check.c picked conditions that are shared by
many DVs, which is the right thing for scalar code. But a vector unit
can check 4 (or 8) conditions in a single pair of loads if they have the
same shape: the same distance between the two words, the same bit
positions, at consecutive values of i. That wants a different choice.
The sha1dc Rust crate has a solver that makes that choice for each
instruction set. It picks a "prefix" of vector groups that runs on every
block, sized so that it rules out as many blocks as possible for what
it costs, and leaves the remaining conditions to a scalar "tail" that
only runs for DVs the prefix left alive. Its plans work out roughly like
this (the last column is the solver's estimate for random blocks):
form prefix tail blocks reaching tail
scalar 70 statements 86 21%
sse2 26 groups of 4 58 9%
avx2 16 groups of 8 44 7%
neon 20 groups of 4 77 17%
The crate only emits Rust, so ubc_check.c carries its plans over as
tables: for each form, the conditions of its prefix (as one entry per
vector group, giving the two words, their shifts, and the bit and DVs
of each lane), and the remaining conditions of each DV for its tail.
The code that runs them is a short loop per form. We ask the compiler to
unroll the prefix loops fully, so that each table entry turns back into
immediates; without that, the forms are 2.2x to 2.4x slower with GCC
and 1.3x to 3.6x slower with clang. With it, GCC's build of the tables
runs the same number of instructions as the crate's plans written out as
straight-line code, at the same speed. The exception so far is clang 18
on x86-64, where the scalar form runs at half that speed (68ns a block
instead of 34ns), since it spills partial masks to the stack. That form
is only the fallback for CPUs without SSE2 or NEON, and Apple clang 21
does not do this.
The tables come from the crate as of commit 426b4afd. The crate is MIT
or Apache-2.0 at your option (sha1collisiondetection itself is MIT); we
use the tables under the MIT license, and the file carries its notice.
Note that the crate stores its schedule backwards on x86 (to save a
shuffle in its SHA-NI code), where we keep it in step order everywhere,
so our SSE2 and AVX2 tables list their lanes in the opposite order from
the crate's.
We now have these backends, in order of preference:
- portable+avx2: if the CPU has AVX2 (and the OS saves the YMM
registers, which we check with xgetbv)
- portable+sse2: always on x86-64, where SSE2 is part of the ABI
- portable+neon: always on arm64
- portable: everywhere else, which still gets the new scalar form
The vector forms need target attributes and intrinsics, so for now
they're only built with GCC 5 or newer and clang. Other compilers (and
32-bit x86, where SSE2 isn't a given) get only the portable one.
With 256MB of random data again, the Xeon picks portable+avx2:
Benchmark 1: test-tool.old sha1
Time (mean ± σ): 801.1 ms ± 82.9 ms [User: 750.1 ms, System: 40.3 ms]
Range (min … max): 673.1 ms … 1006.1 ms 30 runs
Benchmark 2: test-tool.new sha1
Time (mean ± σ): 620.4 ms ± 111.2 ms [User: 571.0 ms, System: 41.9 ms]
Range (min … max): 486.4 ms … 857.2 ms 30 runs
Summary
test-tool.new sha1 ran
1.29 ± 0.27 times faster than test-tool.old sha1
and callgrind counts 16% fewer instructions than before this patch. The
M5 Max picks portable+neon, and hashes 1GiB in 1.22s instead of 1.55s
(median of 9 runs), 1.27x faster.
For testing, the unit test compares every form against sha1dc's own
ubc_check(). Random schedules aren't enough, though: the prefix rules
out most of them, so the tail checks for any one DV would hardly ever
run. So for each DV, the test also generates random schedules until one
keeps that DV alive, and then compares all forms on it with each of its
2560 bits flipped in turn. That's the same approach the crate takes (and
sha1collisiondetection's own tools before it).
Outside of the test suite, I also checked every form against ubc_check()
on a million random schedules and 256 of those witnesses: built with
GCC 13 and clang 18 on x86-64, with Apple clang on arm64 (and with it
targeting x86-64, running the SSE2 form under Rosetta and the AVX2 one
translated to NEON by SIMDe), and with GCC and clang for aarch64 under
qemu.
Signed-off-by: Scott Chacon <scott@gitbutler.net>
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
With the UBC check out of the way, most of what's left is the SHA-1
compression itself, which we still do in portable C. Most x86-64 CPUs
from the last several years (Intel since Ice Lake and some Atoms, AMD
since Zen) have instructions for that: sha1rnds4 does four rounds at a
time, and sha1msg1/sha1msg2 do the message expansion.
There are two catches. The first is that the detection needs the
expanded schedule, which the instructions keep to themselves. That's
easy enough: as each group of four words is used, we store it (with a
shuffle, since SHA-NI holds them in reverse order). Words 16 to 31 come
from sha1msg1/sha1msg2, and the rest from the recurrence
W[t] = (W[t-6] ^ W[t-16] ^ W[t-28] ^ W[t-32]) <<< 2
in plain SSE, in which no word of a group depends on another. That's
what the crate does, too; it notes that sha1msg2 is microcoded on
Sapphire Rapids, where this runs about 10% faster.
The second catch is that the recompression starts from the working
state before step 58 or step 65, and the instructions only let us see
the state between groups of four steps. But the state at step 60 is
just two steps away from 58, and 64 is one step from 65. So we save
those two for every block, and only for a block that the filter flags
(about one in twenty) do we walk them the rest of the way.
While we're here, we can do the recompression of those flagged blocks
with the SHA-1 instructions, too. The portable code runs the partner
block backwards from its state at 58 or 65 to find the chaining value
it started from, then forwards to find the one it ends on, and compares
that with ours. But the instructions only go forwards. So instead we run
the partner forwards from step 60 or 64 to step 80, which tells us the
only input chaining value that could produce our output (it's the output
minus the state at 80, since the feed-forward is just addition). Then we
run forwards from that input to step 60 or 64 and see if we arrive back
where we started. Since each step is a bijection, that's the same test.
The exception is reduced-round detection, which also wants to know the
partner's input chaining value for its own sake. Git never turns that
on, so it just keeps using the portable recompression.
We check for SHA-NI (plus SSSE3 and SSE4.1, which the code also needs)
with cpuid when we pick a backend, and compile with target attributes,
so there's nothing to configure. That gives two new backends in front of
the others, shani+avx2 and shani+sse2, which differ only in the UBC
check.
The unit test gains a check of the compression against SHA-1 steps
written out in the test itself, and of the recompression, which must
accept the partner's real output and reject any other.
With the usual 256MB of random data (the Xeon picks shani+avx2):
Benchmark 1: test-tool.old sha1
Time (mean ± σ): 590.3 ms ± 83.8 ms [User: 541.0 ms, System: 42.6 ms]
Range (min … max): 440.2 ms … 750.7 ms 30 runs
Benchmark 2: test-tool.new sha1
Time (mean ± σ): 285.2 ms ± 26.0 ms [User: 238.9 ms, System: 42.1 ms]
Range (min … max): 223.9 ms … 341.5 ms 30 runs
Summary
test-tool.new sha1 ran
2.07 ± 0.35 times faster than test-tool.old sha1
That makes the whole series so far 2.70 ± 0.35 times faster than plain
sha1dc/ on this machine. Or, for index-pack on the 208MB pack of a clone
of git.git, single-threaded:
Benchmark 1: git.old index-pack --threads=1
Time (mean ± σ): 24.301 s ± 1.102 s [User: 23.739 s, System: 0.219 s]
Range (min … max): 23.557 s … 26.215 s 5 runs
Benchmark 2: git.new index-pack --threads=1
Time (mean ± σ): 12.689 s ± 0.385 s [User: 12.293 s, System: 0.197 s]
Range (min … max): 12.274 s … 13.226 s 5 runs
Summary
git.new index-pack --threads=1 ran
1.92 ± 0.10 times faster than git.old index-pack --threads=1
and with 4 threads:
Benchmark 1: git.old index-pack --threads=4
Time (mean ± σ): 10.450 s ± 0.445 s [User: 27.147 s, System: 0.497 s]
Range (min … max): 9.853 s … 10.782 s 5 runs
Benchmark 2: git.new index-pack --threads=4
Time (mean ± σ): 5.924 s ± 0.214 s [User: 12.693 s, System: 0.527 s]
Range (min … max): 5.591 s … 6.185 s 5 runs
Summary
git.new index-pack --threads=4 ran
1.76 ± 0.10 times faster than git.old index-pack --threads=4
Hashing in-process, shani+avx2 runs at 900 to 1050 MiB/s for messages
of 1KiB and up, against 1130 to 1230 MiB/s for OpenSSL's SHA-1 (which
doesn't detect collisions at all) and 420 to 450 MiB/s for sha1dc/. The
full test suite passes on this machine, as do t0013 and the unit tests
for each of its five backends.
Signed-off-by: Scott Chacon <scott@gitbutler.net>
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Most arm64 CPUs, including Apple's and the Graviton line, have SHA-1
instructions, too, and we can use them the same way as SHA-NI in the
previous patch. It's a little simpler here: the instructions keep the
working state in the natural order, and vsha1h gives us the fifth word
directly, so there's no shuffling to be done.
Knowing whether we can use them is a little trickier:
- If the compiler targets them already (__ARM_FEATURE_SHA2 or
__ARM_FEATURE_CRYPTO, which is the case with Apple's clang on
macOS), we just use them.
- Otherwise we build the two functions that need them with a target
attribute, and check at run time. On Linux that's getauxval(); on
other platforms we don't know how to ask yet, and so don't use them.
- Older compilers only declare the intrinsics when the whole file
targets the instructions, so a target attribute doesn't help there.
We only try the attribute with GCC 9+ and clang 17+.
The one new backend, armv8+neon, goes to the front of the list.
On an Apple M5 Max (macOS, Apple clang), which picks armv8+neon, hashing
1GiB of random data with "test-tool sha1" (median of 9 runs) goes:
sha1dc, before this series 1.46s (~700 MiB/s)
block loop, after 1/4 1.55s
portable+neon, after 2/4 1.22s
armv8+neon, this patch 0.51s (~2 GiB/s)
which is 2.85x overall. For index-pack of the 319MB pack of a clone of
git.git, it goes from 16.1s to 8.7s single-threaded (1.86x), and from
6.7s to 5.9s with all 18 threads. The full test suite passes there, too.
For comparison, the crate reports 81% of plain SHA-1's throughput on an
Apple M4 and 80% on a Graviton4, up from 29% [1].
The run-time check on Linux is only tested under qemu-user so far,
built with GCC 13 and clang 18, in a harness that compares every backend
and UBC form against sha1dc/ (the same comparisons as the unit tests).
I'd be happy to see numbers from anybody with a Graviton or similar
machine.
[1] https://sam.dev/blog/faster-sha1-collision-detection
Signed-off-by: Scott Chacon <scott@gitbutler.net>
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
The C SHA-NI path spends instructions reordering the message schedule, while Rust uses a mirrored schedule. Avoid that overhead without changing the collision checks. Across 30 randomized, AC-powered triplets on the same pack, C's mean fell from 8.201s to 8.053s. The paired mean improvement was 0.147s (95% CI: 0.083-0.211s), leaving C about 1% behind Rust. Assisted-by: GPT-6 Sol Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The generic Windows ARM64 build otherwise keeps portable compression even on processors with SHA-1 instructions. Check the OS-reported capability so acceleration does not require globally targeting crypto instructions. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Sparse 0.6.4 reports "mixing declarations and code" in GCC's intrinsic headers. It ignores diagnostic pragmas, so that warning can only be suppressed for the including translation units. The checker also lacks SIMD builtins such as __builtin_ia32_psrld128. A substitute intrinsic interface would add substantial maintenance for limited analysis value. Confine the portable-only exception to the intrinsic-dependent implementation files. Keep backend selection, scalar code and hardware tests visible to sparse. Normal compilation and the collision tests continue to cover the real implementations. The intrinsic errors are recorded at: https://github.com/dscho/git/actions/runs/39148439733/job/117737029532 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The new Rust crate `sha1dc` (https://crates.io/crates/sha1dc) promises not only type safety but also much better performance compared to the collision-detecting SHA-1 library Git uses at the moment. This performance comes mostly from a SIMD-centric design that relies on features provided by many x86_64 and aarch64 CPUs. Let's optionally use this crate, toggled by the build option `DC_SHA1_RS`. To avoid requiring a shim around the `sha1dc_rs_final()` function just to call `die()` upon a detected collision, pass a pointer to that function to Rust and let it call it directly. This is safe: `die()` is a variadic function, but the Rust code calls it with a simple string without any interpolation required. In a pretty unscientific test on a moderately busy Windows Ryzen 7 machine (UCRT64 GCC 16.2), Rust-backed `git-index-pack.exe` using `sha1dc` 0.1.3 was over three times faster than the C backend by median wall time. Here are the results running five iterations of the `sha1dc` C, Rust v0.1.2, and Rust v0.1.3 backends in randomized, balanced order with `--verify --no-rev-index --threads=1 --object-format=sha1`: SHA1DC backend Median Best of five -------------- ------ ------------ C (default) 32.9s 32.1s Rust 0.1.2 10.9s 10.4s Rust 0.1.3 10.7s 10.5s Version 0.1.3 finished first in four of five triplets, but its median advantage over 0.1.2 was only about 2.4%, and the best 0.1.2 run was faster. In other words, the difference is mostly in the noise. On the same machine, using WSL ("Windows Subsystem for Linux") with the same packfile copied to Linux' ext4 filesystem: SHA1DC backend Median Best of five -------------- ------ ------------ C (default) 23.971s 23.435s Rust 0.1.2 9.123s 8.492s Rust 0.1.3 9.020s 8.491s This is overall faster because of the ext4 vs NTFS performance characteristics, but the same finding holds true: the Rust version of `sha1dc` is dramatically faster. Studying the runs with the Linux perf tools reveals that with the C backend, over 70% of the total time is spent in `git_hash_update()`, with either version of the Rust backend it is around 30%. A comparable test on an M4 Mac yields these results: SHA1DC backend Median Best of five -------------- ------ ------------ C (default) 8.035s 8.009s Rust 0.1.2 4.359s 4.338s Rust 0.1.3 4.361s 4.268s Note that the `sha1dc` crate still requires a significantly newer Rust version than Git's existing Rust support requires: 1.87 instead of 1.63 (https://crates.io/api/v1/crates/sha1dc/0.1.3). Assisted-by: GPT-6 Sol Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The Rust `sha1dc` create is really new. While it produced only correct hashes in my hands, before unleashing this to the masses, we need to provide an "escape hatch" in case it doesn't do the right thing. Therefore, when building with `DC_SHA1_RS`, use the Rust `sha1dc` by default, yet also offer to use the C version of `sha1dc` via `core.sha1dcBackend=c` (and `core.sha1dcBackend=rust` to select Rust explicitly). This is made possible by a set of function pointers that are initialized upon the first call to the `git_hash_init()` function. Note that the order in which `hex.h` and `sha1dc_git.h` are included in `sha1dc_git.c` now have to be turned the other way round, to avoid redefining the `platform_SHA*` constants. Assisted-by: GPT-6 Sol Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
I am about to introduce logic that needs to perform some initialization once, and once only, even if called concurrently. This is a perfect job for `pthread_once()`, but Git's source code currently lacks a Win32 shim. So let's add one! Also provide a trivial shim for `NO_PTHREAD` builds. Assisted-by: GPT-6 Sol Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `sha1dc_init` function pointer initially points to a function that determines which sha1dc backend to use. Naturally, this initialization should only run once. To allow for that function to be called concurrently in multiple threads, we need to use a pthread primitive to ensure that the `sha1dc_*()` function pointers are initialized exactly once. Unfortunately, this requires quite a bit of non-DRY code to prevent data races when different threads run `initial_init()` concurrently (see https://en.cppreference.com/c/language/memory_model#Threads_and_data_races). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Wrapping the initial calls in pthread_once does not protect the callers' loads of the function pointers while the initializer replaces them. Follow Scott's initialization pattern, using one atomic pointer to an immutable backend, instead of rewriting the callable pointers. Unlike CPU detection, selecting C or Rust reads the repository's lazily initialized configuration. Keep that work serialized with pthread_once. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git already reserves a SHA1_CTX for each SHA-1 computation, but the Rust backend allocates another context and frees it on finalization. That additional lifetime management is costly for small hashes and repeatedly incurs heap first-use costs in short-lived Git processes. In uninstrumented Windows ARM64 comparisons with sha1dc 0.1.5, using the existing storage increased the 64-byte complete-hash rate from about 5.98 million to 7.47 million hashes per second (24.9%). In 12 balanced trials on a 146 MB pack, single-threaded index-pack --verify fell from 10.446 s to 10.219 s (2.2%). The C control also improved between binaries; the Rust gain relative to that control was 0.130 s. Avoid the redundant allocation while keeping Rust's private layout out of the C interface. Reject incompatible storage before selecting the Rust backend, rather than relying on a fixed representation across crate versions and target ABIs. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Assisted-by: GPT-6.1 Sol
Git for Windows should retain its existing collision detector while making accelerated C and Rust selectable at runtime. Keep original C as the default, including in Rust-enabled builds, and retain accelerated C when Rust is disabled. Rust SHA1DC needs Rust 1.87, while Git's other Rust components support Rust 1.49. Preserve the separate DC_SHA1_RS opt-in rather than raising the compiler requirement for all Rust-enabled builds. Backend selection must be thread-safe and must not cache incomplete repository configuration. Trace2 hashes the hostname before command-line and repository setup, so that startup hash must not freeze the choice. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SHA-1 instructions and SIMD features provided by many x86_64 and
aarch64 CPUs can make collision-detecting SHA-1 hashing much faster.
In #6438, I proposed optionally using the Rust
sha1dccrate to take advantage ofthis. Git for Windows v2.56 is still built without Rust, though, so that
change alone cannot currently improve the binaries we ship.
Scott Chacon's accelerated C series offers that acceleration without
requiring Rust.
Let's integrate the accelerated C implementation alongside the optional
Rust support, superseding #6438. Keep the original C implementation as the
default for now, with
core.sha1dcBackendselectingc,accel, orrust. The Rust backend still requires the explicitDC_SHA1_RS=1buildoption and Rust 1.87 or newer.
The C implementation includes the mirrored SHA-NI schedule improvement.
The Rust hasher uses the existing context storage instead of a boxed
allocation, with a runtime size/alignment check.
The branch is based on
v2.56.0.windows.2for reuse in Microsoft Git.CI passed for
d619ab7b805: 45 jobs passed and two Visual Studio jobswere skipped. See https://github.com/dscho/git/actions/runs/39205160592.
The default CI builds do not opt into the Rust SHA1DC backend. The
Rust-enabled runtime code was also built and tested natively on Windows
ARM64: collision tests passed, as did all 12 hash unit tests with each
of the C and Rust backends.
Sparse 0.6.4 requires portable-only checks for
ubc_check.spandx86.spbecause it cannot evaluate GCC's SIMD builtins. That exception is confined
to those checker targets; normal builds retain the SIMD implementations.