Split arrow dependency from vortex-array#8717
Conversation
Replace the arrow-string backed LIKE executor with a native implementation over canonical VarBinView arrays, dropping the arrow-string dependency from vortex-array. Patterns compile once into a LikePattern: wildcard-free patterns and prefix/suffix/substring shapes use direct byte comparisons (memchr's SIMD substring search for contains, ASCII case-insensitive variants for ILIKE over ASCII data), everything else translates the SQL pattern to an anchored regex over the value bytes, using the same translation as arrow-string so semantics are unchanged. Non-constant pattern children reuse the previous row's compiled pattern while the pattern bytes repeat. Evaluation exploits the view layout: equality is a single masked 16-byte compare for needles that fit inline, the prefix path rejects lanes with a branch-free masked compare of the view's inline 4-byte prefix without touching the data buffers, the suffix path slices exactly the trailing needle bytes out of the view or its buffer, and contains/equality reject on the view length before dereferencing. Benchmarked against the previous arrow implementation (benches/like.rs, 64Ki strings, median): exact 100us -> 18us, prefix 144us -> 48us, suffix 171us -> 105us, contains 876us -> 732us, regex 924us -> 744us, ilike contains 2.52ms -> 2.03ms, per-row patterns 824us -> 565us. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
… flagged words - Reduce the like benchmark input to 2Ki strings so every benchmark stays well under 1ms on CodSpeed's simulated runner (the worst case, ilike, measured 15.8ms at the previous size). - Box the memmem::Finder inside LikePattern::Contains: the searcher is ~300 bytes and dominated the enum size (clippy::large_enum_variant). - Bump the workspace regex pin to 1.12 so direct-minimal-versions agrees with datafusion's floor now that vortex-array depends on regex. - Rename the "hel%" test patterns to "he%" (same matches on the test data) to satisfy the typos spell checker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
|
this pr includes #8709 and should only merge after that one |
Merging this PR will improve performance by 14.5%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | chunked_varbinview_into_canonical[(1000, 10)] |
170 µs | 206 µs | -17.49% |
| ❌ | Simulation | compare_int_constant |
268.4 µs | 298.8 µs | -10.17% |
| ⚡ | Simulation | chunked_bool_canonical_into[(1000, 10)] |
26.3 µs | 16.6 µs | +58.16% |
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[128] |
273.6 ns | 215.3 ns | +27.1% |
| ⚡ | Simulation | chunked_varbinview_opt_canonical_into[(1000, 10)] |
205.3 µs | 169.3 µs | +21.27% |
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[1024] |
333.9 ns | 275.6 ns | +21.17% |
| ⚡ | Simulation | chunked_varbinview_opt_into_canonical[(1000, 10)] |
219.7 µs | 183.5 µs | +19.74% |
| ⚡ | Simulation | chunked_varbinview_canonical_into[(100, 100)] |
258.6 µs | 223.2 µs | +15.9% |
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[2048] |
427.8 ns | 369.4 ns | +15.79% |
| ⚡ | Simulation | chunked_varbinview_into_canonical[(100, 100)] |
308 µs | 271.5 µs | +13.44% |
| ⚡ | Simulation | eq_i64_constant |
298.6 µs | 268.8 µs | +11.11% |
| 🆕 | Simulation | ilike_contains |
N/A | 782.5 µs | N/A |
| 🆕 | Simulation | like_contains |
N/A | 153.2 µs | N/A |
| 🆕 | Simulation | like_exact |
N/A | 43.6 µs | N/A |
| 🆕 | Simulation | like_per_row_patterns |
N/A | 169.7 µs | N/A |
| 🆕 | Simulation | like_prefix |
N/A | 50.4 µs | N/A |
| 🆕 | Simulation | like_regex |
N/A | 622.9 µs | N/A |
| 🆕 | Simulation | like_suffix |
N/A | 64.1 µs | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing rk/vortexarrow (e0383fd) with develop (d96e4ad)
Footnotes
-
8 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Vortex-array doesn't need arrow for any particular functionality. We want to be
arrow compatible and support conversion from one to another but that logic
doesn't need to live with array implementations.
Instead we move all arrow related logic to vortex-arrow crate that users can
depend on which can depend on other crates not just vortex-array
N.B. we still have arrow-buffer dependency in vortex-array since we use their
i256. Having that dependency doesn't seem particularly probelmatic