Skip to content

Split arrow dependency from vortex-array#8717

Open
robert3005 wants to merge 5 commits into
developfrom
rk/vortexarrow
Open

Split arrow dependency from vortex-array#8717
robert3005 wants to merge 5 commits into
developfrom
rk/vortexarrow

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Vortex-array doesn't need arrow for any particular functionality. We want to be
arrow compatible and support conversion from one to another but that logic
doesn't need to live with array implementations.

Instead we move all arrow related logic to vortex-arrow crate that users can
depend on which can depend on other crates not just vortex-array

N.B. we still have arrow-buffer dependency in vortex-array since we use their
i256. Having that dependency doesn't seem particularly probelmatic

robert3005 and others added 5 commits July 10, 2026 09:52
Replace the arrow-string backed LIKE executor with a native implementation
over canonical VarBinView arrays, dropping the arrow-string dependency from
vortex-array.

Patterns compile once into a LikePattern: wildcard-free patterns and
prefix/suffix/substring shapes use direct byte comparisons (memchr's SIMD
substring search for contains, ASCII case-insensitive variants for ILIKE
over ASCII data), everything else translates the SQL pattern to an anchored
regex over the value bytes, using the same translation as arrow-string so
semantics are unchanged. Non-constant pattern children reuse the previous
row's compiled pattern while the pattern bytes repeat.

Evaluation exploits the view layout: equality is a single masked 16-byte
compare for needles that fit inline, the prefix path rejects lanes with a
branch-free masked compare of the view's inline 4-byte prefix without
touching the data buffers, the suffix path slices exactly the trailing
needle bytes out of the view or its buffer, and contains/equality reject
on the view length before dereferencing.

Benchmarked against the previous arrow implementation (benches/like.rs,
64Ki strings, median): exact 100us -> 18us, prefix 144us -> 48us,
suffix 171us -> 105us, contains 876us -> 732us, regex 924us -> 744us,
ilike contains 2.52ms -> 2.03ms, per-row patterns 824us -> 565us.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
… flagged words

- Reduce the like benchmark input to 2Ki strings so every benchmark stays
  well under 1ms on CodSpeed's simulated runner (the worst case, ilike,
  measured 15.8ms at the previous size).
- Box the memmem::Finder inside LikePattern::Contains: the searcher is
  ~300 bytes and dominated the enum size (clippy::large_enum_variant).
- Bump the workspace regex pin to 1.12 so direct-minimal-versions agrees
  with datafusion's floor now that vortex-array depends on regex.
- Rename the "hel%" test patterns to "he%" (same matches on the test data)
  to satisfy the typos spell checker.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005

Copy link
Copy Markdown
Contributor Author

this pr includes #8709 and should only merge after that one

@codspeed-hq

codspeed-hq Bot commented Jul 10, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 14.5%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 9 improved benchmarks
❌ 2 regressed benchmarks
✅ 1653 untouched benchmarks
🆕 7 new benchmarks
⏩ 8 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_varbinview_into_canonical[(1000, 10)] 170 µs 206 µs -17.49%
Simulation compare_int_constant 268.4 µs 298.8 µs -10.17%
Simulation chunked_bool_canonical_into[(1000, 10)] 26.3 µs 16.6 µs +58.16%
Simulation bitwise_not_vortex_buffer_mut[128] 273.6 ns 215.3 ns +27.1%
Simulation chunked_varbinview_opt_canonical_into[(1000, 10)] 205.3 µs 169.3 µs +21.27%
Simulation bitwise_not_vortex_buffer_mut[1024] 333.9 ns 275.6 ns +21.17%
Simulation chunked_varbinview_opt_into_canonical[(1000, 10)] 219.7 µs 183.5 µs +19.74%
Simulation chunked_varbinview_canonical_into[(100, 100)] 258.6 µs 223.2 µs +15.9%
Simulation bitwise_not_vortex_buffer_mut[2048] 427.8 ns 369.4 ns +15.79%
Simulation chunked_varbinview_into_canonical[(100, 100)] 308 µs 271.5 µs +13.44%
Simulation eq_i64_constant 298.6 µs 268.8 µs +11.11%
🆕 Simulation ilike_contains N/A 782.5 µs N/A
🆕 Simulation like_contains N/A 153.2 µs N/A
🆕 Simulation like_exact N/A 43.6 µs N/A
🆕 Simulation like_per_row_patterns N/A 169.7 µs N/A
🆕 Simulation like_prefix N/A 50.4 µs N/A
🆕 Simulation like_regex N/A 622.9 µs N/A
🆕 Simulation like_suffix N/A 64.1 µs N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/vortexarrow (e0383fd) with develop (d96e4ad)

Open in CodSpeed

Footnotes

  1. 8 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant