Repository navigation
Calcite adapter: close push-down and type-mapping gaps - #550
Merged
Merged
Conversation
VortexTable reopened and memory-mapped the file for every getRowType, scan, totalRows and aggregate call, so one query opened it several times. The struct dtype and chunk row count are now read on first use and cached (a Vortex file is write-once). Column statistics stay uncached: folding them walks every column's zones, which only the aggregate rule needs. Also drops "demo" from the module description. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
VortexTable did not override getStatistic, so Calcite saw an unknown row count and guessed table sizes when ordering joins. It now reports the row count read from chunk metadata (no decode). Cross-checked against a vortex-jni-written multi-batch file: the statistic and SELECT COUNT(*) both match what Rust wrote. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DType.Extension kept its metadata as a slice of the reader's file mapping, so a dtype held past VortexReader.close() threw "Already closed" on first use, e.g. reading a timestamp's unit or zone from a cached SQL schema. The constructor now copies off-heap metadata to the heap; the existing 64 KiB cap is checked first, so a crafted file cannot force a large copy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
toSqlType threw on any column that was not a primitive, string or bool, so a file with a single timestamp could not be queried at all, not even its other columns. Decimal maps to DECIMAL(p, s), binary to VARBINARY, vortex.date/time/uuid to DATE/TIME/UUID, vortex.timestamp to TIMESTAMP (the instant as a UTC wall clock, zoned or not: Avatica's JDBC getters shift TIMESTAMP WITH LOCAL TIME ZONE by the JVM zone), any other extension to its storage type. Nested types map to ANY so the rest of the table stays queryable; selecting one fails loudly. Cross-checked against a vortex-jni-written file with date, time, timestamp (with and without zone), decimal and binary columns. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The filtered fold required one zone per chunk, which only the legacy vortex.stats zone map has. Rust's vortex.zoned (our default since #447, and every Rust-written file) uses fixed 8192-row zones while a chunk spans many of them, so every filtered COUNT/MIN/MAX on such a file fell back to a full scan. The fold now works per zone: IN zones fold from statistics, OUT zones are skipped, and a boundary zone folds only its own rows through the new zero-copy Chunk.slice(offset, length), even when it straddles a chunk edge. Files without a placeable zone map fall back to per-chunk stats as before. SUM still scans on zoned files: Rust records no zone sum. Also fixes a latent wrong MIN/MAX: an IN zone that has values but no min/max stat (zoned string columns keep only truncated bounds) was silently skipped by the fold; it now makes the extreme unknown, so the rule falls back to the scan. Cross-checked on a vortex-jni-written file: the plan has no scan and COUNT/MIN/MAX match the data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…e scan Three predicate shapes never reached the scan, so they pruned no chunk and Calcite re-checked every row: - an IN list: Calcite expands SEARCH to an OR of equalities, and OR was not translated. A single-column OR now becomes one column predicate (Predicate.Or); an OR over several columns, or with any untranslatable disjunct, stays with Calcite in both lenient and strict walks, since dropping a disjunct would lose rows. - a literal on the left (5 < x): the comparison is reversed. - IS NULL / IS NOT NULL: translated on the lenient pruning path too, so chunks with a zero null count are skipped. The aggregate fold now classifies single-column AND/OR predicates from zone statistics instead of treating them as boundary zones. Cross-checked on a vortex-jni-written file with a nullable column. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Since #447 the fixture was written with the default vortex.zoned zone map: 8192-row zones with no sum, and the writer repartitions the 100 batches into 8 chunks. The fold could not answer SUM on that layout, so boundaryFold fell back to the scan and the A/B timed the baseline twice. With the zoned fold on this branch the fold succeeds but reports a null sum, and the benchmark threw a NullPointerException. - Write the fixture for core2025.10.0, the last edition before vortex.zoned: one vortex.stats zone per batch, with sums. - Fail setup when the fold cannot answer SUM where interior zones exist, so the benchmark cannot silently go stale again. - Run the honesty check only in the boundaryFold trial. Decoding through the fold in the baseline's JVM polluted the reader's type profiles: fullScanBaseline medium ran 2.99-3.04 ms with the check, 1.08-1.28 ms without. Single fork, ms/op: range fold main fold branch scan main scan branch wide 0.354 0.361 4.888 4.945 medium 0.308 0.347 1.127 1.119 narrow 0.743 0.774 0.705 0.738 The fold is about 14x faster than the scan on wide and at parity on narrow, where it has no interior zone and abandons to the scan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the Calcite adapter gaps, one commit each, every one with a vortex-jni cross-check in
CalciteInteropIntegrationTest/DateInteropIntegrationTest:getStatistic()reports the row count to the plannerANYinstead of making the whole table unreadablevortex.zonedzones (new zero-copyChunk.slice); fixes a latent wrong MIN/MAX when an IN zone lacks the statINlists, literal-on-the-left comparisons andIS NULLpushed into the scanCalciteBoundaryAggregateBenchmarkhad been timing the scan against itself since Writer emits legacy vortex.stats layout; Rust's default writer emits vortex.zoned #447 (default zoned fixture has no sums); pinned to legacy zones, setup fails if the fold abandons, and the honesty check no longer pollutes the baseline JVM (scan medium 3.0 → 1.1 ms)./mvnw verify: 8729 tests, 0 failures.🤖 Generated with Claude Code