You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(reader): cache decoded strings per dictionary entry in VarBinDictArray #551
VarBinDictArray.getString(i) copies the entry's bytes into a fresh byte[] and decodes a fresh String on every call. On a low-cardinality column that means one decode per row instead of one per dictionary entry.
Found while profiling SQL over a 4M-row file with YourKit (#550). The sym column has 8 distinct values, and getString was about 11% of wall time. VarBinSlicedArray.getString showed up too, but it only delegates to the dictionary array, so that figure is the same cost seen through the wrapper.
The cache belongs in the array, not in the Calcite enumerator. Every caller that reads strings through getString goes through it: Chunk.as, the CLI, the JDBC/CSV export and Calcite.
Proposal
Hold a lazily allocated String[], one slot per dictionary entry, and decode each entry on its first read.
Unsynchronized writes are safe because String is immutable. A race at worst decodes an entry twice.
getBytes keeps returning a fresh copy, because callers may mutate it.
The dictionary size comes from an untrusted offsets buffer. Cache only up to a fixed number of entries (for example 64K) and decode per call above that, so a malformed file cannot force a huge heap allocation (security contract, ADR 0004).
VarBinDictArray is a record, and a record cannot hold a private cache field. It would become a final class with the same accessors. It has one construction site (VarBinArray.ofDict) and nothing deconstructs it.
limited(rows) could share the parent's cache, since it reuses the same dictionary.
Check
A unit test showing repeated reads of one code return the same instance.
A dictionary over the cap still reads correctly.
Before/after numbers on the full-scan and GROUP BY sym SQL workload.
Activity
perf(performance): benchmark reads of a dictionary-encoded string column
perf(reader): decode each dictionary string once, not once per row
perf(reader): filter dictionary strings once per value, not once per row