[quinnj]

Summary

• preserve per-batch dictionary snapshots, accept undeclared stream replacements, and read dictionary deltas in streams, files, and range scans; resume Arrow.append from the final dictionary state even when an update follows the last record batch; fixes Arrow.Table uses last dictionary for all batches in streaming decode, resulting in incorrect reads #610
• replace the Arrow 2.x internals with a validated core, new IPC reader and writer, C Data and C Stream support, and a materialized public table facade
• compile each Tables.Scan once, keep storage and public-domain plans separate, preserve stable empty-result schemas, and support sparse byte-range reads with footer statistics
• deepen column construction around one recursive adapter for native, retained-schema, dictionary, Union, and ArrowTypes-backed columns
• retain automatic ArrowTypes custom serialization, public-domain lifting, extension metadata, and nested custom-type round trips
• use registered DataStrings 1.0 as the StringView representation package and keep Arrow.jl's export surface small
• use DataDecimals 1.0 and Durations 1.4 for supported top-level decimal, interval, and timestamp columns; every zone-naive timestamp reads as Durations.Timestamp{P} at the column unit (a written DateTime column reads back as an instant-equal Timestamp{Millisecond}) and every timezone-declared timestamp as Durations.ZonedTimestamp{P,Z} (no TimeZones.jl requirement), both writing back zero-conversion with every comparison operator pushing down; preserve wire descriptors on rewrite
• make the public value domain uniform at every depth: temporal, decimal, and interval leaves materialize as the same public values inside struct, list, map, and union rows and under run-end-encoded and dictionary wrappers as at the top level; the writer accepts them back at every depth (fresh and retained), scan literal lowering is transparent through run-end encoding, and the reader allocation budget covers the dynamic conversion work at every seam (docs/dev/DESIGN-nested-public-values.md is the design contract)
• discard inherited batch statistics when rewriting edited tables so filters cannot skip changed rows
• reject invalid writer options before opening destination files, release owned files after failures, and preserve the original error
• save only referenced Utf8View/BinaryView bytes, retain shared ranges, and avoid sorting and per-entry copies when referenced bytes already cover each copied buffer prefix
• remove JSON3 and StructTypes integration and support claims
• deepen deterministic fuzzing, Apache Arrow corpus coverage, PyArrow/nanoarrow oracles, C Data checks, allocation limits, trim checks, documentation, and package administration

Current validation

Latest head: 0ffdf1436664228629cbe619249011a415d216f7.

Appending to a valid IPC stream with a trailing dictionary replacement or delta previously either wrote incorrect values or rejected a matching dictionary. The reader now retains the final dictionary map; append resumes that map while earlier record batches keep their original snapshots. Public file and IO regressions cover replacements, deltas, and uncompressed/LZ4/Zstd streams.

• Full package suites pass on Linux Julia 1.10.12 and 1.13.0 with four threads and bounds checks, including CloudStore integration, all four acceptance batteries, and Aqua.
• All 624 focused dictionary and incremental-writer assertions pass on Julia 1.10.12 and 1.13.1. These include 90 assertions covering trailing updates and refusal of a mismatching pool without altering the destination.
• Six independently generated PyArrow 25.0.1 streams reproduce the append defect on the prior head; the candidate passes all 30 public reproduction controls. PyArrow full validation confirms exact values and row counts in all six output streams after two public appends each.
• Pinned Apache Arrow corpus: 388 pass, 0 fail, 63 declared skips. PyArrow 25.0.1/nanoarrow 0.9.0 oracle: 315 pass, 0 fail, 20 declared skips. C Data/C Stream oracle: 147 pass, 0 fail, 8 declared skips. These ran on Linux Julia 1.12.6.
• Strict Documenter 1.19.0 and the exact CI JuliaFormatter 2.12.4 check pass.
• An independent private review covered the complete runtime/test diff and coverage workflow changes.

All 55 hosted checks pass at 0ffdf1436664228629cbe619249011a415d216f7: both CI runs and the conformance run completed successfully. All 34 native coverage uploads were accepted for apache/arrow-julia at that exact commit, and all 34 reports finished processing, with aggregate coverage at 91.93%. Prior green job logs show coverage uploads were rejected with Token required because branch is protected. Uploading jobs now use scoped OIDC for repository-owned runs and fail on upload errors; public forks retain tokenless upload support. Coverage percentage thresholds and processing paths are unchanged. The notification count now matches one 17-report CI run, so main commits and branches without a duplicate PR run can report coverage; all 34 uploads from this PR's push and PR runs were independently verified. The scheduled nightly upload path receives the same authentication repair; scheduled execution is not part of the local validation above.

Dependency

JuliaData/Tables.jl#380 is merged and released as Tables.jl 1.14.0. Commit dd8c272 removes the temporary source and CI pins, raises the Tables.jl compatibility floor to 1.14, and resolves Tables from the General registry everywhere (package, CI workflows, conformance image, docs).

Breaking changes

This is the Arrow.jl 3.0 rewrite. Important changes include:

• Julia 1.10 is now required.
• Arrow.Table and Arrow.Stream return materialized Julia vectors instead of lazy Arrow vector wrappers.
• Arrow.write is eager and whole-buffer. Arrow.Writer and IPC-stream append are reimplemented with fixed-schema semantics; the curried write and tobuffer compatibility forms remain. Obsolete writer tuning keywords are ignored with warnings.
• Arrow.write(io, table) writes file format by default; use file=false for stream format.
• ArrowTypes-based custom serialization remains automatic. ArrowTypes stays exported only as an Arrow 2.x compatibility exception. Packages that define mappings should depend on and import ArrowTypes directly. Arrow.ArrowTypes remains as a qualified compatibility binding.
• most APIs are no longer exported and must be called through Arrow.

Fresh declared heterogeneous Julia Union columns are written as canonical dense Arrow Unions. A retained Union rewrite fails closed when materialization has lost its original route. Nested retained dictionary pools follow the same fail-closed rule.

See CHANGELOG.md and docs/src/migration.md for the complete list and migration guidance.

Release preparation

• DataStrings, DataDecimals, and Durations 1.4 (Timestamp/ZonedTimestamp) resolve from General; the bundled ArrowStrings package and its release steps are removed
• the Apache release-candidate template and CI now enforce a vote window of at least 72 hours
• Arrow.jl remains at 3.0.0-DEV pending the rewrite merge and Apache release process

Co-authored by Codex

AI disclosure: This work was prepared with assistance from OpenAI Codex.

🤖 Generated with Claude Code