From 4b0f349709a922155b77ded88a38d8133762e1d3 Mon Sep 17 00:00:00 2001 From: Jimmy Angelakos Date: Fri, 25 Sep 2026 18:27:13 +0100 Subject: [PATCH 1/5] build: compute TIMESTAMPTZ partition transforms in UTC --- DUCKDB_1.5_PATCHED.md | 44 ++++++++++++----- DUCKDB_1.5_UNPATCHED.md | 16 ++++++- docker/Dockerfile.duckdb15-base | 11 ++++- ...eberg-timestamptz-utc-transforms-v15.patch | 48 +++++++++++++++++++ .../build-extensions.sh | 5 +- .../coldfront-duckdb-extensions/common.sh | 2 +- 6 files changed, 108 insertions(+), 18 deletions(-) create mode 100644 docker/iceberg-timestamptz-utc-transforms-v15.patch diff --git a/DUCKDB_1.5_PATCHED.md b/DUCKDB_1.5_PATCHED.md index c293c32..73f26d3 100644 --- a/DUCKDB_1.5_PATCHED.md +++ b/DUCKDB_1.5_PATCHED.md @@ -12,14 +12,15 @@ and verified*. The cold-tier compactor's own story lives in > `abfss://` Iceberg **reads** (`read_avro` on `abfss`); the base then layers > ColdFront's patches on top. -## The two patch families the base carries +## The patch families the base carries | Patch family | Files | Purpose | Without it | |---|---|---|---| | **Bakery-aware commit-refresh** | `docker/iceberg-bakery-aware-commit-refresh-v15.patch` | makes the async parquet-upload ordering safe → the **no-409** guarantee for concurrent cold writers, at contended-upload throughput | cold writes still work and still never 409 — they fall back to serialized (claim-first) uploads (see [DUCKDB_1.5_UNPATCHED.md](DUCKDB_1.5_UNPATCHED.md)) | | **Strict-reader interop** (upstreamable) | `docker/iceberg-manifest-list-format-version-v15.patch`, `docker/iceberg-data-file-format-v15.patch` | make the manifests duckdb-iceberg *writes* readable by strict Apache readers (apache/iceberg-go) | the cold-tier **compactor cannot read the table** - see [docs/compaction.md](docs/compaction.md). pg_duckdb's own reads/writes are unaffected. | +| **TIMESTAMPTZ transforms in UTC** (port of upstream d3c3348271) | `docker/iceberg-timestamptz-utc-transforms-v15.patch` | year/month/day/hour of a TIMESTAMPTZ partition column are computed on the UTC instant, as the Iceberg spec, duckdb-iceberg's own pruning and iceberg-go take them | a session outside UTC files rows within the zone offset of a boundary in the neighbouring partition, and a UTC-bounded read on the column **prunes them away** | -All three patches apply cleanly to a **pristine** `duckdb-iceberg` @ `5edc45f0` +All four patches apply cleanly to a **pristine** `duckdb-iceberg` @ `5edc45f0` (branch `v1.5-variegata`); `docker/Dockerfile.duckdb15-base` `git apply --check`s each before applying, failing the build loudly on patch rot. @@ -144,7 +145,26 @@ The compactor itself (usage, backends, maintenance steps) is documented in [docs/compaction.md](docs/compaction.md). The interop patches are independent of the bakery patch. -## 4. NOT shipped — no `Commit(ClientContext&)` rewrite +## 4. TIMESTAMPTZ partition transforms in UTC (one patch, a port) + +`iceberg-timestamptz-utc-transforms-v15.patch` ports upstream duckdb-iceberg +d3c3348271 (PR #1361, on `main` only: neither `v1.5-variegata` up to `890b78a9` +nor the duckdb-iceberg that DuckDB v1.5.5 ships, `45163a28`, carries it). At the +pinned ref a partitioned write computes `year/month/day/hour` of a TIMESTAMPTZ +column as `date_diff` on the TIMESTAMPTZ itself, which ICU evaluates in the +session's time zone, and pg_duckdb sets that zone from PostgreSQL's `TimeZone`. +The Iceberg spec, duckdb-iceberg's own read-side pruning +(`iceberg_transform.hpp`) and iceberg-go all take the UTC instant, so from a +session outside UTC a row within the zone offset of a boundary lands in the +neighbouring partition and a UTC-bounded predicate on the column prunes it +away (reproduced: from `America/New_York`, 2026-04-01 02:00 UTC was filed under +March and `ts >= '2026-04-01 00:00+00'` did not return it). The patch binds a +TIMESTAMPTZ source as TIMESTAMP through DuckDB's default cast, which +reinterprets the stored UTC microseconds without ICU, before `date_diff`. +`ci/journey.sh` TC-186 fails without it. Dropped when `ICEBERG_REF` reaches a +ref that carries the fix. + +## 5. NOT shipped — no `Commit(ClientContext&)` rewrite v1.5's `IcebergTransaction::Commit` already copies the caller's `ClientConfig` into its commit-time connection, so `s3_access_key_id` etc. are available; a @@ -157,19 +177,19 @@ patches only. --- -## 5. Version pins (do not drift) +## 6. Version pins (do not drift) | Component | Pin | Notes | |---|---|---| | pg_duckdb | **merged PR #1025** (`c04e6a2`) | no released tag carries 1.5.x; `git checkout c04e6a2`. Its duckdb submodule is the v1.5.4 tag (`08e34c4`). | | DuckDB | **v1.5.4 tag** (`08e34c4`) | pinned by pg_duckdb @ `c04e6a2`; the iceberg build re-pins ITS duckdb submodule to the same tag so the extension ABI matches the engine. The `duckdb.*` GUCs + PRE_COMMIT iceberg-commit deferral ColdFront relies on are unchanged. | -| duckdb-iceberg | **`v1.5-variegata` @ `5edc45f0`** | extension code the three patches target — kept fixed, so the patches apply unchanged. The build re-pins its duckdb submodule to the v1.5.4 tag (the branch tracks duckdb `main`, which drifts off the release; verified: `5edc45f0` compiles clean against v1.5.4). Transaction code lives in `src/catalog/rest/transaction/`. | +| duckdb-iceberg | **`v1.5-variegata` @ `5edc45f0`** | extension code the four patches target — kept fixed, so the patches apply unchanged. The build re-pins its duckdb submodule to the v1.5.4 tag (the branch tracks duckdb `main`, which drifts off the release; verified: `5edc45f0` compiles clean against v1.5.4). Transaction code lives in `src/catalog/rest/transaction/`. | | avro | **`7f423d69`** | the pin `v1.5-variegata` uses. | | azure | **`v1.5-variegata` @ `563589b2`** | the ABI-matched sibling of iceberg's branch. **NOT `main`** — azure `main` collides at link (`multiple definition of duckdb::FileFlags::FILE_FLAGS_NULL_IF_NOT_EXISTS`). | | postgres_scanner | duckdb-postgres **`6b2b12ca`** | the `postgres` ext; built bundled (ABI-matched, stamped v1.5.4), **shipped** in the image (never downloaded). Its vcpkg `libpq` build needs **flex** + **bison**. | | libcurl | **build 8.12.0** (≥ 7.77) | **REQUIRED** — DuckDB 1.5.4 httpfs uses `CURLSSLOPT_AUTO_CLIENT_CERT` (≥ 7.77); the pgEdge base ships 7.76.1. 8.12.0 fixes CVE-2025-0665 (the 8.11.1 resolver SIGABRT); runtime still pins httplib regardless. | -## 6. Build — `docker/Dockerfile.duckdb15-base` +## 7. Build — `docker/Dockerfile.duckdb15-base` The base build *is* the recipe; read it as the source of truth. Its non-obvious requirements (each a real build failure if missing): @@ -193,7 +213,7 @@ requirements (each a real build failure if missing): Cold base build is ~30–60 min (vcpkg compiles the Azure SDK + libpq from source); incremental rebuilds after a patch change recompile only the iceberg extension. -## 7. Install / GUCs / image wiring (base/app split) +## 8. Install / GUCs / image wiring (base/app split) The expensive, **stable** compiles live in the **base** image, published to `ghcr.io/pgedge/coldfront-duckdb-base:pg{16,17,18}`. The thin **app** image layers @@ -235,7 +255,7 @@ Building the app locally pulls the published base `ghcr.io/pgedge/coldfront-duckdb-base:pg`, or uses a locally-built base tagged the same. -## 8. v1.5 architecture notes (verified against source) +## 9. v1.5 architecture notes (verified against source) - `IcebergTransaction::Commit()` opens a fresh `temp_con` but **copies the caller's config** (settings like `s3_access_key_id`, not the secret catalog). @@ -251,7 +271,7 @@ base tagged the same. works by refreshing metadata at commit time rather than re-stamping fields, and why its no-409 correctness is proven by the 3-node bench, not assumed. -## 9. Azure secret (`TYPE azure`) +## 10. Azure secret (`TYPE azure`) Verified against duckdb-azure `src/azure_secret.cpp` + the built extension. There is **no `ACCOUNT_KEY` parameter** — a shared-key account key is supplied only in @@ -273,7 +293,7 @@ One secret serves both `abfss://` (ADLS Gen2 / dfs) and `az://` (blob). live `CREATE PERSISTENT SECRET` is exercised only on the 1.5.x image, not in pg_regress — a green regress run does **not** prove azure I/O). -## 10. CI coverage — why azure is creds-gated, not hermetic +## 11. CI coverage — why azure is creds-gated, not hermetic `ci/matrix.sh` runs the same storage-agnostic journey under s3 (hermetic, SeaweedFS, always) and under azure (creds-gated) across the full grid: both @@ -286,7 +306,7 @@ PENDING — never silently skipped); the storage-divergent code (secret renderin config selection) is covered with no creds by the unit + pg_regress layer on every PR. -## 11. Cutover vs cold-write serialization +## 12. Cutover vs cold-write serialization `coldfront.cutover_archive` acquires the **same bakery** the cold-write path takes (same `v_armed` gate, same `coldfront_iceberg:` key) on its @@ -306,7 +326,7 @@ inversion forms, the **cutover** yields first (100 ms), frees the bakery, the writer commits, and the harness retries the cutover; the writer is never the victim. -## 12. Reverting to UNPATCHED +## 13. Reverting to UNPATCHED No code change — flip to stock by unsetting `coldfront.iceberg_bakery_patch` (the gate goes false → claim-first even if the async flag stays on). To run a diff --git a/DUCKDB_1.5_UNPATCHED.md b/DUCKDB_1.5_UNPATCHED.md index e063b90..f67b492 100644 --- a/DUCKDB_1.5_UNPATCHED.md +++ b/DUCKDB_1.5_UNPATCHED.md @@ -15,11 +15,12 @@ It is still a locally-built (unsigned) extension; there is no signed upstream ## The build delta (vs the patched base) In `docker/Dockerfile.duckdb15-base`, drop the `COPY` + `git apply --check` + -`git apply` of all three patches: +`git apply` of all four patches: - `iceberg-bakery-aware-commit-refresh-v15.patch` - `iceberg-manifest-list-format-version-v15.patch` - `iceberg-data-file-format-v15.patch` +- `iceberg-timestamptz-utc-transforms-v15.patch` Everything else — libcurl, vcpkg deps, the pins, the extension config, the runtime stage — is identical. In `docker/entrypoint.sh`, leave @@ -67,6 +68,19 @@ version/content/format from table metadata, never from the Avro keys iceberg-go checks. So an unpatched cold tier reads and writes fine through PostgreSQL; it just can't be compacted by the go-native compactor. +## Consequence 3: partitioned cold tables are only correct from UTC sessions + +Every tiered cold table, and a decoupled one created with `p_partition_cols`, +is partitioned by `month(ts)` or `day(ts)`. Stock duckdb-iceberg at the pinned +ref computes that transform with `date_diff` on the TIMESTAMPTZ itself, which +ICU evaluates in the session's time zone (pg_duckdb sets it from PostgreSQL's +`TimeZone`), while its own read-side pruning and iceberg-go take the UTC +instant. From a session outside UTC, a row within the zone offset of a month +boundary is filed in the neighbouring partition, and a UTC month-bounded read +prunes it away. The fourth patch, a port of upstream d3c3348271, binds the +column as the UTC TIMESTAMP it holds before the transform. Unpatched, every +cold writer, the archiver included, has to run with `TimeZone = 'UTC'`. + ## When unpatched is acceptable - You don't run the compactor (low cold-write volume, or you compact externally diff --git a/docker/Dockerfile.duckdb15-base b/docker/Dockerfile.duckdb15-base index b5a0b4c..8be89a7 100644 --- a/docker/Dockerfile.duckdb15-base +++ b/docker/Dockerfile.duckdb15-base @@ -105,7 +105,7 @@ RUN dnf install -y --setopt=install_weak_deps=False \ flex bison \ && dnf clean all WORKDIR /build -# ICEBERG_REF is the extension code the three patches target (kept fixed so they +# ICEBERG_REF is the extension code the four patches target (kept fixed so they # apply unchanged). Its duckdb submodule tracks duckdb main, which drifts off the # release, so after init we pin that submodule to the v1.5.4 TAG — the same engine # pg_duckdb links — and build the extension against it. @@ -140,13 +140,20 @@ COPY docker/iceberg-manifest-list-format-version-v15.patch /tmp/iceberg-mlist-fm # (iceberg-go's compaction scan) match it case-sensitively. Inert to pg_duckdb # reads (reader stores verbatim; ducklake lowercases). See the patch header. COPY docker/iceberg-data-file-format-v15.patch /tmp/iceberg-fileformat-v15.patch +# TIMESTAMPTZ transform patch, a port of upstream d3c3348271 (on main only): +# year/month/day/hour of a TIMESTAMPTZ partition column are computed on the UTC +# instant, as the Iceberg spec, duckdb-iceberg's own pruning and iceberg-go take +# them, not in the session's time zone. See the patch header. +COPY docker/iceberg-timestamptz-utc-transforms-v15.patch /tmp/iceberg-tstz-utc-v15.patch RUN cd /build/duckdb-iceberg \ && git apply --check /tmp/iceberg-bakery-v15.patch \ && git apply /tmp/iceberg-bakery-v15.patch \ && git apply --check /tmp/iceberg-mlist-fmtver-v15.patch \ && git apply /tmp/iceberg-mlist-fmtver-v15.patch \ && git apply --check /tmp/iceberg-fileformat-v15.patch \ - && git apply /tmp/iceberg-fileformat-v15.patch + && git apply /tmp/iceberg-fileformat-v15.patch \ + && git apply --check /tmp/iceberg-tstz-utc-v15.patch \ + && git apply /tmp/iceberg-tstz-utc-v15.patch # vcpkg triplet is derived from the build arch (uname -m): x86_64→x64-linux, # aarch64→arm64-linux. Host and target match here (the stage runs on the target # arch, native or emulated), so HOST and TARGET triplets are the same value. diff --git a/docker/iceberg-timestamptz-utc-transforms-v15.patch b/docker/iceberg-timestamptz-utc-transforms-v15.patch new file mode 100644 index 0000000..f2eb660 --- /dev/null +++ b/docker/iceberg-timestamptz-utc-transforms-v15.patch @@ -0,0 +1,48 @@ +# iceberg-timestamptz-utc-transforms-v15.patch +# +# Applies to: duckdb-iceberg @ 5edc45f0 (branch v1.5-variegata), DuckDB v1.5.x. +# +# A port of upstream duckdb-iceberg d3c3348271 ("fix: use UTC semantics for Iceberg +# TIMESTAMPTZ transforms", PR #1361), which is on main only: v1.5-variegata does not +# carry it (checked up to 890b78a9), nor does the duckdb-iceberg that DuckDB v1.5.5 +# ships (45163a28). +# +# Problem: a partitioned write computes year/month/day/hour of a TIMESTAMPTZ column +# as date_diff on the TIMESTAMPTZ itself, which ICU counts in the session's time +# zone, and pg_duckdb sets that zone from PostgreSQL's TimeZone. The Iceberg spec, +# duckdb-iceberg's own partition pruning (iceberg_transform.hpp) and iceberg-go all +# take the UTC instant. So from a session outside UTC, a row within the zone offset +# of a boundary is filed in the neighbouring partition, and a predicate on the +# column prunes it away: from an America/New_York session, 2026-04-01 02:00 UTC is +# filed under March, and "ts >= '2026-04-01 00:00+00'" does not return it. +# +# Fix: bind a TIMESTAMPTZ source as TIMESTAMP through DuckDB's default cast, which +# reinterprets the stored UTC microseconds and never consults ICU, before date_diff. +# Upstream also covers TIMESTAMP_TZ_NS, a type DuckDB 1.5 does not have. +# +# ci/journey.sh's New York-session write (TC-186) fails without this patch. Drop the +# patch when ICEBERG_REF moves to a ref that carries the fix: git apply --check then +# fails on the changed lines. +# +diff --git a/src/execution/operator/iceberg_insert.cpp b/src/execution/operator/iceberg_insert.cpp +index 1fcba7803..3fb0502f8 100644 +--- a/src/execution/operator/iceberg_insert.cpp ++++ b/src/execution/operator/iceberg_insert.cpp +@@ -574,8 +574,15 @@ static unique_ptr GetDateDiffFunction(ClientContext &context, const + } else { + children.push_back(make_uniq(Value::DATE(Date::FromDate(1970, 1, 1)))); + } +- // Third argument: the source column +- children.push_back(CreateColumnReference(copy_input, col_type, col_idx)); ++ // Third argument: the source column. A TIMESTAMPTZ source is bound as the UTC ++ // TIMESTAMP it holds: the default cast reinterprets it without consulting ICU, ++ // where date_diff on a TIMESTAMPTZ counts in the session's time zone. Iceberg ++ // defines the temporal transforms on the UTC instant. ++ auto source = CreateColumnReference(copy_input, col_type, col_idx); ++ if (col_type.id() == LogicalTypeId::TIMESTAMP_TZ) { ++ source = BoundCastExpression::AddDefaultCastToType(std::move(source), LogicalType::TIMESTAMP); ++ } ++ children.push_back(std::move(source)); + + ErrorData error; + FunctionBinder binder(context); diff --git a/packaging/coldfront-duckdb-extensions/build-extensions.sh b/packaging/coldfront-duckdb-extensions/build-extensions.sh index a05877e..aa8846c 100644 --- a/packaging/coldfront-duckdb-extensions/build-extensions.sh +++ b/packaging/coldfront-duckdb-extensions/build-extensions.sh @@ -45,12 +45,13 @@ git clone --depth 1 --branch "v${DUCKDB_VERSION}" --recurse-submodules "${DUCKDB git clone "${VCPKG_REPO}" "${BUILD_ROOT}/vcpkg" "${BUILD_ROOT}/vcpkg/bootstrap-vcpkg.sh" -disableMetrics -# extension set (avro/azure/postgres_scanner refs) + the 3 ColdFront patches live +# extension set (avro/azure/postgres_scanner refs) + the 4 ColdFront patches live # in the repo's docker/ dir; git apply --check fails loudly on patch rot. cp "${CWD}/docker/iceberg-azure-extension-config-v15.cmake" "$ICE/extension_config.cmake" for p in iceberg-bakery-aware-commit-refresh-v15 \ iceberg-manifest-list-format-version-v15 \ - iceberg-data-file-format-v15; do + iceberg-data-file-format-v15 \ + iceberg-timestamptz-utc-transforms-v15; do git -C "$ICE" apply --check "${CWD}/docker/${p}.patch" git -C "$ICE" apply "${CWD}/docker/${p}.patch" done diff --git a/packaging/coldfront-duckdb-extensions/common.sh b/packaging/coldfront-duckdb-extensions/common.sh index 2045dd9..0f52e82 100644 --- a/packaging/coldfront-duckdb-extensions/common.sh +++ b/packaging/coldfront-duckdb-extensions/common.sh @@ -23,7 +23,7 @@ export DUCKDB_EXT_VERSION="$DUCKDB_VERSION" export DUCKDB_EXT_BUILDNUM="${COMPONENT_BUILDNUM:-1}" # --- Build pins ------------------------------------------------------------------- -# duckdb-iceberg: the 3 patches target ICEBERG_REF (branch fetched first so the +# duckdb-iceberg: the 4 patches target ICEBERG_REF (branch fetched first so the # ref resolves). avro/azure/postgres_scanner refs live in the extension_config # cmake (packaging copies docker/iceberg-azure-extension-config-v15.cmake). # ICEBERG_REF is read from the base-image Dockerfile — the single source of truth, From ffaa589bf0c104e8d7a2a9eca4925042e53c6d95 Mon Sep 17 00:00:00 2001 From: Jimmy Angelakos Date: Fri, 25 Sep 2026 18:27:13 +0100 Subject: [PATCH 2/5] fix: scope a data file's position deletes to its partition in the compactor --- cmd/compactor/compact.go | 31 +++++++++++++++++++++++++++ cmd/compactor/compact_test.go | 40 +++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/cmd/compactor/compact.go b/cmd/compactor/compact.go index dac619d..3db50c8 100644 --- a/cmd/compactor/compact.go +++ b/cmd/compactor/compact.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "iter" + "reflect" "strings" "github.com/apache/arrow-go/v18/arrow" @@ -180,6 +181,7 @@ func planCompaction(ctx context.Context, cat *rest.Catalog, ns, name string, tar if err != nil { return nil, nil, fmt.Errorf("plan files for %s.%s: %w", ns, name, err) } + tasks = scopeDeletes(tasks) cfg := compaction.DefaultConfig() if targetSize > 0 { @@ -206,6 +208,35 @@ func planCompaction(ctx context.Context, cat *rest.Catalog, ns, name string, tar return tbl, &planResult{groups: groups, plan: plan, sorted: sorted, sortKey: sortKey}, nil } +// scopeDeletes keeps, for each data file, only the position-delete files that can +// reference its rows under the Iceberg spec: those of its own partition, and +// those of no partition. iceberg-go v0.6.0 attaches position deletes by sequence +// number and the delete file's file_path bounds alone (table/scanner.go, +// matchDeletesToData), never by partition, and duckdb-iceberg 5edc45f0 writes +// those bounds under DuckDB's FILENAME_FIELD_ID (2147483646) rather than the +// spec's file_path field (2147483546), so every delete file is attached to every +// data file: one delete file then sits in several partition groups, which +// ReplaceFiles refuses, and rewriting one partition would remove the delete +// files of a partition the planner skipped, resurrecting its deleted rows. +// TC-190 in ci/journey.sh asserts the engine's bound key; when it moves to the +// spec's field, or iceberg-go matches by partition, this keeps what iceberg-go +// already kept and can go. +func scopeDeletes(tasks []table.FileScanTask) []table.FileScanTask { + out := make([]table.FileScanTask, 0, len(tasks)) + for _, task := range tasks { + own := task.File.Partition() + kept := make([]iceberg.DataFile, 0, len(task.DeleteFiles)) + for _, df := range task.DeleteFiles { + if len(df.Partition()) == 0 || reflect.DeepEqual(df.Partition(), own) { + kept = append(kept, df) + } + } + task.DeleteFiles = kept + out = append(out, task) + } + return out +} + // rewrite executes the planned compaction as a single atomic rewrite snapshot // and commits it to the catalog. // diff --git a/cmd/compactor/compact_test.go b/cmd/compactor/compact_test.go index 93789f0..5e3befb 100644 --- a/cmd/compactor/compact_test.go +++ b/cmd/compactor/compact_test.go @@ -5,6 +5,7 @@ import ( "testing" "github.com/apache/iceberg-go" + "github.com/apache/iceberg-go/table" ) func testSchema() *iceberg.Schema { @@ -63,3 +64,42 @@ func TestLoadTableErr_OtherErrorsPassThrough(t *testing.T) { t.Error("non-404 error was reported as a missing table") } } + +// deleteFile builds a position-delete manifest entry in the given partition. +func deleteFile(t *testing.T, spec iceberg.PartitionSpec, path string, partition map[int]any) iceberg.DataFile { + t.Helper() + b, err := iceberg.NewDataFileBuilder(spec, iceberg.EntryContentPosDeletes, path, iceberg.ParquetFile, + partition, nil, nil, 1, 100) + if err != nil { + t.Fatal(err) + } + return b.Build() +} + +// A data file keeps only the position-delete files that can reference its rows: +// those of its own partition, and those of no partition at all. Every other +// delete file is one iceberg-go attached by sequence number alone. +func TestScopeDeletes(t *testing.T) { + spec := iceberg.NewPartitionSpec(iceberg.PartitionField{ + SourceIDs: []int{2}, FieldID: 1000, Name: "month_ts_2", Transform: iceberg.MonthTransform{}}) + march, april := map[int]any{1000: int32(674)}, map[int]any{1000: int32(675)} + data, err := iceberg.NewDataFileBuilder(spec, iceberg.EntryContentData, "data/month_ts_2=674/d.parquet", + iceberg.ParquetFile, march, nil, nil, 10, 1000) + if err != nil { + t.Fatal(err) + } + own := deleteFile(t, spec, "data/month_ts_2=674/own-deletes.parquet", march) + other := deleteFile(t, spec, "data/month_ts_2=675/other-deletes.parquet", april) + global := deleteFile(t, spec, "data/global-deletes.parquet", map[int]any{}) + + tasks := scopeDeletes([]table.FileScanTask{{File: data.Build(), DeleteFiles: []iceberg.DataFile{other, own, global}}}) + + got := make([]string, 0, 2) + for _, df := range tasks[0].DeleteFiles { + got = append(got, df.FilePath()) + } + want := []string{"data/month_ts_2=674/own-deletes.parquet", "data/global-deletes.parquet"} + if len(got) != len(want) || got[0] != want[0] || got[1] != want[1] { + t.Fatalf("kept %v, want %v", got, want) + } +} From 6c3bd71001eb00cdd2823bf568e31fca054e583d Mon Sep 17 00:00:00 2001 From: Jimmy Angelakos Date: Fri, 25 Sep 2026 18:27:13 +0100 Subject: [PATCH 3/5] feat: partition cold tables the way the hot tier is --- README.md | 5 +- ci/journey.sh | 154 +++++++++++++++++- cmd/archiver/main.go | 64 +++++--- cmd/archiver/main_test.go | 25 +++ docs/architecture.md | 22 +-- docs/architecture_decoupled.md | 20 ++- docs/architecture_tiered.md | 40 +++-- docs/architecture_vectors.md | 13 +- docs/changelog.md | 11 ++ docs/compaction.md | 23 ++- docs/index.md | 3 +- docs/installation.md | 28 ++-- docs/object_store.md | 3 +- docs/usage.md | 16 +- docs/usage_vectors.md | 7 + extension/coldfront/Makefile | 3 +- extension/coldfront/coldfront--1.0.sql | 82 ++++++++-- .../test/expected/partition_clause.out | 89 ++++++++++ .../coldfront/test/sql/partition_clause.sql | 46 ++++++ 19 files changed, 547 insertions(+), 107 deletions(-) create mode 100644 extension/coldfront/test/expected/partition_clause.out create mode 100644 extension/coldfront/test/sql/partition_clause.sql diff --git a/README.md b/README.md index ce15487..f49c88e 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,8 @@ SELECT coldfront.set_storage_secret('admin', 'adminsecret', 'seaweedfs:8333'); -- Decoupled (iceberg-only) table, stored entirely in Iceberg on S3: SELECT coldfront.create_iceberg_table('public', 'events', - '[{"name":"id","type":"bigint"},{"name":"ts","type":"timestamptz"},{"name":"note","type":"text"}]'::jsonb); + '[{"name":"id","type":"bigint"},{"name":"ts","type":"timestamptz"},{"name":"note","type":"text"}]'::jsonb, + '{month(ts)}'); INSERT INTO events VALUES (1, now(), 'hello'); SELECT count(*) FROM events; ``` @@ -233,7 +234,7 @@ against: |-----------|---------|---------| | PostgreSQL | 16, 17, or 18 | Database with native partitioning (stock upstream; no fork) | | pg_duckdb | 1.5.4 (PR #1025) | Iceberg reads + writes via DuckDB in-process | -| duckdb-iceberg | `v1.5-variegata` @ `5edc45f0`, patched | Iceberg catalog/IO for DuckDB; carries ColdFront's three patches (see [DUCKDB_1.5_PATCHED.md](DUCKDB_1.5_PATCHED.md)) | +| duckdb-iceberg | `v1.5-variegata` @ `5edc45f0`, patched | Iceberg catalog/IO for DuckDB; carries ColdFront's four patches (see [DUCKDB_1.5_PATCHED.md](DUCKDB_1.5_PATCHED.md)) | | Lakekeeper | latest | Iceberg REST catalog (Rust binary) | | S3-compatible store | any | SeaweedFS, MinIO, GCS, Azure Blob, etc. | diff --git a/ci/journey.sh b/ci/journey.sh index 4064d09..5896a35 100755 --- a/ci/journey.sh +++ b/ci/journey.sh @@ -128,6 +128,22 @@ vended_creds() { [ "$BACKEND" = vended ] || [ "$BACKEND" = azure-vended ]; } # rather than inventing a literal. hot_days() { echo $(( ( $(date -u +%s) - $(date -u -d "$(date -u +%Y-%m-01) -1 month" +%s) ) / 86400 )); } +# ice_files : live data files (delete files and entries a later +# snapshot removed excluded) whose path matches the regex, counted through the +# table's own metadata scan. Addressed as the attached catalog table, which +# reads identically on static and vended credentials: a bare object-store path +# cannot authenticate under vending. +ice_files() { + q "$HOST" "SELECT coldfront.ensure_attached(); SELECT r['n'] FROM duckdb.query('SELECT count(*) AS n FROM iceberg_metadata(''$1'') WHERE status <> ''DELETED'' AND content NOT LIKE ''%DELETES'' AND regexp_matches(file_path, ''$2'')') AS t(r);" | tail -1 +} + +# ice_spec : the table's partition spec as transform:column terms in +# field order (e.g. "identity:region,month:ts"), empty for an unpartitioned +# table or one that has no data yet. +ice_spec() { + q "$HOST" "SELECT coldfront.ensure_attached(); SELECT r['s'] FROM duckdb.query('SELECT string_agg(t || '':'' || c, '','' ORDER BY id) AS s FROM (SELECT DISTINCT partition_field_id AS id, partition_field_transform AS t, partition_source_columns[1] AS c FROM iceberg_partition_stats(''$1''))') AS t(r);" | tail -1 +} + # assert_register_rejected