From 2c6629ab7bdb94b1c544ca88781af1d3488ad7e6 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Wed, 11 Nov 2026 07:08:36 -0500 Subject: [PATCH] Restructure the installation guide and move testing and CI to its own page --- README.md | 15 +- docs/configure_lakekeeper.md | 2 +- docs/installation.md | 1014 ++++++++++++++++++++-------------- docs/testing_ci.md | 184 ++++++ mkdocs.yml | 1 + 5 files changed, 799 insertions(+), 417 deletions(-) create mode 100644 docs/testing_ci.md diff --git a/README.md b/README.md index 2c61a94c..59577de8 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,9 @@ The ColdFront documentation consists of the following guides: PostgreSQL column types, including arrays, that a cold tier supports. - [Compacting the Cold Tier](docs/compaction.md) - Documents cold-tier table maintenance: compaction, snapshot expiry, and orphan-file removal. + - [Testing and Continuous Integration](docs/testing_ci.md) - Documents the + pre-commit gate, the full test matrix, storage-backend gating, and + publishing the base image. - Developer Resources - [Formal Model - ColdFront Decoupled-Mode Bakery (TLA+/PlusCal)](docs/formal/README.md) - [Release Notes - Changelog](docs/changelog.md) @@ -373,7 +376,7 @@ creates. The deployed image is for development and testing. It trusts every connection without a password. For information about the image's environment variables and a smoke test, see -[Building the Image on Docker](docs/installation.md#building-the-image-on-docker). +[Building ColdFront from Source on Docker](docs/installation.md#building-coldfront-from-source-on-docker). When the extensions are created, continue with [Connecting a Docker Installation to Lakekeeper and an Object Store](#connecting-a-docker-installation-to-lakekeeper-and-an-object-store). @@ -382,9 +385,11 @@ When the extensions are created, continue with Before building ColdFront on bare metal, install the pg_duckdb PostgreSQL extension (built on DuckDB 1.5.4, with ColdFront's pg_duckdb and DuckDB patches -applied) and a patched duckdb-iceberg DuckDB extension. For the prerequisites -and the patches, see -[Building ColdFront from Source](docs/installation.md#building-coldfront-from-source). +applied) and a patched duckdb-iceberg DuckDB extension. For the prerequisites, +see +[Building ColdFront from Source on Bare Metal](docs/installation.md#building-coldfront-from-source-on-bare-metal); +for the patches, see +[Understanding ColdFront's DuckDB Stack](docs/installation.md#understanding-coldfronts-duckdb-stack). You also need a running PostgreSQL 16, 17, or 18 server, `pg_config`, the PostgreSQL server development headers, the libpq headers and library, `make`, `gcc`, Go 1.26.10 or later, and golangci-lint. @@ -456,7 +461,7 @@ PostgreSQL server development headers, the libpq headers and library, `make`, `CREATE EXTENSION` commands in each database that uses ColdFront. For more information, see -[Building ColdFront on Bare Metal](docs/installation.md#building-coldfront-on-bare-metal). +[Building ColdFront from Source on Bare Metal](docs/installation.md#building-coldfront-from-source-on-bare-metal). When the extensions are created, continue with [Connecting a Package or Bare-Metal Installation to Lakekeeper and an Object Store](#connecting-a-package-or-bare-metal-installation-to-lakekeeper-and-an-object-store). diff --git a/docs/configure_lakekeeper.md b/docs/configure_lakekeeper.md index 51947291..c6ce3851 100644 --- a/docs/configure_lakekeeper.md +++ b/docs/configure_lakekeeper.md @@ -237,7 +237,7 @@ The image writes the server settings ColdFront needs into `postgresql.conf` when it initializes a new data directory. A server installed from packages or built another way sets them itself, as the [package](configuration.md#configuring-postgresql) and -[bare-metal](installation.md#building-coldfront-on-bare-metal) sections of +[bare-metal](installation.md#building-coldfront-from-source-on-bare-metal) sections of installation.md show: - `shared_preload_libraries = 'pg_duckdb,coldfront'` loads both extensions at diff --git a/docs/installation.md b/docs/installation.md index 6af830c9..e9358d3d 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -7,58 +7,67 @@ setting up PostgreSQL, Lakekeeper, and ColdFront itself. ## Using a Package to Install ColdFront -ColdFront's packages come from the pgEdge package repository. Access to the -repository requires a certificate from pgEdge; request one and install it as -described in the -[pgEdge Enterprise Postgres documentation](https://docs.pgedge.com/enterprise/). -Then configure the pgEdge repository by following the steps in the Enterprise -Postgres documentation: - -- For RHEL, Oracle Linux, AlmaLinux, and Rocky Linux, see - [Configuring the Repository on Enterprise Linux](https://docs.pgedge.com/enterprise/el/configure-repo/). -- For Debian and Ubuntu, see - [Configuring the Repository on Debian and Ubuntu](https://docs.pgedge.com/enterprise/debian/configure-repo/). - -The documentation details the platform requirements, disabling the PGDG -repository, and creating the pgEdge repository. - -If you already have a pgEdge Enterprise Postgres server installed that you -would like to use, configure the pgEdge repository and install only ColdFront, -continuing with +ColdFront's packages come from the pgEdge package repository. Before you +install the packages, set up access to the repository: + +1. Request and install a certificate from pgEdge, as described in the + [pgEdge Enterprise Postgres documentation](https://docs.pgedge.com/enterprise/). +2. Configure the pgEdge repository by following the steps in the Enterprise + Postgres documentation: + + - For RHEL, Oracle Linux, AlmaLinux, and Rocky Linux, see + [Configuring the Repository on Enterprise Linux](https://docs.pgedge.com/enterprise/el/configure-repo/). + - For Debian and Ubuntu, see + [Configuring the Repository on Debian and Ubuntu](https://docs.pgedge.com/enterprise/debian/configure-repo/). + +If you already have a PostgreSQL server installed that you would like to use, +configure the pgEdge repository and install only ColdFront, continuing with [Installing the ColdFront Packages](#installing-the-coldfront-packages). ColdFront packages from pgEdge require pgEdge's PostgreSQL build. ### Installing PostgreSQL with Packages -Before installing ColdFront, you will need to install pgEdge Enterprise Postgres -and start the server. On RHEL, Rocky Linux, or AlmaLinux, the package does not -create a database cluster, so create and start one as root: +After you configure the pgEdge repository, install pgEdge Enterprise Postgres +and start the server. The following steps walk you through the installation: + +1. Install pgEdge Enterprise Postgres. On RHEL, Rocky Linux, or AlmaLinux, the + package does not create a database cluster, so the installation is followed + by commands that create and start the cluster: + + ```bash + sudo dnf install -y pgedge-enterprise-postgres_18 + sudo /usr/pgsql-18/bin/postgresql-18-setup initdb + sudo systemctl enable --now postgresql-18 + ``` + + On Ubuntu or Debian, install the package, then check for a cluster with + `sudo pg_lsclusters`: + + ```bash + sudo apt update + sudo apt install -y pgedge-enterprise-postgres-18 + sudo pg_lsclusters + ``` -```bash -sudo dnf install -y pgedge-enterprise-postgres_18 -sudo /usr/pgsql-18/bin/postgresql-18-setup initdb -sudo systemctl enable --now postgresql-18 -``` +2. Confirm that the server is running before you continue: -On Ubuntu or Debian, install the package, then check for a cluster with -`sudo pg_lsclusters`. If it lists no `18/main` cluster, create and start one: + ```bash + sudo -u postgres psql -c "SELECT version();" + ``` -```bash -sudo apt update -sudo apt install -y pgedge-enterprise-postgres-18 -sudo pg_lsclusters -sudo pg_createcluster 18 main --start # only if pg_lsclusters lists none -``` + The command displays the PostgreSQL version. If the server is not running + and `sudo pg_lsclusters` lists no `18/main` cluster, use the following + command to create and start the cluster: -On either platform, confirm that the server is running before you continue: + ```bash + sudo pg_createcluster 18 main --start + ``` -```bash -sudo -u postgres psql -c "SELECT version();" -``` + !!! hint -The command prints the PostgreSQL version. A connection error means the server -is not running; check `sudo systemctl status postgresql-18` on RHEL-family -systems or `sudo pg_lsclusters` on Ubuntu and Debian. + A connection error means the server is not running; check + `sudo systemctl status postgresql-18` on RHEL-family systems or + `sudo pg_lsclusters` on Ubuntu and Debian. For details, see the pgEdge Enterprise Postgres installation guide for [Enterprise Linux](https://docs.pgedge.com/enterprise/el/installing/) or @@ -66,93 +75,301 @@ For details, see the pgEdge Enterprise Postgres installation guide for ### Installing the ColdFront Packages -With PostgreSQL running, install the ColdFront extension package for your -PostgreSQL major version, the ColdFront command-line tools (the archiver, the -partitioner, and the compactor), and Lakekeeper (the Iceberg catalog). The -extension package depends on pg_duckdb and ColdFront's DuckDB extensions, so -the package manager installs those as well. The following table shows each -package's purpose and its name on RHEL-family and Debian-family systems: - -| Package | RHEL, Rocky Linux, AlmaLinux | Ubuntu, Debian | -|---|---|---| -| pgEdge Enterprise Postgres | `pgedge-enterprise-postgres_` | `pgedge-enterprise-postgres-` | -| ColdFront extension | `pgedge-coldfront_` | `pgedge-postgresql--coldfront` | -| ColdFront's DuckDB extensions | `pgedge-coldfront-duckdb-extensions` | `pgedge-coldfront-duckdb-extensions` | -| pg_duckdb | `pgedge-pg-duckdb_` | `pgedge-postgresql--pg-duckdb` | -| ColdFront command-line tools (archiver, partitioner, compactor) | `pgedge-coldfront` | `pgedge-coldfront` | -| Lakekeeper | `pgedge-lakekeeper` | `pgedge-lakekeeper` | - -The pg_duckdb packages and ColdFront's DuckDB extensions package are patched -builds, not the upstream releases. Together they include ColdFront's patches to -duckdb-iceberg, to pg_duckdb, and to DuckDB itself. -[Preparing the DuckDB Base Image](#preparing-the-duckdb-base-image) describes -each patch and why ColdFront applies it. - -For PostgreSQL 18 on RHEL, Rocky Linux, or AlmaLinux: - -```bash -sudo dnf install -y pgedge-coldfront_18 pgedge-coldfront pgedge-lakekeeper -``` - -For PostgreSQL 18 on Ubuntu or Debian: - -```bash -sudo apt install -y pgedge-postgresql-18-coldfront pgedge-coldfront \ - pgedge-lakekeeper -``` +With PostgreSQL running, install the ColdFront packages for your PostgreSQL +major version: + +1. Install the ColdFront extension. On RHEL, Rocky Linux, or AlmaLinux, use + the command: + + ```bash + sudo dnf install -y pgedge-coldfront_18 + ``` + + On Ubuntu or Debian, use the command: + + ```bash + sudo apt install -y pgedge-postgresql-18-coldfront + ``` + + For another PostgreSQL major version, replace `18` in the package name + with that version. + +2. Install the ColdFront command-line tools and the Lakekeeper catalog. On + RHEL, Rocky Linux, or AlmaLinux, use the command: + + ```bash + sudo dnf install -y pgedge-coldfront pgedge-lakekeeper + ``` + + On Ubuntu or Debian, use the command: + + ```bash + sudo apt install -y pgedge-coldfront pgedge-lakekeeper + ``` The `pgedge-lakekeeper` package installs Lakekeeper but does not start it; you configure and start it as described in [Configuring Lakekeeper](configure_lakekeeper.md). -## Building ColdFront from Source +!!! hint -This section walks you through building ColdFront from source, either in -Docker on top of the published base image or on bare metal. + The ColdFront extension package depends on pg_duckdb and ColdFront's + DuckDB extensions, so the package manager installs them as well. Both are + patched builds rather than the upstream releases; they include ColdFront's + patches to duckdb-iceberg, pg_duckdb, and DuckDB. + [Understanding ColdFront's DuckDB Stack](#understanding-coldfronts-duckdb-stack) + describes each patch and why ColdFront applies it. -ColdFront runs on a **DuckDB 1.5.x** stack: PostgreSQL + pg_duckdb (DuckDB -1.5.4) and a **patched** duckdb-iceberg that includes ColdFront's seven -patches: +## Building ColdFront from Source on Bare Metal -- the bakery-aware commit-refresh patch (the no-409 guarantee for - concurrent cold-tier writers). -- two strict-reader interop patches, so apache/iceberg-go, the cold-tier - compactor, can read the manifests duckdb-iceberg writes. -- a port of an upstream fix that computes the time partitions of a - `timestamptz` column in UTC. -- a port of another that numbers a new table schema above the highest - schema id. -- a patch that keeps a table's vended storage secret across binds, so a - statement that reads the table twice never has a read go out unsigned. -- a port of an upstream fix that makes a failed manifest read fail the query - instead of aborting the backend. +Before you build ColdFront on bare metal, make sure that you have the +following: -ColdFront also patches pg_duckdb and the DuckDB that pg_duckdb compiles, to -close two security vulnerabilities. [Patching pg_duckdb](#patching-pg_duckdb) -and [Patching DuckDB](#patching-duckdb) explain each patch and the -vulnerability it closes. +- A PostgreSQL 16, 17, or 18 server. +- `pg_config` and the PostgreSQL server development headers. +- The libpq client headers and library (`libpq-dev` or `libpq-devel`). +- `make` and `gcc`. +- Go 1.26.10 or later (pinned in + [go.mod](https://github.com/pgEdge/ColdFront/blob/main/go.mod)), to build the + archiver and the partitioner; `make build` produces `./bin/archiver` and + `./bin/partitioner`. +- golangci-lint, to build the compactor with `make compactor`. -No released pg_duckdb tag includes DuckDB 1.5.x yet, so the stack is built -from a pinned upstream PR plus ColdFront's patches - all from sources you -can fetch. +### Preparing pg_duckdb and the DuckDB Extensions + +When the prerequisites are in place, start by building and installing pg_duckdb +on DuckDB 1.5.4 and the patched DuckDB extensions, with the patches that +[Understanding ColdFront's DuckDB Stack](#understanding-coldfronts-duckdb-stack) +describes. The following steps reproduce the compile steps in +[`docker/Dockerfile.duckdb15-base`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15-base) +on a RHEL-family host running PostgreSQL 18; adjust the package names and the +PostgreSQL major version for your system. + +1. Install the build tools and libraries: + + ```bash + sudo dnf install -y pgedge-postgresql18-devel gcc gcc-c++ make cmake \ + ninja-build git redhat-rpm-config openssl-devel libcurl-devel \ + lz4-devel zlib-devel libicu-devel pkgconf-pkg-config python3 \ + perl-IPC-Cmd flex bison autoconf libtool zip unzip wget tar + export PATH=/usr/pgsql-18/bin:$PATH + export PG_CONFIG=/usr/pgsql-18/bin/pg_config + ``` + + !!! hint + + DuckDB 1.5.4 needs libcurl 7.77 or later to compile. Run + `curl-config --version` to check the installed version. On a system + with an older version, the following script installs libcurl 8.12.0 + under `/usr/local`, leaving the system libcurl in place, and makes the + newer version available to the build and to PostgreSQL: + + ```bash + wget https://curl.se/download/curl-8.12.0.tar.gz + tar xf curl-8.12.0.tar.gz && cd curl-8.12.0 + ./configure --with-openssl --prefix=/usr/local --disable-static \ + --without-libpsl --without-libssh2 --without-nghttp2 \ + --without-brotli --without-zstd + make -j"$(nproc)" && sudo make install + echo /usr/local/lib | sudo tee /etc/ld.so.conf.d/curl-local.conf + sudo ldconfig + export PATH=/usr/local/bin:$PATH + export CMAKE_PREFIX_PATH=/usr/local + cd .. + ``` + +2. Clone the ColdFront repository, which holds the patch files in its `docker` + directory, and set `COLDFRONT_SRC` to the location of the copy: + + ```bash + git clone https://github.com/pgEdge/ColdFront "$HOME/coldfront" + export COLDFRONT_SRC=$HOME/coldfront + ``` + +3. Fetch pg_duckdb at the pinned commit, apply ColdFront's `EXTRACT` + field-quoting patch to pg_duckdb (see + [Patching pg_duckdb](#patching-pg_duckdb)) and its brotli patch to the DuckDB + that pg_duckdb compiles (see [Patching DuckDB](#patching-duckdb)), and then + build and install pg_duckdb: + + ```bash + git clone https://github.com/duckdb/pg_duckdb && cd pg_duckdb + git checkout c04e6a2dcf4e999abb921da1ba2f8335dad644e0 + git submodule update --init --recursive + git apply "$COLDFRONT_SRC/docker/pg_duckdb-extract-field-quoting.patch" && + git -C third_party/duckdb apply \ + "$COLDFRONT_SRC/docker/duckdb-brotli-block-switch-v15.patch" && + make -j"$(nproc)" with_llvm=no && + sudo make install with_llvm=no PG_CONFIG="$PG_CONFIG" + cd .. + ``` + + The `&&` operators chain the patch, build, and install commands, so if + either patch fails to apply, pg_duckdb is not built or installed. + + If the compiler rejects `-fexcess-precision=standard` for C++, append the + line + `override CXXFLAGS := $(filter-out -fexcess-precision=standard,$(CXXFLAGS))` + to `pg_duckdb/Makefile.global` and run `make` again. + +4. Fetch duckdb-iceberg at the pinned commit, replace its DuckDB submodule with + DuckDB 1.5.4, and set up vcpkg: + + ```bash + git clone https://github.com/duckdb/duckdb-iceberg && cd duckdb-iceberg + git checkout 5edc45f0 + git submodule update --init --recursive + rm -rf duckdb + git clone --depth 1 --branch v1.5.4 --recurse-submodules \ + https://github.com/duckdb/duckdb duckdb + git clone https://github.com/microsoft/vcpkg ../vcpkg + ../vcpkg/bootstrap-vcpkg.sh -disableMetrics + cp "$COLDFRONT_SRC/docker/iceberg-azure-extension-config-v15.cmake" \ + extension_config.cmake + ``` + +5. Apply ColdFront's duckdb-iceberg patches in the following order; + [Patching duckdb-iceberg](#patching-duckdb-iceberg) describes each patch: + + ```bash + ( + for p in iceberg-bakery-aware-commit-refresh-v15 \ + iceberg-manifest-list-format-version-v15 \ + iceberg-data-file-format-v15 \ + iceberg-timestamptz-utc-transforms-v15 \ + iceberg-schema-id-allocation-v15 \ + iceberg-vended-secret-keep-v15 \ + iceberg-manifest-read-error-v15; do + git apply "$COLDFRONT_SRC/docker/$p.patch" || + { echo "failed: $p"; exit 1; } + done + ) + ``` + + The parentheses run the loop in a subshell, so a failed patch stops the + loop with a nonzero status without closing your terminal. If a patch + fails, fix it before you continue to the next step. + +6. Build the iceberg, avro, azure, and postgres_scanner extensions: + + ```bash + case "$(uname -m)" in + x86_64) VCPKG_TRIPLET=x64-linux ;; + aarch64) VCPKG_TRIPLET=arm64-linux ;; + esac + export VCPKG_ROOT=$PWD/../vcpkg + export VCPKG_TOOLCHAIN_PATH=$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake + export VCPKG_TARGET_TRIPLET=$VCPKG_TRIPLET VCPKG_HOST_TRIPLET=$VCPKG_TRIPLET + export USE_MERGED_VCPKG_MANIFEST=1 + export EXT_CONFIG=$PWD/extension_config.cmake + export OVERRIDE_GIT_DESCRIBE=v1.5.4 + make -j"$(nproc)" release + ``` + +7. Copy the built extensions into the pg_duckdb extension directory under the + PostgreSQL data directory, as + [`docker/entrypoint.sh`](https://github.com/pgEdge/ColdFront/blob/main/docker/entrypoint.sh) + does. Set `PGDATA` to your data directory: + + ```bash + case "$(uname -m)" in + x86_64) DUCKDB_PLATFORM=linux_amd64 ;; + aarch64) DUCKDB_PLATFORM=linux_arm64 ;; + esac + EXTDIR="$PGDATA/pg_duckdb/extensions/v1.5.4/$DUCKDB_PLATFORM" + sudo -u postgres mkdir -p "$EXTDIR" + for e in iceberg avro azure postgres_scanner; do + sudo install -o postgres -g postgres -m 644 \ + "build/release/extension/$e/$e.duckdb_extension" "$EXTDIR/" + done + ``` + +### Building and Configuring the ColdFront Extension + +Then, after you prepare pg_duckdb and the DuckDB extensions, use the following +steps to build, install, and configure the `coldfront` extension, which is a +standard PGXS C extension: + +1. Build and install the extension: + + ```bash + cd "$COLDFRONT_SRC/extension/coldfront" + make # needs pg_config and the PostgreSQL server dev headers on PATH + sudo make install PG_CONFIG="$(command -v pg_config)" + ``` + +2. Set the following parameters in `postgresql.conf`: + + ```ini + shared_preload_libraries = 'pg_duckdb,coldfront' + # The locally built iceberg extension is unsigned. + duckdb.allow_unsigned_extensions = true + duckdb.autoinstall_known_extensions = false + duckdb.postgres_role = 'coldfront_duckdb' + coldfront.warehouse = '' + coldfront.lakekeeper_endpoint = 'http://:8181/catalog' + coldfront.local_pg_dsn = 'host=/var/run/postgresql dbname= user=' + # Optional, and only with the patched duckdb-iceberg: async cold-write ordering. + coldfront.iceberg_async_parquet = on + coldfront.iceberg_bakery_patch = on + ``` + +3. Restart PostgreSQL so that the server loads pg_duckdb and ColdFront: + + ```bash + sudo systemctl restart postgresql-18 + ``` + + !!! hint + + ColdFront loads only through `shared_preload_libraries`. On a server that + does not preload it, `CREATE EXTENSION coldfront` fails with the error + `coldfront must be loaded via shared_preload_libraries`; add `coldfront` + after `pg_duckdb` in that setting and restart the server. + +4. Create the `coldfront_duckdb` role that `duckdb.postgres_role` names. Only + members of that role can use DuckDB; if you leave the parameter out, only + superusers can use DuckDB, and `coldfront.grant_app_access()` returns an + error. Create the role before you grant access to an application role: + + ```sql + CREATE ROLE coldfront_duckdb NOLOGIN; + ``` -### Prerequisites +See the [Configuring PostgreSQL](configuration.md#configuring-postgresql) +section of the Configuring ColdFront guide for the remaining parameters, and +[Connecting to ColdFront](connecting.md) for granting application roles access +with `coldfront.grant_app_access()`. -The following table lists the prerequisites for each build path: +## Building ColdFront from Source on Docker -| For | You need | -|---|---| -| Docker build (below) | Docker, network access (GitHub, ghcr.io, quay.io, curl.se, and the distribution's RPM repositories), a few GB of disk and RAM, and 30-60 minutes for the base compile. | -| The archiver and partitioner (all paths) | Go 1.26.10+ (pinned in [go.mod](https://github.com/pgEdge/ColdFront/blob/main/go.mod)) and `make` (`make build` produces `./bin/archiver` and `./bin/partitioner`). | -| The compactor and the CI gate | golangci-lint for `make compactor`; `./run-ci-local.sh` also needs Docker and mkdocs with mkdocs-material. | -| Bare metal (near the end of this section) | A PostgreSQL 16, 17, or 18 server, `pg_config`, PostgreSQL server dev headers, libpq client headers and library (libpq-dev / libpq-devel), `make`, and `gcc`. | +!!! warning + + The Docker image is built for development and testing. Its `pg_hba.conf` + trusts every connection from any address without a password, so do not + expose the containers to an untrusted network. + +Before you build ColdFront on Docker, make sure that you have the following: + +- Docker. +- Network access to GitHub, ghcr.io, quay.io, curl.se, and the distribution's + RPM repositories. +- A few GB of disk space and RAM, and 30 to 60 minutes for the base compile. +- Go 1.26.10 or later (pinned in + [go.mod](https://github.com/pgEdge/ColdFront/blob/main/go.mod)) and `make`, + to build the archiver and the partitioner; `make build` produces + `./bin/archiver` and `./bin/partitioner`. +- golangci-lint, to build the compactor with `make compactor`. ### Preparing the DuckDB Base Image +When the prerequisites are in place, start with the DuckDB base image, which +holds the components that take longest to compile. [`docker/Dockerfile.duckdb15-base`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15-base) contains the recipe for the base image: it fetches the requirements, applies -ColdFront's patches, and compiles a set of components. The following -table shows each component and its source: +the ColdFront patches that [Understanding ColdFront's DuckDB +Stack](#understanding-coldfronts-duckdb-stack) describes, and compiles a set of +components. The following table shows each component and its source: | Component | Source | |---|---| @@ -164,108 +381,21 @@ table shows each component and its source: | vcpkg | `github.com/microsoft/vcpkg` | | Base images | `ghcr.io/pgedge/pgedge-postgres:-spock5-minimal` and `quay.io/pypa/manylinux_2_28_` | -The base build runs as three Docker stages: the first builds libcurl and -pg_duckdb, with ColdFront's pg_duckdb patch applied and its brotli patch -applied to pg_duckdb's DuckDB; the second clones duckdb-iceberg at the pinned -ref, applies ColdFront's seven patches, and compiles the iceberg, avro, azure, -and -postgres_scanner extensions under vcpkg; the third assembles the runtime. The -build `git apply --check`s each patch before applying it, so it fails loudly on -patch rot rather than silently shipping stock iceberg (which fails with HTTP -409 under concurrency and writes manifests strict Apache readers reject). +The base image is built in three Docker stages: -#### Patching duckdb-iceberg +1. The first stage builds libcurl, patches pg_duckdb and its bundled copy of + DuckDB, and then builds pg_duckdb. +2. The second stage clones duckdb-iceberg at the pinned commit, applies + ColdFront's patches (see [Patching duckdb-iceberg](#patching-duckdb-iceberg)), + and compiles the iceberg, avro, azure, and postgres_scanner extensions with + vcpkg. +3. The third stage assembles the runtime image. -ColdFront applies seven patches to duckdb-iceberg. The following table -describes what each one does: - -| Patch | What it does | -|---|---| -| `iceberg-bakery-aware-commit-refresh-v15` | Refreshes the table head at the commit POST (and re-creates the table's storage secret in the commit context, needed under catalog credential vending) so concurrent cold writers never get a Lakekeeper 409 (the no-409 guarantee). It also lets writers queued behind the first commit into a never-written table proceed instead of failing as "already outdated". | -| `iceberg-manifest-list-format-version-v15` | Adds the spec-optional `format-version` key to the manifest-list metadata so strict Apache readers parse the entries as v2. | -| `iceberg-data-file-format-v15` | Upper-cases the data-file format in the manifest to match the spec enum strict readers check case-sensitively. | -| `iceberg-timestamptz-utc-transforms-v15` | Computes the year/month/day/hour partition of a `timestamptz` column on the UTC instant rather than in the session's time zone (a port of upstream duckdb-iceberg d3c3348271). | -| `iceberg-schema-id-allocation-v15` | Numbers a new table schema one above the highest schema id in the metadata rather than above the current one, so a column change after a `DROP COLUMN` that returned the table to an earlier schema is accepted (a port of upstream duckdb-iceberg c1cfe2ef). | -| `iceberg-vended-secret-keep-v15` | Keeps a table's vended storage secret when the same transaction binds the table again, instead of replacing it, so a manifest read running on another thread never finds the secret missing between the drop and the write and goes unsigned to the default endpoint. | -| `iceberg-manifest-read-error-v15` | Makes a failed manifest read fail the query rather than abort the backend: the shared scan state's destructor no longer lets the task error escape (a port of upstream duckdb-iceberg 45f323eb). | - -The bakery patch is mandatory for the no-409 guarantee. The two interop patches -make the manifests duckdb-iceberg writes readable by strict Apache readers such -as apache/iceberg-go, the cold-tier compactor; they are inert to pg_duckdb's -own reads. The fourth is what makes a partitioned cold table correct when -written from a session whose time zone is not UTC. The fifth lets a table take -a new column after a change that returned it to an earlier schema. The sixth -keeps a table's vended secret across the binds of one transaction, so a read -on another thread never goes out unsigned. The seventh makes a failed manifest -read fail the query rather than abort the backend. The -canonical recipe - every source pin and compile step - is -[`docker/Dockerfile.duckdb15-base`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15-base) -itself. - -#### Patching pg_duckdb - -ColdFront applies one patch to pg_duckdb, `pg_duckdb-extract-field-quoting`. -pg_duckdb turns the query it runs on DuckDB into SQL text with its vendored -copies of PostgreSQL's `ruleutils.c`, and those copies write the field name of -`EXTRACT(field FROM x)` as it is. PostgreSQL's parser accepts any string as the -field, so an unquoted field would reach DuckDB as SQL and run there with the -session's attachments and secrets. The patch quotes the field name, as -PostgreSQL's own `ruleutils.c` does since its fix for CVE-2026-15741; a valid -field such as `year` stays bare, and any other string is refused as a unit. The -pg_duckdb packages built from `packaging/pg_duckdb` apply the same patch. - -#### Patching DuckDB - -ColdFront patches DuckDB because no DuckDB release up to and including 2.0.0 -fixes CVE-2026-111497. The vulnerability is an out-of-bounds write in the -decoder of brotli 1.1.0, which DuckDB 1.5.4 includes. DuckDB's Parquet reader -decodes every Brotli-compressed page with that decoder. A crafted Parquet file -read through DuckDB can therefore trigger the out-of-bounds write inside a -PostgreSQL backend. The `duckdb-brotli-block-switch-v15` patch ports brotli -1.1.1's fix into the copy of DuckDB 1.5.4 that pg_duckdb compiles. The patched -decoder refuses the crafted stream. The pg_duckdb RPM and Debian packages apply -the same patch. - -## Building the Image on Docker - -Build the stack in two stages, the prebuilt base and the thin app layer: - -```bash -git clone && cd coldfront - -# 1. Build the base (fetches the deps above, applies our patches, compiles -# pg_duckdb on DuckDB 1.5.4 + the patched duckdb-iceberg). ~30–60 min, -# needs network + a few GB of disk/RAM. Repeat with =16 / =17 for those majors. -docker build -f docker/Dockerfile.duckdb15-base --build-arg PG_MAJOR=18 \ - -t ghcr.io/pgedge/coldfront-duckdb-base:pg18 . - -# 2. Build the thin coldfront app layer + bring up the stack (seconds - it only -# compiles the coldfront extension on top of the base). -docker compose up -d --build # end-user single-node stack (ports published) -# (CI uses docker-compose.matrix.yml / docker-compose.mesh.yml - NOT for end-user setup) -``` - -The split keeps app builds fast and always testing current source: the -expensive, stable compiles (pg_duckdb on DuckDB 1.5.4 + the patched -duckdb-iceberg) live in the prebuilt **base**, published to -`ghcr.io/pgedge/coldfront-duckdb-base:pg{16,17,18}`; the **app** build -([`docker/Dockerfile.duckdb15`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15)) -`FROM`s it and compiles the `coldfront` extension in seconds. If you build the -base yourself (step 1) the app layer `FROM`s your local image; otherwise it -`FROM`s the published `ghcr.io/pgedge/coldfront-duckdb-base:pg`. Rebuild -the published base via the -[base-image workflow](https://github.com/pgEdge/ColdFront/blob/main/.github/workflows/base-image.yml) -(`gh workflow run base-image.yml --ref -f push=true`) when its inputs -change; without `push=true` the workflow builds but publishes nothing. CI -builds the app on the tag `ci/base-ref.sh ` prints, `pg-r`, -where the hash covers the base Dockerfile and the files it copies in, and stops -before the app build if that tag is not published, naming the command that -publishes it. The floating `pg` tag moves to the same image only when -the base-image workflow is dispatched from `main` with `push=true` and both -architectures. - -Then follow the [Configuring Lakekeeper](configure_lakekeeper.md) guide: -bootstrap it, create a table, tier it, and verify. +The build checks that each patch applies cleanly before applying it. If a patch +no longer applies, the build stops with an error instead of producing an image +with an unpatched duckdb-iceberg. An unpatched duckdb-iceberg fails with HTTP +409 errors under concurrent writes and writes manifests that strict Apache +Iceberg readers reject. !!! note "pg_duckdb pin" @@ -279,17 +409,79 @@ bootstrap it, create a table, tier it, and verify. need pull access to that image (or substitute an equivalent PostgreSQL base with the same layout). +### Building and Starting the ColdFront Containers + +Then, build the base image and start the ColdFront containers. Docker Compose +builds the ColdFront PostgreSQL image and starts it with Lakekeeper, the +database that Lakekeeper stores its catalog in, and, with the `local-store` +profile, a SeaweedFS object store for testing. Together, these containers make +up the ColdFront stack. + +The build runs in two stages. Splitting it keeps the slow compiles in a base +image that rarely changes, so rebuilding ColdFront itself takes seconds and +always uses the current source: + +1. Clone the ColdFront repository, change to its directory, and set + `PG_MAJOR` to the PostgreSQL major version to build, 16, 17, or 18: + + ```bash + git clone https://github.com/pgEdge/ColdFront coldfront && cd coldfront + export PG_MAJOR=18 + ``` + + The later commands in this section and in + [Verifying the Build](#verifying-the-build) read `PG_MAJOR`, so run them in + the same shell. + +2. Build the base image, which compiles pg_duckdb on DuckDB 1.5.4 and the + patched duckdb-iceberg that + [Understanding ColdFront's DuckDB Stack](#understanding-coldfronts-duckdb-stack) + describes. The build takes 30 to 60 minutes and needs network access and a + few GB of disk space and RAM: + + ```bash + docker build -f docker/Dockerfile.duckdb15-base \ + --build-arg PG_MAJOR="$PG_MAJOR" \ + -t "ghcr.io/pgedge/coldfront-duckdb-base:pg$PG_MAJOR" . + ``` + + pgEdge also publishes the base image as + `ghcr.io/pgedge/coldfront-duckdb-base:pg16`, `:pg17`, and `:pg18`. If you + skip this step, the next step uses the published image. + +3. Build the app image and start the stack. The app image, built from + [`docker/Dockerfile.duckdb15`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15), + starts from your local base image if you built one in step 2, or from the + published image otherwise, and compiles only the `coldfront` extension. + The commands publish each port on the loopback address only, because the + PostgreSQL server trusts every connection and Lakekeeper runs without + authentication: + + ```bash + export COLDFRONT_PG_PORT=127.0.0.1:5432 COLDFRONT_LK_PORT=127.0.0.1:8181 \ + COLDFRONT_S3_PORT=127.0.0.1:8333 + docker compose up -d --build + ``` + + Compose builds the app image for the version in `PG_MAJOR`. The exported + port variables apply to every later Compose command in the same shell, + including the commands in [Verifying the Build](#verifying-the-build). + Without them, Compose publishes the ports on every network interface. The command starts a single-node stack for one + host. The repository's other Compose files, `docker-compose.matrix.yml` + and `docker-compose.mesh.yml`, are for CI rather than for end-user setups. + ### Using Environment Variables -The entrypoint reads the following variables when the container starts. It -writes the server settings they control into `postgresql.conf` only when the -data directory is empty, so a change to one of them takes effect on a fresh -volume or after editing `postgresql.conf` by hand. The following table -describes each variable: +After the stack is running, you can change how the ColdFront container starts +with environment variables. The entrypoint reads the following variables when +the container starts. It writes the server settings they control into +`postgresql.conf` only when the data directory is empty, so a change to one of +them takes effect on a fresh volume or after editing `postgresql.conf` by hand. +The following table describes each variable: | Variable | Default | Effect | |----------|---------|--------| -| `PG_MAJOR` | set by the image | The PostgreSQL major the entrypoint starts. The image sets it from its build argument, so a value given at run time must match. | +| `PG_MAJOR` | set by the image | The PostgreSQL major version that the entrypoint starts. The image sets this variable from the `PG_MAJOR` build argument, so any value you set when you run the container must match the version the image was built for. | | `MESH` | `off` | `on` adds `snowflake` and `spock` to `shared_preload_libraries` and writes the mesh settings: logical WAL, Spock DDL replication, `snowflake.node` and `coldfront.loopback_dsn`. | | `COLDFRONT_WAREHOUSE` | `wh` | The Lakekeeper warehouse name written to `coldfront.warehouse`. | | `COLDFRONT_LAKEKEEPER` | `http://lakekeeper:8181/catalog` | The catalog endpoint written to `coldfront.lakekeeper_endpoint`. | @@ -299,199 +491,197 @@ describes each variable: | `COLDFRONT_DUCKDB_VERSION` | `v1.5.4` | The version directory under `$PGDATA/pg_duckdb/extensions/` that receives the patched DuckDB extensions. | | `COLDFRONT_DUCKDB_PLATFORM` | from `uname -m` | The platform directory under that version, `linux_amd64` or `linux_arm64`. | -The image is built for development and testing: `pg_hba.conf` trusts every -connection from any address without a password. - ### Verifying the Build -A self-contained smoke test confirms the freshly built stack works end to end: -pg_duckdb, the patched duckdb-iceberg, Lakekeeper, and the object store. The -fastest path needs no cloud credentials; bring the stack up with the in-compose -SeaweedFS S3 emulator under the `local-store` profile: - -```bash -docker compose --profile local-store up -d --build -``` - -Bootstrap Lakekeeper, create the `wh` warehouse against the SeaweedFS -credentials in -[`docker/seaweedfs-s3.json`](https://github.com/pgEdge/ColdFront/blob/main/docker/seaweedfs-s3.json), -and seed the `public` namespace: - -```bash -curl -sf -X POST http://localhost:8181/management/v1/bootstrap \ - -H 'Content-Type: application/json' -d '{"accept-terms-of-use":true}' - -WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ - -H 'Content-Type: application/json' -d '{ - "warehouse-name":"wh", - "storage-profile":{"type":"s3","bucket":"iceberg","region":"us-east-1", - "endpoint":"http://seaweedfs:8333","path-style-access":true, - "flavor":"s3-compat","sts-enabled":false,"remote-signing-enabled":false}, - "storage-credential":{"type":"s3","credential-type":"access-key", - "aws-access-key-id":"admin","aws-secret-access-key":"adminsecret"} - }' \ - | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) - -if [ -z "$WID" ]; then - echo "No warehouse ID - check that the warehouse create succeeded." -else - curl -s -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ - -H 'Content-Type: application/json' -d '{"namespace":["public"]}' -fi -``` - -If you run these commands again after the warehouse exists, the create -request fails and `WID` is empty. Set `WID` to the `warehouse-id` of the `wh` -warehouse in the output of -`curl -s http://localhost:8181/management/v1/warehouse`, then rerun the -namespace request. - -Create the extensions, set the cold-store secret, create a decoupled table, -insert a row, and read it back through Iceberg: - -```bash -psql -h localhost -U coldfront -d coldfront <<'SQL' -CREATE EXTENSION IF NOT EXISTS pg_duckdb; -CREATE EXTENSION IF NOT EXISTS coldfront; -SELECT coldfront.set_storage_secret('admin', 'adminsecret', 'seaweedfs:8333'); -SELECT coldfront.create_iceberg_table('public', 'events', - '[{"name":"id","type":"bigint"},{"name":"ts","type":"timestamptz"},{"name":"note","type":"text"}]'::jsonb, - '{month(ts)}'); -INSERT INTO events VALUES (1, now(), 'hello'); -SELECT count(*) FROM events; -SQL -``` - -A row count of 1 read back through Iceberg confirms the full path. For a real -cloud store, drop the `local-store` profile, point the warehouse at your own -bucket, and follow the [Configuring Lakekeeper](configure_lakekeeper.md) guide -for the full tier-and-verify journey. - -## Building ColdFront on Bare Metal - -The `coldfront` extension is a standard PGXS C extension: - -```bash -cd extension/coldfront -make # needs pg_config and the PostgreSQL server dev headers on PATH -sudo make install PG_CONFIG="$(command -v pg_config)" -``` - -You separately need pg_duckdb (DuckDB 1.5.4) and the **patched** iceberg DuckDB -extension installed in your PostgreSQL. Follow the compile steps in -[`docker/Dockerfile.duckdb15-base`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15-base), -including the brotli patch it applies to pg_duckdb's DuckDB before compiling -(see [Patching DuckDB](#patching-duckdb)), and copy the built iceberg, avro, -azure, and postgres_scanner extensions into -`$PGDATA/pg_duckdb/extensions/v1.5.4/linux_/`, as -[`docker/entrypoint.sh`](https://github.com/pgEdge/ColdFront/blob/main/docker/entrypoint.sh) -does. Then set the following in `postgresql.conf`: - -```ini -shared_preload_libraries = 'pg_duckdb,coldfront' -# The locally built iceberg extension is unsigned. -duckdb.allow_unsigned_extensions = true -duckdb.autoinstall_known_extensions = false -duckdb.postgres_role = 'coldfront_duckdb' -coldfront.warehouse = '' -coldfront.lakekeeper_endpoint = 'http://:8181/catalog' -coldfront.local_pg_dsn = 'host=/var/run/postgresql dbname= user=' -# Optional, and only with the patched duckdb-iceberg: async cold-write ordering. -coldfront.iceberg_async_parquet = on -coldfront.iceberg_bakery_patch = on -``` - -coldfront refuses to load any other way: `CREATE EXTENSION coldfront` fails on -a server that does not preload it, with an error that names the setting. - -`duckdb.postgres_role` names the role whose members may run DuckDB. Without -it, only superusers can, and `coldfront.grant_app_access()` fails. Create the -role before you grant access to an application role: - -```sql -CREATE ROLE coldfront_duckdb NOLOGIN; -``` +After you start the stack, a self-contained smoke test confirms that the stack +works end to end: pg_duckdb, the patched duckdb-iceberg, Lakekeeper, and the +object store. The fastest path needs no cloud credentials, because it uses the +SeaweedFS S3 emulator that the stack includes under the `local-store` profile: + +1. In the shell where you exported `PG_MAJOR` and the port variables, bring + the stack up with the `local-store` profile: + + ```bash + docker compose --profile local-store up -d --build + ``` + +2. Wait until `docker compose ps` reports the Lakekeeper container as healthy: + + ```bash + until docker compose ps lakekeeper | grep -q '(healthy)'; do sleep 2; done + ``` + +3. Bootstrap Lakekeeper, create the `wh` warehouse against the SeaweedFS + credentials in + [`docker/seaweedfs-s3.json`](https://github.com/pgEdge/ColdFront/blob/main/docker/seaweedfs-s3.json), + and seed the `public` namespace: + + ```bash + curl -sf -X POST http://localhost:8181/management/v1/bootstrap \ + -H 'Content-Type: application/json' -d '{"accept-terms-of-use":true}' + + WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ + -H 'Content-Type: application/json' -d '{ + "warehouse-name":"wh", + "storage-profile":{"type":"s3","bucket":"iceberg","region":"us-east-1", + "endpoint":"http://seaweedfs:8333","path-style-access":true, + "flavor":"s3-compat","sts-enabled":false,"remote-signing-enabled":false}, + "storage-credential":{"type":"s3","credential-type":"access-key", + "aws-access-key-id":"admin","aws-secret-access-key":"adminsecret"} + }' \ + | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) + + if [ -z "$WID" ]; then + echo "No warehouse ID - check that the warehouse create succeeded." + else + curl -s -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ + -H 'Content-Type: application/json' -d '{"namespace":["public"]}' + fi + ``` + + If you run these commands again after the warehouse exists, the create + request fails and `WID` is empty. Set `WID` to the `warehouse-id` of the + `wh` warehouse in the output of `curl -s + http://localhost:8181/management/v1/warehouse`, then rerun the namespace + request. + +4. Create the extensions, set the cold-store secret, create a decoupled table, + insert a row, and read it back through Iceberg: + + ```bash + psql -h localhost -U coldfront -d coldfront <<'SQL' + CREATE EXTENSION IF NOT EXISTS pg_duckdb; + CREATE EXTENSION IF NOT EXISTS coldfront; + SELECT coldfront.set_storage_secret('admin', 'adminsecret', 'seaweedfs:8333'); + SELECT coldfront.create_iceberg_table('public', 'events', + '[{"name":"id","type":"bigint"},{"name":"ts","type":"timestamptz"},{"name":"note","type":"text"}]'::jsonb, + '{month(ts)}'); + INSERT INTO events VALUES (1, now(), 'hello'); + SELECT count(*) FROM events; + SQL + ``` + + The final query returns a count of `1` when the stack works end to end: + ColdFront wrote the row to the object store through Iceberg and read it + back. + +To set up ColdFront against your own object store, bring the stack up without +the `local-store` profile and complete the following steps: + +1. Bootstrap Lakekeeper, which initializes the catalog so that it accepts + requests, as + [Bootstrapping Lakekeeper](configure_lakekeeper.md#bootstrapping-lakekeeper) + describes. +2. Create a Lakekeeper warehouse that points at your bucket, and an Iceberg + namespace in that warehouse, as the same section describes. +3. Create the `pg_duckdb` and `coldfront` extensions in each database that + uses ColdFront, as + [Configuring PostgreSQL for Lakekeeper](configure_lakekeeper.md#configuring-postgresql-for-lakekeeper) + describes. +4. Store the object store credentials with + `coldfront.set_storage_secret()`, as the same section describes. +5. Create a table and tier it to the object store, as + [Using ColdFront in Tiered Mode](usage_tiered.md) describes. + +## Understanding ColdFront's DuckDB Stack + +ColdFront runs on a DuckDB 1.5 stack that consists of PostgreSQL, pg_duckdb +built on DuckDB 1.5.4, and a patched build of duckdb-iceberg. When you install +ColdFront from packages, the package manager installs these patched +dependencies for you; when you build ColdFront from source, you need to build +and install the patched dependencies yourself. + +The duckdb-iceberg build includes the following ColdFront patches: -See the [Configuring PostgreSQL](configuration.md#configuring-postgresql) -section of the Configuring ColdFront guide for the remaining GUCs, and -the README for the optional non-superuser role that the image sets up. - -## Testing & CI - -One canonical user journey -([ci/journey.sh](https://github.com/pgEdge/ColdFront/blob/main/ci/journey.sh)) -runs identically in every deployment cell; `ci/matrix.sh` drives the cells and -`ci/topo/*.sh` brings up each topology. All cells share the DuckDB 1.5.x app -image -([docker/Dockerfile.duckdb15](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15), -built on the prebuilt -[base](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15-base); -`--build-arg PG_MAJOR=16|17|18`). - -### Pre-Commit Gate - -`./run-ci-local.sh` runs `ci/matrix.sh --quick`: gofmt, go vet, golangci-lint, -shellcheck, hadolint, actionlint, unit tests, build, the compactor module gate -(vet, lint, test, build), `mkdocs build --strict`, the base recipe hash check, -the pg_regress unit layer, and the full journey on one representative cell -(PG18 · vanilla · tiered · s3). -The gate is fast and runs on every commit. GitHub Actions -([.github/workflows/ci.yml](https://github.com/pgEdge/ColdFront/blob/main/.github/workflows/ci.yml)) -runs the identical `ci/matrix.sh` harness: `--quick` on every pull request and -every push to `main`, and `--full` nightly and on demand. GitHub also runs a -gitleaks secret scan, a go-licenses check, and the path-filtered walkthrough -workflow, which the local gate does not. - -On `main` only, GitHub also runs the `aws-vended` cells against a real AWS -bucket and the `azure-vended` cells against a real ADLS Gen2 account -(`ci/matrix.sh --backend `, inside `ci/cloud.sh `). The jobs -hold no cloud credential: each requests a GitHub OIDC token, and Lakekeeper -assumes an IAM role from it (AWS) or signs user-delegation SAS tokens as a -managed identity (Azure, on an account with key access off), vending -per-table credentials to DuckDB. The accounts' identifiers are the variables -of the `cloud-aws` and `cloud-azure` environments, whose branch rules allow -`main` alone. Each cell writes under its own prefix and purges it at -teardown, the job purges every prefix again when it ends, and an hourly -workflow -([.github/workflows/sweep.yml](https://github.com/pgEdge/ColdFront/blob/main/.github/workflows/sweep.yml)) -removes anything older than six hours and fails if it found something. - -### Full Matrix - -`ci/matrix.sh --full` is the release gate, covering PG {16, 17, 18} × {vanilla, -mesh (3-node Spock)} × {tiered, decoupled} × {primary, standby} × {s3, aws, -azure, gcs, vended, azure-vended, aws-vended}, where the three vended backends -run on vanilla only. The ops-hardening and probe-snowflake cells run once, on PG18. The mesh -cells add the cross-node stories - hot visibility via Spock, cold visibility -via the shared Lakekeeper catalog, the R-A bakery serializing concurrent cold -writers (same-node and cross-node) with no 409, and an N×(N-1) probe that the -bakery's `coldfront.claims` table replicates in every direction. - -### Storage-Backend Gating - -The same policy applies locally and in GitHub CI: the hermetic -**SeaweedFS-as-S3** backend (`s3`) always runs - that is the default coverage -with no credentials. The real cloud stores run **only when their credentials -are present in the environment**, else they are reported `PENDING` and never -invoked (no real cloud calls without explicit creds). The following table shows -each backend, the variable whose presence turns it on, and the other variables -it then requires; a lane turned on without the rest fails instead of staying -`PENDING`: - -| Backend | Store | Gate variable | Also required | -|---|---|---|---| -| `s3` | SeaweedFS (in-compose, hermetic) | None; this backend always runs. | None | -| `vended` | SeaweedFS STS credential vending (hermetic, vanilla only) | None; this backend always runs. | None | -| `aws` | Real AWS S3 (native virtual-hosted addressing over HTTPS) | `COLDFRONT_AWS_ACCESS_KEY` | `COLDFRONT_AWS_SECRET_KEY`, `_BUCKET`, `_REGION` | -| `azure` | Real Azure ADLS Gen2 | `COLDFRONT_AZURE_CONNECTION_STRING` | `COLDFRONT_AZURE_ACCOUNT`, `_FILESYSTEM`, `_KEY` | -| `azure-vended` | Real Azure ADLS Gen2 with SAS vending (vanilla only) | `COLDFRONT_AZURE_ACCOUNT` | `COLDFRONT_AZURE_FILESYSTEM`, and either `COLDFRONT_AZURE_KEY` (the account's key as the warehouse credential) or, in CI, the job's OIDC token with `COLDFRONT_AZURE_CLIENT_ID` and `_TENANT_ID` (the managed identity, on an account with key access off) | -| `aws-vended` | Real AWS S3 with STS vending and no stored key (vanilla only) | `COLDFRONT_AWS_DATA_ROLE_ARN` | `COLDFRONT_AWS_EXTERNAL_ID`, `_BUCKET` (with versioning never enabled, since the purge deletes without version ids), `_REGION`; in CI also `_CI_ROLE_ARN` and the job's OIDC token, locally the caller's own AWS session, which must be allowed to assume the data role and, since the cell purges its prefix as the caller, `s3:ListBucket` and `s3:ListBucketMultipartUploads` on the bucket and `s3:DeleteObject` and `s3:AbortMultipartUpload` on the objects under `coldfront-ci/` | -| `gcs` | Real GCS via S3 interoperability (HMAC) | `COLDFRONT_GCS_ACCESS_KEY` | `COLDFRONT_GCS_SECRET_KEY`, `_BUCKET` | - -In GitHub Actions these come from repo secrets; an unset secret arrives empty, -so that backend stays `PENDING`. Fork PRs (no secret access) run -SeaweedFS-only. +- the bakery-aware commit-refresh patch (the no-409 guarantee for + concurrent cold-tier writers). +- two strict-reader interop patches, so apache/iceberg-go, the cold-tier + compactor, can read the manifests duckdb-iceberg writes. +- a port of an upstream fix that computes the time partitions of a + `timestamptz` column in UTC. +- a port of another that numbers a new table schema above the highest + schema id. +- a patch that keeps a table's vended storage secret across binds, so a + statement that reads the table twice never has a read go out unsigned. +- a port of an upstream fix that makes a failed manifest read fail the query + instead of aborting the backend. + +ColdFront also patches pg_duckdb and the DuckDB that pg_duckdb compiles, to +close two security vulnerabilities. [Patching pg_duckdb](#patching-pg_duckdb) +and [Patching DuckDB](#patching-duckdb) explain each patch and the +vulnerability it closes. + +No released pg_duckdb tag includes DuckDB 1.5.x yet, so the stack is built +from a pinned upstream PR plus ColdFront's patches - all from sources you +can fetch. + +### Patching duckdb-iceberg + +ColdFront applies the following patches to duckdb-iceberg: + +| Patch | What it does | +|---|---| +| `iceberg-bakery-aware-commit-refresh-v15` | Refreshes the table head at the commit POST (and re-creates the table's storage secret in the commit context, needed under catalog credential vending) so concurrent cold writers never get a Lakekeeper 409 (the no-409 guarantee). It also lets writers queued behind the first commit into a never-written table proceed instead of failing as "already outdated". | +| `iceberg-manifest-list-format-version-v15` | Adds the spec-optional `format-version` key to the manifest-list metadata so strict Apache readers parse the entries as v2. | +| `iceberg-data-file-format-v15` | Upper-cases the data-file format in the manifest to match the spec enum strict readers check case-sensitively. | +| `iceberg-timestamptz-utc-transforms-v15` | Computes the year/month/day/hour partition of a `timestamptz` column on the UTC instant rather than in the session's time zone (a port of upstream duckdb-iceberg d3c3348271). | +| `iceberg-schema-id-allocation-v15` | Numbers a new table schema one above the highest schema id in the metadata rather than above the current one, so a column change after a `DROP COLUMN` that returned the table to an earlier schema is accepted (a port of upstream duckdb-iceberg c1cfe2ef). | +| `iceberg-vended-secret-keep-v15` | Keeps a table's vended storage secret when the same transaction binds the table again, instead of replacing it, so a manifest read running on another thread never finds the secret missing between the drop and the write and goes unsigned to the default endpoint. | +| `iceberg-manifest-read-error-v15` | Makes a failed manifest read fail the query rather than abort the backend: the shared scan state's destructor no longer lets the task error escape (a port of upstream duckdb-iceberg 45f323eb). | + +The following list explains what each patch contributes: + +- The bakery patch (`iceberg-bakery-aware-commit-refresh-v15`) is required for + the no-409 guarantee. +- The two interop patches (`iceberg-manifest-list-format-version-v15` and + `iceberg-data-file-format-v15`) make the manifests that duckdb-iceberg writes + readable by strict Apache Iceberg readers, such as the cold-tier compactor, + which uses apache/iceberg-go. They have no effect on pg_duckdb's own reads. +- The time zone patch (`iceberg-timestamptz-utc-transforms-v15`) keeps a + partitioned cold table correct when a session whose time zone is not UTC + writes to it. +- The schema ID patch (`iceberg-schema-id-allocation-v15`) lets a table take a + new column after a change that returned it to an earlier schema. +- The vended secret patch (`iceberg-vended-secret-keep-v15`) keeps a table's + vended storage secret across the binds of one transaction, so a read on + another thread never goes out unsigned. +- The manifest read patch (`iceberg-manifest-read-error-v15`) makes a failed + manifest read fail the query instead of aborting the backend. + +[`docker/Dockerfile.duckdb15-base`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15-base) +is the definitive source for every pinned source and compile step. + +### Patching pg_duckdb + +ColdFront applies one patch to pg_duckdb, `pg_duckdb-extract-field-quoting`, +which closes a security vulnerability. + +pg_duckdb converts each query it runs on DuckDB into SQL text, using its own +copies of PostgreSQL's `ruleutils.c`. Those copies write the field name in +`EXTRACT(field FROM x)` exactly as given, without quotes. PostgreSQL's parser +accepts any string as the field name, so a crafted field name could reach +DuckDB as SQL and run there with the session's attachments and secrets. + +The patch quotes the field name, as PostgreSQL's own `ruleutils.c` has done +since its fix for CVE-2026-15741. A valid field name, such as `year`, is +unchanged. Any other string is quoted, so DuckDB rejects it as a field name and +pg_duckdb hands the query back to PostgreSQL, which then rejects the unit. The +pg_duckdb packages built from `packaging/pg_duckdb` include the same patch. + +### Patching DuckDB + +ColdFront applies one patch to DuckDB, `duckdb-brotli-block-switch-v15`, which +closes a security vulnerability that no DuckDB release up to and including +2.0.0 fixes. + +DuckDB 1.5.4 includes brotli 1.1.0, whose decoder has an out-of-bounds write +bug (CVE-2026-111497). DuckDB's Parquet reader uses that decoder for every +Brotli-compressed page, so reading a crafted Parquet file through DuckDB can +trigger the out-of-bounds write inside a PostgreSQL backend. + +The patch ports the fix from brotli 1.1.1 into the copy of DuckDB 1.5.4 that +pg_duckdb compiles, so the decoder rejects the crafted data. The pg_duckdb RPM +and Debian packages include the same patch. ## Next Steps @@ -500,10 +690,12 @@ To go further with ColdFront, consult the following guides: - The [Configuring ColdFront](configuration.md) guide details PostgreSQL and ColdFront's own configuration. - The [Configuring Lakekeeper](configure_lakekeeper.md) guide installs, - configures, and bootstraps it, then installs the extensions each - database needs. -- The [Walkthrough](exploring_coldfront/index.md) guide runs the demo stack hands-on. + configures, and bootstraps Lakekeeper, then installs the extensions that + each database needs. - The [Using ColdFront](using_coldfront/index.md) guide documents the tiered, decoupled, and standalone partitioned modes. -- The [Configuring your Object Store](object_store.md) - guide connects the cold tier to AWS S3. +- The [Configuring your Object Store](object_store.md) guide connects the + cold tier to an S3-compatible store, AWS S3, Google Cloud Storage, or Azure + ADLS Gen2. +- The [Testing and Continuous Integration](testing_ci.md) guide describes how + to run ColdFront's test suite and how CI tests each change. diff --git a/docs/testing_ci.md b/docs/testing_ci.md new file mode 100644 index 00000000..304710c3 --- /dev/null +++ b/docs/testing_ci.md @@ -0,0 +1,184 @@ +# Testing and Continuous Integration + +ColdFront's test harness runs one canonical user journey, +[`ci/journey.sh`](https://github.com/pgEdge/ColdFront/blob/main/ci/journey.sh), +identically in every deployment cell. `ci/matrix.sh` drives the cells, and the +scripts in `ci/topo/` bring up each topology. Every cell uses the same DuckDB +1.5 app image, +[`docker/Dockerfile.duckdb15`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15), +built on the prebuilt +[base image](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15-base) +with `--build-arg PG_MAJOR=16`, `17`, or `18`. To build ColdFront before you +test it, see [Installing ColdFront](installation.md). + +## Pre-Commit Gate + +Before you run the pre-commit gate, make sure that the host has the +following: + +- Git and a clone of the ColdFront repository; run the gate from the top + directory of the clone. +- Go 1.26.10 or later, golangci-lint, and `make`, for the Go checks, the unit + tests, and the build. `make test` runs the tests with the race detector, + which needs cgo, so on Linux the host also needs a C compiler such as + `gcc`. +- mkdocs with mkdocs-material, for the documentation build. +- Docker with the Docker Compose plugin, for the shellcheck, hadolint, and + actionlint checks, which run in containers, and for the stack that the + pg_regress tests and the journey use. +- `curl`, which the journey and topology scripts use to call Lakekeeper. + +The topology scripts and the journey reach PostgreSQL, Lakekeeper, and +SeaweedFS from the host at each container's IP address, so the host must be +able to route traffic to Docker container IP addresses. Docker Engine on Linux +does this; Docker Desktop on macOS and Windows does not, so run the gate on a +Linux host or in a Linux virtual machine that runs Docker Engine. + +Run the gate with `./run-ci-local.sh`, which runs `ci/matrix.sh --quick`. The +gate is fast enough to run on every commit and performs the following checks: + +- The Go checks: gofmt, go vet, golangci-lint, the unit tests, and the build. +- The shell, Dockerfile, and workflow linters: shellcheck, hadolint, and + actionlint. +- The compactor module checks: vet, lint, test, and build. +- The documentation build, `mkdocs build --strict`. +- The base recipe hash check. +- The pg_regress unit tests. +- The full user journey on one representative cell: PostgreSQL 18, vanilla, + tiered mode, and the `s3` backend. + +GitHub Actions +([.github/workflows/ci.yml](https://github.com/pgEdge/ColdFront/blob/main/.github/workflows/ci.yml)) +runs the same `ci/matrix.sh` harness: `--quick` on every pull request and every +push to `main`, and `--full` nightly and on demand. GitHub also runs a gitleaks +secret scan, a go-licenses check, and the path-filtered walkthrough workflow, +which the local gate does not run. + +On the `main` branch only, GitHub also runs two sets of PostgreSQL 18 cells +against real cloud storage: the `aws-vended` cells against an AWS S3 bucket, +and the `azure-vended` cells against an Azure Data Lake Storage (ADLS) Gen2 +account. +These jobs run `ci/matrix.sh --backend ` inside `ci/cloud.sh `. + +The jobs store no cloud credentials. Instead, each job requests a GitHub OIDC +token, and Lakekeeper uses that token to obtain short-lived access: + +- On AWS, Lakekeeper assumes an IAM role. +- On Azure, Lakekeeper signs user-delegation SAS tokens as a managed identity; + the storage account has shared key access turned off. + +Lakekeeper then vends per-table credentials to DuckDB. The account identifiers +are stored as variables in the `cloud-aws` and `cloud-azure` GitHub +environments, whose branch rules allow only `main` to use them. + +The vended cloud cells, `aws-vended` and token-based `azure-vended`, clean up +their test data in three ways: + +- Each cell writes under its own prefix and deletes that prefix at teardown. +- In GitHub Actions, each cloud job deletes every prefix again when it ends. +- An hourly workflow + ([.github/workflows/sweep.yml](https://github.com/pgEdge/ColdFront/blob/main/.github/workflows/sweep.yml)) + removes `coldfront-ci/` prefixes older than six hours from the CI bucket, + `COLDFRONT_AWS_BUCKET`, and the CI filesystem, `COLDFRONT_AZURE_FILESYSTEM` + in `COLDFRONT_AZURE_ACCOUNT`, as the `cloud-aws` and `cloud-azure` + environments set them, and fails if it finds anything to remove. + +When you run a vended cell locally, the job-end purge does not apply. The +sweep cleans the CI bucket and the CI filesystem whatever started the run, so +it also covers a local run that uses them. It does not cover any other bucket, +or any other filesystem in the same storage account. If a local run against +other cloud resources does not finish its teardown, delete the cell's prefix +under `coldfront-ci/` in the bucket or filesystem that the run used. + +The `aws`, `azure`, and `gcs` cells and key-based `azure-vended` runs do not +use these mechanisms, so remove the data that those cells write to your bucket +or container yourself. + +## Full Matrix + +Beyond the pre-commit gate, `ci/matrix.sh --full` is the release gate. It runs +the user journey in every combination of the following dimensions: + +| Dimension | Values | +|---|---| +| PostgreSQL version | 16, 17, and 18 | +| Topology | vanilla (a single node) and mesh (three Spock nodes) | +| Operating mode | tiered and decoupled | +| Role | primary and standby | +| Storage backend | `s3`, `aws`, `azure`, `gcs`, `vended`, `azure-vended`, and `aws-vended` | + +The three vended backends run on the vanilla topology only. The ops-hardening +and probe-snowflake cells run once, on PostgreSQL 18. + +A full run skips a cloud backend whose gate variables are incomplete and still +exits successfully. The run prints a coverage table that lists each cell as +`RUN` or `PENDING` with the reason, so check that table for `PENDING` cells +before you rely on the run for cloud coverage. + +The mesh cells also test behaviour across nodes: + +- Hot data written on one node is visible on the others through Spock + replication. +- Cold data is visible on every node through the shared Lakekeeper catalog. +- The bakery serializes concurrent cold writers, on the same node and across + nodes, without any HTTP 409 conflicts. +- A probe confirms that the bakery's `coldfront.claims` table replicates in + both directions between every pair of nodes. + +## Storage-Backend Gating + +The matrix decides which storage backends to test in the same way locally and +in GitHub CI: + +- The SeaweedFS backends, `s3` and `vended`, need no credentials, because + SeaweedFS runs inside the Compose stack. The full matrix runs both; the + pre-commit gate and the `--quick` pull request job run only `s3`. +- A real cloud backend runs only when its credentials are set in the + environment. Without them, the matrix reports the backend as `PENDING` and + makes no calls to that cloud. + +Each cloud backend has gate variables that turn it on, and other variables +that it then requires. In a full matrix run, a backend whose gate variables +are not all set stays `PENDING`, while a backend whose gate variables are set +but whose other required variables are missing runs, and its cells fail. When +you select one backend with `ci/matrix.sh --backend`, a backend whose gate +variables are not all set makes the script exit with an error instead. The +following table shows each backend, its gate variables, and the other +variables it requires: + +| Backend | Store | Gate variables | Also required | +|---|---|---|---| +| `s3` | SeaweedFS (in the Compose stack) | None; needs no credentials, and runs in every scope. | None | +| `vended` | SeaweedFS STS credential vending (in the Compose stack, vanilla only) | None; needs no credentials, and runs in the full matrix only. | None | +| `aws` | Real AWS S3 (native virtual-hosted addressing over HTTPS) | `COLDFRONT_AWS_ACCESS_KEY` | `COLDFRONT_AWS_SECRET_KEY`, `_BUCKET`, `_REGION` | +| `azure` | Real Azure ADLS Gen2 | `COLDFRONT_AZURE_CONNECTION_STRING` | `COLDFRONT_AZURE_ACCOUNT`, `_FILESYSTEM`, `_KEY` | +| `azure-vended` | Real Azure ADLS Gen2 with SAS vending (vanilla only) | `COLDFRONT_AZURE_ACCOUNT`, and either `COLDFRONT_AZURE_KEY` (the account's key as the warehouse credential) or `COLDFRONT_AZURE_TOKEN_FILE` (the job's OIDC token, which `ci/cloud.sh azure` writes and sets) | `COLDFRONT_AZURE_FILESYSTEM`; with the token file, also `COLDFRONT_AZURE_CLIENT_ID` and `_TENANT_ID` (the managed identity, on an account with key access off), and the cells must run under `ci/cloud.sh azure run` | +| `aws-vended` | Real AWS S3 with STS vending and no stored key (vanilla only) | `COLDFRONT_AWS_DATA_ROLE_ARN`, `COLDFRONT_AWS_EXTERNAL_ID`, and `COLDFRONT_AWS_BUCKET` (with versioning never enabled, since the purge deletes without version ids) | `COLDFRONT_AWS_REGION`. In CI, the cells run under `ci/cloud.sh aws run`, which also requires `_CI_ROLE_ARN` and supplies the job's OIDC token. Locally, the AWS CLI on the host, which exports the caller's own AWS session with `aws configure export-credentials`; the session must be allowed to assume the data role and, since the cell purges its prefix as the caller, needs `s3:ListBucket` and `s3:ListBucketMultipartUploads` on the bucket and `s3:DeleteObject` and `s3:AbortMultipartUpload` on the objects under `coldfront-ci/` | +| `gcs` | Real GCS via S3 interoperability (HMAC) | `COLDFRONT_GCS_ACCESS_KEY` | `COLDFRONT_GCS_SECRET_KEY`, `_BUCKET` | + +In GitHub Actions, the full matrix job reads the `aws`, `azure`, `gcs`, and +key-based `azure-vended` variables from repository secrets. An unset secret +arrives empty, so a backend whose gate variable is unset stays `PENDING`; a +backend whose gate variable is set but whose other variables are not still +runs and fails, for example the `aws` cells when `COLDFRONT_AWS_ACCESS_KEY` is +set and `COLDFRONT_AWS_SECRET_KEY` is not. The full matrix job does not +pass `COLDFRONT_AWS_DATA_ROLE_ARN` or `COLDFRONT_AWS_EXTERNAL_ID`, so +`aws-vended` always stays `PENDING` there. The separate `cloud-aws` and +`cloud-azure` jobs test `aws-vended` and token-based `azure-vended` on +PostgreSQL 18 only. Fork PRs (no secret access) run SeaweedFS-only. + +## Publishing the Base Image + +The app image that every cell uses starts from the published DuckDB base image. +Maintainers rebuild and publish the base image with the following process: + +- When the base image's inputs change, run the + [base-image workflow](https://github.com/pgEdge/ColdFront/blob/main/.github/workflows/base-image.yml) + with `gh workflow run base-image.yml --ref -f push=true`. Without + `push=true`, the workflow builds the image but does not publish it. +- CI builds the app image on the base tag that `ci/base-ref.sh ` + prints, in the form `pg-r`; the hash covers the base Dockerfile + and the files it copies in. When that tag is not published, CI stops before + the app build and names the command that publishes the tag. +- The floating `pg` tag moves to the new image only when the workflow + runs from `main` with `push=true` for both architectures. diff --git a/mkdocs.yml b/mkdocs.yml index 69ff8c8d..8765fc91 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -94,6 +94,7 @@ nav: - Using Vended Credentials: vended_credentials.md - Supported Column Types: supported_types.md - Compacting the Cold Tier: compaction.md + - Testing and Continuous Integration: testing_ci.md - Developer Resources: - Formal Model - ColdFront Decoupled-Mode Bakery (TLA+/PlusCal): formal/README.md