Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci-walkthrough.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ concurrency:

permissions:
contents: read
packages: read # pull the private coldfront-duckdb-base image the app builds FROM
packages: read # pull the coldfront-duckdb-base image the app builds FROM

jobs:
validate:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ on:

permissions:
contents: read
packages: read # pull the private coldfront-duckdb-base image the app builds FROM
packages: read # pull the coldfront-duckdb-base image the app builds FROM

jobs:
quick:
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,8 +119,8 @@ to. See

For compliance environments that cannot store an object-store credential,
`coldfront.set_storage_secret_vended()` runs with no credential in the
database: Lakekeeper mints short-lived per-table credentials at access
time. See [Vended credentials](docs/usage.md#vended-minted-credentials).
database: Lakekeeper issues short-lived per-table credentials at access
time. See [Vended credentials](docs/usage.md#vended-credentials).

## Documentation

Expand Down
6 changes: 3 additions & 3 deletions ci/journey.sh
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ storage_secret_sql() {
if [ "$BACKEND" = azure-vended ]; then
printf "SELECT coldfront.set_storage_secret_vended('azure');"
elif [ "$BACKEND" = vended ]; then
# Vended (minted) creds: no credential stored. Lakekeeper mints per-table
# Vended creds: no credential stored. Lakekeeper issues per-table
# STS creds and ensure_attached() uses ACCESS_DELEGATION_MODE VENDED_CREDENTIALS.
printf "SELECT coldfront.set_storage_secret_vended();"
elif [ "$BACKEND" = azure ]; then
Expand Down Expand Up @@ -114,7 +114,7 @@ storage_yaml() {
fi
}

# vended_creds — true when Lakekeeper mints per-table credentials and no DuckDB
# vended_creds: true when Lakekeeper issues per-table credentials and no DuckDB
# secret exists for the bucket. Reads that address the object store BY PATH
# (glob(), iceberg_metadata('<location>')) cannot authenticate in that mode:
# vending is scoped to a table resolved through the catalog, so a bare path has
Expand Down Expand Up @@ -4141,7 +4141,7 @@ story_iceberg_metadata() {
fi
# Addressed as the attached catalog table, never as a metadata.json path.
# duckdb-iceberg resolves a 3-part name through the catalog and calls
# PrepareIcebergScanFromEntry, which mints that table's secret, so this reads
# PrepareIcebergScanFromEntry, which creates that table's secret, so this reads
# identically on static and vended credentials and needs no backend branch.
# De-quoted because the archiver stores the reference quoted.
cnt=$(q "$HOST" "SELECT coldfront.ensure_attached(); SELECT r['n'] FROM duckdb.query('SELECT count(*) AS n FROM iceberg_metadata(''${ice_ref}'') WHERE file_path LIKE ''%.parquet''') AS t(r);" | tail -1)
Expand Down
2 changes: 1 addition & 1 deletion ci/topo/vanilla.sh
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ elif [ "$BACKEND" = aws ]; then
\"storage-credential\":{\"type\":\"s3\",\"credential-type\":\"access-key\",\"aws-access-key-id\":\"${COLDFRONT_AWS_ACCESS_KEY}\",\"aws-secret-access-key\":\"${COLDFRONT_AWS_SECRET_KEY}\"}
}"
elif [ "$BACKEND" = vended ]; then
# Vended: SeaweedFS mints per-table STS creds. sts-enabled + assume-role-arn
# Vended: SeaweedFS issues per-table STS creds. sts-enabled + assume-role-arn
# (the role/trust-policy in docker/seaweedfs-iam.json); sts-endpoint = the S3
# endpoint. The warehouse's OWN credential stays the long-term admin access-key
# (vending changes only the client plane). Lakekeeper delivers the temp creds in
Expand Down
4 changes: 2 additions & 2 deletions cmd/archiver/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -254,13 +254,13 @@ func coldSecretSQL(cfg *config.Config) string {
// staticCredsConfigured reports whether the config carries static object-store
// credentials to build a DuckDB secret from (an S3 access key or an Azure
// connection string). False means a vended deployment: the YAML carries no
// credentials and Lakekeeper mints them per table. (Config validation has
// credentials and Lakekeeper issues them per table. (Config validation has
// already rejected a partial s3.* config by the time we get here.)
func staticCredsConfigured(cfg *config.Config) bool {
return cfg.S3.AccessKey != "" || cfg.Azure.ConnectionString != ""
}

// storageSecretVended reports whether the cold store uses vended (minted)
// storageSecretVended reports whether the cold store uses vended
// credentials: coldfront.storage_secret.vended is true. A missing row reads as
// false (no cold store configured yet).
func storageSecretVended(ctx context.Context, conn *pgx.Conn) (bool, error) {
Expand Down
6 changes: 3 additions & 3 deletions config.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,11 @@ s3:
# # empty when using azure.
# connection_string: "DefaultEndpointsProtocol=https;AccountName=<account>;AccountKey=<key>;EndpointSuffix=core.windows.net"

# Vended (minted) credentials: OMIT both s3 and azure entirely. When the cold
# Vended credentials: OMIT both s3 and azure entirely. When the cold
# store is vended (SELECT coldfront.set_storage_secret_vended();), Lakekeeper
# mints short-lived per-table credentials at access time, so no credential lives
# issues short-lived per-table credentials at access time, so no credential lives
# in this file, the database, or a DuckDB secret. Keep only the iceberg section
# above. See docs/usage.md#vended-minted-credentials.
# above. See docs/usage.md#vended-credentials.

archiver:
# Tiered tables live in coldfront.partition_config (replicated across a mesh);
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ For deployments that must not store a credential at all,
`coldfront.storage_secret` row that holds no credential and materializes
no secret. The row's `vended` flag drives `coldfront._attach_delegation_mode()`,
so `ensure_attached()` attaches the catalog with
`ACCESS_DELEGATION_MODE VENDED_CREDENTIALS`: Lakekeeper mints short-lived
`ACCESS_DELEGATION_MODE VENDED_CREDENTIALS`: Lakekeeper issues short-lived
per-table credentials (S3 STS, or Azure SAS) that `duckdb-iceberg`
consumes directly. A static row attaches with `ACCESS_DELEGATION_MODE
NONE` (the persistent secret supplies the credential); the vended path
Expand Down
2 changes: 1 addition & 1 deletion docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ and this project adheres to

- `coldfront.drop_iceberg_table()` drops a decoupled or tiered table, with
purge or keep-files for the stored objects.
- Vended (minted) object-store credentials, so cold access can use
- Vended object-store credentials, so cold access can use
short-lived credentials issued by Lakekeeper instead of static keys.
- Cross-tier row relocation: an UPDATE that moves a row's partition key
across the cutoff now moves the row between tiers.
Expand Down
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,8 @@ to. See

For compliance environments that cannot store an object-store credential,
`coldfront.set_storage_secret_vended()` runs with no credential in the
database: Lakekeeper mints short-lived per-table credentials at access
time. See [Vended credentials](usage.md#vended-minted-credentials).
database: Lakekeeper issues short-lived per-table credentials at access
time. See [Vended credentials](usage.md#vended-credentials).

## Least-privilege application roles

Expand Down
8 changes: 4 additions & 4 deletions docs/object_store.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ Before you begin, gather the following:
>
> This applies to the warehouse's own credential. A deployment that
> must not store any object-store credential in the database can use
> vended credentials instead, where the warehouse mints short-lived
> vended credentials instead, where the warehouse issues short-lived
> per-table credentials at access time; see
> [usage.md](usage.md#vended-minted-credentials).
> [usage.md](usage.md#vended-credentials).

- **Permissions** - the key needs read/write/list on the bucket
(`GetObject` / `PutObject` / `DeleteObject` / `ListBucket`). Example
Expand Down Expand Up @@ -154,8 +154,8 @@ The `key-prefix` is an arbitrary path inside the bucket; `coldfront` is
just an example.

**Vended-credentials variant.** To run ColdFront with no stored
credential ([usage.md](usage.md#vended-minted-credentials)), the
warehouse mints per-table STS credentials instead of handing the client
credential ([usage.md](usage.md#vended-credentials)), the
warehouse issues per-table STS credentials instead of handing the client
a static key. Two things change from the warehouse above.

First, an IAM role scoped to the bucket. Lakekeeper assumes it per table
Expand Down
12 changes: 6 additions & 6 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -622,12 +622,12 @@ It writes the same `coldfront.storage_secret` row (replicated,
The Azure cold tier is subject to the soft-delete / change-feed
restriction in [Gotchas](#gotchas).

## Vended (minted) credentials
## Vended credentials

Vended credentials let a deployment run with no object-store credential
stored in the database, in a DuckDB secret file, or in an archiver
config; this suits compliance environments that forbid persisting
long-term keys. Lakekeeper mints a short-lived, per-table credential
long-term keys. Lakekeeper issues a short-lived, per-table credential
(an S3 STS access key, secret, and session token) at read and write
time, and ColdFront uses it directly. The long-term credential lives
only in the Lakekeeper warehouse.
Expand All @@ -645,7 +645,7 @@ reads the same row and skips its own credential setup, so a vended
deployment omits the `s3:`/`azure:` block from the archiver config
entirely. The compactor likewise needs no credential in its config.

Vended mode targets the two clouds that mint scoped credentials: AWS S3
Vended mode targets the two clouds that issue scoped credentials: AWS S3
(STS) and Azure ADLS Gen2 (SAS). The `set_storage_secret_vended()` call
above enables AWS S3; the same call with `'azure'` enables Azure through
the identical path:
Expand All @@ -654,7 +654,7 @@ the identical path:
SELECT coldfront.set_storage_secret_vended('azure');
```

Vending requires a Lakekeeper warehouse configured to mint credentials:
Vending requires a Lakekeeper warehouse configured to vend credentials:

- AWS S3: `flavor: aws` with `sts-enabled: true`, an `assume-role-arn`
for a bucket-scoped IAM role, and an `external-id` on the warehouse
Expand All @@ -667,10 +667,10 @@ Vending requires a Lakekeeper warehouse configured to mint credentials:
`azure-system-identity`.

Google Cloud Storage over the S3-interoperability endpoint has no STS to
mint against, so GCS stays on static HMAC credentials.
issue short-lived credentials, so GCS stays on static HMAC credentials.

The compactor runs fully under vended credentials: compaction, snapshot
expiry, and orphan-file reclaim all use the minted per-table credentials.
expiry, and orphan-file reclaim all use the vended per-table credentials.

Switching a running deployment between static and vended credentials
changes the attach mode, which is fixed per PostgreSQL backend at attach
Expand Down
5 changes: 5 additions & 0 deletions examples/walkthrough/guide.sh
Original file line number Diff line number Diff line change
Expand Up @@ -980,6 +980,9 @@ EOSQL
[[ "$a" =~ ^[Nn]$ ]] || teardown_tiered
fi
}
# demo_decoupled: create and exercise a lake-only table, then demonstrate
# adopting and releasing an external Iceberg table. The adoption fixture stays
# in the catalog; interactive users can remove the lake-only table on exit.
demo_decoupled() {
ensure_coldfront_setup # silent — ColdFront may not be installed yet if this demo ran first
header "Decoupled — a table whose data lives in the lake, not in Postgres"
Expand Down Expand Up @@ -1379,6 +1382,8 @@ demo_distributed() {
fi
}

# reset_demos: restore the single-node stack and attempt to remove each demo's
# PostgreSQL and Iceberg tables, including the external adoption fixture.
reset_demos() {
# Restore the single-node stack first (no-op if already there). Covers the mesh
# case AND a failed switch that left ACTIVE_STACK=none with a mesh still up, so
Expand Down
35 changes: 18 additions & 17 deletions extension/coldfront/coldfront--1.0.sql
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ CREATE TABLE IF NOT EXISTS coldfront.storage_secret (
url_style text NOT NULL DEFAULT 'path',
use_ssl boolean NOT NULL DEFAULT false,
connection_string text, -- azure CONFIG provider: AccountName/AccountKey (shared key)
vended boolean NOT NULL DEFAULT false, -- true ⇒ Lakekeeper mints per-table creds; this row stores none
vended boolean NOT NULL DEFAULT false, -- true ⇒ Lakekeeper issues per-table creds; this row stores none
CONSTRAINT ss_type_enum CHECK (storage_type IN ('s3','azure')),
CONSTRAINT ss_s3_creds CHECK (vended OR storage_type <> 's3'
OR (key_id IS NOT NULL AND secret IS NOT NULL)),
Expand Down Expand Up @@ -251,7 +251,7 @@ BEGIN
-- pg_duckdb when this ATTACH (TYPE ICEBERG, ...) fires, gated by
-- duckdb.autoinstall_known_extensions / autoload_known_extensions. No
-- explicit install or per-session LOAD needed. ACCESS_DELEGATION_MODE is
-- VENDED_CREDENTIALS for a vended cold store (Lakekeeper mints per-table
-- VENDED_CREDENTIALS for a vended cold store (Lakekeeper issues per-table
-- creds), else NONE (the persistent secret supplies them).
PERFORM duckdb.raw_query(format(
'ATTACH IF NOT EXISTS %L AS ice (TYPE ICEBERG, ENDPOINT %L, '
Expand Down Expand Up @@ -468,7 +468,7 @@ RETURNS text
LANGUAGE plpgsql IMMUTABLE AS $$
DECLARE opts text;
BEGIN
-- Vended rows carry no credentials: Lakekeeper mints per-table creds and
-- Vended rows carry no credentials: Lakekeeper issues per-table creds and
-- duckdb-iceberg creates the DuckDB secret from them, so there is nothing to
-- materialize here.
IF r.vended THEN
Expand Down Expand Up @@ -624,7 +624,7 @@ $$;
-- must not store object-store credentials. It writes a credential-less row
-- (vended = true) so nothing is materialized as a DuckDB secret; instead
-- ensure_attached() turns on Iceberg REST credential vending and Lakekeeper
-- mints short-lived per-table credentials at read/write time. The Lakekeeper
-- issues short-lived per-table credentials at read/write time. The Lakekeeper
-- warehouse keeps its own long-term credential (the client plane holds none).
-- p_storage_type selects s3 (default; AWS + S3-compatible) or azure (so
-- ensure_attached still LOADs the azure extension for the vended SAS). Same
Expand All @@ -645,7 +645,7 @@ $$;

-- _attach_delegation_mode(): the DuckDB ICEBERG ATTACH access-delegation mode
-- for this node. VENDED_CREDENTIALS when the stored row is vended (Lakekeeper
-- mints per-table creds), else NONE (the persistent secret supplies them).
-- issues per-table creds), else NONE (the persistent secret supplies them).
-- Absent row ⇒ NONE (no cold store configured yet).
CREATE OR REPLACE FUNCTION coldfront._attach_delegation_mode() RETURNS text
LANGUAGE sql STABLE AS $$
Expand Down Expand Up @@ -2818,8 +2818,9 @@ BEGIN
END;
$$;

-- Everything adoption refuses before it reads a catalog or writes a row, given
-- the name the wrapper view would take and the Iceberg reference behind it.
-- Validate the proposed wrapper name and Iceberg reference before describing
-- the Iceberg table or creating the view and registry row. Refuses a missing PG
-- schema, an occupied relation or Iceberg reference, or an unconfigured catalog.
CREATE OR REPLACE FUNCTION coldfront._adopt_preflight(
p_schema text,
p_relname text,
Expand Down Expand Up @@ -2909,11 +2910,9 @@ BEGIN
END;
$$;

-- coldfront.adopt_iceberg_table: bring a table that already exists in the Iceberg
-- catalog under a PG wrapper view, so it reads and writes like one
-- create_iceberg_table provisioned. Nothing is created on the catalog and no data
-- is moved: the table stays exactly where it is, and releasing it later leaves it
-- untouched.
-- coldfront.adopt_iceberg_table: bring an existing Iceberg table under a PG
-- wrapper view for reads, optionally enabling writes. Nothing is created in the
-- catalog and no data is moved: releasing it later leaves the table untouched.
--
-- p_schema PG schema to put the wrapper view in.
-- p_table the view's name, and the Iceberg table's name.
Expand Down Expand Up @@ -3429,10 +3428,10 @@ BEGIN
END;
$$;

-- Peer-side trigger: fires when spock applies an originator's claim
-- INSERT into this node's local coldfront.claims (REPLICA-only — does
-- NOT fire on the originator's own local INSERT). Runs Ricart-Agrawala's
-- defer rule:
-- Peer-side trigger: fires when spock applies an originator's claim INSERT or
-- poke UPDATE to this node's coldfront.claims (REPLICA-only, not on the
-- originator). Reaps orphaned local claims before applying Ricart-Agrawala's
-- defer rule; an UPDATE without a reap keeps the earlier ack/defer decision:
-- * If this node has its own pending claim with SMALLER ticket on the
-- same iceberg_table → DEFER (queue in coldfront.deferred_acks).
-- * Otherwise → ack immediately (INSERT into coldfront.claim_acks,
Expand Down Expand Up @@ -4085,6 +4084,8 @@ $$;
--
-- Both end states are announced with a NOTICE, because one call yielding two
-- outcomes is worth stating out loud rather than leaving to documentation.
-- An adopted read-only relation cannot be dropped; release it instead to leave
-- its Iceberg table intact.
--
-- p_purge has no default because both choices are irreversible in opposite
-- directions: true deletes the data and metadata objects, which for the cold
Expand Down Expand Up @@ -4679,7 +4680,7 @@ $$;
COMMENT ON SCHEMA coldfront IS 'pgEdge ColdFront: transparent PostgreSQL to Apache Iceberg tiering, plus decoupled iceberg-only tables.';
COMMENT ON TABLE coldfront.tiered_views IS 'Registry (keyed by schema, relname) of views the coldfront DML hook handles — tiered (hot+cold) and decoupled (iceberg-only).';
COMMENT ON TABLE coldfront.archive_watermark IS 'Per-tiered-table (schema, table) hot/cold cutoff: ts >= cutoff is hot (PG), ts < cutoff is cold (Iceberg).';
COMMENT ON TABLE coldfront.storage_secret IS 'Cold-store credential; materialized as a DuckDB PERSISTENT SECRET, replicated by value across a Spock mesh, excluded from pg_dump. A vended row stores no credential and materializes nothing: Lakekeeper mints per-table creds and ensure_attached uses ACCESS_DELEGATION_MODE VENDED_CREDENTIALS.';
COMMENT ON TABLE coldfront.storage_secret IS 'Cold-store credential; materialized as a DuckDB PERSISTENT SECRET, replicated by value across a Spock mesh, excluded from pg_dump. A vended row stores no credential and materializes nothing: Lakekeeper issues per-table creds and ensure_attached uses ACCESS_DELEGATION_MODE VENDED_CREDENTIALS.';
COMMENT ON TABLE coldfront.partition_config IS 'Name-keyed per-table partition/tiering lifecycle config (period, hot_period, retention); replicates by value so every mesh node reads identical config.';
COMMENT ON TABLE coldfront.claims IS 'Ricart-Agrawala bakery: a writer''s outstanding iceberg-commit claim (iceberg_table, snowflake ticket); deleted on release.';
COMMENT ON TABLE coldfront.claim_acks IS 'Ricart-Agrawala bakery: per-peer acknowledgements of a claim, replicated back to the originating writer.';
Expand Down
5 changes: 4 additions & 1 deletion extension/coldfront/src/coldfront.c
Original file line number Diff line number Diff line change
Expand Up @@ -3604,7 +3604,8 @@ cf_own_duckdb_temp_dir(void)

/*
* The coldfront post-parse-analyze hook: rewrite INSERT/UPDATE/DELETE on a
* registered tiered view to the hot/cold/dual emit path.
* writable registered view to the hot/cold/dual emit path. Reject DML on an
* adopted read-only view before it reaches Iceberg.
*
* The hook is registered cluster-wide via shared_preload_libraries, so it
* also fires in databases/sessions where CREATE EXTENSION coldfront was
Expand Down Expand Up @@ -3895,6 +3896,8 @@ coldfront_loopback(PG_FUNCTION_ARGS)
PG_RETURN_TEXT_P(cstring_to_text(val));
}

/* Queue a bakery ticket for release after the outer transaction commits or
* aborts; the transaction callback drains it rather than releasing it here. */
PG_FUNCTION_INFO_V1(coldfront_enqueue_release);
Datum
coldfront_enqueue_release(PG_FUNCTION_ARGS)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
-- White-box test for vended (minted) credential mode. Vended rows carry NO
-- White-box test for vended credential mode. Vended rows carry NO
-- object-store credentials: _build_storage_secret_opts returns NULL (nothing is
-- materialized as a DuckDB secret), the CHECK guards allow the creds-NULL row,
-- and _attach_delegation_mode() reports VENDED_CREDENTIALS so ensure_attached
Expand Down
2 changes: 1 addition & 1 deletion extension/coldfront/test/sql/storage_secret_vended.sql
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
-- White-box test for vended (minted) credential mode. Vended rows carry NO
-- White-box test for vended credential mode. Vended rows carry NO
-- object-store credentials: _build_storage_secret_opts returns NULL (nothing is
-- materialized as a DuckDB secret), the CHECK guards allow the creds-NULL row,
-- and _attach_delegation_mode() reports VENDED_CREDENTIALS so ensure_attached
Expand Down
Loading
Loading