From a294a97f91ee97efb9ee1735d2b1161ab62f2ac9 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 14 Jul 2026 15:24:54 +0300 Subject: [PATCH 1/4] docs(unified-tls-pki): fix the canonical trust-anchor name and its owner Implementation surfaced two errors in the accepted design. The canonical projection cannot be named -ca-cert. Percona PSMDB creates a Secret of exactly that name and puts a private key in it, so source and target collide on the very engine that most needs the projection; Strimzi publishes its key-free CA under a different name again. The document itself warned three times that this name means opposite things on different engines, used that to reject name-based source selection, and then picked it for the target anyway. The canonical object is now -tenant-ca, which no shipped operator claims. The projection also cannot be owner-referenced to the application CR. The apps.cozystack.io kinds are virtual: an application is stored as a HelmRelease, so there is no object in etcd to reference. The HelmRelease is the only real owner, and deleting the application deletes it, which garbage-collects the projection. Assisted-By: Claude Signed-off-by: Aleksei Sviridkin --- design-proposals/external-database-exposure/README.md | 4 ++-- design-proposals/unified-tls-pki/README.md | 10 +++++----- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md index 742352d..3e33f6e 100644 --- a/design-proposals/external-database-exposure/README.md +++ b/design-proposals/external-database-exposure/README.md @@ -16,7 +16,7 @@ This is the design-proposal artifact required by `cozystack/cozystack#2816`, and ## Scope and related proposals -- **Depends on:** `design-proposals/unified-tls-pki` — provides the `-ca-cert` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract, and the path differs by engine — redis self-publishes a key-free `-ca-cert` through its forked operator, while postgres and mongodb obtain theirs through the extraction controller in `unified-tls-pki`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. +- **Depends on:** `design-proposals/unified-tls-pki` — provides the `-tenant-ca` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract, and the path differs by engine — redis self-publishes a key-free `-ca-cert` through its forked operator, while postgres and mongodb obtain theirs through the extraction controller in `unified-tls-pki`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. - **Related:** `design-proposals/structured-external-exposure` (community pull request #29) — replaces the chart-level `external` boolean with a structured, additive `expose` list riding `ExposureClass` / `ServiceExposure` (`cozystack/cozystack#3081`), and lists Gateway/SNI consolidation as the forward-compatible future path. The two proposals meet at the trigger surface: once `expose` lands, "expose this database via SNI-passthrough" is naturally one `expose` entry (an exposure class or scope backed by the tenant Gateway) rather than a new chart toggle, and the per-release route rendering naturally belongs to that orchestration layer. This proposal defines the Gateway-side mechanics either trigger drives; it does not depend on `expose` landing first. - **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, end-to-end TLS). - **Referenced, not designed here:** WS6 east-west / in-cluster CNI encryption (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`). It is complementary defense-in-depth for pod-to-pod traffic and is explicitly out of scope (see Non-goals). @@ -136,7 +136,7 @@ One subtlety the "single certificate" framing hides: an external client doing `s WS5 adds nothing beyond two hooks that already exist: -1. **Trust-anchor delivery** — the `-ca-cert` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. +1. **Trust-anchor delivery** — the `-tenant-ca` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. 2. **SAN coverage** — the chart already injects the external hostname into the operator-issued certificate (postgres `serverAltDNSNames` in `db.yaml`, gated on TLS and external). Under this proposal the hook injects **both** name shapes — `..` and `.` — so the certificate is valid for the subdomain scheme shipping now and the flat scheme arriving with Cilium 1.20 (§6). Stated plainly, where this does **not** work: diff --git a/design-proposals/unified-tls-pki/README.md b/design-proposals/unified-tls-pki/README.md index 5203902..4f8e1f5 100644 --- a/design-proposals/unified-tls-pki/README.md +++ b/design-proposals/unified-tls-pki/README.md @@ -20,7 +20,7 @@ This proposal is the umbrella design for the work tracked by epic `cozystack/coz - **Edge, open:** `cozystack/cozystack#2990` (propagate the operator wildcard to per-tenant termination points — the PR implementing issue `cozystack/cozystack#2820`). - **Workstreams (issues):** `cozystack/cozystack#2812` and `cozystack/cozystack#2400` (closed, edge wildcard); `cozystack/cozystack#2814` (converge per-app TLS and close the trust-anchor delivery gap — the first consumer of this contract); `cozystack/cozystack#2815` (external DB exposure via Gateway TLS-passthrough); `cozystack/cozystack#2816` (end-to-end TLS for databases); `cozystack/cozystack#2977` (opt-in east-west encryption). Throughout this document a `cozystack/cozystack#NNNN` reference is an issue unless called out as a PR. - **Per-app TLS series (open):** `cozystack/cozystack#2729` (redis), `cozystack/cozystack#2692` (mongodb), `cozystack/cozystack#2683` (rabbitmq), `cozystack/cozystack#2682` (opensearch), `cozystack/cozystack#2680` (mariadb). These are the pull requests that should land *after* this contract is accepted, not before; several currently propose handing tenants a key-bearing Secret, which this contract exists to correct (see Security). -- **Related (consumers of this contract):** `design-proposals/external-database-exposure` — the sibling proposal under the same epic; its end-to-end TLS story has external clients validate against the `-ca-cert` object defined here. `design-proposals/structured-external-exposure` (community pull request #29) — the structured `expose` model whose "external implies TLS" rule likewise points clients at this trust anchor. Both consume this proposal's output and define no PKI of their own. +- **Related (consumers of this contract):** `design-proposals/external-database-exposure` — the sibling proposal under the same epic; its end-to-end TLS story has external clients validate against the `-tenant-ca` object defined here. `design-proposals/structured-external-exposure` (community pull request #29) — the structured `expose` model whose "external implies TLS" rule likewise points clients at this trust anchor. Both consume this proposal's output and define no PKI of their own. All repository paths below refer to the `cozystack/cozystack` repository; paths attributed to an open PR (for example the wildcard-secret reconciler in PR `cozystack/cozystack#2990`) are not yet on `main`. @@ -134,7 +134,7 @@ The takeaway: only **kafka** (native) and **redis** (forked operator) are at the ### 4. The uniform consume contract -Every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object: a Secret named `-ca-cert`, containing only `ca.crt`, stamped with `internal.cozystack.io/tenantresource: "true"`. The platform already has the building block — `cozy-lib.tls.caCertSecret` renders exactly this object and fails closed if the input contains a private key. +Every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object: a Secret named `-tenant-ca`, containing only `ca.crt`, stamped with `internal.cozystack.io/tenantresource: "true"`. The name matters, and an earlier revision of this document got it wrong: `-ca-cert` cannot be the canonical name, because Percona PSMDB already creates a Secret of exactly that name and puts a private key in it — source and target would collide on the one engine that most needs the projection — while Strimzi publishes its key-free CA under `-clients-ca-cert` instead. `-tenant-ca` is claimed by no operator this platform ships, and it is symmetric with the `internal.cozystack.io/tenant-ca` marker the controller stamps on every projection. The platform already has the building block — `cozy-lib.tls.caCertSecret` renders exactly this object and fails closed if the input contains a private key. This is where the label-filtered projection matters. Because `tenantsecrets` delivers the whole Secret `Data`, the helper's fail-closed guard is not a nicety — it is the boundary that keeps a server or CA private key out of a tenant's hands. Kafka's `-clients-ca-cert` is the shape to match: a CA certificate, no key, readable by the tenant. @@ -151,16 +151,16 @@ The controller is deliberately **engine-agnostic**: it does not branch on cert-m **(a) An explicit source-selection label, engine-agnostic, not a name convention.** The owner of the CA-bearing Secret stamps `internal.cozystack.io/publish-ca-cert: "true"`, with an optional `internal.cozystack.io/publish-ca-cert-key` annotation naming the key to lift (default `ca.crt`). For the cert-manager-minting charts this rides `Certificate.spec.secretTemplate.labels`, so even the asynchronously-created CA Secret carries the marker from the moment cert-manager writes it. The label, not the name, is the contract — which matters because the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and the name `-ca-cert` is overloaded: key-free for the redis fork, key-bearing for PSMDB. A name convention would mis-handle one of them; the label plus a content check does not. -**(b) A small extraction controller.** Its watch/upsert skeleton can follow the wildcard-secret reconciler from PR `cozystack/cozystack#2990` (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — lineage provides those (part (c)) — and, being **intra-namespace**, it can do something that reconciler cannot. For each label-selected source Secret it upserts a `type: Opaque` Secret named `-ca-cert` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. It does four security-load-bearing things and nothing more: +**(b) A small extraction controller.** Its watch/upsert skeleton can follow the wildcard-secret reconciler from PR `cozystack/cozystack#2990` (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — lineage provides those (part (c)) — and, being **intra-namespace**, it can do something that reconciler cannot. For each selected source Secret it upserts a `type: Opaque` Secret named `-tenant-ca` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. It does four security-load-bearing things and nothing more: - **Key off the label and the content, never the name.** Select sources by the `publish-ca-cert` label; before writing, verify the lifted value is a CA certificate and carries no `-----BEGIN … PRIVATE KEY-----` header. This is what lets one controller serve a key-free redis source it should ignore and a key-bearing PSMDB source it must strip, both named `-ca-cert`-adjacent, without confusion. - **Tolerate operator-created, asynchronous sources.** The source may be created by the CNPG or PSMDB operator after the chart renders (not just by cert-manager). The controller waits on the watch event for the labeled source; it does not assume the chart owns the Secret and does not error or busy-loop before the source exists. - **Sanitize at write time, not just render time.** The `cozy-lib.tls.caCertSecret` helper's fail-closed guard runs at *chart-render* time; this controller writes at *runtime*, so it must itself copy only the single `ca.crt` key (an explicit whitelist) and re-assert the no-private-key check on every write. It never copies the whole `Data`. -- **Owner-ref the projected Secret to the application instance CR**, resolved from the `app.kubernetes.io/instance` label already on the source Secret. This is where intra-namespace beats the `cozystack/cozystack#2990` reconciler, which tracks its replicas by a management label plus a back-reference annotation *because* a cross-namespace replica cannot carry a valid `ownerReference`. Our projection is same-namespace, so a real owner-reference is valid: Kubernetes garbage-collects the `-ca-cert` on app deletion for free, and the controller needs no prune logic of its own. (It does not owner-ref the *source* `-ca`: cert-manager and the DB operators do not own their output Secrets, so a walk from the source would dead-end before the app.) +- **Owner-ref the projected Secret to the release's `HelmRelease`.** An earlier revision of this document said to owner-ref the application instance CR; that is not implementable. The `apps.cozystack.io` kinds are virtual — served by the aggregated apiserver, and a tenant application is *stored* as a `HelmRelease` whose `spec.values` is the application spec verbatim — so there is no application object in etcd and therefore no UID to reference. The `HelmRelease` is the only real object in the chain, and it is enough: deleting the application through the aggregated API deletes the `HelmRelease`, which garbage-collects the projection with it. This is where intra-namespace beats the `cozystack/cozystack#2990` reconciler, which tracks its replicas by a management label plus a back-reference annotation *because* a cross-namespace replica cannot carry a valid `ownerReference`. Our projection is same-namespace, so a real owner-reference is valid and the controller needs no prune logic of its own. `blockOwnerDeletion` stays false: a trust anchor must never delay the teardown of the application it belongs to. (It does not owner-ref the *source* `-ca`: cert-manager and the DB operators do not own their output Secrets, so a walk from the source would dead-end before the app.) **(c) Marking stays in `spec.secrets`, by label not by name.** Because `ApplicationDefinition.spec.secrets.include` accepts a label selector (`internal/lineagecontrollerwebhook/matcher.go`), one generic entry — `matchLabels: {internal.cozystack.io/tenant-ca: "true"}`, stamped by the controller on every projected Secret — covers every engine with no per-release `resourceName` templating. The lineage admission webhook then does the rest: on admission it walks the projected Secret's `ownerReferences` to the owning application and authoritatively stamps `internal.cozystack.io/tenantresource` to `true` (and to `false` should the Secret ever stop matching). -What this *reuses* rather than rebuilds: the label selector in `ApplicationDefinition.spec.secrets` (`internal/lineagecontrollerwebhook/matcher.go`), the lineage webhook's owner-reference walk and authoritative `tenantresource` stamping (`internal/lineagecontrollerwebhook/webhook.go`, `pkg/lineage/lineage.go`), the private-key guard in `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`), and native Kubernetes garbage collection via the owner reference. The irreducibly new work is the extraction step itself — read one key, write a key-free copy, re-copy on rotation — the same job an operator does natively on the engines that already ship a key-free CA object (kafka's `-clients-ca-cert`, the redis fork's `-ca-cert`). +What this *reuses* rather than rebuilds: the label selector in `ApplicationDefinition.spec.secrets` (`internal/lineagecontrollerwebhook/matcher.go`), the lineage webhook's owner-reference walk and authoritative `tenantresource` stamping (`internal/lineagecontrollerwebhook/webhook.go`, `pkg/lineage/lineage.go`), the private-key guard in `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`), and native Kubernetes garbage collection via the owner reference on the `HelmRelease`. The irreducibly new work is the extraction step itself — read one key, write a key-free copy, re-copy on rotation — the same job an operator does natively on the engines that already ship a key-free CA object (kafka's `-clients-ca-cert`, the redis fork's `-ca-cert`) — those two still need the projection, because their CA lands under an engine-specific name rather than the canonical one; for them the extraction step is a straight copy, with nothing to strip. ### 6. Per-engine application order From 5cbcdcbabf81f4b39d6f6aa998862f50d09c5e18 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 14 Jul 2026 19:24:45 +0300 Subject: [PATCH 2/4] docs(unified-tls-pki): document the required release label on CA sources Implementation showed the source-selection label is not enough on its own. The platform runs cert-manager with enableCertificateOwnerRef set to false, so a cert-manager-issued Secret carries no ownerReference back to its application and the lineage graph cannot tell which release it belongs to. The consume contract therefore requires a second label, internal.cozystack.io/publish-ca-cert-release, carrying the release name, stamped alongside the opt-in label in the same secretTemplate block. Assisted-By: Claude Signed-off-by: Aleksei Sviridkin --- design-proposals/unified-tls-pki/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design-proposals/unified-tls-pki/README.md b/design-proposals/unified-tls-pki/README.md index 4f8e1f5..0a7e1a7 100644 --- a/design-proposals/unified-tls-pki/README.md +++ b/design-proposals/unified-tls-pki/README.md @@ -149,7 +149,7 @@ The alternative to one controller is to fork each upstream operator the way redi The controller is deliberately **engine-agnostic**: it does not branch on cert-manager-vs-operator-owned, and it does not key off Secret names. Its contract is three pieces, only the middle of which is new code. -**(a) An explicit source-selection label, engine-agnostic, not a name convention.** The owner of the CA-bearing Secret stamps `internal.cozystack.io/publish-ca-cert: "true"`, with an optional `internal.cozystack.io/publish-ca-cert-key` annotation naming the key to lift (default `ca.crt`). For the cert-manager-minting charts this rides `Certificate.spec.secretTemplate.labels`, so even the asynchronously-created CA Secret carries the marker from the moment cert-manager writes it. The label, not the name, is the contract — which matters because the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and the name `-ca-cert` is overloaded: key-free for the redis fork, key-bearing for PSMDB. A name convention would mis-handle one of them; the label plus a content check does not. +**(a) An explicit source-selection label, engine-agnostic, not a name convention.** The owner of the CA-bearing Secret stamps `internal.cozystack.io/publish-ca-cert: "true"`, with an optional `internal.cozystack.io/publish-ca-cert-key` annotation naming the key to lift (default `ca.crt`). It also stamps `internal.cozystack.io/publish-ca-cert-release` with the release name: implementation showed the release cannot be inferred on the label leg, because the platform runs cert-manager with `enableCertificateOwnerRef: false`, so a cert-manager-issued Secret carries no ownerReference back to its owning application and the lineage graph cannot resolve which release it belongs to. The release label supplies that link explicitly. For the cert-manager-minting charts all three markers ride `Certificate.spec.secretTemplate.labels`, so even the asynchronously-created CA Secret carries them from the moment cert-manager writes it. The label, not the name, is the contract — which matters because the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and the name `-ca-cert` is overloaded: key-free for the redis fork, key-bearing for PSMDB. A name convention would mis-handle one of them; the label plus a content check does not. **(b) A small extraction controller.** Its watch/upsert skeleton can follow the wildcard-secret reconciler from PR `cozystack/cozystack#2990` (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — lineage provides those (part (c)) — and, being **intra-namespace**, it can do something that reconciler cannot. For each selected source Secret it upserts a `type: Opaque` Secret named `-tenant-ca` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. It does four security-load-bearing things and nothing more: @@ -208,7 +208,7 @@ Two consequences for the controller. First, it writes at runtime, after chart-re ## Open questions - **Labelling operator-created sources.** The cert-manager charts stamp the source label via `Certificate.spec.secretTemplate.labels`. Operators that create the CA Secret themselves (CNPG `-ca`, PSMDB `-ca-cert`) may not let the chart label their output. For those, the controller needs either an operator that supports output labels or a small per-engine source-name configuration as a fallback — this is the one place the engine-agnostic label is not yet sufficient, and it should be resolved before postgres/mongodb convergence. -- The exact namespace convention for the `-ca-cert` object (per-release in the app namespace is assumed here) and the final label/annotation names. This proposal uses `internal.cozystack.io/publish-ca-cert` on the source and `internal.cozystack.io/tenant-ca` on the projected copy, matching the `internal.cozystack.io/` convention of the existing `tenantresource` and `managed-by-cozystack` markers. +- The exact namespace convention for the `-ca-cert` object (per-release in the app namespace is assumed here) and the final label/annotation names. This proposal uses `internal.cozystack.io/publish-ca-cert` and `internal.cozystack.io/publish-ca-cert-release` on the source and `internal.cozystack.io/tenant-ca` on the projected copy, matching the `internal.cozystack.io/` convention of the existing `tenantresource` and `managed-by-cozystack` markers. - How this intersects with per-tenant wildcard propagation (issue `cozystack/cozystack#2820`, implemented by PR `cozystack/cozystack#2990`). That is a *cross-namespace* replication problem; the extraction controller here is *intra-namespace, per-release*, so the two do not share a mechanism — but they should share the management-label and foreign-collision conventions. Note that `clustersecret-operator` is packaged in the repo but is not wired into a default bundle, so it is not an installed primitive this design can assume. ## Alternatives considered From f8cc231320ca6dc25c0c0ec0a6667d94903b17ae Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 16 Jul 2026 21:12:08 +0300 Subject: [PATCH 3/4] docs(unified-tls-pki): make the canonical name collision-free by construction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The trust anchor was named -tenant-ca on the grounds that no operator the platform ships claims it. That is false. It collides across releases: for an application foo the projection is postgres-foo-tenant-ca, and for a sibling application foo-tenant CloudNativePG's own CA secret is postgres-foo-tenant plus -ca, which is the same string. Both names are legal and several Postgres instances in one namespace is ordinary. The collision has a direction no guard can refuse. When the projection is written first, CloudNativePG rejects the key-free secret with 'missing ca.key secret data' and the sibling's PKI never completes, so that application never starts, blaming a secret its owner never created. The controller that wrote first was within its rights, so there is nothing to refuse; and before this contract existed both applications worked. The name is now .tenant-ca, which is disjoint by character class rather than by survey: application names are DNS-1035 labels and cannot contain a dot, release prefixes are dot-free, and secret names are DNS-1123 subdomains where a dot is legal. No prefix, application name and operator suffix can therefore produce a dotted name, for any operator, now or later. This is the second time the survey argument failed — the previous name was rejected because Percona claims it. Recording the reasoning so the third name is not chosen the same way. Assisted-By: Claude Signed-off-by: Aleksei Sviridkin --- .../external-database-exposure/README.md | 6 ++-- design-proposals/unified-tls-pki/README.md | 36 ++++++++++--------- 2 files changed, 23 insertions(+), 19 deletions(-) diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md index 3e33f6e..8cc692c 100644 --- a/design-proposals/external-database-exposure/README.md +++ b/design-proposals/external-database-exposure/README.md @@ -16,7 +16,7 @@ This is the design-proposal artifact required by `cozystack/cozystack#2816`, and ## Scope and related proposals -- **Depends on:** `design-proposals/unified-tls-pki` — provides the `-tenant-ca` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract, and the path differs by engine — redis self-publishes a key-free `-ca-cert` through its forked operator, while postgres and mongodb obtain theirs through the extraction controller in `unified-tls-pki`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. +- **Depends on:** `design-proposals/unified-tls-pki` — provides the `.tenant-ca` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract. The engines reach that point by one path: the extraction controller reads each engine's CA source and writes the canonical `.tenant-ca` — copying verbatim from redis and kafka's own key-free CA object, and stripping the key from postgres and mongodb's key-bearing Secret — rather than any engine publishing that canonical name itself. Either way the external client verifies against the one canonical `.tenant-ca`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. - **Related:** `design-proposals/structured-external-exposure` (community pull request #29) — replaces the chart-level `external` boolean with a structured, additive `expose` list riding `ExposureClass` / `ServiceExposure` (`cozystack/cozystack#3081`), and lists Gateway/SNI consolidation as the forward-compatible future path. The two proposals meet at the trigger surface: once `expose` lands, "expose this database via SNI-passthrough" is naturally one `expose` entry (an exposure class or scope backed by the tenant Gateway) rather than a new chart toggle, and the per-release route rendering naturally belongs to that orchestration layer. This proposal defines the Gateway-side mechanics either trigger drives; it does not depend on `expose` landing first. - **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, end-to-end TLS). - **Referenced, not designed here:** WS6 east-west / in-cluster CNI encryption (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`). It is complementary defense-in-depth for pod-to-pod traffic and is explicitly out of scope (see Non-goals). @@ -107,7 +107,7 @@ flowchart TB P2["postgres db2
presents CNPG server cert"] P3["redis cache
presents operator server cert"] end - CA["<release>-ca-cert (ca.crt only)
projected via tenantsecrets"] + CA["<release>.tenant-ca (ca.crt only)
projected via tenantsecrets"] C1 -->|"raw TLS, SNI"| L1 C2 -->|"raw TLS, SNI"| L1 C3 -->|"raw TLS, SNI"| L2 @@ -136,7 +136,7 @@ One subtlety the "single certificate" framing hides: an external client doing `s WS5 adds nothing beyond two hooks that already exist: -1. **Trust-anchor delivery** — the `-tenant-ca` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. +1. **Trust-anchor delivery** — the `.tenant-ca` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. 2. **SAN coverage** — the chart already injects the external hostname into the operator-issued certificate (postgres `serverAltDNSNames` in `db.yaml`, gated on TLS and external). Under this proposal the hook injects **both** name shapes — `..` and `.` — so the certificate is valid for the subdomain scheme shipping now and the flat scheme arriving with Cilium 1.20 (§6). Stated plainly, where this does **not** work: diff --git a/design-proposals/unified-tls-pki/README.md b/design-proposals/unified-tls-pki/README.md index 0a7e1a7..3e14f50 100644 --- a/design-proposals/unified-tls-pki/README.md +++ b/design-proposals/unified-tls-pki/README.md @@ -20,7 +20,7 @@ This proposal is the umbrella design for the work tracked by epic `cozystack/coz - **Edge, open:** `cozystack/cozystack#2990` (propagate the operator wildcard to per-tenant termination points — the PR implementing issue `cozystack/cozystack#2820`). - **Workstreams (issues):** `cozystack/cozystack#2812` and `cozystack/cozystack#2400` (closed, edge wildcard); `cozystack/cozystack#2814` (converge per-app TLS and close the trust-anchor delivery gap — the first consumer of this contract); `cozystack/cozystack#2815` (external DB exposure via Gateway TLS-passthrough); `cozystack/cozystack#2816` (end-to-end TLS for databases); `cozystack/cozystack#2977` (opt-in east-west encryption). Throughout this document a `cozystack/cozystack#NNNN` reference is an issue unless called out as a PR. - **Per-app TLS series (open):** `cozystack/cozystack#2729` (redis), `cozystack/cozystack#2692` (mongodb), `cozystack/cozystack#2683` (rabbitmq), `cozystack/cozystack#2682` (opensearch), `cozystack/cozystack#2680` (mariadb). These are the pull requests that should land *after* this contract is accepted, not before; several currently propose handing tenants a key-bearing Secret, which this contract exists to correct (see Security). -- **Related (consumers of this contract):** `design-proposals/external-database-exposure` — the sibling proposal under the same epic; its end-to-end TLS story has external clients validate against the `-tenant-ca` object defined here. `design-proposals/structured-external-exposure` (community pull request #29) — the structured `expose` model whose "external implies TLS" rule likewise points clients at this trust anchor. Both consume this proposal's output and define no PKI of their own. +- **Related (consumers of this contract):** `design-proposals/external-database-exposure` — the sibling proposal under the same epic; its end-to-end TLS story has external clients validate against the `.tenant-ca` object defined here. `design-proposals/structured-external-exposure` (community pull request #29) — the structured `expose` model whose "external implies TLS" rule likewise points clients at this trust anchor. Both consume this proposal's output and define no PKI of their own. All repository paths below refer to the `cozystack/cozystack` repository; paths attributed to an open PR (for example the wildcard-secret reconciler in PR `cozystack/cozystack#2990`) are not yet on `main`. @@ -96,7 +96,7 @@ flowchart TB end subgraph interior["Interior tier — per-engine operator-owned PKI"] OP["DB operator owns CA + server cert
(CNPG / Strimzi / PSMDB / cert-manager chart graph)"] - CACERT["<release>-ca-cert
(ca.crt only, no private key)"] + CACERT["<release>.tenant-ca
(ca.crt only, no private key)"] OP --> CACERT end subgraph tenant["Tenant"] @@ -119,7 +119,7 @@ The contract is a per-engine table. It is the artifact the per-app pull requests | Engine | PKI owner | CA-bearing Secret today | Key in that Secret? | Self-publishes key-free `ca.crt`? | `ca.crt` to tenant today? | Operator certificate capability | | --- | --- | --- | --- | --- | --- | --- | | kafka (Strimzi) | operator | `-cluster-ca-cert`, `-clients-ca-cert` | no | **yes** | yes — clean reference | `generateCertificateAuthority: false` BYO-CA + per-listener `brokerCertChainAndKey`; public cert on external listener, own-CA mTLS internally | -| redis (forked operator) | operator fork | `-ca-tls` (chart cert-manager) | yes | **yes** → operator emits `-ca-cert` | yes | forked `redis-operator` `spec.tls.caCertSecretName` outputs a key-free `ca.crt`-only Opaque Secret (v1.4.0+) | +| redis (open PR) | chart + cert-manager | `-ca-tls` (chart cert-manager) | yes | **not yet** — planned via a forked operator proposed in the open pull request, which would emit `-ca-cert` | no | stock `redis-operator` has no key-free certificate output today; the open pull request proposes forking it to add `spec.tls.caCertSecretName`, outputting a key-free `ca.crt`-only Opaque Secret | | postgres (CloudNativePG) | operator | `-ca` (operator-created, async) | **yes — `ca.key`** | no | no — tenant gets passwords-only `-credentials` (gap) | `serverCASecret` / `serverTLSSecret` BYO; single server cert, no edge/internal split | | mongodb (Percona PSMDB) | operator | `-ca-cert` (operator-created at runtime; the chart renders no TLS objects) | **yes — cert-manager `isCA`** | no | no (gap; name collides with redis, opposite shape) | `spec.tls.issuerConf` BYO issuer; auto-mints a cert-manager chain | | nats | chart + cert-manager | `-ca` | yes | no | no | — | @@ -130,11 +130,15 @@ The contract is a per-engine table. It is the artifact the per-app pull requests | clickhouse (Altinity) | — (BYO-cert) | supplied cert mount | n/a | no | no (not in TLS series yet) | mounts a supplied cert; no native issuance | | kubernetes (Kamaji) | Kamaji | Kamaji-owned, not swappable | n/a | n/a | no — out of model | control-plane CA, kubeconfig-pinned, non-swappable | -The takeaway: only **kafka** (native) and **redis** (forked operator) are at the target today — they self-publish a key-free `ca.crt`. **Every other engine has a key-bearing CA Secret and delivers no `ca.crt` to the tenant** — one uniform delivery gap. The gap does **not** split along mint-versus-consume: postgres and mongodb are operator-owned, yet their CA Secret is key-bearing (and operator-created, hence asynchronous), so they need the same key-free projection as the cert-manager-minting engines. The cert-manager-minting engines additionally never reach the tenant at all. Kamaji anchors the other end of the table: a PKI owner so absolute that the only sane unification is the one this contract chooses — leave the CA alone, unify how trust reaches the client. +The takeaway: only **kafka** (native) is at the target today — it self-publishes a key-free `ca.crt`; **redis** would join it once the forked operator proposed in its open per-app pull request lands and starts emitting `-ca-cert`. **Every other engine has a key-bearing CA Secret and delivers no `ca.crt` to the tenant** — one uniform delivery gap. The gap does **not** split along mint-versus-consume: postgres and mongodb are operator-owned, yet their CA Secret is key-bearing (and operator-created, hence asynchronous), so they need the same key-free projection as the cert-manager-minting engines. The cert-manager-minting engines additionally never reach the tenant at all. Kamaji anchors the other end of the table: a PKI owner so absolute that the only sane unification is the one this contract chooses — leave the CA alone, unify how trust reaches the client. ### 4. The uniform consume contract -Every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object: a Secret named `-tenant-ca`, containing only `ca.crt`, stamped with `internal.cozystack.io/tenantresource: "true"`. The name matters, and an earlier revision of this document got it wrong: `-ca-cert` cannot be the canonical name, because Percona PSMDB already creates a Secret of exactly that name and puts a private key in it — source and target would collide on the one engine that most needs the projection — while Strimzi publishes its key-free CA under `-clients-ca-cert` instead. `-tenant-ca` is claimed by no operator this platform ships, and it is symmetric with the `internal.cozystack.io/tenant-ca` marker the controller stamps on every projection. The platform already has the building block — `cozy-lib.tls.caCertSecret` renders exactly this object and fails closed if the input contains a private key. +Every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object: a Secret named `.tenant-ca`, containing only `ca.crt`, stamped with `internal.cozystack.io/tenant-ca: "true"` and made tenant-visible through the lineage webhook's `tenantresource` verdict. + +The name took three revisions, and the reason is worth recording because it is the same mistake twice. `-ca-cert` was rejected because Percona PSMDB already creates a Secret of that exact name and puts a private key in it — a collision across ENGINES. Its replacement `-tenant-ca` was justified as "claimed by no operator this platform ships", and that was false: it collides across RELEASES. For an application `foo` the projection is `postgres-foo-tenant-ca`; for a sibling application `foo-tenant`, CloudNativePG's own CA is `postgres-foo-tenant` + `-ca` — the same string. Both names are legal and several Postgres instances in one namespace is the normal model. Nor is it merely a guard problem: if the projection lands first, CNPG rejects the key-free Secret (`missing ca.key secret data`) and the sibling's PKI never completes, so the sibling application never comes up — silently, blaming a Secret its owner never created. That is a regression, since before this contract existed the name was unclaimed and both applications worked, and no guard can refuse it, because the controller that wrote first was legitimate. + +The canonical name is therefore `.tenant-ca`, and it is collision-free by CHARACTER CLASS rather than by survey. Application names are validated as DNS-1035 labels (`[a-z]([-a-z0-9]*[a-z0-9])?`), so a dot is structurally impossible in one; every release prefix is dot-free; and Secret names are DNS-1123 subdomains, where a dot is legal. So `` is dot-free by construction and can never equal a dotted name — for any application name, any operator, and any suffix an operator may add in future. That is a proof. "No operator claims it today" is a guess with an expiry date, and it expired twice. The platform already has the building block — `cozy-lib.tls.caCertSecret` renders exactly this object and fails closed if the input contains a private key. This is where the label-filtered projection matters. Because `tenantsecrets` delivers the whole Secret `Data`, the helper's fail-closed guard is not a nicety — it is the boundary that keeps a server or CA private key out of a tenant's hands. Kafka's `-clients-ca-cert` is the shape to match: a CA certificate, no key, readable by the tenant. @@ -142,7 +146,7 @@ This is where the label-filtered projection matters. Because `tenantsecrets` del The contract in §4 fixes the output object. The remaining work is the input path, and it splits on a single question — **does the engine's operator already emit a key-free `ca.crt` object?** -- **Engines that self-publish a key-free CA** need no platform machinery. `kafka` (Strimzi, native) and `redis` (forked operator via `caCertSecretName`) are here: they converge by matching the canonical name and stamping the tenant-resource label. They opt **out** of the controller below simply by not stamping the source label. +- **Engines that self-publish a key-free CA** need no platform machinery. `kafka` (Strimzi, native) is here today: it converges by matching the canonical name and stamping the tenant-resource label. `redis` would join it once its open per-app pull request lands a forked operator with `caCertSecretName`. Either engine opts **out** of the controller below simply by not stamping the source label. - **Every other engine** — nats, qdrant, mariadb, opensearch, rabbitmq (cert-manager chains, chart-rendered, key-bearing CA) and mongodb, postgres (operator-rendered, key-bearing CA, asynchronous) — has a key-bearing CA Secret and no key-free output. None can feed the helper at chart-render time (see "The problem"). These opt **in** via an explicit source label and are served by **one engine-agnostic extraction controller**. The alternative to one controller is to fork each upstream operator the way redis was forked (adding a `caCertSecretName`-style key-free output). Forking CloudNativePG, PSMDB, mariadb-operator, opster, and the rabbitmq cluster-operator is far more surface to own than a single small controller, and redis's fork was a reaction to a review blocker, not a designed-up-front pattern. So the controller is the target for every non-self-publishing engine. @@ -151,9 +155,9 @@ The controller is deliberately **engine-agnostic**: it does not branch on cert-m **(a) An explicit source-selection label, engine-agnostic, not a name convention.** The owner of the CA-bearing Secret stamps `internal.cozystack.io/publish-ca-cert: "true"`, with an optional `internal.cozystack.io/publish-ca-cert-key` annotation naming the key to lift (default `ca.crt`). It also stamps `internal.cozystack.io/publish-ca-cert-release` with the release name: implementation showed the release cannot be inferred on the label leg, because the platform runs cert-manager with `enableCertificateOwnerRef: false`, so a cert-manager-issued Secret carries no ownerReference back to its owning application and the lineage graph cannot resolve which release it belongs to. The release label supplies that link explicitly. For the cert-manager-minting charts all three markers ride `Certificate.spec.secretTemplate.labels`, so even the asynchronously-created CA Secret carries them from the moment cert-manager writes it. The label, not the name, is the contract — which matters because the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and the name `-ca-cert` is overloaded: key-free for the redis fork, key-bearing for PSMDB. A name convention would mis-handle one of them; the label plus a content check does not. -**(b) A small extraction controller.** Its watch/upsert skeleton can follow the wildcard-secret reconciler from PR `cozystack/cozystack#2990` (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — lineage provides those (part (c)) — and, being **intra-namespace**, it can do something that reconciler cannot. For each selected source Secret it upserts a `type: Opaque` Secret named `-tenant-ca` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. It does four security-load-bearing things and nothing more: +**(b) A small extraction controller.** Its watch/upsert skeleton can follow the wildcard-secret reconciler from PR `cozystack/cozystack#2990` (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — lineage provides those (part (c)) — and, being **intra-namespace**, it can do something that reconciler cannot. For each selected source Secret it upserts a `type: Opaque` Secret named `.tenant-ca` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. It does four security-load-bearing things and nothing more: -- **Key off the label and the content, never the name.** Select sources by the `publish-ca-cert` label; before writing, verify the lifted value is a CA certificate and carries no `-----BEGIN … PRIVATE KEY-----` header. This is what lets one controller serve a key-free redis source it should ignore and a key-bearing PSMDB source it must strip, both named `-ca-cert`-adjacent, without confusion. +- **Key off the label and the content, never the name.** Select sources by the `publish-ca-cert` label; before writing, decode the source in a `pem.Decode` loop, parse each block with `x509.ParseCertificate`, and rebuild the chain only from blocks that validate as certificates, rather than detecting a private key by header. This is what lets one controller serve a key-free redis source it should ignore and a key-bearing PSMDB source it must strip, both named `-ca-cert`-adjacent, without confusion. - **Tolerate operator-created, asynchronous sources.** The source may be created by the CNPG or PSMDB operator after the chart renders (not just by cert-manager). The controller waits on the watch event for the labeled source; it does not assume the chart owns the Secret and does not error or busy-loop before the source exists. - **Sanitize at write time, not just render time.** The `cozy-lib.tls.caCertSecret` helper's fail-closed guard runs at *chart-render* time; this controller writes at *runtime*, so it must itself copy only the single `ca.crt` key (an explicit whitelist) and re-assert the no-private-key check on every write. It never copies the whole `Data`. - **Owner-ref the projected Secret to the release's `HelmRelease`.** An earlier revision of this document said to owner-ref the application instance CR; that is not implementable. The `apps.cozystack.io` kinds are virtual — served by the aggregated apiserver, and a tenant application is *stored* as a `HelmRelease` whose `spec.values` is the application spec verbatim — so there is no application object in etcd and therefore no UID to reference. The `HelmRelease` is the only real object in the chain, and it is enough: deleting the application through the aggregated API deletes the `HelmRelease`, which garbage-collects the projection with it. This is where intra-namespace beats the `cozystack/cozystack#2990` reconciler, which tracks its replicas by a management label plus a back-reference annotation *because* a cross-namespace replica cannot carry a valid `ownerReference`. Our projection is same-namespace, so a real owner-reference is valid and the controller needs no prune logic of its own. `blockOwnerDeletion` stays false: a trust anchor must never delay the teardown of the application it belongs to. (It does not owner-ref the *source* `-ca`: cert-manager and the DB operators do not own their output Secrets, so a walk from the source would dead-end before the app.) @@ -164,15 +168,15 @@ What this *reuses* rather than rebuilds: the label selector in `ApplicationDefin ### 6. Per-engine application order -`postgres` goes first (under the tracking issue `cozystack/cozystack#2814`): it is the engine the epic most wants to see converged, and it exercises the controller against the hardest input — an **operator-created, asynchronous** CA Secret — so validating it there validates the mechanism everywhere. Convergence for postgres means publishing a key-free `-ca-cert` extracted from CNPG's key-bearing `-ca`, while leaving the passwords-only `-credentials` untouched. `kafka` is already at the target and needs only documentation. `redis` (`cozystack/cozystack#2729`) converges by shape adaptation: its forked operator self-publishes a key-free `-ca-cert` via `caCertSecretName`, so it does not stamp the source label and the controller leaves it alone. `mongodb`, then the cert-manager-minting engines (nats, qdrant, and then rabbitmq, mariadb, opensearch) adopt the controller via the source label. `kubernetes` (Kamaji) is explicitly out. +`postgres` goes first (under the tracking issue `cozystack/cozystack#2814`): it is the engine the epic most wants to see converged, and it exercises the controller against the hardest input — an **operator-created, asynchronous** CA Secret — so validating it there validates the mechanism everywhere. Convergence for postgres means publishing a key-free `.tenant-ca` extracted from CNPG's key-bearing `-ca`, while leaving the passwords-only `-credentials` untouched. `kafka` is already at the target and needs only documentation. `redis` (`cozystack/cozystack#2729`) converges by shape adaptation: its forked operator self-publishes a key-free `-ca-cert` via `caCertSecretName`, which the controller copies verbatim into the canonical `.tenant-ca` — a straight copy with nothing to strip, the same treatment kafka gets. `mongodb`, then the cert-manager-minting engines (nats, qdrant, and then rabbitmq, mariadb, opensearch) adopt the controller via the source label. `kubernetes` (Kamaji) is explicitly out. ## User-facing changes -A tenant sees one canonical, key-free trust-anchor object per managed application — `-ca-cert`, carrying only `ca.crt` — through the dashboard resource map and the `tenantsecrets` projection, in the same shape across every engine. An operator sees one interface to choose the edge certificate source. There is no new tenant-authored input. +A tenant sees one canonical, key-free trust-anchor object per managed application — `.tenant-ca`, carrying only `ca.crt` — through the dashboard resource map and the `tenantsecrets` projection, in the same shape across every engine. An operator sees one interface to choose the edge certificate source. There is no new tenant-authored input. ## Upgrade and rollback compatibility -This document changes nothing at runtime; it records a target. The pull requests that implement it are individually backward-compatible in the engine's own PKI: per-app TLS is opt-in (tri-state), external exposure is opt-in, and the extraction controller adds a new object without altering existing Secrets. postgres convergence is purely additive — it publishes a new key-free `-ca-cert` and leaves the passwords-only `-credentials` in place (removing it would drop the tenant's connection passwords). Reverting any implementing PR removes the new `-ca-cert` object and the controller that maintains it, leaving the engine's existing PKI untouched. +This document changes nothing at runtime; it records a target. The pull requests that implement it are individually backward-compatible in the engine's own PKI: per-app TLS is opt-in (tri-state), external exposure is opt-in, and the extraction controller adds a new object without altering existing Secrets. postgres convergence is purely additive — it publishes a new key-free `.tenant-ca` and leaves the passwords-only `-credentials` in place (removing it would drop the tenant's connection passwords). Reverting any implementing PR removes the new `.tenant-ca` object and the controller that maintains it, leaving the engine's existing PKI untouched. ## Security @@ -190,13 +194,13 @@ Two consequences for the controller. First, it writes at runtime, after chart-re - The per-release CA Secret does not exist yet (operator-created, asynchronous) → the controller waits for the watch event on the labeled source; it does not error or busy-loop. - A foreign Secret already occupies the target name → the controller leaves it untouched (management-label guard) and emits a Warning Event on the application, so the operator sees the collision. - The CA rotates → the controller re-copies `ca.crt` on the next source change; no chart re-render is required. -- The application is deleted → the projected `-ca-cert` is garbage-collected by Kubernetes via its owner reference to the application instance; the controller needs no delete path. +- The application is deleted → the projected `.tenant-ca` is garbage-collected by Kubernetes via its owner reference to the `HelmRelease` that backs the application; the controller needs no delete path. ## Testing - The helper is already covered by `packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml`, including the fail-closed assertions. - The extraction controller gets an envtest/Ginkgo suite (its skeleton mirrors `internal/controller/wildcardsecret/reconciler_test.go`): a labeled source appearing after the consumer (cert-manager and operator-created), a source whose `ca.crt` is swapped (rotation), a key-free source that must be left byte-for-byte (no spurious rewrite), a key-bearing source that must be stripped to `ca.crt`, a foreign-name collision (asserting the Secret is untouched and a Warning Event is emitted), a source value that smuggles a private-key header (asserting the controller refuses to write), and owner-reference-driven garbage collection on app deletion. -- Each per-app pull request adds helm-unittest fixtures asserting the `-ca-cert` shape and label, plus an end-to-end check under `hack/e2e-apps/` that a tenant can read `ca.crt`, cannot read any object carrying `tls.key`, and can verify the server. +- Each per-app pull request adds helm-unittest fixtures asserting the `.tenant-ca` shape and label, plus an end-to-end check under `hack/e2e-apps/` that a tenant can read `ca.crt`, cannot read any object carrying `tls.key`, and can verify the server. ## Rollout @@ -208,7 +212,7 @@ Two consequences for the controller. First, it writes at runtime, after chart-re ## Open questions - **Labelling operator-created sources.** The cert-manager charts stamp the source label via `Certificate.spec.secretTemplate.labels`. Operators that create the CA Secret themselves (CNPG `-ca`, PSMDB `-ca-cert`) may not let the chart label their output. For those, the controller needs either an operator that supports output labels or a small per-engine source-name configuration as a fallback — this is the one place the engine-agnostic label is not yet sufficient, and it should be resolved before postgres/mongodb convergence. -- The exact namespace convention for the `-ca-cert` object (per-release in the app namespace is assumed here) and the final label/annotation names. This proposal uses `internal.cozystack.io/publish-ca-cert` and `internal.cozystack.io/publish-ca-cert-release` on the source and `internal.cozystack.io/tenant-ca` on the projected copy, matching the `internal.cozystack.io/` convention of the existing `tenantresource` and `managed-by-cozystack` markers. +- The exact namespace convention for the `.tenant-ca` object (per-release in the app namespace is assumed here). - How this intersects with per-tenant wildcard propagation (issue `cozystack/cozystack#2820`, implemented by PR `cozystack/cozystack#2990`). That is a *cross-namespace* replication problem; the extraction controller here is *intra-namespace, per-release*, so the two do not share a mechanism — but they should share the management-label and foreign-collision conventions. Note that `clustersecret-operator` is packaged in the repo but is not wired into a default bundle, so it is not an installed primitive this design can assume. ## Alternatives considered @@ -218,10 +222,10 @@ Two consequences for the controller. First, it writes at runtime, after chart-re - **A custom `valuesFrom` pointing at `-ca`.** Rejected: `expectedValuesFrom()` pins every application HelmRelease to the single `cozystack-values` Secret and overwrites drift. - **Forking each upstream operator (the redis path) to emit a key-free CA.** Rejected as the general mechanism: it works for redis because the fork is small, but applying it to CloudNativePG, PSMDB, mariadb-operator, opster, and the rabbitmq cluster-operator is far more surface to own than one engine-agnostic controller. - **cert-manager `trust-manager`.** Considered and rejected as the mechanism, recorded as the validating prior art. trust-manager is the canonical "watch a CA source, materialize a key-free copy, never touch `tls.key`" operator and confirms the copy-model is the mainstream choice for trust-anchor distribution. But it does not fit operationally: its `Bundle`/`ClusterBundle` is cluster-scoped and fans out to namespaces by selector (not one object per release); it reads its sources from its own trust namespace, not from an arbitrary per-release tenant namespace; and it targets ConfigMaps first (Secret targets are opt-in). Modelling per-release, intra-namespace extraction on it would mean one cluster-scoped Bundle per database release reading a source it cannot natively see — against the grain. The small intra-namespace controller is a better fit. -- **External Secrets Operator (kubernetes provider).** The closest prior art of the copy-model options, and mechanically sound end-to-end: a `SecretStore` with `provider.kubernetes` plus a per-release `ExternalSecret` selecting a single key (`data[].remoteRef.property: ca.crt`, not `dataFrom`) materializes exactly the key-free `-ca-cert` object. Selecting one property is a structural no-private-key whitelist, `target.template` stamps the marker label, `refreshInterval` tolerates the operator-created asynchronous source, and the chart renders a *declaration* of the projection rather than its data — sidestepping the async/render-time wall the same way the controller does. The lineage chain resolves too: ESO owner-refs its output Secret to the `ExternalSecret`, itself a chart-rendered object, so the ownership walk reaches the owning application the way it does for any chart object, and garbage collection follows. Not adopted as the mechanism for one operational reason and two costs: ESO is packaged but **opt-in** — `cozystack.external-secrets-operator` renders only when listed in `bundles.enabledPackages`, which defaults to empty (`packages/core/platform/values.yaml`) — so the platform's trust-anchor contract would hinge on a component an operator may not run, and adopting it means first promoting ESO to a required, always-on platform dependency; each consuming namespace needs a `SecretStore` plus a ServiceAccount with read access to source Secrets (a per-namespace privileged-read fan-out the intra-namespace controller does not add); and refresh is poll-based (`refreshInterval`) where the controller is watch-driven, so rotation propagation is bounded by the polling period. If ESO later graduates to a default platform component, it is the natural retirement path for the extraction controller — recorded with the same status as the field-filter below. +- **External Secrets Operator (kubernetes provider).** The closest prior art of the copy-model options, and mechanically sound end-to-end: a `SecretStore` with `provider.kubernetes` plus a per-release `ExternalSecret` selecting a single key (`data[].remoteRef.property: ca.crt`, not `dataFrom`) materializes exactly the key-free `.tenant-ca` object. Selecting one property is a structural no-private-key whitelist, `target.template` stamps the marker label, `refreshInterval` tolerates the operator-created asynchronous source, and the chart renders a *declaration* of the projection rather than its data — sidestepping the async/render-time wall the same way the controller does. The lineage chain resolves too: ESO owner-refs its output Secret to the `ExternalSecret`, itself a chart-rendered object, so the ownership walk reaches the owning application the way it does for any chart object, and garbage collection follows. Not adopted as the mechanism for one operational reason and two costs: ESO is packaged but **opt-in** — `cozystack.external-secrets-operator` renders only when listed in `bundles.enabledPackages`, which defaults to empty (`packages/core/platform/values.yaml`) — so the platform's trust-anchor contract would hinge on a component an operator may not run, and adopting it means first promoting ESO to a required, always-on platform dependency; each consuming namespace needs a `SecretStore` plus a ServiceAccount with read access to source Secrets (a per-namespace privileged-read fan-out the intra-namespace controller does not add); and refresh is poll-based (`refreshInterval`) where the controller is watch-driven, so rotation propagation is bounded by the polling period. If ESO later graduates to a default platform component, it is the natural retirement path for the extraction controller — recorded with the same status as the field-filter below. - **A name-convention CA-distribution controller** (the controller watches `-ca` because its name is deterministic, and carries its own marking, garbage-collection, and collision guard). Rejected in favor of §5's explicit source label plus lineage reuse: the CA Secret names are non-uniform and `-ca-cert` is overloaded across engines, so a name convention would mis-handle at least one; and the marking/GC it re-implements is already provided by `spec.secrets` and owner references. - **A field filter on the projection itself (the principled root fix — deferred to its own proposal).** Today `tenantsecrets` delivers the whole Secret `Data`, which is the only reason a key-free *copy* must be materialized at all. A per-key field filter on the projection (or the `spec.secrets` selector) would let a single `ca.crt` key be projected straight out of a key-bearing Secret — no controller, no second object — and would generalize to the operator-owned engines. It is **not** a local change, which is why it is deferred rather than adopted here: the projection is **writable** at the registry level (`pkg/registry/core/tenantsecret/rest.go` implements Create/Update/Patch/Delete), and the write path (`tenantToSecret`) replaces the underlying Secret's `Data` **wholesale** (`out.Data = ts.Data`). A read-side key filter without a matching write-side filter would, the moment any principal with write access used the filtered view, silently drop the keys it could not see. Tenant roles grant only `get/list/watch` today, but that read-only posture lives in a different package (`packages/system/cozystack-basics` clusterroles), so a field filter's safety is entangled with a write path and an RBAC posture defined elsewhere. That makes it a redesign of the tenant-secret API with its own blast-radius analysis — a separate design proposal — not a rider on the TLS rollout. Recorded here as the future simplification the extraction controller could later retire, explicitly **not** a dependency of this proposal. -- **A native `ClusterTrustBundle` (KEP-3257).** Considered, recorded as the direction Kubernetes itself is taking for trust anchors (cluster-scoped, world-readable, public-only by construction — the API server rejects PEM with a private key — with central rotation). It does not fit this use case: it is consumed through the pod-mount `clusterTrustBundle` volume projection, whereas a Cozystack tenant reads `-ca-cert` as a named Secret object through the dashboard and `tenantsecrets`. Different consumer model; a copy object is what an object-by-name consumer needs. +- **A native `ClusterTrustBundle` (KEP-3257).** Considered, recorded as the direction Kubernetes itself is taking for trust anchors (cluster-scoped, world-readable, public-only by construction — the API server rejects PEM with a private key — with central rotation). It does not fit this use case: it is consumed through the pod-mount `clusterTrustBundle` volume projection, whereas a Cozystack tenant reads `.tenant-ca` as a named Secret object through the dashboard and `tenantsecrets`. Different consumer model; a copy object is what an object-by-name consumer needs. - **A general-purpose cluster-secret replication operator / a copy-issuer webhook.** Rejected during the edge work (`cozystack/cozystack#2990`, `cozystack/cozystack#2812`) in favor of native references and a purpose-built reconciler with a tight ownership guard that move only the Secret name, never key material. --- From 42f14c2b5a4b04e23a7956f3e67e82e95ca46282 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Mon, 31 Aug 2026 15:26:59 +0300 Subject: [PATCH 4/4] docs(unified-tls-pki): reconcile the proposal with the merged controller MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The sentinel-based §5 rework predates the extraction controller's actual merge and drifted from it on three points: the redis row and its §5(a) framing still claimed a key-free self-published CA in present tense, although the forked operator that would emit it is still an open pull request; the content guard described a private-key-header check where the merged controller instead decodes each PEM block and rebuilds the chain only from blocks that parse as an x509 certificate; and three places promised a ValidatingAdmissionPolicy pinning sentinel writes to helm-controller, which was never built — RBAC denying tenants any verb on internal.cozystack.io is the actual control. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/unified-tls-pki/README.md | 72 ++++++++++++++-------- 1 file changed, 47 insertions(+), 25 deletions(-) diff --git a/design-proposals/unified-tls-pki/README.md b/design-proposals/unified-tls-pki/README.md index 3e14f50..6bbcba2 100644 --- a/design-proposals/unified-tls-pki/README.md +++ b/design-proposals/unified-tls-pki/README.md @@ -130,7 +130,7 @@ The contract is a per-engine table. It is the artifact the per-app pull requests | clickhouse (Altinity) | — (BYO-cert) | supplied cert mount | n/a | no | no (not in TLS series yet) | mounts a supplied cert; no native issuance | | kubernetes (Kamaji) | Kamaji | Kamaji-owned, not swappable | n/a | n/a | no — out of model | control-plane CA, kubeconfig-pinned, non-swappable | -The takeaway: only **kafka** (native) is at the target today — it self-publishes a key-free `ca.crt`; **redis** would join it once the forked operator proposed in its open per-app pull request lands and starts emitting `-ca-cert`. **Every other engine has a key-bearing CA Secret and delivers no `ca.crt` to the tenant** — one uniform delivery gap. The gap does **not** split along mint-versus-consume: postgres and mongodb are operator-owned, yet their CA Secret is key-bearing (and operator-created, hence asynchronous), so they need the same key-free projection as the cert-manager-minting engines. The cert-manager-minting engines additionally never reach the tenant at all. Kamaji anchors the other end of the table: a PKI owner so absolute that the only sane unification is the one this contract chooses — leave the CA alone, unify how trust reaches the client. +The takeaway: only **kafka** (native) already publishes a key-free `ca.crt` under its own name today; **redis** reaches that once its forked operator lands in `cozystack/cozystack#2729`. Neither is yet at the canonical `.tenant-ca` — both still need the controller to copy their key-free output across under that name (§5). **Every other engine has a key-bearing CA Secret and delivers no `ca.crt` to the tenant** — one uniform delivery gap. The gap does **not** split along mint-versus-consume: postgres and mongodb are operator-owned, yet their CA Secret is key-bearing (and operator-created, hence asynchronous), so they need the same key-free projection as the cert-manager-minting engines. The cert-manager-minting engines additionally never reach the tenant at all. Kamaji anchors the other end of the table: a PKI owner so absolute that the only sane unification is the one this contract chooses — leave the CA alone, unify how trust reaches the client. ### 4. The uniform consume contract @@ -142,33 +142,53 @@ The canonical name is therefore `.tenant-ca`, and it is collision-free This is where the label-filtered projection matters. Because `tenantsecrets` delivers the whole Secret `Data`, the helper's fail-closed guard is not a nicety — it is the boundary that keeps a server or CA private key out of a tenant's hands. Kafka's `-clients-ca-cert` is the shape to match: a CA certificate, no key, readable by the tenant. -### 5. Delivery: one engine-agnostic extraction controller +### 5. Delivery: a declared sentinel and one engine-agnostic controller -The contract in §4 fixes the output object. The remaining work is the input path, and it splits on a single question — **does the engine's operator already emit a key-free `ca.crt` object?** +The contract in §4 fixes the output object; this section fixes the input path — how the key-free `.tenant-ca` gets written for an engine whose operator does not already emit it under that name. Everything in this section is **internal**: no tenant and no consumer observes any of it. A tenant still reads exactly the object §4 defines — `.tenant-ca`, containing only `ca.crt` — and nothing here changes what that object is, how it is labelled, or how it is consumed. The delivery mechanism lives entirely below the line §4 draws. -- **Engines that self-publish a key-free CA** need no platform machinery. `kafka` (Strimzi, native) is here today: it converges by matching the canonical name and stamping the tenant-resource label. `redis` would join it once its open per-app pull request lands a forked operator with `caCertSecretName`. Either engine opts **out** of the controller below simply by not stamping the source label. -- **Every other engine** — nats, qdrant, mariadb, opensearch, rabbitmq (cert-manager chains, chart-rendered, key-bearing CA) and mongodb, postgres (operator-rendered, key-bearing CA, asynchronous) — has a key-bearing CA Secret and no key-free output. None can feed the helper at chart-render time (see "The problem"). These opt **in** via an explicit source label and are served by **one engine-agnostic extraction controller**. +The input path splits on a single question — **does the engine's operator already emit a key-free `ca.crt` object?** — and the platform's standing preference (source from a key-free object where one exists; strip only when forced to) follows from it: + +- **Engines whose operator already publishes a key-free CA under its own name.** `kafka` (Strimzi, native) is here today: its CA already lands key-free, under `-clients-ca-cert` rather than the canonical `.tenant-ca`, so the controller still copies it across through a declared projection — a straight copy with nothing to strip. `redis` joins this group once its open per-app pull request lands a forked operator with `caCertSecretName`; on `main` today the operator is stock and emits no CA object, so redis is not there yet. +- **Every other engine** — nats, qdrant, mariadb, opensearch, rabbitmq (cert-manager chains, chart-rendered, key-bearing CA) and mongodb, postgres (operator-rendered, key-bearing CA, asynchronous) — has a key-bearing CA Secret and no key-free output. None can feed the helper at chart-render time (see "The problem"). The projection sources from the key-bearing CA Secret, and the controller strips everything but `ca.crt`. The alternative to one controller is to fork each upstream operator the way redis was forked (adding a `caCertSecretName`-style key-free output). Forking CloudNativePG, PSMDB, mariadb-operator, opster, and the rabbitmq cluster-operator is far more surface to own than a single small controller, and redis's fork was a reaction to a review blocker, not a designed-up-front pattern. So the controller is the target for every non-self-publishing engine. -The controller is deliberately **engine-agnostic**: it does not branch on cert-manager-vs-operator-owned, and it does not key off Secret names. Its contract is three pieces, only the middle of which is new code. +The mechanism is two pieces: a **declaration** the chart renders, and a small **controller** that reconciles it. An earlier revision of this proposal delivered the input path with a *label* on the CA Secret — the owner stamped `publish-ca-cert` and the controller selected sources by that label. An API-owner review of the implementation (`cozystack/cozystack#3299`) found that this emulates object semantics with a label: the thing being declared is "project this Secret's `ca.crt` as a trust anchor", a first-class intent that deserves a first-class object with a name, a status, and its own admission rules — not a boolean smuggled onto someone else's Secret. The mechanism below is that object. + +**(a) The owner declares the projection with a namespaced sentinel.** The app chart renders a small namespaced custom resource — working name `TenantProjection`, group `internal.cozystack.io` — that names the source Secret explicitly instead of stamping a label on it: + +```yaml +apiVersion: internal.cozystack.io/v1alpha1 +kind: TenantProjection +metadata: + name: +spec: + projections: + - type: CACert + sourceSecretName: -ca # the engine's CA-bearing Secret + sourceKey: ca.crt # key to lift (defaults to ca.crt) +``` -**(a) An explicit source-selection label, engine-agnostic, not a name convention.** The owner of the CA-bearing Secret stamps `internal.cozystack.io/publish-ca-cert: "true"`, with an optional `internal.cozystack.io/publish-ca-cert-key` annotation naming the key to lift (default `ca.crt`). It also stamps `internal.cozystack.io/publish-ca-cert-release` with the release name: implementation showed the release cannot be inferred on the label leg, because the platform runs cert-manager with `enableCertificateOwnerRef: false`, so a cert-manager-issued Secret carries no ownerReference back to its owning application and the lineage graph cannot resolve which release it belongs to. The release label supplies that link explicitly. For the cert-manager-minting charts all three markers ride `Certificate.spec.secretTemplate.labels`, so even the asynchronously-created CA Secret carries them from the moment cert-manager writes it. The label, not the name, is the contract — which matters because the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and the name `-ca-cert` is overloaded: key-free for the redis fork, key-bearing for PSMDB. A name convention would mis-handle one of them; the label plus a content check does not. +The sentinel is a **declaration**, not a copy of the data: the chart renders it at template time, so it exists in the Flux digest and is reconciled like any other chart object, even though the source Secret it names is created asynchronously by cert-manager or the DB operator long after render. Naming the source explicitly — rather than relying on a name convention or a label the operator must be coaxed into stamping — is what keeps the mechanism engine-agnostic: the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and `-ca-cert` is overloaded (key-free for the redis fork, key-bearing for PSMDB), and the declaration accommodates all of them because the chart author who knows the source name simply writes it down. It also dissolves the label design's release-attribution problem: cert-manager runs with `enableCertificateOwnerRef: false`, so a cert-manager-issued Secret carries no ownerReference back to its application and the label leg had to add a separate `-release` label to say which release it belonged to; the sentinel is itself a chart object in the release's namespace, attributed to its release the same way every chart object is (see (c)), so no such marker is needed. -**(b) A small extraction controller.** Its watch/upsert skeleton can follow the wildcard-secret reconciler from PR `cozystack/cozystack#2990` (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — lineage provides those (part (c)) — and, being **intra-namespace**, it can do something that reconciler cannot. For each selected source Secret it upserts a `type: Opaque` Secret named `.tenant-ca` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. It does four security-load-bearing things and nothing more: +**(b) A small extraction controller reconciles the sentinel.** The controller watches `TenantProjection` objects (not Secret labels) and the source Secrets they name. For each `CACert` projection it upserts a `type: Opaque` Secret named `.tenant-ca` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. Its watch/upsert skeleton can follow the wildcard-secret reconciler from PR `cozystack/cozystack#2990` (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — ownership and lineage provide those (part (c)). It does four security-load-bearing things and nothing more: -- **Key off the label and the content, never the name.** Select sources by the `publish-ca-cert` label; before writing, decode the source in a `pem.Decode` loop, parse each block with `x509.ParseCertificate`, and rebuild the chain only from blocks that validate as certificates, rather than detecting a private key by header. This is what lets one controller serve a key-free redis source it should ignore and a key-bearing PSMDB source it must strip, both named `-ca-cert`-adjacent, without confusion. -- **Tolerate operator-created, asynchronous sources.** The source may be created by the CNPG or PSMDB operator after the chart renders (not just by cert-manager). The controller waits on the watch event for the labeled source; it does not assume the chart owns the Secret and does not error or busy-loop before the source exists. +- **Read the source the sentinel names, and check its content.** Read only the Secret named in `sourceSecretName`, lift only `sourceKey`, and before writing decode it in a `pem.Decode` loop, parse each block with `x509.ParseCertificate`, and rebuild the projected value only from blocks that validate as a certificate — never copying the input verbatim, so no byte the parser did not accept as a certificate can reach the tenant. That verifies the lifted value is a certificate, not that it is specifically a CA certificate. The sentinel says *which* Secret; the content check says *whether to strip* — which is what lets one controller straight-copy a key-free redis source and strip a key-bearing PSMDB source without confusing the two, even though both sit under `-ca-cert`-adjacent names. +- **Tolerate operator-created, asynchronous sources.** The named source may be created by the CNPG or PSMDB operator (or by cert-manager) after the chart renders. The controller waits on the watch event for that source; it does not assume the chart owns the Secret and does not error or busy-loop before the source exists. The sentinel exists from render time, so there is always a durable declaration to reconcile against — and to carry status (part (e)) — while the source is still missing. - **Sanitize at write time, not just render time.** The `cozy-lib.tls.caCertSecret` helper's fail-closed guard runs at *chart-render* time; this controller writes at *runtime*, so it must itself copy only the single `ca.crt` key (an explicit whitelist) and re-assert the no-private-key check on every write. It never copies the whole `Data`. -- **Owner-ref the projected Secret to the release's `HelmRelease`.** An earlier revision of this document said to owner-ref the application instance CR; that is not implementable. The `apps.cozystack.io` kinds are virtual — served by the aggregated apiserver, and a tenant application is *stored* as a `HelmRelease` whose `spec.values` is the application spec verbatim — so there is no application object in etcd and therefore no UID to reference. The `HelmRelease` is the only real object in the chain, and it is enough: deleting the application through the aggregated API deletes the `HelmRelease`, which garbage-collects the projection with it. This is where intra-namespace beats the `cozystack/cozystack#2990` reconciler, which tracks its replicas by a management label plus a back-reference annotation *because* a cross-namespace replica cannot carry a valid `ownerReference`. Our projection is same-namespace, so a real owner-reference is valid and the controller needs no prune logic of its own. `blockOwnerDeletion` stays false: a trust anchor must never delay the teardown of the application it belongs to. (It does not owner-ref the *source* `-ca`: cert-manager and the DB operators do not own their output Secrets, so a walk from the source would dead-end before the app.) +- **Owner-ref the projected Secret to the sentinel.** The projection takes an `ownerReference` to the `TenantProjection` that declared it — not to the application-instance CR, and not to the `HelmRelease` directly. The application-instance CR cannot be referenced: the `apps.cozystack.io` kinds are virtual — served by the aggregated apiserver, with a tenant application *stored* as a `HelmRelease` whose `spec.values` is the application spec verbatim — so there is no application object in etcd and no UID to point at. The sentinel, by contrast, is a real chart-rendered object in the same namespace, so a real owner-reference is valid and garbage collection is native (part (c)). `blockOwnerDeletion` stays false: a trust anchor must never delay the teardown of the application it belongs to. (The projection does not owner-ref the *source* CA Secret: cert-manager and the DB operators do not own their output Secrets, so a walk from the source would dead-end before reaching the app.) + +**(c) Attribution, lineage, and garbage collection: the sentinel is a plain chart object.** The sentinel itself carries **no** `ownerReferences`. Flux stamps every object it renders with the `helm.toolkit.fluxcd.io/name` label naming the release, and that label — not an owner-reference — is how the sentinel is attributed to its release, exactly as for any chart-rendered object. So the lineage walk resolves the chain **Secret → sentinel → HelmRelease → application** with no change to the webhook: the projected Secret owner-refs the sentinel, the sentinel's release label names the `HelmRelease`, and the `HelmRelease` backs the application. Garbage collection is native and needs no delete path in the controller: deleting the application deletes the `HelmRelease`, Flux prunes the chart's objects including the sentinel, and Kubernetes then garbage-collects the projection the sentinel owns. Marking the projection tenant-visible is unchanged from §4 — the controller stamps `internal.cozystack.io/tenant-ca: "true"` on it, one generic `ApplicationDefinition.spec.secrets.include` entry (`matchLabels: {internal.cozystack.io/tenant-ca: "true"}`) matches it by label (`internal/lineagecontrollerwebhook/matcher.go`), and the lineage webhook then walks the projection's owner chain and authoritatively stamps `internal.cozystack.io/tenantresource` to `true` (and to `false` should the Secret ever stop matching). + +**(d) Admission: RBAC is the control, not a ValidatingAdmissionPolicy.** No tenant role grants any verb on `internal.cozystack.io`, so a tenant cannot create, update, or delete a `TenantProjection` to aim the controller at a Secret of its choosing — the only way a sentinel enters the cluster is by being rendered into a release. The extraction controller itself holds only `get`/`list`/`watch` on `tenantprojections`; it never creates or deletes one, since that stays the chart's (helm-controller's) job. -**(c) Marking stays in `spec.secrets`, by label not by name.** Because `ApplicationDefinition.spec.secrets.include` accepts a label selector (`internal/lineagecontrollerwebhook/matcher.go`), one generic entry — `matchLabels: {internal.cozystack.io/tenant-ca: "true"}`, stamped by the controller on every projected Secret — covers every engine with no per-release `resourceName` templating. The lineage admission webhook then does the rest: on admission it walks the projected Secret's `ownerReferences` to the owning application and authoritatively stamps `internal.cozystack.io/tenantresource` to `true` (and to `false` should the Secret ever stop matching). +**(e) A broken declaration is observable.** The sentinel carries `status.conditions`, and the controller sets `Ready`. When a declaration cannot be satisfied — the named source Secret does not exist, or an upstream bump renames the CA Secret out from under `sourceSecretName` — the controller records `Ready=False, Reason=SourceNotFound` on the sentinel. This is a concrete advantage over the label design, which had no object to carry status: there, a source that never appeared could only make the controller requeue silently, invisible until someone noticed the trust anchor was missing. With the sentinel the failure is a queryable condition on a named object — alertable, and pointing at the exact broken release. -What this *reuses* rather than rebuilds: the label selector in `ApplicationDefinition.spec.secrets` (`internal/lineagecontrollerwebhook/matcher.go`), the lineage webhook's owner-reference walk and authoritative `tenantresource` stamping (`internal/lineagecontrollerwebhook/webhook.go`, `pkg/lineage/lineage.go`), the private-key guard in `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`), and native Kubernetes garbage collection via the owner reference on the `HelmRelease`. The irreducibly new work is the extraction step itself — read one key, write a key-free copy, re-copy on rotation — the same job an operator does natively on the engines that already ship a key-free CA object (kafka's `-clients-ca-cert`, the redis fork's `-ca-cert`) — those two still need the projection, because their CA lands under an engine-specific name rather than the canonical one; for them the extraction step is a straight copy, with nothing to strip. +What this *reuses* rather than rebuilds: Flux's release-name label and its native pruning (for attribution and garbage collection), the lineage webhook's owner-reference walk and authoritative `tenantresource` stamping (`internal/lineagecontrollerwebhook/webhook.go`, `pkg/lineage/lineage.go`), the label selector in `ApplicationDefinition.spec.secrets` (`internal/lineagecontrollerwebhook/matcher.go`), and the private-key guard in `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`). The irreducibly new work is the `TenantProjection` sentinel and the extraction step that reconciles it — read one key from the named source, write a key-free copy, re-copy on rotation, report readiness — the same job an operator does natively on the engines that already ship a key-free CA object (kafka's `-clients-ca-cert`, the redis fork's `-ca-cert`). Those two still declare a sentinel, because their CA lands under an engine-specific name rather than the canonical one; for them the extraction step is a straight copy, with nothing to strip. ### 6. Per-engine application order -`postgres` goes first (under the tracking issue `cozystack/cozystack#2814`): it is the engine the epic most wants to see converged, and it exercises the controller against the hardest input — an **operator-created, asynchronous** CA Secret — so validating it there validates the mechanism everywhere. Convergence for postgres means publishing a key-free `.tenant-ca` extracted from CNPG's key-bearing `-ca`, while leaving the passwords-only `-credentials` untouched. `kafka` is already at the target and needs only documentation. `redis` (`cozystack/cozystack#2729`) converges by shape adaptation: its forked operator self-publishes a key-free `-ca-cert` via `caCertSecretName`, which the controller copies verbatim into the canonical `.tenant-ca` — a straight copy with nothing to strip, the same treatment kafka gets. `mongodb`, then the cert-manager-minting engines (nats, qdrant, and then rabbitmq, mariadb, opensearch) adopt the controller via the source label. `kubernetes` (Kamaji) is explicitly out. +`postgres` goes first (under the tracking issue `cozystack/cozystack#2814`): it is the engine the epic most wants to see converged, and it exercises the controller against the hardest input — an **operator-created, asynchronous** CA Secret — so validating it there validates the mechanism everywhere. Convergence for postgres means publishing a key-free `.tenant-ca` extracted from CNPG's key-bearing `-ca`, while leaving the passwords-only `-credentials` untouched. `kafka` already publishes a key-free `ca.crt` under its own name, so it converges by declaring a `TenantProjection` sentinel that names `-clients-ca-cert` as the source — a straight copy with nothing to strip. `redis` (`cozystack/cozystack#2729`) converges the same way once its forked operator ships: its `caCertSecretName` output becomes the sentinel's source, and the controller copies it verbatim into the canonical `.tenant-ca`. `mongodb`, then the cert-manager-minting engines (nats, qdrant, and then rabbitmq, mariadb, opensearch) converge by declaring a `TenantProjection` sentinel that names their CA Secret. `kubernetes` (Kamaji) is explicitly out. ## User-facing changes @@ -184,35 +204,36 @@ The trust boundary is precise: a tenant receives `ca.crt` and never receives `tl No merged engine leaks a private key to a tenant today. The merged charts withhold their key-bearing Secrets, and the tenant-facing objects (postgres `-credentials`, the nats/qdrant credentials Secrets) are passwords-only. The live instance of the risk is not in `main` — it is in the **in-flight per-app PRs**, several of which currently propose labelling a key-bearing Secret to tenants: mariadb (`cozystack/cozystack#2680`) projects the CA private key `-ca-tls`, mongodb (`cozystack/cozystack#2692`) projects the key-bearing `-ca-cert`, and rabbitmq (`cozystack/cozystack#2683`) and opensearch (`cozystack/cozystack#2682`) project the leaf key. This contract exists to stop them landing that way; the consume object and the extraction controller are what let those PRs deliver `ca.crt` without the key. -Two consequences for the controller. First, it writes at runtime, after chart-render, so it cannot lean on the helper's render-time guard alone: it must whitelist the single `ca.crt` key and re-assert the no-private-key check itself on every write. Second, it adds a new trust surface — read access to per-release CA Secrets, write access for the key-free copy — and must never overwrite a Secret it did not create, surfacing any name collision as a Warning Event rather than failing silently. +Two consequences for the controller. First, it writes at runtime, after chart-render, so it cannot lean on the helper's render-time guard alone: it must whitelist the single `ca.crt` key and re-assert the no-private-key check itself on every write. Second, it adds a new trust surface — read access to per-release CA Secrets, write access for the key-free copy — and must never overwrite a Secret it did not create, surfacing any name collision as a Warning Event rather than failing silently. The declaration that drives it is itself locked down: no tenant role grants any verb on `internal.cozystack.io`, so a tenant cannot forge a `TenantProjection` aiming the controller at a Secret of their choosing — RBAC alone is the control, with no admission policy narrowing the writer further. ## Failure and edge cases - The helper's input PEM contains a private-key header → render fails closed; the chart does not deploy a key-bearing Secret. -- The source `ca.crt` somehow carries a private-key header at runtime → the controller refuses to write the copy (runtime whitelist plus guard); no key-bearing Secret is ever projected. -- The labeled source is a key-free Secret that should be left alone (a self-publishing engine mis-stamped) → the content check sees no private key to strip and the controller still writes only `ca.crt`; no key is ever exposed. -- The per-release CA Secret does not exist yet (operator-created, asynchronous) → the controller waits for the watch event on the labeled source; it does not error or busy-loop. +- The source `ca.crt` somehow carries a block that does not parse as an `x509.Certificate` at runtime → the controller refuses to write the copy (the `pem.Decode`/`x509.ParseCertificate` guard rebuilds the projection only from validated blocks); no key-bearing Secret is ever projected. +- The sentinel names a key-free source that should be left alone (a self-publishing engine) → the content check sees no private key to strip and the controller still writes only `ca.crt`; no key is ever exposed. +- The named source Secret does not exist yet (operator-created, asynchronous) → the controller waits for the watch event on that source and records `Ready=False, Reason=SourceNotFound` on the sentinel meanwhile; it does not error or busy-loop. +- An upstream bump renames the CA Secret out from under `sourceSecretName` → the source never reappears under the declared name, and the sentinel surfaces `Ready=False, Reason=SourceNotFound` — a queryable, alertable condition rather than a silent requeue. - A foreign Secret already occupies the target name → the controller leaves it untouched (management-label guard) and emits a Warning Event on the application, so the operator sees the collision. - The CA rotates → the controller re-copies `ca.crt` on the next source change; no chart re-render is required. -- The application is deleted → the projected `.tenant-ca` is garbage-collected by Kubernetes via its owner reference to the `HelmRelease` that backs the application; the controller needs no delete path. +- The application is deleted → Flux prunes the sentinel with the rest of the chart's objects, and Kubernetes garbage-collects the projected `.tenant-ca` via its owner reference to the sentinel; the controller needs no delete path. ## Testing - The helper is already covered by `packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml`, including the fail-closed assertions. -- The extraction controller gets an envtest/Ginkgo suite (its skeleton mirrors `internal/controller/wildcardsecret/reconciler_test.go`): a labeled source appearing after the consumer (cert-manager and operator-created), a source whose `ca.crt` is swapped (rotation), a key-free source that must be left byte-for-byte (no spurious rewrite), a key-bearing source that must be stripped to `ca.crt`, a foreign-name collision (asserting the Secret is untouched and a Warning Event is emitted), a source value that smuggles a private-key header (asserting the controller refuses to write), and owner-reference-driven garbage collection on app deletion. -- Each per-app pull request adds helm-unittest fixtures asserting the `.tenant-ca` shape and label, plus an end-to-end check under `hack/e2e-apps/` that a tenant can read `ca.crt`, cannot read any object carrying `tls.key`, and can verify the server. +- The extraction controller gets an envtest/Ginkgo suite (its skeleton mirrors `internal/controller/wildcardsecret/reconciler_test.go`): a sentinel whose named source appears after it (cert-manager and operator-created), a source whose `ca.crt` is swapped (rotation), a key-free source that must be left byte-for-byte (no spurious rewrite), a key-bearing source that must be stripped to `ca.crt`, a foreign-name collision (asserting the Secret is untouched and a Warning Event is emitted), a source value that smuggles a private-key header (asserting the controller refuses to write), a sentinel whose source never appears (asserting `Ready=False, Reason=SourceNotFound` on `status.conditions`), and owner-reference-driven garbage collection of the projection when the sentinel is pruned. +- Each per-app pull request adds helm-unittest fixtures asserting the chart renders the `TenantProjection` sentinel with the right `sourceSecretName`, and asserting the projected `.tenant-ca` shape and label, plus an end-to-end check under `hack/e2e-apps/` that a tenant can read `ca.crt`, cannot read any object carrying `tls.key`, and can verify the server. +- No dedicated admission fixture is needed: RBAC is the control, and no tenant-facing ClusterRole grants any verb on `internal.cozystack.io`, so a tenant cannot write a `TenantProjection` by any path. ## Rollout 1. Edge — `cozystack/cozystack#2988` and `cozystack/cozystack#2989` merged; per-tenant propagation `cozystack/cozystack#2990` still open. 2. This contract — accepted. -3. Extraction controller — implemented and tested. +3. Sentinel CRD and extraction controller — implemented and tested. 4. Per-app convergence — postgres first (tracked by `cozystack/cozystack#2814`), then the remaining per-app TLS pull requests onto the contract. ## Open questions -- **Labelling operator-created sources.** The cert-manager charts stamp the source label via `Certificate.spec.secretTemplate.labels`. Operators that create the CA Secret themselves (CNPG `-ca`, PSMDB `-ca-cert`) may not let the chart label their output. For those, the controller needs either an operator that supports output labels or a small per-engine source-name configuration as a fallback — this is the one place the engine-agnostic label is not yet sufficient, and it should be resolved before postgres/mongodb convergence. -- The exact namespace convention for the `.tenant-ca` object (per-release in the app namespace is assumed here). +- The exact namespace convention for the `.tenant-ca` object (per-release in the app namespace is assumed here) and the final names of the new API. This proposal uses the working name `TenantProjection` in group `internal.cozystack.io` for the sentinel and `internal.cozystack.io/tenant-ca` on the projected copy, matching the `internal.cozystack.io/` convention of the existing `tenantresource` and `managed-by-cozystack` markers; the group, kind, and the `spec.projections[].type` vocabulary are not yet frozen. - How this intersects with per-tenant wildcard propagation (issue `cozystack/cozystack#2820`, implemented by PR `cozystack/cozystack#2990`). That is a *cross-namespace* replication problem; the extraction controller here is *intra-namespace, per-release*, so the two do not share a mechanism — but they should share the management-label and foreign-collision conventions. Note that `clustersecret-operator` is packaged in the repo but is not wired into a default bundle, so it is not an installed primitive this design can assume. ## Alternatives considered @@ -223,7 +244,8 @@ Two consequences for the controller. First, it writes at runtime, after chart-re - **Forking each upstream operator (the redis path) to emit a key-free CA.** Rejected as the general mechanism: it works for redis because the fork is small, but applying it to CloudNativePG, PSMDB, mariadb-operator, opster, and the rabbitmq cluster-operator is far more surface to own than one engine-agnostic controller. - **cert-manager `trust-manager`.** Considered and rejected as the mechanism, recorded as the validating prior art. trust-manager is the canonical "watch a CA source, materialize a key-free copy, never touch `tls.key`" operator and confirms the copy-model is the mainstream choice for trust-anchor distribution. But it does not fit operationally: its `Bundle`/`ClusterBundle` is cluster-scoped and fans out to namespaces by selector (not one object per release); it reads its sources from its own trust namespace, not from an arbitrary per-release tenant namespace; and it targets ConfigMaps first (Secret targets are opt-in). Modelling per-release, intra-namespace extraction on it would mean one cluster-scoped Bundle per database release reading a source it cannot natively see — against the grain. The small intra-namespace controller is a better fit. - **External Secrets Operator (kubernetes provider).** The closest prior art of the copy-model options, and mechanically sound end-to-end: a `SecretStore` with `provider.kubernetes` plus a per-release `ExternalSecret` selecting a single key (`data[].remoteRef.property: ca.crt`, not `dataFrom`) materializes exactly the key-free `.tenant-ca` object. Selecting one property is a structural no-private-key whitelist, `target.template` stamps the marker label, `refreshInterval` tolerates the operator-created asynchronous source, and the chart renders a *declaration* of the projection rather than its data — sidestepping the async/render-time wall the same way the controller does. The lineage chain resolves too: ESO owner-refs its output Secret to the `ExternalSecret`, itself a chart-rendered object, so the ownership walk reaches the owning application the way it does for any chart object, and garbage collection follows. Not adopted as the mechanism for one operational reason and two costs: ESO is packaged but **opt-in** — `cozystack.external-secrets-operator` renders only when listed in `bundles.enabledPackages`, which defaults to empty (`packages/core/platform/values.yaml`) — so the platform's trust-anchor contract would hinge on a component an operator may not run, and adopting it means first promoting ESO to a required, always-on platform dependency; each consuming namespace needs a `SecretStore` plus a ServiceAccount with read access to source Secrets (a per-namespace privileged-read fan-out the intra-namespace controller does not add); and refresh is poll-based (`refreshInterval`) where the controller is watch-driven, so rotation propagation is bounded by the polling period. If ESO later graduates to a default platform component, it is the natural retirement path for the extraction controller — recorded with the same status as the field-filter below. -- **A name-convention CA-distribution controller** (the controller watches `-ca` because its name is deterministic, and carries its own marking, garbage-collection, and collision guard). Rejected in favor of §5's explicit source label plus lineage reuse: the CA Secret names are non-uniform and `-ca-cert` is overloaded across engines, so a name convention would mis-handle at least one; and the marking/GC it re-implements is already provided by `spec.secrets` and owner references. +- **A source-selection label on the CA Secret (the earlier design of this proposal, superseded).** The owner stamped `internal.cozystack.io/publish-ca-cert` on its CA Secret and the controller selected sources by that label. It was adopted first, then replaced by the declared `TenantProjection` sentinel of §5 after an API-owner review (`cozystack/cozystack#3299`) found it emulates object semantics: a label carries no name, no status, and no admission surface of its own, so a broken declaration could only requeue silently, and the release a cert-manager Secret belonged to had to be supplied by a second `-release` label because the Secret carried no ownerReference. The declared sentinel is the chosen mechanism — a first-class namespaced object with `status.conditions`, RBAC-restricted writes, and a source named **by name** rather than by label. Declaring the source by name also closes this proposal's former open question "Labelling operator-created CA sources": CNPG and PSMDB need not label their operator-created output at all, because the chart names that Secret in the sentinel's `sourceSecretName`. The implementation briefly carried a second discovery leg alongside the label — a `spec.caCert` field on the cluster-scoped, per-kind `ApplicationDefinition` — with the mirror-image flaw: a cluster-scoped object standing in for a per-namespace, per-release fact. Both legs collapse into the single namespaced sentinel. +- **A name-convention CA-distribution controller** (the controller watches `-ca` because its name is deterministic, and carries its own marking, garbage-collection, and collision guard). Rejected in favor of §5's declared sentinel plus lineage reuse: the CA Secret names are non-uniform and `-ca-cert` is overloaded across engines, so a name convention would mis-handle at least one; the sentinel names the source explicitly instead, and the marking/GC it re-implements is already provided by `spec.secrets` and owner references. - **A field filter on the projection itself (the principled root fix — deferred to its own proposal).** Today `tenantsecrets` delivers the whole Secret `Data`, which is the only reason a key-free *copy* must be materialized at all. A per-key field filter on the projection (or the `spec.secrets` selector) would let a single `ca.crt` key be projected straight out of a key-bearing Secret — no controller, no second object — and would generalize to the operator-owned engines. It is **not** a local change, which is why it is deferred rather than adopted here: the projection is **writable** at the registry level (`pkg/registry/core/tenantsecret/rest.go` implements Create/Update/Patch/Delete), and the write path (`tenantToSecret`) replaces the underlying Secret's `Data` **wholesale** (`out.Data = ts.Data`). A read-side key filter without a matching write-side filter would, the moment any principal with write access used the filtered view, silently drop the keys it could not see. Tenant roles grant only `get/list/watch` today, but that read-only posture lives in a different package (`packages/system/cozystack-basics` clusterroles), so a field filter's safety is entangled with a write path and an RBAC posture defined elsewhere. That makes it a redesign of the tenant-secret API with its own blast-radius analysis — a separate design proposal — not a rider on the TLS rollout. Recorded here as the future simplification the extraction controller could later retire, explicitly **not** a dependency of this proposal. - **A native `ClusterTrustBundle` (KEP-3257).** Considered, recorded as the direction Kubernetes itself is taking for trust anchors (cluster-scoped, world-readable, public-only by construction — the API server rejects PEM with a private key — with central rotation). It does not fit this use case: it is consumed through the pod-mount `clusterTrustBundle` volume projection, whereas a Cozystack tenant reads `.tenant-ca` as a named Secret object through the dashboard and `tenantsecrets`. Different consumer model; a copy object is what an object-by-name consumer needs. - **A general-purpose cluster-secret replication operator / a copy-issuer webhook.** Rejected during the edge work (`cozystack/cozystack#2990`, `cozystack/cozystack#2812`) in favor of native references and a purpose-built reconciler with a tight ownership guard that move only the Secret name, never key material.