[Andrei Kvapil (kvaps)]

New section under Operations: Encryption at Rest, in next/ only. The etcd guide needs Talos v1.14, and the Keycloak guide relies on the DEK-set check from cozystack/cozystack#4818, which is on main and not in v1.7.

Three pages:

• Overview: what Cozystack stores where, what is encrypted by default and where the keys are.
• etcd with KMS v2: management cluster etcd encrypted with Vault Transit through vault-kubernetes-kms running as a Talos static pod. It also encrypts Flux HelmReleases, because application values (passwords included) live in spec.values. Short notes for aggregated API servers (the Cozystack API has no storage of its own, so it is already covered) and for tenant clusters, where KMS is not possible yet and the page shows the local-key option through the existing controlPlane.apiServer values. Rotation, rollback back to the Talos secretbox setup, troubleshooting.
• Keycloak: keycloak-kms-proxy with Vault Transit, including encrypting existing rows in a maintenance window, and what to do on an installation that already ran the proxy without a DEK set.

Every field, flag and command was checked against the source at pinned versions: Talos v1.14.2, Talm v0.35.0, vault-kubernetes-kms v1.4.0, keycloak-kms-proxy v0.2.3, Kubernetes 1.34, and the kubernetes and keycloak charts on main. The tenant values and the Keycloak values render with helm template. The same setup runs on a production installation. Not run end to end on a clean cluster: the Talm node-file flow with $patch: delete for secretboxEncryptionSecret, and the local-key option for tenant clusters.

[netlify]

✅ Deploy Preview for cozystack ready!

Name	Link
🔨 Latest commit	e7c6f4b
🔍 Latest deploy log	https://app.netlify.com/projects/cozystack/deploys/6af1ef4b84bb65f42e635c22
😎 Deploy Preview	https://deploy-preview-742--cozystack.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

Important

• Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 9d348163-e0f3-4384-ab54-c72ddf49b11c

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[Aleksei Sviridkin (lexfrei)]

Andrei Kvapil (@kvaps) NOT LGTM. The pages themselves hold up against the code I checked; the Keycloak page has to wait for cozystack/cozystack#4818, which I just sent back with changes requested.

The Keycloak page says "The chart refuses to enable encryption without it". On cozystack main 52f542c4 that is false: deksetSecretName is optional there and the proxy falls back to an in-memory DEK. Since next/ is published from main, merging this first would document a guard that does not exist. #4818 may also still change in ways this page should mention, for example how the static KEK Secret survives disabling encryption. So please re-check the page against what #4818 merges with.

The PR description starts with "Do not merge until cozystack/cozystack#4818 is merged." This repo uses the description as the merge commit message, so that line would stay in main's history. Please drop it before merging.

What I checked on the Keycloak page and found correct: every encryption.* key and its default against packages/system/keycloak/values.yaml, the backfill flags (generate-dekset, encrypt-rows, -vault-mount, -dsn) against keycloak-kms-proxy v0.2.3, the encrypted columns against internal/config/fields.go, and the metrics port and kkp_decrypt_failures_total. The Vault policy is enough too, because the proxy only calls encrypt/ and decrypt/ on the key. The suspend step is safe as well: the Package reconciler keeps the HelmRelease's spec.suspend as it finds it. For the tenant part, --encryption-provider-config is not one of the flags the kubernetes chart drops or refuses, and secret volumes are allowed in extraVolumes.

One non-blocking note: the prerequisites say "Talos v1.14.1 runs etcd v3.7.1", but next already pins Talos v1.14.2, so the etcd version in that sentence should follow the pinned Talos.

[Andrei Kvapil (kvaps)]

Aleksei Sviridkin (@lexfrei) merged main and fixed the Talos version, v1.14.2 runs the same etcd 3.7.1. Re-checked the Keycloak page against #4818 as it is now: the page covers only Vault Transit, so the keep annotation and the stricter KEK rule don't change anything there. I'll drop the "Do not merge" line from the description once #4818 is merged.

[Aleksei Sviridkin (lexfrei)]

Andrei Kvapil (@kvaps) NOT LGTM. The Talos version fix is correct: Talos v1.14.2 sets DefaultEtcdVersion = "3.7.1" (pkg/machinery/constants/constants.go:443), and main pins v1.14.2. The other two points from my last review are still open, and a full pass found four more problems.

Still open from the last review:

1. The Keycloak page still describes a guard that main does not have. The alert at keycloak.md:28 says "The chart refuses to enable encryption without it". On cozystack main b3662866 the only guard is for replicas > 1 (packages/system/keycloak/templates/proxy.yaml:55). helm template with enabled: true, kms.backend: vault-transit and no deksetSecretName renders without error there, and fails on #4818 (b62f1caf) as the page says. v1.7.0-rc.2 does not have that guard either, and next/ becomes the v1.7 docs when that release is cut. So #4818 has to merge first, or the sentence has to name the release that has the guard. The same applies to the body of commit 8da73b3 ("The keycloak chart now refuses..."). This repo merges with merge commits, so that sentence lands in main as written.
2. The description still starts with "Do not merge until...", and the description becomes the merge commit message. Dropping it once #4818 merges is fine, I only note that it is still there.

New:

3. The Keycloak page has nothing for users who already run encryption without a DEK set. encryption with an optional deksetSecretName has shipped since v1.6.0. Anyone who enabled it that way will get the render failure from #4818 on upgrade and will probably come to this page. Its steps don't help them. A new DEK set cannot open values written under the old in-memory DEK, and encrypt-rows skips every value that already starts with $KKP$ (cmd/backfill/encrypt.go:103 in v0.2.3). Those rows stay unreadable. Please add a warning for this case and the recovery path. As far as I can tell, the only one is restoring the backup taken before encryption was enabled.
4. The Vault token in the Keycloak prerequisites can't run the backfill. Line 34 asks for a token that "can create a Transit key and use it once to encrypt". Step 3 runs encrypt-rows with the same $VAULT_TOKEN, and that command unwraps the DEK set through kms.OpenCipher (internal/kms/envelope.go:68-79). The unwrap calls transit/decrypt/keycloak (internal/kms/vault.go:99). With the token scoped as described, the backfill fails with 403 in the middle of the maintenance window, while Keycloak is down. The token needs update on transit/decrypt/keycloak too. "Once" is also not accurate, because generate-dekset wraps more than one key.
5. The node files also hold the secretbox key in plain text, and the Git advice misses it. Until step 7 that key (etcd-kms.md:211) decrypts every Secret written before the switch. The alert at :231-232 moves only the pod definition and the two AppRole IDs into templates and encrypted values. Following it as written still commits the secretbox key to the repo history. Please add the key to that list.
6. The rollback section loses every HelmRelease. etcd-kms.md:444 says to rewrite "all Secrets" before removing the kms provider. Step 5 moved HelmReleases to KMS too, so after a rollback done exactly as written, they all become unreadable once the provider is gone, and Flux loses every application. The rewrite has to include the HelmRelease command from step 5. Separately, with identity first the rollback leaves Secrets in plain text, which is weaker than the secretbox setup the cluster started from. Putting the secretbox key first brings it back to the starting point.

Non-blocking:

• The private CA advice at etcd-kms.md:229 doesn't work as written. The static pod mounts only /var/kms, so a CA file placed anywhere else on the node is not visible to -vault-ca-cert. Either say to put it under /var/kms, or show a hostPath mount for it.
• The rotation section (etcd-kms.md:427) has the same Secrets-only wording before raising min_decryption_version. Also, the check in step 6 only looks at the k8s:enc:kms:v2: prefix and can't show the key version. A way to confirm nothing is left on an old version would help.
• The Talm prerequisite is stronger than the Talm docs. "Earlier releases cannot render configuration for Talos v1.14 nodes" vs docs/configuration/talos-versions.md:22 at v0.35.0, which says v1.14 nodes are served by a v1.13 contract.
• The plugin also calls auth/token/renew-self (pkg/vault/lease.go:100 at v1.4.0). It works only because AppRole tokens get the default policy, so one sentence for roles with token_no_default_policy would be enough.
• The pod name keycloak-db-1 at keycloak.md:183 may not exist after a failover or instance replacement. --selector cnpg.io/cluster=keycloak-db,cnpg.io/instanceRole=primary avoids that.

I checked these and found them correct: the vault-kubernetes-kms v1.4.0 binary path, flags and env names, the KubeEtcdEncryptionConfig shape and the exact conflict error in Talos v1.14.2, UID 65534 for kube-apiserver, and the Kubernetes range against the Talos support matrix. Same for the tenant controlPlane.apiServer values, which render into the KamajiControlPlane on main and are not refused, and for the Cozystack API having no etcd of its own. The Keycloak resource names match main, and every internal link and anchor resolves in a local build with next/ enabled.

[IvanHunters]

LGTM.

I verified every tool flag, chart value and version claim in these three pages against upstream source at the pinned versions, rather than trusting the prose: vault-kubernetes-kms v1.4.0 (flags, env, default health port, the /live behaviour), keycloak-kms-proxy v0.2.3 (backfill subcommands and flags, the $KKP$ marker and backfill idempotency, buildCipher, the encrypted columns and their modes), the cozystack.keycloak and kubernetes chart values on main, and the Talos 1.14 / Kubernetes / etcd facts. Everything checks out. Notably etcd 3.7.1 in Talos v1.14.2 is correct (constants.go + the v1.14.2 image list), the keycloak encryption.* values and the deksetSecretName guard are present on main, and all internal links resolve (the deploy preview builds green).

A few optional, non-blocking precision notes are inline. None of them is wrong at the version this page pins today (Talos v1.14.2); they are about claims that are a little broader than the thing they rest on.

One thing outside the pages themselves: the PR description still says "do not merge until cozystack/cozystack#4818 is merged", but #4818 merged on 2026-10-27, so that gate is already lifted.

[IvanHunters]

[MINOR] DRBD module claim is version-fragile and already reverted upstream

[MINOR] This is correct at the Talos version the page pins today (v1.14.2), but the blanket phrasing "Talos v1.14 no longer loads kernel modules on demand" holds only for v1.14.0-v1.14.4. Talos v1.14.5 (2026-10-26) restored on-demand loading, so from that patch drbd_transport_tcp no longer needs to be listed, and this whole step becomes unnecessary. Consider scoping the sentence to the pinned version, or noting that v1.14.5 reverted it. Separately, from Talm v0.36.0 the cozystack preset already carries drbd_transport_tcp in its built-in module list, so the manual extraKernelModules entry is redundant there.

[IvanHunters]

[NIT] Talm v0.35.0 prerequisite reason is stronger than verified

[NIT] Requiring Talm v0.35.0+ is right (it is the first release built against the Talos v1.14 machinery). The justification "Earlier releases cannot render configuration for Talos v1.14 nodes" is the part I could not confirm: a v1.14 node still accepts a v1.13-contract v1alpha1 config (the deprecated fields are not removed), and this guide deliberately keeps talosVersion at v1.13. What is genuinely gated on a newer Talm is rendering the v1.14 contract with typed Kube* documents (Talm v0.36.0). Optional: soften the reason, or keep just "v0.35.0 is the first release built against Talos v1.14".

[IvanHunters]

[NIT] KMS v1 is disabled by default, not unconditionally

[NIT] KMS v1 is disabled by default since Kubernetes 1.29; it can still be force-enabled with --feature-gates=KMSv1=true. The practical advice (apiVersion: v2 is what you want) is unaffected; this is just precision.

[IvanHunters]

[NIT] The deksetSecretName guard is conditional on encryption.fields

[NIT] The chart's fail on an empty deksetSecretName only fires when encryption.fields is not "disabled" (templates/proxy.yaml on main). With fields: disabled (passthrough, no column encryption) the chart renders without a DEK set. That is defensible, since fields: disabled means nothing is being encrypted, so "refuses to enable encryption without it" is true in spirit. Flagging only in case you want the alert to be exact.

[Andrei Kvapil (kvaps)]

Aleksei Sviridkin (@lexfrei) thanks, all of it is in.

1. #4818 is merged. It is not in v1.7.0 (the backport batch skipped it as breaking), and next/ now builds the release after v1.7 from main, where the check exists, so the sentence holds. Made it exact for fields: disabled too.
2. Dropped the "Do not merge" line, the description is rewritten. The branch is squashed into one commit, so the old commit body is gone as well.
3. Added "If Encryption Is Already On Without a DEK Set": what is lost, that the backfill skips $KKP$ values, the pre-encryption backup as the only recovery, and not to restart the running proxy.
4. The token now needs update on both encrypt/ and decrypt/, without "once".
5. The Git advice now covers the secretbox key.
6. Rollback goes back to the Talos secretbox setup in two phases, rewrites HelmReleases too, and only then removes the KMS bits.

Non-blocking: the CA goes under /var/kms via machine.files (needs a reboot), rotation talks about Secrets and HelmReleases and says the check can't show the key version, the Talm reason is softened, and the CNPG primary is picked by selector. On renew-self: when it fails, the plugin logs in again (pkg/vault/lease.go), so token_no_default_policy costs an extra login, not an outage. I wrote it that way.

IvanHunters took all four: the DRBD step is scoped to v1.14.0 to v1.14.4 with the v1.14.5 and Talm v0.36.0 notes, plus the Talm reason, KMS v1 "by default" and the fields condition.