Terraform to spin up a PDS - #2236
Draft
fastfadingviolets wants to merge 8 commits into
Draft
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
fastfadingviolets
added a commit
to hyphacoop/maple
that referenced
this pull request
Sep 9, 2026
…ecksums off (codeforboston#174) Review of codeforboston#2236 found the boot script fragile in five ways; all fixed here. - Secrets: fetch_secret had no retry and swallowed curl's stderr, so one Secret Manager blip left the PDS down until a human rebooted, and 403/404/ network all printed the same line. Now: --retry 5 --retry-all-errors, the HTTP status in the log (404 no version, 403 grant unapplied, 000 network), and a pds-startup-retry.timer that re-runs the script every 3 minutes until /run/pds/pds.env exists. secrets.sh no longer needs to reset the VM. Exit status is 0 only when every gap is "no version yet". - A flock around the whole run so the timer cannot race a slow first boot. - apt: DPkg::Lock::Timeout=600 — apt-daily/unattended-upgrades hold the lock on a fresh Debian 12 image and, under set -e, ended the run before the disk was mounted or the unit installed. - Data disk: format only on blkid exit 2 ("no filesystem"); exit 0 with an empty type or any other failure refuses. The old empty-string test could not tell "no filesystem" from "probe failed". - Blobs: AWS SDK v3 >= 3.729 (bundled in pds 0.4.5027) sends CRC32 flexible checksums by default and GCS's XML API rejects the signature, so uploadBlob 4xx'd while _health stayed green. AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION=when_required turn that off. - Container logs capped (10m x 3, as upstream's installer does) on the 10 GB boot disk; LOG_ENABLED=true so the PDS logs at all. - Comment correction: dockerd persists the container env on the boot disk; "never on the persistent disk" now says "never on the data disk". Rendered with dev values: bash -n and shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BuwSnBqJkHM4qWekjr6TjD
…over runbook A Terraform root at infra/gcp/dns for the org's public zone: signed from day one (ECDSA P-256), every record as served today, keyed "<name> <type>" and checked byte for byte, TTLs included, against ns-cloud-e1.googledomains.com on 2026-09-09. One zone, in digital-testimony-prod. README.md is the runbook: permissions, then the switchover. DNSSEC is on at the registrar today, so the order is build, diff, DNSSEC off and wait, swap name servers, re-sign with the new DS. Squarespace's name-server screen bundles the DNSSEC-off step with the NS change; the runbook keeps them apart. dns-zone-checks.yml runs fmt and validate on the root via a setup-terraform composite action with a root-agnostic provider cache key. The lock file records linux_amd64, darwin_arm64 and darwin_amd64 hashes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013yrzeyMUG1Y7X1SxSVNDQ3
Walking the live zone's NSEC3 chain turned up at least seventeen owner names against the five this file declared. Five of the missing ones are recovered here: the two SendGrid domain-authentication CNAMEs, and the mail.mapletestimony.org SPF/verification TXT with its two Firebase DKIM CNAMEs. Without them the name-server swap turns outbound password-reset and digest mail unsigned and out of SPF alignment. Four CNAMEs are still unaccounted for; their hashes did not reverse. The header comment now says so plainly instead of claiming the set is complete, and names where to read them off. Also: - README step 3 drove its diff loop off records.tf, so a name that was never copied out of the panel was never queried and the check passed clean right up to cutover. It now drives off the step 1 inventory, and carries an NSEC3 census command to count against. - README step 6 checked the ad flag against whatever resolver resolv.conf names, which usually does not validate. Pinned to 8.8.8.8. - outputs.tf: google_dns_keys returns retired key signing keys too, so one() over the whole list would fail at plan time after the first KSK roll, taking down the module and the step 6 command with it. Filter on is_active. - zone.tf: the comment justified algorithm 13 as what Squarespace signs with. The live DS is 60530 8 2, RSASHA256. Corrected, with why the algorithm change is safe across the cutover. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013AMn3BkyiY1PxhFxPA83eh
PDS on a Compute Engine VM with a persistent data disk, the firehose consumer on Cloud Run, two environments mirroring the Firebase projects, secrets in Secret Manager, one Terraform root per environment with IAM grants inside it so an editor's apply fails only the grants. PDS hostnames are sibling names under the parent domain, one delegated zone each: pds-dev for dev, pds reserved for prod. The parent zone is the Cloud DNS zone infra/gcp/dns owns in the prod project; the delegation is an NS record the PDS root writes there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQJGuzu2yoqRFPCvW9Dnq4
MAPLE's DID is minted outside the PDS by the identity tool, with the ops key in Cloud KMS (HSM secp256k1) signed by named people and never by a service account, and the recovery key offline. Section 6a records what the PDS actually requires at activation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQJGuzu2yoqRFPCvW9Dnq4
Phase 1 changes nothing. For the day the PDS opens native signup: the PDS rotation key moves into Cloud KMS through an upstream contribution, a fleet recovery key is held offline, and accounts minted before the move are migrated one scripted operation each. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQJGuzu2yoqRFPCvW9Dnq4
One root, one state per environment (envs/<env>.*), applied by people; no workflow applies it. Delivers the dev PDS at pds-dev.mapletestimony.org with pds.mapletestimony.org reserved for prod. - PDS on an e2-small VM with a separate data disk and snapshot policy; the startup script retries on a timer instead of rebooting, guards the disk before mounting, and renders pds.env into /run (tmpfs) so secrets never touch the data disk or its snapshots. - Blobs in a GCS bucket through the PDS's S3 blobstore; the HMAC key is minted out of band by scripts/secrets.sh so it never enters state. - Secret Manager holds the PDS secrets as resources only; secrets.sh adds the versions, and the VM picks them up within minutes without a reboot. - MAPLE's PLC ops key is a Cloud KMS HSM secp256k1 key; only the people in identity_signers (user:/group:, enforced) may sign, never a service account, so the VM cannot move the identity. - IAM lives in iam.tf with *_iam_member only, written so an editor's apply creates everything and fails only the grants; an owner's apply then plans exactly the grants. - The PDS hostname is apply-once (it lands in the DID document) and the delegated zone carries prevent_destroy to enforce it. - The delegation is an NS record the root writes into the parent zone infra/gcp/dns owns in digital-testimony-prod (parent_zone in the tfvars, the value of that root's `zone` output). Writing it needs dns.admin there and follows the same editor/owner rule as the grants; until the parent zone's name-server swap the record is silent. The dns README notes that the delegated names are expected only on the new servers. - scripts/bootstrap.sh enables the APIs and creates the state bucket, idempotently; the same bucket the dns root uses in prod, under a different prefix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQJGuzu2yoqRFPCvW9Dnq4
fmt and validate on every PR that touches infra/gcp; a dev `plan -refresh=false` for same-repo PRs, run in a terraform-plan GitHub environment whose key is the state-bucket reader named by ci_planner in envs/dev.tfvars. The plan is config-versus-state only, so it needs no access to the parent zone's project. A missing bucket skips the plan; a 403 fails it. No workflow applies. infra/gcp/dns is excluded from the path filters: it is its own root with its own workflow. The setup-terraform action already covers both roots. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQJGuzu2yoqRFPCvW9Dnq4
fastfadingviolets
force-pushed
the
atproto-pds-terraform
branch
from
September 9, 2026 19:41
cab947e to
ca5919c
Compare
The old README explained the rules the stack follows and why; those live in ADR 0001 and in the comments next to the code. The README is now the operator's runbook only: who needs which role, the apply steps and the done check, an inventory of what each file creates, and how to roll back a config change, a secret, the data, the state, or an environment. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQJGuzu2yoqRFPCvW9Dnq4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a short summary of the changes, and a reference to the original issue using
#and the issue number, like #1Checklist
firestore.indexes.json(Please do not only create indexes through the Firebase Web UI, even though the error messages may reccommend it - indexes created this way may be obliterated by subsequent deploys)Screenshots
Add some screenshots highlighting your changes.
Known issues
If you've run against limitations or caveats, include them here. Include follow-up issues as well.
Steps to test/reproduce
For each feature or bug fix, create a step by step list for how a reviewer can test it out. E.g.: