From 403bf490dfe65885e557345bd9ba99b5791ddd04 Mon Sep 17 00:00:00 2001 From: Andrew Miller Date: Mon, 14 Sep 2026 18:22:09 -0400 Subject: [PATCH] Add postgres-r2: durable Postgres in a CVM, disk- and node-loss drills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Postgres in a dstack CVM that treats the disk as a cache: wal-g ships every WAL segment to an S3-compatible bucket (Cloudflare R2), encrypted under a key the app derives from the KMS — nothing outside the enclave ever holds it. Two drills. Disk loss: wipe the volume, redeploy the same app as a standby, promote. Node loss: `phala cvms replicate` a standby onto another node (same app id -> same derived key, no connection to the primary), SIGKILL the primary mid-write, promote, count acked writes lost. Real numbers: 429/771 lost at archive_timeout=60s, 16/739 at 15s (the bound is the guarantee, the sample depends on where the kill lands). verify.sh checks the three claims live. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BVSS9AUWZVgcHbmLr4KvBt --- postgres-r2/README.md | 253 +++++++++++++++++++++++++++++++++ postgres-r2/docker-compose.yml | 138 ++++++++++++++++++ postgres-r2/verify.sh | 72 ++++++++++ 3 files changed, 463 insertions(+) create mode 100644 postgres-r2/README.md create mode 100644 postgres-r2/docker-compose.yml create mode 100755 postgres-r2/verify.sh diff --git a/postgres-r2/README.md b/postgres-r2/README.md new file mode 100644 index 0000000..542bbf5 --- /dev/null +++ b/postgres-r2/README.md @@ -0,0 +1,253 @@ +# Postgres that survives losing its disk, or its node + +Your CVM was just redeployed. Where did your database go? + +It's gone. A dstack app volume is cache-grade storage — the developer guide says so — and a +database sitting on one was never durable, only uninterrupted. This example stops treating the +disk as the database: Postgres ships every write-ahead log segment to an S3-compatible bucket +(Cloudflare R2 here) with [wal-g](https://github.com/wal-g/wal-g), and the disk becomes a cache +you can delete. + +Then it goes one step further. The same archive is enough to stand up a second copy of the +database on a **different node**, one that has never spoken to the first. Kill the first node +and promote the second. What that costs, in acknowledged writes, is measured below. + +The part that makes it a TEE example rather than an ops recipe: **the encryption key is derived +from the app's identity, not handed to it.** The bucket holds bytes nobody can read — not +Cloudflare, not you — and the only thing that turns them back into a database is attested code +running under the same app id. On a second node, that means the same app id on a second node. + +## Run it + +```bash +phala deploy -n pg-r2 -c docker-compose.yml \ + -e AWS_ENDPOINT=https://.r2.cloudflarestorage.com \ + -e AWS_ACCESS_KEY_ID=... -e AWS_SECRET_ACCESS_KEY=... \ + -e WALG_S3_PREFIX=s3:///pg + +./verify.sh pg-r2 +``` + +No `POSTGRES_PASSWORD` and no encryption key in that command — both are derived at boot from +`GetKey` on the guest agent socket. The only secrets you pass are the bucket credentials, which +belong to Cloudflare's side of the arrangement and cannot be derived. + +`PG_ROLE` defaults to `primary`. The other value is `standby`, and both drills below use it. + +## Drill 1: the disk + +The deployment is not the lesson. This is: + +```bash +# 1. leave a canary +psql "$DSN" -c "CREATE TABLE canary(t timestamptz); INSERT INTO canary VALUES (now())" + +# 2. wait for the segment to land (archive_timeout, 60s by default), then destroy the disk +phala ssh pg-r2 -- 'docker rm -f $(docker ps -q); docker volume rm _pgdata' + +# 3. bring the same app back as a standby, built from the archive alone +phala deploy --cvm-id pg-r2 -c docker-compose.yml -e PG_ROLE=standby \ + -e AWS_ENDPOINT=... -e AWS_ACCESS_KEY_ID=... -e AWS_SECRET_ACCESS_KEY=... \ + -e WALG_S3_PREFIX=s3:///pg + +# 4. it comes up read-only, replaying the archive; make it the primary again +psql "$DSN" -c "SELECT pg_promote()" + +# 5. the canary is still there +psql "$DSN" -c "SELECT * FROM canary" +``` + +Nothing was copied from the old node and no key was carried across. The app re-derived the key +because it is the same app. + +## Drill 2: the node + +Same idea, but the second copy is built on another node *before* anything goes wrong, and the +first node is killed the hard way while a client is writing to it. + +```bash +# 1. a standby on a second node, sharing the primary's app id +cat > standby.env <.r2.cloudflarestorage.com +AWS_ACCESS_KEY_ID=... +AWS_SECRET_ACCESS_KEY=... +WALG_S3_PREFIX=s3:///pg +EOF +phala cvms replicate pg-r2 --node-id -e standby.env +``` + +`replicate` creates a second instance of the *same app* (same app id, same compose hash) on +the node you name. It gets the bucket credentials from the env file because those cannot be +derived. It does not get the archive key. It boots, asks `GetKey` for `/pg-r2/walg/v1`, gets +the same 32 bytes the primary got, and `wal-g backup-fetch` reads the primary's encrypted +archive with a key nobody handed it. Then it sits in recovery pulling each new segment as it +lands. + +```bash +# 2. confirm it owes the primary nothing +psql "$STANDBY_DSN" -tAc "SHOW primary_conninfo" # empty +psql "$STANDBY_DSN" -tAc "SELECT count(*) FROM pg_stat_activity WHERE backend_type='walreceiver'" # 0 +psql "$STANDBY_DSN" -tAc "SELECT pg_is_in_recovery()" # t + +# 3. write through the gateway, recording every id the server acknowledged +yes 'INSERT INTO drill DEFAULT VALUES RETURNING id;' \ + | psql "$DSN" -tAq -v ON_ERROR_STOP=1 | grep -E '^[0-9]+$' > acked.txt & + +# 4. mid-write, SIGKILL the primary. No archive flush, no restart. +phala ssh pg-r2 -- 'docker rm -f $(docker ps -q)' + +# 5. promote the standby and count what did not make it +psql "$STANDBY_DSN" -c "SELECT pg_promote(true, 120)" +psql "$STANDBY_DSN" -tAc "SELECT id FROM drill ORDER BY id" > survived.txt +comm -23 <(sort -u acked.txt) <(sort -u survived.txt) | wc -l + +# 6. the old primary is a dead node now; treat it as one +phala cvms delete pg-r2 --force +``` + +The standby's DSN is the primary's app id on the other node's gateway; `phala cvms list` will +show you both instances. Step 3 goes through the gateway on purpose: an ack is only an ack if +the client received it, and the client is the only place that survives the primary dying. + +> [!WARNING] +> **A different app cannot read this archive.** `GetKey` derives from app identity, so deploying +> this compose as a *new* CVM produces a different key and wal-g fails with `corrupted chunk` — +> that is decryption failing, not a damaged object. Measured on two CVMs running this exact +> example: +> +> ``` +> pg-r2-example /pg-r2/walg/v1 -> 736c54f314defc07… +> pg-r2-restored /pg-r2/walg/v1 -> 160f640db26d2529… +> ``` +> +> `phala cvms replicate` is the supported way to get a second instance under the same app id. +> The alternatives are `phala deploy --custom-app-id --nonce `, or a supplied +> `WALG_LIBSODIUM_KEY` — and the moment you supply one, someone outside the enclave is holding +> the key you were trying not to have. Decide before your first backup, not after your last. + +## What `verify.sh` proves + +It is handed nothing. It re-derives the superuser password and logs in with it, then re-derives +the archive key — which it must, because without it wal-g cannot read the bucket at all. + +``` +== 1. keys are derived, not stored + superuser path, twice : b60a400df24a9af7… / b60a400df24a9af7… same + archive-key path : 736c54f314defc07… unrelated + logging in with the re-derived password: postgres + +== 2. the archive is current + segments archived : 6 (0 failed) + newest segment is : 72s old + waiting to archive : 0 .ready files + +== 3. the bucket holds ciphertext + a live segment on disk : 16d10600010000000000000200000000 + the object in the bucket: d830dc91fa382fe4f6fea96b3008cfb1 + the object after wal-g : 16d10600010000000000000600000000 + WAL magic is 16d1 — the bucket copy does not have it, the wal-g copy does +``` + +Every WAL page opens with the same magic for a given server version, so the magic is what +carries across segments; comparing whole heads would prove nothing. + +`verify.sh` reaches the CVM over `phala ssh`, which a replicated instance does not accept (it +is not issued your SSH key). Run it against the primary. Check the standby over its DSN, as in +Drill 2 step 2. + +## The numbers, with denominators + +Both drills below were run on real CVMs on two nodes, writing through the gateway at roughly +12–13 inserts/s, killing the primary with `docker rm -f` (SIGKILL: nothing flushes, nothing +restarts), then promoting a standby that had derived its own key. + +| `archive_timeout` | acknowledged | lost | as time, at the observed write rate | +|---|---:|---:|---| +| 60 s | 771 | **429** | about 33 s of writes | +| 15 s | 739 | **16** | about 1 s of writes | + +- **RPO — what a failure costs.** Whatever has not reached the archive yet, which is at most + `archive_timeout` of writes. That bound is the guarantee. The *sample* depends on where the + kill lands in the archive cycle: 429 used more than half the 60 s window, 16 used almost none + of the 15 s one, and a rerun would land somewhere else inside each bound. Quote the bound. + What the two rows do show is that the knob works: lowering the timeout narrows the window, at + the cost of more and smaller objects in the bucket. +- **RTO — how long the rebuild takes.** Restore time tracks the *compressed archive*, not the + logical database: **333 MB of incompressible rows came back in 29.3 s** (about 11 MB/s), while + 1.18 GB of repetitive rows took 15.2 s because its archive is nearly empty, and an empty + database hits a fixed-cost floor near 12 s. With a standby already caught up, promotion itself + is sub-second and the RTO is your detection time. Measure your own data before promising + anyone either number. +- **What the idle standby costs.** A caught-up standby polls the bucket for the next segment. + At Postgres's default 5 s retry that came to ~19 HEAD requests/s, 0 bytes retrieved, about + 1.7 M class B operations a day — roughly $0.42/day on R2, and the whole bill. The compose + sets `wal_retrieve_retry_interval=30s`, which cuts it about 6×. Promotion drains the restore + command first, so this interval does not change what a failover loses. + +## After the failover + +The promoted node is now a primary with one copy of the data and no standby. `archive_mode=on` +is inert while in recovery and arms the moment `pg_promote` runs, so it resumes shipping WAL to +the bucket on its own; without that a promoted standby would be a single copy with no archive, +and the next failure would lose everything. Check it: + +```bash +psql "$DSN" -tAc "SELECT archived_count, last_archived_time FROM pg_stat_archiver" +``` + +Then give it a standby of its own with another `phala cvms replicate`, and delete the dead +instance. A restart of the promoted container comes back as a primary: the standby staging +only runs on an empty data directory, and promotion removed `standby.signal`. + +## How it works + +- **Keys from `GetKey`.** `POST /GetKey` on `/var/run/dstack.sock` with a path returns 32 bytes of + hex — exactly wal-g's libsodium key size — plus a signature chain. Same app and path, same key + on every boot, on every node running the app, and after a total rebuild; different path, + unrelated key. The derivation path carries the domain separation. +- **It fails closed.** No socket, or a derivation that returns something unusable, and the + container exits. A database that would ship plaintext WAL into someone else's bucket should not + start at all. +- **The standby never talks to the primary.** `primary_conninfo` is set empty on purpose; + `restore_command = wal-g wal-fetch` is its only source. Recovery depends on the bucket and + the app's key, and on nothing that dies with the primary. +- **Archiving is configured on both roles.** So a promoted standby re-arms without a redeploy. +- **The base backup is pushed with the derived key in its environment.** WAL alone restores + nothing; it needs a base to replay onto. The entrypoint pushes one on first boot from the + process that holds the key. Do not push one by hand with `docker exec ... wal-g backup-push`: + that shell has no key, wal-g writes an unencrypted base, and the standby's fetch fails with + `corrupted chunk` because it is trying to decrypt plaintext. +- **wal-g is pinned by sha256** and verified before it runs. Pulling an unverified binary into a + measured enclave at boot gives away most of what the measurement was for. +- **TLS terminates inside the enclave**, so a client talks to Postgres rather than to the gateway. + Connect with `sslnegotiation=direct` (libpq 17+) — the gateway routes `5432s` by peeking the TLS + SNI, and libpq's default handshake gets dropped with a misleading "server closed the connection + unexpectedly". +- **`archive_timeout` is a startup argument.** It has the highest precedence, so `ALTER SYSTEM` + cannot change it; pass `-e ARCHIVE_TIMEOUT=15` and redeploy. + +## Not covered here + +**RPO 0.** An archive-only standby cannot have what the primary never archived. Closing the +window rather than bounding it means streaming replication between the nodes, which is a +different design: the standby then depends on a live connection to the primary, and the key +story has to cover the stream as well as the bucket. + +**A second failover.** After promotion the new primary archives on a new timeline. Building a +fresh standby from that archive, and failing over to it, was not exercised here. + +**Detection.** The drill promotes by hand. Nothing here decides that the primary is dead. + +**Archive rollback.** The bucket holds ciphertext, and a freshness check catches an archiver that +has stopped — but nothing here detects a storage provider that serves an *older* archive that is +internally consistent. wal-g will restore it and the result looks healthy at the wrong point in +history. Closing that needs a monotonic commitment to the archive head, kept somewhere the storage +provider does not control. It is the honest open problem in this design. + +## Requirements + +Two CVMs' worth of capacity on nodes your workspace can see (`phala nodes list`), each with +egress to your object store and the guest agent socket mounted (the compose does that). +Postgres 17, wal-g 3.0.9, any S3-compatible bucket. A libpq 17 client for the drills; on an +older distribution, `docker run --rm -i postgres:17 psql` works. diff --git a/postgres-r2/docker-compose.yml b/postgres-r2/docker-compose.yml new file mode 100644 index 0000000..f3717bc --- /dev/null +++ b/postgres-r2/docker-compose.yml @@ -0,0 +1,138 @@ +# Postgres in a CVM whose disk is disposable. +# +# The data directory is a cache. Durability lives in object storage: wal-g ships every WAL +# segment to an S3-compatible bucket (Cloudflare R2 here), encrypted under a key this app +# derives from the KMS. A node that loses its disk re-derives that key from its own identity +# and rebuilds from the archive — nothing outside the enclave ever held the key. +# +# Deploy: +# phala deploy -n pg-r2 -c docker-compose.yml \ +# -e AWS_ENDPOINT=https://.r2.cloudflarestorage.com \ +# -e AWS_ACCESS_KEY_ID=... -e AWS_SECRET_ACCESS_KEY=... \ +# -e WALG_S3_PREFIX=s3:///pg +# +# Note there is no POSTGRES_PASSWORD and no encryption key in that command. Both are derived. +# PG_ROLE defaults to primary. Deploy or replicate THE SAME app with -e PG_ROLE=standby to +# build a second copy from the archive alone — on a fresh disk here, or on another node via +# `phala cvms replicate` — which stays in recovery until you SELECT pg_promote(). A different +# app derives a different key and cannot read this archive; see the README. That is the +# intended property, not a limitation to work around casually. +services: + postgres: + image: postgres:17 + restart: always + stop_grace_period: 60s + ports: + - "5432:5432" + environment: + # Where the WAL goes. R2 speaks S3, so wal-g needs no special support — just the + # endpoint, path-style addressing, and a region that R2 ignores. + WALG_S3_PREFIX: "${WALG_S3_PREFIX:?set WALG_S3_PREFIX, e.g. s3://my-bucket/pg}" + AWS_ENDPOINT: "${AWS_ENDPOINT:?set AWS_ENDPOINT}" + AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID:?}" + AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY:?}" + AWS_S3_FORCE_PATH_STYLE: "true" + AWS_REGION: auto + WALG_COMPRESSION_METHOD: lz4 + # Force a segment even when writes are slow, so the recovery point stays bounded. + # This interval IS the RPO: a failover loses whatever has not been archived yet. + ARCHIVE_TIMEOUT: "${ARCHIVE_TIMEOUT:-60}" + # primary: initdb and archive. standby: build from the archive on a fresh volume (this + # node, or another node via `phala cvms replicate`) and stay in recovery until promoted. + PG_ROLE: "${PG_ROLE:-primary}" + POSTGRES_DB: app + volumes: + - pgdata:/var/lib/postgresql/data + # The guest agent. This is what makes the keys derivable rather than supplied. + - /var/run/dstack.sock:/var/run/dstack.sock:ro + entrypoint: + - bash + - -c + - | + set -euo pipefail + WALG_URL=https://github.com/wal-g/wal-g/releases/download/v3.0.9/wal-g-pg-20.04-amd64 + WALG_SHA=965a7147852435bc4b54f272799b63df826b54ef65cf7c4fc688c82b96c0c4eb + SOCK=/var/run/dstack.sock + fatal() { echo "[init] FATAL: $$*" >&2; exit 1; } + + # --- derive a key from this app's identity ------------------------------------- + # GetKey returns 32 bytes of hex for a given path: the same app and path give the + # same key on every boot and after a total rebuild, and a different path gives an + # unrelated key. So the derivation path carries the domain separation, and a + # restored node re-derives its own archive key instead of being issued one. + derive() { + curl -sf --unix-socket "$$SOCK" -X POST -H 'Content-Type: application/json' \ + -d "{\"path\":\"$$1\",\"purpose\":\"$$2\"}" http://dstack/GetKey \ + | sed -n 's/.*"key":"\([0-9a-f]\{64\}\).*/\1/p' + } + + apt-get update -qq + apt-get install -y -qq --no-install-recommends curl ca-certificates >/dev/null + + [ -S "$$SOCK" ] || fatal "no dstack socket at $$SOCK — this example derives its keys from it" + WALG_LIBSODIUM_KEY=$$(derive /pg-r2/walg/v1 walg) + POSTGRES_PASSWORD=$$(derive /pg-r2/superuser/v1 password) + export WALG_LIBSODIUM_KEY POSTGRES_PASSWORD + # Fail closed. A database that would ship plaintext WAL to someone else's bucket + # should not start at all — this is the one failure that cannot be a warning. + [ $${#WALG_LIBSODIUM_KEY} -eq 64 ] || fatal "key derivation failed — refusing to ship unencrypted WAL" + [ $${#POSTGRES_PASSWORD} -eq 64 ] || fatal "password derivation failed" + echo "[init] keys derived from the KMS; nothing outside this enclave has held them" + + # --- wal-g, pinned and verified -------------------------------------------------- + curl -fsSL -o /usr/local/bin/wal-g "$$WALG_URL" + echo "$$WALG_SHA /usr/local/bin/wal-g" | sha256sum -c - >/dev/null \ + || fatal "wal-g checksum mismatch — refusing to run an unverified binary in a TEE" + chmod +x /usr/local/bin/wal-g + + # --- TLS so the connection terminates inside the enclave, not at the gateway ----- + mkdir -p /etc/pgtls + openssl req -x509 -newkey rsa:2048 -nodes -days 90 -subj "/CN=pg-r2" \ + -keyout /etc/pgtls/server.key -out /etc/pgtls/server.crt 2>/dev/null + chown postgres:postgres /etc/pgtls/server.* + chmod 600 /etc/pgtls/server.key + + # --- a standby builds from the archive and owes the primary nothing -------------- + # backup-fetch + standby.signal (not recovery.signal): it stays in recovery, + # replaying each new segment as it lands, until someone runs pg_promote(). No + # primary_conninfo — the bucket and this app's derived key are its only inputs, so a + # standby on another node survives the primary's node dying with it. The 30s retry + # keeps a caught-up standby from polling the bucket ~19 HEAD/s at the 5s default. + if [ "$${PG_ROLE}" = standby ] && [ ! -s "$$PGDATA/PG_VERSION" ]; then + echo "[init] building standby from $$WALG_S3_PREFIX — no connection to the primary" + mkdir -p "$$PGDATA"; chown postgres:postgres "$$PGDATA"; chmod 700 "$$PGDATA" + gosu postgres wal-g backup-fetch "$$PGDATA" LATEST + gosu postgres touch "$$PGDATA/standby.signal" + { echo "restore_command = 'wal-g wal-fetch %f %p'"; + echo "primary_conninfo = ''"; + echo "wal_retrieve_retry_interval = '30s'"; } \ + | gosu postgres tee -a "$$PGDATA/postgresql.auto.conf" >/dev/null + fi + + # --- take a base backup once the server is up ------------------------------------ + # WAL alone restores nothing; it needs a base to replay onto. Loudly, because a + # silent archiving failure is invisible for exactly as long as it takes to matter. + # Primary only: a standby is in recovery and cannot push one; after you promote it, + # push a fresh base so its own standby has something to fetch. + if [ "$${PG_ROLE}" != standby ]; then + ( + for _ in $$(seq 1 60); do pg_isready -q -U postgres -d postgres && break; sleep 2; done + if gosu postgres wal-g backup-list 2>/dev/null | grep -q base_; then + echo "[init] base backup already present" + elif gosu postgres wal-g backup-push "$$PGDATA"; then + echo "[init] initial base backup pushed" + else + echo "[init] ERROR: backup-push failed — there is nothing to restore onto" >&2 + fi + ) & + fi + + exec docker-entrypoint.sh postgres \ + -c listen_addresses='*' \ + -c ssl=on -c ssl_cert_file=/etc/pgtls/server.crt -c ssl_key_file=/etc/pgtls/server.key \ + -c wal_level=replica \ + -c archive_mode=on \ + -c "archive_command=wal-g wal-push %p" \ + -c "archive_timeout=$${ARCHIVE_TIMEOUT}" +volumes: + pgdata: diff --git a/postgres-r2/verify.sh b/postgres-r2/verify.sh new file mode 100755 index 0000000..b5f3466 --- /dev/null +++ b/postgres-r2/verify.sh @@ -0,0 +1,72 @@ +#!/usr/bin/env bash +# Check the three claims this example makes, against a running CVM. +# +# ./verify.sh pg-r2 +# +# 1. The keys are derived, not stored — this script re-derives the superuser password from +# the KMS and logs in with it. Nothing was handed to it. +# 2. The archive is current — how many segments, and how stale the newest one is. +# 3. The bucket holds ciphertext — the archived object is compared against a real WAL +# segment on disk, and against the same object after wal-g decrypts it. +# +# Needs the phala CLI and a CVM deployed from docker-compose.yml here. +set -euo pipefail +CVM=${1:?usage: verify.sh } + +INNER=$(cat <<'EOF' +set -euo pipefail +sock=/var/run/dstack.sock +derive() { + curl -sf --unix-socket "$sock" -X POST -H 'Content-Type: application/json' \ + -d "{\"path\":\"$1\",\"purpose\":\"$2\"}" http://dstack/GetKey \ + | sed -n 's/.*"key":"\([0-9a-f]\{64\}\).*/\1/p' +} + +echo "== 1. keys are derived, not stored" +a=$(derive /pg-r2/superuser/v1 password) +b=$(derive /pg-r2/superuser/v1 password) +c=$(derive /pg-r2/walg/v1 walg) +[ -n "$a" ] || { echo " derivation failed"; exit 1; } +echo " superuser path, twice : ${a:0:16}… / ${b:0:16}… $([ "$a" = "$b" ] && echo same || echo DIFFERENT)" +echo " archive-key path : ${c:0:16}… $([ "$a" = "$c" ] && echo SAME-AS-PASSWORD || echo unrelated)" +export PGPASSWORD="$a" +Q() { psql "postgresql://postgres@127.0.0.1:5432/app?sslmode=require" -tAqc "$1"; } +echo " logging in with the re-derived password: $(Q 'SELECT current_user')" + +echo +echo "== 2. the archive is current" +Q "SELECT ' segments archived : '||archived_count||' ('||failed_count||' failed)' FROM pg_stat_archiver" +Q "SELECT ' newest segment is : '||coalesce(round(extract(epoch FROM now()-last_archived_time))||'s old','nothing archived yet') FROM pg_stat_archiver" +Q "SELECT ' waiting to archive : '||count(*)||' .ready files' FROM pg_ls_archive_statusdir() WHERE name LIKE '%.ready'" + +echo +echo "== 3. the bucket holds ciphertext" +# This shell was handed nothing, so wal-g cannot read the archive until the key is +# re-derived — which is the claim, demonstrated by needing to do it. +export WALG_LIBSODIUM_KEY="$c" +seg=$(Q "SELECT substr(last_archived_wal,1,24) FROM pg_stat_archiver") +if [ -z "$seg" ]; then echo " nothing archived yet — write something and switch WAL first"; exit 0; fi +# The listing also holds a backup-label object for this segment, and it sorts first. +# Match the segment itself, whatever compression suffix it carries. +obj=$(gosu postgres wal-g st ls wal_005/ | awk -v s="$seg" '$NF ~ "^"s"\\.[a-z0-9]+$" {print $NF; exit}') +[ -n "$obj" ] || { echo " $seg not found in the bucket listing"; exit 1; } +# Written to a file rather than piped: head closing the pipe would SIGPIPE wal-g, and +# under pipefail that reads as a failure when nothing actually went wrong. +rm -f /tmp/stored.bin +gosu postgres wal-g st cat "wal_005/$obj" > /tmp/stored.bin +stored=$(head -c 16 /tmp/stored.bin | od -An -tx1 | tr -d " \n") +rm -f /tmp/seg && gosu postgres wal-g wal-fetch "$seg" /tmp/seg >/dev/null +decrypted=$( head -c 16 /tmp/seg | od -An -tx1 | tr -d " \n") +local_seg=$(ls "$PGDATA/pg_wal" | grep -E '^[0-9A-F]{24}$' | head -n1) +onwal=$( head -c 16 "$PGDATA/pg_wal/$local_seg" | od -An -tx1 | tr -d " \n") +magic=${onwal:0:4} +echo " segment : $seg" +echo " a live segment on disk : $onwal" +echo " the object in the bucket: $stored" +echo " the object after wal-g : $decrypted" +echo " WAL magic is $magic — the bucket copy $([ "${stored:0:4}" = "$magic" ] && echo 'HAS IT (not encrypted!)' || echo 'does not have it'), the wal-g copy $([ "${decrypted:0:4}" = "$magic" ] && echo does || echo 'does NOT')" +EOF +) + +B=$(printf '%s' "$INNER" | base64 -w0) +phala ssh "$CVM" -- "docker exec \$(docker ps --format '{{.Names}}' | head -n1) bash -c 'echo $B | base64 -d | bash'"