diff --git a/config/exporters/azcopy.env.example b/config/exporters/azcopy.env.example index 281d4dd..56b4efb 100644 --- a/config/exporters/azcopy.env.example +++ b/config/exporters/azcopy.env.example @@ -1,6 +1,9 @@ # Install as /etc/backmaster/exporters/azcopy/nsys-postgres.env. # The destination must include the Azure Blob container and may include a prefix. AZCOPY_DESTINATION=https://REPLACE_ME.blob.core.windows.net/backups/nsys-postgresql +# Keep the catalogue/ prefix outside all Azure lifecycle archive/delete rules. +# Existing destinations need one catalogue-import or a forced backup with a +# unique name after every writer is upgraded; see docs/exporters/azcopy.md. # AzCopy logs and job plans default to the instance state directory: # /var/lib/backmaster/INSTANCE/azcopy/{logs,plans}. Override only with absolute, diff --git a/docs/exporters/azcopy.md b/docs/exporters/azcopy.md index 50bd3f6..d6c7f49 100644 --- a/docs/exporters/azcopy.md +++ b/docs/exporters/azcopy.md @@ -118,6 +118,9 @@ AZCOPY_DESTINATION/ │ └── BACKUP_NAME/ │ ├── payload/... + checksums.sha256, or backup.ARCHIVE │ └── manifest.json +├── catalogue/ +│ └── CREATED_EPOCH/ +│ └── BACKUP_NAME.json └── objects/ └── wal/... ``` @@ -125,14 +128,82 @@ AZCOPY_DESTINATION/ `publish` uploads every regular files-layout object or the configured archive to its exact destination blob. It verifies an archive against the SHA-256 in the manifest, skips `manifest.json` during the artifact pass, and uploads it -separately, last. Only a directory containing that manifest is a committed -catalogue entry. A retry overwrites an interrupted partial upload safely and -recommits the manifest. - -Catalogue operations use `azcopy list` to locate manifests, download only those -small files, and read their embedded creation epoch and backup name. This makes -freshness, daily serial allocation, and retention independent from blob listing -order and last-modified timestamps. +separately, after the payload. It then uploads an identical manifest to +`catalogue/CREATED_EPOCH/BACKUP_NAME.json` as the final publication step. +`publish` succeeds only when both manifests are durable. If catalogue publication +fails, the core keeps the `.ready` stage; the next run retries that same stage. + +Catalogue records are uploaded with `--block-blob-tier=Hot`. Keep `catalogue/` +outside every Azure lifecycle archival and deletion rule: only backup data under +`basebackups/` and recovery data under `objects/` should follow those rules. +Uploading as Hot does not override a lifecycle rule that archives the blob later. + +Freshness and health checks list catalogue keys, select the largest numeric +creation epoch, and download **only that record**. They also check that its +per-backup manifest still exists by listing names, without downloading it. +Directory-name order, host suffixes, blob last-modified times, and the storage +tiers of older backups do not affect selection. If the newest record cannot be +read or validated, the check fails; it does not silently fall back to an older one. + +Daily serial allocation lists per-backup manifest names for the requested date. +It includes legacy backups and never downloads their manifests. Retention uses +catalogue keys without downloading historical records or archived manifests. +All listing failures propagate as errors, rather than an empty backup store. + +Backup names identify one generation. A retry with the same sealed manifest is +allowed, but publishing a different manifest under an already committed name is +rejected. Use `daily-time` or `daily-serial` for multiple backups per day. With +`daily`, a forced second backup using the same name must use a different naming +mode or wait until the next day. + +## Migrating existing backups + +Upgrade the AzCopy exporter on **all nodes sharing the destination** before +resuming timers. Older exporters do not publish catalogue records; mixing old +and new writers can make freshness checks miss newly written backups. +Pause backup and health timers, wait for active runs to finish, and perform the +migration with one operator. Direct exporter commands, including import and +retention, do not acquire the core's Consul lock. + +An empty catalogue with existing per-backup manifests produces an explicit +migration error. It is not treated as an empty store and never triggers a scan +of historical manifest contents. Choose one of these bootstrap methods: + +1. Import the **newest verified completed backup** while its manifest is readable: + + ```bash + sudo -u postgres backmaster exporter production-postgres catalogue-import BACKUP_NAME + ``` + + Replace `BACKUP_NAME` with the exact remote directory name. Only that manifest + is downloaded. Import validates its name and integer creation epoch, then + publishes the Hot catalogue record. Importing the same generation is + idempotent. An archived manifest must be rehydrated before it can be imported; + other archived backups do not need rehydration. + +2. Create one fresh backup with a unique name: + + ```bash + sudo -u postgres backmaster run production-postgres --force + ``` + + Use `daily-time` or `daily-serial` in the instance configuration to avoid a + collision with an existing daily backup. Prepare the state directory and use + the actual service user as described in [Operations](../guides/operations.md#direct-cli-runs). + The command still takes the Consul lock. If a `.ready` stage exists, it resumes + that stage first; check its age and run again if a fresh backup is needed. + +After bootstrap, run `backmaster health production-postgres`, confirm the new +record is online, and resume the timers. Existing backups without catalogue +records remain untouched. Numeric retention reports their count and does not +count them toward `MINIMUM_REDUNDANCY`. Import older backups individually to +enroll them in retention, or manage them separately with Azure lifecycle rules. +Importing an old backup makes it eligible for deletion on the next retention run. + +Never seed the catalogue from an arbitrary older backup and assume it represents +the newest backup. Import all recent candidates if their order is uncertain, or +use the forced-backup method. A successfully indexed backup makes the catalogue +authoritative for freshness; legacy unindexed entries are not read automatically. ## Commands @@ -142,7 +213,8 @@ order and last-modified timestamps. | `exporter INSTANCE connectivitycheck` | Authenticates and lists the destination | | `exporter INSTANCE latest-epoch` | Prints the newest committed creation epoch; exits 3 if empty | | `exporter INSTANCE next-serial DATE` | Returns the next committed serial for the UTC date | -| `exporter INSTANCE publish STAGE` | Uploads a sealed stage and commits its manifest last | +| `exporter INSTANCE publish STAGE` | Uploads a sealed stage, its manifest, and finally its catalogue record | +| `exporter INSTANCE catalogue-import NAME` | Imports one existing readable manifest into the online catalogue | | `exporter INSTANCE retain` | Applies committed-backup and WAL retention | | `exporter INSTANCE healthcheck` | Validates committed-backup freshness | | `exporter INSTANCE put-file SOURCE KEY` | Uploads a recovery object below `objects/` | @@ -166,10 +238,17 @@ you need a direct CLI backup, prepare that directory as described in ## Retention -Base-backup retention sorts committed manifest data newest-first. It always +Base-backup retention sorts indexed creation epochs newest-first. It always preserves the newest `MINIMUM_REDUNDANCY` entries, then recursively removes additional entries older than `RETENTION_DAYS`. +It removes the catalogue record before deleting the backup directory. If the +record cannot be removed, it leaves the backup intact. If the directory removal +fails afterward, the run reports failure and the remaining data is unindexed; +inspect it before explicitly importing it again or completing its removal. +Manual deletion or external lifecycle deletion must also remove corresponding +catalogue records. Archival alone does not require any catalogue changes. + Despite its name, `MINIMUM_REDUNDANCY` does not configure Azure replication. It is the minimum number of committed Backmaster backups retained in this instance's destination. With `RETENTION_DAYS=14` and `MINIMUM_REDUNDANCY=2`, the diff --git a/docs/exporters/index.md b/docs/exporters/index.md index 0792cf1..81994d5 100644 --- a/docs/exporters/index.md +++ b/docs/exporters/index.md @@ -38,6 +38,13 @@ health checks, and retention must ignore payloads without a committed manifest. An exporter must return success from `publish` only after the manifest is durable. The core removes a ready stage only after that success. +An exporter may also maintain a separate online catalogue. The AzCopy exporter +publishes an epoch-keyed catalogue record after the per-backup manifest and +reports success only after both are durable. This allows historical backup +contents and manifests to move to Azure Archive without breaking freshness or +retention. See its [migration guide](azcopy.md#migrating-existing-backups) before +upgrading an existing destination; the rclone catalogue behavior is unchanged. + `publish` must accept both core stage layouts. A files stage contains `payload/`, `checksums.sha256`, and `manifest.json`. An archive stage contains the archive named by `.artifact.file` and `manifest.json`; its payload and diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md index 3510ef8..b597609 100644 --- a/docs/guides/troubleshooting.md +++ b/docs/guides/troubleshooting.md @@ -175,6 +175,25 @@ absolute, protected writable directories outside the backup stage and allowed by the unit's systemd sandbox. AzCopy files may contain storage names and operational metadata. +### Archived historical manifest blocks the latest-backup check + +Older AzCopy exporters download every historical manifest to determine freshness. +If a manifest moves to Azure Archive, the run can fail with `invalid remote +manifest` and a missing temporary file even when newer backups are readable. + +Upgrade every writer to the exporter with the separate online `catalogue/` +prefix. Follow [Migrating existing backups](../exporters/azcopy.md#migrating-existing-backups): +import the newest verified readable backup by name, or force a fresh backup with +a unique name. The updated exporter reports a migration error until this step +is complete; it does not guess a timestamp from directory ordering. Keep the +catalogue outside lifecycle archival and deletion rules. + +An unreadable newest catalogue record or failed listing is a storage error, +not an empty store. A `preserving ... unindexed backup(s)` retention warning +means legacy backups are left intact until individually imported or managed +separately. If catalogue publication fails, retain the local `.ready` stage and +rerun after fixing access; the core resumes publication automatically. + ## Disk space Check the stage filesystem: diff --git a/exporters/azcopy/exporter b/exporters/azcopy/exporter index 08b2b39..eee8736 100755 --- a/exporters/azcopy/exporter +++ b/exporters/azcopy/exporter @@ -89,9 +89,13 @@ object_url() { } azcopy_upload() { + local -a tier=() + # Catalogue records must be readable even when the account defaults to a + # colder tier. Lifecycle policies must also exclude catalogue/. + [[ "$2" != catalogue/* ]] || tier=(--block-blob-tier=Hot) azcopy copy "$1" "$(object_url "$2")" --from-to=LocalBlob \ --overwrite=true --check-length=true --put-md5 \ - --output-level=essential --log-level=ERROR /dev/null - name="$(jq -er '.backup_name' "$temporary/manifest.json")" || \ - die "invalid remote manifest: $path" - directory="${path%/manifest.json}" - [[ "$name" == "$directory" && "$name" =~ ^[A-Za-z0-9._-]+$ && "$name" != *..* ]] || \ - die "manifest name does not match its remote directory: $path" - jq -er '[.created_epoch, .backup_name] | @tsv' \ - "$temporary/manifest.json" - done < <(list_manifest_paths) + [[ "$path" =~ ^(0|[1-9][0-9]{0,14})/([^/]+)\.json$ ]] || \ + die "invalid catalogue path: $path" + epoch="${BASH_REMATCH[1]}"; name="${BASH_REMATCH[2]}" + validate_backup_name "$name" + rows+="${epoch}"$'\t'"${name}"$'\n' + done <<<"$paths" + [[ -n "$rows" ]] || return 0 + duplicates="$(printf '%s' "$rows" | cut -f2 | sort | uniq -d)" + [[ -z "$duplicates" ]] || die "conflicting catalogue records for: $duplicates" + printf '%s' "$rows" | LC_ALL=C sort -t$'\t' -k1,1nr -k2,2 } -latest_epoch() { - local latest - latest="$(manifest_rows | cut -f1 | sort -n | tail -1)" - [[ -n "$latest" ]] || exit 3 - printf '%s\n' "$latest" +download_manifest() { + local key="$1" destination="$2" + rm -f -- "$destination" || return 1 + azcopy_download "$key" "$destination" >/dev/null || \ + die "remote manifest download failed: $key (check access and archive tier)" + [[ -s "$destination" ]] || die "remote manifest download produced no file: $key" +} + +manifest_epoch() { + local file="$1" name="$2" + jq -ers --arg name "$name" ' + select(length == 1) | .[0] | + select(type == "object" and .backup_name == $name) | + .created_epoch | select(type == "number") | + select(. >= 0 and . <= 999999999999999 and . == floor) + ' "$file" || die "invalid manifest name or creation epoch: $name" } +latest_epoch() ( + local rows names epoch name temporary actual + rows="$(catalogue_rows)" || exit 1 + if [[ -z "$rows" ]]; then + names="$(manifest_names)" || exit 1 + [[ -z "$names" ]] || die "legacy backups need catalogue migration: import the newest verified backup with catalogue-import NAME, or run one backup with --force and a unique name" + exit 3 + fi + IFS=$'\t' read -r epoch name <<<"$rows" + names="$(manifest_names)" || exit 1 + grep -Fxq -- "$name" <<<"$names" || \ + die "catalogue references a missing backup manifest: $name" + temporary="$(mktemp -d)" || exit 1 + trap 'rm -rf -- "$temporary"' EXIT + download_manifest "catalogue/$epoch/$name.json" "$temporary/manifest.json" || exit 1 + actual="$(manifest_epoch "$temporary/manifest.json" "$name")" || exit 1 + [[ "$actual" == "$epoch" ]] || die "catalogue epoch mismatch: $name" + printf '%s\n' "$epoch" +) + next_serial() { - local day="$1" latest + local day="$1" latest names [[ "$day" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] || \ die "invalid serial date: $day" - latest="$(manifest_rows | cut -f2 | \ + names="$(manifest_names)" || return 1 + latest="$(printf '%s\n' "$names" | \ sed -nE "s/^${day}-([0-9]+)(-|$).*/\\1/p" | sort -n | tail -1)" - printf '%d\n' "$(( ${latest:-0} + 1 ))" + # Explicit decimal arithmetic handles serials 008 and 009 correctly. + [[ ${#latest} -le 15 ]] || die "serial out of range: $day" + printf '%d\n' "$(( 10#${latest:-0} + 1 ))" } -publish() { - local stage="$1" name source relative layout archive expected actual +# A backup name identifies one generation. Replacing it with a different epoch +# would leave an old catalogue record referring to the new payload. +check_catalogue_identity() { + local name="$1" epoch="$2" rows existing_epoch existing_name + rows="$(catalogue_rows)" || return 1 + while IFS=$'\t' read -r existing_epoch existing_name; do + [[ "$existing_name" == "$name" ]] || continue + [[ "$existing_epoch" == "$epoch" ]] || \ + die "backup name already committed: $name; use daily-time or daily-serial" + done <<<"$rows" +} + +catalogue_import() ( + local name="$1" temporary epoch + validate_backup_name "$name" + temporary="$(mktemp -d)" || exit 1 + trap 'rm -rf -- "$temporary"' EXIT + download_manifest "basebackups/$name/manifest.json" "$temporary/manifest.json" || exit 1 + epoch="$(manifest_epoch "$temporary/manifest.json" "$name")" || exit 1 + check_catalogue_identity "$name" "$epoch" || exit 1 + azcopy_upload "$temporary/manifest.json" "catalogue/$epoch/$name.json" || \ + die "catalogue import failed: $name" +) + +publish() ( + local stage="$1" name epoch source relative layout archive expected actual names temporary [[ -r "$stage/manifest.json" ]] || die "invalid stage: $stage" layout="$(jq -er '.artifact.layout // "files"' "$stage/manifest.json")" || \ die "invalid stage manifest: $stage" @@ -161,30 +242,70 @@ publish() { *) die "unsupported artifact layout: $layout" ;; esac name="$(jq -er '.backup_name' "$stage/manifest.json")" - [[ "$name" =~ ^[A-Za-z0-9._-]+$ && "$name" != *..* ]] || \ - die "invalid backup name" + validate_backup_name "$name" + epoch="$(manifest_epoch "$stage/manifest.json" "$name")" || exit 1 + check_catalogue_identity "$name" "$epoch" || exit 1 + names="$(manifest_names)" || exit 1 + if grep -Fxq -- "$name" <<<"$names"; then + temporary="$(mktemp -d)" || exit 1 + trap 'rm -rf -- "$temporary"' EXIT + download_manifest "basebackups/$name/manifest.json" "$temporary/manifest.json" || exit 1 + # Retries of the same sealed stage are allowed; other name collisions + # must not overwrite a committed backup (including a legacy backup). + jq -e -s 'length == 2 and .[0] == .[1]' \ + "$stage/manifest.json" "$temporary/manifest.json" >/dev/null || \ + die "backup name already committed with a different manifest: $name" + fi while IFS= read -r -d '' source; do relative="${source#"$stage"/}" [[ "$relative" != manifest.json ]] || continue - azcopy_upload "$source" "basebackups/$name/$relative" + azcopy_upload "$source" "basebackups/$name/$relative" || \ + die "backup upload failed: $name/$relative" done < <(find "$stage" -type f -print0 | sort -z) - azcopy_upload "$stage/manifest.json" "basebackups/$name/manifest.json" -} + azcopy_upload "$stage/manifest.json" "basebackups/$name/manifest.json" || \ + die "backup manifest upload failed: $name" + azcopy_upload "$stage/manifest.json" "catalogue/$epoch/$name.json" || \ + die "catalogue publication failed: $name; ready stage must be retried" +) retain() { - local cutoff rows keep epoch name wal_cutoff + local cutoff rows keep epoch name wal_cutoff names indexed unindexed if [[ "$RETENTION_DAYS" != unlimited ]]; then cutoff=$(( $(date +%s) - RETENTION_DAYS * 86400 )) - rows="$(manifest_rows | sort -nr)" + rows="$(catalogue_rows)" || return 1 + names="$(manifest_names)" || return 1 + # Stale records must not count toward minimum redundancy and cause + # deletion of the remaining real backups. Validate before any removal. + while IFS=$'\t' read -r epoch name; do + [[ -n "$epoch" ]] || continue + grep -Fxq -- "$name" <<<"$names" || \ + die "catalogue references a missing backup manifest: $name" + done <<<"$rows" + indexed="$(cut -f2 <<<"$rows")" + unindexed=0 + while IFS= read -r name; do + [[ -n "$name" ]] || continue + if ! grep -Fxq -- "$name" <<<"$indexed"; then + unindexed=$((unindexed + 1)) + fi + done <<<"$names" + if (( unindexed > 0 )); then + echo "azcopy exporter: preserving $unindexed unindexed backup(s); use catalogue-import NAME to include them in retention" >&2 + fi keep=0 while IFS=$'\t' read -r epoch name; do [[ -n "$epoch" ]] || continue keep=$((keep + 1)) if (( keep > MINIMUM_REDUNDANCY && epoch < cutoff )); then + # Remove discoverability first. If payload deletion then fails, + # it is preserved as an unindexed backup, never a false success. + azcopy remove "$(object_url "catalogue/$epoch/$name.json")" \ + --output-level=essential --log-level=ERROR >"$TEST_LOG" + if [[ -n "${TEST_UNREADABLE:-}" && "$key" == $TEST_UNREADABLE ]]; then + echo 'BlobArchived (simulated)' >&2 + exit 1 + fi + [[ -z "${TEST_MISSING_DOWNLOAD:-}" || "$key" != "$TEST_MISSING_DOWNLOAD" ]] || exit 0 + cp "$TEST_REMOTE/$key" "$destination" else key="$(key_from_url "$destination")" + if [[ -n "${TEST_UPLOAD_FAILURE:-}" && "$key" == $TEST_UPLOAD_FAILURE ]]; then + echo 'upload failure (simulated)' >&2 + exit 1 + fi + if [[ "$key" == catalogue/* ]]; then + [[ "$*" == *--block-blob-tier=Hot* ]] || exit 1 + fi mkdir -p "$(dirname "$TEST_REMOTE/$key")" cp "$source" "$TEST_REMOTE/$key" printf 'PUT %s\n' "$destination" >>"$TEST_LOG" @@ -80,13 +92,23 @@ case "$command" in list) key="$(key_from_url "$1")" root="$TEST_REMOTE/$key" + if [[ -n "${TEST_LIST_FAILURE:-}" && "$key" == "$TEST_LIST_FAILURE" ]]; then + # Include a valid-looking partial result: callers must discard it. + printf '1785683663/2026-08-02-006-axon.json; Content Length: 100\n' + echo 'listing failure (simulated)' >&2 + exit 1 + fi [[ -d "$root" ]] || exit 0 find "$root" -type f -printf '%P; Content Length: %s\n' | sort ;; remove) key="$(key_from_url "$1")" printf 'REMOVE %s %s\n' "$1" "$*" >>"$TEST_LOG" - if [[ "$key" == basebackups/* ]]; then + if [[ -n "${TEST_REMOVE_FAILURE:-}" && "$key" == $TEST_REMOVE_FAILURE ]]; then + echo 'removal failure (simulated)' >&2 + exit 1 + fi + if [[ "$key" == basebackups/* || "$key" == catalogue/* ]]; then rm -rf -- "$TEST_REMOTE/$key" fi ;; @@ -97,18 +119,43 @@ chmod +x "$temporary/bin/azcopy" run_exporter() { HOME=/var/lib/postgresql BACKMASTER_STATE_DIRECTORY="$temporary/state" \ - PATH="$temporary/bin:$PATH" TEST_REMOTE="$temporary/remote" \ + PATH="$temporary/bin:$PATH" TEST_REMOTE="${TEST_REMOTE_OVERRIDE:-$temporary/remote}" \ TEST_LOG="$temporary/log" TEST_STATE="$temporary/state" \ EXPORTER_CONFIG="$temporary/config.env" \ "$ROOT/exporters/azcopy/exporter" "$@" } +expect_failure() { + local expected_status="$1" message="$2" status + shift 2 + if "$@" >"$temporary/output" 2>"$temporary/error"; then + echo "unexpected success: $*" >&2 + exit 1 + else + status=$? + fi + [[ "$status" == "$expected_status" ]] || { + printf 'expected status %s, got %s: %s\n' "$expected_status" "$status" "$*" >&2 + cat "$temporary/error" >&2 + exit 1 + } + [[ -z "$message" ]] || grep -Fq -- "$message" "$temporary/error" "$temporary/output" || { + printf 'missing error %s: %s\n' "$message" "$*" >&2 + cat "$temporary/error" "$temporary/output" >&2 + exit 1 + } +} + run_exporter connectivitycheck [[ -d "$temporary/state/azcopy/logs" && \ -d "$temporary/state/azcopy/plans" ]] || { echo "AzCopy work directories were not created in instance state" >&2 exit 1 } +# Import the two original fixtures so the existing retention assertions cover +# indexed backups. Import never needs any other historical manifest. +run_exporter catalogue-import 2026-08-02-005-myelin >/dev/null +run_exporter catalogue-import 2026-07-20-004-axon >/dev/null run_exporter publish "$temporary/stage" >/dev/null backup="$temporary/remote/basebackups/2026-08-02-006-axon" [[ -f "$backup/payload/databases/one.dump" && \ @@ -121,10 +168,12 @@ backup="$temporary/remote/basebackups/2026-08-02-006-axon" exit 1 } last_put="$(grep '^PUT ' "$temporary/log" | tail -1)" -[[ "$last_put" == *'/basebackups/2026-08-02-006-axon/manifest.json?sig=test' ]] || { - echo "manifest was not uploaded last or SAS placement is invalid" >&2 +[[ "$last_put" == *'/catalogue/1785683663/2026-08-02-006-axon.json?sig=test' ]] || { + echo "catalogue was not uploaded last or SAS placement is invalid" >&2 exit 1 } +[[ "$(grep '^PUT ' "$temporary/log" | tail -2 | head -1)" == \ + *'/basebackups/2026-08-02-006-axon/manifest.json?sig=test' ]] grep '/payload/space%20name?sig=test' "$temporary/log" >/dev/null || { echo "payload path was not URL encoded" >&2 exit 1 @@ -184,4 +233,190 @@ if run_exporter retain >/dev/null 2>&1; then exit 1 fi +# Regression fixtures are independent of the original retention fixtures. +sed -i 's/^RETENTION_DAYS=invalid$/RETENTION_DAYS=1/' "$temporary/config.env" +export TEST_REMOTE_OVERRIDE="$temporary/regression" +mkdir -p "$TEST_REMOTE_OVERRIDE/basebackups/2026-08-06-axon" \ + "$TEST_REMOTE_OVERRIDE/basebackups/2026-09-21-009-myelin" +old_name=2026-08-06-axon +new_name=2026-09-21-009-myelin +new_epoch="$(date +%s)" +printf '{"backup_name":"%s","created_epoch":1}\n' "$old_name" \ + >"$TEST_REMOTE_OVERRIDE/basebackups/$old_name/manifest.json" +jq -n --arg name "$new_name" --argjson epoch "$new_epoch" \ + '{schema:2,backup_name:$name,created_epoch:$epoch}' \ + >"$TEST_REMOTE_OVERRIDE/basebackups/$new_name/manifest.json" +export TEST_UNREADABLE="basebackups/$old_name/*" +: >"$temporary/log" +expect_failure 1 'legacy backups need catalogue migration' run_exporter latest-epoch +[[ ! -s "$temporary/log" ]] # No legacy manifest was downloaded. +[[ "$(run_exporter next-serial 2026-09-21)" == 10 ]] +[[ ! -s "$temporary/log" ]] # Serial discovery only lists names, including legacy. +run_exporter catalogue-import "$new_name" >/dev/null +[[ "$(grep -c '^GET ' "$temporary/log")" == 1 ]] +if grep -Fq "$old_name" "$temporary/log"; then + echo "import downloaded an unrelated old manifest" >&2; exit 1 +fi + +# Even if all per-backup manifests are archived, only the newest catalogue +# record is downloaded by freshness and health checks. +export TEST_UNREADABLE='basebackups/*' +# A payload file named manifest.json is not itself a committed backup. +mkdir -p "$TEST_REMOTE_OVERRIDE/basebackups/$new_name/payload/nested" +printf '{}\n' >"$TEST_REMOTE_OVERRIDE/basebackups/$new_name/payload/nested/manifest.json" +: >"$temporary/log" +[[ "$(run_exporter latest-epoch)" == "$new_epoch" ]] +[[ "$(grep -c '^GET ' "$temporary/log")" == 1 ]] +grep -Fq "/catalogue/$new_epoch/$new_name.json?" "$temporary/log" +run_exporter healthcheck | grep -q '^OK:' +[[ "$(run_exporter next-serial 2026-09-21)" == 10 ]] +: >"$temporary/log" +run_exporter retain >"$temporary/output" 2>"$temporary/error" +grep -q 'preserving 1 unindexed backup' "$temporary/error" +[[ -f "$TEST_REMOTE_OVERRIDE/basebackups/$old_name/manifest.json" ]] +[[ ! -s "$temporary/log" ]] # No old download, and the sole indexed backup is kept. + +# List errors cannot turn into "empty", partial success, serial reuse, or deletes. +for prefix in catalogue basebackups; do + if [[ "$prefix" == catalogue ]]; then verb=latest-epoch; else verb=next-serial; fi + TEST_LIST_FAILURE="$prefix" expect_failure 1 'remote listing failed' \ + run_exporter "$verb" 2026-09-21 + : >"$temporary/log" + TEST_LIST_FAILURE="$prefix" expect_failure 1 'remote listing failed' run_exporter retain + [[ ! -s "$temporary/log" ]] +done +TEST_LIST_FAILURE=catalogue expect_failure 2 'unable to determine latest' run_exporter healthcheck + +# An unreadable/corrupt newest catalogue record is an error, never an older +# fallback or a false "no completed backup" result. +catalogue_key="catalogue/$new_epoch/$new_name.json" +TEST_UNREADABLE="$catalogue_key" expect_failure 1 'remote manifest download failed' \ + run_exporter latest-epoch +TEST_MISSING_DOWNLOAD="$catalogue_key" expect_failure 1 'download produced no file' \ + run_exporter latest-epoch +cp "$TEST_REMOTE_OVERRIDE/$catalogue_key" "$temporary/good-manifest" +printf 'broken JSON\n' >"$TEST_REMOTE_OVERRIDE/$catalogue_key" +expect_failure 1 'invalid manifest name or creation epoch' run_exporter latest-epoch +for bad_epoch in '"123"' -1 1.5 null; do + jq --argjson epoch "$bad_epoch" '.created_epoch=$epoch' "$temporary/good-manifest" \ + >"$TEST_REMOTE_OVERRIDE/$catalogue_key" + expect_failure 1 'invalid manifest name or creation epoch' run_exporter latest-epoch +done +jq '.created_epoch=2' "$temporary/good-manifest" >"$TEST_REMOTE_OVERRIDE/$catalogue_key" +expect_failure 1 'catalogue epoch mismatch' run_exporter latest-epoch +cp "$temporary/good-manifest" "$TEST_REMOTE_OVERRIDE/$catalogue_key" +mv "$TEST_REMOTE_OVERRIDE/basebackups/$new_name/manifest.json" "$temporary/missing-manifest" +expect_failure 1 'missing backup manifest' run_exporter latest-epoch +: >"$temporary/log" +expect_failure 1 'missing backup manifest' run_exporter retain +[[ ! -s "$temporary/log" ]] # Stale records cannot count toward redundancy. +mv "$temporary/missing-manifest" "$TEST_REMOTE_OVERRIDE/basebackups/$new_name/manifest.json" +mkdir -p "$TEST_REMOTE_OVERRIDE/catalogue/0" +cp "$temporary/good-manifest" "$TEST_REMOTE_OVERRIDE/catalogue/0/$new_name.json" +expect_failure 1 'conflicting catalogue records' run_exporter latest-epoch +expect_failure 1 'conflicting catalogue records' run_exporter retain +rm "$TEST_REMOTE_OVERRIDE/catalogue/0/$new_name.json" +expect_failure 1 'invalid backup name' run_exporter catalogue-import ../outside +expect_failure 1 'invalid backup name' run_exporter catalogue-import . + +# Empty stores are distinct from failed listings and legacy-only stores. +mkdir -p "$temporary/empty" +TEST_REMOTE_OVERRIDE="$temporary/empty" expect_failure 3 '' run_exporter latest-epoch +TEST_REMOTE_OVERRIDE="$temporary/empty" TEST_LIST_FAILURE=basebackups \ + expect_failure 1 'remote listing failed' run_exporter latest-epoch + +# Publication failure must leave the previous latest untouched. The same sealed +# stage can then retry after its base manifest exists but catalogue record does not. +mkdir -p "$temporary/retry-stage/payload" +printf 'backup\n' >"$temporary/retry-stage/payload/dump" +printf 'checksums\n' >"$temporary/retry-stage/checksums.sha256" +retry_epoch=$((new_epoch + 1)) +jq -n --argjson epoch "$retry_epoch" \ + '{schema:2,backup_name:"retry-backup",created_epoch:$epoch}' \ + >"$temporary/retry-stage/manifest.json" +TEST_UPLOAD_FAILURE='basebackups/retry-backup/payload/*' \ + expect_failure 1 'backup upload failed' run_exporter publish "$temporary/retry-stage" +[[ ! -e "$TEST_REMOTE_OVERRIDE/basebackups/retry-backup/manifest.json" ]] +TEST_UPLOAD_FAILURE='basebackups/retry-backup/manifest.json' \ + expect_failure 1 'backup manifest upload failed' run_exporter publish "$temporary/retry-stage" +[[ ! -e "$TEST_REMOTE_OVERRIDE/catalogue/$retry_epoch/retry-backup.json" ]] +TEST_UPLOAD_FAILURE='catalogue/*' expect_failure 1 'catalogue publication failed' \ + run_exporter publish "$temporary/retry-stage" +[[ "$(run_exporter latest-epoch)" == "$new_epoch" ]] +[[ -s "$TEST_REMOTE_OVERRIDE/basebackups/retry-backup/manifest.json" ]] +export TEST_UNREADABLE="basebackups/$old_name/*" +run_exporter publish "$temporary/retry-stage" >/dev/null +run_exporter publish "$temporary/retry-stage" >/dev/null # Idempotent retry. +[[ "$(run_exporter latest-epoch)" == "$retry_epoch" ]] +jq '.created_epoch+=1' "$temporary/retry-stage/manifest.json" >"$temporary/changed-manifest" +cp "$temporary/changed-manifest" "$temporary/retry-stage/manifest.json" +expect_failure 1 'backup name already committed' run_exporter publish "$temporary/retry-stage" + +# Imported old records need no body reads for retention; unindexed backups remain +# untouched. Keep the two newest indexed backups regardless of lexicographic name. +unset TEST_UNREADABLE +run_exporter catalogue-import "$old_name" >/dev/null +export TEST_UNREADABLE='basebackups/*' +: >"$temporary/log" +TEST_REMOVE_FAILURE='catalogue/1/*' expect_failure 1 'catalogue removal failed' run_exporter retain +[[ -s "$TEST_REMOTE_OVERRIDE/basebackups/$old_name/manifest.json" ]] +[[ -s "$TEST_REMOTE_OVERRIDE/catalogue/1/$old_name.json" ]] +run_exporter retain >/dev/null +[[ ! -e "$TEST_REMOTE_OVERRIDE/basebackups/$old_name" ]] +[[ ! -e "$TEST_REMOTE_OVERRIDE/catalogue/1/$old_name.json" ]] +[[ -s "$TEST_REMOTE_OVERRIDE/$catalogue_key" ]] +if grep -q '^GET ' "$temporary/log"; then + echo "retention downloaded a historical manifest" >&2; exit 1 +fi + +# Exercise the actual core: catalogue failure preserves .ready; the next run +# retries publication without preparing again, and a third run skips as fresh. +mkdir -p "$temporary/core/config/instances.d" "$temporary/core/install/drivers/fake" \ + "$temporary/core/install/exporters/azcopy" "$temporary/core/remote/basebackups/old-archive" +ln -s "$ROOT/exporters/azcopy/exporter" "$temporary/core/install/exporters/azcopy/exporter" +cat >"$temporary/core/install/drivers/fake/driver" <<'DRIVER' +#!/usr/bin/env bash +set -eu +[[ "$1" == prepare ]] || exit 64 +printf 'prepared\n' >>"$TEST_DRIVER_CALLS" +printf 'dump\n' >"$2/dump" +DRIVER +cat >"$temporary/bin/consul" <<'CONSUL' +#!/usr/bin/env bash +while (($#)); do + case "$1" in + service/*) shift; exec "$@" ;; + *) shift ;; + esac +done +exit 64 +CONSUL +chmod +x "$temporary/bin/consul" "$temporary/core/install/drivers/fake/driver" +cat >"$temporary/core/config/instances.d/test.env" <"$temporary/core/remote/basebackups/old-archive/manifest.json" +run_core() { + TEST_REMOTE="$temporary/core/remote" TEST_LOG="$temporary/log" \ + TEST_STATE="$temporary/core/staging/test" TEST_DRIVER_CALLS="$temporary/core/driver-calls" \ + PATH="$temporary/bin:$PATH" BACKMASTER_CONFIG_ROOT="$temporary/core/config" \ + BACKMASTER_INSTALL_ROOT="$temporary/core/install" "$ROOT/bin/backmaster" run test "$@" +} +export TEST_UNREADABLE=basebackups/old-archive/manifest.json +TEST_UPLOAD_FAILURE='catalogue/*' expect_failure 1 'catalogue publication failed' run_core --force +ready="$(find "$temporary/core/staging/test" -maxdepth 1 -name '*.ready' -type d -print -quit)" +[[ -n "$ready" && -s "$ready/manifest.json" && -s "$ready/payload/dump" ]] +run_core "" >/dev/null +[[ ! -e "$ready" && "$(wc -l <"$temporary/core/driver-calls")" == 1 ]] +run_core "" | grep -q 'result=skipped reason=fresh_backup_exists' +[[ "$(wc -l <"$temporary/core/driver-calls")" == 1 ]] + echo "AzCopy exporter tests passed"