Move public changelog registry ownership to the scrubber Lambda

[cotti]

Why

Phase 1 of elastic/docs-eng-team#688. The public registry.json is produced client-side against the private bucket and pass-through copied by the scrubber Lambda, which makes it a log of upload operations rather than a description of the public bucket. Every known consistency gap follows from that: ad-hoc uploads leave it stale (the 2026-07-22 incident), --skip-etag-check re-uploads can never reach a stale public manifest, deletions orphan entries, failed scrubs leave phantom entries, and the recorded ETag is documented as useless.

What

The scrubber Lambda becomes the sole producer of the public registry.json, computed by reconciling against the current state of the buckets — registry = f(state), never f(event):

• Events are triggers; state decides. The handler ignores the event type: each distinct key in a batch gets one object-level reconcile against the private bucket (present → scrub current content and copy; absent → delete the public copy), followed by a post-write HEAD validation that redoes the work if the source changed mid-flight (closes the older-read-writes-last race between overlapping invocations). Each distinct group then gets one registry reconcile from the public listing.
• Group reconcile lists the group's prefix (/-delimited so verbatim branches don't sweep nested pools, paginated), reuses entries whose recorded ETag still matches, recomputes the rest from the scrubbed public YAMLs (amends always recomputed — their target depends on the parent), and writes back with optimistic concurrency (If-Match/If-None-Match: *, bounded jittered retries on 412/409, batch-item failure on exhaustion). An empty group's manifest is conditionally deleted — absent ≠ empty for consumers. A manifest with a newer schema_version is reported and left untouched.
• The registry pass-through is retired in the same deploy: registry-key events only schedule a group reconcile, so client-authored JSON no longer reaches the public bucket uninspected (strengthens the single-writer invariant from Changelog: Bundle upload support and Private-Public Bundling SQS Lambda #3163).
• producer field on the manifest identifies the reconcile algorithm version; any mismatch — including every legacy pass-through manifest at cutover — forces a full metadata recompute and a write even when entries come out identical. ETags recorded are now the public object's, finally usable by CDN consumers.
• The Lambda's top-level handler logic moved to testable classes in Elastic.Changelog (ScrubberProcessor, RegistryReconciler, ChangelogContentScrubber); Program.cs is a thin adapter. Per-invocation reconcile metrics are emitted as CloudWatch EMF (the docs-eng-team#688 Phase 0 observability item that could only land with the reconciler). The FakeS3 test fake (stateful, MD5 ETags, real conditional PUT/DELETE semantics, delimiter+pagination, interleaving hooks) is harvested from Add changelog registry state discovery and reconciliation commands #3717 per the disposition table in the issue.

The guarantee is convergence, not atomicity: any successfully processed event for a group repairs all accumulated drift in that group. Consumers' listed-but-missing tolerance stays. The CLI upload path is untouched in this phase (Phase 3 retires the client-side refresh); rollback = revert the Lambda deploy.

Requires the IAM grants from elastic/docs-infra#360 (s3:GetObject/s3:ListBucket on the public bucket) at deploy time.

Part of elastic/docs-eng-team#688 (Phase 1).

🤖 Generated with Claude Code

[Mpdreamz]

This is overly defensive IMO, we use AOT trimming and source generated json contracts heavily. We should not fallback to handcrafted json emission.

These are emitted as cloudwatch metrics, should we push (some of) them as OTEL metrics too?

[Mpdreamz]

Request changes: drop per-product changelogs/<product>/registry.json

Pushing back on the core assumption in this stack — that we need (and should reconcile) a per-product / per-group changelogs/<product>/registry.json as a description of every changelog object under that prefix.

Why we don't need it

Release-note discovery is not "walk the registry and see what's there." The automated release discover moments (see elastic/docs-eng-team#698) always start from a list of PRs:

• Prestage → query for list of PRs
• GitHub release → already have a list of PRs
• Serverless release → query for list of PRs

Given that list, we can hit CloudFront and decide whether a checked-in changelog exists by whether the object returns 404. If it 404s, changelog bundle inlines a changelog for that PR into the bundle anyway.

The changelogs/ tree on S3 is only a mirror of explicitly checked-in changelogs in repositories. Those objects are not mandatory to exist at bundle time. A growing changelogs/<product>/registry.json (or changelogs/products/registry.json) would just accumulate forever for little consumer value — please remove that model and the reconcile/heal machinery built around it.

A lot of the code in this PR (group listing → entry-level reconcile → per-group manifest write/delete, producer/schema cutover heal, etc.) exists to keep that per-product registry correct. With the assumption dropped, that surface can shrink substantially.

What we do want instead

Keep (or introduce) two shallow folder-level registries on S3 for cache opt-out only:

• changelogs/registry.json
• bundles/registry.json (or bundle/registry.json — match existing key layout)

Shape:

{
  "<folder>": "<etag>"
}

Where <etag> is the ETag of the last-touched file under that folder. Caching layers can then bail out eagerly before diving into a per-folder / per-product bundle registry.

Please rework this PR around that thinner contract: scrubber remains the public-bucket writer for changelog objects, but it should maintain the shallow folder→etag map — not a full per-product object inventory.