Repository navigation
Sync upstream runtime, release, and Helm updates - #103
Merged
Merged
Conversation
…#3110) Workers are always unprivileged now, so delegation never fails to apply. Fail at startup if it can't rather than quietly running actors without their cgroups, which drops their cpu limits and checkpoint cache isolation, and drop the flag both workers threaded through. <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) eliminates dead code, reduces unnecessary complexity for implementing PRs like agent-substrate#3106 (comment) Fixes # <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: AI assisted, I reviewed and tested it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated --------- Co-authored-by: Davanum Srinivas <davanum@gmail.com>
…ubstrate#3105) <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Related to agent-substrate#1378 Pointedly punts code changes to avoid the churn at this time, just docs and agent instructions to drive consistency for future PRs. That can be a follow-up. It's trivial, but there are lot of files to patch and some of it will naturally land from other PRs once agents.md is updated. For prior art see firecracker's docs. no `micro-VM`, only `microVM`, this also matches our github label. I made a stylistic choice to use `MicroVM` to start sentences. `microvm` (all lowercase) remains preferred for API keys and references to API keys. Mainly this eliminates `micro-VM`. <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: Yes, human reviewed. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated
…-substrate#2991) For agent-substrate#1399. > [!IMPORTANT] > Recreate every existing Substrate PostgreSQL database after this merges, including dev databases built from `main`. ateapi can't upgrade a database created before the squash. > > `verify-migrations` fails on this PR because the squash changes migrations already on `main`. It isn't a required check yet, so it doesn't block the merge. After this merges, a maintainer should add it to the required checks in the `main` ruleset. New clusters run every migration at startup. Squashing `000001` to `000003` into `000001_initial.sql` cuts that to one. I applied the three original files and the squashed file to empty databases, and their `pg_dump --schema-only` output matched. Nothing detects an old database. A v0.6.0 database still starts, because its schema matches and `schema_migrations` already records versions 1 to 3. Goose would then treat the next `000002` and `000003` as applied and skip them. A v0.2.0 or v0.3.0 database lacks the access policy tables and also starts, because the store accepts a ledger that is ahead of the binary. `hack/verify/postgresql-migrations.sh` now compares against the merge base with the target branch, instead of the latest v1+ tag. Clusters built from `main` also record migration versions, so an edit to a merged but unreleased migration was skipped there too. The check moves out of `run-tests` into its own `verify-migrations` job, so a failure no longer stops the unit tests and verifiers. The job runs on pull requests and merge groups, not on pushes to `main`. `hack/verify-all.sh` runs the script without a base ref, which checks only the file rules. I can move the migration check to a follow-up PR if that's easier to review. If the squash merges first, neither PR fails `verify-migrations`. AI disclosure: Claude Code made the changes and this description.
agent-substrate#3155) Flake fix. `TestKillMatching` reads `/proc/<pid>/cmdline` right after `Start`. `Start` returns when the exec closes the child's status pipe, and the kernel does that before it sets up the new argv, so the file can still read empty and `killMatching` finds nothing: `kill_test.go:41: killed 0, want 1` ([2026-10-19](https://github.com/agent-substrate/substrate/actions/runs/39340595841/job/118360866503), [2026-10-23](https://github.com/agent-substrate/substrate/actions/runs/39956725255)). Poll for the command line before calling `killMatching`. Pinned to one CPU on Linux 6.8 (`taskset -c 0 go test -count=3000`), the old test failed 4 of 3000 runs with `killed 0`, the fixed one 0 of 3000.
agent-substrate#3148) Why is this needed? * Bugfix: After an actor's generation N has crashed, atelet may still upload it's snapshot from generation N-1, ateapi now will accept this snapshot report * updated the `actorOnNode` check to only check the assigned node. Today, each actor's local snapshots are cleared during Detach, we don't have any actors with old snapshots on a different node. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated
agent-substrate#3015) When `atelet` restarts (or loses the `CheckpointWorkload` response) after `ateworker` has already finished writing the snapshot into `checkpoint-state/` and torn down the sandbox/VM, `atelet` has not yet moved `checkpoint-state/` into `local-checkpoints/<uuid>/` or recorded the snapshot in `snapshottracker`. When the control plane retries `Checkpoint`, the replacement `atelet` calls `ateworker.CheckpointWorkload` again against a workload that no longer exists (`runsc checkpoint: exit status 128`). This PR records an atomic completion marker (`checkpoint-state/.checkpoint-done.json`) at the end of `CheckpointWorkload` in both `ateworker-gvisor` and `ateworker-microvm`. If a retry arrives when the workload is already unhosted (`lookupActor(actorUID) == nil`), `ReuseCheckpoint` verifies that the marker's fidelity matches and all listed `SnapshotFiles` / `DataSnapshotFiles` are still present in `checkpoint-state/`, returning the cached response immediately instead of re-running `runsc checkpoint` or cloud-hypervisor snapshot. Once `atelet` moves the snapshot files into `local-checkpoints/<uuid>/` or resets the actor directory via `resetActorDirs`, the marker is invalidated/removed automatically. Part of agent-substrate#2836 - [x] Tests pass
…ead of 300 (agent-substrate#3141) size10's `max_connections = 300` limits how many ate-api-server replicas can run with a useful pool. Each replica holds its read/write pool plus 6 owner, watch and readiness connections, so eight replicas fit only about 30 pool connections each. In the cold resume benchmark on the fake data plane (8 replicas at 8 CPUs, size10 Postgres), pool size set the ceiling: | Pool per replica | Connections for 8 replicas | Ceiling (cycles/s) | |---|---|---| | 30 | 288 | about 6500 | | 55 | 488 | about 7350 | | 80 | 688 | about 7420 | AI use: AI-assisted (Claude Code) for the benchmark runs, the analysis and this change. - [x] Tests pass - [ ] Docs updated
Amends the release workflow to include a build and publish step for the system artifacts required for consumers. Part of agent-substrate#1871 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ly (agent-substrate#3177) The shipped gVisor config pinned the 2026-10-17 nightly for google/gvisor@b79256080dec, which saves and restores multiple filesystem checkpoint bundles. Release 20261019.0 is the first tagged release that contains it, so the config can point at a release again. **Change.** `manifests/ate-install/sandboxconfig-gvisor.yaml` replaces `v4` with `v5`, release 20261019.0 for amd64 and arm64, as `defaultVersion`, the way the previous bumps did. `docs/api-guide.md` mirrors it. The sha256 values come from the downloaded tarballs, which match the published `.sha512` files. **Breaking change.** A MEMORY snapshot taken under `v4` cold boots on its next resume: it keeps its files and loses its memory state. Recut golden snapshots after upgrading. Please add the `breaking-change` label. **Tests.** `go test ./api/infra/k8s/ate/v1/` applies the edited manifest against the CRD and the admission policy; `./cmd/ate-setup/...` and `./cmd/atelet/` pass. CI's e2e lanes install this manifest and boot, suspend and restore gVisor actors on it. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated
…-substrate#3152) <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Upgrading to current Ubuntu in limactl (agent-substrate#3149) on macOS led to emulation errors with rosetta. Using qemu is super slow. We should just cross-compile, pretty much always, that's a key benefit of buildx and multi-stage builds. It's WAY faster. Compiling under emulation is major kludge. <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: Yes, I let it draft this. Then hand edited, bash and dockerfiles I'm quite familiar with :-) <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated
A WorkerPool name can be up to 253 characters, but `ate-controller` puts it in the `ate.dev/worker-pool` label on the worker pods and the pool's NetworkPolicy. Label values are capped at 63 characters. A longer name is accepted, and then every reconcile of the pool fails, with the error only in the controller log. The CRD now rejects names longer than 63 characters at create time. Came up while reviewing agent-substrate#3130. AI use: Claude Code wrote the change, the test rows and this description. I reviewed them.
… actor on its worker (agent-substrate#3106) On some nodes, one gVisor actor using too much memory takes down every actor on its worker, not just itself. See agent-substrate#3092 for details. We can't turn that off from inside the pod, so this makes sure the kernel never OOM-kills inside the worker. Instead: - the actor's cgroup gets `memory.high` at its memory limit instead of `memory.max`, so going over the limit slows the actor down instead of killing it - the worker watches for actors stuck over their limit and kills them itself with `cgroup.kill`, which systemd ignores - the actor shows up as crashed with "killed for exceeding its memory limit" "Stuck" means the actor spent at least a quarter of the last 2 seconds stalled on memory (from `memory.pressure`) while hitting its own limit. We also kill right away if it gets more than 16MiB (or 1/16 of its limit, if smaller) over, since a throttled actor keeps creeping up slowly. Just sampling memory usage didn't work, the kernel keeps pulling a throttled actor back just under the limit between allocations. I searched for prior art like meta's oomd, systemd-oomd, and lmkd. Those projects have a slightly different goal/tuning with machine-wide oom, so we borrow some ideas but we have tighter time thresholds. Part of agent-substrate#3092. The other half of that issue, the whole worker pod running out of memory, is separate. This is pre-factored into the shared cgroup package, microVM doesn't currently limit cgroups outside of the microVM, I will sequence that change next, keeping this PR scoped to the mechanism, and the critical gVisor update. AI use: AI-assisted, I reviewed and tested it. Tested on GKE 1.37.1 (runc 1.5.2, systemd 254): an actor allocating 400MiB against a 256MiB limit was killed alone about a second after hitting its limit, and its neighbour and the worker kept running. Without this change the same thing restarts the whole worker. - [x] Tests pass - [x] Docs updated
…mages pinned by digest (agent-substrate#3181) Envoy v1.39.5 drops the compromised eMudhra intermediate CAs, and so do the prometheus, otel-collector, cloud-sql-proxy and kata releases below. The CSI NFS proxy pulled the floating `envoyproxy/envoy:v1.34-latest` tag and the Cloud SQL proxy an unpinned one. The kind overlay images and the kata assets were several releases behind. **Change.** One commit per area. - Envoy dataplane image v1.39.5, plus the egress TLS e2e helper and the nighthawk driver that pin the same digest. - CSI NFS proxy and the kind hostpath testing proxy on the dataplane's Envoy digest, cloud-sql-proxy 2.26.3 by digest. - kind overlay: rustfs 1.0.3, aws-cli 2.38.5, prometheus v3.15.1, otel-collector-contrib 0.163.2. - kata assets 4.3.1: 6.18 guest kernel, Ubuntu 26.04 rootfs, virtiofsd still 1.14.0, cloud-hypervisor still v53.0. The micro-VM SandboxConfig becomes `v2` with digests from `assemble.sh` on amd64 and arm64 hosts. The kata agent protos are unchanged since 3.31.0. **Breaking change.** A micro-VM actor whose MEMORY snapshot, its own or its template's golden, was taken on the kata 4.1.0 assets restores at ROOTFS fidelity on its next resume: a fresh guest with its rootfs writes and volumes, no memory state. Recreate templates whose golden you want warm again. Please add the `breaking-change` label. **Tests.** `go test ./api/infra/k8s/ate/v1/ ./cmd/ate-setup/... ./cmd/atelet/` and shellcheck pass. All 19 e2e suites pass under the micro-VM class on an amd64 KVM host with the new assets. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated
…ntimes.json (agent-substrate#3143) Fixes agent-substrate#3139 On microVM, `workerSandboxSyncer.versionAttrs` only probed `/proc/cpuinfo` and wrote `sandbox-runtimes.json` without checking whether a version's assets could be fetched and verified. A `SandboxConfig` rollout with a broken asset URL or SHA256 on `defaultVersion` overwrote `sandbox-runtimes.json` right away, and workers advertised a `defaultRuntime` that failed at `Run` or `Restore`. Stage each microVM version's assets through `ensureSandboxAssets` into a scratch directory in `buildFile` (with `probeFetchTimeout` raised to 10m) and record verified bundle hashes in `workerSandboxSyncer.microvmAssets`. Always stage the default version (a cache hit once prewarmed) so promoting an evicted version to `defaultVersion` verifies it before unpinning the previous default. Skip `ensureMicroVMAssets` for restorable versions already in `microvmAssets`; the memo is per `atelet` process, so a restart re-checks each version (re-downloading any restorable version evicted before the restart). A failed `defaultVersion` fetch or SHA256 check fails `buildFile` before `s.write` runs, while a failed restorable version is left out with `errIncomplete` and retried. As with gVisor, a SHA256 mismatch is not cached or memoized, so retries re-download until the `SandboxConfig` is fixed. Update `docs/sandbox-asset-cache.md` and the prewarmer comments to match. AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated
…ent-substrate#3064) ### Summary Follow-up to @BenTheElder's review comment on agent-substrate#2746 (agent-substrate#2746 (comment)): update the `--snapshot-cache-dir` flag help text in `cmd/atelet/snapshotcache.go` to reflect that cache hits fall back to copies when the per-actor restore directories (`/var/lib/ate/snapshots/<uid>/restore-state`) are on another mount.
…substrate#3089) ## What this PR does and why we need it ? Following agent-substrate#2977 (`5d5dd7ae`), `atenet-router` no longer exposes plaintext HTTP on `:80` and requires SPIFFE mTLS on `:443` (`https://atenet-router.ate-system.svc`), verifying the server leaf against `/run/servicedns-ca/ca.crt` (SAN `atenet-router.ate-system.svc`) and presenting `/run/podidentity.podcert.ate.dev/credential-bundle.pem`. This PR updates all benchmark data-plane clients to dial `atenet-router` over SPIFFE mTLS on `:443`: - **`internal/benchmarking/boomer/boomerutil`**: - Add `DefaultRouterURL = "https://atenet-router.ate-system.svc"` and `RouterTLSConfig()`, cached with `sync.OnceValues` and returning a `Clone()` configured with `RootCAs` from `credbundle.ParsePool("/run/servicedns-ca/ca.crt")`, `GetClientCertificate` from `credbundle.ClientLoader("/run/podidentity.podcert.ate.dev/credential-bundle.pem")`, and `ServerName: "atenet-router.ate-system.svc"`. - Add unit tests in `client_test.go` covering mTLS handshake success, mismatched `ServerName` rejection, and missing credential files. - **`cmd/benchmarking/boomer-worker`**: - Default `--router-url` to `boomerutil.DefaultRouterURL` and fail fast on startup if `--router-url` does not start with `https://`. - Configure `http.Transport.TLSClientConfig` via `boomerutil.RouterTLSConfig()` and keep `ForceAttemptHTTP2 = false` so Go does not implicitly enable HTTP/2 ALPN when `TLSClientConfig` is non-nil (preserving HTTP/1.1 connection-pool semantics for `glutton` and `durdir`). - **`internal/benchmarking/boomer/ateenv`**: - Update `routerGRPCTarget` to require `https://` and default to `:443`. - Dial `atenet-router` with `credentials.NewTLS(tlsCfg)` from `boomerutil.RouterTLSConfig()`. - **`benchmarking/nighthawk-ingress` & `benchmarking/nighthawk-egress`**: - Update `DEFAULT_ROUTER_URL` to `https://atenet-router.ate-system.svc:443` and configure `_warm_actor` `requests.Session` with `verify = "/run/servicedns-ca/ca.crt"` and `cert = "/run/podidentity.podcert.ate.dev/credential-bundle.pem"`. - Reject non-`https://` URIs in `build_spec_dict` and populate `tls_context` (`sni: "atenet-router.ate-system.svc"`, `tls_certificates` client cert + key from `/run/podidentity.podcert.ate.dev/credential-bundle.pem`, `trusted_ca` from `/run/servicedns-ca/ca.crt`, and `match_typed_subject_alt_names` exact `DNS` match on `atenet-router.ate-system.svc`). - **`benchmarking/locust/tests/counter_demo.py`**: - Switch `ROUTER_URL` to `https://atenet-router.ate-system.svc` and configure `http_session.verify` and `http_session.cert`. ## Testing - Unit tests: `go test -race -count=1 ./cmd/benchmarking/... ./internal/benchmarking/... ./benchmarking/...`, `benchmarking/automation` unit tests, and Nighthawk proto round-trip & URI validation tests (`test_spec_round_trips_through_real_protos`, `test_rejects_non_https_uri`). - Live GKE cluster smoke tests (`glutton`, `durdir`, and `ate-env` on gVisor): 0 failures across all runs; verified `atenet-router` Envoy metrics (`ssl_handshake > 0`, `ssl_fail_verify_* == 0`, HTTP/1.1 connections for `glutton`/`durdir`, and HTTP/2 connections for `ate-env`). Fixes agent-substrate#3088 - [x] Tests pass - [x] Docs updated
…es to v2.22.0 (agent-substrate#3196) The distroless and alpine digests in `.ko.yaml` still trust the two roots the eMudhra compromise distrusted, so every ko image does. The kind overlay ran Jaeger all-in-one 1.55, and Jaeger v1 had its last release in December 2025. **Change.** Three commits. `.ko.yaml` and the GCP Secret Manager plugin's `.ko.yaml` move to the current distroless static and base digests and to alpine 3.24.3, whose bundles drop AAA Certificate Services and emSign ECC Root CA - G3. The kind overlay runs `jaegertracing/jaeger:2.22.0`, which with no config is the all-in-one: OTLP on 4317 and 4318, the UI on 16686, memory storage. Two doc phrases drop "all-in-one". **Tests.** `go test ./cmd/ate-setup/...` passes. On a kind cluster the v2 image rolls out, logs the all-in-one default, answers on the UI and the v3 API, and lists the components' traces from the collector. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated
…s in progress (agent-substrate#3131) Fixes agent-substrate#3129 `UpdateActor` now takes the actor lease, like suspend, resume, revert, delete and tag. A caller that races one of them gets Aborted and can retry. This is to prevent an actor's workerSelector or actorTemplate from getting updated during other operations such as suspend/resume. AI use: AI-assisted with Claude Code, which drafted the code, test, and this description. I reviewed them. - [x] Tests pass - [ ] Docs updated
…3028) ko built every image on a base that runs as root, so runAsNonRoot would have failed every pod with CreateContainerConfigError. Switch defaultBaseImage to static-debian13:nonroot, pinned by digest, and set runAsNonRoot on the control plane containers. Also set allowPrivilegeEscalation: false and drop all capabilities on containers that lacked them. ate-api-server keeps NET_BIND_SERVICE for :443. The envoy container in atenet-router sets runAsUser: 65532 because its image has no USER. ate-controller gets a read-only root filesystem. atelet and its objectstore-plugin sidecar stay root and only get allowPrivilegeEscalation: false. <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Fixes part of agent-substrate#2936 <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: Ai-assisted, I reviewed and tested it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [ ] Docs updated
Part of agent-substrate#2715 First of the PRs that measure how many CONNECT tunnels the egress gateway can hold and what each costs. This PR adds `benchmark-egress`, one binary with two subcommands: `actor`, which runs inside each Actor and holds one tunnel per endpoint through the gateway, and `target`, the Deployment behind the endpoint Services. The driver, the deploy script and the orchestrator entry follow in later PRs. Design and details: https://github.com/ygao-g/substrate/tree/egress-tunnels-workload/cmd/benchmarking/egress. The README there is the design doc: what the benchmark measures (tunnel, blended, settled), the topology, the traffic shape over a run, what the actor does and reports, what a run computes, and the decisions with the alternatives they displaced. More specifically, - **Pacing.** The gateway's stream idle timeout is 5 minutes by default, so every tunnel has to stay busy. With C endpoints the actor runs one goroutine per endpoint: endpoint i opens i intervals after `/start`, then each is requested once per period of C × `intervalMs`. - **Report.** The actor records its four latency distributions as OTel histograms on a private MeterProvider behind a manual reader in delta temporality, and serves them as window cuts: `POST /collect?window=N` answers the counters and the distributions since the previous cut, each distribution as the fields of an OTel histogram data point with the bounds in the payload, so the driver merges across actors with `Point.Merge` and refuses an image that buckets differently. The actor caches the last cut by id: a lost answer is asked for again and costs a retry, not a window; a skipped or older id is 409. `GET /stats` serves the cumulative counters for a driver's gates. `POST /stop` is the last cut. - **Target.** One Deployment behind every endpoint Service, one backend port per Service. For HTTPS it serves a certificate from a throwaway CA name-constrained to the endpoint domain. ## Overview of Commits. 1. the binary: Cobra root, no subcommands yet 2. the contract: the routes and the `/start` fields, the `Stats` counters, `Point` with `Merge` and `Quantile`, the shared bounds, the collect route and window id, protocol version 2 3. the name-constrained CA and certificate the target serves 4. the actor subcommand: the loop, the recorder, `/stats`, `/collect` with window ids, `/stop`, the protocol header, `classify` 5. the target subcommand 6. the README ## Testing Unit tests cover the window ids (first cut is 1, the last id answers again byte for byte, the next id cuts, anything else is 409 with the next id named), that a repeated `/collect` consumes nothing and the sample between two answers lands in the next window, the cut invariant under concurrent reads (each window's samples equal the growth of successes since the previous cut), that a `/collect` racing a `/stop` is never a 500, `Merge` refusing differing bounds by index, and the payload budget at C = 1024 (`/stats` under 1 KiB, a cut under 12 KiB). Ran on GKE at 250 × 1000 and 1000 × 1000 HTTPS endpoints through one gateway replica, with the actor serving cuts: 0 errors and 0 redials in the settled window, every actor answering every round, 6.0 M settled requests at 10,070 req/s on 8.13 gateway cores at 1000 × 1000, and p99 within one bucket step of the earlier exposition-based run. The target served 1000 Services on the port range from one image. Prepared with help from an AI coding agent; reviewed and verified by the author.
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring the fork through upstream
ffed85512, including sandbox and snapshot lifecycle fixes, release publishing, and the egress benchmark.Keep Helm aligned with the upstream security contexts, runtime and image updates, and WorkerPool name validation. Preserve the fork's manual image, chart, and binary publishing in
release-fork.yamlalongside upstream's release workflow.Upgrade requirements are documented in the chart README: databases from before the upstream migration squash must be recreated; old gVisor memory snapshots need a compatible runtime; and existing Helm installs must delete the completed RustFS bucket initializer Job before its image changes. Please apply the
breaking-changelabel.Fork release workflows
flowchart LR tag["Version tag"] --> automatic["release.yaml"] automatic --> draft["Draft GitHub release"] manual["Manual release-fork.yaml"] --> images["Fork component images"] manual -->|create_release enabled| assets["Helm charts and kubectl-ate release"]Validation passed: Go race tests,
hack/verify-all.sh, Helm lint, manifest and CRD parity checks, and all 38 Helm unit tests.Draft follow-up:
hack/verify/postgresql-migrations.sh origin/mainrejects the migration files changed or deleted by the imported upstream squash. This needs resolution before merging.Merge with a merge commit to preserve upstream ancestry.
AI use: AI-assisted integration, review, validation, and PR preparation.