diff --git a/doc/howto/DEVELOPMENT.md b/doc/howto/DEVELOPMENT.md index 7d5c91a14..8b4d11219 100644 --- a/doc/howto/DEVELOPMENT.md +++ b/doc/howto/DEVELOPMENT.md @@ -98,7 +98,7 @@ brew install grpcurl | `make e2e-test` | Run end-to-end tests | | `make proto` | Regenerate protobuf files | | `make gazelle` | Update BUILD.bazel files | -| `make local-submitqueue-start` | Start full stack (Gateway + Orchestrator + MySQL) | +| `make local-submitqueue-start` | Start full workflow stack (Gateway + Orchestrator + Runway + two MySQL databases) | | `make local-submitqueue-ps` | Show running containers and ports | | `make local-submitqueue-logs` | View logs from all services | | `make local-stop` | Stop all services | @@ -109,10 +109,10 @@ brew install grpcurl ```bash # Run tests for a single package -bazel test //gateway/controller:controller_test +bazel test //submitqueue/gateway/controller:go_default_test # Run a single test function -bazel test //gateway/controller:controller_test --test_filter=TestLand +bazel test //submitqueue/gateway/controller:go_default_test --test_filter=TestLand # Run Gateway integration tests only make integration-test-submitqueue-gateway @@ -120,7 +120,7 @@ make integration-test-submitqueue-gateway # Run Orchestrator integration tests only make integration-test-submitqueue-orchestrator -# Run extension integration tests only +# Run SubmitQueue and shared extension integration tests make integration-test-extensions # Run unit tests without cache diff --git a/doc/howto/TESTING.md b/doc/howto/TESTING.md index 643c107d5..c85d941e5 100644 --- a/doc/howto/TESTING.md +++ b/doc/howto/TESTING.md @@ -1,6 +1,6 @@ # Testing -All testing (automated and manual) uses **containerized environments** for consistency and reproducibility. +Integration, end-to-end, and local/manual service testing use **containerized environments** for consistency and reproducibility. Unit tests run directly under Bazel and do not require Docker. ## Prerequisites @@ -24,13 +24,13 @@ SubmitQueue uses **two separate databases** to demonstrate proper architectural ### 1. Application Database - **Purpose**: Business data (requests, counters, batches) - **Schema**: `submitqueue/extension/storage/mysql/schema`, `platform/extension/counter/mysql/schema` -- **Used by**: Gateway (stores requests), Orchestrator (reads/updates request state) +- **Used by**: Gateway (receipts, request logs, and read models), Orchestrator (requests, batches, builds, and counters) - **Connection**: `MYSQL_DSN` ### 2. Queue Database - **Purpose**: Messaging infrastructure (queue messages, offsets, partition leases) - **Schema**: `platform/extension/messagequeue/mysql/schema` -- **Used by**: Gateway (publishes), Orchestrator (consumes) +- **Used by**: Gateway (publishes and consumes), Orchestrator (publishes and consumes), Runway (consumes merge work and publishes results) - **Connection**: `QUEUE_MYSQL_DSN` **Why separate?** @@ -54,7 +54,7 @@ make test # Integration tests (Docker required) make integration-test-submitqueue-gateway # Gateway in isolation make integration-test-submitqueue-orchestrator # Orchestrator in isolation -make integration-test-extensions # All extension tests +make integration-test-extensions # SubmitQueue and shared extension tests make integration-test # All integration tests # E2E tests (Docker required) @@ -73,16 +73,16 @@ make build-all-linux # Build Linux binaries for the local docker- - Speed: Fast (< 1s typically) **2. Integration Tests** - Service in isolation with real dependencies -- Location: `test/integration/submitqueue//` (e.g., `gateway/`, `orchestrator/`, `extension//`) +- Location: `test/integration/{submitqueue,stovepipe,extension}/...` - Run: `make integration-test-submitqueue-gateway`, `make integration-test-submitqueue-orchestrator`, `make integration-test-submitqueue-consumer`, or `make integration-test-extensions` - Containers: MySQL + one service or the extension's dependencies - Tests one service isolated from others -**3. E2E Tests** - Complete workflows across all services -- Location: `test/e2e/submitqueue/` +**3. E2E Tests** - Complete domain workflows +- Location: `test/e2e/{submitqueue,stovepipe,runway}/` - Run: `make e2e-test` -- Containers: MySQL + all services -- Tests cross-service communication +- Containers: Each suite's required services and dependencies; SubmitQueue E2E includes Gateway, Orchestrator, Runway, and MySQL +- Tests end-to-end behavior, including cross-service communication where applicable ### How Automated Tests Work @@ -111,7 +111,7 @@ Project name format: ``` sq-test-{context}-{shortid} │ │ │ -│ │ └─ 6-char hex timestamp (unique per test run) +│ │ └─ 6 hex digits derived from the current time │ └─────────── Test context (domain-qualified — see convention) └─────────────────── Namespace prefix ``` @@ -139,14 +139,14 @@ Shared (cross-domain) suites carry no domain segment — e.g. the shared queue e | Stovepipe | `svc-stovepipe` | `sq-test-svc-stovepipe-abc123-stovepipe-service-1` | | SubmitQueue storage extension | `ext-submitqueue-storage-mysql` | `sq-test-ext-submitqueue-storage-mysql-2ce1d0-mysql-1` | | Counter extension (shared) | `ext-counter-mysql` | `sq-test-ext-counter-mysql-…-mysql-1` | -| SubmitQueue changestore extension | `ext-submitqueue-changestore-mysql` | `sq-test-ext-submitqueue-changestore-mysql-…-mysql-1` | +| Stovepipe storage extension | `ext-stovepipe-storage-mysql` | `sq-test-ext-stovepipe-storage-mysql-…-mysql-1` | | Shared queue extension | `ext-messagequeue-sql` | `sq-test-ext-messagequeue-sql-a1b2c3-mysql-1` | | SubmitQueue consumer (core) | `core-submitqueue-consumer` | `sq-test-core-submitqueue-consumer-…-mysql-1` | | SubmitQueue e2e (full stack) | `e2e-submitqueue` | `sq-test-e2e-submitqueue-def456-gateway-service-1` | ### Parallel execution -Every suite gets a unique project name (`{context}-{shortid}`) and every compose service publishes **ephemeral host ports** (`- "3306"`, `- "8080"`), so suites are fully isolated and run **in parallel**. `make integration-test` runs all suites concurrently via `--test_output=errors` (`--test_output=streamed` would force bazel to serialize them). The domain-qualified context is what keeps container names unambiguous when many run at once. +Each suite normally gets a distinct project name (`{context}-{shortid}`), where the short suffix is derived from the low 24 bits of the current nanosecond timestamp. It is useful for separating concurrent runs but is not a guaranteed unique identifier. Every compose service publishes **ephemeral host ports** (`- "3306"`, `- "8080"`), so suites can run **in parallel**. `make integration-test` runs suites concurrently via `--test_output=errors` (`--test_output=streamed` would force Bazel to serialize them). The domain-qualified context keeps container names understandable when many run at once. ### Debugging with Container Names @@ -173,7 +173,7 @@ docker exec -it sq-test-ext-counter-2ce1d0-mysql-1 \ ### Quick Start ```bash -# Start all services (Gateway + Orchestrator + 2 MySQL DBs) +# Start the full workflow stack (Gateway + Orchestrator + Runway + 2 MySQL DBs) make local-submitqueue-start # See running containers and endpoints @@ -199,9 +199,10 @@ grpcurl -plaintext -d '{"message": "hello"}' localhost: uber.submitqueue.g # Test Land API grpcurl -plaintext -d '{ "queue": "test-queue", - "change": {"source": "github", "ids": ["PR-123"]}, + "change": {"uris": ["github://github.com/owner/repo/pull/123/0123456789abcdef0123456789abcdef01234567"]}, "strategy": "REBASE" -}' localhost: uber.submitqueue.gateway.SubmitQueueGateway/Land +}' -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \ + localhost: uber.submitqueue.gateway.SubmitQueueGateway/Land # Stop make local-submitqueue-gateway-stop @@ -252,20 +253,16 @@ mysql -h127.0.0.1 -P -uroot -proot submitqueue brew install grpcurl # macOS # OR: go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest -# List services (use port from make local-submitqueue-ps) -grpcurl -plaintext localhost: list - -# Describe a service -grpcurl -plaintext localhost: describe uber.submitqueue.gateway.SubmitQueueGateway - -# Call Ping -grpcurl -plaintext -d '{"message": "test"}' \ +# Gateway reflection currently cannot resolve one imported descriptor, so pass the local proto. +grpcurl -plaintext -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \ + -d '{"message": "test"}' \ localhost: uber.submitqueue.gateway.SubmitQueueGateway/Ping # Call Land -grpcurl -plaintext -d '{ - "queue": "my-queue", - "change": {"source": "github", "ids": ["PR-456"]}, +grpcurl -plaintext -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \ + -d '{ + "queue": "test-queue", + "change": {"uris": ["github://github.com/owner/repo/pull/456/0123456789abcdef0123456789abcdef01234567"]}, "strategy": "REBASE" }' localhost: uber.submitqueue.gateway.SubmitQueueGateway/Land ``` @@ -380,7 +377,7 @@ assert.Equal(s.T(), "expected", resp.Value) 1. Add test to `test/e2e/submitqueue/suite_test.go` 2. Use all service clients -3. Use `require.Eventually()` for async operations +3. Use the suite's polling helper for async operations; it retries until the condition holds and relies on the Bazel test timeout rather than adding a second hardcoded deadline 4. Run: `make e2e-test` --- diff --git a/service/README.md b/service/README.md index 939808e29..72c76bb46 100644 --- a/service/README.md +++ b/service/README.md @@ -5,7 +5,7 @@ Runnable gRPC servers and clients that wire each domain's controllers and extens Each domain has its own subdirectory with a dedicated README: - [`submitqueue/`](submitqueue/README.md) — the multi-service SubmitQueue domain (Gateway + Orchestrator). -- [`stovepipe/`](stovepipe/README.md) — the single-service Stovepipe domain (ingest → process pipeline). +- [`stovepipe/`](stovepipe/README.md) — the single-service Stovepipe domain (ingest → process → build → buildsignal → record). - [`runway/`](runway/README.md) — the single-service Runway landing service (consumes the merge queues). ## Services @@ -13,8 +13,8 @@ Each domain has its own subdirectory with a dedicated README: | Service | Port | Domain | RPCs | Backing stores | |---------|------|--------|------|----------------| | **SubmitQueue Gateway** | 8081 | `submitqueue` | `Ping`, `Land`, `Cancel`, `GetRequestSummaryByID`, `GetRequestSummaryByChangeURI`, `List`, `GetRequestHistoryByID`, `GetRequestHistoryByChangeURI` | MySQL app + queue | -| **SubmitQueue Orchestrator** | 8082 | `submitqueue` | `Ping` (+ consumes pipeline topics: start, cancel, validate, batch, dependency-analysis, speculate, build, buildsignal, submitqueue-merge, conclude, log, plus DLQ topics, and the two Runway signal topics) | MySQL app + queue | -| **Stovepipe** | 8083 | `stovepipe` | `Ping`, `Ingest` (+ consumes the process, build, buildsignal, and record topics, plus DLQ topics) | MySQL storage + queue | +| **SubmitQueue Orchestrator** | 8082 | `submitqueue` | `Ping` (+ consumes start, cancel, validate, merge-conflict-check-signal, batch, dependency-analysis, speculate, build, buildsignal, submitqueue-merge, merge-signal, conclude, submitqueue-hook, and paired DLQ topics) | MySQL app + queue | +| **Stovepipe** | 8083 | `stovepipe` | `Ping`, `Ingest` (+ consumes process, build, buildsignal, record, stovepipe-hook, and paired DLQ topics) | MySQL storage + queue | | **Runway** | 8086 | `runway` | `Ping` (+ consumes merge-conflict-check & runway-merge topics) | MySQL queue | Ports above are the `go run` defaults; under Docker Compose each server listens on `:8080` inside its container and is published on a random ephemeral host port (use `make local-*-ps` / `docker port` to discover it). @@ -24,11 +24,11 @@ Ports above are the `go run` defaults; under Docker Compose each server listens ``` service/ ├── submitqueue/ -│ ├── docker-compose.yml # Full stack (Gateway + Orchestrator + 2x MySQL) +│ ├── docker-compose.yml # Full stack (Gateway + Orchestrator + Runway + 2x MySQL) │ ├── gateway/ │ │ ├── server/ # Gateway server entry point + Dockerfile + compose -│ │ │ └── queues.yaml # Per-queue extension profiles -│ │ └── client/ # Gateway ping client +│ │ │ └── queues.yaml # Gateway valid-queue registry +│ │ └── client/ # Gateway command-line client │ └── orchestrator/ │ ├── server/ # Orchestrator server entry point + Dockerfile + compose │ └── client/ # Orchestrator ping client @@ -47,7 +47,7 @@ service/ ### Docker Compose (recommended) ```bash -# Full SubmitQueue stack (Gateway + Orchestrator + MySQL) +# Full SubmitQueue workflow stack (Gateway + Orchestrator + Runway + MySQL) make local-submitqueue-start make local-submitqueue-gateway-start # gateway-only stack make local-submitqueue-orchestrator-start # orchestrator-only stack @@ -98,7 +98,7 @@ go run ./service/runway/server ```bash # Go clients -go run ./service/submitqueue/gateway/client -addr localhost:8081 -message "hello" +go run ./service/submitqueue/gateway/client -addr localhost:8081 ping -message "hello" go run ./service/submitqueue/orchestrator/client -addr localhost:8082 -message "hello" go run ./service/stovepipe/client -addr localhost:8083 -message "hello" go run ./service/runway/client -addr localhost:8086 -message "hello" @@ -110,10 +110,7 @@ make run-client-stovepipe make run-client-runway ``` -Client flags: -- `-addr`: Server address (default: service-specific port) -- `-message`: Message to send in the ping request -- `-timeout`: Request timeout (default: 5s) +The Gateway client is command-oriented (`ping`, `land`, `status`, `list`, and `watch`); run it without arguments for command-specific usage. The other clients are Ping clients with `-addr`, `-message`, and `-timeout` flags. ### grpcurl @@ -125,17 +122,19 @@ go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest ``` ```bash -# Ping -grpcurl -plaintext -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping +# Ping services whose reflection descriptors resolve normally grpcurl -plaintext -d '{"message": "hello"}' localhost:8082 uber.submitqueue.orchestrator.SubmitQueueOrchestrator/Ping grpcurl -plaintext -d '{"message": "hello"}' localhost:8083 uber.submitqueue.stovepipe.Stovepipe/Ping grpcurl -plaintext -d '{"message": "hello"}' localhost:8086 uber.runway.Runway/Ping -# List / describe services (reflection is registered on every server) -grpcurl -plaintext localhost:8081 list -grpcurl -plaintext localhost:8081 describe uber.submitqueue.gateway.SubmitQueueGateway +# Gateway: use the repository client, or supply its proto explicitly +go run ./service/submitqueue/gateway/client -addr localhost:8081 ping -message "hello" +grpcurl -plaintext -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \ + -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping ``` +The Gateway registers reflection, but its generated descriptor currently cannot resolve one imported file name. Do not use Gateway reflection-based `list` or `describe`; use the client or pass `-import-path` and `-proto` to grpcurl. + ## API Reference ### Gateway Service @@ -183,4 +182,3 @@ The gateway owns request receipts, current-status projections, and request-log r | Method | Description | |--------|-------------| | `Ping` | Health check | - diff --git a/service/stovepipe/README.md b/service/stovepipe/README.md index 7fc53a571..ae5c011fa 100644 --- a/service/stovepipe/README.md +++ b/service/stovepipe/README.md @@ -7,13 +7,13 @@ Runnable wiring for the **Stovepipe** domain — a single-service domain (the do - **process consumer** (`TopicKeyProcess`) — reloads the persisted `Request` from storage and runs the process stage (`stovepipe/controller/process`). - **build consumer** (`TopicKeyBuild`) — reloads the persisted `Request` and triggers the build-runner, then publishes to `buildsignal`. - **buildsignal consumer** (`TopicKeyBuildSignal`) — polls/records the build's terminal status and releases the queue's in-flight slot, then publishes to `record`. -- **record consumer** (`TopicKeyRecord`) — writes the whole-repo validation fact, and advances the queue's last-green bookmark and promotion ref. -- **hook consumer** (`TopicKeyHook`) — hands each lifecycle event to the hooks `hookResolver` returns (`platform/hook`). Nothing publishes to this topic yet and the resolver returns only `noop`, so events are accepted and discarded. Its topic name is domain-qualified (`stovepipe-hook`) because the key is shared across domains. -- **DLQ reconciler** — for each internal topic, a `_dlq` consumer that drives stuck requests to a conservative terminal state so the queue's slot is freed. +- **record consumer** (`TopicKeyRecord`) — writes the whole-repository validation fact and, for a green fact, advances the queue's last-green bookmark and promotion ref. +- **hook consumer** (`TopicKeyHook`) — receives `validation.repository.started` from process and `validation.repository.recorded` or `validation.repository.cancelled` from record. The current resolver invokes only `noop`, so these events have no external side effect. Its topic name is domain-qualified (`stovepipe-hook`) because the key is shared across domains. +- **DLQ consumers** — registered for process, build, buildsignal, record, and hook, with stage-specific reconciliation behavior. -The ingest → process → build → buildsignal → record hop stays inside one service and one store, so the queue messages carry only request **IDs**; the consumers reload from storage (the source of truth), which keeps messages small and redelivery idempotent. The process, build, buildsignal, and record topic keys and their internal wire contract are owned by the domain under `stovepipe/core/messagequeue/`. +The ingest → process → build → buildsignal → record flow stays inside one service and one store, so messages carry thin identifiers: request IDs on process, build, and record; a build ID on buildsignal. Consumers reload the full entity from storage, which keeps messages small and redelivery idempotent. The topic keys and internal wire contract are owned by the domain under `stovepipe/core/messagequeue/`. -Stovepipe therefore needs two MySQL databases: a **storage** database (the `queue`, `request`, `request_uri`, and `build` tables) and a **queue** database (messaging infrastructure). +Stovepipe therefore needs two MySQL databases: a **storage** database (the `queue`, `request`, `request_uri`, `request_log`, `build`, and `validation_fact` tables) and a **queue** database (messaging infrastructure). ## Wiring notes @@ -35,14 +35,15 @@ stovepipe/ └── main.go # Ping client (default :8083) ``` -The Stovepipe controllers live under [`stovepipe/controller/`](../../stovepipe/controller) (subdirectories for `ingest`, `process`, `build`, `buildsignal`, `record`, and `dlq`) and its extensions under [`stovepipe/extension/`](../../stovepipe/extension); this directory only contains the runnable wiring and a Docker Compose stack for manual testing. +The Stovepipe controllers live under [`stovepipe/controller/`](../../stovepipe/controller): `ingest.go` contains the RPC controller, while `process/`, `build/`, `buildsignal/`, `record/`, and `dlq/` contain queue controllers. Its extensions live under [`stovepipe/extension/`](../../stovepipe/extension); this directory only contains runnable wiring and a Docker Compose stack for manual testing. ## Configuration | Variable | Required | Description | Default | |---------------------|----------|------------------------------------------|----------------------| -| `STORAGE_MYSQL_DSN` | yes | Storage database DSN (`request`, `request_uri`) | — | +| `STORAGE_MYSQL_DSN` | yes | Storage database DSN | — | | `QUEUE_MYSQL_DSN` | yes | Queue database DSN | — | +| `QUEUE_LOG_LEVEL` | no | Message-queue logger level | `info` | | `PORT` | no | gRPC listen address | `:8083` | | `HOSTNAME` | no | Subscriber name for the queue consumers | `stovepipe-` | @@ -93,4 +94,4 @@ grpcurl -plaintext -d '{"message": "hello"}' localhost:8083 uber.submitqueue.sto ## Shutdown -The server handles `SIGINT` / `SIGTERM` gracefully: it drains in-flight RPCs, then stops the process consumer (30s timeout). It exits `0` on clean shutdown, `143` (128 + SIGTERM) when stopped by signal, and `1` on startup/runtime errors (details on stderr). Shutdown errors override the signal exit code. +The server handles `SIGINT` / `SIGTERM` gracefully: it drains in-flight RPCs, then stops the primary pipeline consumer followed by the DLQ consumer (30-second limit for each). It exits `0` on clean shutdown, `143` (128 + SIGTERM) when stopped by signal, and `1` on startup/runtime errors (details on stderr). Shutdown errors override the signal exit code. diff --git a/service/submitqueue/README.md b/service/submitqueue/README.md index 1cdb2bea1..cc80c582b 100644 --- a/service/submitqueue/README.md +++ b/service/submitqueue/README.md @@ -1,38 +1,43 @@ # SubmitQueue Services -Runnable wiring for the **SubmitQueue** domain's two services — the Gateway (entry point for land requests) and the Orchestrator (coordinates the pipeline) — wired with MySQL-backed extensions and runnable via Docker Compose. +Runnable wiring for the **SubmitQueue** domain's two services — the Gateway (entry point for land requests) and the Orchestrator (coordinates the pipeline) — wired with MySQL-backed extensions. The full Docker Compose workflow also starts Runway, which performs merge-conflict checks and merges. ## Starting ### Docker Compose (recommended) ```bash -make local-submitqueue-start # builds binaries and starts all services +make local-submitqueue-start # builds binaries and starts Gateway, Orchestrator, Runway, and both databases make local-submitqueue-ps # verify services are running make local-submitqueue-logs # view logs ``` ### Standalone -Each service requires two MySQL databases (app and queue) and is configured via environment variables: - -| Variable | Required | Description | Default | -|--------------------|----------|--------------------------------------|--------------------------------------| -| `MYSQL_DSN` | yes | App database DSN | — | -| `QUEUE_MYSQL_DSN` | yes | Queue database DSN | — | -| `PORT` | no | gRPC listen address | `:8081` (gateway), `:8082` (orchestrator) | -| `HOSTNAME` | no | Subscriber name for queue consumers | `orchestrator-` (orchestrator only) | -| `GITHUB_TOKEN` | no | GitHub API token for merge checker | — (orchestrator only) | -| `GITHUB_GRAPHQL_URL` | no | GitHub GraphQL endpoint | `https://api.github.com/graphql` (orchestrator only) | +Each service requires the application and queue MySQL databases. Service-specific configuration is supplied through environment variables: + +| Variable | Service | Required | Description | Default | +|----------|---------|----------|-------------|---------| +| `MYSQL_DSN` | both | yes | Application database DSN | — | +| `QUEUE_MYSQL_DSN` | both | yes | Queue database DSN | — | +| `QUEUE_CONFIG_PATH` | Gateway | yes | YAML registry of accepted queue names | — | +| `PROFILES_CONFIG_PATH` | Orchestrator | no | YAML extension profiles and per-queue routing | built-in example profiles | +| `HOSTNAME` | both | no | Base subscriber name; Gateway appends its PID | time-based service name | +| `QUEUE_LOG_LEVEL` | both | no | Message-queue logger level | `info` | +| `CONSUMER_GATE_DIR` | both | no | Directory holding per-consumer gate files | gating disabled when unset | +| `GITHUB_TOKEN` | Orchestrator | profile-dependent | Default credential for GitHub integrations selected by a profile | — | +| `GITHUB_BASE_URL` | Orchestrator | no | GitHub API base URL | `https://api.github.com` | +| `PORT` | both | no | gRPC listen address | `:8081` (Gateway), `:8082` (Orchestrator) | ```bash export MYSQL_DSN='root:root@tcp(127.0.0.1:3306)/submitqueue?parseTime=true' export QUEUE_MYSQL_DSN='root:root@tcp(127.0.0.1:3307)/submitqueue?parseTime=true' # Start gateway (default :8081) +export QUEUE_CONFIG_PATH=service/submitqueue/gateway/server/queues.yaml go run ./service/submitqueue/gateway/server -# Start orchestrator (default :8082) +# Start orchestrator with its built-in example profiles (default :8082) go run ./service/submitqueue/orchestrator/server ``` @@ -41,7 +46,7 @@ go run ./service/submitqueue/orchestrator/server Both services handle `SIGINT` (Ctrl+C) and `SIGTERM` gracefully: 1. The gRPC server stops accepting new connections and drains in-flight RPCs. -2. The orchestrator additionally stops its queue consumers (30s timeout). +2. The Gateway stops its request-log consumer, and the Orchestrator stops its complete queue pipeline (30-second limit for each). 3. The process exits with a code reflecting the outcome (see below). To stop Docker Compose services: diff --git a/stovepipe/core/messagequeue/README.md b/stovepipe/core/messagequeue/README.md index 15889fe19..e0c437c7a 100644 --- a/stovepipe/core/messagequeue/README.md +++ b/stovepipe/core/messagequeue/README.md @@ -7,7 +7,6 @@ Payloads are defined in proto3 (`proto/`, generated into `protopb/`) and seriali ## Stages - **process** (`TopicKeyProcess`, `ProcessRequest`) — ingest publishes the minted request id here once it accepts a new head; the process controller reloads the `Request` from storage and decides the build strategy. Only the id travels: producer and consumer share the store, so messages stay small and redelivery is idempotent. -- **build** (`TopicKeyBuild`, `BuildRequest`) — process/analyze publishes the request id here once its build scope (`BuildStrategy`/`BaseURI`) is decided; the build controller reloads the `Request`, triggers the build, and persists the resulting `Build`. Partitioned by request id. -- **buildsignal** (`TopicKeyBuildSignal`, `BuildSignal`) — build publishes the build id here after triggering; buildsignal re-publishes to itself between polls until the build reaches a terminal status. Partitioned by build id, so each build's poll loop is an independent partition. See [doc/rfc/stovepipe/steps/build.md](../../../doc/rfc/stovepipe/steps/build.md) and [buildsignal.md](../../../doc/rfc/stovepipe/steps/buildsignal.md). - -See [doc/rfc/messagequeue-contract.md](../../../doc/rfc/messagequeue-contract.md) for the contract conventions and `api/runway/messagequeue` for the external reference example. +- **build** (`TopicKeyBuild`, `BuildRequest`) — process publishes the request id here once its build scope (`BuildStrategy`/`BaseURI`) is decided; the build controller reloads the `Request`, triggers the build, and persists the resulting `Build`. Partitioned by request id. +- **buildsignal** (`TopicKeyBuildSignal`, `BuildSignal`) — build publishes the build id here after triggering; buildsignal holds the delivery between polls until the build reaches a terminal status. Partitioned by build id, so each build's poll loop is an independent partition. +- **record** (`TopicKeyRecord`, `Record`) — buildsignal publishes the request id after persisting the terminal outcome; record reloads the request and records its validation fact. Partitioned by request id.