docs: align runtime and contributor guides with implementation

doc/howto/DEVELOPMENT.md

@@ -98,7 +98,7 @@ brew install grpcurl
 | `make e2e-test` | Run end-to-end tests |
 | `make proto` | Regenerate protobuf files |
 | `make gazelle` | Update BUILD.bazel files |
-| `make local-submitqueue-start` | Start full stack (Gateway + Orchestrator + MySQL) |
+| `make local-submitqueue-start` | Start full workflow stack (Gateway + Orchestrator + Runway + two MySQL databases) |
 | `make local-submitqueue-ps` | Show running containers and ports |
 | `make local-submitqueue-logs` | View logs from all services |
 | `make local-stop` | Stop all services |
@@ -109,18 +109,18 @@ brew install grpcurl
 
 ```bash
 # Run tests for a single package
-bazel test //gateway/controller:controller_test
+bazel test //submitqueue/gateway/controller:go_default_test
 
 # Run a single test function
-bazel test //gateway/controller:controller_test --test_filter=TestLand
+bazel test //submitqueue/gateway/controller:go_default_test --test_filter=TestLand
 
 # Run Gateway integration tests only
 make integration-test-submitqueue-gateway
 
 # Run Orchestrator integration tests only
 make integration-test-submitqueue-orchestrator
 
-# Run extension integration tests only
+# Run SubmitQueue and shared extension integration tests
 make integration-test-extensions
 
 # Run unit tests without cache

doc/howto/TESTING.md

@@ -1,6 +1,6 @@
 # Testing
 
-All testing (automated and manual) uses **containerized environments** for consistency and reproducibility.
+Integration, end-to-end, and local/manual service testing use **containerized environments** for consistency and reproducibility. Unit tests run directly under Bazel and do not require Docker.
 
 ## Prerequisites
 
@@ -24,13 +24,13 @@ SubmitQueue uses **two separate databases** to demonstrate proper architectural
 ### 1. Application Database
 - **Purpose**: Business data (requests, counters, batches)
 - **Schema**: `submitqueue/extension/storage/mysql/schema`, `platform/extension/counter/mysql/schema`
-- **Used by**: Gateway (stores requests), Orchestrator (reads/updates request state)
+- **Used by**: Gateway (receipts, request logs, and read models), Orchestrator (requests, batches, builds, and counters)
 - **Connection**: `MYSQL_DSN`
 
 ### 2. Queue Database
 - **Purpose**: Messaging infrastructure (queue messages, offsets, partition leases)
 - **Schema**: `platform/extension/messagequeue/mysql/schema`
-- **Used by**: Gateway (publishes), Orchestrator (consumes)
+- **Used by**: Gateway (publishes and consumes), Orchestrator (publishes and consumes), Runway (consumes merge work and publishes results)
 - **Connection**: `QUEUE_MYSQL_DSN`
 
 **Why separate?**
@@ -54,7 +54,7 @@ make test
 # Integration tests (Docker required)
 make integration-test-submitqueue-gateway       # Gateway in isolation
 make integration-test-submitqueue-orchestrator  # Orchestrator in isolation
-make integration-test-extensions    # All extension tests
+make integration-test-extensions    # SubmitQueue and shared extension tests
 make integration-test               # All integration tests
 
 # E2E tests (Docker required)
@@ -73,16 +73,16 @@ make build-all-linux                # Build Linux binaries for the local docker-
 - Speed: Fast (< 1s typically)
 
 **2. Integration Tests** - Service in isolation with real dependencies
-- Location: `test/integration/submitqueue/<area>/` (e.g., `gateway/`, `orchestrator/`, `extension/<ext>/`)
+- Location: `test/integration/{submitqueue,stovepipe,extension}/...`
 - Run: `make integration-test-submitqueue-gateway`, `make integration-test-submitqueue-orchestrator`, `make integration-test-submitqueue-consumer`, or `make integration-test-extensions`
 - Containers: MySQL + one service or the extension's dependencies
 - Tests one service isolated from others
 
-**3. E2E Tests** - Complete workflows across all services
-- Location: `test/e2e/submitqueue/`
+**3. E2E Tests** - Complete domain workflows
+- Location: `test/e2e/{submitqueue,stovepipe,runway}/`
 - Run: `make e2e-test`
-- Containers: MySQL + all services
-- Tests cross-service communication
+- Containers: Each suite's required services and dependencies; SubmitQueue E2E includes Gateway, Orchestrator, Runway, and MySQL
+- Tests end-to-end behavior, including cross-service communication where applicable
 
 ### How Automated Tests Work
 
@@ -111,7 +111,7 @@ Project name format:
 ```
 sq-test-{context}-{shortid}
 │       │         │
-│       │         └─ 6-char hex timestamp (unique per test run)
+│       │         └─ 6 hex digits derived from the current time
 │       └─────────── Test context (domain-qualified — see convention)
 └─────────────────── Namespace prefix
 ```
@@ -139,14 +139,14 @@ Shared (cross-domain) suites carry no domain segment — e.g. the shared queue e
 | Stovepipe | `svc-stovepipe` | `sq-test-svc-stovepipe-abc123-stovepipe-service-1` |
 | SubmitQueue storage extension | `ext-submitqueue-storage-mysql` | `sq-test-ext-submitqueue-storage-mysql-2ce1d0-mysql-1` |
 | Counter extension (shared) | `ext-counter-mysql` | `sq-test-ext-counter-mysql-…-mysql-1` |
-| SubmitQueue changestore extension | `ext-submitqueue-changestore-mysql` | `sq-test-ext-submitqueue-changestore-mysql-…-mysql-1` |
+| Stovepipe storage extension | `ext-stovepipe-storage-mysql` | `sq-test-ext-stovepipe-storage-mysql-…-mysql-1` |
 | Shared queue extension | `ext-messagequeue-sql` | `sq-test-ext-messagequeue-sql-a1b2c3-mysql-1` |
 | SubmitQueue consumer (core) | `core-submitqueue-consumer` | `sq-test-core-submitqueue-consumer-…-mysql-1` |
 | SubmitQueue e2e (full stack) | `e2e-submitqueue` | `sq-test-e2e-submitqueue-def456-gateway-service-1` |
 
 ### Parallel execution
 
-Every suite gets a unique project name (`{context}-{shortid}`) and every compose service publishes **ephemeral host ports** (`- "3306"`, `- "8080"`), so suites are fully isolated and run **in parallel**. `make integration-test` runs all suites concurrently via `--test_output=errors` (`--test_output=streamed` would force bazel to serialize them). The domain-qualified context is what keeps container names unambiguous when many run at once.
+Each suite normally gets a distinct project name (`{context}-{shortid}`), where the short suffix is derived from the low 24 bits of the current nanosecond timestamp. It is useful for separating concurrent runs but is not a guaranteed unique identifier. Every compose service publishes **ephemeral host ports** (`- "3306"`, `- "8080"`), so suites can run **in parallel**. `make integration-test` runs suites concurrently via `--test_output=errors` (`--test_output=streamed` would force Bazel to serialize them). The domain-qualified context keeps container names understandable when many run at once.
 
 ### Debugging with Container Names
 
@@ -173,7 +173,7 @@ docker exec -it sq-test-ext-counter-2ce1d0-mysql-1 \
 ### Quick Start
 
 ```bash
-# Start all services (Gateway + Orchestrator + 2 MySQL DBs)
+# Start the full workflow stack (Gateway + Orchestrator + Runway + 2 MySQL DBs)
 make local-submitqueue-start
 
 # See running containers and endpoints
@@ -199,9 +199,10 @@ grpcurl -plaintext -d '{"message": "hello"}' localhost:<PORT> uber.submitqueue.g
 # Test Land API
 grpcurl -plaintext -d '{
   "queue": "test-queue",
-  "change": {"source": "github", "ids": ["PR-123"]},
+  "change": {"uris": ["github://github.com/owner/repo/pull/123/0123456789abcdef0123456789abcdef01234567"]},
   "strategy": "REBASE"
-}' localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Land
+}' -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \
+  localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Land
 
 # Stop
 make local-submitqueue-gateway-stop
@@ -252,20 +253,16 @@ mysql -h127.0.0.1 -P<QUEUE_PORT> -uroot -proot submitqueue
 brew install grpcurl  # macOS
 # OR: go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest
 
-# List services (use port from make local-submitqueue-ps)
-grpcurl -plaintext localhost:<PORT> list
-
-# Describe a service
-grpcurl -plaintext localhost:<PORT> describe uber.submitqueue.gateway.SubmitQueueGateway
-
-# Call Ping
-grpcurl -plaintext -d '{"message": "test"}' \
+# Gateway reflection currently cannot resolve one imported descriptor, so pass the local proto.
+grpcurl -plaintext -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \
+  -d '{"message": "test"}' \
   localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Ping
 
 # Call Land
-grpcurl -plaintext -d '{
-  "queue": "my-queue",
-  "change": {"source": "github", "ids": ["PR-456"]},
+grpcurl -plaintext -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \
+  -d '{
+  "queue": "test-queue",
+  "change": {"uris": ["github://github.com/owner/repo/pull/456/0123456789abcdef0123456789abcdef01234567"]},
   "strategy": "REBASE"
 }' localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Land
 ```
@@ -380,7 +377,7 @@ assert.Equal(s.T(), "expected", resp.Value)
 
 1. Add test to `test/e2e/submitqueue/suite_test.go`
 2. Use all service clients
-3. Use `require.Eventually()` for async operations
+3. Use the suite's polling helper for async operations; it retries until the condition holds and relies on the Bazel test timeout rather than adding a second hardcoded deadline
 4. Run: `make e2e-test`
 
 ---

service/README.md

@@ -5,16 +5,16 @@ Runnable gRPC servers and clients that wire each domain's controllers and extens
 Each domain has its own subdirectory with a dedicated README:
 
 - [`submitqueue/`](submitqueue/README.md) — the multi-service SubmitQueue domain (Gateway + Orchestrator).
-- [`stovepipe/`](stovepipe/README.md) — the single-service Stovepipe domain (ingest → process pipeline).
+- [`stovepipe/`](stovepipe/README.md) — the single-service Stovepipe domain (ingest → process → build → buildsignal → record).
 - [`runway/`](runway/README.md) — the single-service Runway landing service (consumes the merge queues).
 
 ## Services
 
 | Service | Port | Domain | RPCs | Backing stores |
 |---------|------|--------|------|----------------|
 | **SubmitQueue Gateway** | 8081 | `submitqueue` | `Ping`, `Land`, `Cancel`, `GetRequestSummaryByID`, `GetRequestSummaryByChangeURI`, `List`, `GetRequestHistoryByID`, `GetRequestHistoryByChangeURI` | MySQL app + queue |
-| **SubmitQueue Orchestrator** | 8082 | `submitqueue` | `Ping` (+ consumes pipeline topics: start, cancel, validate, batch, dependency-analysis, speculate, build, buildsignal, submitqueue-merge, conclude, log, plus DLQ topics, and the two Runway signal topics) | MySQL app + queue |
-| **Stovepipe** | 8083 | `stovepipe` | `Ping`, `Ingest` (+ consumes the process, build, buildsignal, and record topics, plus DLQ topics) | MySQL storage + queue |
+| **SubmitQueue Orchestrator** | 8082 | `submitqueue` | `Ping` (+ consumes start, cancel, validate, merge-conflict-check-signal, batch, dependency-analysis, speculate, build, buildsignal, submitqueue-merge, merge-signal, conclude, submitqueue-hook, and paired DLQ topics) | MySQL app + queue |
+| **Stovepipe** | 8083 | `stovepipe` | `Ping`, `Ingest` (+ consumes process, build, buildsignal, record, stovepipe-hook, and paired DLQ topics) | MySQL storage + queue |
 | **Runway** | 8086 | `runway` | `Ping` (+ consumes merge-conflict-check & runway-merge topics) | MySQL queue |
 
 Ports above are the `go run` defaults; under Docker Compose each server listens on `:8080` inside its container and is published on a random ephemeral host port (use `make local-*-ps` / `docker port` to discover it).
@@ -24,11 +24,11 @@ Ports above are the `go run` defaults; under Docker Compose each server listens
 ```
 service/
 ├── submitqueue/
-│   ├── docker-compose.yml              # Full stack (Gateway + Orchestrator + 2x MySQL)
+│   ├── docker-compose.yml              # Full stack (Gateway + Orchestrator + Runway + 2x MySQL)
 │   ├── gateway/
 │   │   ├── server/                     # Gateway server entry point + Dockerfile + compose
-│   │   │   └── queues.yaml             # Per-queue extension profiles
-│   │   └── client/                     # Gateway ping client
+│   │   │   └── queues.yaml             # Gateway valid-queue registry
+│   │   └── client/                     # Gateway command-line client
 │   └── orchestrator/
 │       ├── server/                     # Orchestrator server entry point + Dockerfile + compose
 │       └── client/                     # Orchestrator ping client
@@ -47,7 +47,7 @@ service/
 ### Docker Compose (recommended)
 
 ```bash
-# Full SubmitQueue stack (Gateway + Orchestrator + MySQL)
+# Full SubmitQueue workflow stack (Gateway + Orchestrator + Runway + MySQL)
 make local-submitqueue-start
 make local-submitqueue-gateway-start        # gateway-only stack
 make local-submitqueue-orchestrator-start   # orchestrator-only stack
@@ -98,7 +98,7 @@ go run ./service/runway/server
 
 ```bash
 # Go clients
-go run ./service/submitqueue/gateway/client -addr localhost:8081 -message "hello"
+go run ./service/submitqueue/gateway/client -addr localhost:8081 ping -message "hello"
 go run ./service/submitqueue/orchestrator/client -addr localhost:8082 -message "hello"
 go run ./service/stovepipe/client -addr localhost:8083 -message "hello"
 go run ./service/runway/client -addr localhost:8086 -message "hello"
@@ -110,10 +110,7 @@ make run-client-stovepipe
 make run-client-runway
 ```
 
-Client flags:
-- `-addr`: Server address (default: service-specific port)
-- `-message`: Message to send in the ping request
-- `-timeout`: Request timeout (default: 5s)
+The Gateway client is command-oriented (`ping`, `land`, `status`, `list`, and `watch`); run it without arguments for command-specific usage. The other clients are Ping clients with `-addr`, `-message`, and `-timeout` flags.
 
 ### grpcurl
 
@@ -125,17 +122,19 @@ go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest
 ```
 
 ```bash
-# Ping
-grpcurl -plaintext -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping
+# Ping services whose reflection descriptors resolve normally
 grpcurl -plaintext -d '{"message": "hello"}' localhost:8082 uber.submitqueue.orchestrator.SubmitQueueOrchestrator/Ping
 grpcurl -plaintext -d '{"message": "hello"}' localhost:8083 uber.submitqueue.stovepipe.Stovepipe/Ping
 grpcurl -plaintext -d '{"message": "hello"}' localhost:8086 uber.runway.Runway/Ping
 
-# List / describe services (reflection is registered on every server)
-grpcurl -plaintext localhost:8081 list
-grpcurl -plaintext localhost:8081 describe uber.submitqueue.gateway.SubmitQueueGateway
+# Gateway: use the repository client, or supply its proto explicitly
+go run ./service/submitqueue/gateway/client -addr localhost:8081 ping -message "hello"
+grpcurl -plaintext -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \
+  -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping
 ```
 
+The Gateway registers reflection, but its generated descriptor currently cannot resolve one imported file name. Do not use Gateway reflection-based `list` or `describe`; use the client or pass `-import-path` and `-proto` to grpcurl.
+
 ## API Reference
 
 ### Gateway Service
@@ -183,4 +182,3 @@ The gateway owns request receipts, current-status projections, and request-log r
 | Method | Description |
 |--------|-------------|
 | `Ping` | Health check |
-</content>