sbalabanov · Oct 27, 2026
@@ -215,7 +215,7 @@ The `project_facts_recorded` event also projects its retained timestamp into `Re
 | Ingest | Create accepted Request, then retain accepted. | An existing Request ensures accepted before process publication. |
 | Process | CAS to superseded or processing, then retain that state. | An existing state is reconstructed from Request context before ack or build publication. |
 | Build | Create Build after runner acceptance, then retain `build_triggered`. | An identical existing Build ensures the event before buildsignal publication. |
-| Buildsignal | Persist terminal Build and retain `build_finished`; CAS the Request outcome and retain its terminal state. | Existing terminal Build and Request outcome each ensure their own entry before record publication. |
+| Buildsignal | Persist terminal Build and retain `build_finished`; CAS the Request outcome and retain its terminal state. A validation timeout retains only the terminal state entry. | Existing terminal Build and Request outcome each ensure their own entry before record publication. |
 | Record | Create or verify the whole-repository fact, then retain `validation_fact_recorded`; after recording project facts, retain one `project_facts_recorded` event. | An identical whole-repository fact and project-fact batch ensure their events before bookmark or promotion work. |
 | Record DLQ | Retain `record_abandoned`; for failed Requests, create or verify the repository fact and retain `validation_fact_recorded` before publishing its hook. | Existing facts remain authoritative; stable log and hook IDs make replay idempotent without resolving projects or replaying promotion. |
 | Build analysis DLQ | Retain `build_analysis_abandoned` without rerunning detection or changing validation state. | One stable event per build deduplicates recovery deliveries; only state reads and history retention retry. |

@@ -52,7 +52,7 @@ For a delivery carrying request id `R`:
      either domain — the shape is deferred until then, not decided here.
    - failure -> return raw; classifier decides (transient runner blip retryable, bad URI not).
 
-6. Persist Build{ID: buildID.ID, RequestID: R.ID, Status: accepted, Version: 1}
+6. Persist Build{ID: buildID.ID, RequestID: R.ID, Status: accepted, CreatedAt: now, Version: 1}
    via BuildStore.Create.
    - the row carries no scope; it is recoverable from the Request's immutable fields
      (see the entity table).
@@ -301,6 +301,7 @@ Either could be adopted independently: the idempotency token, if a backend that
 | `ID` | The build's own key — the runner-assigned id returned by `Trigger` (a Buildkite build number, a CI-gateway job id); opaque, never parsed or derived | no |
 | `RequestID` | The `Request` this build validates (`Build`→`Request` navigation) | no |
 | `Status` | `accepted / running / succeeded / failed / cancelled` | **yes** — `buildsignal` |
+| `CreatedAt` | Unix milliseconds when the row was created; zero on rows created before it was recorded | no |
 | `Version` | `int32` optimistic-locking version | **yes** — with `Status` |
 
 The row deliberately carries no scope: `R.URI`, `R.BaseURI`, and `R.BuildStrategy` — immutable and reachable through `RequestID` — fully determine what a build ran against.

@@ -59,6 +59,10 @@ For a delivery carrying build id `B`:
      - ErrVersionMismatch -> return with its declaration-level retryable classification (a concurrent writer moved the row; reload and re-check).
      - with the write-once rule, accepted -> running -> {succeeded|failed|cancelled} is monotonic
        by mechanism, not by assumption about the backend.
+   - stored status still non-terminal and max_validation_ms elapsed since Build.CreatedAt ->
+     validation timeout: the Build keeps that status, and step 7 runs without a build_finished
+     entry or terminal analysis publish, recording R failed with reason validation_timeout and
+     no terminal build id (see Fail-closed interaction).
 
 7. If the stored status is terminal:
    a. If R does not already carry an outcome, release the queue's build slot:
@@ -165,7 +169,7 @@ The window to guard is between persisting status (step 6) and ack (steps 7–8);
 
 A build that never reaches terminal `Status` — runner outage, a build the runner lost — must not wedge its `Request` forever, since callers gate deployments on greenness reaching a recorded terminal state. `buildsignal` does not implement the forcing function: per [workflow.md](../workflow.md#fail-closed-on-unprocessable-work) and the `in_flight_count` slot lifecycle in [process.md](process.md#concurrency-lifecycle), a `Request` stuck at `buildsignal` past `MaxAttempts` dead-letters, and the DLQ reconciler forces a conservative terminal `failed` and releases the Queue's slot. This is the same posture SubmitQueue's build/buildsignal pair relies on: terminal status is what releases the slot and lets validation progress.
 
-One boundary of that posture is worth stating: the `MaxAttempts` path fires only when polls *fail*. A runner that keeps answering a healthy non-terminal status forever — a hung build on a backend with no timeout of its own — never errors, so the hold chain (whose redeliveries deliberately do not count toward the retry limit) re-polls indefinitely and nothing dead-letters; SubmitQueue's poll loop shares this property. Bounding it requires a poll deadline — a `max_validation_ms` past which `buildsignal` treats the build as failed and lets the normal terminal path run — which pairs naturally with the lease idea [process.md](process.md#per-queue-concurrency-gate) floats for `in_flight_count`. Deferred with it; until then a too-old non-terminal `Build` is an operational alert, not a self-healing path.
+The `MaxAttempts` path fires only when polls *fail*. A runner that keeps answering a healthy non-terminal status — a hung build on a backend with no timeout of its own — never errors, and hold redeliveries deliberately do not count toward the retry limit, so a deadline bounds that case instead: the per-queue `max_validation_ms` (`QueueConfig.MaxValidationMs`) caps the time from `Build.CreatedAt` to a terminal status. When a poll leaves the `Build` non-terminal past the deadline, step 7 runs as for a polled failure, except that the request records the reason `validation_timeout` and no terminal build id, no `build_finished` entry is retained, and no terminal analysis is published, since the build established nothing. The `Build` keeps its last polled status, because the runner reported no failure; nothing lists non-terminal builds, and the terminal request is never reopened. A retry after a failure partway through polls again, finds the build still non-terminal and past the deadline, and takes the same path with the same reason, while step 7a–b no-op once the outcome is recorded. If the build reached a real terminal status in the meantime and the request has no outcome yet, that verdict is recorded instead. Zero disables the deadline, as does a zero `CreatedAt` on a `Build` created before creation times were recorded. The runner's build is not cancelled (see [build.md](build.md#cancellation-defined-not-yet-called)) and may keep running on its backend. SubmitQueue's poll loop has no equivalent deadline.
 
 ## Entity, storage, and queue additions
 

@@ -67,7 +67,7 @@ A slot is held from admit until the build goes terminal (`process → build →
 
 **Liveness — a stuck Request wedges the whole Queue.** The slot is shared per-Queue, so a Request admitted but never driven terminal holds the only slot and stalls the whole Queue. The fail-closed path in [workflow.md](../workflow.md#fail-closed-on-unprocessable-work) prevents this: every admitted Request terminates — `build`/`buildsignal` errors retry to `MaxAttempts`, then dead-letter, and the DLQ reconciler forces a conservative terminal `failed`, freeing the slot. Residual risk is operational: a **poison DLQ message** the reconciler can't process loops forever (it runs always-retryable), wedging that Queue until an operator removes it. The gate makes the blast radius the whole Queue, not one Request — monitor and alert on the DLQ.
 
-**Future alternative — time-bounded leases.** To self-heal instead of relying on operator cleanup, `in_flight_count` could become a set of `{owner, expires_at}` leases: the gate counts only unexpired leases, terminal transitions drop the owner's lease, and a leaked lease is reclaimed on expiry — bounding any stall to a `max_validation_ms`. A *list* of leases would also generalize to `max_concurrent > 1`. Deferred; `in_flight_count` plus the fail-closed path is enough for MVP.
+**Future alternative — time-bounded leases.** To self-heal instead of relying on operator cleanup, `in_flight_count` could become a set of `{owner, expires_at}` leases: the gate counts only unexpired leases, terminal transitions drop the owner's lease, and a leaked lease is reclaimed on expiry — bounding a stall whose request never reaches a polled build, which `buildsignal`'s `max_validation_ms` deadline cannot (see [buildsignal.md](buildsignal.md#fail-closed-interaction)). A *list* of leases would also generalize to `max_concurrent > 1`. Deferred; `in_flight_count` plus the fail-closed path is enough for MVP.
 
 `max_concurrent` is a scalar, and `= 1` is an MVP simplicity choice, **not** a correctness requirement — see [Raising `max_concurrent`](#raising-max_concurrent-speculative-validation).
 

@@ -403,7 +403,7 @@ func registerPrimaryControllers(
 	}
 	count++
 
-	buildSignalController := buildsignal.NewController(logger, scope, store, materializer, brf, registry, stovepipemq.TopicKeyBuildSignal, "stovepipe-buildsignal")
+	buildSignalController := buildsignal.NewController(logger, scope, store, materializer, queueconfigdefault.NewStore(), brf, registry, stovepipemq.TopicKeyBuildSignal, "stovepipe-buildsignal")
 	if err := c.Register(buildSignalController); err != nil {
 		return count, fmt.Errorf("failed to register buildsignal controller: %w", err)
 	}

@@ -48,6 +48,7 @@ go_test(
         "//stovepipe/extension/buildissue/mock:go_default_library",
         "//stovepipe/extension/buildrunner:go_default_library",
         "//stovepipe/extension/buildrunner/mock:go_default_library",
+        "//stovepipe/extension/queueconfig/mock:go_default_library",
         "//stovepipe/extension/storage:go_default_library",
         "//stovepipe/extension/storage/mock:go_default_library",
         "@com_github_stretchr_testify//assert:go_default_library",

@@ -21,6 +21,7 @@ package build
 import (
 	"context"
 	"fmt"
+	"time"
 
 	"github.com/uber-go/tally"
 	entityqueue "github.com/uber/submitqueue/platform/base/messagequeue"
@@ -49,6 +50,7 @@ type Controller struct {
 	registry      consumer.TopicRegistry
 	topicKey      consumer.TopicKey
 	consumerGroup string
+	now           func() time.Time
 }
 
 // _opName is the metric operation name shared by every emit in this file.
@@ -74,6 +76,7 @@ func NewController(
 		registry:      registry,
 		topicKey:      topicKey,
 		consumerGroup: consumerGroup,
+		now:           time.Now,
 	}
 }
 
@@ -143,6 +146,7 @@ func (c *Controller) Process(ctx context.Context, delivery consumer.Delivery) er
 		ID:        buildID.ID,
 		RequestID: request.ID,
 		Status:    entity.BuildStatusAccepted,
+		CreatedAt: c.now().UnixMilli(),
 		Version:   1,
 	}
 	if err := store.GetBuildStore().Create(ctx, build); err != nil {

@@ -30,6 +30,7 @@ import (
 	"github.com/uber/submitqueue/stovepipe/extension/buildissue"
 	buildissuemock "github.com/uber/submitqueue/stovepipe/extension/buildissue/mock"
 	"github.com/uber/submitqueue/stovepipe/extension/buildrunner"
+	queueconfigmock "github.com/uber/submitqueue/stovepipe/extension/queueconfig/mock"
 	"github.com/uber/submitqueue/stovepipe/extension/storage"
 	storagemock "github.com/uber/submitqueue/stovepipe/extension/storage/mock"
 	"go.uber.org/mock/gomock"
@@ -121,7 +122,7 @@ func TestBuildAnalysisPipeline(t *testing.T) {
 	analysisController := buildanalysis.NewController(zap.NewNop().Sugar(), tally.NoopScope,
 		staticStorageFactory{store: m.store}, detectorFactory, buildController.registry, stovepipemq.TopicKeyBuildAnalysis, "analysis")
 	statusController := buildsignal.NewController(zap.NewNop().Sugar(), tally.NoopScope,
-		staticStorageFactory{store: m.store}, m.materializer, m.runnerFactory, buildController.registry,
+		staticStorageFactory{store: m.store}, m.materializer, queueconfigmock.NewMockStore(ctrl), m.runnerFactory, buildController.registry,
 		stovepipemq.TopicKeyBuildSignal, "signal")
 
 	delivered := func(msg entityqueue.Message) *consumermock.MockDelivery {

@@ -20,6 +20,7 @@ import (
 	"database/sql/driver"
 	"errors"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -50,6 +51,7 @@ const (
 	testHeadURI = "git://repo/monorepo/main/head"
 	testBaseURI = "git://repo/monorepo/main/base"
 	testBuildID = "bk-1"
+	testNowMs   = int64(1700000000000)
 )
 
 func queueContext() context.Context {
@@ -105,6 +107,7 @@ func newController(t *testing.T, ctrl *gomock.Controller) (*Controller, buildMoc
 	require.NoError(t, err)
 
 	c := NewController(zap.NewNop().Sugar(), scope, staticStorageFactory{store: m.store}, m.materializer, m.runnerFactory, registry, stovepipemq.TopicKeyBuild, "stovepipe-build")
+	c.now = func() time.Time { return time.UnixMilli(testNowMs) }
 	return c, m
 }
 
@@ -225,6 +228,7 @@ func TestProcess(t *testing.T) {
 					ID:        testBuildID,
 					RequestID: testID,
 					Status:    entity.BuildStatusAccepted,
+					CreatedAt: testNowMs,
 					Version:   1,
 				}
 				createCall := m.buildStore.EXPECT().Create(gomock.Any(), build).Return(nil)
@@ -244,6 +248,7 @@ func TestProcess(t *testing.T) {
 					ID:        testBuildID,
 					RequestID: testID,
 					Status:    entity.BuildStatusAccepted,
+					CreatedAt: testNowMs,
 					Version:   1,
 				}
 				createCall := m.buildStore.EXPECT().Create(gomock.Any(), build).Return(nil)

@@ -16,6 +16,7 @@ go_library(
         "//stovepipe/core/requestlog:go_default_library",
         "//stovepipe/entity:go_default_library",
         "//stovepipe/extension/buildrunner:go_default_library",
+        "//stovepipe/extension/queueconfig:go_default_library",
         "//stovepipe/extension/storage:go_default_library",
         "@com_github_uber_go_tally//:go_default_library",
         "@org_uber_go_zap//:go_default_library",
@@ -41,6 +42,7 @@ go_test(
         "//stovepipe/entity:go_default_library",
         "//stovepipe/extension/buildrunner:go_default_library",
         "//stovepipe/extension/buildrunner/mock:go_default_library",
+        "//stovepipe/extension/queueconfig/mock:go_default_library",
         "//stovepipe/extension/storage:go_default_library",
         "//stovepipe/extension/storage/mock:go_default_library",
         "@com_github_stretchr_testify//assert:go_default_library",