Report upstream circuit breaker state in /health and /metrics

README.md

@@ -652,7 +652,7 @@ Recently cached:
 | Endpoint | Description |
 |----------|-------------|
 | `GET /` | Dashboard (web UI) |
-| `GET /health` | Health check (JSON; HTTP 200 healthy, 503 unhealthy) |
+| `GET /health` | Health check and upstream circuit breaker state (JSON; HTTP 200 healthy, 503 unhealthy) |
 | `GET /stats` | Cache statistics (JSON) |
 | `GET /metrics` | Prometheus metrics |
 | `GET /npm/*` | npm registry protocol |
@@ -895,8 +895,14 @@ The proxy exposes Prometheus metrics at `GET /metrics`. All metric names are pre
 | `proxy_storage_errors_total` | counter | `operation` | Storage read/write failures |
 | `proxy_active_requests` | gauge | | In-flight requests |
 | `proxy_health_probe_failures_total` | counter | `step` | Storage health probe failures by failing step (`write`, `size`, `read`, `verify`, `delete`). |
+| `proxy_circuit_breaker_state` | gauge | `registry` | Artifact-fetch circuit breaker state per upstream host (0 closed, 2 open). Published once that host's breaker has tripped. |
+| `proxy_circuit_breaker_trips_total` | counter | `registry` | Circuit breaker trips per upstream host. |
 
-Cache size and artifact count are refreshed every 60 seconds. The remaining metrics update on each request.
+Cache size and artifact count are refreshed every 60 seconds. Circuit breaker state is read from the fetcher on each scrape of `/metrics` and each `/health` request, so `proxy_circuit_breaker_trips_total` counts the trips visible between those reads — a breaker that opens and recovers entirely between two scrapes is not counted. The remaining metrics update on each request.
+
+The breaker metrics carry one series per upstream host, but only for hosts whose breaker has tripped at least once since startup. A breaker is created per host the proxy fetches artifacts from, and for some ecosystems that host comes from upstream metadata rather than from configuration (composer takes it from a package's `dist.url`, helm from the chart URLs in `index.yaml`), so publishing every host would let upstream content grow the series count for the lifetime of the process. Once a host has tripped it keeps reporting, so a recovery still shows up as a transition to 0 rather than as a series that vanishes. `/health` is not a persistent time series and lists every breaker, tripped or not.
+
+Alert on `proxy_circuit_breaker_state == 2` sustained for more than a few minutes: while a breaker is open, artifact downloads for that upstream fail with HTTP 502 on every cache miss, and only a single probe request per backoff interval reaches the upstream. Cached artifacts keep serving, and so does metadata for the same ecosystem (metadata does not go through the circuit breaker), so installs fail in a way that looks like a partial upstream outage.
 
 ### Health Check
 
@@ -908,12 +914,20 @@ Cache size and artifact count are refreshed every 60 seconds. The remaining metr
   "checks": {
     "database": {"status": "ok"},
     "storage":  {"status": "ok"}
+  },
+  "circuit_breakers": {
+    "registry.npmjs.org": "closed",
+    "static.crates.io":   "open"
   }
 }
 ```
 
 Failing checks include an `"error"` field. Storage failures also include a `"step"` field identifying which probe step failed (`write`, `size`, `read`, `verify`, `delete`). When the database check fails, the storage entry reports `{"status": "skipped"}` so the response always carries the same key set.
 
+`circuit_breakers` reports the state of each upstream's artifact-fetch circuit breaker (`"open"` or `"closed"`), keyed by upstream host. The key is omitted until the proxy has fetched an artifact from at least one upstream, and a host appears only once a breaker has been created for it. Breakers trip after repeated upstream failures and retry the upstream after an exponential backoff. While one is open, artifact downloads for that host return HTTP 502 on a cache miss without contacting the upstream; already-cached artifacts are still served from storage, since the cache is checked before the fetcher. A breaker is reported as `"open"` throughout its backoff, including the half-open window in which it admits one probe request to test recovery. Breaker state is per process and in memory, so a restart always clears it — worth knowing when a breaker stays open after the upstream has recovered.
+
+An open breaker does **not** set `status` to `"error"` or change the HTTP status code: it reports a specific upstream refusing to serve, not this proxy being unfit to receive traffic, and failing the readiness probe over one unhealthy upstream would pull the pod out of rotation for every other ecosystem too. Use `proxy_circuit_breaker_state` for alerting on it.
+
 Storage probe results are cached for `health.storage_probe_interval` (default 30s) to bound the cost of probing remote backends. A probe holds an internal mutex for up to 10 seconds (the hardcoded per-probe timeout), so `/health` is intended as a Kubernetes **readiness** probe rather than a liveness probe — a slow S3 round-trip should pull the pod from rotation, not restart it.
 
 Scrape config for Prometheus:

docs/architecture.md

@@ -279,7 +279,7 @@ HTTP server setup, web UI, and API handlers.
 - Web UI under `/ui`: dashboard, package browser, source browser, version comparison
 - Templates are embedded in the binary via `//go:embed`
 - Enrichment API for package metadata, vulnerability scanning, and outdated detection
-- Health, stats, and Prometheus metrics endpoints. `/health` runs an active write → size-check → read → verify → delete probe against the storage backend and returns a structured JSON response (`HealthResponse`) with `"ok"` / `"error"` status per subsystem. Probe results are cached (default 30 s, configurable via `health.storage_probe_interval`) to avoid overwhelming remote backends.
+- Health, stats, and Prometheus metrics endpoints. `/health` runs an active write → size-check → read → verify → delete probe against the storage backend and returns a structured JSON response (`HealthResponse`) with `"ok"` / `"error"` status per subsystem. Probe results are cached (default 30 s, configurable via `health.storage_probe_interval`) to avoid overwhelming remote backends. The response also carries a `circuit_breakers` map reporting each upstream host's artifact-fetch breaker as `"open"` or `"closed"`; the same state is published as the `proxy_circuit_breaker_state` gauge on each `/metrics` scrape. An open breaker leaves the overall status `"ok"` — it describes an upstream, not this proxy.
 
 ### `internal/metrics`
 

docs/swagger/docs.go

@@ -538,6 +538,13 @@ const docTemplate = `{
                         "$ref": "#/definitions/server.HealthCheck"
                     }
                 },
+                "circuit_breakers": {
+                    "description": "CircuitBreakers reports the state (\"open\" or \"closed\") of each upstream\nregistry's artifact-fetch circuit breaker, omitted when no breaker has\nbeen created yet. An open breaker fails every artifact fetch for that\nhost without contacting it, but says nothing about this proxy's own\nhealth, so it does not change Status.",
+                    "type": "object",
+                    "additionalProperties": {
+                        "type": "string"
+                    }
+                },
                 "status": {
                     "type": "string"
                 }

docs/swagger/swagger.json

@@ -531,6 +531,13 @@
                         "$ref": "#/definitions/server.HealthCheck"
                     }
                 },
+                "circuit_breakers": {
+                    "description": "CircuitBreakers reports the state (\"open\" or \"closed\") of each upstream\nregistry's artifact-fetch circuit breaker, omitted when no breaker has\nbeen created yet. An open breaker fails every artifact fetch for that\nhost without contacting it, but says nothing about this proxy's own\nhealth, so it does not change Status.",
+                    "type": "object",
+                    "additionalProperties": {
+                        "type": "string"
+                    }
+                },
                 "status": {
                     "type": "string"
                 }

internal/server/breakers.go

@@ -0,0 +1,111 @@
+package server
+
+import (
+	"log/slog"
+	"sync"
+
+	"github.com/git-pkgs/proxy/internal/metrics"
+)
+
+// Gauge values for proxy_circuit_breaker_state. The fetcher reports only open
+// or closed, so half-open (1) is never published.
+const (
+	breakerGaugeClosed = 0
+	breakerGaugeOpen   = 2
+)
+
+const (
+	breakerStateOpen   = "open"
+	breakerStateClosed = "closed"
+)
+
+// breakerStateSource reports circuit breaker state per registry host, keyed by
+// host, with values breakerStateOpen or breakerStateClosed. Implemented by
+// fetch.CircuitBreakerFetcher.
+type breakerStateSource interface {
+	GetBreakerState() map[string]string
+}
+
+// breakerMonitor mirrors the artifact fetcher's per-registry circuit breaker
+// state into Prometheus metrics, the health report, and the log.
+//
+// Breaker state lives only in the fetcher's memory. While a breaker is open
+// every artifact fetch for that host that misses the cache fails without
+// reaching the upstream, which looks identical to an upstream outage from the
+// outside: metadata still serves (it does not go through the fetcher), other
+// registries still serve, and /health reports the database and storage as fine.
+// Publishing the state makes that distinguishable.
+type breakerMonitor struct {
+	source breakerStateSource
+	logger *slog.Logger
+
+	// mu guards seen, which holds one entry per registry that has tripped at
+	// least once in this process — the only registries published as metrics.
+	mu   sync.Mutex
+	seen map[string]string
+}
+
+func newBreakerMonitor(source breakerStateSource, logger *slog.Logger) *breakerMonitor {
+	if logger == nil {
+		logger = slog.Default()
+	}
+	return &breakerMonitor{
+		source: source,
+		logger: logger,
+		seen:   map[string]string{},
+	}
+}
+
+// snapshot returns the current state of every breaker the fetcher has created,
+// keyed by registry host, and mirrors it into the breaker metrics as a side
+// effect. It returns nil for a nil monitor so callers that build a Server
+// without a fetcher (tests) need no special case.
+//
+// Only registries that have tripped at least once are published as metrics.
+// The fetcher creates a breaker per host it fetches from, and for some
+// ecosystems that host comes from upstream metadata rather than configuration
+// (composer takes it from a package's dist.url, helm from the chart URLs in
+// index.yaml), so publishing every host would let upstream content grow the
+// series count for the life of the process. A host that has never tripped
+// carries no information a series could convey; once it trips it keeps
+// reporting, including the 0 that marks its recovery. /health is a per-request
+// response rather than a persistent series, so it reports every breaker.
+//
+// Trips are counted on the closed→open transitions observed between calls,
+// because the fetcher exposes current state rather than trip events: a breaker
+// that opens and recovers entirely between two calls is not counted.
+func (m *breakerMonitor) snapshot() map[string]string {
+	if m == nil || m.source == nil {
+		return nil
+	}
+
+	states := m.source.GetBreakerState()
+
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	for registry, state := range states {
+		previous, published := m.seen[registry]
+
+		switch {
+		case state == breakerStateOpen && previous != breakerStateOpen:
+			metrics.RecordCircuitBreakerTrip(registry)
+			m.logger.Error("circuit breaker open, artifact fetches for this registry "+
+				"fail without contacting it", "registry", registry)
+		case state == breakerStateClosed && previous == breakerStateOpen:
+			m.logger.Info("circuit breaker closed", "registry", registry)
+		case state == breakerStateClosed && !published:
+			// Never tripped: nothing to publish.
+			continue
+		}
+
+		gauge := breakerGaugeClosed
+		if state == breakerStateOpen {
+			gauge = breakerGaugeOpen
+		}
+		metrics.UpdateCircuitBreakerState(registry, gauge)
+		m.seen[registry] = state
+	}
+
+	return states
+}