Problem Statement
The gateway's Postgres pool is hardcoded to max_connections(10) (crates/openshell-server/src/persistence/postgres.rs), with no config or env path. It's a single pool shared by every DB-backed RPC, so 10 is a hard, client-side ceiling on gateway↔DB concurrency — independent of how large the database is.
In cluster mode with external Postgres (workload.kind=deployment + server.externalDbSecret), a fleet of sandbox supervisors polling GetSandboxConfig / GetInferenceBundle / provider-env easily exceeds 10 concurrent connections. Symptoms:
sqlx::pool::acquire slow-threshold warnings; multi-second acquire latency on top of every DB-backed RPC./readyz becomes latency-sensitive to pool pressure, since it also acquires from this pool (PostgresStore::ping()). (This surfaced as readiness flapping in our case, but the root cause there was the default readiness-probe timeout being too short to tolerate a slow acquire — operator-tunable — not the pool itself. Noted only because it shows readiness is coupled to pool saturation.)
Scaling the database up doesn't help: the pool never opens more than 10 connections, leaving the extra capacity idle.
Proposed Design
Make the pool options configurable via environment variables, each defaulting to today's value (no change for existing deployments). Every one is read at startup and drops straight into the Helm chart as a container env: value — no image rebuild or config-file edit — consistent with OPENSHELL_DB_URL:
OPENSHELL_DB_MAX_CONNECTIONS — the pool cap (default 10). The primary fix.OPENSHELL_DB_ACQUIRE_SLOW_THRESHOLD_SECS — the sqlx slow-acquire warn threshold (currently the sqlx default of 2s), so operators can tune or quiet that log line independently of the pool size.- Optionally
OPENSHELL_DB_MIN_CONNECTIONS and OPENSHELL_DB_ACQUIRE_TIMEOUT_SECS.
Thread these into the pool builder (Store::connect → PostgresStore::connect → PgPoolOptions). They may also be mirrored in gateway.toml [openshell.gateway.database] (as log_level is today), with the env vars taking precedence.
Operator note: total connections = pool_size × replica_count, which must stay under the server's max_connections.
Agent Investigation
- Single shared pool:
Store::connect (crates/openshell-server/src/lib.rs) builds one PostgresStore/PgPool at startup.
- Hardcoded cap:
PgPoolOptions::new().max_connections(10).connect(url) (persistence/postgres.rs) — connect takes only the URL, no pool knob.
- Shared by all handlers: e.g.
handle_get_sandbox_config (grpc/policy.rs) does several store reads per request, each from the same 10 slots.
Checklist
Problem Statement
The gateway's Postgres pool is hardcoded to
max_connections(10)(crates/openshell-server/src/persistence/postgres.rs), with no config or env path. It's a single pool shared by every DB-backed RPC, so 10 is a hard, client-side ceiling on gateway↔DB concurrency — independent of how large the database is.In cluster mode with external Postgres (
workload.kind=deployment+server.externalDbSecret), a fleet of sandbox supervisors pollingGetSandboxConfig/GetInferenceBundle/ provider-env easily exceeds 10 concurrent connections. Symptoms:sqlx::pool::acquireslow-threshold warnings; multi-second acquire latency on top of every DB-backed RPC./readyzbecomes latency-sensitive to pool pressure, since it also acquires from this pool (PostgresStore::ping()). (This surfaced as readiness flapping in our case, but the root cause there was the default readiness-probe timeout being too short to tolerate a slow acquire — operator-tunable — not the pool itself. Noted only because it shows readiness is coupled to pool saturation.)Scaling the database up doesn't help: the pool never opens more than 10 connections, leaving the extra capacity idle.
Proposed Design
Make the pool options configurable via environment variables, each defaulting to today's value (no change for existing deployments). Every one is read at startup and drops straight into the Helm chart as a container
env:value — no image rebuild or config-file edit — consistent withOPENSHELL_DB_URL:OPENSHELL_DB_MAX_CONNECTIONS— the pool cap (default 10). The primary fix.OPENSHELL_DB_ACQUIRE_SLOW_THRESHOLD_SECS— the sqlx slow-acquire warn threshold (currently the sqlx default of 2s), so operators can tune or quiet that log line independently of the pool size.OPENSHELL_DB_MIN_CONNECTIONSandOPENSHELL_DB_ACQUIRE_TIMEOUT_SECS.Thread these into the pool builder (
Store::connect→PostgresStore::connect→PgPoolOptions). They may also be mirrored ingateway.toml[openshell.gateway.database](aslog_levelis today), with the env vars taking precedence.Operator note: total connections =
pool_size × replica_count, which must stay under the server'smax_connections.Agent Investigation
Store::connect(crates/openshell-server/src/lib.rs) builds onePostgresStore/PgPoolat startup.PgPoolOptions::new().max_connections(10).connect(url)(persistence/postgres.rs) —connecttakes only the URL, no pool knob.handle_get_sandbox_config(grpc/policy.rs) does several store reads per request, each from the same 10 slots.Checklist