Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions docs/polyglot/server-config-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,74 @@ when you configure `DW_V2_MATCHING_ROLE_QUEUE_WAKE` or a dedicated
`workflow:v2:repair-pass --loop` daemon. Those settings change where
ready-task discovery runs, not the worker-protocol contract itself.

## Additional published PHP controls {#additional-published-php-controls}

This reference covers the published **PHP Server 2.11.0** contract, including the controls below. Check the image you actually run: the independently selected installation version can be older. `unset` means no environment value is supplied; its effect depends on the control. Defaults and aliases come from the [published contract](https://github.com/durable-workflow/server/blob/36d16273604755f3dd0743cab331cc7c62e51d8e/config/dw-contract.php). The versioned contract snapshot is checked against every locale during the documentation build. This reference does not change the selected installation tuple.

### Namespace HTTP admission {#namespace-request-controls}

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_NAMESPACE_MAX_REQUESTS_PER_MINUTE` | `unset` | Optional per-minute control-plane request cap for each namespace. Unset means unlimited. | `WORKFLOW_SERVER_NAMESPACE_MAX_REQUESTS_PER_MINUTE` |
| `DW_NAMESPACE_MAX_CONCURRENT_REQUESTS` | `unset` | Optional concurrent control-plane request cap for each namespace. Unset means unlimited. | `WORKFLOW_SERVER_NAMESPACE_MAX_CONCURRENT_REQUESTS` |
| `DW_NAMESPACE_HARD_MAX_REQUESTS_PER_MINUTE` | `unset` | Non-bypassable ceiling for namespace control-plane requests per minute, including namespace overrides. | `WORKFLOW_SERVER_NAMESPACE_HARD_MAX_REQUESTS_PER_MINUTE` |
| `DW_NAMESPACE_HARD_MAX_CONCURRENT_REQUESTS` | `unset` | Non-bypassable ceiling for concurrent namespace control-plane requests, including namespace overrides. | `WORKFLOW_SERVER_NAMESPACE_HARD_MAX_CONCURRENT_REQUESTS` |
| `DW_NAMESPACE_ADMISSION_OVERRIDES` | `{}` | JSON object keyed by namespace with optional request-rate and concurrent-request overrides. | `WORKFLOW_SERVER_NAMESPACE_ADMISSION_OVERRIDES` |

[Namespace request admission](./task-queue-admission.md#namespace-request-admission).

### Namespace durable-state quotas {#namespace-durable-controls}

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_NAMESPACE_DURABLE_LIMITS` | `{}` | JSON object of default per-namespace durable row limits across workflow instances, runs, history, tasks, timers, waits, commands, schedules, workers, and streams. Lifetime and pending/open limits are independent; pending workflow tasks count queued ready rows because active leases have separate admission limits. An empty object is unlimited. | `WORKFLOW_SERVER_NAMESPACE_DURABLE_LIMITS` |
| `DW_NAMESPACE_DURABLE_HARD_LIMITS` | `{}` | JSON object of non-bypassable ceilings for the namespace durable row limit fields, including per-namespace overrides. An empty object adds no operator hard ceilings. | `WORKFLOW_SERVER_NAMESPACE_DURABLE_HARD_LIMITS` |
| `DW_NAMESPACE_DURABLE_OVERRIDES` | `{}` | JSON object keyed by namespace with optional namespace durable row limit overrides. Hard limits remain authoritative. | `WORKFLOW_SERVER_NAMESPACE_DURABLE_OVERRIDES` |

[Namespace durable-state quotas](./task-queue-admission.md#namespace-durable-state).

### External payload quotas and S3 transport {#external-payload-quotas-and-s3}

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_EXTERNAL_PAYLOAD_COMPLETION_MAX_BYTES` | `unset` | Distinct bytes allowed across immutable upload slots for one leased completion; unset uses DW_EXTERNAL_PAYLOAD_MAX_BYTES. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_COMPLETION_MAX_BYTES` |
| `DW_EXTERNAL_PAYLOAD_MAX_BYTES_PER_NAMESPACE` | `unset` | Optional cumulative external-payload byte cap for each namespace. Ready and in-progress objects count toward the cap; unset means unlimited. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_MAX_BYTES_PER_NAMESPACE` |
| `DW_EXTERNAL_PAYLOAD_MAX_OBJECTS_PER_NAMESPACE` | `unset` | Optional external-payload object-count cap for each namespace. Ready and in-progress objects count toward the cap; unset means unlimited. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_MAX_OBJECTS_PER_NAMESPACE` |
| `DW_EXTERNAL_PAYLOAD_HARD_MAX_BYTES_PER_NAMESPACE` | `unset` | Non-bypassable cumulative external-payload byte ceiling for each namespace, including namespace overrides. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_HARD_MAX_BYTES_PER_NAMESPACE` |
| `DW_EXTERNAL_PAYLOAD_HARD_MAX_OBJECTS_PER_NAMESPACE` | `unset` | Non-bypassable external-payload object-count ceiling for each namespace, including namespace overrides. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_HARD_MAX_OBJECTS_PER_NAMESPACE` |
| `DW_EXTERNAL_PAYLOAD_NAMESPACE_OVERRIDES` | `{}` | JSON object keyed by namespace with optional max_bytes and max_objects external-payload quota overrides. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_NAMESPACE_OVERRIDES` |
| `DW_EXTERNAL_PAYLOAD_S3_ACCESS_KEY_ID` | `unset` | Optional access-key ID for the runtime-managed S3-compatible external-payload disk. May be omitted when workload identity supplies credentials. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_ACCESS_KEY_ID` |
| `DW_EXTERNAL_PAYLOAD_S3_SECRET_ACCESS_KEY` | `unset` | Optional secret access key for the runtime-managed S3-compatible external-payload disk. Must be set with the access-key ID. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_SECRET_ACCESS_KEY` |
| `DW_EXTERNAL_PAYLOAD_S3_SESSION_TOKEN` | `unset` | Optional session token for temporary credentials on the runtime-managed S3-compatible external-payload disk. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_SESSION_TOKEN` |
| `DW_EXTERNAL_PAYLOAD_S3_REGION` | `us-east-1` | Region used by the runtime-managed S3-compatible external-payload disk. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_REGION` |
| `DW_EXTERNAL_PAYLOAD_S3_BUCKET` | `unset` | Bucket used by the runtime-managed S3-compatible external-payload disk. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_BUCKET` |
| `DW_EXTERNAL_PAYLOAD_S3_ENDPOINT` | `unset` | Optional HTTPS endpoint for an S3-compatible external-payload service. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_ENDPOINT` |
| `DW_EXTERNAL_PAYLOAD_S3_USE_PATH_STYLE_ENDPOINT` | `false` | Use path-style bucket addressing for the runtime-managed S3-compatible external-payload service. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_USE_PATH_STYLE_ENDPOINT` |

### Caller-namespace service boundaries {#service-caller-namespace-controls}

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_SERVICE_BOUNDARY_NAMESPACE_RATE_LIMIT_PER_MINUTE` | `unset` | Optional shared per-minute service-call limit aggregated across each caller namespace. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_NAMESPACE_RATE_LIMIT_PER_MINUTE` |
| `DW_SERVICE_BOUNDARY_NAMESPACE_MAX_IN_FLIGHT` | `unset` | Optional shared in-flight service-call limit aggregated across each caller namespace. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_NAMESPACE_MAX_IN_FLIGHT` |
| `DW_SERVICE_BOUNDARY_HARD_RATE_LIMIT_PER_MINUTE` | `unset` | Non-bypassable per-minute ceiling for service-call boundary and caller-namespace policies. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_HARD_RATE_LIMIT_PER_MINUTE` |
| `DW_SERVICE_BOUNDARY_HARD_MAX_IN_FLIGHT` | `unset` | Non-bypassable in-flight ceiling for service-call boundary and caller-namespace policies. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_HARD_MAX_IN_FLIGHT` |
| `DW_SERVICE_BOUNDARY_NAMESPACE_OVERRIDES` | `{}` | JSON object of caller-namespace service-call rate and in-flight overrides; hard ceilings still apply. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_NAMESPACE_OVERRIDES` |

### Coordination, storage and runtime {#coordination-storage-and-runtime-controls}

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_OBSERVABILITY_CONTEXT_ENABLED` | `false` | Enable optional bounded trace context persistence, delivery and capability discovery when its optional API and schema are installed. | - |
| `DW_OBSERVABILITY_BAGGAGE_ALLOWLIST` | `[]` | JSON list of baggage keys permitted on run origins; default empty. Links never carry baggage. | - |
| `DW_STORAGE_ADMISSION_FILE` | `unset` | Optional read-only local storage-pressure observation file; invalid or stale observations fence writes. | - |
| `DW_STORAGE_ADMISSION_SOURCE` | `unset` | Expected storage resource identity in the admission observation. | - |
| `DW_STORAGE_ADMISSION_MAX_AGE_SECONDS` | `15` | Maximum age of a storage observation in seconds (1-300). | - |
| `DW_RUNTIME_CREDENTIALS_ENABLED` | `false` | Enable durable, database-backed, namespace-bound runtime bearer credentials managed through the admin API. | `WORKFLOW_SERVER_RUNTIME_CREDENTIALS_ENABLED` |
| `DW_WORKER_LONG_POLL_MAX_CONCURRENT_PER_NAMESPACE` | `unset` | Optional per-node cap for concurrent held workflow/activity long-poll waits from one namespace. The effective cap cannot exceed the global worker wait cap. Configured namespace isolation fails closed when shared cache authority is unavailable. | `WORKFLOW_SERVER_WORKER_LONG_POLL_MAX_CONCURRENT_PER_NAMESPACE` |
| `DW_QUERY_TASK_POLL_MAX_CONCURRENT_PER_NAMESPACE` | `unset` | Optional per-node cap for concurrent held query-task long-poll waits from one namespace. The effective cap cannot exceed the global query-task wait cap. Configured namespace isolation fails closed when shared cache authority is unavailable. | `WORKFLOW_SERVER_QUERY_TASK_POLL_MAX_CONCURRENT_PER_NAMESPACE` |
| `DW_V1_ENABLED` | `false` | Enable legacy embedded v1 migration and queue-drain behavior. Standalone Server defaults to v2-only; enable only when deliberately preserving v1 state. | `WORKFLOW_V1_ENABLED` |

## Runtime Infrastructure Variables

Runtime infrastructure variables are framework and dependency controls. The
Expand Down
61 changes: 61 additions & 0 deletions docs/polyglot/task-queue-admission.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,67 @@ An open application breaker raises `NonRetryableError` before the payment call;

Keep protected downstream calls in **ordinary queued activities**. Inline local activities create no ordinary activity task, so queue, namespace and budget-group activity dispatch caps do not meter them. A prepared local-activity claim validates execution authority, not this dispatch budget. Embedded Laravel mode has no standalone Server admission; use its own deployment controls. See [local activities](../features/local-activities.md).

## Namespace request admission {#namespace-request-admission}

The following contract is for published **PHP Server 2.11.0**. Check your actual image and the [versioned configuration reference](./server-config-reference.md#additional-published-php-controls). Namespace HTTP budgets are separate from worker slots, task dispatch, held worker polls and durable-state quotas. The HTTP budget belongs to the request namespace resolved from `Durable-Workflow-Namespace` or `DW_DEFAULT_NAMESPACE`, including namespace-management requests whose path names another namespace.

Authenticated requests that reach namespace admission count on these control and management route families, including their descendants: `/api/namespaces`, `/api/storage`, `/api/workflows`, `/api/activities`, `/api/bridge-adapters`, `/api/workers`, `/api/worker-sessions`, `/api/task-queues`, `/api/deployments`, `/api/schedules`, `/api/search-attributes`, `/api/service-endpoints`, `/api/service-catalog`. Plural `/api/workers` management requests count.

Singular `/api/worker/*` registration, polling and outcome routes do not use this HTTP budget. Health/readiness, `/api/cluster/info`, admin `/api/system/*`, runtime credentials and external-payload transport also stay outside this gate; their own authentication, capacity and storage checks still apply. Requests refused earlier by authentication, protocol or namespace resolution never reach this counter.

Rate admission uses fixed server-clock minute buckets, not a smooth or rolling rate. A budget of N can admit N requests just before a boundary and another N just after it. Requests that fail later still count; rate admission happens before the concurrency check, so a concurrency refusal can consume minute budget. A concurrency slot remains held until the request finishes and cleanup releases it; it does not count running workflows.

Across API replicas, use the same shared Redis store/key prefix, synchronized clocks and consistent loaded configuration. The shared namespace budget is not multiplied by adding replicas. A local file cache only coordinates processes with compatible writable access to its files; it is not the multi-node baseline. Per-node held worker/query-poll caps remain separate. [Restart affected roles after environment changes](./server-config-reference.md#how-changes-take-effect).

```dotenv
DW_NAMESPACE_MAX_REQUESTS_PER_MINUTE=100
DW_NAMESPACE_MAX_CONCURRENT_REQUESTS=8
DW_NAMESPACE_HARD_MAX_REQUESTS_PER_MINUTE=150
DW_NAMESPACE_HARD_MAX_CONCURRENT_REQUESTS=10
DW_NAMESPACE_ADMISSION_OVERRIDES='{"payments":{"max_requests_per_minute":200,"max_concurrent_requests":12},"reports":{"max_requests_per_minute":40,"max_concurrent_requests":3}}'
```

For existing namespaces, this example resolves `payments` to 150 requests/minute and 10 concurrent requests, `reports` to 40/3, and other namespaces to 100/8. Overrides can raise or lower defaults, but the 150/10 hard caps still apply. This configuration does not create namespaces or reserve throughput.

For these HTTP controls, unset means no corresponding configured limit; a hard cap alone can still impose a limit. A null override falls back to the default. Values must be positive integers: zero is invalid. PHP implementation ceilings are 10000000 requests/minute and 1024 concurrent requests, independent of the actual HTTP pool capacity. Published PHP 2.11.0 silently treats malformed `DW_NAMESPACE_ADMISSION_OVERRIDES` JSON as an empty override map, falling back to global defaults. Validate the JSON before deployment; if the defaults and hard caps are unset, the intended namespace cap is lost. This [confirmed product gap](https://github.com/durable-workflow/durable-workflow.github.io/issues/335#issuecomment-6369150521) remains separate from the documentation update.

| Reason | HTTP | Retry hint in seconds |
| --- | --- | --- |
| `namespace_request_rate_exhausted` | `429` | Until the next minute, 1–60 |
| `namespace_request_concurrency_exhausted` | `429` | 1 |
| `namespace_request_admission_unavailable` | `503` | 1 |

The JSON `retry_after_seconds` and `Retry-After` header agree. Unavailable configured coordination or invalid numeric limits fails closed with 503. Inspect namespace-scoped admin metrics even when ordinary requests are exhausted; `dw_namespace_request_admission_rejections` reports the finite reasons. Apply bounded caller retry/backoff policy and preserve idempotency; an uncertain transport outcome is not permission to resubmit a write automatically. See [timeout budgets](../deployment.md#worker-long-poll-timeout-budgets).

## Namespace durable-state quotas {#namespace-durable-state}

`DW_NAMESPACE_DURABLE_LIMITS` sets default retained-row limits, `DW_NAMESPACE_DURABLE_OVERRIDES` changes them per namespace, and `DW_NAMESPACE_DURABLE_HARD_LIMITS` adds non-bypassable ceilings. Empty objects add no limits. Overrides can raise or lower defaults; null falls back to the default. Unlike HTTP admission, a durable limit of zero is valid and prevents new allocations. In this example, existing `payments` can hold 250 open runs, other namespaces 100, under a 500 hard ceiling.

```dotenv
DW_NAMESPACE_DURABLE_LIMITS='{"max_open_workflow_runs":100,"max_open_workflow_run_waits":200,"max_pending_workflow_timers":200}'
DW_NAMESPACE_DURABLE_HARD_LIMITS='{"max_open_workflow_runs":500}'
DW_NAMESPACE_DURABLE_OVERRIDES='{"payments":{"max_open_workflow_runs":250}}'
```

Use `max_<resource>` fields for the resources below. Counts belong to the namespace owning each row, or its linked workflow/run. Total retained rows and open/pending subsets are independent limits. Enforcement uses database-authoritative counts while serializing mutations on the namespace row.

| Resource | What counts |
| --- | --- |
| `workflow_instances`, `workflow_runs` | All retained workflow instance/run rows, including terminal runs. |
| `open_workflow_runs` | Runs with `pending`, `running` or `waiting` status. |
| `workflow_history_events`, `workflow_commands`, `workflow_tasks` | Retained history, command and workflow-task rows. |
| `pending_workflow_tasks` | Queued workflow-task rows with `ready` status; active leases have separate admission controls. |
| `workflow_timers`, `pending_workflow_timers` | All retained timers, and the `pending` subset respectively. |
| `workflow_run_waits`, `open_workflow_run_waits` | All retained run waits, and the `open` subset respectively. |
| `schedules`, `schedule_history_events`, `worker_registrations` | Retained schedules, schedule history and worker-registration rows; stale registrations count until removed. |
| `workflow_streams`, `workflow_stream_items` | Durable and inbound stream rows, and their item rows. |

A parent waiting for a child or service call still consumes an open run in its own namespace. A child adds its own run; in the same namespace, plan for both open runs and their waits/timers. A service operation that creates a workflow consumes rows in that workflow’s namespace; signal/update operations on an existing run do not automatically create a second run. Service-call boundary and external-payload quotas are separate controls.

Exhaustion returns 429 `namespace_<resource>_exhausted`. Read `retryable`: some quotas are nonretryable and provide no retry hint; retryable durable refusals advertise 60 seconds, without promising capacity then. Failure to evaluate a configured quota returns 503 `namespace_durable_state_quota_unavailable` with a one-second hint. Retained-row pressure may require retention or a capacity/configuration change, not a tight retry loop.

Select the namespace with the request header when reading admin metrics. `dw_namespace_durable_state_usage` and the JSON `limits`, `usage`, `remaining`, `configuration_status` and `measurement_status` expose that scope; namespace is request scope rather than an unbounded metric label. Unavailable measurements are not zero usage or unlimited capacity. See the [complete configuration reference](./server-config-reference.md#namespace-durable-controls).

## Server Configuration

Set global caps when every queue should share the same ceiling:
Expand Down
Loading
Loading