Skip to content

Serve the Prometheus metrics summary like published PHP 2.11 - #685

Closed
rmcdaniel wants to merge 2 commits into
rust-325-system-metricsfrom
rust-325-prometheus-metrics
Closed

rmcdaniel wants to merge 2 commits into
rust-325-system-metricsfrom
rust-325-prometheus-metrics

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Part of #325. Stacked on #684 (system metrics), which is stacked on train #675.

Summary

Native now serves GET /api/system/prometheus-metrics as published Server 2.11 does (PrometheusMetricsSummary::snapshot). Prometheus exporters scrape it for:

  • workflow series by task queue and workflow type: started, completed, failed, cancelled and terminated totals, running runs, and a latency histogram;
  • activity series by task queue, workflow type and activity type, with the same counters and histogram;
  • task-queue state: ready, due, delayed, leased and expired tasks, tasks added and dispatched in the last minute, and fresh worker pollers and slots.

Each family keeps the first 100 series in PHP's order. The summary discloses observed, reported and suppressed series and the suppressed starts, and switches to lower_bound precision at the limit. Native refuses the series-limit settings, so the limit is always PHP's default. Task and worker times compare in whole seconds, as Laravel binds them.

Decisions

  • Cumulative latency buckets. PHP binds each bucket bound as text. SQLite orders every number before every text value, so PHP on SQLite counts every duration in every bucket. That is filed as Prometheus latency buckets on SQLite count every duration in every bucket #683 and handed to General. Native compares with numeric bounds and reports the cumulative histogram on every backend. The fixture records PHP's SQLite shape as a reference defect without adopting it. Native tests pin the buckets with fixed durations on SQLite, PostgreSQL and MySQL.
  • Undispatched activity tasks. PHP's worker completion publishes only the timer tasks it creates. An activity task scheduled by a workflow task therefore keeps null last_dispatched_at and last_dispatch_attempt_at, and its queue's dispatched counter stays 0. Native stamped it dispatched. It now leaves those columns null for that path, as PHP does. Other task paths keep their stamps.

Fixture first

The prometheus-metrics fixture reads a fresh namespace's summary, then runs a scenario and reads it again:

  • a worker with four workflow and three activity slots registers on the first queue;
  • a run there completes after one activity;
  • a second run there is terminated;
  • a run waits on a second queue with no worker.

Counters, task-queue state, worker capacity and the disclosure are exact. Each counted run and activity lasts at least 1.1 s, so latency sums and the empty low buckets are checked against that bound.

Verification

  • Local three-way parity: all 53 fixtures pass on embedded PHP and native, including prometheus-metrics. Published PHP passed prometheus-metrics as well. Its only difference was storage-test, an intermittent 503 from PHP's payload registration on the loaded host; that fixture passed in the previous local run.
  • Native suite on SQLite with --all-targets: 649 passed. The new tests pass on PostgreSQL and MySQL too.
  • Contract unit test: 22 cases, including both histogram shapes and mutations.

GET /api/system/prometheus-metrics reports the namespace's workflow,
activity and task-queue series with latency histograms, task state and
worker capacity, each family capped at 100 series with its suppression
disclosed, as PrometheusMetricsSummary does.

Latency buckets compare durations with numeric bounds on every backend.
Published PHP on SQLite counts every duration in every bucket
(server#683); the fixture records that reference defect without adopting
it.

As in PHP, an activity task scheduled by a workflow task is no longer
stamped dispatched: PHP's completion publishes only timer tasks.
@rmcdaniel

Copy link
Copy Markdown
Member Author

Superseded by #696, which carries this change unchanged, rebased on main. Its tests pass in #696 on SQLite, PostgreSQL and MySQL. The branch stays until #696 merges and this head is in main.

@rmcdaniel rmcdaniel closed this Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant