Skip to content

Serve external payload cleanup and its pass like published PHP 2.11 - #686

Closed
rmcdaniel wants to merge 3 commits into
rust-325-prometheus-metricsfrom
rust-325-payload-cleanup
Closed

rmcdaniel wants to merge 3 commits into
rust-325-prometheus-metricsfrom
rust-325-payload-cleanup

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Part of #325. Stacked on #685 (Prometheus summary), on #684, on train #675.

Summary

Native now serves the external payload cleanup routes as published Server 2.11 does. Operators use them to reclaim abandoned payload uploads.

  • GET /api/system/external-payload-cleanup: the namespace's cleanup metrics (expired backlog, batch bounds, totals and last pass). Native already reported these inside operator metrics.
  • POST /api/system/external-payload-cleanup/pass takes limit (nullable|integer|min:1|max:1000, default 100). It then:
    1. visits expired completion-upload allowances. One whose activity can still complete (task leased to the same owner, attempt able to continue) gets another abandoned-upload period. Any other is dropped.
    2. reclaims expired, unclaimed references, oldest first, each under its object lock. A shared object loses only the reference's row. A sole reference also deletes its object.
    3. reports processed, deleted, preserved, skipped and blocked counts, backlog and scan pressure, and answers 207 when anything was blocked.
    4. adds the pass to the namespace's totals, with whole-second times as Eloquent stores them.

Decisions

  • Native stores objects only with the local driver. A reference without an enabled local policy, or whose object lies outside the current root, is blocked rather than deleted.
  • Native never takes an online-backup hold. A hold row left active blocks deletion, as in PHP. A missing row means no hold. (PHP's migration seeds that row, so PHP would report a storage failure.)
  • An allowance whose context native cannot read (native holds activity contexts only) cannot resume, so it is dropped once expired.

Fixture first, and probe hardening

The payload-cleanup fixture covers the status, passes, recorded totals and limit validation. Every answer was checked against a fresh published 2.11.1 server. Deletion needs expired rows and stored objects, which an HTTP probe cannot create within the abandoned-upload period. Native tests on SQLite, PostgreSQL and MySQL cover it instead: shared, live and retained references, the limit and backlog, an active backup hold, a disabled policy, and an allowance renewed while its activity runs and dropped after it completes.

Locally, published PHP on SQLite refused a namespace delete with its retryable 503 backend_lock_pressure in two of four parity runs, and a storage diagnostic with a retryable 503 in two. The namespace-delete and payload-cleanup probes now retry lock pressure through the shared middleware. The storage diagnostic retries a retryable 503 up to ten times.

Verification

  • Local three-way parity: all 54 fixtures pass on embedded PHP and native, including payload-cleanup. Published PHP differed only on namespace-delete (the 503 above, before the retry).
  • Native suite on SQLite with --all-targets: 652 passed. The new tests pass on PostgreSQL and MySQL too.
  • Contract unit test: 17 cases, including mutations.

GET /api/system/external-payload-cleanup reports the namespace's cleanup
metrics. POST .../pass renews expired completion-upload allowances whose
activity can still complete and drops the rest, then reclaims the
namespace's expired, unclaimed references under their object locks: a
shared object keeps its file, a sole reference deletes it. Each pass is
recorded in the namespace's totals; a blocked reference answers 207.

Native deletes through the local driver only, honours a backup hold that
PHP left active, and treats a missing hold row as no hold.
Locally, published PHP on SQLite refused a namespace delete with its
retryable 503 backend_lock_pressure in two of four parity runs, and a
storage diagnostic with a retryable 503 in two. The namespace-delete and
payload-cleanup probes now retry lock pressure through the shared
middleware, and the storage diagnostic retries a retryable 503 up to ten
times.
@rmcdaniel

Copy link
Copy Markdown
Member Author

Superseded by #696, which carries this change unchanged, rebased on main. Its tests pass in #696 on SQLite, PostgreSQL and MySQL. The branch stays until #696 merges and this head is in main.

@rmcdaniel rmcdaniel closed this Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant