[reatang]

Pull Request

Description

Operators could not tell from the Site Replication page whether replication was actually working. The server already reports peer reachability, object replication failures, queue depth, undelivered configuration changes, unfinished lifecycle operations, and configuration drift between sites, but the page only rendered configuration fields. Worse, three fields read like health signals while carrying none:

• The black "Enabled" badge in the Sync column is the per-peer sync setting, not link health. It stays "Enabled" while the peer is down.
• Replicated Objects / Replicated Size are in-memory counters of the node that answered the request. They reset on restart, cover only that node, and are not limited to site replication targets.
• Last Online was read from the local site's own entry, so it always showed the current time.

This PR is the P0 (console-only) part of rustfs/backlog#2922. All data comes from endpoints the page already calls (site-replication/info and site-replication/status); no server change is needed.

• Replication Health summary at the top: Healthy / Needs attention / Unhealthy / Unknown, plus the reasons. An unreachable peer is Unhealthy. An unfinished lifecycle operation (site removal, endpoint update, service account rotation), undelivered configuration changes, object replication failures in the last hour, or configuration drift is Needs attention. Long server error strings are clamped to two lines and shown in full on hover.
• Connection column in the peer table: Online / Offline / Unknown, with last-online time and accumulated downtime for offline peers. The sync column is renamed Sync Setting and rendered as plain text.
• Failures and Backlog section: current and peak queue, failures in the last hour and since start, active workers, and pending/failed configuration deliveries, labelled as node-local counters.
• Consistency list under Replicated Scope: buckets, policies, users, groups, and ILM expiry rules that are missing on a site or differ between sites, with the differing settings. The comparison is skipped while any peer is unreachable, because an unreachable peer reports empty metadata and would flag everything as missing.
• Resync Status item in each peer's action menu, fetched on demand. It is intentionally not polled: the server takes the site replication lifecycle lock and persists resync state on every status call.
• Quiet auto refresh every 30 seconds while the tab is visible, with the last update time shown. Quiet refreshes never overlap and never show the loading state.
• Removed the misleading Replicated Objects / Replicated Size summary and the local Last Online field.

Implementation notes:

• Health rules live in lib/site-replication-health.ts as pure functions so they are unit tested independently of the page. Peers without a metric entry (servers before fix(admin): report real peer health in site replication status rustfs#7024 only report the local entry) stay Unknown instead of reading as healthy.
• hooks/use-site-replication.ts now normalizes PeerErrors, PendingOperation, RetryStats, BucketStats, PolicyStats, UserStats, GroupStats, ILMExpiryStats, metric failed windows, and queued, plus the resync status counters.
• The duration formatter that was local to the status page moves to lib/functions.ts so both pages share it.
• 40 new strings are added to all 14 locale files. The Chinese wording follows the existing "重同步" term.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Code refactoring
• Performance improvement
• Test improvements
• Security fix

Testing

• Unit tests added/updated
• Manual testing completed

pnpm test:run

• tests/lib/site-replication-health.test.ts (11 cases): healthy group; offline peer via metric and via peer error (critical, keeps last-online time and downtime); unreachable peer suppresses the drift comparison; peers without a metric entry stay unknown; pending operation, retry backlog, and last-hour failures degrade health while an empty retry queue and older failures do not; drift lists missing and mismatched buckets, policies, and users; info-only fallback when the status endpoint is unavailable; local site counts as online without its own metric entry.
• tests/lib/functions.test.ts covers the shared formatDuration.
• pnpm test:run (799 pass), pnpm type-check, pnpm lint, pnpm format:check, and pnpm install --frozen-lockfile all pass.
• Manual run against a local two-site RustFS lab built from current rustfs main (scripts/test/site_replication_smoke.py):
  1. Both sites up: Healthy, both peers Online, consistency clean.
  2. Stopped site B: within one auto-refresh cycle the page turned Unhealthy, site B showed Offline with last-online time and downtime, and the consistency check reported that it is skipped.
  3. Created a bucket on site A while site B was down: "2 configuration changes are waiting to be delivered to peer sites" appeared.
  4. Restarted site B: Needs attention, with the new bucket listed as missing on site B and differing in versioning and replication config.
  5. After the server replayed the queued changes: back to Healthy without a manual refresh.
  6. Resync Status dialog checked both before any resync and after a completed one.
  7. Mobile width (375 px): no horizontal page scroll.

Checklist

• Code follows the project's style guidelines
• Self-review completed
• TypeScript types are properly defined
• All commit messages are in English (Conventional Commits)
• All existing tests pass
• No new dependencies added, or they are justified

Related Issues

Refs rustfs/backlog#2922 (P0 only; the backend data items in P1 and the alerting guide in P2 remain open)

Screenshots (if applicable)

All screenshots are from the same local two-site lab, before (current main) and after this PR.

Both sites healthy

Before	After

Site B stopped. Before: the page looks exactly like the healthy state; Sync still says "Enabled" and Last Online shows the current time. After: Unhealthy with reasons, and site B is Offline with its last-online time.

Before	After

Site B back, queued changes not yet delivered (after only)

Needs attention	Consistency list

Mobile (after)

Additional Notes

• The resync status call is on demand only, not part of the 30-second refresh, because the server takes the lifecycle lock and writes resync state on every status request.
• Follow-ups tracked in rustfs/backlog#2922 (P1): cluster-wide replication counters scoped to site replication targets, separate control-plane and data-plane reachability per peer, per-peer failure and backlog attribution, and a fix for the replication uptime value. Peer error strings are currently the server's debug-formatted S3Error, which is why the page clamps them.
• The Replicated Objects and Replicated Size locale keys are no longer used by this page but are left in place to avoid churning existing keys.
• Screenshots live on the reatang/evidence/site-replication-health branch and are referenced by commit SHA.