feat: Backfill the canister-level monotonic consumed cycles

[mraszyk]

CanisterMetrics::consumed_cycles_monotonic (added in #11465) is absent in
checkpoints written before it, and decodes as zero. Left alone it would only
start accruing from the round it was added in, so every canister that predates
it under-reports its lifetime consumption forever.

The consumed_cycles gauge predates it and holds the full history, so the
monotonic value can be derived from it. SystemState::outstanding_prepayments()
sums the prepayments that have been added to the gauge but whose refund has not
been observed yet -- the ones held in an open Callback
(prepayment_for_response_execution plus prepayment_for_call_transmission,
falling back to prepayment_for_response_transmission for callbacks created
before April 2026, as the refund path itself does) or in the
prepaid_execution_cycles of an aborted execution or install_code. All of
them are recorded in the replicated state, which yields the invariant

consumed_cycles - outstanding_prepayments() == consumed_cycles_monotonic

whenever no execution is in progress or paused, and makes
SystemState::migrate_consumed_cycles_to_monotonic() exact. Because the
invariant holds at every checkpoint rather than just the first one, the backfill
is also idempotent: a no-op for a canister already backfilled, and self-healing
if a downgrade dropped the field.

A callback created before prepayment_for_call_transmission was stored in
callbacks (#9859, April 2026) is no exception, even though the fallback accounts
for the response transmission part of that prepayment only, leaving out the
xnet_call_performed_fee plus the request bytes fee. That call fee is part of
the gauge, so the first backfill credits it while the callback is still open --
the gauge being what the canister really consumed -- and executing the response
then adds exactly the prepayments that outstanding_prepayments() reported,
keeping the monotonic value in step with the gauge. Only a canister still short
of the gauge, i.e. one not backfilled yet or whose field a downgrade dropped,
lags by that fee, which is the case the backfill exists for anyway.

The scheduler runs it on checkpoint rounds only, after
abort_all_paused_executions: a paused execution generally holds its prepayment
in memory rather than in the state, and backfilling from a gauge that covers it
would overestimate, so outstanding_prepayments() returns None for any paused
execution -- including a paused response execution, whose prepayments the
callback carried by the task does record, since the only caller runs once there
are none left. Aborting materializes those prepayments into the task queues. The
pass takes a mutable reference to a canister only when it has something to
write, so it stays read-only once the subnet has been backfilled.

The three states this considers unreachable -- a canister whose outstanding
prepayments cannot be derived, outstanding prepayments above the gauge, and a
monotonic value already above the gauge net of them -- are reported through
debug_assert_or_critical_error! and a new
scheduler_consumed_cycles_invariant_broken error counter, since all of them
would otherwise be silent in production. debug_assert_or_critical_error! now
passes its message to the debug_assert! as well, so the panic it raises under
debug assertions names the report that tripped rather than reading
assertion failed: false; that is what lets the tests here expect one specific
report.

The tests span the three levels this touches: outstanding_prepayments() and
the backfill are covered by unit tests in rs/replicated_state; scheduler tests
drive real prepayments through checkpoint rounds, a response callback paused and
then aborted among them, to assert that the abort leaves behind a task that
prepays nothing of its own and a callback that is no longer registered; and an
ExecutionTest executes the response of a legacy callback, tying the fallback
of outstanding_prepayments() to the one that
ResponseHelper::apply_initial_refunds() applies -- the two fall back
independently of each other, in different crates -- and asserts that a backfill
taken while that callback is still open credits its call fee, so that executing
the response leaves the monotonic value in step with the gauge. Turning a callback into a legacy
one needs a new reset_prepayment_for_call_transmission testing hook, as no
production code writes such a callback any more.

No metric or behaviour changes: nothing outside of tests reads
consumed_cycles_monotonic yet. A follow-up
(mraszyk/consumed-cycles-as-counter, #11416) exports the monotonic total under
the existing replicated_state_consumed_cycles_since_replica_started gauge and
is stacked on this branch.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Saturating subtraction can silently hide an outstanding-prepayments invariant violation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Backfills canister-level monotonic consumed cycles from existing accounting state during checkpoint rounds.

Changes:

• Derives outstanding prepayments from callbacks and aborted executions.
• Runs idempotent checkpoint migration with invariant-error metrics.
• Adds unit and scheduler coverage for migration scenarios.

File summaries

File	Description
rs/state_layout/src/state_layout/proto.rs	Documents legacy checkpoint decoding.
rs/replicated_state/src/canister_state/tests.rs	Tests prepayment derivation and migration.
rs/replicated_state/src/canister_state/system_state.rs	Implements derivation and backfill.
rs/execution_environment/src/scheduler/tests/metrics.rs	Tests checkpoint integration.
rs/execution_environment/src/scheduler/scheduler_metrics.rs	Adds invariant error counter.
rs/execution_environment/src/scheduler.rs	Executes and validates migration.
rs/execution_environment/src/canister_manager.rs	Documents deletion accounting invariant.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

The migration depends on accounting invariants spanning callbacks, execution lifecycle states, checkpointing, and subnet operations.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

The deletion-accounting documentation incorrectly assumes the monotonic value has already been backfilled.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

A critical invariant-error branch lacks direct regression coverage.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The replicated cycles-accounting migration depends on a subtle system-wide invariant and warrants final human validation.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Two unconstrained panic tests can pass without exercising their intended invariant branches.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

rs/execution_environment/src/scheduler/tests/metrics.rs:1212

• This unconstrained should_panic can pass on any setup assertion failure before the checkpoint round (for example, if outstanding unexpectedly becomes zero), without exercising the invariant-error branch. Constrain the expected panic from debug_assert!(false) so setup regressions fail the test instead of producing a false positive.

This issue also appears on line 1238 of the same file.

rs/execution_environment/src/scheduler/tests/metrics.rs:1238

• This unconstrained should_panic also treats failures in the ingress/setup assertions as success, so the test can pass without reaching the Ordering::Less report. Constrain it to the panic emitted by the target debug_assert!(false) (or catch the unwind only around the checkpoint call).

#[should_panic]

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

It changes consensus-critical cycle accounting and checkpoint migration across several execution states, warranting final human review.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The checkpoint migration depends on intricate cross-crate cycles-accounting invariants that warrant final human validation.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟢 Approval recommended

The migration is guarded, idempotent, and comprehensively covers outstanding and legacy prepayment states.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to e349119.

Security Overview

• 🔎 Scanned files: 8 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/f5e9d2bb-29a4-4b64-902a-391d65fd8b2d?scanId=616fa4af-d797-4eb0-b38b-9aec161ec1d7&codeScanTypes=PrScan&tab=issues

Detected Code Changes

Change Type	Relevant files
Enhancement	► rs/execution_environment/src/execution/response/tests.rs       Add test execute_response_of_legacy_callback_settles_the_outstanding_prepayments ► rs/execution_environment/src/scheduler.rs       Import additional debug_assert_or_critical_error and migrate backfill logic on checkpoint rounds ► rs/execution_environment/src/scheduler/scheduler_metrics.rs       Add consumed_cycles_invariant_broken metric ► rs/execution_environment/src/scheduler/tests/metrics.rs       Introduce tests for consumed_cycles invariant and checkpoint backfill behavior ► rs/execution_environment/src/util.rs       Update debug_assert_or_critical_error macro to include formatted message ► rs/replicated_state/src/canister_state/system_state.rs       Add outstanding_prepayments() method documentation and logic ► rs/replicated_state/src/canister_state/system_state/call_context_manager.rs       Expose reset_prepayment_for_call_transmission in testing interface ► rs/replicated_state/src/canister_state/tests.rs       Add tests for outstanding_prepayments_of_open_callbacks and related scenarios ► rs/replicated_state/src/canister_state/system_state.rs       Add detailed comments and new methods related to prepayments and monotonic backfill
Refactor	► rs/execution_environment/src/scheduler.rs       Rearrange imports and align backfill call with new migrate_consumed_cycles_to_monotonic function ► rs/execution_environment/src/scheduler/scheduler_metrics.rs       Integrate new invariant-related metric into SchedulerMetrics construction

[schneiderstefan]

I wonder if this PR actually makes it better. Previously, there was a disconnect between the monotonic and the non-monotonic metrics. But the monotonic metrics and the per-usecase monotonic metrics (consumed_cycles_by_use_cases_monotonic) were consistent. Now this is no longer true, and we don't have the information to backfill the per-usecase metrics.

At the very least, this should be documented as a shortfall of the per-usecase metrics.

[mraszyk]

But the monotonic metrics and the per-usecase monotonic metrics (consumed_cycles_by_use_cases_monotonic) were consistent.

Not really because the monotonic aggregate was only introduced on Tuesday by #11465 and thus its value is currently lower than the aggregate over the monotonic per-use-case break-down.

Now this is no longer true, and we don't have the information to backfill the per-usecase metrics.

We do have it and I plan to follow-up on this in a separate PR. We only need to backfill the Instructions and RequestAndResponseTransmission use cases (the others are charged directly without any non-monotonic prepayment/refund separation) and we have the information available - Instructions as prepaid instructions in callbacks and aborted executions and RequestAndResponseTransmission as prepayments for response transmission in callbacks (this PR sums them all up to update the gauge and backfill the aggregate, the follow-up PR would separately update the Instructions gauge by deducting prepaid instructions and the ReqeustAndResponseTransmissions gauge by deducting prepaid response transmission cycles - the request tranmission cycles being already settled in the gauge as noted on this PR, too).

[Copilot]

🔵 Needs a closer look

The accounting migration spans checkpoint scheduling, legacy state compatibility, refunds, and persisted metrics, requiring final human validation.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 0 new
• Review effort level: Balanced