Skip to content

Use monotonic clock for timeouts; harden db-sync progress wait - #3593

Merged
mkoura merged 2 commits into
masterfrom
monotonic_time
Aug 6, 2026
Merged

Use monotonic clock for timeouts; harden db-sync progress wait#3593
mkoura merged 2 commits into
masterfrom
monotonic_time

Conversation

@mkoura

@mkoura mkoura commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Use monotonic clock for durations and deadlines

Replace time.time() with time.monotonic() where values are used only for
elapsed-time measurement or in-process deadlines, so timeouts are immune to
wall-clock jumps (NTP sync, manual adjustment):

  • dbsync_utils: retry_query timeout and db-sync progress poll
  • test_tx_many_utxos: UTxO generation duration
  • test_blocks: epoch-deadline loops; variables renamed from *_timestamp to
    *_deadline as they no longer hold wall-clock timestamps

Wall-clock time.time() is intentionally kept where timestamps are persisted,
shared across processes or compared to file mtimes (logfiles offset and
ignore-rule files, status db, tx-name generation).

Harden wait_for_db_sync_completion

While converting the clock usage, review uncovered real bugs in the polling
logic:

  • query_db_sync_progress crashed with TypeError on an empty block table:
    the aggregate query returns a (None,) row, so float(None) raised and the
    else 0.0 branch was dead. This was hit routinely, as the only caller
    (restart_with_config) recreates the database right before polling. The
    function now returns None in that case (annotated float | None).
  • A None progress is now retryable: the initial query is retried by
    retry_query, and the poll loop keeps polling with the last known progress
    until the deadline. A genuine 0.0% progress no longer aborts the wait.
  • The loop logs the value it compares instead of issuing a second redundant DB
    query per iteration.
  • Timeout raises DbSyncTimeoutError (a TimeoutError subclass) for module
    consistency and reports actual elapsed time.
  • Docstring fixes: hardcoded 99% vs the expected_progress parameter, typo.

Replace time.time() with time.monotonic() where values are used only
for elapsed-time measurement or in-process deadlines, so timeouts are
immune to wall-clock jumps (NTP, manual adjustment):

* dbsync_utils: retry_query timeout and db-sync progress poll
* test_tx_many_utxos: UTxO generation duration
* test_blocks: epoch-deadline loops; rename *_timestamp to *_deadline
  as the values are no longer wall-clock timestamps

Wall-clock time.time() is kept where timestamps are persisted, shared
across processes or compared to file mtimes (logfiles, status_db,
skip_after rules, unique-name generation).
@mkoura
mkoura requested a lite review from Copilot and removed request for ArturWieczorek and saratomaz August 6, 2026 11:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates timeout/deadline handling to use monotonic time (avoiding wall-clock jumps) and hardens db-sync progress polling so empty databases and None progress are handled correctly without aborting prematurely.

Changes:

  • Switched elapsed-time and deadline calculations from time.time() to time.monotonic() in db-sync utilities and relevant tests.
  • Updated db-sync progress querying to return None when the block table is empty, and adjusted the polling loop to treat that as retryable/pollable.
  • Standardized db-sync timeout behavior to raise DbSyncTimeoutError (a TimeoutError subclass) and improved progress logging behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
cardano_node_tests/utils/dbsync_utils.py Uses monotonic deadlines for retries/polling and improves db-sync completion waiting logic and error semantics.
cardano_node_tests/utils/dbsync_queries.py Returns None (instead of crashing) when sync progress cannot be computed due to an empty block table.
cardano_node_tests/tests/test_tx_many_utxos.py Uses monotonic timing for reporting UTxO generation duration.
cardano_node_tests/tests/test_blocks.py Uses monotonic deadlines for epoch/test loops while keeping wall-clock time for persisted tx name generation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cardano_node_tests/utils/dbsync_utils.py
* Fix query_db_sync_progress crash on empty block table: the aggregate
  query returns a (None,) row, so float(None) raised TypeError. Return
  None instead, matching the documented behavior, and annotate the
  return type as float | None.
* Treat None progress as retryable: the initial query raises
  DbSyncNoResponseError (retried by retry_query), the poll loop keeps
  polling with the last known progress. A genuine 0.0% progress no
  longer aborts the wait.
* Log the progress value used by the loop instead of issuing a second
  redundant DB query per iteration.
* Raise DbSyncTimeoutError (TimeoutError subclass) for module
  consistency and report actual elapsed time in the message.
* Express the total time budget as a precomputed deadline.
* Fix docstring: hardcoded 99% vs the expected_progress parameter and
  a typo.
@mkoura
mkoura merged commit 636fa28 into master Aug 6, 2026
3 checks passed
@mkoura
mkoura deleted the monotonic_time branch August 6, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants