Use monotonic clock for timeouts; harden db-sync progress wait - #3593
Merged
Conversation
Replace time.time() with time.monotonic() where values are used only for elapsed-time measurement or in-process deadlines, so timeouts are immune to wall-clock jumps (NTP, manual adjustment): * dbsync_utils: retry_query timeout and db-sync progress poll * test_tx_many_utxos: UTxO generation duration * test_blocks: epoch-deadline loops; rename *_timestamp to *_deadline as the values are no longer wall-clock timestamps Wall-clock time.time() is kept where timestamps are persisted, shared across processes or compared to file mtimes (logfiles, status_db, skip_after rules, unique-name generation).
mkoura
requested
a lite review from Copilot
and removed request for
ArturWieczorek and
saratomaz
August 6, 2026 11:32
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates timeout/deadline handling to use monotonic time (avoiding wall-clock jumps) and hardens db-sync progress polling so empty databases and None progress are handled correctly without aborting prematurely.
Changes:
- Switched elapsed-time and deadline calculations from
time.time()totime.monotonic()in db-sync utilities and relevant tests. - Updated db-sync progress querying to return
Nonewhen theblocktable is empty, and adjusted the polling loop to treat that as retryable/pollable. - Standardized db-sync timeout behavior to raise
DbSyncTimeoutError(aTimeoutErrorsubclass) and improved progress logging behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
cardano_node_tests/utils/dbsync_utils.py |
Uses monotonic deadlines for retries/polling and improves db-sync completion waiting logic and error semantics. |
cardano_node_tests/utils/dbsync_queries.py |
Returns None (instead of crashing) when sync progress cannot be computed due to an empty block table. |
cardano_node_tests/tests/test_tx_many_utxos.py |
Uses monotonic timing for reporting UTxO generation duration. |
cardano_node_tests/tests/test_blocks.py |
Uses monotonic deadlines for epoch/test loops while keeping wall-clock time for persisted tx name generation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
* Fix query_db_sync_progress crash on empty block table: the aggregate query returns a (None,) row, so float(None) raised TypeError. Return None instead, matching the documented behavior, and annotate the return type as float | None. * Treat None progress as retryable: the initial query raises DbSyncNoResponseError (retried by retry_query), the poll loop keeps polling with the last known progress. A genuine 0.0% progress no longer aborts the wait. * Log the progress value used by the loop instead of issuing a second redundant DB query per iteration. * Raise DbSyncTimeoutError (TimeoutError subclass) for module consistency and report actual elapsed time in the message. * Express the total time budget as a precomputed deadline. * Fix docstring: hardcoded 99% vs the expected_progress parameter and a typo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use monotonic clock for durations and deadlines
Replace
time.time()withtime.monotonic()where values are used only forelapsed-time measurement or in-process deadlines, so timeouts are immune to
wall-clock jumps (NTP sync, manual adjustment):
dbsync_utils:retry_querytimeout and db-sync progress polltest_tx_many_utxos: UTxO generation durationtest_blocks: epoch-deadline loops; variables renamed from*_timestampto*_deadlineas they no longer hold wall-clock timestampsWall-clock
time.time()is intentionally kept where timestamps are persisted,shared across processes or compared to file mtimes (logfiles offset and
ignore-rule files, status db, tx-name generation).
Harden
wait_for_db_sync_completionWhile converting the clock usage, review uncovered real bugs in the polling
logic:
query_db_sync_progresscrashed withTypeErroron an emptyblocktable:the aggregate query returns a
(None,)row, sofloat(None)raised and theelse 0.0branch was dead. This was hit routinely, as the only caller(
restart_with_config) recreates the database right before polling. Thefunction now returns
Nonein that case (annotatedfloat | None).Noneprogress is now retryable: the initial query is retried byretry_query, and the poll loop keeps polling with the last known progressuntil the deadline. A genuine 0.0% progress no longer aborts the wait.
query per iteration.
DbSyncTimeoutError(aTimeoutErrorsubclass) for moduleconsistency and reports actual elapsed time.
expected_progressparameter, typo.