Skip to content

fix(builder): abandon the load operation when a resource load times out - #1553

Open
joaodinissf wants to merge 1 commit into
masterfrom
fix/parallel-loader-timeout-count
Open

joaodinissf wants to merge 1 commit into
masterfrom
fix/parallel-loader-timeout-count

Conversation

@joaodinissf

@joaodinissf joaodinissf commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Why the change

A resource load timeout currently leaves the build running with an incomplete index: a different, healthy resource is silently not indexed. With this change the timeout fails the build visibly instead, and the next build is a full build. It does not change how long the builder waits.

Special things to note

  • Behaviour change: a load timeout while writing resource descriptions now cancels the build (rolled back, next build is a full build). On master that phase carried on with an incomplete index; the linking phase already cancelled this way on master, so both phases now behave the same.
  • A load that always stalls longer than the timeout (300 s with no result from any worker) now fails every build instead of quietly degrading the index.
  • An interrupted wait is handled like a timeout. Without that, the operation would never finish: an interrupted wait delivers no result, so nothing would count it off.
  • Raising the timeout alone does not fix the miscount, it only makes it rarer: on master every timeout leaves the counter one short for the rest of the operation.

Change outline

The owed-result counter in ParallelLoadOperation.next():

 result = resourceQueue.poll(waitTime)      # null on timeout or interrupt
-toProcess--                                # also without a result: counter now one short
 if result == null
+  cancel()                                 # toProcess = 0, shutdownNow; "remaining loads are abandoned"
   throw LoadOperationException(null, TimeoutException)
+toProcess--

What the builder does after a timeout:

 MonitoredClusteringBuilderState
   linking loop     timeout -> log, then !hasNext() -> cancel build      (unchanged)
   writeResources   timeout -> log
-                    loop ends when hasNext() is false: rest silently unindexed
+                    isAbandonedByTimeout(ex, loadOperation) -> OperationCanceledException

Tests (ParallelResourceLoaderTest, each run against an unbounded queue and production's synchronous hand-off):

  • timeoutAbandonsEveryOutstandingLoad: one timeout ends the operation and queued loads never start. Fails on master.
  • interruptedWaitAbandonsEveryOutstandingLoad: an interrupted wait ends the operation too and keeps the interrupt flag. Fails on master.
  • failedLoadDoesNotAbandonTheOperation: an ordinary load failure keeps the other loads.
  • resultWithinTimeoutIsDelivered: sanity check.
  • BuilderLoadTimeoutTest: only a timeout with nothing left to load counts as abandonment; a timeout with loads left, an ordinary load failure and other wrapped exceptions do not.

🤖 Generated with Claude Code

joaodinissf added a commit that referenced this pull request Sep 26, 2026
formal/README.md describes the method, target status, models and how to
reproduce. formal/BUGS.md catalogues 51 findings (49 confirmed, 1
plausible, 1 refuted) plus 9 observations, all verified by three
independent skeptics, with traces, test status, fix plans, a proposed
fix-PR sequence and links to the fix PRs opened so far (#1550, #1551,
#1552, #1553). REPORT.md is the chronological log of rounds 1-2. The
patches are reference fixes used to show each disabled test turns green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
if (result != null) {
toProcess--;
consecutiveTimeouts = 0;
} else if (++consecutiveTimeouts >= MAX_CONSECUTIVE_TIMEOUTS) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not get the point of this PR, could we not just increase the timeout if we want to try longer?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal isn't to wait longer, it's what happens once a timeout fires. On master the build carries on and silently drops a different, healthy resource from the index: the timed-out wait still counts as a delivered result, so the loop ends one short. Nothing is logged for the resource that went missing, and the index stays incomplete until the next full build. With this change a timeout (or an interrupted wait) cancels the build instead, so the next one is a full build on a consistent index. This is already how the linking phase handles it on master. A longer timeout would only make the silent case rarer.

Posted by Claude Code on behalf of @joaodinissf.

@joaodinissf
joaodinissf force-pushed the fix/parallel-loader-timeout-count branch from 4282d4c to 95ccf71 Compare September 28, 2026 19:15
@joaodinissf joaodinissf changed the title fix(builder): keep waiting for a timed-out resource load instead of dropping it fix(builder): abandon the load operation when a resource load times out Sep 28, 2026
joaodinissf added a commit that referenced this pull request Sep 29, 2026
formal/README.md describes the method, target status, models and how to
reproduce. formal/BUGS.md catalogues 51 findings (49 confirmed, 1
plausible, 1 refuted) plus 9 observations, all verified by three
independent skeptics, with traces, test status, fix plans, a proposed
fix-PR sequence and links to the fix PRs opened so far (#1550, #1551,
#1552, #1553). REPORT.md is the chronological log of rounds 1-2. The
patches are reference fixes used to show each disabled test turns green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ParallelLoadOperation.next() decremented the outstanding-result counter
even when poll() timed out. From then on the counter was one short of the
results still coming, so the builder either aborted the cluster as if it
had been cancelled or, in writeResources, left some other resource
silently unindexed. A longer timeout makes this rarer but cannot prevent
it.

Decrement only when a result is delivered. A timeout cannot be attributed
to a URI, and a load that never finishes would otherwise keep the builder
waiting forever, so a timeout now cancels the whole operation and says so.
An interrupted wait is handled the same way: it delivers no result either,
and without a result nothing would ever count it off.
writeResources treats that like the linking phase already did on master:
it cancels the build instead of dropping the remaining resources, so the
next build is a full build rather than one with a stale index. The check
is a public predicate, isAbandonedByTimeout, so that it can be tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joaodinissf
joaodinissf force-pushed the fix/parallel-loader-timeout-count branch from 95ccf71 to 75b16e8 Compare October 25, 2026 06:30
@joaodinissf
joaodinissf marked this pull request as ready for review October 25, 2026 06:30

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants