fix(throttler): keep scheduling queued tasks after a task fails - #11060
fix(throttler): keep scheduling queued tasks after a task fails#11060Ishkirat-Singh wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request ensures that the throttler continues scheduling queued tasks after a task fails by calling this.process() in onTaskFailed. However, the reviewer pointed out that this change can cause this.finish() to be called multiple times, leading to memory leaks and redundant execution because this.waits is never cleared. It is recommended to clear this.waits inside this.finish() to resolve this issue.
| this.finish(error); | ||
| // A failed task frees its concurrency slot just like a fulfilled one. Without this, | ||
| // tasks still waiting behind it are never scheduled and their promises never settle. | ||
| this.process(); |
There was a problem hiding this comment.
With the introduction of this.process() in onTaskFailed, the throttler will now continue to execute remaining tasks after a failure. This means this.finish() can be called multiple times (either via subsequent failures or when the queue eventually becomes idle and calls finishIfIdle()).
Currently, this.finish() iterates over this.waits but never clears the array:
private finish(err?: TaskError): void {
this.waits.forEach((p) => {
if (err) {
return p.reject(err);
}
this.finished = true;
return p.resolve();
});
}Because this.waits is not cleared, this leads to:
- Memory Leak: References to already-settled promises and their handlers are retained in
this.waitsindefinitely. - Redundant Execution / Double Settling: Subsequent calls to
finish()will attempt to resolve or reject the same already-settled promises again. - Inconsistent Behavior: If a caller calls
wait()after a failure has occurred but before the queue is idle, that new promise will be resolved successfully when the queue becomes idle, despite the earlier failure.
Recommendation:
Please update this.finish() to clear the this.waits array after settling them:
private finish(err?: TaskError): void {
this.waits.forEach((p) => {
if (err) {
return p.reject(err);
}
this.finished = true;
return p.resolve();
});
this.waits = [];
}There was a problem hiding this comment.
Done in 428c571: finish() now hands the registered waiters off before settling them, with a test asserting they are released.
onTaskFulfilled calls process() to start the next waiting task, but onTaskFailed did not. A task that exhausted its retries (or timed out) therefore released its concurrency slot without refilling it, so any task still waiting behind it was never scheduled and its promise never settled. For QueueExecutor callers that means a deploy can hang on an unsettled promise and, once nothing else is pending, exit without reporting an error. Related: firebase#11040
finish() can now run more than once, so hand the registered waiters off before settling them instead of keeping them for the throttler's lifetime.
8aec507 to
428c571
Compare
Description
Throttler.onTaskFulfilledcallsprocess()so the next waiting task is scheduled, butonTaskFaileddid not. A task that exhausted its retries (or timed out) therefore released its concurrency slot without refilling it, and any task still waiting behind it was never scheduled. Forrun()-style callers such asQueueExecutor, that task's promise never settles: a deploy can hang on an unsettled promise and, once nothing else is pending, exit without ever reporting an error.This makes
onTaskFailedresume scheduling the same wayonTaskFulfilleddoes.finish(error)still rejectswait()callers first, so their behaviour is unchanged; the difference is that tasks already added to the throttler keep running, which is what theclose()docstring promises ("any tasks already added to the throttler will continue to run").Related: #11040. The source-token deadlock described there was addressed in #11044; this is a separate defect in the throttler with the same "silent hang, exit 0" signature, found while investigating that report.
Scenarios Tested
should keep scheduling waiting tasks after a task failstosrc/throttler/throttler.spec.ts; it runs for bothQueueandStack. Withconcurrency: 1, task 1 fails and task 2 is queued behind it; the test asserts task 2's promise settles.npx mocha src/throttler/throttler.spec.ts src/throttler/queue.spec.ts src/throttler/stack.spec.ts: 39 passing. Without the fix the new test fails (2 failing, one per implementation).Sample Commands
N/A.