Skip to content

fix(throttler): keep scheduling queued tasks after a task fails - #11060

Open
Ishkirat-Singh wants to merge 2 commits into
firebase:mainfrom
Ishkirat-Singh:fix/throttler-resume-after-failure
Open

fix(throttler): keep scheduling queued tasks after a task fails#11060
Ishkirat-Singh wants to merge 2 commits into
firebase:mainfrom
Ishkirat-Singh:fix/throttler-resume-after-failure

Conversation

@Ishkirat-Singh

Copy link
Copy Markdown

Description

Throttler.onTaskFulfilled calls process() so the next waiting task is scheduled, but onTaskFailed did not. A task that exhausted its retries (or timed out) therefore released its concurrency slot without refilling it, and any task still waiting behind it was never scheduled. For run()-style callers such as QueueExecutor, that task's promise never settles: a deploy can hang on an unsettled promise and, once nothing else is pending, exit without ever reporting an error.

This makes onTaskFailed resume scheduling the same way onTaskFulfilled does. finish(error) still rejects wait() callers first, so their behaviour is unchanged; the difference is that tasks already added to the throttler keep running, which is what the close() docstring promises ("any tasks already added to the throttler will continue to run").

Related: #11040. The source-token deadlock described there was addressed in #11044; this is a separate defect in the throttler with the same "silent hang, exit 0" signature, found while investigating that report.

Scenarios Tested

  • Added should keep scheduling waiting tasks after a task fails to src/throttler/throttler.spec.ts; it runs for both Queue and Stack. With concurrency: 1, task 1 fails and task 2 is queued behind it; the test asserts task 2's promise settles.
  • npx mocha src/throttler/throttler.spec.ts src/throttler/queue.spec.ts src/throttler/stack.spec.ts: 39 passing. Without the fix the new test fails (2 failing, one per implementation).

Sample Commands

N/A.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures that the throttler continues scheduling queued tasks after a task fails by calling this.process() in onTaskFailed. However, the reviewer pointed out that this change can cause this.finish() to be called multiple times, leading to memory leaks and redundant execution because this.waits is never cleared. It is recommended to clear this.waits inside this.finish() to resolve this issue.

Comment on lines 300 to +303
this.finish(error);
// A failed task frees its concurrency slot just like a fulfilled one. Without this,
// tasks still waiting behind it are never scheduled and their promises never settle.
this.process();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

With the introduction of this.process() in onTaskFailed, the throttler will now continue to execute remaining tasks after a failure. This means this.finish() can be called multiple times (either via subsequent failures or when the queue eventually becomes idle and calls finishIfIdle()).

Currently, this.finish() iterates over this.waits but never clears the array:

  private finish(err?: TaskError): void {
    this.waits.forEach((p) => {
      if (err) {
        return p.reject(err);
      }
      this.finished = true;
      return p.resolve();
    });
  }

Because this.waits is not cleared, this leads to:

  1. Memory Leak: References to already-settled promises and their handlers are retained in this.waits indefinitely.
  2. Redundant Execution / Double Settling: Subsequent calls to finish() will attempt to resolve or reject the same already-settled promises again.
  3. Inconsistent Behavior: If a caller calls wait() after a failure has occurred but before the queue is idle, that new promise will be resolved successfully when the queue becomes idle, despite the earlier failure.

Recommendation:
Please update this.finish() to clear the this.waits array after settling them:

  private finish(err?: TaskError): void {
    this.waits.forEach((p) => {
      if (err) {
        return p.reject(err);
      }
      this.finished = true;
      return p.resolve();
    });
    this.waits = [];
  }

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 428c571: finish() now hands the registered waiters off before settling them, with a test asserting they are released.

onTaskFulfilled calls process() to start the next waiting task, but
onTaskFailed did not. A task that exhausted its retries (or timed out)
therefore released its concurrency slot without refilling it, so any task
still waiting behind it was never scheduled and its promise never settled.

For QueueExecutor callers that means a deploy can hang on an unsettled
promise and, once nothing else is pending, exit without reporting an error.

Related: firebase#11040
finish() can now run more than once, so hand the registered waiters off
before settling them instead of keeping them for the throttler's lifetime.
@Ishkirat-Singh
Ishkirat-Singh force-pushed the fix/throttler-resume-after-failure branch from 8aec507 to 428c571 Compare September 10, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants