Skip to content

[Messaging] sendEach() remains pending after an HTTP/2 stream closes without a response #3325

Open

Description

@eger1393

[Messaging] sendEach() remains pending after an HTTP/2 stream closes without a response

[READ] Step 1: Are you in the right place?

This report concerns the request lifecycle in the Admin Node.js SDK. The reproduction uses a local HTTP/2 server, one fake registration token, and fake credentials. It does not require a Firebase project or send any requests to Firebase.

Related, but not necessarily duplicate:

[REQUIRED] Step 2: Describe your environment

  • Operating System version: Windows, build 10.0.26200, x64.
  • Firebase SDK version: firebase-admin@14.5.1; also reproduced with 14.0.0.
  • Firebase Product: Cloud Messaging.
  • Node.js version: v26.13.0.
  • NPM version: 11.20.0.

An earlier equivalent local-server harness also reproduced the behavior on Linux in Docker with Node.js v22.23.5 and firebase-admin@14.0.0.

As of November 19, 2026, npm's latest tag points to 14.5.1. All three cases below were executed against a full, isolated installation of 14.5.1, with the resolved SDK path and installed version checked before running. The same reproduction also failed with 14.0.0.

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

In an otherwise empty directory:

npm init -y
npm install firebase-admin@14.5.1

Save the code below as repro.cjs, then run:

node repro.cjs ok
node repro.cjs disconnect
node repro.cjs cancel

The ok case is a control: the local server returns a complete JSON response and sendEach() resolves successfully.

In disconnect, the server receives the request and destroys the HTTP/2 session without sending response headers. In cancel, it closes the stream with NGHTTP2_CANCEL, also without sending response headers.

Expected: sendEach() settles with a rejected Promise or a failed message result after the stream closes. It must not continue waiting on a closed stream.

Actual: the client stream emits close, but sendEach() is still pending after the reproduction's 20-second watchdog. Neither error nor timeout is observed on that client stream in these runs. Both failure cases exit with code 1.

Example output:

{"mode":"ok","elapsedMs":13,"requests":1,"events":[],"outcome":"resolved","successCount":1,"failureCount":0}

{"mode":"disconnect","elapsedMs":20001,"requests":1,"events":["close"],"outcome":"still-pending"}
sendEach did not settle after stream close

{"mode":"cancel","elapsedMs":20016,"requests":1,"events":["close"],"outcome":"still-pending"}
sendEach did not settle after stream close

These outputs are from the 14.5.1 run. The control exits with code 0; each failure case exits with code 1 because its assertion detects the unresolved operation.

The watchdog only observes the unresolved operation and lets the script clean up. It does not change the SDK's request timeout or retry configuration.

Relevant Code:

The test temporarily redirects http2.connect() to a local cleartext HTTP/2 server and adjusts :scheme for that local connection. It does not change the SDK's response/error handling, stream timeout, retry behavior, or Promise implementation. All connection attempts must target the expected FCM hostname before being redirected locally.

const assert = require('node:assert/strict');
const http2 = require('node:http2');
const { initializeApp, deleteApp } = require('firebase-admin/app');
const { getMessaging } = require('firebase-admin/messaging');

const mode = process.argv[2] || 'disconnect';
assert.ok(['ok', 'disconnect', 'cancel'].includes(mode));
const server = http2.createServer();
const serverSessions = new Set();
const events = [];
let requests = 0;
let clientSession;

server.on('session', session => {
  serverSessions.add(session);
  session.on('error', () => {});
});
server.on('stream', stream => {
  requests++;
  stream.on('error', () => {});
  stream.resume();
  stream.on('end', () => {
    if (mode === 'disconnect') stream.session.destroy();
    else if (mode === 'cancel') stream.close(http2.constants.NGHTTP2_CANCEL);
    else {
      stream.respond({ ':status': 200, 'content-type': 'application/json' });
      stream.end(JSON.stringify({ name: 'projects/local/messages/test' }));
    }
  });
});

(async () => {
  await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
  const originalConnect = http2.connect;
  http2.connect = url => {
    assert.equal(new URL(url).hostname, 'fcm.googleapis.com');
    clientSession = originalConnect(`http://127.0.0.1:${server.address().port}`);
    const originalRequest = clientSession.request.bind(clientSession);
    clientSession.request = (headers, ...args) => {
      // Route to local cleartext HTTP/2 without changing SDK lifecycle handling.
      const stream = originalRequest({ ...headers, ':scheme': 'http' }, ...args);
      for (const event of ['close', 'aborted', 'error', 'timeout']) {
        stream.on(event, () => events.push(event));
      }
      return stream;
    };
    return clientSession;
  };

  const app = initializeApp({
    projectId: 'local',
    credential: {
      getAccessToken: async () => ({ access_token: 'fake-local-token', expires_in: 3600 }),
    },
  }, 'local-hang-repro');
  let timer;
  const started = Date.now();
  try {
    const result = await Promise.race([
      getMessaging(app).sendEach([{
        token: 'fake-registration-token', notification: { title: 'Local test' },
      }]).then(
        response => ({
          outcome: 'resolved', successCount: response.successCount, failureCount: response.failureCount,
        }),
        error => ({ outcome: 'rejected', code: error.code }),
      ),
      new Promise(resolve => {
        timer = setTimeout(() => resolve({ outcome: 'still-pending' }), 20000);
      }),
    ]);
    console.log(JSON.stringify({ mode, elapsedMs: Date.now() - started, requests, events, ...result }));
    assert.equal(requests, 1);
    if (mode !== 'ok') assert.ok(events.includes('close'));
    assert.notEqual(result.outcome, 'still-pending', 'sendEach did not settle after stream close');
    if (mode === 'ok') assert.equal(result.successCount, 1);
    else if (result.outcome === 'resolved') assert.equal(result.failureCount, 1);
  } finally {
    clearTimeout(timer);
    http2.connect = originalConnect;
    clientSession?.destroy();
    for (const session of serverSessions) session.destroy();
    await new Promise(resolve => server.close(resolve));
    await deleteApp(app);
  }
})().catch(error => { console.error(error.message); process.exitCode = 1; });

Suspected cause

AsyncHttp2Call.execute() subscribes to response and error and installs a stream timeout, but does not settle the operation when the stream closes before a response. In these reproductions, close is the only observed terminal stream event. sendEach() then remains blocked on Promise.allSettled(requests).

This is a hypothesis supported by the local reproduction, not a claim that we captured the exact network event in production.

Impact

We investigated this after a production sendEach() call for 108 messages failed to return and blocked later scheduled sender runs. Restarting the application restored subsequent runs, but the delivery outcome of that batch remained unknown. This report does not establish whether those messages reached FCM or prove that the production incident had exactly the same trigger as the local reproduction.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions