Skip to content

Failed initialize: connect() rejects before the transport has closed, so a retried stdio connect() fails with Connection closed #3194

Open

Description

@prkomb

When the legacy initialize handshake fails (for example, the server answers with a protocol version the client does not support), Client.connect() calls void this.close() in _legacyHandshake (packages/client/src/client/client.ts, the catch block at the end of the handshake) and rejects at once. StdioClientTransport.close() waits for the child process to exit (up to 2 s, then SIGTERM and up to 2 s more, then SIGKILL), so on stdio:

  • client.transport is still set when connect() rejects, because the child has not exited yet;
  • a connect() retried straight away on a new transport fails with Connection closed when the first child's exit lands during the new handshake, which it did on every run here. The first child's close event runs the old transport's onclose wrapper from Protocol.connect(), and Protocol._onclose() clears the new transport and rejects its in-flight initialize. The second server process is then left running, because client.close() no longer has a transport to close.
Reproduction (real spawned servers)

Save as repro.mjs in the repository root after pnpm install && pnpm build:all, then run node repro.mjs.

// node repro.mjs: the client; it spawns itself as two stdio servers.
import { Client } from '@modelcontextprotocol/client';
import { StdioClientTransport } from '@modelcontextprotocol/client/stdio';
import { Server } from '@modelcontextprotocol/server';
import { StdioServerTransport } from '@modelcontextprotocol/server/stdio';

const role = process.argv[2];
if (role) {
    const server = new Server({ name: role, version: '0' }, { capabilities: { tools: {} } });
    if (role === 'bad') {
        server.setRequestHandler('initialize', () => ({ protocolVersion: '1999-01-01', capabilities: {}, serverInfo: { name: 'bad', version: '0' } }));
    }
    server.setRequestHandler('tools/list', () => ({ tools: [] }));
    await server.connect(new StdioServerTransport());
} else {
    const spawn = role => new StdioClientTransport({ command: process.execPath, args: [process.argv[1], role] });
    const client = new Client({ name: 'repro', version: '0' });
    try {
        await client.connect(spawn('bad'));
    } catch (error) {
        console.log('1st connect rejected:', error.message, '| client.transport set:', client.transport !== undefined);
    }
    try {
        await client.connect(spawn('good'));
        console.log('2nd connect resolved; tools/list:', JSON.stringify(await client.listTools()));
    } catch (error) {
        console.log('2nd connect rejected:', error.code, error.message);
    }
    await client.close();
}

On main (919bb8e, packages built from the workspace), Node 26.13.0; the process never exits, timeout 30 ends it:

1st connect rejected: Server's protocol version is not supported: 1999-01-01 | client.transport set: true
2nd connect rejected: CONNECTION_CLOSED Connection closed

With this change it exits on its own:

1st connect rejected: Server's protocol version is not supported: 1999-01-01 | client.transport set: false
2nd connect resolved; tools/list: {"tools":[]}

The remaining stdio known failure on lifecycle:version:reject-unsupported in test/e2e/requirements.ts records the same symptom.

A fix with tests is ready on prkomb:fix-client-await-close-on-failed-initialize: it awaits the close and ignores an error from it (await this.close().catch(() => {})), as the other connect() failure paths in client.ts already do (await transport.close().catch(() => {}) after a failed probe, await this.close().catch(() => {}) after an aborted listen), so connect() still rejects with the handshake error.

The cost is that a failed connect() now waits for whatever the transport's close() does: on stdio normally the few milliseconds the server takes to exit, up to about 4 s for a server that ignores stdin EOF and SIGTERM. In that last case close() returns after SIGKILL without waiting for the exit, so a late close can still reach a retried connection, as it already can after a plain client.close() and connect(); the branch leaves that alone.

GitHub didn't let me open a second pull request here while my e2e stdio harness fix #3192 is open, so I'll open this one once #3192 is merged or closed; with both applied, that stdio known failure can be removed.

Versions: main at 919bb8e, Node 26.13.0. v1.x has the same void this.close() line; #2770 handles its rejection there without waiting for the close.

Activity

  1. A7mdMe commented on Nov 17, 2026

    @A7mdMe

    Spot-on analysis @prkomb. This is a critical edge case for agent orchestrators, test harnesses, and automated failover runners that retry connection attempts on stdio or switch between fallback servers.

    Why void this.close() Creates the Race Condition

    When _legacyHandshake fails in packages/client/src/client/client.ts:

    1. void this.close() is fired un-awaited in the catch block while connect() rejects synchronously.
    2. At this moment, client.transport is still bound to the dying transport.
    3. When the application catch block immediately retries connect(newTransport):
      • The first child process exits a few milliseconds later during the second handshake.
      • The first transport's child process close event fires.
      • Its onclose listener (registered in Protocol.connect) invokes Protocol._onclose().
      • _onclose() assumes the active connection was severed, so it clears this.transport = undefined, aborts pending requests, and rejects the in-flight initialize on the new transport with CONNECTION_CLOSED Connection closed.
      • Crucially, the second child process is now orphaned and left running, because client.close() has no active transport reference to terminate!

    The Deeper Issue: Awaiting close() Still Needs Epoch Fencing

    While changing void this.close() to await this.close().catch(() => {}) resolves the normal exit timing, a subtle race still exists in edge cases:

    • If a child process is spawned via shell wrappers (npx, sh, cmd.exe), SIGTERM may terminate the wrapper while the real child process takes several seconds or ignores termination until SIGKILL.
    • If close() returns after its kill timeout without waiting indefinitely for exit, that late child exit event can still fire into the client during attempt $N+1$.

    To achieve rock-solid isolation across retried connections, transports should be guarded by Monotonic Epoch Fencing:

    Connection 1 (Epoch 1) -> Handshake Rejects -> Teardown Triggered
         │
         ▼ (Late exit event fires during Connection 2)
    Epoch Check: Event Epoch (1) !== Current Epoch (2) -> DROP STALE EVENT (Safe!)
    

    Drop-in Client Wrapper (For Production Runners & Orchestrators Today)

    For anyone building retry loops or server failovers today, you can wrap the client or transport factory with an epoch fence and teardown synchronizer:

    import { Client } from '@modelcontextprotocol/sdk/client/index.js';
    import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
    
    export class TransportEpochFence {
      private currentEpoch = 0;
    
      public fenceTransport<T extends { onclose?: () => void; onerror?: (err: Error) => void }>(transport: T): T {
        this.currentEpoch++;
        const epoch = this.currentEpoch;
        const originalOnClose = transport.onclose;
        const originalOnError = transport.onerror;
    
        transport.onclose = () => {
          // Drop stale event if a newer connection attempt has already started
          if (epoch !== this.currentEpoch) return;
          if (originalOnClose) originalOnClose.call(transport);
        };
    
        transport.onerror = (err) => {
          if (epoch !== this.currentEpoch) return;
          if (originalOnError) originalOnError.call(transport, err);
        };
    
        return transport;
      }
    
      public async connectSafely(client: Client, transportFactory: () => StdioClientTransport): Promise<void> {
        const rawTransport = transportFactory();
        const fencedTransport = this.fenceTransport(rawTransport);
    
        try {
          await client.connect(fencedTransport);
        } catch (err) {
          // Guarantee old transport is fully closed and awaited before propagating rejection
          try {
            await fencedTransport.close();
          } catch {}
    
          // Disarm client.transport if SDK left it populated
          if ((client as any).transport === fencedTransport) {
            (client as any).transport = undefined;
          }
          throw err;
        }
      }
    }
    
    // Usage in failover / retry orchestrators:
    const fence = new TransportEpochFence();
    
    try {
      await fence.connectSafely(client, () => spawnTransport('bad-server'));
    } catch (err) {
      // Transport is 100% dead, child process awaited, and client.transport is undefined
      console.log('1st connect rejected cleanly:', err.message);
    }
    
    // 2nd connect will never be aborted by late exit events from the 1st server!
    await fence.connectSafely(client, () => spawnTransport('good-server'));

    Upstream PR awaiting this.close() will be a great fix for the core SDK. Looking forward to reviewing the PR once #3192 lands.


    Disclosure: I maintain MCP Production Kit, an open-source kit and commercial suite providing stdio lifecycle guards, monotonic epoch fencing, and slowloris lifecycle defenses for production MCP architectures.

  2. prkomb commented on Nov 17, 2026

    @prkomb
    Author

    I tried the TransportEpochFence wrapper above against the repro in this issue, and it doesn't work around it on 1.32.2 or 2.4.0:

    • client.transport only has a getter, so (client as any).transport = undefined throws TypeError: Cannot set property transport of #<Protocol> which has only a getter, and connectSafely rejects with that instead of the handshake error.
    • The epoch check doesn't stop the teardown. Protocol.connect() keeps the transport's earlier onclose and calls its own _onclose() after it, so a late close still clears the new connection. With the assignment removed and a first server that only exits on SIGKILL, the retried connection on 2.4.0 had still lost its transport 4.5 s later.
    • A second close() on the same StdioClientTransport returns at once (the first call already dropped its process), so the await in the catch doesn't wait for the exit either.

    On 1.32.2 the retry fails sooner, with "Already connected to a transport", because client.transport is still set when connect() rejects.

    What did work on both versions, for a bad server that exits normally and for one that only exits on SIGKILL, is waiting for the failed transport's own onclose before retrying:

    const closed = new Promise<void>(resolve => {
        transport.onclose = () => resolve();
    });
    try {
        await client.connect(transport);
    } catch (error) {
        if (client.transport === transport) await closed;
        throw error;
    }

    connect() calls an onclose that was set before it, right before its own cleanup clears client.transport, so once closed resolves a retry connects normally (in the SIGKILL case after about 4 s). The client.transport === transport check skips the wait when the transport is already gone or was never taken, as with 1.32.2's "Already connected". It relies on the transport firing onclose once it has closed, which StdioClientTransport does, a failed spawn included.

  3. robertwmccarty-caelum commented on Nov 17, 2026

    @robertwmccarty-caelum

    I tested an additional connection-lifecycle invariant against main at 919bb8ea88b6e10e72c2f8b6218fad4cfdc40b2c.

    @prkomb's proposed await this.close().catch(() => {}) is the right first step for the failed-initialize race, and I don't want to duplicate that work. The second boundary is late callbacks from a superseded transport: Protocol.connect() currently installs an onclose wrapper that unconditionally calls this._onclose(). If an old transport signals close after a subsequent connection is active, that wrapper can clear the new this._transport and reject its in-flight work. Delayed old onmessage and onerror callbacks can similarly reach the current protocol instance.

    A minimal internal fence to evaluate is capturing the transport for each connect() and checking this._transport === transport before handling its close/error/message events; it does not change the public API. I added a deterministic regression that (1) holds a failed transport's close() pending until released, then (2) successfully reconnects, fires the old transport's retained onclose/onerror/onmessage handlers, and checks that the new connection still answers ping() and no stale error leaks through.

    A candidate implementation combining the reporter's await-close repair and these guards passed the affected client and core-internal Vitest suites, both package typechecks, lint, and a full workspace build in an isolated pinned-source run. This is controlled callback testing, not yet proof for every real child-process shutdown timing. The proposed work is complementary to the existing branch; I'm happy to align on whether the extra Protocol-level fence belongs in the same PR or a follow-up.

  4. robertwmccarty-caelum commented on Nov 17, 2026

    @robertwmccarty-caelum

    Follow-up with the real spawned-process result, including a counterexample to my earlier candidate.

    On pinned 919bb8e, a candidate with await this.close().catch(() => {}) and stale-transport event guards passed the normal failed-initialize retry but failed against a stdio server that ignored stdin EOF and SIGTERM. After the transport escalated to SIGKILL, StdioClientTransport.close() could resolve before its final onclose callback, leaving client.transport pointing at the spent transport when connect() had already rejected. Thus awaiting close() is necessary but not sufficient.

    The refined candidate also settles Protocol's connection state after a transport's close() promise resolves if that transport is still current:

    override async close(): Promise<void> {
      const closingTransport = this.transport;
      try {
        await super.close();
      } finally {
        if (closingTransport && this.transport === closingTransport) this._onclose();
        this._resetConnectionState();
      }
    }

    Together with the transport-identity guards in base Protocol.connect(), a genuinely late close then sees the old transport is no longer active and cannot invalidate a subsequent connection. This is internal-only; there is no new public API. The error from the original failed initialize is still propagated.

    Verified: actual spawned Node stdio repros both passed on the revised candidate (ordinary bad server and resistant server requiring forced termination), along with the full affected client/core-internal suites, typechecks, source lint, and full workspace build. These tests do not prove the absence of every possible scheduler race or settle the broader API-contract decision for explicit close. Credit for the original await-close idea belongs to @prkomb.

  5. added
    v1Issues / PRs related to v1.x
    v2Ideas, requests and plans for v2 of the SDK which will incorporate major changes and fixes
    on Nov 18, 2026
  6. prkomb commented on Nov 18, 2026

    @prkomb
    Author

    @robertwmccarty-caelum thanks, that matches what I see, and it isn't limited to a failed initialize. On today's main (5ee54ee3), with a connected stdio server that ignores both stdin EOF and SIGTERM, a plain client.close() returns after about 4.0 s (once SIGKILL is sent, without waiting for the exit). An immediate connect() to a second, well-behaved server then rejects within about 10 ms with CONNECTION_CLOSED, and client.transport is not the new transport afterwards (3 runs out of 3). With a first server that exits normally, the same sequence connects fine.

    The same gap shows up in #3198: client.transport is still set when that close() returns, so with #3198's guard applied, listTools() right after it still resolves [], while after a normal close it rejects with Not connected.

    The await on this issue's branch only covers the normal exit after a failed initialize, as you found. The forced-kill case comes from close() returning before the transport's onclose, on any close, so it needs something like your Protocol-level check and close() change. I'd keep that as a separate PR: it changes core-internal for every transport, so it's worth a maintainer's view first, and the draft #2470 (moving connection-scoped state into a private Connection owner) may be where it fits. If you want to take it, go ahead; my branch stays limited to the await. It still applies cleanly on 5ee54ee3, where the client tests pass and e2e has no unexpected failures.

  7. robertwmccarty-caelum commented on Nov 19, 2026

    @robertwmccarty-caelum

    @prkomb Thanks for checking explicit close on 5ee54ee3, including the normal-exit control. I can take the follow-up. One design question for the maintainers before opening a core PR: should close() settle the current connection before returning and ignore later events from its old transport on main, or should that behavior land with the connection owner in #2470? I’d keep your failed-initialize await in its own branch. The acceptance check for the follow-up is that an immediate reconnect can complete a ping and a delayed callback from the previous transport leaves that new connection usable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    v1Issues / PRs related to v1.xv2Ideas, requests and plans for v2 of the SDK which will incorporate major changes and fixes

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions