Skip to content

feat(http): route the node:http client's simplest shape onto turnloop - #11091

Closed
proggeramlug wants to merge 1 commit into
mainfrom
turnloop/http-client-lane1
Closed

proggeramlug wants to merge 1 commit into
mainfrom
turnloop/http-client-lane1

Conversation

@proggeramlug

Copy link
Copy Markdown
Contributor

First lane of moving the node:http / node:https CLIENT off reqwest.
perry-ext-http -> reqwest is one of only four tokio edges reached ALWAYS
rather than as a fallback, and six of the repo's seventeen live in this crate.

No tokio edge is removed yet — tokio_inventory.py is unchanged at 17/7,
which is correct: reqwest still serves everything outside lane 1. Only the
annotations move.

What this does

client_turnloop.rs drives turnloop_http::client::Http1Connection over
perry_ffi::turnloop_net on the agent's own loop. It is offered every exchange
in dispatch_request_snapshot, immediately before dispatch_request — three
hunks in lib.rs, nothing else moved. PendingHttpEvent was already
transport-agnostic, so the lane emits the same
ResponseHead/ResponseChunk/ResponseEnd/TransportError and the existing
drain, agent-admission release included, is untouched.

Two behaviours fall out rather than being ported: Node's client never follows a
3xx (driving the codec directly means nothing follows anything), and
res.statusMessage is unchanged — http1::Head carries no reason phrase, so
the canonical one stands in, exactly as reqwest did.

Decline set

Narrow and named, per the pattern fetch established: a non-default agent
handle, a non-empty body, an explicit timeout, NODE_USE_ENV_PROXY=1,
TE: trailers / Connection: Upgrade / Expect: 100-continue, non-http://,
!available(), and whatever client::Request::new refuses.

The three bypass headers are declined on the same predicates those bypass
modules trigger on
, so the routing cannot disagree with itself.

One decline that was not anticipated: an explicit Host header.
Request::head strips a caller's host and substitutes the URL authority
(a Fetch rule); reqwest sends what the caller set. A unit test pins that the
codec really does rewrite it.

Keep-alive is deliberately out of this lane. Releasing a connection before
Event::End is the framing-misattribution hazard, and it deserves its own
change. The cost is invisible to JS: req.reusedSocket and
agent.sockets/freeSockets are fed by agent.rs's facade pool, already
decoupled from the physical connection.

Verification

  • cargo check --workspace --all-targets (UI excludes): exit 0, re-run at the
    final tree state
  • cargo test -p perry-ext-http: 148 lib + 2 integration, 0 failed
  • tests/turnloop_client_exchange.rs, new end-to-end binary in its own process,
    asserting liveness three ways — try_dispatch returned true,
    completed_total() moved, and a plain TcpListener saw a well-formed head.
    Sabotage-proven: forcing the lane to decline makes it fail with the
    intended message, so a green run means the lane actually ran
  • Gap suite --filter test_gap_http_: 4/4 pass, 100% parity, three times
  • Liveness in the real compiled binary (temporary probe, removed before commit):
    the lane accepted the request and output was byte-identical to node v26.5.1
  • check_file_size.sh, addr_class_inventory.py, gc_runtime_root_holders.py,
    check_gc_env_knobs.py, check_node_version_consistency.py: OK. rustfmt
    clean, zero clippy and zero rustc warnings in the new files

scripts/run_lint_gates.sh was not run — it is broken at the base commit
(Install cargo-xwin extraction error, fixed separately in #11081), so the
individual gates were run instead.

Also fixed

A per-request reserved-id leak — free_handle_id missing from on_closed,
the #6441 exhaustion shape that ws, ext-net and http-server each already carry
an arm for.

Corrections to the roadmap this lane was scoped from

Several long-standing claims in tokio_inventory.json turn out to be wrong, and
the annotations are updated:

  • agent.rs is not "a second connection pool layered over reqwest's own".
    It is a per-origin admission engine — FIFO waiter queue, maxSockets /
    maxTotalSockets / maxFreeSockets, socket facades — sitting above the
    transport, with its own unit tests. reqwest owns only the physical connection.
    Its reqwest coupling is 9 call sites in 3 functions. It is not the blocker
    the inventory describes and it barely shrinks. The per-name FIFO that the
    migration plan assumed would need building already exists.
  • The "three raw tokio::net::TcpStream bypasses" list names the wrong
    third.
    Measured: plain_client.rs (TE: trailers), continue_client.rs
    (Expect), and client_upgrade.rs (Connection: Upgrade, node:http client never emits 'upgrade': a 101 Switching Protocols response is delivered as 'response' and the socket is lost #10468).
    client_connect_override.rs (agent.createConnection) reaches zero
    tokio — it runs on perry-ext-net's raw_net vtable. All three real ones are
    deletable, since Event::Upgrade exists too.
  • agent.createConnection is still blocked, for a different reason than
    recorded. Detached::from_fd + Loop::attach are real in turnloop, but a
    binding crate cannot reach them: there is zero attach / adopt /
    from_fd / Detached surface in perry_ffi::turnloop_net or
    perry-runtime/src/turnloop_net/abi.rs. That lane needs a perry-ffi ABI
    addition first.
  • test_gap_fetch_expect_continue_header is unaffected — fetch: shorthand { headers } drops a Headers instance before js_fetch_headers_to_json, silently sending no headers #11024 is a codegen
    defect (shorthand { headers } dropping a Headers instance), not transport,
    so its skip-list entry is not stale.

Remaining lanes

Lane Scope Removes
2 request bodies + Lifecycle deadlines POST/PUT/PATCH
3 TLS via perry_tls_session, needs per-request config rather than fetch's process-wide OnceLock all https — the biggest single surface
4 keep-alive + Agent: Pool, release only after End closes #10328
5 proxy CONNECT via Route small
6 fold in the three bypass modules 3 tokio TcpStream users
7 agent.createConnection blocked on the ABI addition above

Lane 2 wants wants_step() / is_mid_message() / eof_is_clean(), which do
not exist in alpha.6 — this lane expresses the progress loop without them and
bounds zero-consumed steps so a non-advancing decoder cannot spin the agent's
loop. It should be sequenced behind the turnloop bump (#11083).

`client_turnloop::try_dispatch` drives `turnloop_http::client::Http1Connection`
over `perry_ffi::turnloop_net` on the agent's own loop, and is offered every
exchange in `dispatch_request_snapshot` before the reqwest path. Lane 1 takes
cleartext `http://` with no body, no explicit Agent, no per-request timeout and
no proxy; everything else is a named decline that falls through to reqwest
unchanged, which is the coexistence rule `fetch` already uses.

The JS surface does not move: `PendingHttpEvent` was already transport-agnostic,
so the lane emits the same ResponseHead/ResponseChunk/ResponseEnd/TransportError
the reqwest task emitted and the existing drain handles them, agent admission
included.

Keep-alive is deliberately not in this lane — releasing a socket before
`Event::End` is what misattributes framing, and that hazard gets a change of its
own. `scripts/tokio_inventory.json` keeps the reqwest edge but its `reached_when`
no longer says "always", and its `blocker` is corrected: agent.rs is an
admission engine above the transport, not a duplicate of reqwest's pool, and the
third raw-TcpStream bypass is `client_upgrade.rs`, not `agent.createConnection`.
@coderabbitai

coderabbitai Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 50 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used all 8 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a4c11b7b-8fc3-42d0-afbb-341e3a11f899

📥 Commits

Reviewing files that changed from the base of the PR and between e27f0a0 and 6063c9d.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (8)
  • CLAUDE.md
  • Cargo.toml
  • changelog.d/11091-node-http-client-turnloop-lane-1.md
  • crates/perry-db-turnloop/src/lib.rs
  • crates/perry-ext-http/src/client_turnloop.rs
  • crates/perry-ext-http/src/lib.rs
  • crates/perry-ext-http/tests/turnloop_client_exchange.rs
  • scripts/tokio_inventory.json

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@proggeramlug
proggeramlug force-pushed the turnloop/http-client-lane1 branch from 06e010c to 6063c9d Compare September 23, 2026 05:03
proggeramlug pushed a commit that referenced this pull request Sep 23, 2026
Resolving #11091's Cargo.lock conflict by taking main's lockfile
silently discarded #11067's group bump, which had been cherry-picked
earlier in the same assembly: mongodb 3.9.1->3.9.0, clap 4.6.7->4.6.6,
toml 1.1.6->1.1.5, http 1.5.0->1.4.1, hickory-proto 0.26.3->0.26.2,
cc 1.4.6->1.4.5, uuid 1.26.1->1.26.0.

Worth noting WHICH gate caught it. `lock_no_downgrade.py --vs
origin/main` stayed GREEN, correctly: relative to main nothing moved
backwards, the bumps simply never happened. Only `tokio_inventory.py`
saw it, because it compares against its own recorded list rather than
against the merge base:

  Cargo.lock version change: mongodb ['3.9.1'] -> ['3.9.0']

Restored by taking #11067's lockfile and re-resolving the merged
manifests on top, rather than hand-merging. Verified:
  cargo metadata --locked            rc=0 (lock satisfies every manifest)
  tokio_inventory.py                 unchanged
  lock_no_downgrade.py --vs main     no resolved version moved backwards
  rustls 0.23.45, getrandom 0.4.3    both at or above their floors
@proggeramlug

Copy link
Copy Markdown
Contributor Author

Landed on main in merge train 265 (#11108), released as v0.5.1648 at 9d26936298.

Cherry-picked from this PR's head 6063c9d548 and validated as one tree with 15 other PRs — CI 22/22 green, all 6 gap-suite shards. A train rebase gives the commits new SHAs, so GitHub cannot auto-close the source PR; closing by hand.

Nothing needed from you. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perry-ext-http's AGENT_CLIENTS cache never evicts

1 participant