Skip to content

feat(net): network modules — HTTP client/server + WebSocket client, C/Rust cores, ESP-IDF host, TLS - #299

Open
HalfSweet wants to merge 17 commits into
pocket-stack:mainfrom
HalfSweet:feat/network-v2
Open

feat(net): network modules — HTTP client/server + WebSocket client, C/Rust cores, ESP-IDF host, TLS#299
HalfSweet wants to merge 17 commits into
pocket-stack:mainfrom
HalfSweet:feat/network-v2

Conversation

@HalfSweet

@HalfSweet HalfSweet commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds network support to PocketJS as a working stack: three spec-pinned guest modules, a portable C reference core, the Rust reference client core, deterministic + browser + ESP-IDF hosts, and TLS. First target is ESP-IDF; the design keeps other platforms in view. The in-repo documentation is docs/NET.md and site/content/docs/net.md; the pinned boundary is contracts/spec/{net,ws,httpd}.ts.

This replaces the NET v1 SDK surface with the v2 modules. globalThis.net and op codes 1–5 are preserved and evolved in place; the old @pocketjs/framework/net value exports and net.http are removed (no in-repo consumers).

What's here

  • Contractscontracts/spec/net.ts evolved to v2 (ops 6–9 readInto/limits/write/endBody, op 2 retired, streaming headers/readable/end/error events, the shared error vocabulary with the four tls_* codes); new ws.ts and httpd.ts. gen-rust.ts emits net/ws/httpd; new gen-c.ts emits engine/net/include/pocketjs/net/spec.h. Both mirrors are under the tests/contract.ts drift guard.
  • SDK@pocketjs/framework/net (support module: NetworkError, AbortController/AbortSignal, URL, getNetworkLimits, shared types), net/http (fetch, Headers, Request, Response, BodyStream, serve), net/websocket (connect). A per-module Guest Binding drains one poll per tick from the framework service pump, settles Promises and calls handlers inside that pump, and copies bodies through readInto. New @pocketjs/framework/headless runs the frame transaction without a UI.
  • Portable C coreengine/net: HTTP/1.1 client + server, RFC 6455 client, strict framing profile, bounded receive/send queues with backpressure, the immutable policy, tick queues with a per-tick budget and readable-before-terminal ordering, and the TLS handshake state machine. The only host interfaces are pnet_driver_ops (a BSD/lwIP driver under drivers/posix) and an optional pnet_tls_ops TLS provider.
  • Rust coreengine/crates/pocket-net rewritten to the v2 boundary over an HttpClientBackend; mount installs the six ops via rquickjs.
  • Hosts — deterministic hosts/sim/{net,httpd,ws}.ts; hosts/web/net.js moved to the streaming v2 contract; hosts/esp-idf with a QuickJS-ng owner task, a network task under one runtime lock, the net/ws/httpd bindings, AtomS3R + Tab5 bring-up, and the smoke firmware.
  • TLSpnet_tls_ops + pnet_runtime_create_tls: https:/wss: and the "tls" feature are enabled only when a provider is present; the core owns the deadline/cancel/policy and fails closed with tls_clock_untrusted before I/O when the wall clock is untrusted; no plaintext fallback. Reference NativeTlsProvider over OpenSSL (engine/net/drivers/openssl) and an ESP-TLS provider (hosts/esp-idf/components/pocketjs_net_esptls, IDF certificate bundle).
  • Capabilitiescontracts/spec/platforms.ts replaces net.http with the role-split network.http.client(.tls) / network.http.server(.tls) / network.websocket.client(.tls) ids. No stock target advertises them yet: a target appends an id only when its native host ships and tests the module.

Execution model (unchanged)

Network facts enter the guest only at frame boundaries: the host runs begin_tick before each frame(), the service pump polls each module once inside frame(), and Promise reactions run in that tick's job drain. No independent network turn, no wake, no ref()/unref(); docs/RUNTIMES.md Law 3 is untouched.

Validation

  • Softwarebunx tsc --noEmit, the contracts drift guard, and the full bun tools/test.ts suite (unit, wasm-host unit, vue-sfc, octane, and the sim determinism stages) are green. Net SDK coverage: tests/net.test.ts, tests/net-httpd.test.ts, tests/net-websocket.test.ts, tests/net-web.test.js.
  • C corectest --test-dir engine/net/build under ASan/UBSan: unit (154 checks: HTTP/1.1 head/body strict framing, URL, policy, JSON, UTF-8, base64/SHA-1, tick queue), host (163 checks: both cores over real loopback sockets against scripted peers), tls (16 checks: an in-process OpenSSL PKI + HTTPS/WSS peers — valid chain, unknown CA, expired, hostname mismatch, untrusted clock, development-insecure refusal, no plaintext fallback, WSS echo).
  • Rust corecargo test -p pocket-net (7 tests: ordering, budget truncation, cancel/late completions, refusals, mount).
  • Hardware — ESP-IDF v6.0.2 (7101770dc6db), AtomS3R (ESP32-S3-PICO-1-N8R8) and Tab5 (ESP32-P4 rev 1.3 + ESP32-C6 over SDIO). Each passes 26/26: the plaintext suite (GET/POST/JSON/chunked/404, redirect follow+manual, a 200 KB body through an 8 KiB queue at ~350–370 KiB/s, aggregate limit, headers timeout, permission_denied, connect refused, WebSocket echo, board-to-board GET/POST/JSON/stream/404, continuous pings) against an independent workstation peer (tools/net-peer.ts) and each other, plus a TLS suite: HTTPS/1.1 to a public host with a valid chain from the IDF certificate bundle (SNTP-synced clock) and badssl.com's expired / wrong-host / self-signed / untrusted-root endpoints, all failing closed. A 12-minute board-to-board soak (43,200 frames, ~330 round trips each way) ended with zero failures and flat heap (guest ≈357 KB, core ≈4 KB).

Tab5 notes captured in hosts/esp-idf/README.md: rev 1.3 silicon (CONFIG_ESP32P4_SELECTS_REV_LESS_V3 + REV_MIN_100), the C6 SDIO preset with active-high reset on GPIO15, the WLAN rail on the PI4IOE5V6408 IO expander, and FREERTOS_HZ=1000.

Out of scope (staged / future)

manifest format 3 + requiresOneOf, the compiler surface-demand derivation and the globalThis.net/ws/httpd direct-reference rejection, advertising a network capability from a stock target, record/replay tooling, the PSP network substrate, HTTP Server / WebSocket Server admission on hardware, and the TLS extensions (custom CA, client auth, ALPN, TLS 1.3 requirement, revocation). The ESP-TLS async path exposes the Mbed TLS verify flags inconsistently, so on hardware a hostname mismatch is reported precisely while other certificate faults collapse to tls_handshake_failed (always fail-closed); the precise per-fault codes are proven in the desktop OpenSSL conformance suite.

Notes for reviewers

This supersedes the experimental stack in #282: the boundary is frame-bounded (no dispatcher/turn ABI), delivery is a bounded poll inside frame(), and the native cores never call QuickJS.

…odule specs

- net.ts: ops 6-9 (readInto/limits/write/endBody), retire op 2, streaming
  headers/readable/end/error events, portable ceilings, the shared error
  vocabulary with the four tls_* codes and NetworkError categories
- ws.ts: WebSocket Client ops/events/limits (globalThis.ws)
- httpd.ts: HTTP Server ops/events/limits (globalThis.httpd)
- gen-rust.ts emits net/ws/httpd modules; new gen-c.ts emits the C mirror
  engine/net/include/pocketjs/net/spec.h, both under the drift guard
@pocketjs/framework/net is the support module (NetworkError, AbortController,
AbortSignal, URL, getNetworkLimits, shared types); net/http provides fetch,
Headers, Request, Response, BodyStream and serve over the v2 net/httpd
namespaces; net/websocket provides connect over ws. A Network Guest Binding
per module drains one poll per tick from the service pump, settles Promises
and calls handlers inside that pump, and copies bodies through readInto.

hosts/sim gains deterministic net/httpd/ws hosts; tests cover delivery
order, streaming, backpressure, cancellation, error mapping and refusals.
mountHeadless() installs a frame handler that runs the fixed prefix of the
frame transaction — virtual clock, service pumps, effect delivery, app hook —
without a UI root, for display-less hosts and the network smoke firmware.
Browser fetch stays the transport; the adapter now speaks the v2 ops
(readInto/limits), keeps a bounded per-handle receive queue with reader
backpressure, freezes readable watermarks at beginFrame(), and maps hidden
redirects to unsupported per the browser profile.
engine/net is the native implementation of the v2 module boundaries in
portable C99: platform/driver interfaces, the shared async runtime (bounded
allocator accounting, tick queues with readable-before-terminal ordering and
per-tick budgets, connections, resolve+connect dialer with per-address policy
checks), strict HTTP/1.1 head/body parsing, URL and policy handling, and the
HTTP Client core behind pnet_http_* (redirect policy, timeouts, backpressure
through the receive queue, readInto). Includes a BSD-socket driver shared by
POSIX hosts and lwIP, and a unit-test binary under ASan/UBSan.
pnet_httpd_* implements listen/stop, bounded accept, strict request parsing
(400/413/414/431/503 refusals), request-body streaming into a bounded queue,
respond/write/endBody with keep-alive, chunked streaming, HEAD discard,
Expect: 100-continue, handler/idle/keep-alive deadlines, peer-disconnect and
abort reporting, and graceful stop. The host test drives both cores over real
loopback sockets against scripted peers.
pnet_ws_* implements the RFC 6455 client: upgrade handshake with accept-key,
subprotocol and extension checks, masked client frames, unmasked-only server
frames, fragment reassembly with incremental UTF-8 validation, native pong
replies, bounded receive/send queues with drain, 1002/1007/1009/1013 local
closes reported as error → close, close handshake with a deadline and
terminate. The host harness gains a scripted WebSocket peer covering echo,
fragmentation, control frames, limit and protocol violations, transport loss,
handshake refusals, terminate and backpressure.
…nd Tab5

- pocketjs_net_core: engine/net + the lwIP/BSD driver as an IDF component
- pocketjs_esp_host: guest owner task (PSRAM heap, fixed-rate frame ticks
  with begin_tick, job drain), globalThis.net/ws/httpd bindings, network
  task under one runtime lock, wind-down frames on stop
- pocketjs_board: STA + DHCP for the AtomS3R (native) and Tab5 (C6 over
  SDIO via esp_hosted/esp_wifi_remote, WLAN rail on the IO expander)
- examples/net-smoke: headless smoke app + firmware template (rev 1.3 and
  hosted sdkconfig), tools/net-peer.ts as the independent workstation peer
- core fixes found on hardware: httpd end is a readable barrier, connect
  failures map to connect, WebSocket keeps reading after a local close and
  ignores data after its Close frame, deferred write shutdown

Both boards pass 20/20 against the peer and each other.
NetCore now speaks spec v2: HttpClientBackend reports streaming
Headers/Body/End/Error completions, the core keeps per-handle bounded
receive queues with backpressure hints, freezes readable watermarks at
begin_tick (inserted before the handle's barrier event), applies the
per-tick event/byte budget, serves readInto and limits(), enforces the
connect policy and the shared error vocabulary; the mount feature installs
the six v2 ops through rquickjs. Fixture-backed tests cover ordering,
budget truncation, cancel/late completions, refusals and the mount.
…bilities

docs/NET.md and site/content/docs/net.md now document the v2 modules
(net/http, net/websocket, the support module), delivery at frame
boundaries, streaming bodies, errors, limits, ownership and testing;
RUNTIMES.md and concepts.md follow. contracts/spec/platforms.ts replaces
net.http with the role-split network.* capability ids (no stock target
advertises them).
globalThis.net comes with the policy; ws and httpd are mounted only when the
host admits those roles (both default on for the smoke).
The core gains a TlsProvider interface (pnet_tls_ops) and
pnet_runtime_create_tls: https:/wss: are accepted and the "tls" feature is
advertised only when a provider is present. The connection layer runs a
non-blocking client handshake between the plain connect and reporting open,
routes application I/O through the session, owns the connect deadline and
cancellation, and fails closed with tls_clock_untrusted before any I/O when
the platform reports the wall clock untrusted. SNI equals the authorized
hostname and is the DNS-ID/IP-ID the certificate must match; a plaintext
fallback never happens.

engine/net/drivers/openssl implements the reference NativeTlsProvider (TLS
1.2 min, renegotiation/tickets off, verify failures mapped to the four
stable tls_* codes). A new tls conformance harness stands up an in-process
OpenSSL PKI and HTTPS/WSS peers covering a valid chain, unknown CA, expired
cert, hostname mismatch, no plaintext fallback, an untrusted clock, the
development-insecure refusal, and a WSS echo.
pocketjs_net_esptls wraps ESP-TLS + the IDF certificate bundle as a
pnet_tls_ops over the driver's connected lwIP socket (non_block=false so the
Mbed TLS handshake progresses asynchronously without ESP-TLS owning the
connect). The host creates the runtime with pnet_runtime_create_tls when
network_tls is set and exposes a wall-clock-trusted hook backed by time();
the board syncs SNTP before the guest starts. The smoke gains a TLS suite
against a public host plus badssl.com's expired/wrong-host/self-signed/
untrusted-root endpoints.

Both boards: 26/26 (20 plaintext + 6 TLS). HTTPS/1.1 to a public host with a
valid chain succeeds; every bad certificate fails closed. docs/NET.md and the
esp-idf README document the TLS boundary and the gate.
@HalfSweet
HalfSweet force-pushed the feat/network-v2 branch 2 times, most recently from 51294e5 to 2fa7f8e Compare August 18, 2026 16:01
The network architecture and guest-boundary v2 design notes are internal
artifacts and are no longer committed. Point the API docs (docs/NET.md,
site/content/docs/net.md) at the in-repo pinned specs (contracts/spec/*),
and rewrite the source-comment citations to be self-describing: each now
states the rule it used to reference by section number, with no dead
repo-relative path. Also correct the ESP-TLS provider header, which
described a socket-fd duplicate the implementation does not make.
The Phase 1A/1B/1C staged-gate terminology came from the internal design
notes that are no longer committed. State what each module actually
delivers and the plain rule that a target advertises a capability only when
its native host ships and tests the module, instead of naming rollout
phases. No stock target advertises the network capabilities yet.
@HalfSweet
HalfSweet marked this pull request as ready for review August 19, 2026 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant