feat(net): network modules — HTTP client/server + WebSocket client, C/Rust cores, ESP-IDF host, TLS - #299
Open
HalfSweet wants to merge 17 commits into
Open
feat(net): network modules — HTTP client/server + WebSocket client, C/Rust cores, ESP-IDF host, TLS#299HalfSweet wants to merge 17 commits into
HalfSweet wants to merge 17 commits into
Conversation
…odule specs - net.ts: ops 6-9 (readInto/limits/write/endBody), retire op 2, streaming headers/readable/end/error events, portable ceilings, the shared error vocabulary with the four tls_* codes and NetworkError categories - ws.ts: WebSocket Client ops/events/limits (globalThis.ws) - httpd.ts: HTTP Server ops/events/limits (globalThis.httpd) - gen-rust.ts emits net/ws/httpd modules; new gen-c.ts emits the C mirror engine/net/include/pocketjs/net/spec.h, both under the drift guard
@pocketjs/framework/net is the support module (NetworkError, AbortController, AbortSignal, URL, getNetworkLimits, shared types); net/http provides fetch, Headers, Request, Response, BodyStream and serve over the v2 net/httpd namespaces; net/websocket provides connect over ws. A Network Guest Binding per module drains one poll per tick from the service pump, settles Promises and calls handlers inside that pump, and copies bodies through readInto. hosts/sim gains deterministic net/httpd/ws hosts; tests cover delivery order, streaming, backpressure, cancellation, error mapping and refusals.
mountHeadless() installs a frame handler that runs the fixed prefix of the frame transaction — virtual clock, service pumps, effect delivery, app hook — without a UI root, for display-less hosts and the network smoke firmware.
Browser fetch stays the transport; the adapter now speaks the v2 ops (readInto/limits), keeps a bounded per-handle receive queue with reader backpressure, freezes readable watermarks at beginFrame(), and maps hidden redirects to unsupported per the browser profile.
engine/net is the native implementation of the v2 module boundaries in portable C99: platform/driver interfaces, the shared async runtime (bounded allocator accounting, tick queues with readable-before-terminal ordering and per-tick budgets, connections, resolve+connect dialer with per-address policy checks), strict HTTP/1.1 head/body parsing, URL and policy handling, and the HTTP Client core behind pnet_http_* (redirect policy, timeouts, backpressure through the receive queue, readInto). Includes a BSD-socket driver shared by POSIX hosts and lwIP, and a unit-test binary under ASan/UBSan.
pnet_httpd_* implements listen/stop, bounded accept, strict request parsing (400/413/414/431/503 refusals), request-body streaming into a bounded queue, respond/write/endBody with keep-alive, chunked streaming, HEAD discard, Expect: 100-continue, handler/idle/keep-alive deadlines, peer-disconnect and abort reporting, and graceful stop. The host test drives both cores over real loopback sockets against scripted peers.
pnet_ws_* implements the RFC 6455 client: upgrade handshake with accept-key, subprotocol and extension checks, masked client frames, unmasked-only server frames, fragment reassembly with incremental UTF-8 validation, native pong replies, bounded receive/send queues with drain, 1002/1007/1009/1013 local closes reported as error → close, close handshake with a deadline and terminate. The host harness gains a scripted WebSocket peer covering echo, fragmentation, control frames, limit and protocol violations, transport loss, handshake refusals, terminate and backpressure.
…nd Tab5 - pocketjs_net_core: engine/net + the lwIP/BSD driver as an IDF component - pocketjs_esp_host: guest owner task (PSRAM heap, fixed-rate frame ticks with begin_tick, job drain), globalThis.net/ws/httpd bindings, network task under one runtime lock, wind-down frames on stop - pocketjs_board: STA + DHCP for the AtomS3R (native) and Tab5 (C6 over SDIO via esp_hosted/esp_wifi_remote, WLAN rail on the IO expander) - examples/net-smoke: headless smoke app + firmware template (rev 1.3 and hosted sdkconfig), tools/net-peer.ts as the independent workstation peer - core fixes found on hardware: httpd end is a readable barrier, connect failures map to connect, WebSocket keeps reading after a local close and ignores data after its Close frame, deferred write shutdown Both boards pass 20/20 against the peer and each other.
NetCore now speaks spec v2: HttpClientBackend reports streaming Headers/Body/End/Error completions, the core keeps per-handle bounded receive queues with backpressure hints, freezes readable watermarks at begin_tick (inserted before the handle's barrier event), applies the per-tick event/byte budget, serves readInto and limits(), enforces the connect policy and the shared error vocabulary; the mount feature installs the six v2 ops through rquickjs. Fixture-backed tests cover ordering, budget truncation, cancel/late completions, refusals and the mount.
…bilities docs/NET.md and site/content/docs/net.md now document the v2 modules (net/http, net/websocket, the support module), delivery at frame boundaries, streaming bodies, errors, limits, ownership and testing; RUNTIMES.md and concepts.md follow. contracts/spec/platforms.ts replaces net.http with the role-split network.* capability ids (no stock target advertises them).
globalThis.net comes with the policy; ws and httpd are mounted only when the host admits those roles (both default on for the smoke).
The core gains a TlsProvider interface (pnet_tls_ops) and pnet_runtime_create_tls: https:/wss: are accepted and the "tls" feature is advertised only when a provider is present. The connection layer runs a non-blocking client handshake between the plain connect and reporting open, routes application I/O through the session, owns the connect deadline and cancellation, and fails closed with tls_clock_untrusted before any I/O when the platform reports the wall clock untrusted. SNI equals the authorized hostname and is the DNS-ID/IP-ID the certificate must match; a plaintext fallback never happens. engine/net/drivers/openssl implements the reference NativeTlsProvider (TLS 1.2 min, renegotiation/tickets off, verify failures mapped to the four stable tls_* codes). A new tls conformance harness stands up an in-process OpenSSL PKI and HTTPS/WSS peers covering a valid chain, unknown CA, expired cert, hostname mismatch, no plaintext fallback, an untrusted clock, the development-insecure refusal, and a WSS echo.
pocketjs_net_esptls wraps ESP-TLS + the IDF certificate bundle as a pnet_tls_ops over the driver's connected lwIP socket (non_block=false so the Mbed TLS handshake progresses asynchronously without ESP-TLS owning the connect). The host creates the runtime with pnet_runtime_create_tls when network_tls is set and exposes a wall-clock-trusted hook backed by time(); the board syncs SNTP before the guest starts. The smoke gains a TLS suite against a public host plus badssl.com's expired/wrong-host/self-signed/ untrusted-root endpoints. Both boards: 26/26 (20 plaintext + 6 TLS). HTTPS/1.1 to a public host with a valid chain succeeds; every bad certificate fails closed. docs/NET.md and the esp-idf README document the TLS boundary and the gate.
HalfSweet
force-pushed
the
feat/network-v2
branch
2 times, most recently
from
August 18, 2026 16:01
51294e5 to
2fa7f8e
Compare
The network architecture and guest-boundary v2 design notes are internal artifacts and are no longer committed. Point the API docs (docs/NET.md, site/content/docs/net.md) at the in-repo pinned specs (contracts/spec/*), and rewrite the source-comment citations to be self-describing: each now states the rule it used to reference by section number, with no dead repo-relative path. Also correct the ESP-TLS provider header, which described a socket-fd duplicate the implementation does not make.
HalfSweet
force-pushed
the
feat/network-v2
branch
from
August 18, 2026 16:09
2fa7f8e to
62e5843
Compare
The Phase 1A/1B/1C staged-gate terminology came from the internal design notes that are no longer committed. State what each module actually delivers and the plain rule that a target advertises a capability only when its native host ships and tests the module, instead of naming rollout phases. No stock target advertises the network capabilities yet.
HalfSweet
marked this pull request as ready for review
August 19, 2026 02:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds network support to PocketJS as a working stack: three spec-pinned guest modules, a portable C reference core, the Rust reference client core, deterministic + browser + ESP-IDF hosts, and TLS. First target is ESP-IDF; the design keeps other platforms in view. The in-repo documentation is
docs/NET.mdandsite/content/docs/net.md; the pinned boundary iscontracts/spec/{net,ws,httpd}.ts.This replaces the NET v1 SDK surface with the v2 modules.
globalThis.netand op codes 1–5 are preserved and evolved in place; the old@pocketjs/framework/netvalue exports andnet.httpare removed (no in-repo consumers).What's here
contracts/spec/net.tsevolved to v2 (ops 6–9readInto/limits/write/endBody, op 2 retired, streamingheaders/readable/end/errorevents, the shared error vocabulary with the fourtls_*codes); newws.tsandhttpd.ts.gen-rust.tsemits net/ws/httpd; newgen-c.tsemitsengine/net/include/pocketjs/net/spec.h. Both mirrors are under thetests/contract.tsdrift guard.@pocketjs/framework/net(support module:NetworkError,AbortController/AbortSignal,URL,getNetworkLimits, shared types),net/http(fetch,Headers,Request,Response,BodyStream,serve),net/websocket(connect). A per-module Guest Binding drains onepollper tick from the framework service pump, settles Promises and calls handlers inside that pump, and copies bodies throughreadInto. New@pocketjs/framework/headlessruns the frame transaction without a UI.engine/net: HTTP/1.1 client + server, RFC 6455 client, strict framing profile, bounded receive/send queues with backpressure, the immutable policy, tick queues with a per-tick budget andreadable-before-terminal ordering, and the TLS handshake state machine. The only host interfaces arepnet_driver_ops(a BSD/lwIP driver underdrivers/posix) and an optionalpnet_tls_opsTLS provider.engine/crates/pocket-netrewritten to the v2 boundary over anHttpClientBackend;mountinstalls the six ops via rquickjs.hosts/sim/{net,httpd,ws}.ts;hosts/web/net.jsmoved to the streaming v2 contract;hosts/esp-idfwith a QuickJS-ng owner task, a network task under one runtime lock, thenet/ws/httpdbindings, AtomS3R + Tab5 bring-up, and the smoke firmware.pnet_tls_ops+pnet_runtime_create_tls:https:/wss:and the"tls"feature are enabled only when a provider is present; the core owns the deadline/cancel/policy and fails closed withtls_clock_untrustedbefore I/O when the wall clock is untrusted; no plaintext fallback. ReferenceNativeTlsProviderover OpenSSL (engine/net/drivers/openssl) and an ESP-TLS provider (hosts/esp-idf/components/pocketjs_net_esptls, IDF certificate bundle).contracts/spec/platforms.tsreplacesnet.httpwith the role-splitnetwork.http.client(.tls)/network.http.server(.tls)/network.websocket.client(.tls)ids. No stock target advertises them yet: a target appends an id only when its native host ships and tests the module.Execution model (unchanged)
Network facts enter the guest only at frame boundaries: the host runs
begin_tickbefore eachframe(), the service pump polls each module once insideframe(), and Promise reactions run in that tick's job drain. No independent network turn, no wake, noref()/unref();docs/RUNTIMES.mdLaw 3 is untouched.Validation
bunx tsc --noEmit, the contracts drift guard, and the fullbun tools/test.tssuite (unit, wasm-host unit, vue-sfc, octane, and the sim determinism stages) are green. Net SDK coverage:tests/net.test.ts,tests/net-httpd.test.ts,tests/net-websocket.test.ts,tests/net-web.test.js.ctest --test-dir engine/net/buildunder ASan/UBSan:unit(154 checks: HTTP/1.1 head/body strict framing, URL, policy, JSON, UTF-8, base64/SHA-1, tick queue),host(163 checks: both cores over real loopback sockets against scripted peers),tls(16 checks: an in-process OpenSSL PKI + HTTPS/WSS peers — valid chain, unknown CA, expired, hostname mismatch, untrusted clock, development-insecure refusal, no plaintext fallback, WSS echo).cargo test -p pocket-net(7 tests: ordering, budget truncation, cancel/late completions, refusals, mount).7101770dc6db), AtomS3R (ESP32-S3-PICO-1-N8R8) and Tab5 (ESP32-P4 rev 1.3 + ESP32-C6 over SDIO). Each passes 26/26: the plaintext suite (GET/POST/JSON/chunked/404, redirect follow+manual, a 200 KB body through an 8 KiB queue at ~350–370 KiB/s, aggregate limit, headers timeout, permission_denied, connect refused, WebSocket echo, board-to-board GET/POST/JSON/stream/404, continuous pings) against an independent workstation peer (tools/net-peer.ts) and each other, plus a TLS suite: HTTPS/1.1 to a public host with a valid chain from the IDF certificate bundle (SNTP-synced clock) and badssl.com's expired / wrong-host / self-signed / untrusted-root endpoints, all failing closed. A 12-minute board-to-board soak (43,200 frames, ~330 round trips each way) ended with zero failures and flat heap (guest ≈357 KB, core ≈4 KB).Tab5 notes captured in
hosts/esp-idf/README.md: rev 1.3 silicon (CONFIG_ESP32P4_SELECTS_REV_LESS_V3+REV_MIN_100), the C6 SDIO preset with active-high reset on GPIO15, the WLAN rail on the PI4IOE5V6408 IO expander, andFREERTOS_HZ=1000.Out of scope (staged / future)
manifest format 3 +
requiresOneOf, the compiler surface-demand derivation and theglobalThis.net/ws/httpddirect-reference rejection, advertising a network capability from a stock target, record/replay tooling, the PSP network substrate, HTTP Server / WebSocket Server admission on hardware, and the TLS extensions (custom CA, client auth, ALPN, TLS 1.3 requirement, revocation). The ESP-TLS async path exposes the Mbed TLS verify flags inconsistently, so on hardware a hostname mismatch is reported precisely while other certificate faults collapse totls_handshake_failed(always fail-closed); the precise per-fault codes are proven in the desktop OpenSSL conformance suite.Notes for reviewers
This supersedes the experimental stack in #282: the boundary is frame-bounded (no dispatcher/turn ABI), delivery is a bounded
pollinsideframe(), and the native cores never call QuickJS.