Any foreign C thread → actor send is silently dropped (SPSC from_queues[MAX_CORES] illegitimate 2nd producer)

[paul-hammant]

Summary

A fire-and-forget actor ! Msg issued from a C thread that is neither a scheduler thread nor the single main thread is enqueued to a single-producer queue slot it does not own, and the message is silently lost — no crash, no diagnostic, the receiving actor's arm simply never runs.

Severity: high. std.actors' own module header (std/actors/module.ae:26-30) documents whereis-then-send from a handler as a first-class pattern, and that pattern is silently broken wherever the handler runs off-scheduler (the whole std.http worker pool, std.worker pools, embedding host threads, C-library callback threads).

Discovered: building selaenium's grid hub — a std.http handler sending to a registry actor (a = actors.whereis("ctr"); a ! Inc {}).

Symptom

An HTTP handler on a std.http pool thread does actors.whereis(name) ! Msg{}. The handler completes normally (200, no crash), whereis returns a valid non-null ref, but the actor's receive arm never executes. The identical send from an on-scheduler context (inside another actor's receive) works.

Minimal reproduction

One .ae file, std.http + std.actors, no external deps — send the SAME message to the SAME actor from two origins in one binary:

message Inc {}
actor Ctr { state n = 0
    receive { Inc() -> { println("[Ctr] Inc received"); n = n + 1 } } }

// (A) ON-SCHEDULER: from inside Srv's receive, before the blocking server loop
StartS(raw) -> { c = actors.whereis("ctr")  c ! Inc {}  http_server_start_raw(raw) }   // WORKS

// (B) POOL THREAD: from the http handler
inc_handler(req, res, ud) { c = actors.whereis("ctr")  c ! Inc {}  ... }               // DROPPED

[Ctr] Inc received fires exactly once (from A), never from B, across any number of POSTs. (Pool-thread stdout is block-buffered — run under stdbuf -oL; the handler DOES run, it's the send that's lost.)

Toolchain: ae 0.681.0, also present at HEAD.

Root cause (file:line)

1. A std.http pool worker is a bare pthread_create'd thread (std/net/aether_http_pool.c:121). It never registers a scheduler identity, so its TLS current_core_id stays at the module default -1 (runtime/scheduler/multicore_scheduler.c:238).
2. aether_send_message reads my_core = current_core_id (== -1) (runtime/actors/aether_send_message.c:274); my_core >= 0 is false, so it calls scheduler_send_remote(actor, msg, from_core=-1) (line 279).
3. scheduler_send_remote's from_core < 0 path computes from_idx = MAX_CORES (multicore_scheduler.c:1794) and enqueues to schedulers[target].from_queues[MAX_CORES] (1806-1810).
4. But from_queues[] is SPSC. Header contract, multicore_scheduler.h:153: "Per-sender SPSC channels: from_queues[src] is written ONLY by core src." [MAX_CORES] is the reserved slot for the ONE main thread; queue_enqueue loads tail memory_order_relaxed assuming one fixed producer. A pool worker is a second, different OS thread writing that single-producer slot — an illegitimate concurrent producer. Its tail publication isn't correctly observed by the consumer, and the message is dropped.

This is exactly why main's startup sends work (main is the legitimate sole producer of [MAX_CORES]) while a pool worker's send is lost, though both pass from_core = -1 down the identical path.

Not a lost-wakeup / timing issue

Scheduler threads park with pthread_cond_timedwait (multicore_scheduler.c:1008) and re-poll q = 0..MAX_CORES every loop (line 577). A validly-enqueued message would drain within the park timeout regardless of any sched_wake race. It never drains ⇒ it was never validly enqueued. The defect is structural (wrong queue for the producer), not a wake race.

The runtime already shows the intended discipline

std/net/aether_actor_bridge.c:52-58 (ae_io_await) hits the same current_core_id < 0 case and deliberately routes through a real core rather than the foreign slot:

int core = current_core_id;
if (core < 0) { /* actor still assigned to a scheduler core; route through it */ core = 0; }

The send path is missing this discipline. (Corroborating: the counter on this path is atomic — main_thread_sent via atomic_fetch_add, multicore_scheduler.c:1745 — while the queue write right below it is not given equivalent multi-writer safety.)

Suggested fixes (cheapest general fix first)

1. (preferred, general) A real MPSC / locked ingress for from_core < 0 sends that aren't the main thread: a small mutex-guarded per-core "external inbox," drained alongside from_queues in the scheduler loop. Correct for ANY number of foreign producers (std.http pool, std.worker pools, embedding hosts), not just one.
2. Have runtime-owned worker pools acquire a real core-id identity on thread start, so their sends take the on-scheduler path with a legitimate from_idx.
3. (minimum) Detect the genuinely-foreign-thread case in scheduler_send_remote and fall back to a locked mailbox_send / overflow path instead of the SPSC slot.

Leaning (1): the general fix that unblocks handler→actor hand-off for every C-thread producer — the shape real servers need. This is runtime/scheduler C, so it's a design call for the runtime owner.

Related (this arc)

• Doc bug: the http actor-dispatch API that WOULD let a handler run on-scheduler (a legitimate producer) is only half-wired from released .ae — separate doc-fix PR + asks/REBUTTAL-http-actor-dispatch-doc-is-not-a-usable-ae-api.md. It dovetails: actor-dispatch mode "fixes" the send only because the handler then runs on-scheduler.
• Reclamation gap: std.sync: expose a reclamation primitive (atomic int) to .ae for pool-owned COW structures #2082 (std.sync). Together these three are the Aether-side items the distributed-grid build surfaced.

Released workaround (used by the grid): don't hand off to an actor from a pool handler — mutate a lock-free snapshot.cas structure on the pool thread. That's what selaenium's grid registry now does.