Skip to content

fix(rtc): create the FfiClient singleton under a lock - #918

Merged
longcw merged 1 commit into
mainfrom
longc/ffi-client-singleton-lock
Nov 5, 2026
Merged

longcw merged 1 commit into
mainfrom
longc/ffi-client-singleton-lock

Conversation

@longcw

@longcw longcw commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Problem: Two threads that read FfiClient.instance at the same time each create their own FfiClient, and FFI events reach only the queue of the last one. A Room.connect() that listens on the other queue never sends ReadyForRoomEventRequest, so after 15 s the FFI server panics and the process exits with every room in it.

Fix: FfiClient.instance returns an existing client without a lock, and creates a new one only under a class-level lock that checks _instance again. Every thread gets the same client, so every connect callback reaches its caller.

Fixes livekit/agents#7545.

Context for reviewing and coding agents

How to see it

Two threads each run their own event loop and call Room.connect() to two rooms at the same moment, after a threading.Barrier. On main, one room connects, the other hangs, and the process logs timed out waiting for ReadyForRoomEventRequest after ConnectCallback and exits with SIGTERM, in 8 of 8 runs. With this change, both rooms connect and the process exits cleanly in 5 of 5 runs. Two threads that only read FfiClient.instance behind a barrier get two objects on main in 5 of 5 runs, and one object with this change.

Who hits it

The agents worker runs jobs as threads when job_executor_type=JobExecutorType.THREAD, which is the default on Windows. Two dispatches that arrive together make two jobs call ctx.connect() together. The first connect() in the process is also the first read of FfiClient.instance, so the race window is the whole of FfiClient.__init__, which loads the library and calls livekit_ffi_initialize.

Why the lock is here and not in the caller

FfiClient.instance is the only place that creates the client. A caller cannot serialize the first read, because Room.connect() makes it internally.

What this does not change

A missing ReadyForRoomEventRequest for one room still makes livekit-ffi call send_panic, and ffi_event_callback then sends SIGTERM to the whole process. That escalation belongs in rust-sdks, and this change removes the cause seen in livekit/agents#7545.

Two threads that read FfiClient.instance at the same time each created
their own client, and FFI events reached only one of their queues. A
Room.connect() listening on the other queue never sent
ReadyForRoomEventRequest, so the FFI server panicked after 15 s and the
process exited with every room in it.

Fixes livekit/agents#7545.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@longcw
longcw merged commit 9beff01 into main Nov 5, 2026
36 of 51 checks passed
@longcw
longcw deleted the longc/ffi-client-singleton-lock branch November 5, 2026 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Worker panics (FFI Panic: ReadyForRoomEventRequest timeout) when two jobs' ctx.connect() run concurrently, killing all jobs on the worker

2 participants