Skip to content

Fix stale and overlapping remote participant updates - #1253

Open
hiroshihorie wants to merge 3 commits into
mainfrom
hiroshi/participant-update-races
Open

hiroshihorie wants to merge 3 commits into
mainfrom
hiroshi/participant-update-races

Conversation

@hiroshihorie

Copy link
Copy Markdown
Member

Fixes #1248.

Two races let a remote participant end up without tracks the server says it has.

Stale updates were applied. Participant.updateFromInfo rejects an info with an older version for the same sid, but RemoteParticipant has had its early return commented out since #435. A stale info still reconciled tracks and removed every publication that only the newer info had. It now returns early, like LocalParticipant and the JS SDK (livekit/client-sdk-js#663). The server only raises the version for a given sid and a rejoin gets a new sid, so no legitimate update is dropped.

Updates were applied concurrently. The room signal listener is not synchronized, so ParticipantUpdate handlers overlap across awaits. Right after joining, every update that arrives while the engine sets up its peer connections waits for RoomConnectedEvent, and they all resume together. Later on, frames read from the socket together are dispatched before either handler finishes. This has two outcomes:

  • An older update removes a track that a newer one added. This is the sequence in the issue.
  • Two updates for a new participant both miss the lookup and create two RemoteParticipants, each with its own ParticipantConnectedEvent. The app keeps the first instance, which the room no longer updates.

Participant creation from the JoinResponse and ParticipantUpdate handling now run under one lock. Pending track flushing runs after the lock is released, because addSubscribedMediaTrack can wait for a later update to publish the track.

Testing

  • test/core/participant_update_test.dart drives the room through the mock websocket. It covers four cases:

    • a stale update after a newer one
    • the join sequence from the issue: v2 [], then v3 [A], v4 [A], v5 [A, B] queued behind the JoinResponse
    • two queued updates for a new participant
    • a pending track that waits for a later update to publish it

    The first three fail on main. The last one fails if flushing moves inside the lock.

  • flutter test passes and flutter analyze is clean.

  • E2E against livekit-server --dev 1.13.8. Three lk publishers keep joining and publishing two tracks. Meanwhile a subscriber joins 60 times and checks every remote participant against the newest ParticipantInfo the server sent.

    • main fails 32 of 60 trials: 33 duplicate ParticipantConnectedEvents, and 9 times the room holds a different instance than the event delivered.
    • This branch passes 60 of 60.
    • The scenario is not in e2e-flutter yet.

Participant.updateFromInfo returns false when the incoming info has an
older version for the same participant sid, but the remote override had
its early return commented out. The stale info then went on to
reconcile tracks and removed every publication that only the newer info
contained.

Return early like LocalParticipant and the JS SDK already do. The
server only increases the version for a given sid and a rejoin gets a
new sid, so this never drops a legitimate update.
The room signal listener is not synchronized, so each ParticipantUpdate
runs _onParticipantUpdateEvent concurrently with the ones before it.
Right after joining this is the common case: the server sends the
JoinResponse and then flushes the updates it queued, and every one of
them parks on the RoomConnectedEvent wait and resumes at once. Their
track reconciliation then interleaves across awaits, and an older
update removes the publication a newer one added. The same window lets
two updates for a new participant both miss the lookup in
_getOrCreateRemoteParticipant and create two instances, each with its
own ParticipantConnectedEvent.

Run JoinResponse participant creation and ParticipantUpdate handling
under one lock. Pending track flushing moves out of the locked section
because addSubscribedMediaTrack can wait for a later update to publish
the track, which would otherwise block on the lock until the publish
timeout.
@hiroshihorie
hiroshihorie marked this pull request as ready for review October 22, 2026 21:14

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] Stale and overlapping ParticipantInfo updates remove newer remote track publications (2.9.0)

2 participants