[Feature] Improve LingBot WebRTC streaming controls and lifecycle#12
Merged
Conversation
Introduce streaming telemetry, the session-owned scheduler, and actor execution primitives for long-running LingBot generation. The scheduler foundation includes its actor runtime and baseline tests; later commits migrate services and add lifecycle hardening.
Route LingBot service sessions through the actor-based streaming scheduler and update the pipeline lifecycle around the new runtime. This connects the scheduler foundation to bidirectional generation while keeping session state actor-owned.
Add actor health barriers, resource-group permits, and scheduler validation for bounded long-running streaming sessions. Include regression coverage and benchmarking support for admission, failure, and lifecycle behavior.
Use browser-provided reference images consistently when constructing LingBot stream sessions and camera-control geometry. Keep the interactive service configuration aligned with the selected input image.
Route bidirectional generated frames directly through RTP, add bounded video buffering, and preserve the most recent frame while an interactive stream is idle. Expose video transport settings through the service configuration and cover raw-frame routing behavior.
Bound outgoing media buffers, retain the last rendered frame for idle interactive sessions, and close abandoned bidirectional sessions after DataChannel or disconnect deadlines. Wire lifecycle settings through the API and keep pipeline session cleanup off the event loop.
Store causal KV cache cursors in mutable scalar tensors so their updates survive shallow copies across actor and FSDP call boundaries. Add regression coverage for cache cursor mutation semantics.
Replace stale directional buffering with latest browser control snapshots, keep movement and rotation mappings distinct, and tie each chunk HUD to its applied control snapshot. Add reset-pose handling, bounded control cleanup, native frame metadata, and translation scale configuration for the v2 stream example.
lzx1413
marked this pull request as ready for review
July 20, 2026 12:23
Document the actor-based streaming scheduler, resource placement, native bidirectional frame transport, and current LingBot v2 camera-control workflow. Refresh the English and Chinese setup guides, examples, TURN instructions, and stream-service references.
Load LingBot VAE, text encoder, and DiT through ModuleManager with explicit checkpoint shards and converter-based DiT construction. Replace the shared VAE runtime placement with independent encode and decode runtime configurations, and expose their device IDs in the v1 and v2 examples. Update scheduler documentation and coverage for conversion, preloaded modules, placement, and examples. Verification: - ruff check - git diff --check - pytest -q tests/unit/models/test_lingbot_world_fast_dit_converter.py tests/unit/pipelines/lingbot_world_fast tests/unit/pipelines/lingbot_world_v2 - Four-GPU offline LingBot v1/v2 generation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Improve LingBot streaming reliability and interactive WebRTC control behavior.
The PR introduces the session-aware actor scheduler for LingBot streaming, native RTP frame delivery with bounded latest-frame buffering, stronger WebRTC session cleanup, and synchronized camera-control snapshots. It also refreshes the v1/v2 examples and stream-server documentation.
Motivation
Long-running LingBot sessions need bounded memory, isolated per-session state, and predictable cleanup. The previous control path could retain stale directional input, while generated-frame metadata and the HUD could diverge from the controls actually applied to a chunk.
This change also removes an unnecessary JPEG/base64 round trip for bidirectional LingBot output and makes session progress and queue behavior observable.
Type of Change
Changes Made
framesinto RTP, with bounded latest-frame queues that drop stale frames under client backpressure.applied_controlsmetadata aligned with the controls submitted to the model.WASD/ arrows) from rotation (IJKL), add reset-pose behavior, and set realtime translation scale to3.0for both v1 and v2 examples.Testing
Test commands: