Skip to content

[Feature] Improve LingBot WebRTC streaming controls and lifecycle#12

Merged
lzx1413 merged 10 commits into
mainfrom
refact_lingbot
Jul 20, 2026
Merged

[Feature] Improve LingBot WebRTC streaming controls and lifecycle#12
lzx1413 merged 10 commits into
mainfrom
refact_lingbot

Conversation

@lzx1413

@lzx1413 lzx1413 commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Description

Improve LingBot streaming reliability and interactive WebRTC control behavior.

The PR introduces the session-aware actor scheduler for LingBot streaming, native RTP frame delivery with bounded latest-frame buffering, stronger WebRTC session cleanup, and synchronized camera-control snapshots. It also refreshes the v1/v2 examples and stream-server documentation.

Motivation

Long-running LingBot sessions need bounded memory, isolated per-session state, and predictable cleanup. The previous control path could retain stale directional input, while generated-frame metadata and the HUD could diverge from the controls actually applied to a chunk.

This change also removes an unnecessary JPEG/base64 round trip for bidirectional LingBot output and makes session progress and queue behavior observable.

Type of Change

  • Bug fix
  • New feature
  • Performance improvement
  • Code refactoring
  • Documentation update

Changes Made

  • Add a session-aware actor streaming scheduler, health barriers, resource permits, and scheduler validation coverage.
  • Preserve LingBot KV-cache cursors across actor/FSDP boundaries.
  • Route bidirectional output as native frames into RTP, with bounded latest-frame queues that drop stale frames under client backpressure.
  • Add WebRTC DataChannel-open timeout, disconnect grace handling, bounded pipeline-close calls, and session lifecycle metrics.
  • Expose runtime progress, latency, queue high-watermark, dropped-frame, and scheduler telemetry through the DataChannel and demo UI.
  • Replace stale directional-command buffering with latest control-state snapshots.
  • Keep the per-chunk HUD and applied_controls metadata aligned with the controls submitted to the model.
  • Separate translation (WASD / arrows) from rotation (IJKL), add reset-pose behavior, and set realtime translation scale to 3.0 for both v1 and v2 examples.
  • Update English and Chinese stream-server, scheduler, TURN, and LingBot v2 documentation.

Testing

  • Targeted unit tests pass
  • WebRTC integration tests pass
  • Manual stream-server/TURN startup and health-check validation performed
  • Benchmarks/validation tooling added
  • New regression tests added
  • Documentation updated

Test commands:

.venv/bin/python -m pytest -q \
  tests/unit/orchestrator/test_parallel_worker_stage_actor.py \
  tests/unit/orchestrator/test_streaming_pipeline_orchestrator.py \
  tests/unit/pipelines/lingbot_world_fast/test_control_alignment.py \
  tests/unit/pipelines/lingbot_world_fast/test_runtime_baseline.py \
  tests/unit/pipelines/lingbot_world_fast/test_service_action_loop.py \
  tests/unit/pipelines/lingbot_world_fast/test_stream_example.py \
  tests/unit/service/test_service_routes.py \
  tests/unit/service/test_webrtc_session_manager.py

.venv/bin/python -m pytest -q \
  tests/integration/test_webrtc_api.py::TestWebRTCBidirectional

git diff --check

## Checklist

- [x] Code follows project formatting and lint rules
- [x] Commit hooks pass
- [ ] Full pytest tests/ suite run
- [x] New tests added for new functionality
- [x] Documentation updated
- [x] Commit history is split by scheduler, transport, lifecycle, controls, and docs

## Related Issues

N/A

## GPU Architecture Support

No kernels were added or modified.

## Performance Impact

Bidirectional LingBot output now bypasses generated-frame JPEG/base64 serialization and uses bounded latest-frame buffering. Under a slow client, stale frames are intentionally dropped to preserve freshness and bound memory use.

lzx1413 added 8 commits July 20, 2026 12:12
Introduce streaming telemetry, the session-owned scheduler, and actor execution primitives for long-running LingBot generation.

The scheduler foundation includes its actor runtime and baseline tests; later commits migrate services and add lifecycle hardening.
Route LingBot service sessions through the actor-based streaming scheduler and update the pipeline lifecycle around the new runtime.

This connects the scheduler foundation to bidirectional generation while keeping session state actor-owned.
Add actor health barriers, resource-group permits, and scheduler validation for bounded long-running streaming sessions.

Include regression coverage and benchmarking support for admission, failure, and lifecycle behavior.
Use browser-provided reference images consistently when constructing LingBot stream sessions and camera-control geometry.

Keep the interactive service configuration aligned with the selected input image.
Route bidirectional generated frames directly through RTP, add bounded video buffering, and preserve the most recent frame while an interactive stream is idle.

Expose video transport settings through the service configuration and cover raw-frame routing behavior.
Bound outgoing media buffers, retain the last rendered frame for idle interactive sessions, and close abandoned bidirectional sessions after DataChannel or disconnect deadlines.

Wire lifecycle settings through the API and keep pipeline session cleanup off the event loop.
Store causal KV cache cursors in mutable scalar tensors so their updates survive shallow copies across actor and FSDP call boundaries.

Add regression coverage for cache cursor mutation semantics.
Replace stale directional buffering with latest browser control snapshots, keep movement and rotation mappings distinct, and tie each chunk HUD to its applied control snapshot.

Add reset-pose handling, bounded control cleanup, native frame metadata, and translation scale configuration for the v2 stream example.
@lzx1413 lzx1413 changed the title feat(lingbot): add WebRTC runtime telemetry [Feature] Improve LingBot WebRTC streaming controls and lifecycle Jul 20, 2026
@lzx1413
lzx1413 marked this pull request as ready for review July 20, 2026 12:23
lzx1413 added 2 commits July 20, 2026 12:33
Document the actor-based streaming scheduler, resource placement, native bidirectional frame transport, and current LingBot v2 camera-control workflow.

Refresh the English and Chinese setup guides, examples, TURN instructions, and stream-service references.
Load LingBot VAE, text encoder, and DiT through ModuleManager with explicit checkpoint shards and converter-based DiT construction.

Replace the shared VAE runtime placement with independent encode and decode runtime configurations, and expose their device IDs in the v1 and v2 examples. Update scheduler documentation and coverage for conversion, preloaded modules, placement, and examples.

Verification:
- ruff check
- git diff --check
- pytest -q tests/unit/models/test_lingbot_world_fast_dit_converter.py tests/unit/pipelines/lingbot_world_fast tests/unit/pipelines/lingbot_world_v2
- Four-GPU offline LingBot v1/v2 generation
@lzx1413
lzx1413 merged commit 35af611 into main Jul 20, 2026
5 checks passed
@lzx1413
lzx1413 deleted the refact_lingbot branch July 21, 2026 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant