[pblazej]

Summary

livekit-telemetry — the shared client-telemetry core — and its UniFFI surface for the Swift, Kotlin and Dart SDKs. Wire spec (events, attributes, cadence, upload policy): livekit-telemetry/SPEC.md.

#1566 stacks on this PR: it takes every telemetry name from the protocol registry.

Changes

• livekit-ffi/src/core_modules/telemetry.rs: configure and configure-pulled, scopes, spans, stats, the app API, the opt-out, device state
• log_forward.rs: the core's own warnings and errors copied into telemetry
• Dart tests
• The telemetry_ping example and the README's local testing guide
• One lifecycle task carries out configure, shutdown and the opt-out's purge one at a time, in the order they were sent; telemetry_configure waits until its pipeline is installed
• A replacing exporter starts only once the one before it stopped (drained within its bound, or cancelled and joined), so two exporters never replay a shared cache at once
• telemetry_shutdown returns once every exporter queued before it stopped, and a shutdown that resumes late never ends a newer lifecycle
• The pull queue never hands out or keeps a request once finish returned, and a pulled configure refused after the opt-out finishes its queue
• Platform log lines passed to telemetry_log or TelemetryScope.log get the same credential masking as the core's own copied lines

What a platform calls, and when

When	Call
SDK init	telemetry_configure(config, transport, instruments) — storage and tuning only; Dart: telemetry_configure_pulled(config, []) + queue.try_next()
Room created	telemetry_scope()
Connect, and every token refresh	scope.set_server(url, token) — cheap, idempotent
Room lifecycle	scope.start(SpanName, parent) → step / set_track / end; subscribe_started / subscribed (tracks already in the room at join: subscribe_started at connect, so lk.subscribe measures from join; a lone subscribed still opens the span as a fallback) / subscribe_failed; track_ended(sid); set_room; disconnected(reason)
Stats	every scope.stats_poll_interval_ms(): scope.record_peer_stats(report, {trackId → sid}, ts) per peer connection
OS signals	telemetry_set_device_state (thermal Unknown / low power None without a source), telemetry_device_event, telemetry_log
App API (the only public surface)	scope.emit_custom(name, {k: v}), scope.set_attribute(key, value?), telemetry_disable() — synchronous, in effect when it returns (TODO on platforms: token discussion); telemetry_is_disabled() reads that opt-out process-wide (every thread and Dart isolate), for platforms to check right before each getStats or submit

No endpoint, header or sink exists in any client-language API. Local end-to-end tests set LK_TELEMETRY_ENDPOINT (read by the core; see the README).

Local testing against a local OpenTelemetry backend (LGTM)

Everything can be tried against a local OpenTelemetry backend (LGTM) in a couple of minutes; no LiveKit server is needed for the Rust example.

# 1. Local OTel backend (OTLP/HTTP on :4318, UI on :3000)
docker run -d --name lk-lgtm -p 3000:3000 -p 4318:4318 grafana/otel-lgtm

# 2. Record a small session (lk.connect with its checkpoints, lk.publish, lk.subscribe ended by
#    first media, a custom event) and export it to the local backend; it prints the trace id
LK_TELEMETRY_ENDPOINT=http://localhost:4318 cargo run -p telemetry_ping

# 3. Optional: the same through the Dart bindings (pull queue)
(cd livekit-ffi && cargo make dart-package) && cd livekit-ffi/packages/dart && dart pub get \
  && LK_TELEMETRY_ENDPOINT=http://localhost:4318 dart test test/live_collector_test.dart

# 4. Optional: the file cache — stop the backend, run step 2 with LK_TELEMETRY_DIR=/tmp/lk-telemetry,
#    start the backend again and run once more to watch the cached batches replay

Then open http://localhost:3000 → Explore: Loki {service_name="telemetry_ping"} for the log records (otel.event.name: custom.ping, lk.rtc.stats.sample, lk.telemetry.report), Tempo with the printed trace id for the spans (lk.connect, lk.publish, lk.subscribe). Platform e2e tests set LK_TELEMETRY_ENDPOINT the same way (with livekit-server --dev for a real Room).

Verification

At a2558a40, from a clean checkout:

• cargo fmt --check
• cargo clippy -p livekit-telemetry --all-targets -- -D warnings, also with --all-features and as a cargo hack --feature-powerset --depth 2 run
• cargo test -p livekit-telemetry: 275 unit, 2 doc
• RUSTDOCFLAGS="-D warnings" cargo doc -p livekit-telemetry --no-deps
• cargo test -p livekit-ffi --no-default-features --features core-modules: 34
• cargo clippy -p livekit-ffi --no-default-features --features core-modules --all-targets: only the 7 warnings that main (6473ec17) has
• cargo clippy -p telemetry_ping --all-targets -- -D warnings
• Kotlin: cargo make --profile release android-package in livekit-ffi generates the bindings and assembleRelease succeeds, and a fresh ./gradlew compileReleaseKotlin --rerun-tasks passes. android-check-size fails: the arm64 library is 2,298,256 bytes against a limit of 1,769,472 bytes (main: 1,593,408 bytes)
• Dart: cargo make dart-package, dart pub get, dart test: 7 pass, 1 skip. The skipped live collector test passes against a local collector

Stages 2–7 pass the same livekit-telemetry gates on their own tips (see each PR).

[xianshijing-lk]

there can be a race condition here, like serve() hands a request to the host after finish() has returned:

finish() (751) sets finished inside the inflight lock. A serve already holding that lock has read finished == false, so it inserts and returns Some(export) while finish blocks behind it; finish then clears the map. next() has no re-check on the Some path. The doc on inflight ("nothing is handed out or kept once finish returned") is only half true — the existing test nothing_is_served_after_finish_returns asserts the map is empty but never checks what serve returned.

Impact: the host performs one more upload — carrying Authorization: Bearer — after the queue was finished, and since its entry is gone, complete() is ignored and the exporter retries → duplicate batch. Not an opt-out leak: serve's separate process && global::is_disabled() check covers that path. This is the shutdown/queue-replacement path.

I reproduced it deterministically at both levels, and this 7-line fix makes both go green with all 18 of the PR's telemetry tests still passing:

pub fn finish(&self) {

• // Before the lock: a serve already holding it then sees the flag on its way out.
• self.finished.store(true, Ordering::SeqCst);
  {
  let mut inflight = self.inflight.lock()...;

•    self.finished.store(true, Ordering::SeqCst);
     inflight.clear();
  

  }
  // in serve(), just before the insert:
  +if self.finished.load(Ordering::SeqCst) {

• return None;
  +}
  inflight.retain(|_, done| !done.is_closed());

[pblazej]

Fixed in 137e6292, 9e2a762a and 92baa589: earlier hand-outs remain answerable, new hand-outs/enqueues cannot commit after finish, and abandoned queued requests are discarded. Cleanup waits for an active poller to release the receiver, as documented in 00c216ac.

[xianshijing-lk]

what will happen if telemetry is off ? does telemetry_copy cost anything or trigger some warning logs ?

[pblazej]

Fixed in 3032ffa3 and c57f5c5e: target/level filtering happens first, and the telemetry copy is built only if the call snapshots an installed pipeline. Host logging is unchanged.

[xianshijing-lk]

it looks like this telemetry_configure can block the caller thread ?

This can be very problematic.

@1egoman and @lukasIO have some experience with AIC and Krisp plugins, and it took us a few iterations to fix the register problems. maybe we can get some suggestions here ?

[1egoman]

I need to dig into the telemetry PRs in depth, sorry I still haven't. That being said, is NetTransport::from_registry actually blocking for a signifigant amount of time? From a quick glance at the code it looks like it's all fairly fast synchronous code and no blocking IO or anything like that.

In case it matters, In the AIC plugin, there have been some recent changes to move synchronous blocking work into a background thread, though that is fairly straightforward thing to do in that case because the return value isn't needed, it's "fire and forget". Here it looks like the return value is important so it might be more involved.

If you want to chat about this more feel free to reach out on slack and I'll share some of the non public AIC plugin PR links.

[pblazej]

The measured registry lookup was cheap, but instrument startup can block configure or deadlock if it waits on the caller. 81393b49 documents synchronous configure and its callback restrictions; the early-answer prototype remains excluded because it changes readiness semantics.

[pblazej]

The measured mobile registry path performs a lookup and an Arc clone, rather than a network request. We retained synchronous configure and documented its blocking work in 81393b49; fire-and-forget would weaken installation-before-return semantics.

[xianshijing-lk]

I am concerned on this approach, how long will telemetry_shutdown() will delay the shutdown ?
I wonder if we should just stop networking and save the data on device, and the data will be uploaded after the app get opened again

[pblazej]

A stalled upload used roughly the default ten-second deadline in the desktop test, and draining generations can accumulate waits. 81393b49 documents the cooperative bounds and cache recovery; the existing drain is retained because persist-only shutdown or a shorter cap changes delivery behavior.