Skip to content

feat(telemetry): trace RPC calls made and handled - #2498

Draft
davidzhao wants to merge 1 commit into
dz/telemetry-startup-shutdownfrom
dz/telemetry-rpc
Draft

davidzhao wants to merge 1 commit into
dz/telemetry-startup-shutdownfrom
dz/telemetry-rpc

Conversation

@davidzhao

@davidzhao davidzhao commented Sep 15, 2026

Copy link
Copy Markdown
Member

Port of livekit/agents#7134. Stacked on #2497.

Description

Neither user-registered RPC methods nor the avatar datastream RPCs appeared in a trace. This installs a tracing interceptor on the job's local participant and turns each RPC into a span.

Span Kind Parent
rpc_call CLIENT the current span, so an RPC issued from a tool nests under function_tool
rpc_handler SERVER the primary session's root span, so it lands on the session timeline; the job's root (job_entrypoint) before or after the session

Attributes: rpc.method, lk.rpc.request_id, lk.rpc.caller_identity, lk.rpc.destination_identity, lk.rpc.payload_size, lk.rpc.response_size, lk.rpc.response_timeout, lk.rpc.error_code, lk.rpc.handler_registered. Request and response bodies are under PII keys lk.pii.rpc.payload / lk.pii.rpc.response, truncated to 1 KiB. RpcError and handler exceptions set error status.

Installed from JobContext.connect() after the room connects, and from RoomIO.start() on every connect for a room connected elsewhere. One interceptor per process; the SDK dedups registrations by identity.

Depends on the SDK

The interceptor hook is LocalParticipant.addRpcInterceptor from livekit/node-sdks#724, not yet in the installed @livekit/rtc-node. Like the Python layer, this degrades to a no-op with one debug log on an SDK without it: the interceptor shapes are declared structurally in telemetry/rpc.ts, install feature-detects the method, and no dependency bump is needed now. Once the SDK ships, bumping @livekit/rtc-node turns tracing on without further code changes.

Note that the FFI layer answers UNSUPPORTED_METHOD for unregistered methods before the SDK's handler runs, so lk.rpc.handler_registered=false is only set on the defensive path where the chain sees that error.

Changes Made

  • telemetry/rpc.ts (new): TracingRpcInterceptor, install; exported as telemetry.rpc.
  • telemetry/traces.ts: StartSpanOptions.kind passed through startSpan / startActiveSpan / startActiveSpanSync.
  • telemetry/trace_types.ts: eleven new attributes (the two lk.pii.rpc.* keys are PII).
  • job.ts, voice/room_io/room_io.ts: install points.

Adaptations from the Python source

  • Python's CancelledError / cancel_reason branch and its tests are not ported: Node promises are not cancelled and the SDK never cancels the handler chain.
  • install(undefined) is tolerated since room.localParticipant is optional in the JS SDK types.

Testing

  • New telemetry/rpc.test.ts (11 tests) with fake continuations: attributes and parenting both directions, truncation and byte sizes, error codes and status, the unregistered flag on the defensive path, handler exceptions, install once / degrade; room_io.test.ts install-on-connect case.
  • Full agents suite green; build, typecheck, lint, API report updated.

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Sep 15, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c64e497

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages
Name Type
@livekit/agents Minor
@livekit/agents-plugin-anam Minor
@livekit/agents-plugin-anthropic Minor
@livekit/agents-plugin-assemblyai Minor
@livekit/agents-plugin-azure Minor
@livekit/agents-plugin-baseten Minor
@livekit/agents-plugin-bey Minor
@livekit/agents-plugin-cartesia Minor
@livekit/agents-plugin-cerebras Minor
@livekit/agents-plugin-deepgram Minor
@livekit/agents-plugin-did Minor
@livekit/agents-plugin-elevenlabs Minor
@livekit/agents-plugin-fishaudio Minor
@livekit/agents-plugin-google Minor
@livekit/agents-plugin-hume Minor
@livekit/agents-plugin-inworld Minor
@livekit/agents-plugin-krisp Minor
@livekit/agents-plugin-lemonslice Minor
@livekit/agents-plugin-liveavatar Minor
@livekit/agents-plugin-livekit Minor
@livekit/agents-plugin-meta Minor
@livekit/agents-plugin-minimax Minor
@livekit/agents-plugin-mistral Minor
@livekit/agents-plugin-mistralai Minor
@livekit/agents-plugin-neuphonic Minor
@livekit/agents-plugin-openai Minor
@livekit/agents-plugin-perplexity Minor
@livekit/agents-plugin-phonic Minor
@livekit/agents-plugin-protoface Minor
@livekit/agents-plugin-resemble Minor
@livekit/agents-plugin-rime Minor
@livekit/agents-plugin-runway Minor
@livekit/agents-plugin-sarvam Minor
@livekit/agents-plugin-silero Minor
@livekit/agents-plugin-soniox Minor
@livekit/agents-plugin-tavus Minor
@livekit/agents-plugins-test Minor
@livekit/agents-plugin-trugen Minor
@livekit/agents-plugin-xai Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Port of livekit/agents#7134. `telemetry.rpc` installs one tracing
interceptor on the job's local participant and turns each RPC into a
span following the OpenTelemetry RPC semantic conventions: `rpc_call`
(CLIENT, under the current span, so an RPC issued from a tool nests
under `function_tool`) and `rpc_handler` (SERVER, under the primary
session's root span). Attributes cover method, ids, identities, payload
and response sizes, the response timeout in seconds, the RpcError code
and whether a handler was registered; request and response bodies are
recorded truncated to 1 KiB under `lk.pii.rpc.*` keys.

Installed from JobContext.connect() after the room connects and from
RoomIO on every connected transition; both are idempotent, the SDK
dedups the singleton interceptor by identity. `StartSpanOptions` gains
`kind`.

Adaptation: the installed @livekit/rtc-node predates
LocalParticipant.addRpcInterceptor (it is in an open SDK PR), so the
interceptor and call shapes are declared structurally here and `install`
feature-detects the hook, degrading to one debug log and a no-op like
the Python layer on an older livekit-rtc. Not ported: the Python
handling of asyncio cancellation of the handler chain (promises are not
cancelled in Node; the SDK maps timeouts on the caller's side).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant