feat(telemetry): trace dispatch, startup, and shutdown

[davidzhao]

Port of livekit/agents#7131. Stacked on #2496.

Description

Before the first turn and after the last one the trace had a single span, job_entrypoint, so "the agent took 4 s to say hello" and "the job hung on shutdown" could not be split into their steps. This adds the dispatch timeline, the startup spans, the shutdown spans, and the join keys for lining the agent trace up with server, SIP, and client events.

Dispatch timeline. The worker stamps receivedAt, acceptedAt, assignedAt on the running job info; the process pool stamps launchedAt. job_entrypoint is back-dated to receivedAt and gets one timestamped event per stage (job_received, job_accepted, job_assigned, process_assigned, entrypoint_started, job_started_on_server) and the seconds between adjacent stages as lk.job.accept_latency, assignment_latency, launch_latency, entrypoint_latency, and their sum dispatch_latency. Unknown stages are skipped, not guessed.

job_entrypoint spans the whole job. Created before the user entrypoint runs and ended after job_shutdown; the entrypoint returning is an entrypoint_returned event, a throwing one is recorded redaction-aware. agent_session is now its child, so one trace covers dispatch → session → teardown. Join keys lk.room_sid, lk.dispatch_id, lk.job.worker_id, lk.job.agent_id on both; a linked SIP participant's sip.* attributes land on agent_session as lk.sip.*, with only sip.phoneNumber under lk.pii..

The trace pipeline is up before the job's first span. The cloud tracer provider used to be created in initRecording inside session.start(), so in a process's first job every earlier span was a non-recording stub. JobContext._prepareTelemetry now creates the provider and a per-job gated exporter when the job starts; the gate holds the job's spans (oldest first, bounded at 1024) until initRecording decides, then uploads (redacting if the job asked for it) or drops them, and drops them at cleanup if the job never registers.

Startup: room_connect, session_start (grouping start_agent_activity, setup_toolsets, wait_for_participant, wait_for_audio_track with track_subscribed / first_frame events and lk.first_frame_delay, publish_audio_output), events on agent_session for participant, connection and state changes. Startup spans are never made current: they are parented explicitly so the long-lived tasks they spawn do not inherit them.

Shutdown: session_close (lk.close_reason, lk.close.drain; drain_agent_activity and on_exit inside), job_shutdown (lk.shutdown.reason, lk.shutdown.user_initiated; children on_session_end, session_end_upload, room_disconnect, one shutdown_callback per callback named by lk.callback.name).

job_entrypoint  (back-dated to job_received; events per stage)
├─ agent_session
│  ├─ session_start
│  │  ├─ room_connect
│  │  ├─ wait_for_participant
│  │  ├─ wait_for_audio_track
│  │  ├─ publish_audio_output
│  │  └─ start_agent_activity
│  │     ├─ setup_toolsets
│  │     └─ on_enter
│  ├─ user_turn / agent_turn ...
│  └─ session_close
│     ├─ drain_agent_activity
│     └─ on_exit
└─ job_shutdown
   ├─ on_session_end
   ├─ session_end_upload
   ├─ room_disconnect
   └─ shutdown_callback (×N)

Changes Made

• ipc/job_trace.ts (new): startJobSpan, recordDispatchTimeline, serverTimestampMs, callbackName.
• telemetry/span_gate.ts (new): JobSpanGateExporter; telemetry/traces.ts: prepareCloudTracer / discardPreparedCloudTracer / flushCloudTraces, tracer.detachedSpan, shared auth/resource/metadata helpers; telemetry/redaction.ts: redactReadableSpan for held spans released after the job chose redaction.
• ipc/job_proc_lazy_main.ts, job.ts, job_lifecycle.ts: prepare telemetry, root span lifecycle, job_shutdown tree, final trace flush before process.exit.
• worker.ts, ipc/proc_pool.ts: dispatch timestamps (epoch ms; structured-cloned through IPC, no proto change).
• voice/agent_session.ts, voice/agent_activity.ts, voice/room_io/*: session_start / session_close, explicit trace contexts, session events, SIP attributes, wait_for_audio_track (later participant switches parent to the session root).
• telemetry/session_context.ts: sessionSpan, sessionStartContext; telemetry/utils.ts: participantAttributes.
• ipc/_preload.ts (new): the framework's one-time warm-up before the user prewarm.

Adaptations from the Python source

• No forkserver in Node: _preload.ts warms only the local inference binding; the rtc-node binding loads at import time and its runtime cannot be started early. asyncio.to_thread cleanup offloading: not applicable.
• entrypoint_cancelled: JS never cancels the entrypoint promise; omitted. pre_connect_audio event: JS RoomIO has no pre-connect audio handler.
• JS registers no framework shutdown callback, so every callback gets a shutdown_callback span.
• New flushJobTraces step: the JS job process never force-flushed traces before process.exit(0); without it the now-late-ending job_entrypoint would be lost.
• AutoSubscribe is rendered as subscribe_all|subscribe_none|video_only|audio_only to match the Python attribute values.

Testing

• New ipc/job_trace.test.ts, telemetry/startup_spans.test.ts, telemetry/prepared_tracer.test.ts, voice/session_lifecycle_span.test.ts; job_lifecycle.test.ts, proc_pool.test.ts, recording_disabled.test.ts, trace_types.test.ts extended.
• Not covered by unit tests (needs a real room): wait_for_audio_track first-frame timing and publish_audio_output.
• Full agents suite green; build, typecheck, lint, API report updated.

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 6ad23e6

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages

Name	Type
@livekit/agents	Minor
@livekit/agents-plugin-anam	Minor
@livekit/agents-plugin-anthropic	Minor
@livekit/agents-plugin-assemblyai	Minor
@livekit/agents-plugin-azure	Minor
@livekit/agents-plugin-baseten	Minor
@livekit/agents-plugin-bey	Minor
@livekit/agents-plugin-cartesia	Minor
@livekit/agents-plugin-cerebras	Minor
@livekit/agents-plugin-deepgram	Minor
@livekit/agents-plugin-did	Minor
@livekit/agents-plugin-elevenlabs	Minor
@livekit/agents-plugin-fishaudio	Minor
@livekit/agents-plugin-google	Minor
@livekit/agents-plugin-hume	Minor
@livekit/agents-plugin-inworld	Minor
@livekit/agents-plugin-krisp	Minor
@livekit/agents-plugin-lemonslice	Minor
@livekit/agents-plugin-liveavatar	Minor
@livekit/agents-plugin-livekit	Minor
@livekit/agents-plugin-meta	Minor
@livekit/agents-plugin-minimax	Minor
@livekit/agents-plugin-mistral	Minor
@livekit/agents-plugin-mistralai	Minor
@livekit/agents-plugin-neuphonic	Minor
@livekit/agents-plugin-openai	Minor
@livekit/agents-plugin-perplexity	Minor
@livekit/agents-plugin-phonic	Minor
@livekit/agents-plugin-protoface	Minor
@livekit/agents-plugin-resemble	Minor
@livekit/agents-plugin-rime	Minor
@livekit/agents-plugin-runway	Minor
@livekit/agents-plugin-sarvam	Minor
@livekit/agents-plugin-silero	Minor
@livekit/agents-plugin-soniox	Minor
@livekit/agents-plugin-tavus	Minor
@livekit/agents-plugins-test	Minor
@livekit/agents-plugin-trugen	Minor
@livekit/agents-plugin-xai	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR