Skip to content

[Agent] Include audio in the native (non-go2rtc) HLS fallback #651

Open

Description

@matteius

Problem and evidence

When go2rtc native HLS isn't available (ENABLE_GO2RTC=OFF, or go2rtc down), LightNVR falls back to its own FFmpeg HLS writer. That writer produces video only, so live HLS has no audio in fallback mode and nothing tells the user.

  • src/video/hls/hls_unified_thread.c:~1361: non-video packets are logged ("Skipping audio packet ...") and dropped.
  • src/video/hls_writer.c:585: /* This writer emits video only. ... */. All packets are remapped to output_ctx->streams[0].
  • The README advertises "HLS Streaming". Recording (MP4) already handles audio, including AAC transcoding via src/video/mp4_writer_utils.c (FAAC/FFmpeg encoder).

Expected behavior and scope

  • When the source has an audio track and the stream's audio is enabled, the fallback HLS writer adds an audio stream to its output and muxes audio packets into the segments.
  • AAC sources pass through (copy). Non-AAC sources (G.711 PCMA/PCMU, etc.) either get transcoded to AAC by reusing the existing audio transcoder in mp4_writer_utils.c, or are skipped with a single log_info per stream. Copy-only plus a clear skip is an acceptable first PR; say which one you did.
  • Map stream indices properly instead of forcing streams[0]. RTSP may list audio before video.
  • Timestamps: rescale audio from its own input timebase, and keep A/V in sync across segment rollover.

Acceptance criteria

  • With go2rtc disabled, an RTSP camera with AAC audio plays in hls.js with audible, in-sync audio.
  • A camera with no audio, or with audio disabled for the stream, produces exactly the same output as today (video only).
  • No regression in segment duration, keyframe-aligned cuts, or the PTS/DTS fixups in hls_writer.c.
  • Unsupported audio codecs don't break video output.

Validation

  • Unit test using a synthetic FFmpeg input with video and AAC (generated in the test, like other tests/unit media tests) that checks the output segment has two streams.
  • Manual check with ENABLE_GO2RTC=OFF and a test RTSP source (e.g. mediamtx + ffmpeg -f lavfi sine).
  • Run the full ctest after building all targets.

Constraints and rollback

  • Don't add CPU-heavy transcoding by default on embedded profiles. Gate transcoding behind the existing per-stream audio settings.
  • Rollback: revert. Video-only behavior is the current baseline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions