WIP: Ffmpeg decoder, encoders and filters. - #10990
Draft
lgirdwood wants to merge 69 commits into
Draft
Conversation
Add a loadable (LLEXT) audio module that wraps FFmpeg. Two build modes: - decoder (default): a compressed elementary stream (FLAC/AAC/Opus, selected via Kconfig) is parsed and decoded to PCM (process_raw_data); - filter (CONFIG_FFMPEG_DEC_FILTER_MODE): a PCM source/sink effect that runs an FFmpeg audio filter graph (afftdn noise reduction). FFmpeg is a west-pinned source (see west.yml) cross-built by ffmpeg.cmake as a CMake ExternalProject, enabling only the Kconfig-selected decoders and filters. The module supplies the libc surface FFmpeg needs that the SOF core does not export to LLEXT (ffmpeg_dec-shims.c), a SOF-heap-backed malloc (ffmpeg_dec-alloc.c), fast single-precision float math (fastmathf.c), and routes av_log() into the Zephyr log. The avfilter graph backend and the PCM effect ops are in ffmpeg_dec-filter.c. Also adds the rimage module manifest (ffmpeg_dec.toml + per-platform includes), the topology widget (ffmpeg_dec.conf), host test tooling (ffmpeg_dec_prepare.sh) and docs (README/TESTING). Verified building and load-ready (all symbols resolved, signed) for ace30 (ptl) with the Zephyr SDK: FLAC and FLAC+AAC decoders, and the afftdn filter effect. Runtime decode/denoise verification is pending hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the ffmpeg_dec component to the host->component->host benchmark harness so a topology instantiating the module can be built and driven (testbench or target). Adds the per-format bench configs (generated with bench_comp_generate.sh), registers the widget include and the ffmpeg_dec32 BENCH_CONFIG in cavs-benchmark-hda.conf, and adds ffmpeg_dec to the s32 component list in tplg-targets-bench.cmake (the filter effect path processes S32). Also drop the codec-setup bytes control from the ffmpeg_dec widget class so it is a plain 1-in/1-out effect; the decoder's STREAMINFO control is added at the instance level in a decoder topology instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CONFIG_FFMPEG_DEC_MP3: enables the libavcodec mp3 decoder and mpegaudio parser in the cross-build, the FFMPEG_DEC_CODEC_MP3 codec id mapping, and the float math layer (the mp3 decoder uses it). Verified building for ace30 (ptl) with decoders [flac,mp3]. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FFmpeg has no native MP3 encoder, so add MP3 encode through libshine, a small fixed-point encoder (good fit for a DSP - no float dependency). libshine is a west-pinned project; ffmpeg.cmake cross-builds it (its lib needs no config.h and its autotools CLI/shared link cannot work bare-metal, so the objects are compiled and archived directly), generates a shine.pc, and enables --enable-libshine --enable-encoder=libshine in FFmpeg. FFmpeg's require_pkg_config link test for -lshine pulls newlib malloc and hence Zephyr-runtime symbols (z_errno_wrap, ...) that only exist at module load; a configure-test-only stub is passed via --extra-ldflags so the test links (--extra-ldflags does not affect the static-archive build). Gated by CONFIG_FFMPEG_ENC_MP3. Verified building for ace30 (ptl): libshine.a cross-builds and libavcodec.a contains the libshine encoder. A module encode path (PCM->MP3) is a separate build mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CONFIG_FFMPEG_DEC_ENCODE_MODE: build the module as an encoder using the modern .process_raw_data path in reverse of the decoder - avcodec_send_frame/avcodec_receive_packet. ffmpeg_dec-encode.c opens the libshine MP3 encoder, converts SOF interleaved S32 to the encoder sample format per frame, and emits the compressed elementary stream. The interface ladder in ffmpeg_dec.c selects encoder / filter / decoder ops by Kconfig. libshine is linked into the module (after libavcodec, which references it) from its own cross-build install dir. ffmpeg.cmake now allows an encoder-only build (no decoder) and makes the decoder configure flags conditional. The encoder path pulls a few more libc symbols, added to the local surface: calloc/posix_memalign (alloc), modf/localtime_r/iconv*/__xpg_strerror_r (shims). fastmathf is now always built for the real backend (its sofm_log2f backs the shims' log10()). Verified building and load-ready (no unresolved externals, signed) for ace30 (ptl) as an encoder-only module. Runtime encode + real-time framing need hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add HIFI.md: analysis of the hot audio-processing paths in the module (decode/ filter/encode) and the linked FFmpeg DSP, and where Xtensa HiFi intrinsics would help. FFmpeg is built --disable-asm (scalar C on Xtensa) and has no _xtensa DSP init, so the generic C kernels run. Documents: the module's own PCM conversion loops (easy HiFi wins we own), FFmpeg's DSP dispatch contexts (float_dsp, flacdsp, mpegaudiodsp, tx) as fork-patch targets mirroring the other arch inits, fastmathf vectorisation, and a cost/benefit priority order. References SOF's existing HiFi kernels (src/math/*_hifi*) as the template. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the webrtc_vad SOF module wrapping libfvad — a standalone C port of the WebRTC Gaussian mixture model VAD (identical to the algorithm shipping in Chrome/WebRTC since 2011). Module characteristics: - Single-source, single-sink PCM pass-through - Accumulates 10/20/30 ms frames (configurable via Kconfig) - Broadcasts NOTIFIER_ID_VAD event on every frame classification - Supports S16_LE and S32_LE at 8/16/32/48 kHz, mono or stereo - Fixed-point; no FPU required - Two backends: real (libfvad) and pass-through stub for CI/LLEXT Build: - libfvad is cross-compiled by src/audio/webrtc_vad/webrtc_vad.cmake - Source is fetched via west (modules/audio/libfvad, pinned SHA 532ab666c20d) - LLEXT packaging supported via llext/ subdirectory Add NOTIFIER_ID_VAD to the notifier_id enum so downstream components (e.g. webrtc_ns2 / RNNoise) can share the same VAD event channel. UUID: c790b11d-5d14-e54e-be36ba4ad732cc14 Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the webrtc_ns SOF module wrapping the classic WebRTC Noise Suppression algorithm (spectral Wiener filter, fixed-point) from the webrtc-audio-processing 0.3.1 library. Module characteristics: - Single-source, single-sink PCM effect - Fixed-point implementation; no FPU required - Operates at 8 or 16 kHz on 10 ms frames (160/80 samples) - Supports S16_LE and S32_LE, mono or stereo (per-channel instances) - Four suppression levels: Mild/Medium/Aggressive/VeryAggressive (configurable at build time or via IPC4 set_configuration) - Two backends: real (WebRTC NS) and pass-through stub for CI/LLEXT - Designed as a build-time alternative to webrtc_ns2 (RNNoise): lower CPU, no 48 kHz constraint, fully fixed-point Build: - WebRTC NS subset is cross-compiled by src/audio/webrtc_ns/webrtc_ns.cmake (6 C files from modules/audio/webrtc-apm) - Source fetched via west (modules/audio/webrtc-apm, tag v0.3.1) - LLEXT packaging supported UUID: 0fc8faef-945f-004b-8d5f315047f1136a Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the webrtc_aec SOF module wrapping WebRTC AECm (Acoustic Echo Canceller Mobile) — the fixed-point Q15 echo canceller designed for mobile and embedded devices. Module characteristics: - Dual-source (2 input pins): pin 0 = microphone capture, pin 1 = playback echo reference - Single-sink: echo-cancelled microphone output - Fixed-point Q15; no FPU required - Operates at 8 or 16 kHz on 10 ms frames - Supports S16_LE and S32_LE, up to WEBRTC_AEC_CHANNELS_MAX channels (one AecmCore handle per channel) - Pipeline-ID heuristic (matching google_rtc_audio_processing) used to distinguish mic vs. reference at prepare() time - Two backends: real (AECm) and pass-through stub for CI/LLEXT Pin binding follows the same pattern as google-rtc-aec: both DAI capture copiers are connected as sources; the firmware resolves mic-vs-ref from pipeline membership. Reference topology: tools/topology/topology2/development/ cavs-nocodec-webrtc-aec.conf (dual SSP: SSP0=mic, SSP2=ref) UUID: e5da7b5b-133a-ba46-b517651d6300bb83 Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the webrtc_ns2 SOF module wrapping xiph/rnnoise — a recurrent neural network (GRU-based) noise suppressor originally developed by Jean-Marc Valin at Mozilla. Module characteristics: - Single-source, single-sink PCM effect - Floating-point; internal float math, hardware FPU beneficial - Hard requirement: pipeline sample rate MUST be 48000 Hz - Processes 480-sample (10 ms) frames; partial periods accumulated internally - Supports S16_LE and S32_LE, mono or stereo (one DenoiseState per channel; up to WEBRTC_NS2_CHANNELS_MAX channels) - Float scale bridging: pipeline samples normalised to ±1.0 are scaled to RNNoise's native ±32768 range and back - VAD dual-use: rnnoise_process_frame() returns per-frame speech probability [0.0, 1.0]; this fires NOTIFIER_ID_VAD events at a configurable threshold (WEBRTC_NS2_VAD_THRESHOLD_PCT) - Two backends: real (RNNoise) and pass-through stub for CI/LLEXT Design notes: - rnnoise_init() used at prepare/reset instead of rnnoise_create() to avoid per-init heap allocation (embedded-friendly path) - Model weights (rnn_data.c) are const float[] placed in .rodata (Flash/ROM); ~85-340 KB depending on model variant - Peak stack ~12-16 KB per rnnoise_process_frame() call - No expf/tanhf in hot path: replaced by 201-entry tansig LUT Build: - RNNoise cross-compiled by src/audio/webrtc_ns2/webrtc_ns2.cmake (6 C files: denoise.c rnn.c rnn_data.c pitch.c celt_lpc.c kiss_fft.c) - Source fetched via west (modules/audio/rnnoise, SHA 70f1d256) - LLEXT packaging supported UUID: eacfacdc-2a87-c942-97e894c917a740db Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Register all four WebRTC audio processing modules in the SOF build
system:
Kconfig (src/audio/Kconfig):
- rsource the four module Kconfig files (alphabetical order)
CMakeLists (src/audio/CMakeLists.txt):
- add_subdirectory() guards for COMP_WEBRTC_{VAD,NS,AEC,NS2}
UUID registry (uuid-registry.txt):
- c790b11d-5d14-e54e-be36ba4ad732cc14 webrtc_vad
- 0fc8faef-945f-004b-8d5f315047f1136a webrtc_ns
- e5da7b5b-133a-ba46-b517651d6300bb83 webrtc_aec
- eacfacdc-2a87-c942-97e894c917a740db webrtc_ns2
west.yml:
- Add remotes: libfvad (dpirch), webrtc-apm (freedesktop.org),
rnnoise (xiph)
- Add project entries with pinned revisions:
libfvad @ 532ab666c20d → modules/audio/libfvad
webrtc-apm v0.3.1 → modules/audio/webrtc-apm
rnnoise @ 70f1d256 → modules/audio/rnnoise
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add Topology 2.0 component widgets, capture pipeline templates, and
standalone development/test topologies for all four WebRTC modules.
Component widgets (include/components/):
webrtc-vad.conf — pass-through VAD; S16/S32; any rate; 1→1
webrtc-ns.conf — fixed-point spectral NS; 8/16 kHz; S16/S32; 1→1
webrtc-aec.conf — AECm echo canceller; 10ms IBS/OBS; 2 input pins
(pin 0 = mic, pin 1 = echo ref); S16/S32; 2→1
webrtc-ns2.conf — RNNoise GRU NS; 48 kHz only; S16/S32; 1→1
Capture pipeline templates (include/pipelines/cavs/):
webrtc-ns-capture.conf — DAI-copier → VAD → NS → module-copier
webrtc-aec-capture.conf — copier → AEC(2-pin) → copier
webrtc-ns2-capture.conf — RNNoise → module-copier (48 kHz locked)
Development/test topologies (development/):
cavs-nocodec-webrtc-ns.conf — single SSP loopback; VAD+NS
cavs-nocodec-webrtc-aec.conf — dual SSP (SSP0=mic, SSP2=ref); AEC
mirrors cavs-nocodec-rtcaec.conf
cavs-nocodec-webrtc-ns2.conf — single 48 kHz SSP; RNNoise
All three development topologies registered in tplg-targets.cmake
targeting TGL nocodec with NHLT preprocessing.
Compile example:
alsatplg -c development/cavs-nocodec-webrtc-aec.conf -D PLATFORM=tgl -o sof-tgl-nocodec-webrtc-aec.tplg
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the root Kconfig that exposes the ffmpeg_dec module configuration tree (codec selection, filter mode, cold-split experimental option) and the standalone ffmpeg.cmake cross-build helper for out-of-tree builds. These files were omitted from the earlier ffmpeg_dec commits and are grouped here to keep all ffmpeg_dec build system pieces together. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…and webrtc modules Create and update README.md documentation for the FFmpeg decoder/encoder/filter module and all four WebRTC audio modules: - ffmpeg_dec/README.md: Added Mermaid architecture diagram, detailed decoder/encoder/filter configurations, build choices, and topology guides. - webrtc_vad/README.md: Included Mermaid data flow, GMM classification details, ring-buffer accumulation behavior, and notification topology mappings. - webrtc_ns/README.md: Documented fixed-point spectral Wiener filter complexity, channel separation, and Kconfig rules. - webrtc_aec/README.md: Described dual-input AECm echo cancellation routing, pipeline-ID heuristics, and layout designs. - webrtc_ns2/README.md: Documented deep-learning/RNNoise GRU topology, 48 kHz lock rules, VAD probability thresholds, and DSP scaling metrics. Each document contains a tailored Mermaid diagram mapping data/control flows. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Document the per-frame AAC-LC decode hot spots for Xtensa HiFi work: IMDCT (libavutil/tx), the AVFloatDSPContext vector kernels (vector_fmul_window and friends), x^(4/3) inverse quant, and TNS, plus SBR/PS for HE-AAC. Includes a toolchain-requirement note: the HiFi intrinsic headers (xt_hifi*.h) and an ace30 core-isa.h with XCHAL_HAVE_HIFI4=1 are only available via the LLVM/xt-clang Xtensa toolchain, not the Zephyr-SDK GCC currently used by ffmpeg.cmake, so the ff_*_init_xtensa kernels need an intrinsic path plus a scalar C fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…loat_dsp
Activate the Xtensa HiFi AVFloatDSPContext kernels (lgirdwood/FFmpeg sof-hifi,
ff_float_dsp_init_xtensa) in the ffmpeg_dec module build:
- ffmpeg.cmake: stop passing --disable-asm to FFmpeg configure. FFmpeg treats
every per-arch optimisation dir -- including our C-intrinsic libavutil/xtensa/
-- as "asm"; --disable-asm forces arch=c and drops them (ARCH_XTENSA=0). There
is no Xtensa assembly, so leaving asm enabled is safe and is what makes
ARCH_XTENSA=1 and builds ff_float_dsp_init_xtensa.
- west.yml: bump ffmpeg to sof-hifi @ a600a78acd (adds libavutil/xtensa HiFi
float_dsp on top of the av_tx table cap).
Verified: pristine ptl (intel_adsp/ace30/ptl) AAC build (FLAC+AAC) reconfigures
FFmpeg with ARCH_XTENSA=1, compiles libavutil/xtensa/float_dsp_init.o, links it
into ffmpeg_dec.llext with no errors or dangerous relocations. Under the
Zephyr-SDK GCC the kernels run as the scalar C fallback (GCC lacks xt_hifi4.h);
the xtfloatx2 SIMD path activates with an xt-clang/XCC build for the ace30 core.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pick up the __XCC__ gate on the Xtensa float_dsp SIMD path so the ffmpeg_dec module builds cleanly with the LLVM Xtensa clang (was: intrinsic branch entered via the xt_hifi4.h wrapper and failed on undeclared xtfloatx2). GCC behaviour is unchanged (scalar fallback either way). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a clang branch to the FFmpeg cross-build so the module can be built with the
LLVM Xtensa clang (needed for any HiFi intrinsics -- the Zephyr-SDK GCC ships no
xt_hifi/ae_* intrinsics header at all). clang is a generic driver, so unlike the
target-specific GCC it needs the target/core/sysroot spelled out, uses the
Zephyr-SDK GNU binutils (ar/as/nm/objcopy), and cannot link the bare-metal
configure *test* executables (no default crt), so those link via the GNU gcc
(--ld). All values are derived from CMAKE_AR (GNU binutils even under clang):
cross-prefix, sysroot, and -mcpu=<core> from the triple. Also disables auto/SLP
vectorisation: the LLVM Xtensa backend cannot lower the v2i32 bswap FFmpeg
byteswap code gets SLP-vectorised into ("Cannot select: v2i32 = bswap"), and
it buys nothing (no packed float SIMD codegen on Xtensa anyway).
The GCC path is unchanged (guarded on CMAKE_C_COMPILER_ID) and re-verified green:
pristine ptl AAC build -> ARCH_XTENSA=1, float_dsp_init.o compiled, ffmpeg_dec.llext
linked. The clang recipe is validated at the FFmpeg-configure/build level: a
standalone decoder-only cross-build (avcodec+avutil+swresample, FLAC) with
~/work/llvm-project clang links clean and produces elf32-xtensa-le objects. A full
firmware clang build additionally requires the Zephyr LLVM toolchain variant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Setting the bytes control max inside the host-gateway-tdfb-drc-capture pipeline class had no effect: alsatplg leaves the widget control at the default 1024, so pushing a larger TDFB blob with sof-ctl failed. Move the max to the widget instantiation in sdw-dmic-generic.conf where it is honored, and keep the original 16384 headroom so the setting also covers larger beamformer blobs that may be added later. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Hook a tdfb blob validator into the model handler so a corrupted or mismatching run-time configuration update is rejected before it can replace the working blob. Capture then continues with the previously set filters instead of being interrupted by bad re-configuration. The TDFB blob is variable size and the per-filter walk in tdfb_init_coef() was not bounded against the IPC payload, so a bad length field could push tdfb_filter_seek() past the buffer. The new validator walks every FIR section and the trailing arrays with byte-bounded steps, requires the layout to exactly match config->size, and rejects blobs whose channels or filter_index values do not fit the running stream. The stream channel counts are cached in tdfb_comp_data so the validator can consult them, and the validator is re-run at prepare time once those counts are known. A malformed initial blob is discarded so the runtime path cannot dereference it. With ingress fully validated, the redundant sanity checks inside tdfb_init_coef() are dropped. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
This patch hooks a drc blob validator into the model handler so a corrupted run-time configuration update is rejected before it can replace the working blob. Playback or capture then continues with the previously set parameters instead of being interrupted by a bad IPC. The DRC configuration is a fixed-size struct sof_drc_config, so the validator requires the IPC payload size to match exactly and the self-declared config->size to agree with it. It is installed in drc_init() when the model handler is created, so every blob swap is covered - including any received while the component is in READY - and the redundant size check that drc_prepare() used to run on the initial blob is dropped. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add native_sim board configuration and support in the build script. This allows building and running tests on the host using Zephyr's native_sim target. native_sim leverages the POSIX architecture, but the libfuzzer support specifically requires CONFIG_ARCH_POSIX_LIBFUZZER to be set. Therefore, this wraps fuzzer-specific code in ipc.c and the build of fuzz.c behind this config to allow clean compilation on the standard native_sim board. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add native_sim board target to the sof-qemu-run scripts, and add an option to additionally run it under valgrind. The default build directory is set to ../build-native_sim Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
When building the firmware for native_sim, debugging allocations with host machine tools like Valgrind is constrained due to Zephyr's internal minimal libc tracking the heap manually via static pools. By bypassing Zephyr's memory interception on native_sim using nsi_host_malloc, dynamically tracked memory can surface appropriately to Valgrind memory checkers without causing a libc heap pool panic. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Keep spinning in case user needs to inspect status via monitor. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
When building the native_sim fuzzer, the host allocator does not possess the strict bounds of the internal Zephyr memory pools. If the fuzzer generates a malformed payload requesting an excessively large size (e.g. 4GB), it passes directly to the host ASAN allocator which aborts due to OOM or protection limits. Adding a 16MB cap allows these to fail gracefully. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Only run sof_run_boot_tests() at boot for QEMU or native_sim (STANDALONE) targets. Other platforms trigger via IPC.
The feed-forward and feedback paths copy frames * channels samples into fixed intermediate buffers but only checked the frame count. Bound the total sample count against the buffer capacity so an unexpected channel count cannot overflow the buffers. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
The remap routines used host-provided channel map entries to index the source frame, guarding only against the -1 unmapped sentinel. Skip any entry outside [0, source channels) so a crafted map cannot read past the source frame. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
The fragmented calibration get path advanced a read offset by a host-controlled index without checking it against the calibration data size, allowing reads past the buffer. Reject offsets at or beyond the data size and fragments that would extend past it. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
When compiling loadable modules (LLEXT) with the LLVM toolchain, we must specify the LLD linker (-fuse-ld=lld). Explicit references to bfd (-fuse-ld=bfd) were removed or guarded behind non-Clang checks so that LLVM builds succeed when generating relocatable/shared objects.
Enable CONFIG_SOF_OS_LINUX_COMPAT_PRIORITY=y in the Panther Lake board configuration. This prevents the firmware from querying UAOL (USB Audio Offload) capabilities on hosts where the kernel has not enabled/supported UAOL. Without this priority set, the capability check accesses unpowered/disabled UAOL registers, triggering a hardware bus error and subsequent DSP panic during the first Host-to-DSP Large Config Get request. Also disable CONFIG_ADSP_IMR_CONTEXT_SAVE to prevent low-power context save failures during runtime suspend.
Enable CONFIG_XTENSA_ADSP_FATAL_BREADCRUMB in the debug overlay so debug builds record the faulting PC/cause/vaddr into HP-SRAM window0, making a crash visible in the host dmesg "Firmware state" line when no console or mtrace output is available. The option defaults to n, so production builds (which do not pull in this overlay) are unaffected. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
This patch fixes a critical firmware load failure (-22 / IPC4_ERROR_INVALID_PARAM) on PTL DSPs when loading LLEXT modules compiled with Clang. The DSP ROM loader requires all memory-resident sections (SHF_ALLOC) to have a valid physical load address (sh_addr > 0). The previous script explicitly ignored the empty .text stub emitted by Clang's linker and also missed some auxiliary sections that had SHF_ALLOC but no other explicit types, leaving their sh_addr as 0. Fixes: - Explicitly seed the addr_map with .text so the empty section receives the valid text_addr offset instead of 0. - Ensure any remaining SHF_ALLOC sections fall back into the readonly lists, mapping them properly and satisfying the ROM loader's memory checks. - Also removes the racy llext_offset_calc.py size file updates.
The Zephyr LLVM toolchain passes -D__XCC__ alongside -D__XCC_CLANG__ to clang for ABI compatibility. format.h only tested __XCC__, so it selected format_hifi3.h (HiFi3 intrinsic sat_int32 using AE_SLAI64S / AE_SRAI64) even when building with clang. LLVM encodes HiFi3 instructions in 8-byte FLIX bundles. On TGL (cavs2.5) which has FeatureHIFI3 but no FLIX execution unit, these bundles are illegal instructions (EXCCAUSE=0) at runtime. Guard the HiFi3 selection with !defined __XCC_CLANG__ so that clang always takes the generic C path (format_generic.h), which compiles to safe scalar code on any Xtensa target. XCC-only builds are unaffected. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pantherlake (ACE30/PTL) bring-up requires several temporary workarounds and fixes until LLEXT ET_REL relocatable module loading is fully supported: * Disable CONFIG_UAOL to prevent early boot crash. The ADSP power management framework resumes all registered devices during boot. On PTL, the USB/UAOL host link is not yet initialised by the host at that point and the UAOL register access (0xd40 range) triggers a PIF bus error (EXCCAUSE_LOAD_STORE_DATA_ERROR), crashing the DSP firmware. * Force core audio components (volume, asrc, crossover, fir, iir, drc, multiband-drc, tdfb, sel, tone, mux, mixin-mixout) to build statically (=y) rather than as loadable modules (=m). The ACE30 platform uses ET_REL relocatable ELF format for LLEXT modules, and the openmodules split-release loader does not yet correctly handle ET_REL relocation at runtime. Building them static is the safe interim approach. * Set CONFIG_LIBRARY_DEFAULT_MODULAR=n to prevent other tristate options from defaulting to =m during this bring-up phase. * Enable CONFIG_ADSP_IMR_CONTEXT_SAVE and drop CONFIG_SOF_OS_LINUX_COMPAT_PRIORITY which is no longer applicable. TODO: remove static overrides once the ET_REL LLEXT loader is fixed. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
ACE30 (Panther Lake) uses relocatable ELF (ET_REL) format for LLEXT modules, where sections have sh_addr=0 at link time because no base address is assigned until runtime allocation. The existing code sets .pre_located = true, which tells llext_load() to treat all section addresses as already mapped in place. For ET_DYN shared libraries (ACE15/ARL and older) this is correct because the module is pre-loaded into IMR at its final VMA. For ET_REL modules the sections have no VMA, so pre_located=true causes the loader to skip relocation and leaves function pointers unresolved, causing faults when the module is first called. Set .pre_located = false when building for CONFIG_SOC_ACE30 so that llext_load() allocates memory for each section and applies relocations before the module is used. All other platforms continue to use the pre-located path. TODO: convert this #ifdef to a runtime capability check once the LLEXT loader supports a clean way to query the ELF type. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…T_SAVE When ADSP IMR context save is disabled (CONFIG_ADSP_IMR_CONTEXT_SAVE=n), the HP-SRAM and L3 heap memory are lost during PM runtime suspend/resume. However, the LLEXT library manager state `lib_manager_dram` resides in the persistent IMR data section (`__imrdata`) and retains stale pointers (such as `lib_manager_dram.ctx` pointing to the now-invalid L3 heap address from the first boot). On resume, `llext_manager_restore_from_dram()` would try to restore libraries from these stale pointers, causing register window underflows and fatal exceptions (EXCCAUSE_ILLEGAL) when executing scheduler work queues on subsequent IPC handling. Guard `llext_manager_store_to_dram()` and `llext_manager_restore_from_dram()` with `IS_ENABLED(CONFIG_ADSP_IMR_CONTEXT_SAVE)`. If context save is disabled, the library manager cleanly re-initializes on resume boot, forcing libraries to reload correctly. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…ngth is 0 Ensure extra_cfg_len subtraction does not underflow when copier_cfg->gtw_cfg.config_length is 0, and guard tlv_buff_size calculation against underflow when ipc_config_size is smaller than basic_size.
…put ELF When linking multiple input objects into an LLEXT module, LLD merges sections sharing identical names (such as .literal.foo). llext_link_helper previously calculated section base addresses (addr_map) using section sizes read from the unlinked input object rather than the linked output ELF (.tmp). This caused section start addresses to underestimate the true size of merged sections, resulting in overlapping section errors in rimage. Fix: read section headers from the linked output ELF (args.file.tmp) after running the linker, so section base addresses are computed using actual merged section sizes. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Export ZEPHYR_BASE in platf_build_environ so that post-build west commands (such as west sign on PTL/ACE30) can discover Zephyr's west extension commands when invoked from arbitrary working directories during CMake build. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…lter mode
The filter-mode process path converts interleaved S32 PCM to planar float on
the way into the FFmpeg graph and back on the way out. On HiFi4/HiFi5 VFPU
cores our LLVM Xtensa clang now provides packed conversion intrinsics, so do
the conversions two lanes at a time:
- S32 -> float: float.sx2(x, 31) == (float)x * 2^-31 (fuses the /2^31)
- float -> S32: trunc.sx2(f, 31) == (int32_t)(f * 2^31) with int32
saturation (fuses the *2^31 and the manual clamp)
Packed values are moved through the AE register file with AE_L32X2/AE_S32X2
(the backend has no generic 64-bit-vector memory op); the strided interleaved
side gathers/scatters via an 8-byte aligned temp while the contiguous planar
side loads/stores packed directly. The scalar C loop is kept as a fallback for
GCC and non-VFPU cores, guarded by FFMPEG_AF_VFPU_CONV.
Emission verified with the built clang for intel_ace30_ptl: the vectorized
bodies contain exactly one float.sx2 / trunc.sx2 with AE loads/stores, scalar
mul.s only in the <=1-element remainder tails.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d path When the SOF build uses the LLVM Xtensa clang, ffmpeg.cmake configures the FFmpeg ExternalProject with clang driven by --target/-mcpu/--sysroot only. clang's --sysroot does not add GCC's internal fixed-include directory (lib/gcc/<triple>/<ver>/include) -- the GNU target driver adds it implicitly, clang does not. That directory is where <xtensa/config/core-isa.h> lives. Without it, __has_include(<xtensa/config/core-isa.h>) fails in libavutil/xtensa/float_dsp_init.c, so XCHAL_HAVE_HIFI*_VFPU stay undefined, the VFPU gate is not taken, and the file silently compiles to scalar C -- i.e. the HiFi packed-float SIMD is dropped from the ace40/ace30 build with no diagnostic. Ask the matching GNU gcc for that directory (-print-file-name=include) and add it to the FFmpeg compile flags, guarded on the xtensa/config subdir actually being present so it is a no-op for other toolchains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…odec Link libavcodec/libswresample/libavutil statically into the base image for the =y (non-LLEXT) build instead of the =m loadable module. The =m LLEXT path is blocked on LLVM by the ET_REL relocation issue; LLD resolves everything at link time for =y, sidestepping runtime relocation entirely. CMakeLists.txt (=y else branch): cross-build the FFmpeg archive via ffmpeg.cmake, expose its headers, and link avcodec (before its deps) into the SOF interface library, ordered after ffmpeg_ext. No -lm/-lc: the base image is bare-metal and libav* libc refs resolve from Zephyrs minimal libc. A new
Flesh out the decoder module: pull compressed input from the source ring into a
padded linear bounce buffer, strip the FLAC container header once so the raw
parser sees a bare frame elementary stream, drive libavcodec send-packet /
receive-frame, and stage decoded PCM into a linear buffer drained into the sink
across as many DP cycles as the sink free space needs. STREAMINFO arrives as a
bytes-control blob via set_configuration and is stored as avctx->extradata.
Two fixes make it actually decode and drain on the target:
* ffmpeg_dec-ffmpeg.c: on this cut-down static libavcodec build
avcodec_alloc_context3() does not apply the AVOption defaults, so
avctx->max_samples is left zero and ff_get_buffer() rejects every audio
frame ("samples per frame N exceeds max_samples 0", -EINVAL). Restore the
upstream INT_MAX default before avcodec_open2(). Also route av_log() into
the Zephyr log with identical-message rate limiting so a stalled stream
cannot flood the 64 KiB mtrace ring.
* ffmpeg_dec.c: implement compress drain. On a DRAIN trigger the kernel sets
pipeline->expect_eos and blocks in compress_drain() until the module reports
completion. Pre-build the COMPR-magic module notification (the same template
the Cadence decoder uses, which sof_ipc4_compr_drain_done matches on), and
once the last PCM is flushed - input drained or the decoder yields no more
output under expect_eos - send it once and mark the sink EOS so the state
propagates to the DAI. Without this cplay's compress_drain() timed out.
Verified end-to-end on Panther Lake (ace30, LX7+HiFi4): a 136 KB FLAC clip
decodes to PCM and cplay returns cleanly ("Close Normally") instead of the
prior 25 s drain timeout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bled get_codec_info() reports the DSP's compress codec capabilities to the driver. Add SND_AUDIOCODEC_FLAC (playback) to the list when CONFIG_FFMPEG_DEC_FLAC is set, so the compress PCM enumerates FLAC as a supported decode format now that the ffmpeg_dec (libavcodec) module provides it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the ffmpeg_dec widget into a compress pipeline. The stock compress path
instantiates the Cadence decoder; ffmpeg_dec needs its own pipeline + platform
instance, gated on a new COMPR_FFMPEG define so the stock compr.conf (COMPRESSED)
is unaffected. They share the COMPR_* pcm/pipeline ids, so enable one or the
other.
* new compr-playback-ffmpeg.conf: clone of compr-playback with the decoder
widget replaced by ffmpeg_dec.
* new platform/intel/compr-ffmpeg.conf: the platform instance, with the FLAC
STREAMINFO bytes control attached at instance level.
* new components/ffmpeg_dec/flac_streaminfo.conf: the STREAMINFO setup blob.
* ffmpeg_dec.conf: change the widget DAPM type "effect" -> "decoder". The
compress kernel path (sof_ipc4_compr_get_module) only finds the codec module
when its DAPM id is decoder/encoder, so a plain effect widget is never
configured on a compress pipeline.
* cavs-mixin-mixout-hda.conf / sof-hda-generic.conf / compr-default.conf:
include the new pipeline and the COMPR_FFMPEG IncludeByKey.
* tplg-targets.cmake: add sof-hda-generic-ffmpeg-compr build target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…alls Replace the return-0 double-libm stubs in the =y build with real newlib math, so lossy decoders (AAC/Opus/Vorbis) get correct math instead of garbage. FLAC is integer-only and never calls libm at runtime, but the references now resolve to real kernels. The Zephyr SDK's pre-built newlib math cannot be linked as-is: the image is compiled -mlongcalls and linked --no-relax, but the SDK objects use direct call8 (there is no -mlongcalls multilib), so an out-of-range inter-object call (e.g. -> memset) fails at link time with "dangerous relocation: call8: call target out of range". So compile newlib's math from source with -mlongcalls -mtext-section-literals (the compiler then emits callx8, unlimited range), which links cleanly under --no-relax. The newlib libm math/ + common/ sources are vendored under libm/ (the exact member set the SDK's libc.a selects, so there are no duplicate definitions; see libm/README for provenance and regeneration). ld/, complex/ and fenv/ are omitted: FFmpeg audio never calls long-double or complex math, and the math/+common/ symbol closure is self-contained (external tail = compiler soft-float builtins + memset, already in the image, + __errno). ffmpeg.cmake compiles libm/newlib_math_sources.cmake into libnewlib_m.a (new newlib_m target); CMakeLists.txt links it last as an archive so only the math objects a decoder actually references are pulled in. ffmpeg_dec-builtin-libc.c drops the fake libm stubs and bridges __errno to Zephyr's per-thread errno (z_impl_z_errno) for the math error paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ace30/40 core-isa reports XCHAL_HAVE_FP=1 (a single-precision scalar FP0 unit), so LLVM lowers plain-C float math in libav* to hardware FP0 ops (mul.s, mov.s, ...). But SOF firmware is built CONFIG_FPU=n: the scalar FP coprocessor is never enabled for any thread (SOF's own float code uses the separate always-available HiFi vector VFPU). An FP0 instruction therefore faults at runtime (EXCCAUSE 0, illegal instruction) the first time a codec runs scalar float math -- e.g. ff_mpadsp_init building the MP3 synthesis window. Integer codecs (FLAC) never emit one, which is why they worked. Disable the LLVM "fp" target feature for FFmpeg's own --cc so it emits soft-float libcalls (__mulsf3, __truncdfsf2, ...) instead. This is scoped to _ff_cc_ff and must NOT reach the newlib libm build (_ff_cc): libm's <fenv.h> selects its FP variant on XCHAL_HAVE_FP and its rur.fcr/rur.fsr inline asm needs the "fp" feature to name the FPU control regs. libm double math is soft-float regardless, and its unreferenced single-precision FP0 variants get GC'd, so libm stays safe on the plain _ff_cc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SOF runs module init and prepare (including the backend configure step) on the EDF work-queue thread. The ffmpeg_dec lossy decoders (MP3/AAC/Opus) build large cos/pow tables in avcodec_open2(), whose call chain is ~13 KiB deep and overflows the 8 KiB CONFIG_STACK_SIZE_EDF default -- spilling past the guard into the adjacent thread's stack and faulting on its next context switch (EXCCAUSE 9, load/store alignment in xtensa_restore_high_regs). This is not the DP thread's stack, so bumping the topology stack_bytes_requirement has no effect. FFMPEG_DEC_FLOAT_MATH is auto-selected by exactly those decoders, so default the EDF stack to 32 KiB when it is set. The integer FLAC decoder does not need it and keeps the 8 KiB default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the runtime pieces to decode an MP3 elementary stream end-to-end: - Pad the decode packet to AV_INPUT_BUFFER_PADDING_SIZE. FFmpeg's bitstream reader over-reads its word cache past the packet payload; the mpegaudio parser can hand back a packet pointing into the unpadded host input, so copy the payload into the padded scratch and zero the tail before decode. - Skip a leading ID3v2 tag in ffmpeg_dec_skip_container(). Real-world MP3 files prepend an ID3v2 tag that the mpegaudio parser would mis-sync on; skip the syncsafe-encoded tag length so the parser sees frames. The FLAC "fLaC" container skip is generalised alongside it. - Advertise SND_AUDIOCODEC_MP3 to the kernel from get_codec_info() when CONFIG_FFMPEG_DEC_MP3 is built (guarded so it is not advertised twice when the Cadence MP3 decoder is also present). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Parametrize the ffmpeg compress topology on COMPR_FFMPEG_CODEC (default flac). FLAC carries a STREAMINFO codec-setup bytes control (applied as libavcodec extradata before avcodec_open2); MP3 is a self-describing elementary stream and needs no extradata, so the control is wrapped in an IncludeByKey and omitted for mp3. The default resolves to flac, so the existing sof-hda-generic-ffmpeg-compr target is byte-for-byte unchanged. Add a sof-hda-generic-ffmpeg-mp3-compr target that overrides COMPR_FFMPEG_CODEC=mp3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The HiFi FIR cores issue 64-bit AE_L32X2/AE_S32X2 circular accesses into the delay line. buffer_start is offset from the delay_lines base by a multiple of 8 bytes, so the base itself must be 8-byte aligned or those paired accesses fault (EXCCAUSE 9). mod_alloc() with alignment 0 may return only a 4-byte-aligned pointer; request the 8-byte alignment explicitly with mod_alloc_align() instead of relying on the allocator default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ffmpeg_dec_emit_frame() copied the decoder's native PCM straight into the sink buffer, assuming the decoder sample format matched the sink. That is false: the SOF IPC4 pipeline propagates the PCM (sink) bit depth back onto this widget's output pin, so the fixed-point mp3 decoder (S16) and 16-bit FLAC (S16) feed an S32 sink. The old memcpy packed two 16-bit samples into each 32-bit slot and sized the frame from the source bytes-per-sample, so the sink received half the bytes it expected filled with mis-strided data -- audible as constant harmonic distortion and half-rate playback. Convert every source sample through a signed Q1.31 intermediate and re-quantise to the negotiated sink frame format (S16/S24_4LE/S32/FLOAT), de-planarising planar layouts in the same pass. Integer source formats cover the fixed-point decoders built today (mp3, FLAC); the float paths cover the planar-float decoders (AAC, Opus) added later. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ffmpeg_dec data-processing module decodes a whole codec frame (up to ~24 ms of audio for MP3) per process() call, but never advertised an output block size. The DP bind path sizes the ring buffer between this module and the downstream LL chain to 3 x mpd.out_buff_size (ipc4 helper.c), so with out_buff_size left at 0 the ring collapsed to ~3 LL periods - far smaller than one decoded frame. As a result process() could only drain a fraction of each decoded frame into the sink per LL period, then dribbled the remainder out at the downstream real-time rate before decoding the next frame. Decode (~17 ms/frame) and the real-time drain (~24 ms/frame) ran serially instead of overlapping, so a 48 kHz MP3 played at ~0.62x real time (measured 16.2 s of wall for 10 s of audio, gap 21.7 ms/frame). Advertise mpd.out_buff_size in init, sized to hold ~2 decoded frames (worst case stereo S32, capped so the 3x ring stays bounded, floored at one frame for large blocks). The ring now holds several frames, letting decode run ahead and overlap the real-time drain. Measured wall for the same clip dropped to 9.9 s - real time - decode cost unchanged (gap 6.6 ms/frame). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
struct idc_payload lives in coherent (SHARED_DATA) memory and is shared across cores, but it is only a byte array with no explicit alignment, so the linker was free to place it (and every per-core slot) on an odd address. Handlers that read the payload as a wider type -- e.g. the cross-core get_attribute path, which reads a uint32 type field -- then take an EXCCAUSE 9 (load/store alignment) fault when a component runs on a secondary core (ptl/ace30). Align the payload to the cache line. That is the granularity the payload is already built around (IDC_MAX_PAYLOAD_SIZE == DCACHE_LINE_SIZE * 2) and it guarantees every per-core slot starts word-aligned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The remote (cross-core) get_attribute handler passed the caller coherent (uncached) IDC buffer straight to comp_get_attribute(), which memcpy()s a whole ipc4_base_module_cfg into it. On Xtensa the compiler lowers that struct copy to wide accesses (AE_L32X2 / 64-bit) that the uncached window does not support, faulting EXCCAUSE 9 -- so binding a component that lives on a secondary core (e.g. ffmpeg_dec offloaded to core 1 on ptl/ace30) crashed. Have the handler fill a local cacheable copy and publish it to the coherent buffer with aligned 32-bit word stores (ipc4_coherent_copy32); do the same for the read-back on the initiator side. Only COMP_ATTR_BASE_CONFIG is supported remotely, so the fixed-size local copy is always correct. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A remote component prepare() can be expensive: a DP decoder one-time codec open builds large trig tables -- measured ~580 ms for mp3 on a secondary core. Waiting for it stalls the initiator core IPC/EDF thread for that whole time, and with the default CONFIG_IDC_TIMEOUT_US the blocking k_p4wq_wait() instead times out mid-prepare (-EAGAIN) and races pipeline teardown into a crash. Send the prepare IDC non-blocking. Ordering is preserved without the wait: each core drains its IDC work through a single in-order p4wq worker, so the prepare completes before the cross-core trigger (START) queued behind it. For a compress stream the START IPC also arrives only after the app first write, giving the remote prepare time to finish off the critical path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ffmpeg_dec_prepare() opened the backend decoder (avcodec_open2), but prepare() is dispatched to the component core over IDC and runs on that core single IDC worker thread. avcodec_open2 builds large trig tables and can take hundreds of ms (~580 ms for mp3), monopolising the IDC worker so a cross-core prepare/trigger on the initiator core stalls or times out. Defer the open to the first process() call, which runs on the module own DP thread (deep 64 KiB stack, off the IDC critical path). The open is gated on cd->configured; reset() flushes but keeps the decoder open, so it runs only once per fresh prepare(). The first process() returns early after opening and decodes on the next cycle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the ffmpeg_dec widget to core 1 to free core-0 headroom, which also runs the DRC/IIR/SRC of the mix path; sharing a core with the decoder left too little margin and caused occasional steady-state glitches. The cross-core placement exercises the IDC get_attribute and prepare/trigger paths, which are fixed in firmware (idc_payload alignment, ipc4_coherent_copy32, non-blocking comp_prepare_remote, deferred avcodec_open2). Requires that firmware. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP build support for llext based and runtime linked ffmpeg audio encoding, decoding and filters as a west module. Builds and links today. Next steps are integration with topology and kernel then optimization.