heap: try VMH when system heap is exhausted

[lyakh]

This is a temporary measure before removing VMH and increasing the system heap for Linux configurations. Currently the system heap allocations can fail in cases when multiple (tyipcally more than 5) streams are started. These are rather extreme testing scenarios, unlikely to occur in real use-cases, but to give them a chance to succeed try to fall back to VMH in such cases.

[Copilot]

🟡 Changes recommended

The new warning/fallback uses an incorrect format specifier for size_t and can misleadingly log “trying VMH” even when VMH is not enabled/initialized.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds a fallback path for SOF/Zephyr heap allocations: when the system heap (sof_heap) cannot satisfy an allocation request, it attempts to retry using the Virtual Memory Heap (VMH) to reduce allocation failures in extreme multi-stream scenarios on Linux configurations.

Changes:

• Add a VMH fallback attempt in rmalloc_align() when allocating from the system heap fails.
• Add a similar VMH fallback attempt in z_impl_sof_heap_alloc() for sof_heap allocation failures.

File summaries

File	Description
zephyr/lib/alloc.c	Adds “system heap → VMH” fallback behavior for aligned allocations in both the generic allocator and the Zephyr syscall implementation.

Review details

Suppressed comments (1)

zephyr/lib/alloc.c:799

• Same as above: %u is the wrong format for bytes (size_t), and this should only log/attempt a VMH fallback when VMH is enabled and initialized; otherwise it logs "trying VMH" but just retries the system heap path.

	if (!ptr && heap == &sof_heap) {
		tr_warn(&zephyr_tr, "system heap allocation of %u failed, trying VMH", bytes);
		ptr = rballoc_align(flags, bytes, alignment);
	}

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[intel-sofci]

PR 11185: test results

Run date: 2026-09-10 12:47 UTC

Tested commit: 700906b7899a5f08cd0da28b9a046978763ba4b0

[wjablon1]

I already attempted to solve the system heap running out-of-memory from another angle (not sure if the root cause is the same #11182).
Your solution seems to be more generic and might be sufficient as a temporary measure. However, the virtual heap is tricky (e.g. you could easily exhaust small block pools and be forced to use big ones for small allocations). So I am concerned about further errors.

[lyakh]

I already attempted to solve the system heap running out-of-memory from another angle (not sure if the root cause is the same #11182). Your solution seems to be more generic and might be sufficient as a temporary measure. However, the virtual heap is tricky (e.g. you could easily exhaust small block pools and be forced to use big ones for small allocations). So I am concerned about further errors.

@wjablon1 I'm not sure if this is the same problem. But yes, this should be more generic since it fixes OOM for the allocator, not only for buffers. Could you check if this fixes your problem too?

[kv2019i]

This is less than ideal and makes already confusing code more confusing, but I can't argue with the rationale. It makes no sense to fail in cases where we have heap memory available (due to the fixed partitioning we now have).

[kv2019i]

It's not very obvious how the free is matched, but it seems this is ok as rfree() does a "is_virtual_heap_pointer(ptr)". If you need to respin, maybe worth a comment (not a showstopper, rfree() is in the same file, so people can look it up as well).

[lyakh]

rballoc()s counterpart is rfree() - that's universal, and of course not a single bit confusing ;-)

[wjablon1]

@lyakh OK I tested and it works for my case. Since my own solution isn't ready, we can go with yours