[Bug]: Linux guest intermittently hangs during boot: expedited RCU grace period never completes because target vCPU is idle with interrupts enabled (lost IPI)

[highoctane305]

Version

7.2.14

Host OS Type

Linux

Host OS name + version

Linux Mint 22.x Cinnamon

Host Architecture

x86

Guest OS Type

Linux

Guest Architecture

x86

Guest OS name + version

Linux Mint 22.x MATE

Component

Other

What happened?

On a fleet of 130+ identically configured lab machines, Linux guests intermittently
hang during boot. The guest console reports:

rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-...D } 60580 jiffies s: 93 root: 0x8/.
rcu: blocking rcu_node structures (internal RCU debug):

The jiffies count climbs indefinitely and the guest never recovers. Every task
that subsequently forks blocks behind the stalled grace period, so boot stops.

The significant finding is that the vCPU that RCU reports as blocking is not
stuck. Reading guest CPU state from the host shows it halted in the idle loop
with interrupts enabled, indistinguishable from the other vCPUs. It is idle and
fully interruptible; it has simply never received the expedited grace period IPI.

This appears to be a lost or undelivered inter-processor interrupt in VirtualBox's
software APIC emulation, not a guest kernel deadlock.

Booting the guest with nox2apic rcupdate.rcu_normal=1 eliminates the hang
completely. This was verified across three labs of ~37 machines each under
production load, against control labs that continued to fail.

Environment

Host

• HP Elite Tower 800
• Intel Core i7-12700 (Alder Lake hybrid: 12 physical cores, 20 logical)
• 32 GB RAM, NVMe storage
• Linux Mint 22.x, kernel 6.17.0-x-generic
• VirtualBox 7.2.14 r174565, Extension Pack installed, but also experienced with other 7.2 versions
• Full VT-x available and in use. HM: Using VT-x implementation 3.0,
  nested paging and unrestricted guest enabled, UseNEMInstead = 0.
  No Hyper-V involvement - these are Linux hosts (see "Ruled out" below).

Guest

• Linux Mint 22.2 (MATE), Ubuntu 24.04 base
• Reproduced on kernel 6.8.0-139-generic and 6.17.0-35-generic and 6.14.0-37-generic
• 4 vCPUs, 8 GB RAM, EFI firmware
• Reproduced on both PIIX3 and ICH9 chipsets
• VMSVGA graphics, 256 MB VRAM, 3D disabled
• PCnet-FAST III NIC, NAT
• Immutable disk (differencing VMDK, discarded each boot)
• Guest Additions 7.2.14 installed
• Effective paravirt provider: KVM

Frequency: intermittent. Occurs on a minority of boots, but reliably affects
several machines per lab session when 30+ machines boot at around the same time. The machines that are affected differ each time - they are not the same machines all the time.

Symptoms

Boot proceeds normally to the point where systemd is starting services, then:

rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-...D } 60580 jiffies s: 93 root: 0x8/.
rcu: blocking rcu_node structures (internal RCU debug):
INFO: task (sd-mkdcreds):492 blocked for more than 122 seconds.
      Tainted: G           W  OE     6.8.0-139-generic #139-Ubuntu
INFO: task kworker/1:0:25 blocked for more than 122 seconds.

The stall message repeats with a rising jiffies count (observed up to 781474
jiffies, roughly 13 minutes, before the machine was reset manually). It does not
recover.

Which CPU is named varies between boots (CPU 1 and CPU 3 both observed), as does
the accompanying root: mask (0x2/, 0x8/).

Which systemd units fail varies between machines and boots - polkit.service,
systemd-logind.service, accounts-daemon.service,
switcheroo-control.service have all been seen. These are not independent
failures: they are whichever services happened to be forking when the block
occurred. Blocked task names are frequently truncated systemd children caught
mid-exec.

Key diagnostic evidence

The reported CPU is idle, not stuck

With a guest in the hung state, from the host:

$ VBoxManage debugvm <vm> info cpumguest

All four vCPUs, including CPU 3 which RCU reports as blocking, show:

rip=ffffffffb245895b rsp=ffffffffb3803e18 rbp=ffffffffb3803e20  ... eflags=00000246
rip=ffffffffb245895b rsp=ffffd2ecc00b7e78 rbp=ffffd2ecc00b7e80  ... eflags=00000246
rip=ffffffffb245895b rsp=ffffd2ecc00bfe78 rbp=ffffd2ecc00bfe80  ... eflags=00000246
rip=ffffffffb245895b rsp=ffffd2ecc00c7e78 rbp=ffffd2ecc00c7e80  ... eflags=00000246

Points to note:

• All four vCPUs are at the same kernel RIP, on per-CPU idle stacks spaced
  0x8000 apart.
• VBoxManage debugvm <vm> info cpumguestinstr disassembles that address as
  c3 retn - the function epilogue reached after hlt returns in the idle loop.
• Every general-purpose register is zeroed, consistent with an idle task.
• eflags=00000246 - bit 9 (IF) is set. Interrupts are enabled on the
  "blocking" CPU.
• Sampled five times over 15+ seconds: values byte-identical, no progress.

This state is indistinguishable from a healthy idle guest. The CPU is not
spinning on a lock, not in a driver, and not executing anything. It is halted
and available, waiting for an interrupt that never arrives.

SysRq confirms the kernel is alive

With kernel.sysrq=1 set:

• SysRq-W (show blocked state) printed sysrq: Show Blocked State to the
  console and then listed no tasks - despite the hung task detector reporting
  two blocked tasks moments earlier. This is consistent with the tasks waiting
  in synchronize_rcu_expedited, which is a killable wait not enumerated by the
  blocked-state dump.
• SysRq-L (backtrace all active CPUs) printed
  sysrq: Show backtrace of all active CPUs and then produced no backtraces at
  all, suggesting the NMIs it sends were also not answered.

Both handlers executed, so the guest kernel is running and servicing keyboard
interrupts on at least one CPU.

Host side is healthy

top -H -p <VirtualBoxVM pid> during the hang shows all EMT threads in S
state at ~2% CPU. No thread is in D state. The host is not blocked on I/O and
the VM process is not spinning.

Associated kernel warning

A WARN_ON in RCU fires on affected guests, and also on some successful boots:

WARNING: CPU: 3 PID: 381 at kernel/rcu/tree_plugin.h:815 rcu_sched_clock_irq+0x3ad/0x3c0
CPU: 3 PID: 381 Comm: (udev-worker) Tainted: G           OE      6.8.0-139-generic #139-Ubuntu
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Call Trace:
 <IRQ>
 update_process_times+0x76/0xb0
 tick_sched_handle+0x28/0x70
 tick_nohz_highres_handler+0x78/0xa0
 __hrtimer_run_queues+0x107/0x250
 hrtimer_interrupt+0xf6/0x250
 __sysvec_apic_timer_interrupt+0x4e/0x120
 sysvec_apic_timer_interrupt+0x8d/0xd0
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1b/0x20
RIP: 0010:__slab_free+0xc8/0x2c0

The call trace contains no VirtualBox modules. It is an APIC timer interrupt
arriving while a udev-worker is in __slab_free, and RCU's consistency check
failing. This is the same warning reported in ticket #22222 (there at
tree_plugin.h:744 on a 6.11 kernel - same function, same (udev-worker)
process, same G OE taint).

This warning appears on healthy boots too, so it is not by itself the hang - but
it indicates RCU is repeatedly finding itself in an unexpected state relative to
timer interrupt delivery.

Relevant VBox.log lines

GIM: Using provider 'KVM' (Implementation version: 0)
APIC0: Switched mode to x2APIC
APIC: fPostedIntrsEnabled=false fVirtApicRegsEnabled=false fSupportsTscDeadline=false
HM: Using VT-x implementation 3.0

The guest CPUID hypervisor leaf 40000001 returns 01000089 - clocksource,
clocksource2, PV_UNHALT and the stable-TSC bit. KVM_FEATURE_STEAL_TIME and
PV_SCHED_YIELD are not advertised.

The APIC is emulated entirely in software with no hardware assist (no posted
interrupts, no virtual APIC registers, no TSC deadline).

Ruled out

Each of the following was tested and does not affect the outcome:

Hypothesis	Test	Result
Guest kernel version	6.8.0-139 and 6.17.0-35	Identical failure on both
Chipset	PIIX3 and ICH9	Identical failure on both
Shared folders (vboxsf)	Removed the only shared folder, one full lab	Still failed at normal rate
Graphics driver	vmwgfx absent from call trace	Not implicated
Host storage	NVMe; host threads never in D state	Not implicated
Hyper-V / NEM fallback	Linux hosts, full VT-x, UseNEMInstead=0	Not applicable
Core dump handling	apport removed, systemd-coredump capped	Symptom persisted

The reporter notes the guest previously ran without incident on kernel 5.19.17,
suggesting changes in RCU's expedited grace period usage and/or tick handling
since then have made a pre-existing race reachable.

Workaround

Adding the following to the guest kernel command line eliminates the hang:

nox2apic rcupdate.rcu_normal=1

rcupdate.rcu_normal=1 disables expedited grace periods entirely for the life of
the boot. The guest confirms this at startup:

rcu: Preemptible hierarchical RCU implementation.
     No expedited grace period (rcu_normal).

With expedited grace periods disabled, RCU waits passively for each CPU to pass
through a quiescent state via the tick rather than demanding acknowledgement by
IPI, so the undelivered IPI no longer blocks anything.

nox2apic forces the guest onto legacy xAPIC. It has not been independently
verified as necessary - the two parameters were deployed together.

Note that rcupdate.rcu_normal_after_boot=1 alone is not sufficient, because
the hang occurs during boot when expedited grace periods are still forced on.

NOTE: Prior to this work, it was observed that manually installing kernel 5.19.17-generic on the guest Linux OS worked without issue. No lock ups were observed on the same machines.

Verification

Configuration	Labs	Machines in use	Boot hangs
nox2apic rcupdate.rcu_normal=1	3	~30+ per lab	0
Shared folder removed, otherwise unchanged	1	~30+	Several
Unchanged (control)	2	~30+	Several

All labs ran the same guest image and the same VirtualBox version, on the same
host hardware model, during the same assessment sessions.

Suggested area to investigate

The evidence points to expedited RCU grace period IPIs occasionally not being
delivered to, or not waking, a halted vCPU. Given that the target CPU is in
hlt with IF set, candidate areas include:

• IPI delivery to a vCPU in the halted activity state under the software APIC
  emulation, particularly in x2APIC mode
• Interaction between IPI delivery and the VMX-preemption-timer-driven wakeup
  path (fUsePeriodicPreemptionTimers=false, cPreemptTimerShift=7)
• Any window between a vCPU signalling halt and the emulation registering it as
  interruptible

The intermittency and the correlation with many VMs starting simultaneously on
separate hosts suggest a timing-sensitive race rather than a systematic error.

Related reports

• Ticket #22222 - "Linux guest kernel warning on boot". Same RCU warning,
  same function, same (udev-worker) context, on Debian 6.11.4.
• Forum topic 112589 - "Ubuntu 22.04 randomly fails to boot". Same expedited
  stall signature with a climbing jiffies count, across VirtualBox 7.0.x, 7.1
  and 7.1.2.
• GitHub issue [Bug]: multiple problems with rcu_prehempt at boot on Debian 13 #567 - multiple rcu_preempt stalls at boot
  on Debian 13 under VirtualBox 7.2.6.

None of these appear to have established that the reported CPU is idle and
interruptible rather than stuck, which is what distinguishes a lost IPI from a
guest-side deadlock.

Secondary issue

VBoxManage modifyvm <vm> --x2apic off also disables the local APIC entirely.
VBoxManage showvminfo subsequently reports:

APIC:                        disabled
X2APIC:                      disabled

and the guest fails to boot with 4 vCPUs. --apic on --x2apic off does not
restore the expected combination; APIC remains disabled. Tested on VirtualBox
7.2.14 r174565 on a Linux host. As noted above, this symptom has been observed on earlier versions of VirtualBox, too.

If disabling x2APIC while retaining the local APIC is intended to be supported,
this looks like a separate bug. It also blocks the most obvious host-side test
of the issue described above.

How can we reproduce this?

The bug report includes various details on the nature of the environment. To increase your chances of seeing this bug you would need to set up a lab of many machines. The report was based on a sample of 166 machines. That said, here is more detailed information:

Evidence attachment: VirtualBox Linux guest boot hang

All output below is verbatim as captured. Provenance is stated for each item,
because not every capture came from a guest in the hung state - please read the
"Captured from" line before using any of it.

---

1. Guest CPU state during the hang (primary evidence)

Captured from: a guest actively hung at boot, ICH9 chipset, guest kernel
6.8.0-139-generic, VirtualBox 7.2.14 r174565, Linux host. Console at the time
was repeating rcu_preempt detected expedited stalls on CPUs/tasks: { 3-...D }
with the jiffies count climbing (781474 jiffies at the last observation, roughly
13 minutes).

$ vboxmanage debugvm Linux_Mint_22.2 info cpumguest
Guest CPUM (VCPU 0) state:
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000 rdx=0000000000000000
rsi=0000000000000000 rdi=0000000000000000 r8 =0000000000000000 r9 =0000000000000000
r10=0000000000000000 r11=0000000000000000 r12=ffffffffb380fb40 r13=0000000000000000
r14=0000000033da2000 r15=00000000d9020000
rip=ffffffffb245895b rsp=ffffffffb3803e18 rbp=ffffffffb3803e20 iopl=0  iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=0000 es=0000 fs=0000 gs=0000 tr=0040      eflags=00000246
cr0=80050033 cr2=7f6efb533010 cr3=105d90002 cr4=000706f0 gdtr=fffffe6bedc2c000:007f ldtr=0000
Guest CPUM (VCPU 1) state:
rax=0000000000000000 rbx=0000000000000001 rcx=0000000000000000 rdx=0000000000000000
rsi=0000000000000000 rdi=0000000000000000 r8 =0000000000000000 r9 =0000000000000000
r10=0000000000000000 r11=0000000000000000 r12=ffff8ad700860000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
rip=ffffffffb245895b rsp=ffffd2ecc00b7e78 rbp=ffffd2ecc00b7e80 iopl=0  iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=0000 es=0000 fs=0000 gs=0000 tr=0040      eflags=00000246
cr0=80050033 cr2=7f6ef748d800 cr3=10a042003 cr4=000706f0 gdtr=fffffe5a4dead000:007f ldtr=0000
Guest CPUM (VCPU 2) state:
rax=0000000000000000 rbx=0000000000000001 rcx=0000000000000000 rdx=0000000000000000
rsi=0000000000000000 rdi=0000000000000000 r8 =0000000000000000 r9 =0000000000000000
r10=0000000000000000 r11=0000000000000000 r12=ffff8ad700865200 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
rip=ffffffffb245895b rsp=ffffd2ecc00bfe78 rbp=ffffd2ecc00bfe80 iopl=0  iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=0000 es=0000 fs=0000 gs=0000 tr=0040      eflags=00000246
cr0=80050033 cr2=7f6efb533010 cr3=10a434006 cr4=000706f0 gdtr=fffffe3b458f0000:007f ldtr=0000
Guest CPUM (VCPU 3) state:
rax=0000000000000000 rbx=0000000000000001 rcx=0000000000000000 rdx=0000000000000000
rsi=0000000000000000 rdi=0000000000000000 r8 =0000000000000000 r9 =0000000000000000
r10=0000000000000000 r11=0000000000000000 r12=ffff8ad700972900 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
rip=ffffffffb245895b rsp=ffffd2ecc00c7e78 rbp=ffffd2ecc00c7e80 iopl=0  iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=0000 es=0000 fs=0000 gs=0000 tr=0040      eflags=00000246
cr0=80050033 cr2=7f6ef75e8484 cr3=105d90005 cr4=000706f0 gdtr=fffffe294935f000:007f ldtr=0000

What this shows. CPU 3 is the CPU that RCU reports as blocking the expedited
grace period ({ 3-...D }, root: 0x8/). It is in exactly the same state as the
three CPUs RCU is content with:

• Same RIP on all four vCPUs.
• Per-CPU idle stacks, spaced 0x8000 apart: ...b7e78, ...bfe78, ...c7e78.
• All general-purpose registers zeroed except rbx/r12, consistent with the
  idle task.
• eflags=00000246 - bit 9 (IF) set. Interrupts are enabled.

The CPU is halted and interruptible, not stuck.

---

2. Instruction at the stalled RIP

Captured from: the same hung guest, same moment.

$ vboxmanage debugvm Linux_Mint_22.2 info cpumguestinstr
CPUM0: 0010:ffffffffb245895b c3                      retn
CPUM1: 0010:ffffffffb245895b c3                      retn
CPUM2: 0010:ffffffffb245895b c3                      retn
CPUM3: 0010:ffffffffb245895b c3                      retn

A single-byte ret - the function epilogue reached after hlt returns in the
idle loop. Not a spin loop, not a pause, not a port access.

---

3. Repeated sampling: no progress over 15 seconds

Captured from: the same hung guest. Five iterations at 3-second intervals.

$ for i in 1 2 3 4 5; do
    vboxmanage debugvm Linux_Mint_22.2 info cpumguest | grep -E '^rip'
    vboxmanage debugvm Linux_Mint_22.2 info cpumguestinstr
    echo ---; sleep 3
  done

rip=ffffffffb245895b rsp=ffffffffb3803e18 rbp=ffffffffb3803e20 iopl=0  iopl=0 nv up ei pl zr na po nc
rip=ffffffffb245895b rsp=ffffd2ecc00b7e78 rbp=ffffd2ecc00b7e80 iopl=0  iopl=0 nv up ei pl zr na po nc
rip=ffffffffb245895b rsp=ffffd2ecc00bfe78 rbp=ffffd2ecc00bfe80 iopl=0  iopl=0 nv up ei pl zr na po nc
rip=ffffffffb245895b rsp=ffffd2ecc00c7e78 rbp=ffffd2ecc00c7e80 iopl=0  iopl=0 nv up ei pl zr na po nc

CPUM0: 0010:ffffffffb245895b c3                      retn
CPUM1: 0010:ffffffffb245895b c3                      retn
CPUM2: 0010:ffffffffb245895b c3                      retn
CPUM3: 0010:ffffffffb245895b c3                      retn
---

Iterations 2 through 5 were byte-for-byte identical to the above and are omitted
for brevity. No vCPU made any progress over the sampling period.

---

4. SysRq results during the hang

Captured from: the same hung guest, with kernel.sysrq=1 set in the image.
Scancodes injected from the host via VBoxManage controlvm ... keyboardputscancode.
Both lines below appeared on the guest console.

[  593.006057] sysrq: Show Blocked State
[  769.012472] sysrq: Show backtrace of all active CPUs

Show Blocked State printed its header and then listed no tasks, despite the
hung task detector having reported (sd-mkdcreds):492 and kworker/1:0:25 as
blocked moments earlier. This is consistent with those tasks waiting in
synchronize_rcu_expedited, which is a killable wait and is not enumerated by
the blocked-state dump.

Show backtrace of all active CPUs printed its header and produced no
backtraces at all, suggesting the NMIs it broadcasts were also not answered.

Both handlers ran, so the guest kernel was alive and servicing keyboard
interrupts throughout.

---

5. RCU warning and call trace

Captured from: a guest that booted successfully, with the workaround
parameters nox2apic rcupdate.rcu_normal=1 already applied, guest kernel
6.8.0-139-generic.

This is an important caveat: the warning was captured on a healthy boot with the
workaround in place. It demonstrates that the warning fires independently of the
hang, and persists with the workaround applied. The W taint flag was present
on every hung boot observed, but the trace itself was never captured on a hung
machine because the console had scrolled past it and scrollback was unavailable.

$ dmesg | grep -A 30 'WARNING: CPU'
[    7.717351] WARNING: CPU: 3 PID: 381 at kernel/rcu/tree_plugin.h:815 rcu_sched_clock_irq+0x3ad/0x3c0
[    7.717356] Modules linked in: nf_reject_ipv4 nft_compat intel_uncore_frequency(-) snd_seq intel_uncore_frequency_common binfmt_misc intel_pmc_core intel_vsec pmt_telemetry nf_tables pmt_class snd_seq_device snd_timer snd nls_iso8859_1 rapl vboxguest(OE) soundcore i2c_piix4 input_leds mac_hid serio_raw sch_fq_codel vmwgfx drm_ttm_helper ttm msr parport_pc ppdev lp parport efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 dm_mirror dm_region_hash dm_log crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 psmouse video ahci pcnet32 libahci mii wmi aesni_intel crypto_simd cryptd
[    7.717383] CPU: 3 PID: 381 Comm: (udev-worker) Tainted: G           OE      6.8.0-139-generic #139-Ubuntu
[    7.717385] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[    7.717386] RIP: 0010:rcu_sched_clock_irq+0x3ad/0x3c0
[    7.717390] Code: 04 25 c0 41 03 00 f0 80 08 08 65 81 25 f0 7f 64 7b ff ff ff 7f e9 a2 fe ff ff 8b 87 38 08 00 00 85 c0 0f 84 23 fd ff ff eb b2 <0f> 0b e9 ed fc ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90
[    7.717391] RSP: 0018:ffffd13100168e68 EFLAGS: 00010086
[    7.717393] RAX: ffff8c09c3460000 RBX: 0000000000000000 RCX: 0000000000000000
[    7.717394] RDX: 00000000ffffffff RSI: ffffffff863ece26 RDI: ffff8c09c3460000
[    7.717395] RBP: ffffd13100168e80 R08: 0000000000000000 R09: 0000000000000000
[    7.717395] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[    7.717396] R13: ffff8c0ad7da4fe0 R14: ffffd1310054fae8 R15: ffff8c0ad7da4a40
[    7.717397] FS:  00007b83072218c0(0000) GS:ffff8c0ad7d80000(0000) knlGS:0000000000000000
[    7.717398] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    7.717399] CR2: 000075498690f3c0 CR3: 0000000103914005 CR4: 00000000000706f0
[    7.717402] Call Trace:
[    7.717403]  <IRQ>
[    7.717405]  update_process_times+0x76/0xb0
[    7.717407]  tick_sched_handle+0x28/0x70
[    7.717410]  tick_nohz_highres_handler+0x78/0xa0
[    7.717411]  ? __pfx_tick_nohz_highres_handler+0x10/0x10
[    7.717413]  __hrtimer_run_queues+0x107/0x250
[    7.717415]  hrtimer_interrupt+0xf6/0x250
[    7.717417]  __sysvec_apic_timer_interrupt+0x4e/0x120
[    7.717420]  sysvec_apic_timer_interrupt+0x8d/0xd0
[    7.717423]  </IRQ>
[    7.717423]  <TASK>
[    7.717424]  asm_sysvec_apic_timer_interrupt+0x1b/0x20
[    7.717426] RIP: 0010:__slab_free+0xc8/0x2c0
[    7.717430] Code: 89 44 24 28 0f 94 44 24 2f 0f b6 44 24 2f 66 45 85 f6 74 07 45 31 ff 84 c0 74 0a 45 31 ff 80 7c 24 28 00 74 44 41 8b 7c 24 08 <4c> 8b 4c 24 58 48 89 de 48 89 54 24 20 4c 8b 44 24 18 e8 c1 bb ff
[    7.717431] RSP: 0018:ffffd1310054fb90 EFLAGS: 00000246

What this shows. An APIC timer interrupt arrives while a (udev-worker) is
in __slab_free; the tick handler runs rcu_sched_clock_irq, and RCU's
consistency check fires. The call trace contains no VirtualBox modules -
vboxguest(OE) and vmwgfx are loaded but are not in the code path.

A second occurrence of the same warning on a different boot of the same machine
fired at [ 13.541392] on CPU 1, PID 390.

This matches ticket #22222, which reports the same warning in the same function
from the same (udev-worker) context on a Debian 6.11.4 guest (at
tree_plugin.h:744 - the line number differs only because of the kernel
version).

---

6. Confirmation the workaround is active

Captured from: a guest booted with nox2apic rcupdate.rcu_normal=1.

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.0-139-generic root=UUID=881c1b26-20c2-4ede-be5a-c5c89449d3b4 ro quiet nox2apic rcupdate.rcu_normal=1

$ dmesg | grep -i rcu
[    0.085068] rcu: Preemptible hierarchical RCU implementation.
[    0.085070] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4.
[    0.085071] 	No expedited grace period (rcu_normal).
[    0.085071] 	Trampoline variant of Tasks RCU enabled.
[    0.085072] 	Rude variant of Tasks RCU enabled.
[    0.085072] 	Tracing variant of Tasks RCU enabled.
[    0.085073] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[    0.085074] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.087584] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.213075] rcu: Hierarchical SRCU implementation.

$ nproc
4

No expedited grace period (rcu_normal) confirms expedited grace periods are
disabled for the life of the boot, so the stall path cannot execute. All four
vCPUs remain online. No Switched to x2APIC message appears, confirming
nox2apic took effect; on unmodified guests the host log shows
APIC0: Switched mode to x2APIC.

---

7. VM configuration

Captured from: the affected VM, powered off. This is the configuration in
use when the hangs occur (chipset shown here is PIIX3; the hang also reproduces
with --chipset ich9).

$ vboxmanage showvminfo Linux_Mint_22.2
Name:                        Linux_Mint_22.2
Guest OS:                    Ubuntu (64-bit)
Memory size:                 8192MB
VRAM size:                   256MB
CPU exec cap:                100%
CPUProfile:                  host
Chipset:                     piix3
Firmware:                    EFI
Number of CPUs:              4
HPET:                        disabled
PAE:                         enabled
Long Mode:                   enabled
Triple Fault Reset:          disabled
APIC:                        enabled
X2APIC:                      enabled
Nested VT-x/AMD-V:           disabled
CPUID overrides:             None
Hardware Virtualization:     enabled
Nested Paging:               enabled
Large Pages:                 disabled
VT-x VPID:                   enabled
VT-x Unrestricted Exec.:     enabled
CPUID Portability Level:     0
ACPI:                        enabled
IOAPIC:                      enabled
BIOS APIC mode:              APIC
UEFI Secure Boot:            disabled
RTC:                         UTC
IOMMU:                       None
Paravirt. Provider:          Default
Effective Paravirt. Prov.:   KVM
Graphics Controller:         VMSVGA
Monitor count:               1
3D Acceleration:             disabled
Storage Controllers:
#0: 'SATA', Type: IntelAhci, Instance: 0, Ports: 2 (max 30), Bootable
NIC 1:                       Attachment: NAT, Type: Am79C973
Pointing Device:             PS/2 Mouse
Keyboard Device:             PS/2 Keyboard
UART 1:                      disabled
Audio:                       enabled (Driver: Null, Controller: AC97, Codec: STAC9700)

The guest disk is immutable, so a fresh differencing image is created on every
start and discarded on shutdown.

---

8. Host-side APIC and paravirtualisation state

Captured from: VBox.log of the affected VM. These lines are identical on
hung and successful boots.

GIM: Using provider 'KVM' (Implementation version: 0)
APIC0: Switched mode to x2APIC
APIC: fPostedIntrsEnabled=false fVirtApicRegsEnabled=false fSupportsTscDeadline=false
HM: Using VT-x implementation 3.0
HM: Enabled nested paging
HM: Enabled unrestricted guest execution
HM: Enabled VMX-preemption timer (cPreemptTimerShift=7)
VMM: fUsePeriodicPreemptionTimers=false

Guest hypervisor CPUID leaf:

Gst: 40000000/0000  40000001 4b4d564b 564b4d56 0000004d
Gst: 40000001/0000  01000089 00000000 00000000 00000000

0x01000089 = bits 0, 3, 7, 24: clocksource, clocksource2, PV_UNHALT, and the
stable-TSC bit. KVM_FEATURE_STEAL_TIME (bit 5) and PV_SCHED_YIELD (bit 13)
are not advertised.

The local APIC is emulated in software with no hardware assist: no posted
interrupts, no virtual APIC register access, no TSC deadline support.

---

Did you upload all of your necessary log files, screenshots, etc.?

• Yes, I've uploaded all pertinent files to this issue.