icd: Do not clear loader environment in extension enumeration - #49
Conversation
vkEnumerateInstanceExtensionProperties() only reads the cached instance extension list and does not call into the Vulkan loader, so the EnvironmentOverride taken at this entry point does not protect any downstream call. The override clears the loader environment variables for the whole process while it is held, which is visible to every thread. Because the Vulkan SC loader resolves drivers from the process environment on each pre-instance call, a driver scan issued by another thread during that window can resolve differently than it otherwise would. Removing the override here closes that window at this entry point. The override also acquires the environment mutex. No synchronization is needed for these reads: every write to instance_extension_list_ happens in the Global constructor, which completes at library load before any exported entry point can run, and the member is private with no accessor. This does not remove the window in general. The remaining EnvironmentOverride sites still modify the process environment for the duration of calls that do use the Vulkan loader.
|
Interestingly, this is the right solution for the wrong reason... It is, indeed, true, that the environment override in the end is unnecessary in However, the reason behind this happening to solve the problem is more subtle. First of all, the whole environment override shenanigans is necessary because of the historical bad decision to not use different environment variable names for Vulkan SC vs Vulkan, that we have to live with. Therefore, the Emulation ICD has to temporarily override them for the duration of calling into the Vulkan loader at points where the loader uses these. The locking inside the Emulation ICD's environment override utility is to ensure that no other API call is invoked simultaneously that may similarly depend on loader environment variables inside the ICD, although it's more of a guard-rail than actual safety, as the environment overrides can still impact other threads calling into the Vulkan SC loader itself (before even reaching the ICD). Unfortunately, this entire environment override trick relies on treating the loader in a somewhat white-box fashion (having to know where it reads environment variables and whether that is appropriately guarded against simultaneous calls). As it turns out, we do indeed have an issue... Let's look at the places where environment overrides are used:
Now the loader does take global locks appropriately in case of (3) and (4). However, interestingly, indeed it does not lock in case of (1), which explains the misbehavior there (I'm not sure if that changed over time or we just did not notice). Nonetheless, that should be fine with this patch applied, as we anyway cache the instance extension list. The only remaining case is (2). However, both (1) and (2) could be "problematic" because of the same underlying question: why ICD scanning is not protected by a mutex in the loader? ICD scanning happens in 3 places:
It is true that the actual ICD scanning itself does not necessarily require a global lock, although I'd argue it's probably not a great idea to scan and load the same drivers from multiple threads simulatenously. Anyway, it does not look like this change would entirely eliminate the possible "environment race conditions" without appropriate loader collaboration (it may be worth raising the question to the Vulkan loader folks why ICD scanning isn't mutexed in general), but certainly solves some of the problems. Honestly, probably, the easiest solution would be if the loader would mutex the Although, because this PR's commit description does not really capture the real issue and the code change contains a somewhat misleading and certainly unnecessary comment, I've created a separate PR to address this problem: #50 (I deemed that this is the easier path than iterating with Claude on the commit message / content). |
What
vkEnumerateInstanceExtensionProperties()takes anicd::EnvironmentOverride, but the function itcalls,
Global::EnumerateInstanceExtensionProperties(), only reads the cached instance extension listand never calls into the Vulkan loader. The override therefore protects no downstream call at this site.
This removes it, and replaces it with a comment explaining why it is absent.
Why it matters
EnvironmentOverrideclears the loader environment variables for the whole process while it is held, sothe effect is visible to every thread rather than only the calling one. The Vulkan SC loader resolves
drivers from the process environment on each pre-instance call, so a driver scan issued by another
thread during that window can resolve to a different driver than the application selected. Where a
second Vulkan SC driver is installed, an application using the two-call enumeration idiom from multiple
threads can then receive a count from one driver and a list from another.
Background and the full investigation are in Vulkan SC WG issue #304 (Khronos GitLab, members only).
Effect
Measured with the reproducing case from the Vulkan SC CTS, alternating a build of this branch against an
unmodified build of the same commit one run each, so that between-session variation applies to both:
This is a reduction in observed incidence, not a guaranteed fix. The four remaining
EnvironmentOverridesites still modify the process environment for the duration of calls that do usethe Vulkan loader, so the window remains open for other call patterns. With no occurrences in 210 runs
the upper bound on the remaining rate at this entry point is roughly 1.4%, which bounds the improvement
rather than demonstrating elimination.
The underlying rate is also unstable: across sessions on the same machine it varied from 8 in 60 runs to
0 in 150 consecutive runs, and running the system under heavy load suppressed it rather than increasing
it. Absolute rates from any single session should not be relied on, and failing to reproduce does not
indicate the window is closed.
Removing the mutex
The override also acquires the environment mutex, so this drops a lock. Every write to
instance_extension_list_occurs in theGlobalconstructor, which completes at library load beforeany exported entry point can run; the member is private, has no accessor, and no other code path writes
it. The reads at this site therefore need no synchronization.
Portions of the investigation and testing behind this change were assisted by Claude (Anthropic).