Skip to content

CPU-tiered x86 deps bundles (fixes #92) - #103

Open
StuartCameronCode wants to merge 9 commits into
mainfrom
deps-cpu-tiers
Open

CPU-tiered x86 deps bundles (fixes #92)#103
StuartCameronCode wants to merge 9 commits into
mainfrom
deps-cpu-tiers

Conversation

@StuartCameronCode

@StuartCameronCode StuartCameronCode commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes #92 (SIGILL on a pre-AVX Mac Pro 5,1). Root cause: libmvtools.dylib's AVX2 translation units build tables in static initializers, which run inside dlopen. R78 autoloads every plugin at core creation, so every job died before it started.

Every x86 deps bundle (macOS, Windows, Linux x64) now ships in two CPU tiers:

  • v3: x86-64-v3 CPUs (Haswell, 2013, and later). Keeps the plain asset names.
  • v2: anything older. Asset names are suffixed, e.g. VapourBox-deps-X-macos-x64-v2.zip.

The choice is made in one place, cpu::cpu_tier() in the worker. --probe-cpu reports it, the app downloads by it, and ctmf_opt derives from it. VAPOURBOX_DEPS_TIER=v2|v3 overrides it. On x86 an unclear answer means v2.

  • Worker: new cpu.rs. The zsmooth explicit-path machinery (locator, templates, script generator) is removed, because each bundle now ships one autoloaded zsmooth.
  • App: the tier appears only in the asset name and version.json, and the install dir stays deps/<platform>. Startup compares the installed tier with the machine's tier (DependencyStatus.wrongTier), so a hardware upgrade replaces the bundle even if it is newer than expected.
  • Build scripts: --tier / -Tier with a single mapping block per script. zsmooth is haswell or x86_64_v2. macOS v2 builds MVTools v24 from source with patches/mvtools-v24-no-avx2.patch.
  • CI: tier matrix in the three build-deps-* workflows. v2 bundles are gated before upload: Intel SDE on Linux (Westmere) and Windows (Sandy Bridge), and Scripts/check-load-time-simd.py on macOS. Nightly runs both tiers.
  • Probe kit: probe-plugin-compat.{py,sh} replaces the macOS-only script, and tests one plugin per process for a user on real hardware.

Before merging

deps-version.json already points at deps-v1.11.0, which is not published yet. Merging first would break ci-test on main and first-run downloads. Publishing order:

  1. Dispatch build-deps-{macos,windows,linux}.yml with version=1.11.0 release_tag=deps-v1.11.0.
  2. Check the release has all 8 assets and their sidecars, and that it is not marked Latest (--latest=false).
  3. Merge.

Testing

Run against the unpublished 1.11.0 bundles via deps_run_id:

  • ci-test: green on macOS arm64, macOS x64, Windows and Linux.
  • nightly (heavy suite): green on all 7 jobs, including x64-v2 on every platform.
  • The build-workflow v2 gates passed, and the macOS gate reports libmvtools with 6 initializers, none using VEX.
  • cargo test passes except the whisper subtitle test, which needs an addon that isn't installed locally.

Not verified

  • No pre-AVX2 hardware was available. Hosted runners are all Haswell or newer, and Rosetta translates AVX2. The macOS v2 result rests on the static check plus the reporter running the probe kit on the Mac Pro.
  • Pre-AVX Windows is unverified. SDE cannot emulate it, because the MSVC runtime reads the host's CPU features. Windows v2 is gated at Sandy Bridge only, which covers AVX without AVX2.

Deviations worth a look

  • Scripts/release.sh no longer packages deps locally and prints the build-deps-* commands instead. A local tree holds only one tier per platform.
  • The MVTools patch goes further than making the selectors return null: it masks X264_CPU_AVX2 and stubs the three AVX2 translation units with functions that abort, so no AVX2-compiled code is in the binary at all.

🤖 Generated with Claude Code

https://claude.ai/code/session_014GLXdGLfPwgYjW1AkonGqN

StuartCameronCode and others added 9 commits September 25, 2026 23:14
@StuartCameronCode @claude
Probe which bundled plugins a CPU can run, one plugin per process
19e3613 
Issue #92's first probe reported every plugin as crashing on a Westmere Mac
Pro, because R78 autoloads the whole plugin directory and MVTools faults in a
static initializer before any filter runs. Loading each plugin by path into a
DISABLE_AUTO_LOADING core, in its own process, makes the result per plugin.

The probe discovers plugins from the bundle instead of a hard-coded list and
runs with the bundle's own Python, so the same script covers macOS, Linux and
Windows. A new workflow runs it under Intel SDE, with zsmooth's AVX2-only
haswell build as a positive control that must crash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GLXdGLfPwgYjW1AkonGqN
@StuartCameronCode @claude
Make an SDE probe run prove it is valid before it counts
d114e43 
The first survey showed why the positive control alone is not enough: on
Windows at a pre-AVX chip, every test "crashed" inside VCRUNTIME140's memcpy,
because Microsoft's runtime chooses its AVX path from what the host kernel
reports, which SDE cannot emulate. The control plugin crashed too, so the run
looked valid while proving nothing.

A run now needs the plugin-free core test to pass under SDE, and the control to
fault inside its own image. The report names the faulting image, and a fault in
an OS runtime DLL is classified as inconclusive rather than blamed on a plugin.
Windows moves to Sandy Bridge (AVX, no AVX2), which is the tier boundary and
where the runtime's AVX path is legal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GLXdGLfPwgYjW1AkonGqN
@StuartCameronCode @claude
Probe: don't pass an empty argument for "no plugin"
8f8188a 
sde.exe drops empty arguments when it relaunches the child, so the core
control's empty plugin path vanished and shifted every argument after it.
Natively the same test passed, which is what made it look like a CPU finding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GLXdGLfPwgYjW1AkonGqN
@StuartCameronCode @claude
Worker: one CPU-tier decision, and zsmooth back to plain autoload
d213717 
The x86 deps bundles are splitting into CPU tiers (issue #92): v3 for
x86-64-v3 CPUs and v2 for anything older, where the prebuilt macOS MVTools
faults with SIGILL inside dlopen because its AVX2 files build their lookup
tables at load time.

cpu::cpu_tier() is the single decision: v3 needs the whole x86-64-v3 set, not
just AVX2, because the v3 plugins are compiled for the whole level.
--probe-cpu reports it for the app to choose a bundle, and ctmf_opt derives
from it, so the worker has one answer to "what is this machine".

With one zsmooth build per bundle, the explicit-path machinery that picked
between two builds (issue #82) goes: the bundle's zsmooth autoloads like every
other plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GLXdGLfPwgYjW1AkonGqN
@StuartCameronCode @claude
App: download the deps bundle for this CPU's tier, and re-check it at…
5f2fee1 
… startup

The tier lives only in which release asset is downloaded (macos-x64-v2 etc.)
and in version.json; the install directory stays deps/<platform>, so the
worker, dev paths and tests need no tier awareness.

The app asks the bundled worker (--probe-cpu) rather than re-deriving the
tier, so both share one definition. VAPOURBOX_DEPS_TIER overrides it, and any
unclear answer on x86 means v2, which runs everywhere and only costs speed.

Startup compares the installed tier with this CPU's and re-downloads on a
mismatch, before the version check, so an install carried over to (or from) a
different machine is replaced even when it is newer than expected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GLXdGLfPwgYjW1AkonGqN
@StuartCameronCode @claude
Build every x86 deps bundle in two CPU tiers, and gate the v2 one
4e83424 
Each download-deps script takes --tier v3|v2 (-Tier on Windows) and maps it in
one block at the top. The only differences, from running every bundled plugin
under Intel SDE and on a real Westmere Mac Pro:
  - zsmooth: the haswell build for v3, an x86_64_v2 build for v2; each bundle
    ships exactly one, in the autoload directory.
  - macOS MVTools: Stefan-Olt's prebuilt for v3; for v2, v24 from source with
    patches/mvtools-v24-no-avx2.patch, which builds none of the AVX2 files
    and masks the AVX2 flag, removing the load-time VEX that crashed dlopen.

The package scripts read the tier from version.json and name the asset from
it. A v2 bundle is gated before it can be published: under SDE on Linux
(Westmere) and Windows (Sandy Bridge, as SDE cannot emulate pre-AVX Windows),
and on macOS by check-load-time-simd.py, which proves statically that no
plugin runs AVX while it loads. Release upload moves after the gate.

release.sh no longer packages deps locally: one checkout holds only one tier
per platform, so it would publish an incomplete release. Deps 1.11.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GLXdGLfPwgYjW1AkonGqN
@StuartCameronCode @claude
Windows deps: quote zig's -Dcpu so the tier's CPU is actually passed
2243ffc 
PowerShell handed zig the literal text "$ZsmoothCpu": a bare native-command
argument that starts with a dash is not expanded, so the v2 build failed with
"unknown CPU".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GLXdGLfPwgYjW1AkonGqN
@StuartCameronCode @claude
Docs: CPU-tiered deps bundles (issue #92)
b4efbf8 
CLAUDE.md states the tier rules (one decision in the worker, the tier only in
the asset name and version.json, one --tier block per script, v2 gated before
publishing) in place of the zsmooth-by-path rule they replace, plus the 1.11.0
history row, the load-time SIGILL debugging tip, and the Rosetta caveat.
ENGINEERING_NOTES records the investigation, including the traps that produced
misleading results on the way. README tells users older CPUs are supported.
The Apple Silicon x64-deps instructions are removed: Homebrew's installer no
longer creates the Intel prefix they depend on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GLXdGLfPwgYjW1AkonGqN
@StuartCameronCode @claude
vapoursynth_integration_test: honour VAPOURBOX_DEPS_DIR
b66a96f 
Its own deps resolver ignored the override that the worker harness and the
app honour, so a run pointed at an unreleased bundle silently tested whatever
was in the repo's deps/ instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GLXdGLfPwgYjW1AkonGqN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Monterey 12.7.6 error vspipe exited with signal 4 (unknown)

1 participant

@StuartCameronCode