Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3381,6 +3381,52 @@ x86_64 build is git-18a9cecb, pre-r31, which is what deps ≤ 1.16.0 shipped
chroma placement by its pixels on every platform; it skips only on a macOS x64
bundle at 1.16.0 or older, the known skew.

### FFT3DFilter: built from source with our determinism patch (task #62)

Every published FFT3DFilter build (myrsloik's R1/R2 releases, yuygfgg's
macOS arm64 prebuilt, upstream HEAD, the AmusementClub fork) gives a
**different output on every run**, so the Noise Reduction FFT3DFilter method,
TemporalDegrain2 (its spatial pass at every setting, and its post pass) and
QTGMC's `Denoiser='fft3dfilter'` never previewed exactly what they encoded.
Two faults, both fixed by `Scripts/patches/fft3dfilter-r2-determinism.patch`:

- **The left border was mirrored before the source line was copied**
(`FramePlaneToCoverbuf`), so the left 10 luma / 6 chroma columns came from
the previous frame's pixels, or uninitialised memory. Upstream regression
da50417 (2018).
- **A per-frame Wiener sigma lived in a shared member** of a filter
registered `fmParallel`: an edge frame (`btcur` 1) could overwrite it while
a middle frame used it, filtering that frame at 1/bt of its sigma. Hit
whenever `bt > 1` with a single sigma (the FFT3D method, TD2's post pass,
QTGMC with NoiseTR > 0).

Rules:

- **Built from source at tag R2 on every platform, patched; never a
prebuilt.** `FFT3DFILTER_TAG` in `download-deps-{macos,linux}.sh` and
`$Fft3dTag` in `download-deps-windows.ps1` stay in step. macOS compiles the
four files directly (its meson.build can't find `fftw3f_threads` on arm64)
and links the bundle's own FFTW; Linux uses meson; Windows cross-compiles
with the Zig the script already pins, against FFTW 3.3.5's DLL through an
import library from its `.def`. All three **fail the build** if the patch
stops applying. Plain C++, so one build serves both x64 tiers.
- **Never `bt=0` (Kalman).** It carries state from frame to frame by design
(upstream says it is "broken due to non-linear access"), so it can never
preview what it encodes. `NoiseReductionParameters::fft3d_effective_bt` clamps to
1..5 and no other call site passes 0; keep it that way.
- **FFTW plans are measured per process** (`measure=1`, the default). Seven
processes agreed on macOS arm64, and so does the push gate's second process
everywhere it runs; if one platform ever disagrees, pass `measure=0`
(FFTW_ESTIMATE) at our call sites and make it the plugin's default in the
patch (havsfunc's QTGMC call).
- `vapoursynth_integration_test.dart`'s `fft3dfilter is deterministic (task
#62)` (push gate) runs each argument class six times in fresh graphs at 8
threads, compares a second process, and checks a flat picture stays flat at
the left edge; it skips on bundles up to 1.17.x.
`integration_fft3dfilter_determinism_test.dart` (heavy) checks that two
previews and the lossless encode agree sample for sample, for the FFT3D
method and TD2 with its post pass (docs/reports/task-62-fft3dfilter-determinism).

## Debugging Tips

1. **Worker crashes**: Run worker standalone with `--config` to isolate
Expand Down Expand Up @@ -3853,3 +3899,4 @@ Full write-ups (root causes, measurements) for each entry are in
| 1.15.0 | 2026-11-03 | **Issue #144**: KNLMeansCL moves to Jaded Encoding Thaumaturgy's fork **v1.3** on every platform (Windows: PyPI wheel, replacing Khanattila's 2018 v1.1.1 zip, which refused 10-bit 4:2:0; macOS/Linux: the `v1.3` tag, replacing unpinned master, which gave unrepeatable results on Apple's OpenCL). Both meson patchers learn v1.3's assigned `run_command` include probe. Published as the tested `deps-v1.15.0-rc1` bundles with only the version changed |
| 1.16.0 | 2026-11-09 | **vszip 22.1.0 → 24.0.0** on every platform (CPU `NLMeans` for Noise Reduction's NLMeans (CPU) and the vine dehalo; macOS x64 sdist now built with Zig 0.17.0). Adds **SangNom** r45 (PyPI wheels; MIT source, GPL-2.0 binaries) on every platform for Anti-aliasing's SangNom method. Both single-file on x64 (no AVX), so `_tierFiles` is unchanged. **rc2: the AI models leave the bundle** — RIFE 4.26's (now the `frame-rate-rife` add-on) and the Real-ESRGAN anime / Real-CUGAN ones (now `upscale-anime`); the plugins stay, and their probes run stand-in models. Published as the tested `deps-v1.16.0-rc2` bundles with only the version changed (`Scripts/promote-deps-rc.py`) |
| 1.17.0 | rc1 2026-11-15, rc2 2026-11-16, rc3 2026-11-17 | **Task #36**: macOS arm64 libdvdread built from 6.1.3 source (patched to find a user's libdvdcss in `/opt/homebrew/lib`, `/usr/local/lib`, `/opt/local/lib`) instead of Homebrew's bottle, which linked `/opt/homebrew/opt/libdvdcss/…` and failed to load on Macs without it; macOS x64 gets the same patched build. Absolute build-machine rpaths stripped from libvapoursynth, vapoursynth.abi3.so, libfftw3f*, libboost_charconv, bm3d, dctfilter and mvtools (inert: nothing used `@rpath`). Windows and Linux binaries unchanged; libdvdread is now a required file on every platform. **rc2** (task #45): macOS x64 FFmpeg evermeet 9.0.1 → martin-riedl 9.0.2 (build 1789931006, pinned by SHA-256), which has SVT-AV1, so AV1 output works on Intel Macs; x264/x265/MP3 bitstreams change there, native codecs byte-identical. **rc3** (task #60): **Bifrost** v3.0 → **4.0** from its PyPI wheels on every platform (was the v3.0 tag's `bifrost.c` built from source on macOS/Linux, v3.0's 7z on Windows), under the same file names; v3.0 left chroma past the last whole 4x4 block uninitialised (different output run to run at width 98, 720x486), v4.0 initialises it and runs at 8–16 bits natively. No AVX, so `_tierFiles` unchanged |
| 1.18.0 | unreleased | **Task #62**: FFT3DFilter built from source at R2 with `patches/fft3dfilter-r2-determinism.patch` on every platform (Windows: Zig cross-compile replacing the R2 release DLL; macOS arm64: replacing yuygfgg's prebuilt), so its output no longer differs from run to run. One build for both x64 tiers; file names unchanged |
13 changes: 11 additions & 2 deletions Scripts/download-deps-linux.sh
Original file line number Diff line number Diff line change
Expand Up @@ -927,11 +927,20 @@ build_plugin "ttempsmooth" \
"libttempsmooth.so" \
"$PLUGIN_BUILD_ENV meson setup build --buildtype=release && ninja -C build"

# FFT3DFilter
# FFT3DFilter, pinned to R2 and patched with
# patches/fft3dfilter-r2-determinism.patch (task #62): every published build
# has two faults that make its output differ from run to run (the left border
# mirrored from the previous frame's pixels; a per-frame sigma in a shared
# member of a parallel filter), so a preview never matched its encode. The
# patch header has both. A patch that stops applying exits the script (the
# `exit` runs inside build_plugin's eval). Keep FFT3DFILTER_TAG in step with
# download-deps-macos.sh and download-deps-windows.ps1.
FFT3DFILTER_TAG="R2"
build_plugin "fft3dfilter" \
"https://github.com/myrsloik/VapourSynth-FFT3DFilter.git" \
"libfft3dfilter.so" \
"$PLUGIN_BUILD_ENV meson setup build --buildtype=release && ninja -C build"
"{ git apply \"$SCRIPT_DIR/patches/fft3dfilter-r2-determinism.patch\" || { echo ' ERROR: patches/fft3dfilter-r2-determinism.patch did not apply to FFT3DFilter $FFT3DFILTER_TAG.' >&2; exit 1; }; } && $PLUGIN_BUILD_ENV meson setup build --buildtype=release && ninja -C build" \
"$FFT3DFILTER_TAG"

# MiscFilters
build_plugin "miscfilters" \
Expand Down
99 changes: 56 additions & 43 deletions Scripts/download-deps-macos.sh
Original file line number Diff line number Diff line change
Expand Up @@ -791,12 +791,6 @@ if [ "$ARCH" = "arm64" ]; then
install_name_tool -id "@loader_path/libttempsmooth.dylib" "$PLUGINS_DIR/libttempsmooth.dylib" 2>/dev/null || true
codesign -s - -f "$PLUGINS_DIR/libttempsmooth.dylib" 2>/dev/null

# FFT3DFilter (also used by MCTemporalDenoise; links FFTW like dfttest)
curl -sL "$YUYGFGG_BASE/lib/libfft3dfilter.dylib" -o "$PLUGINS_DIR/libfft3dfilter.dylib"
install_name_tool -change "@rpath/libfftw3f.3.dylib" "@loader_path/../../lib/libfftw3f.3.dylib" "$PLUGINS_DIR/libfft3dfilter.dylib" 2>/dev/null || true
install_name_tool -change "@rpath/libfftw3f_threads.3.dylib" "@loader_path/../../lib/libfftw3f_threads.3.dylib" "$PLUGINS_DIR/libfft3dfilter.dylib" 2>/dev/null || true
codesign -s - -f "$PLUGINS_DIR/libfft3dfilter.dylib" 2>/dev/null

# VIVTC (inverse telecine - VFM + VDecimate)
curl -sL "$YUYGFGG_BASE/lib/libvivtc.dylib" -o "$PLUGINS_DIR/libvivtc.dylib"
install_name_tool -id "@loader_path/libvivtc.dylib" "$PLUGINS_DIR/libvivtc.dylib"
Expand Down Expand Up @@ -1300,31 +1294,6 @@ PYEOF
cd "$BUILD_DIR"
fi

# fft3dfilter (meson; links Homebrew FFTW, repoint at the bundled lib/ copy).
# Also used by havsfunc MCTemporalDenoise.
if [ "$FORCE" = true ] || [ ! -f "$PLUGINS_DIR/libfft3dfilter.dylib" ]; then
echo ""; echo "=== Building FFT3DFilter (x86_64) ==="
rm -rf fft3dfilter
if git clone --depth 1 https://github.com/myrsloik/VapourSynth-FFT3DFilter.git fft3dfilter \
&& PKG_CONFIG_PATH="$VS_PC_DIR:$BREW_PREFIX/lib/pkgconfig:${PKG_CONFIG_PATH:-}" \
meson setup fft3dfilter/build fft3dfilter --buildtype=release \
&& ninja -C fft3dfilter/build; then
lib=$(find fft3dfilter/build -name "*.dylib" -type f | head -1)
if [ -n "$lib" ]; then
cp "$lib" "$PLUGINS_DIR/libfft3dfilter.dylib"
install_name_tool -change "$BREW_PREFIX/opt/fftw/lib/libfftw3f.3.dylib" \
"@loader_path/../../lib/libfftw3f.3.dylib" "$PLUGINS_DIR/libfft3dfilter.dylib" 2>/dev/null || true
install_name_tool -change "$BREW_PREFIX/opt/fftw/lib/libfftw3f_threads.3.dylib" \
"@loader_path/../../lib/libfftw3f_threads.3.dylib" "$PLUGINS_DIR/libfft3dfilter.dylib" 2>/dev/null || true
codesign -s - -f "$PLUGINS_DIR/libfft3dfilter.dylib" 2>/dev/null || true
echo " Built FFT3DFilter"
else echo " no dylib"; FAILED_PLUGINS+=("fft3dfilter"); fi
else
echo " Failed to build FFT3DFilter"; FAILED_PLUGINS+=("fft3dfilter")
fi
cd "$BUILD_DIR"
fi

# BM3D (core.bm3d - Noise Reduction's BM3D method). Stefan-Olt has no
# current x86_64 build and the PyPI wheel is minos 15.0 (our floor is 12.0),
# so build the same tag as arm64 from source. Its meson.build probes the host
Expand Down Expand Up @@ -1466,18 +1435,6 @@ else
echo " DFTTest: using pre-built ARM64 version from yuygfgg"
fi

# FFT3DFilter
# On ARM64 we use the pre-built version from yuygfgg (downloaded above); its
# from-source build needs fftw3f_threads which isn't reliably discoverable here.
if [ "$ARCH" != "arm64" ]; then
build_plugin "fft3dfilter" \
"https://github.com/myrsloik/VapourSynth-FFT3DFilter.git" \
"libfft3dfilter.dylib" \
"meson setup build --buildtype=release && ninja -C build"
else
echo " FFT3DFilter: using pre-built ARM64 version from yuygfgg"
fi

# MiscFilters
build_plugin "miscfilters" \
"https://github.com/vapoursynth/vs-miscfilters-obsolete.git" \
Expand Down Expand Up @@ -1713,6 +1670,62 @@ else
echo " fmtconv already exists, skipping"
fi

# FFT3DFilter (core.fft3dfilter - Noise Reduction's FFT3DFilter method,
# TemporalDegrain2's spatial and post passes, QTGMC's Denoiser=fft3dfilter),
# built from source on BOTH arches at tag R2 with
# patches/fft3dfilter-r2-determinism.patch (task #62). Every published build,
# R2 included, has two faults that make its output differ from run to run, so
# a preview never matched its encode: the left border was mirrored from the
# previous frame's pixels, and a per-frame sigma lived in a shared member of a
# parallel filter. The patch header has both. Never take a prebuilt (arm64 had
# yuygfgg's, which has both faults). Keep FFT3DFILTER_TAG in step with
# download-deps-linux.sh and download-deps-windows.ps1.
#
# Compiled directly rather than through its meson.build, whose fftw3f_threads
# probe does not find the library on arm64: four .cpp files, upstream's own
# flags (release -O3, C++17, hidden visibility), FFTW's header from Homebrew,
# linked against the bundle's own lib/libfftw3f{,_threads}.3.dylib and then
# repointed at them through @loader_path. Plain C++ with no intrinsics, so one
# build serves both x64 tiers (no _tierFiles entry); MACOSX_DEPLOYMENT_TARGET
# (exported near the top) sets its minos.
FFT3DFILTER_TAG="R2"
cd "$BUILD_DIR"
if [ "$FORCE" = true ] || [ ! -f "$PLUGINS_DIR/libfft3dfilter.dylib" ]; then
echo ""
echo "=== Building FFT3DFilter ($FFT3DFILTER_TAG, patched) ==="
rm -rf fft3dfilter
git clone --depth 1 --branch "$FFT3DFILTER_TAG" -q \
https://github.com/myrsloik/VapourSynth-FFT3DFilter.git fft3dfilter
(cd fft3dfilter && git apply "$SCRIPT_DIR/patches/fft3dfilter-r2-determinism.patch") || {
echo " ERROR: patches/fft3dfilter-r2-determinism.patch did not apply." >&2
echo " FFT3DFilter $FFT3DFILTER_TAG may have changed upstream - re-check the patch." >&2
exit 1
}
FFTW_INC="$(brew --prefix fftw)/include"
if c++ -std=c++17 -O3 -fPIC -fvisibility=hidden -dynamiclib \
-I"$VS_INC_DIR" -I"$FFTW_INC" \
fft3dfilter/FFT3DFilter.cpp fft3dfilter/FFT3DFilterTransform.cpp \
fft3dfilter/fft3dfilter_c.cpp fft3dfilter/Plugin.cpp \
"$LIB_DIR/libfftw3f.3.dylib" "$LIB_DIR/libfftw3f_threads.3.dylib" \
-install_name "@loader_path/libfft3dfilter.dylib" -headerpad_max_install_names \
-o "$PLUGINS_DIR/libfft3dfilter.dylib"; then
# Whatever install name the bundled FFTW carries, point the plugin at
# the copy in deps/<platform>/lib (two levels up from the plugin dir).
for ref in $(otool -L "$PLUGINS_DIR/libfft3dfilter.dylib" | awk 'NR > 1 && /libfftw3f/ {print $1}'); do
install_name_tool -change "$ref" "@loader_path/../../lib/$(basename "$ref")" \
"$PLUGINS_DIR/libfft3dfilter.dylib"
done
codesign -s - -f "$PLUGINS_DIR/libfft3dfilter.dylib" 2>/dev/null || true
echo " Built FFT3DFilter -> libfft3dfilter.dylib"
BUILT_PLUGINS+=("fft3dfilter")
else
echo " Failed to build FFT3DFilter"
FAILED_PLUGINS+=("fft3dfilter")
fi
else
echo " FFT3DFilter already exists, skipping"
fi

# The three plugins below sit outside the arch split above, on BOTH arches.
# They started inside its arm64 branch, where x64 never reached them and the
# packaging guard failed on all three missing dylibs. x64 takes pre-built
Expand Down
Loading
Loading