Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1995,9 +1995,18 @@ original frames of a frame-rate conversion done by blending. Rules, from
- Not exposed: `thresh`, `mode`, `omode`, `speed` (upstream defaults); no
built-in preset turns it on.

- **Its decisions are not stable to a level of input change** (task #64):
the detector thresholds min/max statistics of blend maps, so QTGMC's
rounding at 10 bits (~0.35 levels) moves ~10% of decisions and ~5-8% of
output frames to a neighbouring original, with recovery as good. Never
expect two depths, deinterlacers or plugin versions to pick the same
frames; only the same picture at two depths (an exact shift) does.

`integration_unblend_test.dart` (heavy) checks recovery on the blended
fixtures (`generate_blended_test.sh`) against plain decimation, exact length
and rate, 8 = 10-bit, and preview = encode.
and rate, depth two ways (the same picture at 8 and 10 bits: identical
output; a real 10-bit source: as good a recovery, mostly the same picks),
and preview = encode.

### Frame Rate targets (issue #129)

Expand Down Expand Up @@ -3118,6 +3127,13 @@ Subtitle heavy tests skip when the whisper add-on is absent.
cd worker && cargo run --release -- --config test_job.json
```

> **A heavy test job on a >8-bit fixture sets `inputPixelFormat` and asserts
> the output's `pix_fmt`.** Before the source model (task #50) a job without
> it was decoded as 8-bit 4:2:0, silently, and Unblend's "10-bit = 8-bit"
> test compared the 8-bit run with itself for as long as it existed
> (task #64). The worker now probes the format, but only the assertion makes
> a test fail if its depth stops reaching the pass.

> **The heavy tests run the worker BINARY, not the library**, and
> `app/lib/**/*.g.dart` is gitignored — **run `cargo build` before the heavy
> suite** and `dart run build_runner build` before any Dart test, or you'll
Expand Down
157 changes: 138 additions & 19 deletions app/test/integration_unblend_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,12 @@
/// - length and rate: exactly floor(n * num / den) frames;
/// - preview = encode, at frames near the start and well into the clip,
/// where the detector's window is decoded fresh by the preview;
/// - depth: a 10-bit copy of the fixture gives the 8-bit result.
/// - depth, two ways (task #64): the same picture at 8 and 10 bits (an
/// exact shift, nothing ahead of Unblend that rounds) gives the same frames
/// exactly; a real 10-bit copy of the source, deinterlaced at 10 bits, is
/// unblended as well as the 8-bit one. Not the same frames: QTGMC rounds
/// differently at 10 bits, and srestore's min/max statistics turn that
/// into different picks for ~5-8% of frames, each a neighbouring original.
///
/// Heavy (full encodes): runs in the nightly workflow, not the push gate.
@Tags(['heavy'])
Expand Down Expand Up @@ -63,6 +68,57 @@ Future<List<Uint8List>> _frames(String path, {String? vf}) async {
Future<List<Uint8List>> _truth(int count) => _frames(_fixture('small_clip.mp4'),
vf: "select='between(n\\,$_first\\,${_first + count - 1})',setpts=N/TB");

/// Every frame's luma (Y plane), in 8-bit levels. Decoded in the planar
/// format itself, never through `gray`, which expands the range.
Future<List<Float64List>> _lumas(String path, {bool tenBit = false, String? vf}) async {
final all = await _raw(path, tenBit ? 'yuv420p10le' : 'yuv420p', vf: vf);
const luma = _width * _height, samples = luma * 3 ~/ 2;
final bps = tenBit ? 2 : 1;
final view = ByteData.sublistView(all);
return [
for (var o = 0; o + samples * bps <= all.length; o += samples * bps)
Float64List.fromList([
for (var i = 0; i < luma; i++)
tenBit ? view.getUint16(o + 2 * i, Endian.little) / 4 : all[o + i].toDouble(),
]),
];
}

double _lumaDiff(Float64List a, Float64List b) {
var sum = 0.0;
for (var i = 0; i < a.length; i++) {
sum += (a[i] - b[i]).abs();
}
return sum / a.length;
}

/// For each output frame, the original within +-3 of where it belongs that
/// is nearest in luma, and its distance.
List<(int, double)> _nearestLuma(List<Float64List> out, List<Float64List> truth) => [
for (var m = 0; m < out.length; m++)
[
for (var k = m - 3; k <= m + 3; k++)
if (k >= 0 && k < truth.length) (k, _lumaDiff(out[m], truth[k])),
].reduce((a, b) => a.$2 <= b.$2 ? a : b),
];

/// Every sample of every frame, raw (yuv420p bytes or yuv420p10le words).
Future<Uint8List> _raw(String path, String pixFmt, {String? vf}) async {
final r = await Process.run(
WorkerHarness.ffmpegPath,
['-v', 'error', '-i', path, if (vf != null) ...['-vf', vf], '-fps_mode', 'passthrough',
'-f', 'rawvideo', '-pix_fmt', pixFmt, '-'],
environment: WorkerHarness.ffmpegEnv,
stdoutEncoding: null,
);
if (r.exitCode != 0) throw Exception('decoding $path: ${r.stderr}');
final bytes = r.stdout as List<int>;
return bytes is Uint8List ? bytes : Uint8List.fromList(bytes);
}

Future<String?> _pixFmt(String path) async =>
(await WorkerHarness.firstStream(path, selector: 'v:0', entries: ['pix_fmt']))?['pix_fmt'] as String?;

/// For each output frame, its distance (rgb mean abs diff) to the nearest
/// original within +-3 of where it belongs.
List<double> _nearest(List<Uint8List> out, List<Uint8List> truth) => [
Expand All @@ -76,6 +132,7 @@ List<double> _nearest(List<Uint8List> out, List<Uint8List> truth) => [
VideoJob _job(String name, String input, double fps, {
required ProcessingPipeline pipeline,
ContainerFormat container = ContainerFormat.mkv,
String? pixelFormat,
}) =>
VideoJob(
id: const Uuid().v4(),
Expand All @@ -88,6 +145,10 @@ VideoJob _job(String name, String input, double fps, {
audioMode: AudioMode.none,
),
inputFrameRate: fps,
// Set for every >8-bit fixture: before task #50 a job without it was
// decoded as 8-bit 4:2:0, silently, and a depth test compared the
// 8-bit run with itself (task #64).
inputPixelFormat: pixelFormat,
);

ProcessingPipeline _unblend(FrameRateTarget target) => ProcessingPipeline(
Expand Down Expand Up @@ -150,15 +211,16 @@ void main() {

test('recovers PAL frames from NTSC blending, at 10-bit 4:2:2', () async {
final path = await _encode(_job('pal_unblend', _fixture('blended_pal_5994i.mkv'), 29.97,
pipeline: _unblend(FrameRateTarget.pal25)));
pipeline: _unblend(FrameRateTarget.pal25), pixelFormat: 'yuv422p10le'));
expect(await _pixFmt(path), 'yuv422p10le', reason: 'the 10-bit 4:2:2 path must really run');
final pal = await _truth(125);
final out = await _frames(path);
// 150 frames of 29.97i -> 300 at 59.94p -> floor(300 * 1001 / 2400).
expect(out.length, 125);
final unblended = _nearest(out, pal);
final decimated = _nearest(
await _frames(await _encode(_job('pal_decimate', _fixture('blended_pal_5994i.mkv'), 29.97,
pipeline: _decimate(FrameRateTarget.pal25)))),
pipeline: _decimate(FrameRateTarget.pal25), pixelFormat: 'yuv422p10le'))),
pal);
int clean(List<double> d) => d.where((v) => v <= _cleanLimit).length;
print(' clean: unblend ${clean(unblended)}/${out.length}, '
Expand All @@ -177,30 +239,87 @@ void main() {
expect(v?['nb_read_frames'], '143');
}, timeout: const Timeout(Duration(minutes: 10)));

test('a 10-bit copy of the source gives the 8-bit result', () async {
final ten = '$_outDir/blended_film_50i_10bit.mkv';
test('the same picture at 10 bits gives the 8-bit result exactly', () async {
// The detector's own input, deinterlaced once at 8 bits, and its exact
// 10-bit copy (ffmpeg's 8 -> 10 is a shift by 2). Nothing ahead of
// Unblend rounds, so every measurement and pick must be the same: a
// threshold left in 8-bit units at 10 bits would move most of them.
final eightIn = await _encode(_job('film_deint_8bit', _fixture('blended_film_50i.mkv'), 25,
pipeline: ProcessingPipeline(deinterlace: const QTGMCParameters(enabled: true, tff: true))));
final tenIn = '$_outDir/film_deint_10bit.mkv';
final r = await Process.run(WorkerHarness.ffmpegPath, [
'-y', '-v', 'error', '-i', eightIn, '-c:v', 'ffv1', '-pix_fmt', 'yuv420p10le', tenIn,
], environment: WorkerHarness.ffmpegEnv);
expect(r.exitCode, 0, reason: '${r.stderr}');
final progressive = ProcessingPipeline(
deinterlace: const QTGMCParameters(enabled: false),
unblend: UnblendParameters(enabled: true, target: FrameRateTarget.film23976),
);
final eight = await _encode(
_job('film_exact_8bit', eightIn, 50, pipeline: progressive, pixelFormat: 'yuv420p'));
final ten = await _encode(
_job('film_exact_10bit', tenIn, 50, pipeline: progressive, pixelFormat: 'yuv420p10le'));
expect(await _pixFmt(ten), 'yuv420p10le', reason: 'the 10-bit path must really run');
final a = await _raw(eight, 'yuv420p');
final b = await _raw(ten, 'yuv420p10le');
// 300 frames at 50p -> floor(300 * 480 / 1001).
const frameSamples = _width * _height * 3 ~/ 2;
expect(a.length, 143 * frameSamples);
expect(b.length, a.length * 2);
final words = ByteData.sublistView(b);
var differ = 0;
int? firstFrame;
for (var i = 0; i < a.length; i++) {
if (words.getUint16(2 * i, Endian.little) != a[i] << 2) {
differ++;
firstFrame ??= i ~/ frameSamples;
}
}
expect(differ, 0, reason: '$differ samples differ, first in output frame $firstFrame');
}, timeout: const Timeout(Duration(minutes: 10)));

test('a 10-bit copy of the source is unblended as well as the 8-bit one', () async {
final tenSource = '$_outDir/blended_film_50i_10bit.mkv';
final r = await Process.run(WorkerHarness.ffmpegPath, [
'-y', '-v', 'error', '-i', _fixture('blended_film_50i.mkv'),
'-c:v', 'ffv1', '-pix_fmt', 'yuv420p10le', ten,
'-c:v', 'ffv1', '-pix_fmt', 'yuv420p10le', tenSource,
], environment: WorkerHarness.ffmpegEnv);
expect(r.exitCode, 0, reason: '${r.stderr}');
final eight = await _frames(filmEncode);
final deep = await _frames(await _encode(
_job('film_unblend_10bit', ten, 25, pipeline: _unblend(FrameRateTarget.film23976))));
final ten = await _encode(_job('film_unblend_10bit', tenSource, 25,
pipeline: _unblend(FrameRateTarget.film23976), pixelFormat: 'yuv420p10le'));
expect(await _pixFmt(ten), 'yuv420p10le', reason: 'the 10-bit path must really run');
final truth = await _lumas(_fixture('small_clip.mp4'),
vf: "select='between(n\\,$_first\\,${_first + 143})',setpts=N/TB");
final eight = await _lumas(filmEncode);
final deep = await _lumas(ten, tenBit: true);
expect(deep.length, eight.length);
// The detector measures an 8-bit copy of the luma, so it picks the same
// frames; what differs is only the deinterlacer's rounding at 10-bit.
var worst = 0.0;
for (var m = 0; m < eight.length; m++) {
final d = WorkerHarness.meanAbsDiff(eight[m], deep[m]);
if (d > worst) worst = d;
}
print(' worst 8 vs 10-bit frame difference: ${worst.toStringAsFixed(3)}');
expect(worst, lessThan(1.0), reason: 'a different frame would differ by several levels');
final n8 = _nearestLuma(eight, truth), n10 = _nearestLuma(deep, truth);
// Luma: within 2 levels of an original (report: originals repeated
// through the same chain sit within ~1.5, a blend ~3.6 from either).
int clean(List<(int, double)> d) => d.where((x) => x.$2 <= 2.0).length;
final same = [for (var m = 0; m < eight.length; m++) if (n8[m].$1 == n10[m].$1) m];
final diffs = [for (final m in same) _lumaDiff(eight[m], deep[m])]..sort();
final median = diffs[diffs.length ~/ 2], worst = diffs.last;
print(' clean (luma <= 2.0): 8-bit ${clean(n8)}/${eight.length}, '
'10-bit ${clean(n10)}/${deep.length}; same original ${same.length}/${eight.length}; '
'luma 8 vs 10 where same: median ${median.toStringAsFixed(3)}, max ${worst.toStringAsFixed(3)}');
// Measured on macOS arm64 (task #64): 143/143 at both depths, 136/143
// the same original (the report's probe: 132), median 0.37 and max
// 1.26 levels where the same. Picks differ a little per platform.
expect(clean(n10), greaterThanOrEqualTo((deep.length * 0.95).floor()));
expect(clean(n10), greaterThanOrEqualTo(clean(n8) - (deep.length * 0.02).ceil()));
expect(same.length, greaterThanOrEqualTo((eight.length * 0.85).floor()));
expect(median, lessThanOrEqualTo(0.6));
expect(worst, lessThanOrEqualTo(2.0));
}, timeout: const Timeout(Duration(minutes: 10)));

test('the preview shows the encode\'s frame', () async {
final encoded = await _frames(filmEncode);
// Decoded as the preview converts to RGB (pipeline_executor.rs,
// preview_scale_filter, task #56 F8; the fixture is untagged, so
// limited range): with ffmpeg's default flags the same frame read
// 0.54 apart.
final encoded = await _frames(filmEncode,
vf: 'scale=in_range=tv:out_range=pc:flags=accurate_rnd+full_chroma_int');
// Near the start (the detector starts at the clip's first frame) and
// past the window, where the preview decodes only part of the clip.
for (final t in [3, 30, 60, 100, 140]) {
Expand Down
25 changes: 25 additions & 0 deletions docs/ENGINEERING_NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -5222,3 +5222,28 @@ for 48-px blocks. Ships in deps 1.18.0; the push gate's `fft3dfilter is
deterministic` and the heavy `integration_fft3dfilter_determinism_test.dart`
skip on bundles up to 1.17.x. Report:
[docs/reports/task-62-fft3dfilter-determinism](reports/task-62-fft3dfilter-determinism/index.html).

### Unblend's 8 = 10-bit test was vacuous; a real 10-bit source picks differently (task #64, 2026-11-17)

Report: [docs/reports/2026-11-srestore](reports/2026-11-srestore/index.html), section 8's correction.

- From dc9e36c (task #50) the heavy test "a 10-bit copy of the source gives
the 8-bit result" failed on every platform (13.08 levels on arm64, 13.43
on x86). Nothing in Unblend changed: its job named no `inputPixelFormat`,
and before the source model the worker decoded such a source as 8-bit
4:2:0, so the test had compared the 8-bit run with itself (0.000). The
deps version was ruled out (1.16.0 and 1.17.0-rc2 identical).
- At 10 bits QTGMC rounds differently (luma ~0.35 levels); srestore's
min/max statistics magnify that (`SrBMax` up to 31), 30 of 300 decisions
change, and 7-11 of 143 output frames take a neighbouring original.
Recovery is equal (143/143 clean at both depths). The exact-shift result
(same picture at both depths, identical decisions) stands.
- The test is now two: the same picture at 8 and 10 bits (Deinterlace off on
a lossless 50p intermediate) must give identical output, and a real 10-bit
source must recover as well, with >= 85% of picks the same. Both set
`inputPixelFormat` and assert the output's `pix_fmt`; forcing the 10-bit
job back to `yuv420p` fails the exact test.
- The preview test's own decode had drifted too: task #56 F8 made the
preview convert with `accurate_rnd+full_chroma_int`, which put the same
frame 0.54 rgb levels from a default-flags decode of the encode. The test
now decodes the encode with the preview's flags (0.00 again).
Loading
Loading