Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1946,13 +1946,25 @@ time span `[t, t+1)`; the worker reports `sub_frame`, `sub_frames` and
else the worker takes `totalFrames`; Soft Telecine's last cycle filled as
`pipe_frames` does). `FrameMap::Decimate` declares radius `cycle + 1`.
Which frame of the target's cycle goes depends on the picture: each frame
is tagged with its index, the script reads the tag where the target would
be, picks between the two outputs the walk computed (`after_drop`) and
prints `PREVIEW_DECIMATE:corr=N` for `PREVIEW_INFO`. Stabilize's pitch
is tagged with its index, and a `FrameEval` at the very end
(`_preview_decimate_pick`, `prop_src` the frame where the target would
be) picks between the two outputs the walk computed (`after_drop`) and
prints `PREVIEW_DECIMATE:corr=N` for `PREVIEW_INFO` when vspipe requests
the frame. **No template requests a frame while the script is
evaluated** (task #65): the first version read the tag with `get_frame()`
there, so the target's frame was ready the moment vspipe asked, and
**R78's vspipe counts a frame done before writing it**: its main thread
tore the script down under the frame callback (SIGSEGV, 2-3% of x86 IVTC
previews under load). R80 fixed vspipe (upstream 4ff3f8a0); until the
bundle has it, any preview can still hit it, only far more rarely.
`test_259` allows only the border colour's one-frame swatch. Stabilize's pitch
lookup starts at `preview_source_offset` (`first + skip - pad`).
`integration_ivtc_preview_test.dart` (heavy) checks previews against a
lossless encode by the film frame they hold
(docs/reports/issue-46-ivtc-preview).
lossless encode by the film frame they hold, and repeats the decimating
preview 36 times (docs/reports/issue-46-ivtc-preview,
docs/reports/task-65-ivtc-preview-teardown). An IVTC test needs a source
with pulldown: on progressive frames IVTC keeps each frame unchanged
(`_telecineFixture` in the parity test).
- A source frame with no output of its own (down-conversion, decimation)
shows the output just before it, as the preview always has.

Expand Down
46 changes: 43 additions & 3 deletions app/test/integration_high_bit_depth_filters_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,39 @@ Future<String> _shakenFixture(String pixFmt) async {
return path;
}

/// The master with 3:2 pulldown, at [pixFmt], for IVTC (task #65): its first
/// 20 frames as film, telecined to 25 slots, kept at the file's rate and
/// length (which `_job` declares). The master itself has no pulldown, so VFM
/// matches every frame to itself and VDecimate only drops whole frames: since
/// the preview runs VDecimate in the encode's phase (task #46), IVTC of frame
/// [_previewFrame] is that source frame unchanged (0.000 levels from the
/// passthrough). Here frame 12 is the cycle's mixed-field slot, which the
/// passthrough shows combed and IVTC field-matches (5.2 levels apart; 8 vs
/// 10-bit 0.19). Built from the master, so every depth holds one picture.
Future<String> _telecineFixture(String pixFmt) async {
final path = '$_outDir/telecined_$pixFmt.mkv';
if (File(path).existsSync()) return path;
final master = await _master();
final result = await Process.run(
WorkerHarness.ffmpegPath,
[
'-hide_banner', '-loglevel', 'error', '-y',
'-i', master,
'-vf', 'setfield=tff,trim=end_frame=20,'
'telecine=first_field=t:pattern=23,'
'setpts=N/(${_fps.toInt()}*TB),setfield=tff',
'-r', '${_fps.toInt()}',
'-c:v', 'ffv1', '-pix_fmt', pixFmt,
path,
],
environment: WorkerHarness.ffmpegEnv,
);
expect(result.exitCode, 0,
reason: 'telecined fixture generation failed for $pixFmt: '
'${result.stderr}');
return path;
}

/// The fixture [pass] runs on, at [pixFmt].
Future<String> _fixtureFor(_Pass pass, String pixFmt) =>
(pass.fixture ?? _fixture)(pixFmt);
Expand Down Expand Up @@ -495,8 +528,9 @@ class _Pass {
/// master lacks (null: [_fixture]): [_shakenFixture] for a stabiliser, which
/// has nothing to do on a picture with no global motion; [_scratchedFixture]
/// for DeScratch (task #34); [_crawlFixture] for the dot-crawl and rainbow
/// filters (task #37). The passthrough reference is rendered from the same
/// fixture.
/// filters (task #37); [_telecineFixture] for IVTC, which keeps a frame
/// with no pulldown unchanged (task #65). The passthrough reference is
/// rendered from the same fixture.
final Future<String> Function(String pixFmt)? fixture;

/// Whether the pass needs a working KNLMeansCL OpenCL device. Without one the
Expand Down Expand Up @@ -617,6 +651,9 @@ List<_Pass> _passes() => [
),
// vivtc.VFM is 8-bit only and runs on a downconverted metrics clip, taking
// its pixels from the full-depth clip via clip2 (issue: 10-bit ProRes IVTC).
// On the telecined fixture: the master has no pulldown, so IVTC kept its
// frame unchanged once the preview decimated in the encode's phase
// (task #46, #65), and passed only while the preview showed the wrong one.
_Pass(
'deinterlace_ivtc',
_only(
Expand All @@ -627,6 +664,7 @@ List<_Pass> _passes() => [
),
),
tolerance: 3.0,
fixture: _telecineFixture,
),

// DeScratch is 8-bit only: the pass analyses an 8-bit copy and applies
Expand Down Expand Up @@ -798,7 +836,8 @@ List<_Pass> _passes() => [
// This fixture holds no fade, and on a frame that is not fading the fix
// changes ~0.1 levels by design — so it cannot be required to change
// the picture; the parity check is what this case is for (the fix's
// effect is integration_classical_filters_test's).
// effect is integration_classical_filters_test's). The telecined
// fixture, so the parity run goes through real field matching.
_Pass(
'ivtc_fix_fades',
_only(
Expand All @@ -810,6 +849,7 @@ List<_Pass> _passes() => [
),
),
changesPicture: false,
fixture: _telecineFixture,
),

// Deflicker measures frame brightness against a windowed average, so its
Expand Down
43 changes: 43 additions & 0 deletions app/test/integration_ivtc_preview_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -302,4 +302,47 @@ void main() {
retime: true);
}, timeout: const Timeout(Duration(minutes: 10)));
});

// Task #65: in the 2026-11-16 nightly, decimating IVTC previews exited 1
// on x86: vspipe R78 segfaulted in teardown, under its own frame callback,
// because the preview read VDecimate's drop with get_frame() during script
// evaluation and its one frame came back at once (2-3% of previews with
// four running under CPU load; 0 run one at a time). It is read in the
// graph now (test_259 keeps it there). This runs the preview again and
// again, three at a time, at every cycle phase: too few runs to catch a
// 2% fault reliably, so test_259 is the guard and this the smoke test.
test('IVTC preview exits cleanly, run after run (task #65)', () async {
final job = VideoJob(
id: const Uuid().v4(),
inputPath: _prores.path,
outputPath: '$_outDir/repeat.mkv',
processingPipeline: const ProcessingPipeline(deinterlace: _ivtc),
qtgmcParameters: _ivtc,
encodingSettings: const EncodingSettings(
codec: VideoCodec.ffv1, container: ContainerFormat.mkv, audioMode: AudioMode.none),
detectedFieldOrder: FieldOrder.topFieldFirst,
inputWidth: _prores.width,
inputHeight: _prores.height,
inputPixelFormat: _prores.pixFmt,
inputFrameRate: 30000 / 1001,
totalFrames: _prores.slots,
).toJson();
final failures = <String>[];
final frames = [for (var i = 0; i < 36; i++) (i * 7) % _prores.slots];
for (var i = 0; i < frames.length; i += 3) {
final runs = await Future.wait([
for (final f in frames.skip(i).take(3))
WorkerHarness.runPreview(job, frame: f, label: 'repeat_$f')
.then((pv) => (f, pv)),
]);
for (final (f, pv) in runs) {
if (!pv.success) {
failures.add('frame $f: ${pv.error} | ${pv.errorTail}');
} else if (!pv.logs.contains('decimation dropped a frame at or before the target: ')) {
failures.add('frame $f: no decimation report');
}
}
}
expect(failures, isEmpty, reason: '${failures.length} of ${frames.length} previews');
}, timeout: const Timeout(Duration(minutes: 10)));
}
41 changes: 41 additions & 0 deletions app/test/preview_error_tail_test.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// The heavy tests report a failed preview by PreviewResult.errorTail. It must
// keep the worker's own error line, which carries vspipe's exit status, even
// when more than four lines follow it (task #65: the nightly showed only
// plugin warnings and vspipe's frame count).
import 'package:flutter_test/flutter_test.dart';

import 'support/worker_harness.dart';

void main() {
const status = 'Error generating preview: vspipe exited with signal 11 '
'(SIGSEGV) — a native crash; no Python error was printed';

test('keeps the worker error line when it has scrolled out of the tail', () {
final logs = [
status,
'Core freed but 3 filter instances still exist',
'Plugin already loaded (com.vapoursynth.a)',
'Plugin already loaded (com.vapoursynth.b)',
'Plugin already loaded (com.vapoursynth.c)',
'Output 1 frames in 0.00 seconds (1000.00 fps)',
].join('\n');
final tail = previewErrorTail(logs, 'preview exited with 1');
expect(tail, startsWith(status));
expect(tail, endsWith('Output 1 frames in 0.00 seconds (1000.00 fps)'));
expect(tail, isNot(contains('Core freed')));
});

test('does not repeat a worker error line already in the tail', () {
final logs = ['warning', status, 'Traceback line'].join('\n');
expect(previewErrorTail(logs, null), 'warning | $status | Traceback line');
});

test('without the worker line, the last four lines', () {
final logs = List.generate(6, (i) => 'line $i').join('\n');
expect(previewErrorTail(logs, null), 'line 2 | line 3 | line 4 | line 5');
});

test('no logs: the error', () {
expect(previewErrorTail('\n \n', 'timed out'), 'timed out');
});
}
36 changes: 24 additions & 12 deletions app/test/support/worker_harness.dart
Original file line number Diff line number Diff line change
Expand Up @@ -1212,6 +1212,27 @@ class JobResult {
: 'FAILED: $error (exit code: $exitCode)';
}

/// The last few lines of a failed preview's stderr (the VapourSynth error,
/// without the traceback noise above it), led by the worker's own
/// `Error generating preview:` line when that has scrolled out of them. That
/// line carries vspipe's exit status (task #56 F9b); in the 2026-11-16
/// nightly the tail held only plugin warnings and vspipe's "Output 1 frames"
/// line, so the status that said "crashed after writing" was never shown
/// (task #65).
String previewErrorTail(String logs, String? error) {
final lines = logs
.split('\n')
.map((l) => l.trim())
.where((l) => l.isNotEmpty)
.toList();
// A preview killed at its timeout has no logs; say so rather than nothing.
if (lines.isEmpty) return error ?? '';
final tail = lines.length <= 4 ? lines : lines.sublist(lines.length - 4);
final head = lines.indexWhere((l) => l.contains('Error generating preview:'));
if (head < 0 || head >= lines.length - tail.length) return tail.join(' | ');
return [lines[head], '…', ...tail].join(' | ');
}

/// Outcome of a `--preview` run: the PNG bytes, or an error plus the worker's
/// stderr (which carries the VapourSynth traceback for a failed pass).
class PreviewResult {
Expand All @@ -1223,18 +1244,9 @@ class PreviewResult {

bool get success => png != null && png!.isNotEmpty;

/// The last few stderr lines — the VapourSynth error, without the traceback
/// noise above it.
String get errorTail {
final lines = logs
.split('\n')
.map((l) => l.trim())
.where((l) => l.isNotEmpty)
.toList();
// A preview killed at its timeout has no logs; say so rather than nothing.
if (lines.isEmpty) return error ?? '';
return lines.length <= 4 ? lines.join(' | ') : lines.sublist(lines.length - 4).join(' | ');
}
/// The worker's own error line and the last few stderr lines; see
/// [previewErrorTail].
String get errorTail => previewErrorTail(logs, error);

@override
String toString() => success ? 'SUCCESS (${png!.length} bytes)' : 'FAILED: $error\n$errorTail';
Expand Down
35 changes: 35 additions & 0 deletions docs/ENGINEERING_NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -5181,3 +5181,38 @@ Report: [docs/reports/task-60-bifrost-v4](reports/task-60-bifrost-v4/index.html)
v3.0 (one run each).
- **The earlier 2026-08-15 note "Bifrost is 8-bit only", and its include-farm
note, describe v3.0.**

### The IVTC preview crashed vspipe in teardown; IVTC parity needs pulldown (task #65, 2026-11-18)

Report: [docs/reports/task-65-ivtc-preview-teardown](reports/task-65-ivtc-preview-teardown/index.html).
Both faults came in with task #46 (the preview runs VDecimate in the encode's phase).

- **vspipe R78 counts a frame done before writing it.** `frameDoneCallback`
increments `completedFrames` without the mutex, then writes the frame, and
takes the lock only to notify. `outputNode`'s `condition.wait(lock, pred)`
checks the predicate as soon as it is called, so if the frame completes
before the main thread reaches `wait`, main sees "all done", calls
`freeScript` and the callback segfaults (`outputFrame`,
`frameDoneCallback`; main in Python's GC under `vpy4_freeScript`). Fixed
upstream in 4ff3f8a0, first released in R80 ("Counting at entry … let the
waiter return while the last frame was still being written").
- **A one-frame preview hits it when its frame is ready at once.** #46's
preview read VDecimate's drop with `get_frame()` while the script was
evaluated, which computed the target's graph in advance. With four
previews at a time under CPU load, that template failed 23/1320 (Linux
x64), 21/1320 (x64-v2) and 4/1320 (macOS x64) IVTC previews, every Linux
failure a vspipe core in teardown. The drop is now read in the graph (a
`FrameEval` with `prop_src`): 0/2640 on Linux and 0/1320 on macOS. Run one
at a time, both templates were 0/2280; on Apple Silicon 0/600 (Rosetta).
- **It isn't gone.** Bwdif (no decimation) crashed the same way 2 times in
1200 on macOS x64, one with each template. Until the bundle has R80+, any
preview can.
- **IVTC needs pulldown to test.** The high-bit-depth parity master is
progressive, so IVTC of frame 12 is frame 12 unchanged (0.000 levels from
the passthrough); it "changed the picture" only while the preview showed
the wrong frame. The case now runs on the master telecined 3:2 (5.2 levels;
8 vs 10-bit 0.19).
- **Concurrent previews of one job id share a scratch directory**
(`vapourbox_preview_<id>`). The app gives every preview a fresh id; the
first repro run gave every loop one id and measured the collision, not the
crash.
1 change: 1 addition & 0 deletions docs/reports/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,3 +120,4 @@ to it.
| 2026-11-15 | [9-bit sources: Dehalo's YAHR failed](2026-11-9-bit-sources/index.html) | Built: 9-bit sources decode as 10-bit. CTMF has no 9-bit format, and havsfunc reaches it through `MinBlur(r>=2)` and directly: YAHR, FineDehalo contra, MCTemporalDenoise pfMode 2 and ContraSharpening r2 failed at every size, HQDeringmod and SMDegrain contrasharp at HD. The conversion is lossless (limited range an exact shift); the Noise Reduction CTMF block's own 9-bit guard is removed. `original` output of a 9-bit source is 10-bit (+1.2% FFV1). Windows/Linux by the nightly run only |
| 2026-11-16 | [Task #49: one output capability table, and the output conformance matrix](task-49-output-conformance/index.html) | Built: `output_capabilities.json` read by worker and app; a 444-case generated heavy test. 425/446 pass; three deterministic faults (19 cases) and an intermittent `pipe_source` wrong-frame fault filed as task #58 |
| 2026-11-17 | [Task #60: Bifrost v3.0 → v4.0](task-60-bifrost-v4/index.html) | Built: every platform takes Bifrost 4.0 from its PyPI wheel. v3.0's chroma past the last whole 4x4 block was uninitialised (2–3 results in 3 runs at width 98/718, 720x486); v4.0 gives one, identical to v3.0 inside whole blocks. Runs natively at 8–16 bits, luma bit-exact, 0.1% of chroma decisions differing from 8-bit at the threshold; the 8-bit copy is gone. 4:2:0 at 720x486 with `interlaced=True` is still refused (task #57) |
| 2026-11-18 | [Task #65: the IVTC preview's vspipe crash, and IVTC parity](task-65-ivtc-preview-teardown/index.html) | Fixed: vspipe R78 counts a frame done before writing it, so a one-frame preview whose frame is ready at once can be torn down under its callback (SIGSEGV; fixed upstream in R80). Task #46's evaluation-time `get_frame()` made the IVTC preview's frame ready at once: 2–3% of x86 IVTC previews under load crashed, 0/3960 with the read moved into the graph. Any preview can still hit it rarely (2/1200 Bwdif previews on macOS x64) until the bundle has R80+. The IVTC parity case needed a telecined fixture |
Loading
Loading