diff --git a/CLAUDE.md b/CLAUDE.md index d9f9a5a2..7f2ecf75 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1329,9 +1329,23 @@ the **single** builder of both decoders' arguments (encode + preview): mid-file: ffprobe counts the last segment only, `-ss` can't reach a later segment, and `-copyts` stops at the reset. `source_continuity.rs` is the one decision: a reset is ffmpeg's own (DTS > 0.1 s back or > 10 s forward, - per A/V stream), and the fix is a **same-container stream copy without - `-copyts`**, checked (no reset left, every video packet) before use and - made only when a reset is found — a file without one keeps every argument. + per A/V stream), and the fix is a **same-container stream copy**, checked + (no reset left, every video packet) before use and made only when a reset + is found — a file without one keeps every argument. + - **The picture decides the join (task #58).** Each segment moves by one + offset, every stream alike, so its first picture follows the last + segment's last by one frame (`JoinPlan`, made by the scan from each + packet's DTS, duration and position): `-copyts` and a `setts` filter per + stream (by packet index `N` for a stream that resets, by byte position + `POS` for one that doesn't: subtitles; MPEG audio split from a PES has + no position). A sound packet overhanging the next segment by more than + half of itself is dropped (`noise`), and the log says so. ffmpeg's own + correction (no `-copyts`) took the offset from whichever stream showed + the reset first: a part whose sound ran 16 ms past its last picture put + every later picture off the grid (125 fps, 493 pictures for 100). It is + still the fallback when no plan can be made (no durations, streams + resetting a different number of times) or the planned copy fails its + check. - **The app makes it after analysis, in the background (task #55).** The scan reads every byte (77 s for a 2 h DVD title on a network share), so analysis probes with `--probe-source --defer-scan` (the model @@ -1420,12 +1434,18 @@ the **single** builder of both decoders' arguments (encode + preview): transcription, `OutputWindow`, `SoundCut` and the notes keep the true start. The sidecar decision (`needs_sound_sidecar`) asks with the encoder's start, so `main.rs` measures the priming before the sound copy. + - **A trimmed one stops at the window's length (`-t`), not `-shortest`** + (`encode_end_args`, task #58): read late by P, a sound that ends with the + window (the restored track; a source's sound at a trim to its last + picture) ends P before the picture on ffmpeg's timeline, and + `-shortest` dropped the last picture (49 stored for 50). - The cost is the first P (21–23 ms) of sound, replaced by the encoder's near-silent start-up; a failed probe is a warning, never a refusal; every other job keeps its arguments byte for byte (`a_priming_shift_changes_nothing_outside_convert_into_avi`). `integration_video_start_offset_test.dart` (heavy): AAC/MP3/AC-3 into AVI - store 100 frames and are in sync within 1 ms, restored and trimmed too, + store 100 frames and are in sync within 1 ms, restored and trimmed too + (the matrix counts trimmed pictures since task #58), and `small_clip.mp4` 0..49 stores 50 (docs/reports/2026-11-avi-convert-priming). - **An AAC stream in AVI declares a block size no packet exceeds (task #41).** avienc writes AAC's `nBlockAlign` as 768 × channels, and ffmpeg, @@ -1783,11 +1803,18 @@ Invariants that are easy to break: `PitchFactorFile.sidecarCandidates` or the factor's meaning (`invertFactors`). - **Pitch on a still-interlaced clip clears `_FieldBased`** (a vertical resample would mix the fields; film scans are progressive). Weave alone leaves it. -- **`pipe_source` evicts silently.** It keeps `cache_frames` decoded frames and - returns the *last* frame read for anything already evicted. The cache is - sized from the pipeline's declared radius (`pipe_cache_frames`, never below - the old 40), so a pass with a large window must report it through its - `FrameMap` radius. +- **`pipe_source` raises on a frame it has dropped (task #58).** A pipe reads + forward only, and VapourSynth asks out of order: vspipe requests a frame + whenever any completes, so one late callback let the reader read the whole + clip past it (frames 6–7 served after 99 was read, on a plain job under + load). It keeps every frame until it is served and `cache_frames` behind + the oldest frame not yet served, with a memory cap (512 MB beyond the + window) for frames never asked for; a frame asked for after it was dropped + raises. It used to return the *last* frame read instead, silently. The + window is sized from the pipeline's declared radius (`pipe_cache_frames`, + never below 40), so a pass with a large window must report it through its + `FrameMap` radius, or its job now fails rather than writing wrong frames. + `pipe_source_order_test.dart` (push gate) drives the reader in that order. Tests: `test_211`…`test_218` (script generation, legacy block, offsets, errors, cache size, field marking) and `app/test/integration_stabilize_test.dart` @@ -2509,18 +2536,20 @@ table. Rules: codec the container takes, Restore, Restore + Convert AAC) × source class (plain MP4, late-video TS, delayed AVI, reset PS through the continuity copy) × trim (none, from frame 2, from frame 40), plus ProRes/MOV and - AV1/MP4 rows, from `generate_output_conformance.sh`'s flash/click sources: - ~440 cases, generated from the enums and the table + AV1/MP4 rows and trims to the last picture for the primed Convert codecs + into AVI (`tailCases`), from `generate_output_conformance.sh`'s + flash/click sources: 448 cases, generated from the enums and the table (`test/support/output_matrix.dart`, which also holds the oracle; the push gate's `output_matrix_cases_test.dart` checks the cases). Each checks the header, picture count and first picture, no null frames, every flash's - click within one audio packet, packet timing, and lossless paths sample- - or byte-exact. **A new container fact or source rule gets a case here, not - another one-cell heavy test.** Cases that fail today are `knownFailures` - (id → task, today all task #58); they still run and fail when they pass. - One fault can't be listed: `pipe_source` intermittently serves an evicted - frame as the last one (#58), so the matrix fails a different case or two - per run until that is fixed. One case: + click within one audio packet **of where the source has it** (each + source's own lag, measured: `conf_delayed.avi`'s MP3 is 23 ms late by + libmp3lame's priming, which AVI cannot mark), packet timing, and lossless + paths sample- or byte-exact (a restore against the render's decoder). + **A new container fact or source rule gets a case here, not another + one-cell heavy test.** Cases that fail today are `knownFailures` (id → + task; empty since task #58); they still run and fail when they pass. One + case: `flutter test --tags heavy test/integration_output_conformance_test.dart --plain-name 'avi/convert-aac/delayed-avi/near-start'`. The shared measuring helpers are `test/support/av_sync.dart`. @@ -2549,6 +2578,12 @@ Rules, from docs/reports/2026-11-audio-restoration: lead `adelay` → [DeepFilterNet] → `loudnorm` + `aresample` → `alimiter` → `apad`+`atrim` to the picture's length. Each ffmpeg run gets one `-af` (`every_combination_keeps_the_order_and_one_af_per_run`, 512 × 5 cases). +- **MP1/MP2 are read with the float decoder** (`render_decoder`: + `mp2float`, task #58). FFmpeg's default for them is fixed-point, which + rounds with an error carried from frame to frame, so a render that seeked + differed from the source's decode by one 16-bit step in ~40% of samples; + the float decode is exact from any seek. MP3 and AC-3 default to float + already; Convert and the analysis keep the default. - **Measured traps, each now handled in the plan:** `afftdn` outputs 25 ms late at every rate (`atrim=start_sample=rate/40`); its default `nf=-50` took 0.7 dB off −39 dB hiss — it gets the source's measured floor (quietest tenth diff --git a/app/test/integration_output_conformance_test.dart b/app/test/integration_output_conformance_test.dart index d0269d9f..0b6a3cf4 100644 --- a/app/test/integration_output_conformance_test.dart +++ b/app/test/integration_output_conformance_test.dart @@ -43,15 +43,12 @@ import 'support/worker_harness.dart'; String _resource(String name) => p.join(WorkerHarness.repoRoot, 'Tests', 'TestResources', name); -/// The task that owns the off-grid join (task #58, filed by task #49). -const _offGridJoin = 'off-grid reset join, task #58'; - final _outDir = p.join(Directory.systemTemp.path, 'vapourbox_output_conformance'); /// One source as the app analyses it. class _Source { _Source(this.decodePath, this.model, this.info, this.videoStart, this.payload, this.samples, - this.packetSeconds); + this.renderSamples, this.packetSeconds, this.clickLags); /// What the encode reads: the file, or the app's joined copy of a reset. final String decodePath; @@ -70,6 +67,18 @@ class _Source { /// The decoded sound, f32 mono. final Float32List samples; + /// The decoded sound as Audio Restoration's render decodes it + /// ([MatrixCase.referenceDecoder]): the float decoder for MP2. Null when + /// that is FFmpeg's default ([samples]). + final Float32List? renderSamples; + + /// How far each white frame's click sits after its picture in the source + /// itself, seconds, in [matrixFlashes] order. The delayed AVI's is 23 ms: + /// libmp3lame's priming, which AVI cannot mark, so every player plays it + /// that late. The output keeps the source's timing, so its clicks are + /// expected this far after their flashes (task #58). + final List clickLags; + /// The longest audio packet, seconds. final double packetSeconds; } @@ -88,10 +97,11 @@ Future _payload(String path) async { return Uint8List.fromList(r.stdout as List); } -Future _samples(String path) async { +Future _samples(String path, {String? decoder}) async { final r = await Process.run( WorkerHarness.ffmpegPath, - ['-v', 'error', '-i', path, '-map', '0:a:0', '-ac', '1', '-f', 'f32le', '-'], + ['-v', 'error', if (decoder != null) ...['-c:a', decoder], '-i', path, '-map', '0:a:0', '-ac', '1', + '-f', 'f32le', '-'], stdoutEncoding: null, environment: WorkerHarness.ffmpegEnv, ); @@ -132,8 +142,26 @@ Future<_Source> _prepare(SourceClass s) async { expect(info!.frameCount, matrixPictures, reason: '${s.file}: the app counts every picture'); expect(model.videoStartOffset, closeTo(s.nominalVideoStart, 0.03), reason: '${s.file}: where the picture starts after the sound'); + // Each click against its picture, on the source's own timeline: the + // container's start plus the picture's start plus n frames (not the + // decoded frame's PTS, which a delayed AVI stamps a frame late). + final fmt = await WorkerHarness.ffprobeJson( + ['-v', 'error', '-show_entries', 'format=start_time', '-of', 'json', decode]); + final start = double.tryParse('${(fmt['format'] as Map)['start_time']}') ?? 0; + final beeps = await beepOnsets(decode); + final lags = [ + for (final n in matrixFlashes) + beeps + .map((b) => b - (start + model.videoStartOffset + n / matrixFps)) + .reduce((a, b) => a.abs() < b.abs() ? a : b), + ]; + for (final l in lags) { + expect(l.abs(), lessThan(0.05), reason: '${s.file}: each click near its flash (lags $lags)'); + } + final render = s.renderDecoder; return _Source(decode, model, info, model.videoStartOffset, await _payload(decode), - await _samples(decode), _longest(await _audioTimes(decode))); + await _samples(decode), render == null ? null : await _samples(decode, decoder: render), + _longest(await _audioTimes(decode)), lags); } VideoJob _job(MatrixCase c, _Source s) { @@ -288,7 +316,8 @@ Future _check(MatrixCase c, _Source s) async { reason: '3. pictures ${i - 1}..$i are not one frame apart'); } - // 4. Every white frame has its click, within one audio packet. + // 4. Every white frame has its click, within one audio packet of where the + // source has it (its own lag after the picture, [_Source.clickLags]). final times = await _audioTimes(out); final tolerance = [_longest(times), s.packetSeconds].reduce((a, b) => a > b ? a : b) + 0.001; final e = await events(out); @@ -296,16 +325,18 @@ Future _check(MatrixCase c, _Source s) async { expect(e.flashes, hasLength(want.length), reason: '4. white frames at ${e.flashes}'); for (var i = 0; i < want.length; i++) { expect(e.flashes[i], closeTo(want[i], 0.5 / matrixFps), reason: '4. white frame $i'); + final lag = s.clickLags[matrixFlashes.indexOf(c.trim.flashes[i])]; + final at = e.flashes[i] + lag; final near = e.beeps.isEmpty ? double.infinity - : e.beeps.map((b) => (b - e.flashes[i]).abs()).reduce((a, b) => a < b ? a : b); + : e.beeps.map((b) => (b - at).abs()).reduce((a, b) => a < b ? a : b); final nearest = e.beeps.isEmpty ? null - : e.beeps.reduce((a, b) => (a - e.flashes[i]).abs() < (b - e.flashes[i]).abs() ? a : b); + : e.beeps.reduce((a, b) => (a - at).abs() < (b - at).abs() ? a : b); expect(near, lessThan(tolerance), reason: '4. the click for the flash at ${e.flashes[i].toStringAsFixed(4)} s is at ' - '${nearest?.toStringAsFixed(4)} (${((nearest ?? 0) - e.flashes[i]).toStringAsFixed(4)} s; ' - 'one packet ${tolerance.toStringAsFixed(4)} s)'); + '${nearest?.toStringAsFixed(4)} (${((nearest ?? 0) - e.flashes[i]).toStringAsFixed(4)} s ' + 'after it, the source\'s ${lag.toStringAsFixed(4)}; one packet ${tolerance.toStringAsFixed(4)} s)'); } // 5. The audio packets follow on. AVI stores no duration for its first @@ -335,7 +366,8 @@ Future _check(MatrixCase c, _Source s) async { final fmt = '${a['sample_fmt']}'; final raw = int.tryParse('${a['bits_per_raw_sample']}') ?? 0; final depth = raw > 0 ? raw : (fmt.startsWith('s16') ? 16 : fmt.startsWith('s32') ? 32 : 24); - final why = _sampleExact(await _samples(out), s.samples, depth); + final ref = c.referenceDecoder == null ? s.samples : s.renderSamples!; + final why = _sampleExact(await _samples(out), ref, depth); expect(why, isNull, reason: '6. the samples ($depth-bit): $why'); case LosslessCheck.none: break; @@ -384,25 +416,30 @@ void main() { // picture's grid (2.0 s of MP2 is 83.3 frames) is joined with its second // part's picture 16 ms late, so ffprobe reads 125 fps and the app analyses // 493 pictures. Known failure: fails while the fault is there. + // The first part's sound runs 16 ms past its last picture (a DVD cell's + // may). ffmpeg's own correction joined at the sound's end, so every later + // picture sat 16 ms off the grid and ffprobe read 125 fps; the picture now + // decides the join, and the overhanging sound packet goes (task #58). test('a reset whose sound ends off the picture grid joins on the grid', () async { - Object? failure; - try { - final service = TimestampContinuityService( - workerPath: WorkerHarness.workerPath, environment: WorkerHarness.workerEnv); - final r = await service.prepare(_resource('conf_reset_offgrid.mpg'), const Uuid().v4()); - expect(r.error, isNull); - expect(r.scan!.videoResetPictures, [50]); - addTearDown(r.proxy!.delete); - final info = await FieldOrderDetector() - .getVideoInfo(r.proxy!.path, await WorkerHarness.sourceModel(r.proxy!.path)); - expect(info!.frameRate, 25, reason: 'the copy\'s rate'); - expect(info.frameCount, matrixPictures, reason: 'the copy\'s pictures'); - } catch (e) { - failure = e; + final service = TimestampContinuityService( + workerPath: WorkerHarness.workerPath, environment: WorkerHarness.workerEnv); + final r = await service.prepare(_resource('conf_reset_offgrid.mpg'), const Uuid().v4()); + expect(r.error, isNull); + expect(r.scan!.videoResetPictures, [50]); + addTearDown(r.proxy!.delete); + final info = await FieldOrderDetector() + .getVideoInfo(r.proxy!.path, await WorkerHarness.sourceModel(r.proxy!.path)); + expect(info!.frameRate, 25, reason: 'the copy\'s rate'); + expect(info.frameCount, matrixPictures, reason: 'the copy\'s pictures'); + // Every picture one frame after the last, and each click on its flash. + final e = await events(r.proxy!.path); + expect(e.flashes, hasLength(matrixFlashes.length), reason: 'white frames at ${e.flashes}'); + for (var i = 0; i < matrixFlashes.length; i++) { + expect(e.flashes[i] - e.flashes[0], closeTo((matrixFlashes[i] - matrixFlashes[0]) / matrixFps, 0.002), + reason: 'flash $i on the grid'); + final near = e.beeps.map((b) => (b - e.flashes[i]).abs()).reduce((a, b) => a < b ? a : b); + expect(near, lessThan(0.013), reason: 'the click for flash $i within half a sound packet'); } - expect(failure, isNotNull, - reason: 'a known failure ($_offGridJoin) now passes: make this an ordinary test'); - print(' known failure ($_offGridJoin): $failure'); }); test('a copy the container cannot hold is refused before any work (task #49)', () async { diff --git a/app/test/output_matrix_cases_test.dart b/app/test/output_matrix_cases_test.dart index db255af1..0499b350 100644 --- a/app/test/output_matrix_cases_test.dart +++ b/app/test/output_matrix_cases_test.dart @@ -19,12 +19,13 @@ void main() { final rows = 1 + AudioCodec.values.where((c) => c.supportsContainer(k)).length + 2; final sources = SourceClass.values.length; final copied = SourceClass.values.where((s) => k.copies(s.audioCodec)).length; - expected += ((rows - 1) * sources + copied) * MatrixTrim.values.length; + expected += ((rows - 1) * sources + copied) * MatrixTrim.everyCase.length; } for (final (k, _) in extraVideoRows) { final copied = SourceClass.values.where((s) => k.copies(s.audioCodec)).length; - expected += (SourceClass.values.length + copied) * MatrixTrim.values.length; + expected += (SourceClass.values.length + copied) * MatrixTrim.everyCase.length; } + expected += tailCases.length; expect(cases, hasLength(expected)); print('${cases.length} cases'); }); @@ -50,6 +51,25 @@ void main() { } }); + test('the tail trims are the primed Convert codecs into AVI (task #58)', () { + final tail = cases.where((c) => c.trim == MatrixTrim.tail).toList(); + expect(tail.map((c) => c.id), [ + 'avi/convert-aac/plain/tail', + 'avi/convert-libmp3lame/plain/tail', + 'avi/convert-ac3/plain/tail', + 'avi/restore-convert-aac/plain/tail', + ]); + expect(tail.first.expectedPictures, 50); + expect(tail.first.trim.flashes, [55, 80]); + }); + + test('a restore of MP2 is checked against the float decode (task #58)', () { + MatrixCase find(String id) => cases.singleWhere((c) => c.id == id); + expect(find('mkv/restore/late-video/mid').referenceDecoder, 'mp2float'); + expect(find('mkv/convert-flac/late-video/mid').referenceDecoder, isNull); + expect(find('mkv/restore/plain/mid').referenceDecoder, isNull); + }); + test('every known failure names a real case', () { final ids = cases.map((c) => c.id).toSet(); for (final id in knownFailures.keys) { diff --git a/app/test/pipe_source_order_test.dart b/app/test/pipe_source_order_test.dart new file mode 100644 index 00000000..f498eb01 --- /dev/null +++ b/app/test/pipe_source_order_test.dart @@ -0,0 +1,111 @@ +// pipe_source's reader serves the frame it was asked for, whatever order +// VapourSynth asks in (task #58). +// +// vspipe requests a new frame each time any frame completes, so one early +// frame whose callback runs late lets the rest of the clip be read past it. +// Measured under load: frames 6 and 7 served after frame 99 was read, and the +// old reader, which kept only the 40 frames behind the last one read, served +// them as whatever frame it had read last (2-4 wrong outputs in 160 plain +// runs; docs/reports/task-58-output-conformance-findings). This drives +// `_PipeReader` directly, with a fake pipe of numbered frames, in that order, +// so it fails deterministically on the old reader. It needs the bundle's +// Python (vapoursynth imports), as plugin_format_contracts_test does. +import 'dart:convert'; +import 'dart:io'; + +import 'package:flutter_test/flutter_test.dart'; +import 'package:path/path.dart' as p; + +import 'support/worker_harness.dart'; + +const _script = r''' +import io, json, sys +sys.path.insert(0, sys.argv[1]) +import vapoursynth as vs +import pipe_source + +W, H, N = 16, 16, 100 +FRAME = W * H * 3 // 2 + + +def reader(window=40): + r = pipe_source._PipeReader(W, H, vs.YUV420P8, cache_frames=window) + r._pipe = io.BytesIO(b''.join(bytes([n]) * FRAME for n in range(N))) + return r + + +def tag(r, n): + try: + return r._frame_data(n)[0] + except vs.Error as e: + return 'raised' if 'left the cache' in str(e) else str(e) + + +out = {} + +# Run-ahead: 0..2, then 8..99 (to the end of the pipe), then 3..7. +r = reader() +order = [0, 1, 2] + list(range(8, N)) + list(range(3, 8)) +out['runahead'] = [[n, tag(r, n)] for n in order] + +# The ModifyFrame callback itself, for a late frame. +r = reader() +for n in [0, 1, 2] + list(range(8, N)): + tag(r, n) +blank = vs.core.std.BlankClip(width=W, height=H, format=vs.YUV420P8, length=1).get_frame(0) +f = r.fill_frame(5, blank) +out['callback'] = [f[p][0, 0] for p in range(3)] + +# A frame left behind the temporal window after everything was served. +r = reader() +for n in range(N): + tag(r, n) +out['behindWindow'] = tag(r, 10) +out['insideWindow'] = tag(r, 70) + +# Past the end of the stream: the last real frame (for VDecimate). +out['pastEnd'] = tag(r, N + 5) + +# Frames never asked for, past the memory cap: given up, and raising if +# they are asked for after all. +r = reader() +r._cap = 50 +for n in (0, 1, 2, 99): + tag(r, n) +out['abandoned'] = tag(r, 3) +out['keptNearEnd'] = tag(r, 60) + +print(json.dumps(out)) +'''; + +void main() { + setUpAll(() async { + await WorkerHarness.ensureReady(); + }); + + test('pipe_source serves each frame it was asked for, in any order (task #58)', () async { + final script = File(p.join(Directory.systemTemp.path, 'vb_pipe_order_$pid.py')); + await script.writeAsString(_script); + addTearDown(() { + if (script.existsSync()) script.deleteSync(); + }); + final r = await Process.run( + WorkerHarness.pythonPath, + [script.path, p.join(WorkerHarness.repoRoot, 'worker', 'templates')], + environment: WorkerHarness.workerEnv, + ); + expect(r.exitCode, 0, reason: '${r.stdout}\n${r.stderr}'); + final out = jsonDecode((r.stdout as String).trim().split('\n').last) as Map; + + for (final pair in (out['runahead'] as List).cast()) { + expect(pair[1], pair[0], reason: 'frame ${pair[0]} after a run-ahead to the end of the pipe'); + } + expect(out['callback'], [5, 5, 5], reason: 'fill_frame writes frame 5 into every plane'); + expect(out['behindWindow'], 'raised', + reason: 'a frame dropped behind the window raises; it never comes back as another picture'); + expect(out['insideWindow'], 70); + expect(out['pastEnd'], 99, reason: 'past the end of the stream: the last real frame'); + expect(out['abandoned'], 'raised', reason: 'a frame given up at the memory cap raises'); + expect(out['keptNearEnd'], 60); + }); +} diff --git a/app/test/support/av_sync.dart b/app/test/support/av_sync.dart index 74a846aa..da359568 100644 --- a/app/test/support/av_sync.dart +++ b/app/test/support/av_sync.dart @@ -27,9 +27,19 @@ Future<({List flashes, List beeps, double videoStart})> events( '-v', 'error', '-select_streams', 'v:0', '-show_entries', 'frame=pts_time', '-of', 'json', path, ]); - final pts = [ - for (final f in vf['frames'] as List) double.parse('${f['pts_time']}'), - ]; + // A program stream leaves a picture untimed now and then (a B picture + // whose packet has no PTS): it is one step after the picture before it. + final pts = []; + for (final f in vf['frames'] as List) { + final t = double.tryParse('${f['pts_time']}'); + if (t != null) { + pts.add(t); + } else if (pts.length >= 2) { + pts.add(pts.last + (pts.last - pts[pts.length - 2])); + } else { + pts.add(pts.isEmpty ? 0 : pts.last); + } + } final luma = await Process.run( WorkerHarness.ffmpegPath, ['-v', 'quiet', '-i', path, '-map', '0:v:0', '-fps_mode', 'passthrough', @@ -48,6 +58,15 @@ Future<({List flashes, List beeps, double videoStart})> events( if (sum / (n / 97).ceil() > 200) flashes.add(pts[i]); } + final beeps = await beepOnsets(path); + return (flashes: flashes, beeps: beeps, videoStart: pts.isEmpty ? double.nan : pts.first); +} + +/// The onsets of the first audio stream's beeps on the file's own timeline +/// (loud after 5 ms of quiet; the first 5 ms never count, since a copied +/// track can start inside a beep). Needs no video timestamps, so it also +/// reads a source whose pictures have none (task #58's source lags). +Future> beepOnsets(String path) async { final af = await WorkerHarness.ffprobeJson([ '-v', 'error', '-select_streams', 'a:0', '-show_entries', 'frame=pts_time,nb_samples:stream=sample_rate', '-of', 'json', path, @@ -85,7 +104,7 @@ Future<({List flashes, List beeps, double videoStart})> events( } pos += count; } - return (flashes: flashes, beeps: beeps, videoStart: pts.isEmpty ? double.nan : pts.first); + return beeps; } /// Each white frame has its beep: the first beep from 0.5 s before it to 1.5 s diff --git a/app/test/support/output_matrix.dart b/app/test/support/output_matrix.dart index b64eba52..107dc845 100644 --- a/app/test/support/output_matrix.dart +++ b/app/test/support/output_matrix.dart @@ -41,6 +41,16 @@ enum SourceClass { /// The source's sound, as ffprobe names it. final String audioCodec; + /// The decoder Audio Restoration's render reads this sound with, when it + /// is not FFmpeg's default: the float decoder for MPEG layers I and II, + /// whose fixed-point default rounds differently after a seek (mirrors + /// `audio_restoration::render_decoder`, task #58). + String? get renderDecoder => switch (audioCodec) { + 'mp1' => 'mp1float', + 'mp2' => 'mp2float', + _ => null, + }; + /// How far the picture starts after the sound, as the fixture was made. double get nominalVideoStart => this == lateVideo ? 1.0 : 0.0; } @@ -51,7 +61,15 @@ enum SourceClass { enum MatrixTrim { untrimmed('untrimmed', null, null), nearStart('near-start', 2, 51), - mid('mid', 40, 89); + mid('mid', 40, 89), + + /// To the source's last picture, where its sound ends with the window too. + /// Only in [tailCases]: a Convert into AVI reads its sound late by the + /// encoder's priming, and `-shortest` cut the last picture (task #58). + tail('tail', 50, 99); + + /// The trims every container × audio mode × source runs. + static const everyCase = [untrimmed, nearStart, mid]; const MatrixTrim(this.id, this.start, this.end); final String id; @@ -200,6 +218,11 @@ class MatrixCase { return codec == 'flac' || codec.startsWith('pcm_') ? LosslessCheck.exact : LosslessCheck.none; } + /// The decoder the source's sound is decoded with for [LosslessCheck.exact]: + /// a restored track is the render's decode ([SourceClass.renderDecoder]); + /// a Convert is FFmpeg's default decoder. Null for the default. + String? get referenceDecoder => restores ? source.renderDecoder : null; + /// Whether the AVI AAC block-size rule applies (task #41). bool get checksAacBlockAlign => container.timesAudioByBlockAlign && expectedAudioCodec == 'aac'; @@ -226,7 +249,7 @@ List matrixCases() { void add(ContainerFormat k, VideoCodec v, AudioKind a, [AudioCodec? c]) { for (final s in SourceClass.values) { if ((a == AudioKind.passthrough) && !k.copies(s.audioCodec)) continue; - for (final t in MatrixTrim.values) { + for (final t in MatrixTrim.everyCase) { cases.add(MatrixCase(container: k, video: v, audio: a, audioCodec: c, source: s, trim: t)); } } @@ -245,37 +268,25 @@ List matrixCases() { add(k, v, AudioKind.passthrough); add(k, v, AudioKind.restore); } - return cases; + return [...cases, ...tailCases]; +} + +/// Trims to the last picture: where an encoder priming meets a sound that +/// ends with the window (task #58). AVI, each Convert codec with a priming +/// (and Restore + Convert AAC), the plain source. +List get tailCases { + const k = ContainerFormat.avi; + final v = MatrixCase.defaultVideo(k); + MatrixCase c(AudioKind a, [AudioCodec? codec]) => MatrixCase( + container: k, video: v, audio: a, audioCodec: codec, source: SourceClass.plain, trim: MatrixTrim.tail); + return [ + for (final codec in [AudioCodec.aac, AudioCodec.mp3, AudioCodec.ac3]) c(AudioKind.convert, codec), + c(AudioKind.restoreConvert), + ]; } /// Cases that fail today, by id, with the task that owns the fault. They -/// still run, and fail if they pass, so the list can only shrink. -/// The task that owns what the matrix found (filed by task #49). Its fourth -/// finding, `pipe_source` serving an evicted frame as the last one, is -/// intermittent and so not listed: a different case or two fails each run. -const _t58 = 'task #58'; - -const knownFailures = { - // Restore + Convert AAC into AVI, trimmed: 49 pictures for 50. - 'avi/restore-convert-aac/plain/near-start': _t58, - 'avi/restore-convert-aac/plain/mid': _t58, - 'avi/restore-convert-aac/late-video/near-start': _t58, - 'avi/restore-convert-aac/late-video/mid': _t58, - 'avi/restore-convert-aac/delayed-avi/near-start': _t58, - 'avi/restore-convert-aac/delayed-avi/mid': _t58, - 'avi/restore-convert-aac/reset-ps/near-start': _t58, - 'avi/restore-convert-aac/reset-ps/mid': _t58, - // Restore of an MP2 source whose sound is seeked: off by up to 2^-15. - 'mkv/restore/late-video/near-start': _t58, - 'mkv/restore/late-video/mid': _t58, - 'avi/restore/late-video/near-start': _t58, - 'avi/restore/late-video/mid': _t58, - 'mov+proresProxy/restore/late-video/near-start': _t58, - 'mov+proresProxy/restore/late-video/mid': _t58, - 'mkv/restore/reset-ps/mid': _t58, - 'avi/restore/reset-ps/mid': _t58, - 'mov+proresProxy/restore/reset-ps/mid': _t58, - 'avi/restore/late-video/untrimmed': _t58, - // A delayed AVI's MP3 copied into AVI from frame 2: the click 31 ms late. - 'avi/passthrough/delayed-avi/near-start': _t58, -}; +/// still run, and fail if they pass, so the list can only shrink. Empty since +/// task #58 fixed what the matrix first found (docs/reports/ +/// task-58-output-conformance-findings). +const knownFailures = {}; diff --git a/docs/ENGINEERING_NOTES.md b/docs/ENGINEERING_NOTES.md index 00d5b69a..0b9f4244 100644 --- a/docs/ENGINEERING_NOTES.md +++ b/docs/ENGINEERING_NOTES.md @@ -5222,3 +5222,41 @@ for 48-px blocks. Ships in deps 1.18.0; the push gate's `fft3dfilter is deterministic` and the heavy `integration_fft3dfilter_determinism_test.dart` skip on bundles up to 1.17.x. Report: [docs/reports/task-62-fft3dfilter-determinism](reports/task-62-fft3dfilter-determinism/index.html). + + +### What the output conformance matrix found, fixed (task #58, 2026-11-17) + +Report: [docs/reports/task-58-output-conformance-findings](reports/task-58-output-conformance-findings/index.html). + +- **`pipe_source` served a dropped frame as another picture.** VapourSynth + runs the pipe clip's callbacks out of order (vspipe requests a frame when + any completes), so one late callback let the reader read the whole clip + past it: frames 6–7 served after 99 was read, on a plain job, 6 runs in + 320 under 16-way load. The reader kept 40 frames behind the last one read + and returned the last frame for anything older. It now keeps frames until + they are served (window measured from the oldest frame not yet served, a + 512 MB cap for frames never asked for), and raises on a dropped one: 0 in + 320 on the same load. +- **Restore + Convert AAC into AVI, trimmed: 49 pictures for 50.** Task + #38's priming shift reads every sound late by P; a sound ending with the + window (the restored track, or a source trimmed to its last picture) then + ended before the picture and `-shortest` dropped the last one. Convert + AAC/MP3/AC-3 trimmed to the end had it too. Such a trim now ends with + `-t` the window. +- **The restored track of a seeked MP2 was one 16-bit step off** in ~40% of + samples: FFmpeg's default MP2 decoder is fixed-point, with rounding error + carried between frames, so the samples depend on where decoding started. + The render reads MP1/MP2 with `mp2float`, exact from any seek. (Task + #49's "Convert of the same seek is exact" was the oracle's 16-bit + tolerance.) +- **The delayed AVI's 31 ms late click was the oracle's.** The fixture's + own click is 23 ms late (libmp3lame's priming, unmarked in AVI); the cut + adds 8 ms by its nearest-packet rule. The matrix now expects each click + at the source's own lag. +- **A reset whose sound ran 16 ms past its last picture joined 16 ms off + the grid** (125 fps, 493 pictures for 100): ffmpeg's discontinuity + correction takes its offset from whichever stream shows the reset first. + The scan now makes a `JoinPlan` and the copy applies it with `-copyts` and + `setts` per stream: the picture decides, every stream of a segment moves + alike, an overhanging sound packet is dropped. Streams that reset are cut + by packet index (MPEG audio from a split PES has no byte position). diff --git a/docs/reports/README.md b/docs/reports/README.md index e23c5c8c..36becd68 100644 --- a/docs/reports/README.md +++ b/docs/reports/README.md @@ -121,3 +121,4 @@ to it. | 2026-11-15 | [9-bit sources: Dehalo's YAHR failed](2026-11-9-bit-sources/index.html) | Built: 9-bit sources decode as 10-bit. CTMF has no 9-bit format, and havsfunc reaches it through `MinBlur(r>=2)` and directly: YAHR, FineDehalo contra, MCTemporalDenoise pfMode 2 and ContraSharpening r2 failed at every size, HQDeringmod and SMDegrain contrasharp at HD. The conversion is lossless (limited range an exact shift); the Noise Reduction CTMF block's own 9-bit guard is removed. `original` output of a 9-bit source is 10-bit (+1.2% FFV1). Windows/Linux by the nightly run only | | 2026-11-16 | [Task #49: one output capability table, and the output conformance matrix](task-49-output-conformance/index.html) | Built: `output_capabilities.json` read by worker and app; a 444-case generated heavy test. 425/446 pass; three deterministic faults (19 cases) and an intermittent `pipe_source` wrong-frame fault filed as task #58 | | 2026-11-17 | [Task #60: Bifrost v3.0 → v4.0](task-60-bifrost-v4/index.html) | Built: every platform takes Bifrost 4.0 from its PyPI wheel. v3.0's chroma past the last whole 4x4 block was uninitialised (2–3 results in 3 runs at width 98/718, 720x486); v4.0 gives one, identical to v3.0 inside whole blocks. Runs natively at 8–16 bits, luma bit-exact, 0.1% of chroma decisions differing from 8-bit at the threshold; the 8-bit copy is gone. 4:2:0 at 720x486 with `interlaced=True` is still refused (task #57) | +| 2026-11-17 | [Task #58: what the output conformance matrix found, fixed](task-58-output-conformance-findings/index.html) | Fixed all five: `pipe_source` keeps frames until served and raises on a dropped one (6/320 wrong runs under load → 0); primed AVI trims end with `-t`, not `-shortest` (49 → 50 pictures, Convert trimmed to the end too); MP1/MP2 rendered with the float decoder (exact from any seek); the reset join decided by the picture (`-copyts` + `setts`); the delayed AVI's late click was the oracle's (the source's own 23 ms). `knownFailures` empty | diff --git a/docs/reports/task-49-output-conformance/index.html b/docs/reports/task-49-output-conformance/index.html index d74ad60e..4636a935 100644 --- a/docs/reports/task-49-output-conformance/index.html +++ b/docs/reports/task-49-output-conformance/index.html @@ -95,6 +95,7 @@

Result

pipe_source returns an evicted frame as the last one: pictures 3–7 come out as picture 991–3no, varies A reset whose sound ends off the picture grid joins 16 ms late (125 fps, 493 pictures)1 testyes +

Correction (task #58, 2026-11-17). “Convert to FLAC/PCM of the same seek is exact” was the oracle's tolerance: it differs in the same samples by one 16-bit step and passed because Convert writes 16-bit. The delayed AVI's 31 ms was the source's own 23 ms lag plus an 8 ms cut. All five findings are fixed: task #58's report.

The deterministic 19 are knownFailures: they run, and fail the suite if they start passing. The pipe_source fault cannot be listed, so the matrix fails intermittently until #58 fixes it; it writes wrong pictures into a user's file, so that is the right pressure.

Limits: one platform (macOS arm64), one picture size and rate (25 fps, so a frame is exactly 1,920 samples at 48 kHz and every trim lands on a sample), one content. “Within one audio packet” is the bound the task asked for, not a measure of how close each case is.

diff --git a/docs/reports/task-58-output-conformance-findings/index.html b/docs/reports/task-58-output-conformance-findings/index.html new file mode 100644 index 00000000..732f363a --- /dev/null +++ b/docs/reports/task-58-output-conformance-findings/index.html @@ -0,0 +1,148 @@ + + + + + +Conformance findings + + + +
+

Task #58: what the output conformance matrix found, fixed

+

2026-11-17 · macOS arm64, deps 1.16.0 (bundled FFmpeg 9.0.2) · task #58, filed by task #49 (its report)

+ +
+

Five findings, five causes, all fixed. Two were worker bugs (a dropped frame served as another picture; a primed AVI trim cut short), one a decoder choice in the Audio Restoration render, one the reset join, and one the matrix's own oracle. knownFailures is empty. Each fix was checked against the pre-fix worker, built from dc9e36c: every case it fixes fails there.

+
+ + + + + + + + +
#findingcausefix
4pipe_source served the wrong pictures, intermittentlyVapourSynth runs the pipe clip's callbacks out of order; the reader dropped a frame not yet served and returned the last one read in its placeKeep every frame until it is served; raise on a dropped one
1Restore + Convert AAC into AVI, trimmed: 49 pictures for 50-shortest with a sound read late by the encoder's priming-t the window instead, under a priming
2Restore of a seeked MP2: one 16-bit step offFFmpeg's default MP2 decoder is fixed-point, with rounding state carried between framesThe render reads MP1/MP2 with the float decoder
3Delayed AVI's MP3 copied into AVI from frame 2: click 31 ms lateThe source's own click is 23 ms late (libmp3lame's priming, unmarked in AVI); the cut adds 8 ms, by designOracle: expect each click at the source's own lag
5A reset whose sound ends off the picture grid joins 16 ms lateffmpeg's correction takes its offset from the stream that shows the reset first: here the soundThe picture decides the join (-copyts + setts per stream)
+ +

4. pipe_source served a frame it had dropped as another picture

+

The source is read from a pipe, forward only, and pipe_source kept the 40 frames behind the last one read. A frame asked for after it left that window came back as whatever frame had been read last, silently. The matrix saw it as pictures 3–7 of an output holding source picture 99.

+

Reproduction

+

Nothing but the pipe: conf_plain.mp4 decoded to raw frames, piped to vspipe running a script that is only create_pipe_clip(…, cache_frames=40), each output compared by framemd5 with a direct decode, 16 runs at once (pipe_source_stress.sh):

+ + + + + + +
readerrunswrong runsruns that raised
before (dc9e36c), first sample3206–
before (dc9e36c), second sample3208–
after, first sample32000
after, second sample32000
+

In the failing runs the wrong frames were output 1–7, 4–7, 5–7 or 6–7, each holding whatever the reader had read last (source 53, 69, 74, 99), not always 99. An instrumented reader (not kept) showed the cause: in both failing runs out of 160, frames 6 and 7 reached the callback after the reader had read to frame 100 (92–93 frames behind it); passing runs never fell more than 38 behind. vspipe requests a new frame each time any frame completes, so one early frame whose callback is late lets the whole clip go past it. Sleeping inside the callback for frame 3 did not reproduce it (the callbacks run one at a time, so a sleep there stalls everything): the late frame is one whose callback hasn't started.

+

Fix

+

The reader keeps every frame until it has been served and is cache_frames behind the oldest frame not yet served (not merely behind the last frame read), with a cap for frames the script never asks for (a decimation, a preview's single frame): 512 MB beyond the window, at least 16 frames. A frame asked for after it was dropped raises a VapourSynth error naming it, so a pass whose radius is under-declared now fails the job instead of writing wrong frames. app/test/pipe_source_order_test.dart (push gate) drives the reader in the measured order with a fake pipe, and checks the window, the cap and the end of stream.

+

Limits: the failure rate depends on load, so the counts are one machine's on one afternoon; the deterministic test is the guard.

+ +

1. Restore + Convert AAC into AVI, trimmed: one picture short

+

Task #38 reads every sound input of a Convert into AVI late by the encoder's priming P (AAC: 1024 samples), so that the priming the encoder puts in front lands where the sound was. The restored track is exactly the window long, so read from P it ends P before the picture on ffmpeg's timeline, and -shortest dropped the last picture. The same arguments with plain ffmpeg (shortest_priming.sh, 50 pictures, a 2.000 s WAV):

+
+-ss P on the exact WAV, -shortest (before)    pictures 49, audio 2.005333 s
+no priming seek, -shortest                    pictures 50, audio 2.026667 s
+WAV P longer, -shortest                       pictures 50, audio 2.026667 s
+no -shortest                                  pictures 50, audio 2.005333 s
+-t 2 instead of -shortest (after)             pictures 50, audio 2.005333 s
+
+

Not only Restore. A Convert into AVI trimmed to the source's last picture reads a sound that also ends with the window. The matrix had no such trim; four new cases (avi/convert-{aac,libmp3lame,ac3}/plain/tail, avi/restore-convert-aac/plain/tail, frames 50..99) all store 49 pictures on the pre-fix worker.

+

Fix: under a priming, a trimmed encode ends with -t the window's length in place of -shortest (pipeline_executor::encode_end_args, both builders). Every other job keeps its arguments.

+ +

2. Audio Restoration of a seeked MP2: one 16-bit step off

+

FFmpeg's default decoder for MPEG layers I and II is the fixed-point one (mp2, s16p output), listed ahead of mp2float; layer III defaults to mp3float. The fixed-point synthesis rounds with an error it carries from frame to frame, so a decode that starts at a seek rounds differently from one that starts at 0. No filter is involved (mp2_seek_decoder.sh, conf_late_video.ts from 1.090 s, the render's chain, against a decode of the whole stream):

+
+default_near         vs full_fixed.f32   offset 52321: 41377 of 96000 samples differ, max 1.000 LSB16
+default_near_noaf    vs full_fixed.f32   offset 52321: 41377 of 96000 samples differ, max 1.000 LSB16
+default_near         vs full_float.f32   offset 52321: 95996 of 96000 samples differ, max 1.047 LSB16
+float_near           vs full_float.f32   offset 52321: 0 of 96000 samples differ, max 0.000 LSB16
+float_mid            vs full_float.f32   offset 125281: 0 of 96000 samples differ, max 0.000 LSB16
+
+

Correction to task #49's report. It said Convert to FLAC/PCM of the same seeked MP2 was exact. It differs in the same samples; it passed because the oracle allows one step at the output's depth, and Convert writes 16-bit. The restored track is 24-bit, so the same step failed it.

+

Fix: the render reads MP1/MP2 with mp1float/mp2float (audio_restoration::render_decoder), which is also the more precise decode for an f32 render. Convert and the analysis keep FFmpeg's default. The oracle compares a restore against the render's decoder (MatrixCase.referenceDecoder).

+ +

3. Delayed AVI copied into AVI from frame 2: the oracle, not the copy

+

Where each source's first click sits after its flash, on its own timeline (click_lags.sh):

+
+conf_plain.mp4       click 0.2001 s, flash 0.2000 s: the click +0.1 ms after it
+conf_late_video.ts   click 2.6101 s, flash 2.6100 s: the click +0.1 ms after it
+conf_delayed.avi     click 0.2231 s, flash 0.2000 s: the click +23.1 ms after it
+conf_reset.mpg       click 0.7401 s, flash 0.7400 s: the click +0.1 ms after it
+
+

The delayed AVI's MP3 was written by libmp3lame into AVI, which cannot mark the encoder's start-up (1,105 samples), so every player plays it 23 ms late: real AVI captures with MP3 look like this. Its packets are 24 ms; the in-point 0.080 s falls in the packet at 0.072, which the copy keeps (task #24's nearest-packet cut), 8 ms early, so the content plays 8 ms later: 23 + 8 = 31 ms, as measured. The mid trim's in-point (1.600) keeps the packet at 1.608: 23 − 8 = 15 ms, which passed. The worker keeps the source's timing within half a packet, as designed.

+

Fix (test only): the matrix measures each source's own lag per flash and expects each output click there, within one packet as before.

+ +

5. A reset whose sound ends off the picture grid joined 16 ms late

+

conf_reset_offgrid.mpg resets at picture 50 (2.0 s = 83.3 MP2 frames), so its first part's sound runs 16 ms past its last picture, as a DVD cell's may. ffmpeg's discontinuity correction (the joined copy, without -copyts) takes one offset per segment from whichever stream shows the jump first, using that stream's next timestamp: here the sound's. The worker now makes a JoinPlan from the scan (each packet's DTS, duration and byte position): each segment moves so its first picture follows the last by one frame, every stream alike, applied by setts per stream under -copyts, with a sound packet overhanging the next segment by more than half of itself dropped (reset_join.sh):

+
+ffmpeg conf_reset              rate 25/1    pictures 100, picture steps [0.04], picture 50 at 2.0000 s; sound frames 167, irregular sound steps []
+worker conf_reset              rate 25/1    pictures 100, picture steps [0.04], picture 50 at 2.0000 s; sound frames 167, irregular sound steps []
+ffmpeg conf_reset_offgrid      rate 125/1   pictures 100, picture steps [0.04, 0.056], picture 50 at 2.0160 s; sound frames 168, irregular sound steps []
+worker conf_reset_offgrid      rate 25/1    pictures 100, picture steps [0.04], picture 50 at 2.0000 s; sound frames 167, irregular sound steps [(2.006, 0.032)]
+
+

On the grid the two copies time every picture and sound frame alike. Off it, the picture now stays on the grid, and the sound has one 8 ms gap where its overhanging packet went (the job log says so). The sound after the join is within half a packet either way, which a copy can't improve on.

+

Two traps on the way, both in the plan: the MPEG audio frames the parser splits out of one PES carry no byte position, so a stream that resets is cut by its packet index (N), and only streams without a reset of their own (subtitles) by position (POS); and a packet with no DTS must keep none (if(eq(DTS,NOPTS),NOPTS,…)), or the copy's DTS stop increasing. Without a plan (no durations, streams resetting a different number of times) the copy is exactly as before. The off-grid matrix test is an ordinary test now; integration_seek_alignment_test.dart's seek_reset_* fixtures and the DVD import test pass through the new copy.

+

Limits: two synthetic program streams; no DVD with angles or a broadcast TS with a reset was available. Task #55 (reset scan) changes the same scan and should rebase on this.

+ +

The matrix

+

448 cases now (444 + the four tail trims), plus the off-grid join test and the copy-refusal test: 450 tests. knownFailures is empty.

+ + + + + +
full runpassedfailed
24500
34500
44500
+

Run 1 is void. While it ran I swapped the pre-fix worker binary in and out of worker/target/debug (and its templates into the templates/ folder the worker reads first) to check an unrelated Unblend failure. Overwriting an executed binary in place makes macOS kill it (exit −9) until it is rewritten, so 291 of its cases failed with “Worker exited with code −9”; a fourth run was added. Lesson for the next person: swap a binary with mv, not cp over it, and not while a suite runs.

+

The heavy files that read back in time through pipe_source pass (integration_stabilize_test, integration_ivtc_preview_test, integration_frame_mapping_test without gpu, integration_unblend_test but one), and none raised “left the cache”; so do integration_seek_alignment_test (87) and integration_video_start_offset_test (80). The Unblend case “a 10-bit copy of the source gives the 8-bit result” fails here (13.1 levels) identically with the pre-fix worker and templates, so it is not from this task: filed as task #64. The local bundle is deps 1.16.0 (the app pins 1.17.0-rc2).

+

Output: matrix_macos-arm64.txt. Run with task-49-output-conformance/scripts/run_matrix.sh.

+ +

Not tested

+
    +
  • Windows and Linux: nightly runs the matrix there. The decoder order (fixed MP2 first) is FFmpeg's own, the same on every build of a version, and the render names its decoder anyway.
  • +
  • The cap of pipe_source on a long decimating job at 4K (the cap there is the window plus 16 frames); a radius too small for a pass now raises, which the heavy suite would show: see the matrix and the runs below.
  • +
+
+ + diff --git a/docs/reports/task-58-output-conformance-findings/matrix_macos-arm64.txt b/docs/reports/task-58-output-conformance-findings/matrix_macos-arm64.txt new file mode 100644 index 00000000..1c213dce --- /dev/null +++ b/docs/reports/task-58-output-conformance-findings/matrix_macos-arm64.txt @@ -0,0 +1,10 @@ +# task #58, macOS arm64, deps 1.16.0 (FFmpeg 9.0.2), 2026-11-17: full runs 2, 3, 4 (run 1 void, see the report) +== run 2 +10:34 +450: All tests passed! +failures (case :: first failed check): +== run 3 +09:08 +450: All tests passed! +failures (case :: first failed check): +== run 4 +08:29 +450: All tests passed! +failures (case :: first failed check): diff --git a/docs/reports/task-58-output-conformance-findings/scripts/click_lags.sh b/docs/reports/task-58-output-conformance-findings/scripts/click_lags.sh new file mode 100755 index 00000000..982b7cb2 --- /dev/null +++ b/docs/reports/task-58-output-conformance-findings/scripts/click_lags.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# Finding 3: where each source's click sits after its flash (picture 5, +# 0.2 s after the picture starts), on the source's own timeline: the sound +# decoded from its start against the container start plus the picture's. +set -euo pipefail +source "$(dirname "$0")/common.sh" +for f in conf_plain.mp4 conf_late_video.ts conf_delayed.avi conf_reset.mpg; do + fs=$("$FFPROBE" -v error -show_entries format=start_time -of csv=p=0 "$RES/$f") + vs=$("$FFPROBE" -v error -select_streams v:0 -show_entries stream=start_time -of csv=p=0 "$RES/$f" | head -n1 | tr -d ,) + as=$("$FFPROBE" -v error -select_streams a:0 -show_entries stream=start_time -of csv=p=0 "$RES/$f" | head -n1 | tr -d ,) + "$FFMPEG" -v error -i "$RES/$f" -map 0:a:0 -ac 1 -f f32le - | python3 -c " +import struct, sys +b = sys.stdin.buffer.read(); a = struct.unpack('<%df' % (len(b) // 4), b) +i = next(i for i, v in enumerate(a) if abs(v) > 0.3) +click = $as + i / 48000; flash = $vs + 0.2 +print('%-20s click %.4f s, flash %.4f s: the click %+.1f ms after it' % ('$f', click, flash, (click - flash) * 1000))" +done +echo "conf_delayed.avi's MP3 packets (24 ms each; the near-start in-point 0.080 s keeps the packet at 0.072):" +"$FFPROBE" -v error -select_streams a:0 -show_entries packet=pts_time,duration_time -of csv=p=0 "$RES/conf_delayed.avi" | sed -n 3,6p diff --git a/docs/reports/task-58-output-conformance-findings/scripts/common.sh b/docs/reports/task-58-output-conformance-findings/scripts/common.sh new file mode 100755 index 00000000..04a930aa --- /dev/null +++ b/docs/reports/task-58-output-conformance-findings/scripts/common.sh @@ -0,0 +1,12 @@ +# Shared by the task #58 scripts: the repo, the deps bundle, a scratch dir. +# VAPOURBOX_DEPS_DIR (deps/) as for every heavy test, else the +# repo's own deps/macos-arm64; SCRATCH, else a new temporary directory. +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/../../../.." && pwd)" +DEPS="${VAPOURBOX_DEPS_DIR:-$REPO/deps/macos-arm64}" +FFMPEG="$DEPS/ffmpeg/ffmpeg" +FFPROBE="$DEPS/ffmpeg/ffprobe" +VSPIPE="$DEPS/vapoursynth/vspipe" +RES="$REPO/Tests/TestResources" +SCRATCH="${SCRATCH:-$(mktemp -d)}" +mkdir -p "$SCRATCH" diff --git a/docs/reports/task-58-output-conformance-findings/scripts/mp2_seek_decoder.sh b/docs/reports/task-58-output-conformance-findings/scripts/mp2_seek_decoder.sh new file mode 100755 index 00000000..f665a9c1 --- /dev/null +++ b/docs/reports/task-58-output-conformance-findings/scripts/mp2_seek_decoder.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Finding 2: Audio Restoration's render of conf_late_video.ts's MP2 from a +# seek, against a decode of the whole stream, with FFmpeg's default MP2 +# decoder (fixed-point `mp2`) and with `mp2float`. 1 LSB16 = 2^-15. +set -euo pipefail +source "$(dirname "$0")/common.sh" +cd "$SCRATCH" +TS="$RES/conf_late_video.ts" +"$FFMPEG" -v error -y -c:a mp2 -i "$TS" -map 0:a:0 -f f32le full_fixed.f32 +"$FFMPEG" -v error -y -c:a mp2float -i "$TS" -map 0:a:0 -f f32le full_float.f32 +CHAIN="aresample=first_pts=0,apad=whole_len=96000,atrim=end_sample=96000" +render() { # name, seek, decoder option(s), -af chain + "$FFMPEG" -v error -y $3 -ss "$2" -t 2.6 -i "$TS" -map 0:a:0 ${4:+-af "$4"} -c:a pcm_f32le -f f32le "$1.f32" +} +render default_near 1.090022 "" "$CHAIN" +render default_near_noaf 1.090022 "" "" +render float_near 1.090022 "-c:a mp2float" "$CHAIN" +render float_mid 2.610022 "-c:a mp2float" "$CHAIN" +python3 - <<'PY' +import struct +def load(p): + b = open(p, 'rb').read(); return struct.unpack('<%df' % (len(b) // 4), b) +def cmp(name, ref, near): + a, r = load(name + '.f32')[:96000], load(ref) + best = None + for off in range(near - 3, near + 4): + d = [abs(a[k] - r[k + off]) for k in range(min(len(a), len(r) - off))] + cand = (max(d), sum(x > 1e-9 for x in d), off, len(d)) + best = cand if best is None or cand[0] < best[0] else best + mx, nz, off, n = best + print(f'{name:20s} vs {ref:16s} offset {off}: {nz} of {n} samples differ, max {mx * 32768:.3f} LSB16') +cmp('default_near', 'full_fixed.f32', 52321) +cmp('default_near_noaf', 'full_fixed.f32', 52321) +cmp('default_near', 'full_float.f32', 52321) +cmp('float_near', 'full_float.f32', 52321) +cmp('float_mid', 'full_float.f32', 125281) +PY +"$FFMPEG" -v debug -ss 1.09 -t 0.1 -i "$TS" -map 0:a:0 -f null - 2>&1 | grep -m1 'samplefmt' | sed 's/^.*\(samplefmt:[a-z0-9]*\).*$/default decoder output: \1 (s16p: the fixed-point mp2)/' diff --git a/docs/reports/task-58-output-conformance-findings/scripts/pipe_source_stress.sh b/docs/reports/task-58-output-conformance-findings/scripts/pipe_source_stress.sh new file mode 100755 index 00000000..eb5b6a0d --- /dev/null +++ b/docs/reports/task-58-output-conformance-findings/scripts/pipe_source_stress.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# Finding 4: pipe_source serving a frame it has dropped as another picture. +# A plain pipe clip (no pass at all) of conf_plain.mp4's 100 pictures, run +# RUNS times, PAR at once, each output compared by framemd5 with a direct +# decode. Prints the wrong runs and which source picture each wrong frame +# holds. Usage: +# pipe_source_stress.sh [git-rev] # pipe_source.py at that revision +# # (default: the working tree's) +# RUNS (160) and PAR (16) from the environment. macOS/Linux (vspipe wrapper). +set -euo pipefail +source "$(dirname "$0")/common.sh" +RUNS="${RUNS:-160}"; PAR="${PAR:-16}" +src="$SCRATCH/pipe_source_${1:-worktree}" +mkdir -p "$src" "$src/runs" +if [ -n "${1:-}" ]; then + git -C "$REPO" show "$1:worker/templates/pipe_source.py" > "$src/pipe_source.py" +else + cp "$REPO/worker/templates/pipe_source.py" "$src/pipe_source.py" +fi +cat > "$src/plain.vpy" </dev/null \ + | "$VSPIPE" -c y4m "$src/plain.vpy" - 2>"$src/runs/$1.err" \ + | "$FFMPEG" -v error -y -f yuv4mpegpipe -i - -f framemd5 "$src/runs/$1.md5" || true +} +export -f one; export FFMPEG VSPIPE RES src +seq 1 "$RUNS" | xargs -P "$PAR" -I{} bash -c 'one {}' +python3 - "$src" "$RUNS" <<'PY' +import glob, sys +d, runs = sys.argv[1], int(sys.argv[2]) +def md5s(p): return [l.split(',')[-1].strip() for l in open(p) if not l.startswith('#')] +ref = md5s(d + '/ref.md5'); bad = 0; raised = 0 +for i in range(1, runs + 1): + out = md5s(f'{d}/runs/{i}.md5') if glob.glob(f'{d}/runs/{i}.md5') else [] + err = open(f'{d}/runs/{i}.err').read() + if 'left the cache' in err: raised += 1 + wrong = [(k, ref.index(o) if o in ref else None) for k, o in enumerate(out) if k >= len(ref) or o != ref[k]] + if wrong or len(out) != len(ref): + bad += 1 + print(f'run {i}: {len(out)} frames; wrong (output frame, source picture it holds): {wrong[:8]}') +print(f'wrong runs: {bad} of {runs}; runs that raised: {raised}') +PY diff --git a/docs/reports/task-58-output-conformance-findings/scripts/reset_join.sh b/docs/reports/task-58-output-conformance-findings/scripts/reset_join.sh new file mode 100755 index 00000000..5296ccf1 --- /dev/null +++ b/docs/reports/task-58-output-conformance-findings/scripts/reset_join.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Finding 5: the joined copy of the two reset fixtures, made by ffmpeg's own +# discontinuity correction (the copy before task #58) and by the worker's +# --make-continuous (the picture decides the join). For each: the video +# rate ffprobe reports, the pictures, picture 50's time after picture 0, and +# the sound's irregular steps. WORKER is a vapourbox-worker binary +# (default: worker/target/debug's; a debug worker finds deps/ above it). +set -euo pipefail +source "$(dirname "$0")/common.sh" +WORKER="${WORKER:-$REPO/worker/target/debug/vapourbox-worker}" +for f in conf_reset conf_reset_offgrid; do + "$FFMPEG" -hide_banner -nostdin -y -v error -i "$RES/$f.mpg" -map 0:v -map '0:a?' -c copy -f vob "$SCRATCH/ffmpeg_$f.mpg" + "$WORKER" --make-continuous "$RES/$f.mpg" --output "$SCRATCH/worker_$f.mpg" | grep -o '"message":"[^"]*joining[^"]*"' || true + for kind in ffmpeg worker; do + python3 - "$SCRATCH/${kind}_$f.mpg" "$FFPROBE" "$kind $f" <<'PY' +import json, subprocess, sys +f, P, name = sys.argv[1:4] +def times(sel): + j = json.loads(subprocess.run([P, '-v', 'error', '-select_streams', sel, '-show_entries', + 'frame=best_effort_timestamp_time', '-of', 'json', f], capture_output=True, text=True).stdout) + return [float(x['best_effort_timestamp_time']) for x in j['frames'] if 'best_effort_timestamp_time' in x], len(j['frames']) +rate = subprocess.run([P, '-v', 'error', '-select_streams', 'v:0', '-show_entries', 'stream=r_frame_rate', + '-of', 'csv=p=0', f], capture_output=True, text=True).stdout.strip().strip(',') +(v, nv), (a, na) = times('v:0'), times('a:0') +steps = sorted(set(round(v[i + 1] - v[i], 4) for i in range(len(v) - 1))) +odd = [(round(a[i] - v[0], 4), round(a[i + 1] - a[i], 4)) for i in range(len(a) - 1) if abs(a[i + 1] - a[i] - 0.024) > 1e-4] +print(f'{name:30s} rate {rate:7s} pictures {nv}, picture steps {steps}, picture 50 at {v[50] - v[0]:.4f} s; ' + f'sound frames {na}, irregular sound steps (at, step) {odd}') +PY + done +done diff --git a/docs/reports/task-58-output-conformance-findings/scripts/shortest_priming.sh b/docs/reports/task-58-output-conformance-findings/scripts/shortest_priming.sh new file mode 100755 index 00000000..c3f31b69 --- /dev/null +++ b/docs/reports/task-58-output-conformance-findings/scripts/shortest_priming.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# Finding 1: a 50-picture trim into AVI whose sound (a WAV exactly the +# window long, as Audio Restoration renders it) is read late by AAC's +# priming. -shortest drops the last picture; -t the window keeps it. +set -euo pipefail +source "$(dirname "$0")/common.sh" +cd "$SCRATCH" +"$FFMPEG" -v error -y -f lavfi -i testsrc2=size=160x120:rate=25 -frames:v 50 -pix_fmt yuv420p v50.y4m +"$FFMPEG" -v error -y -f lavfi -i sine=f=200:sample_rate=48000 -t 2.0 -c:a pcm_f32le w.wav +"$FFMPEG" -v error -y -f lavfi -i sine=f=200:sample_rate=48000 -t 2.021333 -c:a pcm_f32le wp.wav +P=0.021334 # 1024 / 48000, rounded up +row() { + local name="$1"; shift + "$FFMPEG" -v error -y -f yuv4mpegpipe -i v50.y4m "$@" -c:v ffv1 -c:a aac out.avi + printf '%-45s pictures %s, audio %s s\n' "$name" \ + "$("$FFPROBE" -v error -select_streams v:0 -count_frames -show_entries stream=nb_read_frames -of csv=p=0 out.avi)" \ + "$("$FFPROBE" -v error -select_streams a:0 -show_entries stream=duration -of csv=p=0 out.avi)" +} +row "-ss P on the exact WAV, -shortest (before)" -ss $P -i w.wav -map 0:v -map 1:a -avoid_negative_ts disabled -shortest +row "no priming seek, -shortest" -i w.wav -map 0:v -map 1:a -shortest +row "WAV P longer, -shortest" -ss $P -i wp.wav -map 0:v -map 1:a -avoid_negative_ts disabled -shortest +row "no -shortest" -ss $P -i w.wav -map 0:v -map 1:a -avoid_negative_ts disabled +row "-t 2 instead of -shortest (after)" -ss $P -i w.wav -map 0:v -map 1:a -avoid_negative_ts disabled -t 2 diff --git a/worker/src/audio_restoration.rs b/worker/src/audio_restoration.rs index abc3953e..e507d12a 100644 --- a/worker/src/audio_restoration.rs +++ b/worker/src/audio_restoration.rs @@ -132,6 +132,25 @@ pub struct SourceAudio { pub sample_rate: u32, /// The restored stream's duration in seconds, when stated or measured. pub duration: Option, + /// The restored stream's codec, as ffprobe names it (`codec_name`). + pub codec: Option, +} + +/// The decoder the render reads a codec with, when it is not FFmpeg's +/// default: MPEG audio layers I and II decode by default with the +/// fixed-point decoders (`mp1`, `mp2`), which round each sample with an error +/// carried from frame to frame, so a decode that starts at a seek differs from +/// one that starts at 0 in up to 40% of samples by one 16-bit step. The float +/// decoders carry no such state: from any seek they give the full decode's +/// samples exactly, which is what makes the restored track the source's sound +/// sample for sample (task #58). Layer III's default (`mp3float`) and AC-3's +/// (`ac3`) are already float. Convert and the analysis keep the default. +pub fn render_decoder(codec: &str) -> Option<&'static str> { + match codec { + "mp1" => Some("mp1float"), + "mp2" => Some("mp2float"), + _ => None, + } } /// The span of the output on the source's timeline: where the picture starts @@ -391,6 +410,9 @@ pub struct AudioPlan { pub read_skip: bool, /// How much to read (`-t`), when the window's length is known. pub read_duration: Option, + /// The decoder the stream is read with, when not FFmpeg's default + /// ([`render_decoder`]). + pub decoder: Option<&'static str>, /// The first ffmpeg run's filters, in order. pub restore_filters: Vec, /// DeepFilterNet's attenuation limit, when it runs. @@ -567,6 +589,7 @@ impl AudioPlan { read_start, read_skip, read_duration, + decoder: source.codec.as_deref().and_then(render_decoder), restore_filters: f, deep_filter, loudness_target, @@ -594,6 +617,9 @@ impl AudioPlan { if let Some(t) = self.read_duration { args.extend(["-t".into(), format!("{:.6}", t + head)]); } + if let Some(d) = self.decoder { + args.extend([format!("-c:a:{}", self.stream), d.to_string()]); + } args.extend(["-i".into(), source.to_string()]); args.extend(["-map".into(), format!("0:a:{}", self.stream)]); args.extend(["-af".into(), self.restore_filters.join(",")]); @@ -1677,7 +1703,7 @@ impl AudioRestorer { let ffprobe = self.deps.ffprobe_path()?; let out = Command::new(&ffprobe) .args(["-v", "error", "-select_streams", "a", "-show_entries", - "stream=channels,sample_rate,duration:stream_tags=DURATION", "-of", "json", input]) + "stream=codec_name,channels,sample_rate,duration:stream_tags=DURATION", "-of", "json", input]) .envs(self.deps.build_environment()) .output() .context("Failed to run ffprobe on the audio")?; @@ -1882,6 +1908,7 @@ pub fn parse_source_probe(json: &str, stream: u32) -> Option { channels: int("channels").unwrap_or(2), sample_rate: int("sample_rate").unwrap_or(48_000), duration, + codec: s.get("codec_name").and_then(|c| c.as_str()).map(str::to_string), }) } @@ -1923,7 +1950,7 @@ mod tests { } fn stereo48() -> SourceAudio { - SourceAudio { streams: 1, channels: 2, sample_rate: 48_000, duration: Some(60.0) } + SourceAudio { streams: 1, channels: 2, sample_rate: 48_000, duration: Some(60.0), codec: None } } fn window(start: f64, d: f64) -> OutputWindow { @@ -2271,12 +2298,32 @@ mod tests { assert!(parse_loudnorm_json("no json").is_none()); } + /// Task 58: MP1/MP2 are rendered with the float decoders, which give the + /// same samples from any seek; every other codec keeps its arguments. + #[test] + fn the_render_reads_mpeg_layers_1_and_2_with_the_float_decoder() { + assert_eq!(render_decoder("mp2"), Some("mp2float")); + assert_eq!(render_decoder("mp1"), Some("mp1float")); + for c in ["mp3", "aac", "ac3", "pcm_s16le", "flac", "opus"] { + assert_eq!(render_decoder(c), None, "{c}"); + } + let p = AudioRestorationParameters { enabled: true, stream_index: 1, ..Default::default() }; + let src = |c: &str| SourceAudio { streams: 2, codec: Some(c.to_string()), ..stereo48() }; + let mp2 = AudioPlan::new(&p, &src("mp2"), window(1.5, 2.0), &Analysis::default(), None); + let args = mp2.restore_args("in.ts", Path::new("a.wav")).join(" "); + assert!(args.contains(" -t 3.000000 -c:a:1 mp2float -i in.ts"), "{args}"); + let aac = AudioPlan::new(&p, &src("aac"), window(1.5, 2.0), &Analysis::default(), None); + let plain = AudioPlan::new(&p, &SourceAudio { streams: 2, ..stereo48() }, window(1.5, 2.0), &Analysis::default(), None); + assert_eq!(aac.restore_args("in.ts", Path::new("a.wav")), plain.restore_args("in.ts", Path::new("a.wav"))); + } + #[test] fn source_probe_reads_streams_and_durations() { - let json = r#"{"streams":[{"channels":2,"sample_rate":"48000","duration":"12.5"}, + let json = r#"{"streams":[{"codec_name":"mp2","channels":2,"sample_rate":"48000","duration":"12.5"}, {"channels":6,"sample_rate":"44100","tags":{"DURATION":"01:00:01.250000000"}}]}"#; let a = parse_source_probe(json, 0).unwrap(); assert_eq!((a.streams, a.channels, a.sample_rate, a.duration), (2, 2, 48_000, Some(12.5))); + assert_eq!(a.codec.as_deref(), Some("mp2")); let b = parse_source_probe(json, 1).unwrap(); assert_eq!((b.channels, b.sample_rate, b.duration), (6, 44_100, Some(3601.25))); assert!(parse_source_probe(json, 2).is_none()); @@ -2317,7 +2364,7 @@ mod tests { // The plan reads from the window's start, with nothing else changed. let plan = AudioPlan::new( &job.effective_pipeline().audio, - &SourceAudio { streams: 1, channels: 2, sample_rate: 48_000, duration: Some(6.1) }, + &SourceAudio { streams: 1, channels: 2, sample_rate: 48_000, duration: Some(6.1), codec: None }, w, &Analysis::default(), None, @@ -2792,7 +2839,7 @@ mod tests { /// delay, or further in, reads with -ss exactly as before. #[test] fn a_delayed_avi_near_its_start_drops_the_head_in_the_chain() { - let src = SourceAudio { streams: 1, channels: 1, sample_rate: 32_000, duration: Some(3.0) }; + let src = SourceAudio { streams: 1, channels: 1, sample_rate: 32_000, duration: Some(3.0), codec: None }; let start = 2.0 * 1001.0 / 30000.0; let delayed = OutputWindow { start, duration: Some(2.0), decoder_delay: 2 }; let plan = AudioPlan::new(&on(), &src, delayed, &Analysis::default(), None); diff --git a/worker/src/pipeline_executor.rs b/worker/src/pipeline_executor.rs index f18505aa..a292b9cf 100644 --- a/worker/src/pipeline_executor.rs +++ b/worker/src/pipeline_executor.rs @@ -1399,13 +1399,9 @@ impl PipelineExecutor { } args.extend(priming); - // When trimming, stop when the shortest stream (the trimmed video) ends. - // Without this, ffmpeg reads the full audio track even if only a small - // video segment was exported. - let is_trimmed = job.start_frame.is_some() || job.end_frame.is_some(); - if is_trimmed { - args.push("-shortest".to_string()); - } + // When trimming, stop with the trimmed video: `-shortest`, or the + // window's length under an encoder priming (see `encode_end_args`). + args.extend(encode_end_args(job, self.encoder_priming.as_ref())); // Embed VapourBox version in output metadata, preserving any existing // comment: the global tags come from input 2, the metadata file. @@ -2036,6 +2032,27 @@ pub fn source_audio_input_seek(job: &VideoJob, priming: Option<&EncoderPriming>) } } +/// How a trimmed encode stops with its picture: `-shortest`, so ffmpeg does +/// not read the rest of the source's sound. Under an encoder priming +/// ([`EncoderPriming`], Convert into AVI) every sound input is read late by +/// the priming P, so a sound that ends with the window (the restored track, +/// always exactly the window long; a source whose sound ends at the out +/// point) ends P before the picture on ffmpeg's timeline, and `-shortest` +/// dropped the last picture: 49 stored for 50 (task #58). There the output +/// stops at the window's length instead (`-t`, rounded up like a seek), which +/// keeps every picture and ends the sound with the picture. Untrimmed: none. +pub fn encode_end_args(job: &VideoJob, priming: Option<&EncoderPriming>) -> Vec { + let is_trimmed = job.start_frame.is_some() || job.end_frame.is_some(); + if !is_trimmed { + return Vec::new(); + } + let window = priming.and_then(|_| audio_restoration::OutputWindow::for_job(job).duration); + match window { + Some(d) => vec!["-t".to_string(), crate::source_decode::seek_arg(d)], + None => vec!["-shortest".to_string()], + } +} + /// The output option an [`EncoderPriming`] needs /// (`audio_restoration::ENCODER_PRIMING_OUTPUT_ARGS`), or none. pub fn encoder_priming_output_args(priming: Option<&EncoderPriming>) -> Vec { @@ -2659,6 +2676,40 @@ mod tests { assert!(joined.contains("-ss 2.221334 -i input.mp4"), "{joined}"); } + /// Task 58: a trimmed Convert into AVI stops at the window's length, not + /// with the shortest stream: every sound input is read late by the + /// priming, so a sound that ends with the window (the restored track) ends + /// before the picture and `-shortest` dropped the last picture. + #[test] + fn a_primed_trim_stops_at_the_window_not_the_shortest_stream() { + let mut job = convert_avi_job(); + job.total_frames = Some(50); + assert!(encode_end_args(&job, Some(&AAC_PRIMING)).is_empty(), "untrimmed: no end option"); + job.start_frame = Some(2); + job.end_frame = Some(51); + assert_eq!(encode_end_args(&job, Some(&AAC_PRIMING)), vec!["-t", "2.000000"]); + assert_eq!(encode_end_args(&job, None), vec!["-shortest"]); + let joined = build_ffmpeg_args_with_priming(&job, Some(AAC_PRIMING)).join(" "); + assert!(joined.contains("-avoid_negative_ts disabled -t 2.000000 -y /tmp/out.avi"), "{joined}"); + assert!(!joined.contains("-shortest"), "{joined}"); + let plain = build_ffmpeg_args_with_priming(&job, None).join(" "); + assert!(plain.contains("-shortest -y /tmp/out.avi") && !plain.contains(" -t "), "{plain}"); + // Without a frame count the window is unknown: -shortest as before. + job.total_frames = None; + assert_eq!(encode_end_args(&job, Some(&AAC_PRIMING)), vec!["-shortest"]); + } + + /// The real builder ends the encode through `encode_end_args`. + #[test] + fn test_real_ffmpeg_builder_ends_through_encode_end_args() { + let src = include_str!("pipeline_executor.rs").replace("\r\n", "\n"); + let start = src.find("fn build_ffmpeg_args(\n").expect("real builder"); + let body = &src[start..]; + let body = &body[..body.find("\n }\n").expect("end of builder")]; + assert!(body.contains("args.extend(encode_end_args(job, self.encoder_priming.as_ref()));")); + assert!(!body.contains("\"-shortest\""), "-shortest only through encode_end_args"); + } + /// The NUT copy is read late by the priming too, and an untrimmed delayed /// AVI needs it once the priming moves its seek off 0 (rule 9). #[test] @@ -3078,6 +3129,7 @@ mod tests { // The encoder's priming keeps every stream's timestamps (mirrors // build_ffmpeg_args; its IVTC `make_zero` is not mirrored here). args.extend(encoder_priming_output_args(priming.as_ref())); + args.extend(encode_end_args(job, priming.as_ref())); // Custom arguments if !settings.custom_ffmpeg_args.is_empty() { diff --git a/worker/src/source_continuity.rs b/worker/src/source_continuity.rs index 4b8d5e57..70f6a899 100644 --- a/worker/src/source_continuity.rs +++ b/worker/src/source_continuity.rs @@ -14,13 +14,22 @@ //! ffmpeg's own discontinuity correction off, so a trim across a reset stops //! there. //! -//! **Frame N is the Nth slot of the continuous timeline, with each segment -//! following the last exactly as ffmpeg's correction joins them.** A source -//! with a reset is stream-copied once, in its own container and without -//! `-copyts`, so that correction rewrites the timestamps continuously (one -//! offset per segment, for every stream alike, so sync inside a segment is -//! kept), and everything decodes the copy. A source without one is never -//! copied, and its every argument stays as it was. +//! **Frame N is the Nth slot of the continuous timeline, each segment's +//! first picture one frame after the last segment's last.** A source with a +//! reset is stream-copied once, in its own container, and everything decodes +//! the copy. The copy shifts each segment by one offset, for every stream +//! alike (so sync inside a segment is kept), and **the picture decides the +//! offset** ([`JoinPlan`], task #58): `-copyts` turns ffmpeg's own +//! correction off and a `setts` bitstream filter per stream applies the +//! offsets by byte position. ffmpeg's correction took its offset from +//! whichever stream showed the reset first, so a first part whose sound ran +//! 16 ms past its last picture put every later picture 16 ms off the grid +//! (ffprobe then read 125 fps). A sound packet left overhanging the next +//! segment by more than half of it is dropped. When no plan can be made (no +//! durations or positions, streams resetting a different number of times) +//! the copy falls back to ffmpeg's correction, without `-copyts`; so does +//! a planned copy that fails or does not check out. A source +//! without a reset is never copied, and its every argument stays as it was. //! //! [`resets_from_packets`] is the one statement of what counts as a reset; //! the app learns it by running `--scan-timestamps`. @@ -51,15 +60,16 @@ pub const STREAMS_PROBE_ARGS: [&str; 6] = [ "-of", "json", ]; -/// ffprobe's arguments (before the input) for every packet's stream, DTS and -/// byte position, one `stream_index,dts_time,pos` line each (ffprobe's own -/// field order, whatever order they are asked in): the whole file is read, -/// so the output is kept to what the rule needs, plus the position, which is -/// how far through the file the scan is (task #55: on a network share the -/// read takes minutes, and the app shows it). +/// ffprobe's arguments (before the input) for every packet's stream, DTS, +/// duration and byte position, one `stream_index,dts_time,duration_time,pos` +/// line each (ffprobe's own field order, whatever order they are asked in): +/// the whole file is read, so the output is kept to what the rule and the +/// join plan (task #58) need, plus the position, which is how far through +/// the file the scan is (task #55: on a network share the read takes +/// minutes, and the app shows it). pub const PACKETS_PROBE_ARGS: [&str; 6] = [ "-v", "error", - "-show_entries", "packet=stream_index,dts_time,pos", + "-show_entries", "packet=stream_index,dts_time,duration_time,pos", "-of", "csv=p=0", ]; @@ -92,6 +102,196 @@ pub struct ContinuityScan { pub resets: Vec, /// Video packets in the file, every video stream (how the copy is checked). pub video_packets: u64, + /// How the copy joins the segments; `None` without a reset, or when the + /// packets don't give what the plan needs (the copy then falls back to + /// ffmpeg's own correction). Absent from a scan an older worker wrote. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub join: Option, +} + +/// From this point on (the stream's packet index, or a byte position; see +/// [`JoinStream::by_position`]), a stream's timestamps move by `offset` +/// seconds (the latest cut a packet is at or after applies). +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct Cut { + pub at: i64, + pub offset: f64, +} + +/// One copied stream's part of a [`JoinPlan`]. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct JoinStream { + /// `video`, `audio` or `subtitle`. + pub codec_type: String, + /// Its index among the file's streams of that type, which is its output + /// index of that type too (`-map 0:v -map 0:a? -map 0:s?` keeps order). + pub type_index: u32, + /// Whether the cuts are byte positions (`POS`) rather than the stream's + /// own packet indices (`N`). A stream that resets is cut at its own + /// reset packet, by index: MPEG audio split from one PES by the parser + /// carries no position after its first frame. A stream without a reset + /// of its own (subtitles) is cut where the picture's reset packet sits + /// in the file. + pub by_position: bool, + pub cuts: Vec, + /// Packet indices dropped: the last packet of a segment that overhangs + /// the next segment's first by more than half of itself. + pub drop: Vec, +} + +/// The joined copy's timeline: each segment shifted so that its first +/// picture follows the previous segment's last by one frame, and every +/// stream of the segment shifted alike. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct JoinPlan { + pub streams: Vec, + /// What the plan did that a log should say (dropped sound). + pub notes: Vec, +} + +/// One row of [`PACKETS_PROBE_ARGS`]' answer. +#[derive(Debug, Clone, Copy)] +struct Packet { + dts: Option, + duration: Option, + pos: Option, +} + +fn field(f: Option<&str>) -> Option { + f.and_then(|s| s.trim().parse::().ok()) +} + +/// The packet rows per stream, in file order, for the streams in `types`. +fn packets_by_stream(csv: &str, types: &HashMap) -> std::collections::BTreeMap> { + let mut out: std::collections::BTreeMap> = Default::default(); + for line in csv.lines() { + let mut fields = line.split(','); + let Some(stream) = field::(fields.next()) else { continue }; + if !types.contains_key(&stream) { + continue; + } + let dts = field::(fields.next()).filter(|t| t.is_finite()); + let duration = field::(fields.next()).filter(|d| d.is_finite() && *d > 0.0); + let pos = field::(fields.next()).filter(|p| *p >= 0); + out.entry(stream).or_default().push(Packet { dts, duration, pos }); + } + out +} + +/// Where a stream's clock jumps: (index of the last packet with a DTS +/// before it, index of the first after it), by [`resets_from_packets`]' rule. +fn reset_points(packets: &[Packet]) -> Vec<(usize, usize)> { + let mut last: Option<(usize, f64)> = None; + let mut out = Vec::new(); + for (i, p) in packets.iter().enumerate() { + let Some(dts) = p.dts else { continue }; + if let Some((j, prev)) = last { + let step = dts - prev; + if step < -RESET_BACKWARD_SECS || step > RESET_FORWARD_SECS { + out.push((j, i)); + } + } + last = Some((i, dts)); + } + out +} + +fn typical_duration(packets: &[Packet]) -> Option { + let mut d: Vec = packets.iter().filter_map(|p| p.duration).collect(); + if d.is_empty() { + return None; + } + d.sort_by(|a, b| a.partial_cmp(b).unwrap()); + Some(d[d.len() / 2]) +} + +/// The [`JoinPlan`] for a packet list (as [`resets_from_packets`] reads it, +/// with durations and positions), or `None` when there is no video reset or +/// the plan can't be made exactly: a reset packet without a position, a +/// video stream without a duration, an audio or video stream resetting a +/// different number of times from the picture. +pub fn join_plan(csv: &str, types: &HashMap) -> Option { + let copied: HashMap = types + .iter() + .filter(|(_, t)| matches!(t.as_str(), "video" | "audio" | "subtitle")) + .map(|(i, t)| (*i, t.clone())) + .collect(); + let streams = packets_by_stream(csv, &copied); + // The grid: the first video stream with a reset. + let (_, grid_packets) = streams + .iter() + .find(|(i, p)| copied[*i] == "video" && !reset_points(p).is_empty())?; + let frame = typical_duration(grid_packets)?; + let mut offsets = Vec::new(); + // The picture's cuts by byte position, for streams without a reset of + // their own; `None` when a reset packet has no position. + let mut grid_positions = Some(Vec::new()); + let mut off = 0.0; + for (from, to) in reset_points(grid_packets) { + let (a, b) = (grid_packets[from], grid_packets[to]); + let step = a.duration.unwrap_or(frame); + off += a.dts? + (to - from) as f64 * step - b.dts?; + offsets.push(off); + grid_positions = grid_positions.and_then(|mut v: Vec| { + v.push(Cut { at: b.pos?, offset: off }); + Some(v) + }); + } + let mut type_counts: HashMap = HashMap::new(); + let mut plan = JoinPlan { streams: Vec::new(), notes: Vec::new() }; + for (&index, packets) in &streams { + let kind = copied[&index].clone(); + let type_index = { + let c = type_counts.entry(kind.clone()).or_insert(0); + *c += 1; + *c - 1 + }; + let resets = if kind == "subtitle" { Vec::new() } else { reset_points(packets) }; + let mut stream = JoinStream { + codec_type: kind.clone(), + type_index, + by_position: false, + cuts: Vec::new(), + drop: Vec::new(), + }; + if resets.is_empty() { + // No reset of its own (subtitles, a stream that ends before + // it): the picture's cuts, by where they sit in the file. + stream.by_position = true; + stream.cuts = grid_positions.clone()?; + } else if resets.len() == offsets.len() { + let dur = typical_duration(packets); + let mut prev_off = 0.0; + for (k, &(from, to)) in resets.iter().enumerate() { + let (a, b) = (packets[from], packets[to]); + stream.cuts.push(Cut { at: to as i64, offset: offsets[k] }); + // How far its last packet before the reset runs into the + // next segment's first, once both are shifted. + if let (Some(d), Some(ad), Some(bd), true) = (a.duration.or(dur), a.dts, b.dts, to == from + 1) { + let overhang = (ad + prev_off + d) - (bd + offsets[k]); + if overhang > d / 2.0 { + stream.drop.push(from as i64); + plan.notes.push(format!( + "{} stream {}: the segment's sound ran {:.0} ms past its last picture; \ + its last packet ({:.0} ms) is dropped at the join", + kind, + type_index, + (overhang) * 1000.0, + d * 1000.0 + )); + } + } + prev_off = offsets[k]; + } + } else { + return None; + } + plan.streams.push(stream); + } + Some(plan) } impl ContinuityScan { @@ -108,11 +308,16 @@ impl ContinuityScan { .filter(|r| r.codec_type == "video") .map(|r| format!("picture {} ({:.2} s -> {:.2} s)", r.packet, r.from, r.to)) .collect(); - if video.is_empty() { + let mut line = if video.is_empty() { format!("{} timestamp reset(s) in the sound", self.resets.len()) } else { format!("timestamps reset at {}", video.join(", ")) + }; + for note in self.join.iter().flat_map(|j| &j.notes) { + line.push_str("; "); + line.push_str(note); } + line } } @@ -124,7 +329,7 @@ pub fn applies_to(format_name: &str) -> bool { .any(|n| TIMESTAMP_SEARCH_FORMATS.contains(&n.trim())) } -/// The resets in a packet list (`stream_index,dts_time[,pos]` lines, in +/// The resets in a packet list (`stream_index,dts_time[,duration_time,pos]` lines, in /// file order), for the audio and video streams in `types` (index -> codec /// type): [`ResetScanner`] over the whole list. pub fn resets_from_packets(csv: &str, types: &HashMap) -> (Vec, u64) { @@ -169,13 +374,14 @@ impl<'a> ResetScanner<'a> { self.position } - /// One `stream_index,dts_time[,pos]` line. + /// One `stream_index,dts_time[,duration_time,pos]` line. pub fn push(&mut self, line: &str) { let mut fields = line.split(','); let Some(stream) = fields.next().and_then(|s| s.trim().parse::().ok()) else { return; }; let dts = fields.next(); + let _duration = fields.next(); if let Some(pos) = fields.next().and_then(|s| s.trim().parse::().ok()) { self.position = Some(self.position.map_or(pos, |p| p.max(pos))); } @@ -304,7 +510,7 @@ pub fn scan_packets_with_progress( mut progress: impl FnMut(f64), ) -> Result { if !applies_to(&format) { - return Ok(ContinuityScan { format, applies: false, resets: vec![], video_packets: 0 }); + return Ok(ContinuityScan { format, applies: false, resets: vec![], video_packets: 0, join: None }); } let size = std::fs::metadata(input).map(|m| m.len()).unwrap_or(0); let mut child = Command::new(ffprobe) @@ -316,6 +522,7 @@ pub fn scan_packets_with_progress( .spawn() .context("failed to run ffprobe")?; let mut scanner = ResetScanner::new(types); + let mut csv = String::new(); if let Some(stdout) = child.stdout.take() { use std::io::BufRead; let mut reported = 0.0f64; @@ -327,7 +534,9 @@ pub fn scan_packets_with_progress( Ok(0) | Err(_) => break, Ok(_) => {} } - scanner.push(&String::from_utf8_lossy(&line)); + let text = String::from_utf8_lossy(&line); + scanner.push(&text); + csv.push_str(&text); if let (Some(pos), true) = (scanner.position(), size > 0) { let f = (pos as f64 / size as f64).min(1.0); if f - reported >= PROGRESS_STEP { @@ -342,8 +551,10 @@ pub fn scan_packets_with_progress( bail!("ffprobe could not list the packets of {}", input); } let (resets, video_packets) = scanner.finish(); + // The join plan needs every packet's row, but only when there is a reset. + let join = if resets.is_empty() { None } else { join_plan(&csv, types) }; progress(1.0); - Ok(ContinuityScan { format, applies: true, resets, video_packets }) + Ok(ContinuityScan { format, applies: true, resets, video_packets, join }) } /// The muxer a continuous copy is written with: the source's own container, @@ -370,15 +581,32 @@ pub fn continuous_copy_args( muxer: &str, output: &str, with_subtitles: bool, + join: Option<&JoinPlan>, ) -> Vec { - let mut args: Vec = ["-hide_banner", "-nostdin", "-y", "-v", "error", "-i", input] + let mut args: Vec = ["-hide_banner", "-nostdin", "-y", "-v", "error"] .iter() .map(|s| s.to_string()) .collect(); + if join.is_some() { + args.push("-copyts".to_string()); + } + args.extend(["-i", input].iter().map(|s| s.to_string())); args.extend(["-map", "0:v", "-map", "0:a?"].iter().map(|s| s.to_string())); if with_subtitles { args.extend(["-map", "0:s?"].iter().map(|s| s.to_string())); } + for s in join.map(|j| j.streams.as_slice()).unwrap_or_default() { + if s.codec_type == "subtitle" && !with_subtitles { + continue; + } + let spec = match s.codec_type.as_str() { + "video" => "v", + "audio" => "a", + _ => "s", + }; + args.push(format!("-bsf:{spec}:{}", s.type_index)); + args.push(join_bsf(s)); + } args.extend( ["-c", "copy", "-map_metadata", "0", "-progress", "pipe:1", "-nostats", "-f", muxer, output] .iter() @@ -387,6 +615,27 @@ pub fn continuous_copy_args( args } +/// The bitstream filters that move one stream onto the joined timeline: +/// `setts` adds the offset of the latest cut a packet is at or after (its +/// packet index `N`, or its byte position `POS`), leaving a missing +/// timestamp missing; `noise`, after it, drops the overhanging packets by +/// index (`n`: `setts` drops nothing, so both count the same packets). +/// Commas inside an option are escaped for the filter list. +pub fn join_bsf(s: &JoinStream) -> String { + let key = if s.by_position { "POS" } else { "N" }; + let mut off = "0".to_string(); + for c in &s.cuts { + off = format!("if(gte({key}\\,{})\\,{:.9}\\,{})", c.at, c.offset, off); + } + let shift = |ts: &str| format!("if(eq({ts}\\,NOPTS)\\,NOPTS\\,{ts}+round(({off})/TB))"); + let mut f = format!("setts=pts={}:dts={}", shift("PTS"), shift("DTS")); + if !s.drop.is_empty() { + let drop: Vec = s.drop.iter().map(|n| format!("eq(n\\,{n})")).collect(); + f.push_str(&format!(",noise=drop={}", drop.join("+"))); + } + f +} + /// Why a finished copy can't be used, or `None` when it can: it must have no /// reset left and every video packet of the source. pub fn copy_problem(source: &ContinuityScan, copy: &ContinuityScan) -> Option { @@ -423,12 +672,12 @@ pub fn make_continuous( .with_context(|| format!("no continuous copy for format {}", source.format))?; let size = std::fs::metadata(input).map(|m| m.len()).unwrap_or(0); let out_str = output.to_string_lossy().to_string(); - let result = (|| -> Result { + let mut attempt = |join: Option<&JoinPlan>| -> Result { let mut last_err = String::new(); let mut done = false; for with_subtitles in [true, false] { let mut child = Command::new(ffmpeg) - .args(continuous_copy_args(input, muxer, &out_str, with_subtitles)) + .args(continuous_copy_args(input, muxer, &out_str, with_subtitles, join)) .envs(env) .stdout(Stdio::piped()) .stderr(Stdio::piped()) @@ -470,7 +719,15 @@ pub fn make_continuous( } progress(1.0); Ok(copy) - })(); + }; + // The planned join first; should its copy fail or not check out, ffmpeg's + // own correction, which joined every source before task #58. + let result = match source.join.as_ref() { + Some(plan) => attempt(Some(plan)).or_else(|planned| { + attempt(None).map_err(|e| e.context(format!("the planned join failed too: {planned:#}"))) + }), + None => attempt(None), + }; if result.is_err() { let _ = std::fs::remove_file(output); } @@ -588,9 +845,10 @@ mod tests { #[test] fn the_position_column_is_read_and_never_goes_back() { - // ffprobe's own order: stream_index, dts_time, pos (a TS line can - // end in an empty field; a PS sound packet often has no position). - let csv = "1,0.529978,2062\n1,0.553978,N/A\n0,1.40,564,\n0,1.44,5076,\n2,9.0,99999\n0,1.48,4000\n"; + // ffprobe's own order: stream_index, dts_time, duration_time, pos + // (a TS line can end in an empty field; a PS sound packet often has + // no position). + let csv = "1,0.529978,0.024,2062\n1,0.553978,0.024,N/A\n0,1.40,0.04,564,\n0,1.44,0.04,5076,\n2,9.0,N/A,99999\n0,1.48,0.04,4000\n"; let t = types(); let mut s = ResetScanner::new(&t); let mut seen = Vec::new(); @@ -614,14 +872,14 @@ mod tests { "0,0.50\n0,0.54\n0,6.46\n0,0.50\n0,0.54\n", "0,1.0\n0,16.0\n0,16.04\n", "0,6.0\n0,N/A\n0,0.5\n", - "1,6.5,10\n1,0.53,20\n2,9.0,30\n2,0.0,40\n3,9.0\n3,0.0\n0,0.5,50\n", + "1,6.5,0.024,10\n1,0.53,0.024,20\n2,9.0,0.04,30\n2,0.0,0.04,40\n3,9.0\n3,0.0\n0,0.5,0.04,50\n", ]; let t = types(); for csv in cases { let mut s = ResetScanner::new(&t); csv.lines().for_each(|l| s.push(l)); assert_eq!(s.finish(), resets_from_packets(csv, &t), "{csv}"); - // The old two-column lines and the new three-column lines agree. + // The old two-column lines and the new four-column lines agree. let two: String = csv .lines() .map(|l| l.split(',').take(2).collect::>().join(",") + "\n") @@ -632,7 +890,7 @@ mod tests { #[test] fn the_packet_probe_asks_for_the_position() { - assert!(PACKETS_PROBE_ARGS.join(" ").contains("packet=stream_index,dts_time,pos")); + assert!(PACKETS_PROBE_ARGS.join(" ").contains("packet=stream_index,dts_time,duration_time,pos")); } #[test] @@ -649,18 +907,135 @@ mod tests { #[test] fn the_copy_keeps_the_container_and_drops_copyts() { - let a = continuous_copy_args("in.vob", "vob", "out.mpg", true); + let a = continuous_copy_args("in.vob", "vob", "out.mpg", true, None); let s = a.join(" "); assert!(!s.contains("copyts"), "{s}"); assert!(s.contains("-map 0:v -map 0:a? -map 0:s? -c copy -map_metadata 0"), "{s}"); assert!(s.ends_with("-f vob out.mpg"), "{s}"); - let t = continuous_copy_args("in.ts", "mpegts", "out.ts", false).join(" "); + let t = continuous_copy_args("in.ts", "mpegts", "out.ts", false, None).join(" "); assert!(!t.contains("0:s?"), "{t}"); assert!(t.ends_with("-f mpegts out.ts"), "{t}"); // No -muxdelay: the copy keeps the source's start time. assert!(!t.contains("muxdelay"), "{t}"); } + /// The off-grid reset (`conf_reset_offgrid.mpg`'s shape): the picture + /// resets after 50 frames at 2.00 s, the sound after 84 packets, running + /// 16 ms past the last picture. Task #58. + fn offgrid_csv() -> String { + let mut rows = Vec::new(); + let mut pos = 0i64; + let mut push = |s: u32, dts: f64, d: f64| { + rows.push(format!("{s},{dts:.6},{d:.6},{pos}")); + pos += 100; + }; + for part in 0..2 { + for i in 0..50 { + push(0, 0.5 + i as f64 * 0.04, 0.04); + } + let sound = if part == 0 { 84 } else { 83 }; + for i in 0..sound { + push(1, 0.529978 + i as f64 * 0.024, 0.024); + } + } + rows.join("\n") + } + + #[test] + fn the_picture_decides_the_join_and_overhanging_sound_is_dropped() { + let plan = join_plan(&offgrid_csv(), &types()).expect("a plan"); + assert_eq!(plan.streams.len(), 2); + let (v, a) = (&plan.streams[0], &plan.streams[1]); + assert_eq!((v.codec_type.as_str(), v.type_index), ("video", 0)); + // Last picture 0.5 + 49 x 0.04 = 2.46, next at 2.50; the second part's + // first picture is at 0.50: everything after moves by exactly 2.00 s. + assert_eq!(v.cuts.len(), 1); + assert!((v.cuts[0].offset - 2.0).abs() < 1e-9, "{:?}", v.cuts); + assert_eq!(v.cuts[0].at, 50, "the second part's first picture"); + assert!(!v.by_position && !a.by_position, "streams that reset are cut by packet index"); + // The sound moves by the picture's offset, at its own reset. + assert!((a.cuts[0].offset - 2.0).abs() < 1e-9); + assert_eq!(a.cuts[0].at, 84); + // Its last packet ends 2.545978; the next part's first starts at + // 2.529978: 16 ms of a 24 ms packet overhang, so it goes. + assert_eq!(a.drop, vec![83]); + assert_eq!(plan.notes.len(), 1, "{:?}", plan.notes); + assert!(v.drop.is_empty()); + } + + #[test] + fn a_join_on_the_grid_drops_nothing_and_two_resets_add_up() { + // Sound and picture end together: 51 pictures = 85 MP2 packets. + let mut rows = Vec::new(); + let mut pos = 0; + for _ in 0..3 { + for i in 0..51 { + rows.push(format!("0,{:.6},0.04,{pos}", 0.5 + i as f64 * 0.04)); + pos += 10; + } + for i in 0..85 { + rows.push(format!("1,{:.6},0.024,{pos}", 0.5 + i as f64 * 0.024)); + pos += 10; + } + } + let plan = join_plan(&rows.join("\n"), &types()).unwrap(); + let offs: Vec = plan.streams[0].cuts.iter().map(|c| c.offset).collect(); + assert_eq!(offs.len(), 2); + assert!((offs[0] - 2.04).abs() < 1e-9 && (offs[1] - 4.08).abs() < 1e-9, "{offs:?}"); + assert!(plan.streams.iter().all(|s| s.drop.is_empty())); + assert!(plan.notes.is_empty()); + } + + #[test] + fn the_summary_says_what_the_join_dropped() { + let mut scan = scan_of(1, 100); + assert!(!scan.summary().contains("dropped")); + scan.join = join_plan(&offgrid_csv(), &types()); + let s = scan.summary(); + assert!(s.starts_with("timestamps reset at picture 150") && s.contains("16 ms past its last picture"), "{s}"); + } + + #[test] + fn no_plan_without_what_it_needs() { + // No durations (an older scan's two columns). + assert!(join_plan("0,6.46\n0,0.50\n", &types()).is_none()); + // No video reset. + assert!(join_plan("0,0.5,0.04,0\n0,0.54,0.04,10\n1,3.0,0.024,20\n1,0.5,0.024,30\n", &types()).is_none()); + // The sound resets twice, the picture once. + let csv = "0,2.46,0.04,0\n0,0.5,0.04,10\n1,2.0,0.024,20\n1,0.5,0.024,30\n1,3.0,0.024,40\n1,0.6,0.024,50\n"; + assert!(join_plan(csv, &types()).is_none()); + } + + #[test] + fn the_join_filters_shift_by_position_and_keep_missing_timestamps() { + let plan = join_plan(&offgrid_csv(), &types()).unwrap(); + let v = join_bsf(&plan.streams[0]); + assert_eq!( + v, + "setts=pts=if(eq(PTS\\,NOPTS)\\,NOPTS\\,PTS+round((if(gte(N\\,50)\\,2.000000000\\,0))/TB))\ + :dts=if(eq(DTS\\,NOPTS)\\,NOPTS\\,DTS+round((if(gte(N\\,50)\\,2.000000000\\,0))/TB))" + ); + assert!(join_bsf(&plan.streams[1]).ends_with(",noise=drop=eq(n\\,83)")); + let a = continuous_copy_args("in.vob", "vob", "out.mpg", true, Some(&plan)).join(" "); + assert!(a.contains("-v error -copyts -i in.vob"), "{a}"); + assert!(a.contains("-bsf:v:0 setts=") && a.contains("-bsf:a:0 setts="), "{a}"); + } + + /// A subtitle stream has no reset of its own: it follows the picture's + /// cuts, and is left out with the subtitles. + #[test] + fn subtitles_follow_the_picture() { + let csv = format!("{}\n2,1.0,,50\n2,0.7,,20000\n", offgrid_csv()); + let plan = join_plan(&csv, &types()).unwrap(); + let sub = plan.streams.iter().find(|s| s.codec_type == "subtitle").unwrap(); + assert!(sub.by_position); + assert_eq!(sub.cuts, vec![Cut { at: 13_400, offset: plan.streams[0].cuts[0].offset }]); + assert!(join_bsf(sub).contains("gte(POS\\,13400)")); + let without = continuous_copy_args("in.vob", "vob", "o.mpg", false, Some(&plan)).join(" "); + assert!(!without.contains("-bsf:s"), "{without}"); + assert!(continuous_copy_args("in.vob", "vob", "o.mpg", true, Some(&plan)).join(" ").contains("-bsf:s:0 ")); + } + fn scan_of(resets: usize, video: u64) -> ContinuityScan { ContinuityScan { format: "mpeg".into(), @@ -675,6 +1050,7 @@ mod tests { }) .collect(), video_packets: video, + join: None, } } diff --git a/worker/src/source_model.rs b/worker/src/source_model.rs index e0608db1..1d67eba0 100644 --- a/worker/src/source_model.rs +++ b/worker/src/source_model.rs @@ -1077,6 +1077,7 @@ mod tests { to: 0.0, }], video_packets: 300, + join: None, }; let mut job = job_with(json!({})); job.source.as_mut().unwrap().continuity = Some(reset.clone()); diff --git a/worker/templates/pipe_source.py b/worker/templates/pipe_source.py index 03c9e686..c8268ea2 100644 --- a/worker/templates/pipe_source.py +++ b/worker/templates/pipe_source.py @@ -56,8 +56,30 @@ } +# Frames read but not yet served are kept beyond the temporal window, up to +# this much memory (at least _MIN_EXTRA_FRAMES of them). Only frames the +# script never asks for (a decimation, a preview's single frame) fill it. +_EXTRA_BYTES = 512 * 1024 * 1024 +_MIN_EXTRA_FRAMES = 16 + + class _PipeReader: - """Reads raw video frames from stdin with a small cache for temporal filters.""" + """Reads raw video frames from stdin, keeping the frames the script may + still ask for. + + A pipe can only be read forward, and VapourSynth does not ask for frames + in order: vspipe requests a new frame whenever any frame completes, so one + early frame whose callback is late (a busy machine) lets the rest of the + clip be read past it -- measured: frames 6 and 7 served after frame 99 was + read (task #58). So a frame is kept until it has been served and is more + than the temporal window behind the oldest frame not yet served (`_low`), + not merely the window behind the last frame read. Frames that are never + asked for pin `_low`; past a memory cap the oldest are dropped anyway. + + A frame asked for after it was dropped raises: it can no longer be read, + and serving any other picture in its place silently writes the wrong frame + into the output (which this reader used to do). + """ def __init__(self, width, height, vs_format, cache_frames=40): self.width = width @@ -81,9 +103,13 @@ def __init__(self, width, height, vs_format, cache_frames=40): self.frame_size = sum(self.plane_sizes) self._pipe = sys.stdin.buffer - self._cache = OrderedDict() - self._cache_limit = max(40, int(cache_frames)) # >= QTGMC TR2 + margin - self._next = 0 + self._cache = OrderedDict() # frame -> bytes, in frame (= read) order + self._window = max(40, int(cache_frames)) # >= QTGMC TR2 + margin + self._cap = self._window + max( + _MIN_EXTRA_FRAMES, _EXTRA_BYTES // max(1, self.frame_size)) + self._next = 0 # the next frame the pipe will deliver + self._low = 0 # the lowest frame not yet served + self._served = set() # frames >= _low already served self._lock = threading.Lock() self._eof = False self._last_frame = None # last successfully read frame data @@ -101,33 +127,67 @@ def _read_one(self): remaining -= len(chunk) return data + def _advance_low(self): + while self._low in self._served: + self._served.discard(self._low) + self._low += 1 + + def _evict(self): + """Drop served frames behind the window, and the oldest past the cap.""" + while self._cache: + k = next(iter(self._cache)) + if k < self._low - self._window: + self._cache.popitem(last=False) + elif len(self._cache) > self._cap: + # Never asked for while the reader went this far ahead: given + # up. Asking for it later raises. + self._cache.popitem(last=False) + if k >= self._low: + self._low = k + 1 + self._served = {s for s in self._served if s > k} + self._advance_low() + else: + break + def _ensure(self, n): - """Read stdin forward until frame n is cached.""" - if self._eof and n not in self._cache: - return - while self._next <= n: + """Read stdin forward until frame n is cached (or the pipe ends).""" + while self._next <= n and not self._eof: data = self._read_one() if data is None: return self._last_frame = data self._cache[self._next] = data self._next += 1 - # Evict old frames - while len(self._cache) > self._cache_limit: - self._cache.popitem(last=False) + self._evict() + + def _frame_data(self, n): + """Frame n's bytes, under the lock. Raises if it was read and dropped.""" + self._ensure(n) + data = self._cache.get(n) + if data is None: + if n < self._next: + raise vs.Error( + f"pipe_source: frame {n} was asked for after it left the " + f"cache (read up to frame {self._next - 1}, oldest frame " + f"not yet served {self._low}, window {self._window}, cap " + f"{self._cap} frames). This is a VapourBox fault: a pass " + "reads further back than its declared radius." + ) + # Past the end of the stream: repeat the last real frame so + # VDecimate can identify it as a duplicate and drop it (blank + # frames would be kept, extending the video duration when + # TOTAL_FRAMES overshoots the actual count). + data = self._last_frame + if n >= self._low: + self._served.add(n) + self._advance_low() + return data def fill_frame(self, n, f): """VapourSynth ModifyFrame callback.""" fout = f.copy() with self._lock: - self._ensure(n) - data = self._cache.get(n) - - if data is None: - # EOF: repeat last real frame so VDecimate can identify it as a - # duplicate and drop it (blank frames would be kept, extending the - # video duration when TOTAL_FRAMES overshoots the actual count). - data = self._last_frame + data = self._frame_data(n) if data is None: return fout # no frames read at all: return blank @@ -175,10 +235,12 @@ def create_pipe_clip(width, height, num_frames, fps_num, fps_den, pix_fmt, fps_num: FPS numerator fps_den: FPS denominator pix_fmt: FFmpeg pixel format string (e.g. 'yuv420p') - cache_frames: Decoded frames kept for out-of-order reads. Must cover - the pipeline's temporal window plus vspipe's read-ahead: a frame - evicted before it is read again comes back as the *last* frame, - silently. The worker sizes it from the pipeline's total radius. + cache_frames: The temporal window: frames kept behind the oldest + frame not yet served, for passes that read back. Frames read ahead + and not yet served are kept as well (see _PipeReader). A frame + asked for after it was dropped raises rather than coming back as + another picture. The worker sizes it from the pipeline's total + radius. Returns: vs.VideoNode backed by stdin pipe diff --git a/worker/tests/continuity_integration_test.rs b/worker/tests/continuity_integration_test.rs index 455bbad2..cea6a588 100644 --- a/worker/tests/continuity_integration_test.rs +++ b/worker/tests/continuity_integration_test.rs @@ -47,6 +47,8 @@ fn reset_fixtures_scan_then_join_into_one_timeline() { .collect(); assert_eq!(video.len(), 1, "{s}"); assert_eq!(video[0]["packet"], 150, "{s}"); + // The picture decides the join (task #58): the scan carries a plan. + assert!(s["join"]["streams"].as_array().is_some_and(|v| !v.is_empty()), "{s}"); let dir = std::env::temp_dir().join(format!("vb_continuity_it_{}", std::process::id())); std::fs::create_dir_all(&dir).unwrap(); @@ -186,3 +188,29 @@ fn the_join_uses_the_models_scan() { assert!(!String::from_utf8_lossy(&r.stdout).contains("Using the source model")); let _ = std::fs::remove_dir_all(&dir); } + +/// A planned join whose copy does not check out falls back to ffmpeg's own +/// correction (how every reset was joined before task #58), rather than +/// leaving the source unjoined. +#[test] +fn a_planned_join_that_fails_falls_back_to_ffmpegs_correction() { + use vapourbox_worker::dependency_locator::DependencyLocator; + use vapourbox_worker::source_continuity::{make_continuous, scan as scan_file}; + let d = DependencyLocator::new().expect("deps"); + let (ffmpeg, ffprobe, env) = (d.ffmpeg_path().unwrap(), d.ffprobe_path().unwrap(), d.build_environment()); + let src = resource("seek_reset_dvd_pal.mpg"); + let src = src.to_str().unwrap(); + let mut source = scan_file(&ffprobe, &env, src).unwrap(); + // A plan that leaves the clock going back: its copy keeps the reset. + let plan = source.join.as_mut().expect("a plan"); + for s in &mut plan.streams { + for c in &mut s.cuts { + c.offset = 0.0; + } + } + let out = std::env::temp_dir().join(format!("vb_continuity_fallback_{}.mpg", std::process::id())); + let copy = make_continuous(&ffmpeg, &ffprobe, &env, src, &source, &out, |_| {}).expect("the fallback joins"); + assert!(copy.resets.is_empty(), "{:?}", copy.resets); + assert_eq!(copy.video_packets, 300); + let _ = std::fs::remove_file(&out); +}