Skip to content

codec-compatibility.md

Latest commit

 

History

History
141 lines (113 loc) · 7.31 KB

codec-compatibility.md

File metadata and controls

141 lines (113 loc) · 7.31 KB
Raw

MLVC codec compatibility contract

This document defines compatibility for the fixed-shape dmc61sbr_e1d1 runtime. Passing a model microbenchmark alone is not codec compatibility. A conforming backend must pass the complete mlvc_codec_compatibility target for both registered profiles.

Model identity

The supported profiles are:

Profile Model version 640x368 feature/y/z channels
mlvc-psnr-v1 dmc61sbr_reglu 256/128/128
mlvc-s-psnr-v1 dmc61sbr_mini_reglu 96/48/48

models/profiles/profiles.json binds each profile to its checkpoint hash and construction parameters. A deployment model directory is one canonical bundle produced by tools/model_package.py assemble; it must not combine metadata or PMFs from separate ONNX and TorchScript conversions.

The bundle manifest records the SHA-256 and size of every graph, weight blob, PMF, and metadata file. metadata.json, gaussian_pmf.json, and bit_estimator_pmf.json are canonicalized from the ONNX export for all four backends. The compatibility runner rejects any bundle/reference identity or artifact-hash mismatch before inference.

Normative bitstream behavior

The container identifier is mlvc-frame-le-v1. Every frame is:

int32_le q_index | uint32_le payload_size | payload_size bytes of rANS data

The entropy identifier is canonical-pmf-v1. Given identical integer-valued z_raw, y_raw_0, and y_raw_1 tensors, every backend must produce the same rANS payload and every backend must decode it identically.

The official Python reference's first frame is required to be bit-exact. A recurrent P-frame payload is not required to be byte-identical across GPU backends: small legal FP16 differences in the preceding feature DPB can cross a rounding boundary and change integer latents. Such a stream is still conforming only if it decodes successfully with the canonical PMFs and its round-trip reconstruction meets the rules below.

Reconstruction and tensor rules

References are generated by tools/model_reference.py from the pinned microsoft/mlvc split-model implementation. The standard fixture contains two frames: frame 0 starts with a zero reference (I-frame), and frame 1 consumes the previous feature (P-frame). This exercises preprocessing, QP shifting, encoder DPB, entropy coding, decoder DPB, and reconstruction. The portable C++ test additionally checks the 64-frame GOP wrap and I-frame reset schedule.

A backend must satisfy all of the following:

  • Decode the canonical Python stream and decode its own encoded stream.
  • Produce the requested frame count and preserve each frame's unshifted q_index in the container.
  • For both canonical decode and self round trip, have maximum YUV sample error at most 1 and per-frame PSNR at least 60 dB against Python.
  • Match all int32 control tensors and preprocessed x tensors exactly.
  • Match frame-0 quantized latents exactly.
  • Keep enforced FP16 feature, reference-feature, and reconstruction tensors at maximum absolute error at most 0.25 and RMSE at most 0.007.
  • Emit and consume the Python reference feature on frame 1 within those tensor limits; this is the explicit recurrent-DPB check.

P-frame encoder quantized latents are recorded for diagnosis but do not use a numeric tolerance because one rounding-boundary difference can be integral. Their correctness is instead covered by rANS decode and the self-round-trip reconstruction requirement.

Running the target

Generate references once for both profiles and place them under models/fixtures/references/<profile>/gray-q21-2f. Configure a build with BUILD_TESTING=ON, then run:

cmake --build build/driver-cubin-release --target mlvc_codec_compatibility

The target reports a missing-fixture error until both reference manifests exist. Individual tests can be run with:

ctest --preset driver-cubin-release -L conformance

tests/conformance/run_codec_conformance.py --result result.json writes all payload sizes, per-frame YUV metrics, and per-tensor errors for audit. The same test applies without backend-specific exceptions to onnxruntime, libtorch, tensorrt, and driver-cubin builds.

For long-sequence audits, generate a 100-frame fixture with the pinned Python split-model implementation and add --diagnostic to keep all per-frame and per-tensor measurements when a contract check fails:

(cd third_party/mlvc && uv run --frozen ../../tools/model_reference.py \
    --model-dir "$PWD/../../models/canonical/mlvc-psnr-v1/640x368" \
    --input "$PWD/../../models/fixtures/references/mlvc-psnr-v1/gray-q21-2f/input.yuv" \
    --width 640 --height 360 --frames 100 --q-index 21 \
    --output-dir /tmp/mlvc-official-100 --execution-provider cuda)
python3 tests/conformance/run_codec_conformance.py --diagnostic \
    --binary build/tensorrt-release/mlvc_demo \
    --model-dir models/canonical/mlvc-psnr-v1/640x368 \
    --reference-dir /tmp/mlvc-official-100 \
    --engine-cache-dir models/generated/engines/a30-fp16 \
    --result models/generated/results/conformance/tensorrt-mlvc-100.json

--diagnostic does not weaken the normal test: it records contract_passed=false instead of stopping at the first mismatch. On the NVIDIA A30 used for the 2026-09-02 release audit, the TensorRT profile-scoped engine produced the following 100-frame results:

Profile Encode fps Decode official stream fps Decode self stream fps Official-stream min PSNR Self-stream min PSNR Max self YUV error Strict failures (official/self)
mlvc-psnr-v1 69.98 72.85 95.70 63.522 dB 60.447 dB 2 0 / 32
mlvc-s-psnr-v1 109.72 116.65 146.38 59.421 dB 49.096 dB 4 9 / 100

Both profiles preserved the unshifted q_index=21 on all 100 containers and reset their feature DPB at frames 0 and 64, matching the official GOP schedule. For the main profile, the frame-0 and frame-64 I-frame payloads were both 107 bytes and bit-exact with Python; y_raw_0 and y_raw_1 were numerically exact across the complete sequence. The remaining main-profile failures are small TensorRT FP16 feature/DPB drift and a two-level YUV sample excursion in the self-encoded stream. MLVC-S shows larger long-term feature drift, so its 100-frame TensorRT result is explicitly not release-contract conforming yet.

The complete JSON reports are kept at models/generated/results/conformance/tensorrt-mlvc-100.json and models/generated/results/conformance/tensorrt-mlvc-s-100.json in the development workspace. These long-sequence numbers are diagnostic evidence, not a relaxed release contract: TensorRT tactic selection and FP16 accumulation can vary between serialized engines. The two-frame CTest remains the required portable compatibility gate, while the 100-frame audit currently shows that MLVC-S does not yet meet the strict 60 dB / one-sample long-sequence target.