ci: build and run the unit test suite on Linux/macOS/Windows - #442
Conversation
|
@gqf2008 Let’s hold off on this PR for now. The current unit tests (and, in fact, most of the test scripts) need a deep cleaning. Many of them should not belong to unittests. I did a test target leakage fix last week, and planned to do more cleanup after the next release. |
|
Understood — that matches exactly what the first CI run surfaced: the noisy failures were all utility/perf/parity tests that probably don't belong in unittests to begin with. Marked as draft. Happy to rebase this on top of your test-target cleanup after the next release — or help with the cleanup itself if that's useful. |
|
@gqf2008 This PR can be reopend. I moved random tests to a new option ENGINE_BUILD_EXTENDED_TESTS so the current unit tests are relatively clean. Also updated the thresholds in audio_dsp_test to 3.0e-5f and 5.0e-6. |
ENGINE_BUILD_TESTS is OFF by default so CI never compiled or ran the unittest suite. Configure with ENGINE_BUILD_TESTS=ON and run ctest after the main targets on the three CMake-driven workflows; build_windows.ps1 gains a -RunTests switch (forces tests ON, builds remaining targets, runs ctest) instead of hardcoding it into the release presets. Rebased after the test-gate split (0xShug0#480-era): the noisy utility/perf tests now live behind ENGINE_BUILD_EXTENDED_TESTS / MODEL_TESTS, so the plain unit gate runs clean with no per-test exclusions.
41566ff to
e2a3c6c
Compare
|
Rebased on the post-split main: the unit gate now runs with no per-test exclusions — everything noisy (audio_utility, rnnoise/zipenhancer, audio_dsp istft) moved to EXTENDED_TESTS/MODEL_TESTS by your split, and the thresholds you updated cover the istft drift. PR is now just the CI wiring: ENGINE_BUILD_TESTS=ON + ctest on Linux (cpu+vulkan matrix), macOS, Windows (ps1 -RunTests). |
|
@gqf2008 Merged! Thanks! |
Summary
The registered unit tests (55+ targets under
ENGINE_BUILD_TESTS) were never compiled or run by CI — the flag defaults to OFF and no workflow ever flipped it. This wires the suite into all three CMake-driven workflows:-DENGINE_BUILD_TESTS=ON, build the remaining test targets, runctest --output-on-failure.mac-build.yml.build_windows.ps1gains a-RunTestsswitch (forces tests ON, builds everything the-Targetpass skipped, runs ctest) so the release presets stay untouched.Known environment-specific exclusions, each annotated in the workflow:
audio_utility_api_teston Debug builds: hits a Debug-only ggml softmax NaN assert inside the denoise models (passes in Release, e.g. the Windows job); excluded until the root cause is chased.rnnoise/zipenhancerutility tests on the Linux vulkan matrix only: they hard-require a Vulkan device, and runner behavior is non-uniform (backend registers with zero devices on Xeon, sometimes not at all on EPYC).mesa-vulkan-drivers+ explicitVK_ICD_FILENAMESare installed for the matrix; the cpu matrix keeps both tests.audio_dsp_testistft variant parity: mean tolerance rode at 2.0e-6 and CI measured 2.0074e-06 — loosened to 3e-5 max / 5e-6 mean, still an order of magnitude below one int16 LSB.fun_asr_nano_assets_test: compared temp paths as strings; canonical forms now compared (macOS /var symlink, Windows 8.3 short names).supertonic_vector_convnext_exp_test: the 5% perf-improvement assertion is noise-bound on shared runners; gated behindSUPERTONIC_ENFORCE_PERF=1, output parity stays unconditional.Also fixes an istft tolerance edge and a Vulkan ICD discovery issue surfaced by the first run.
Verification
Full matrix green on the fork (Linux cpu + Linux vulkan + macOS + Windows, ~59 tests each):
Local macOS arm64:
text_chunking_test,audio_dsp_test,fun_asr_nano_assets_test,supertonic_vector_convnext_exp_testall pass; suite runs in ctest with--output-on-failureand parallelism 4.