Skip to content

ci: build and run the unit test suite on Linux/macOS/Windows - #442

Merged
0xShug0 merged 1 commit into
0xShug0:mainfrom
gqf2008:ci/unit-tests
Sep 7, 2026
Merged

ci: build and run the unit test suite on Linux/macOS/Windows#442
0xShug0 merged 1 commit into
0xShug0:mainfrom
gqf2008:ci/unit-tests

Conversation

@gqf2008

@gqf2008 gqf2008 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

The registered unit tests (55+ targets under ENGINE_BUILD_TESTS) were never compiled or run by CI — the flag defaults to OFF and no workflow ever flipped it. This wires the suite into all three CMake-driven workflows:

  • Linux (cpu + vulkan matrix): configure with -DENGINE_BUILD_TESTS=ON, build the remaining test targets, run ctest --output-on-failure.
  • macOS: same pattern in mac-build.yml.
  • Windows: build_windows.ps1 gains a -RunTests switch (forces tests ON, builds everything the -Target pass skipped, runs ctest) so the release presets stay untouched.

Known environment-specific exclusions, each annotated in the workflow:

  • audio_utility_api_test on Debug builds: hits a Debug-only ggml softmax NaN assert inside the denoise models (passes in Release, e.g. the Windows job); excluded until the root cause is chased.
  • rnnoise/zipenhancer utility tests on the Linux vulkan matrix only: they hard-require a Vulkan device, and runner behavior is non-uniform (backend registers with zero devices on Xeon, sometimes not at all on EPYC). mesa-vulkan-drivers + explicit VK_ICD_FILENAMES are installed for the matrix; the cpu matrix keeps both tests.
  • audio_dsp_test istft variant parity: mean tolerance rode at 2.0e-6 and CI measured 2.0074e-06 — loosened to 3e-5 max / 5e-6 mean, still an order of magnitude below one int16 LSB.
  • fun_asr_nano_assets_test: compared temp paths as strings; canonical forms now compared (macOS /var symlink, Windows 8.3 short names).
  • supertonic_vector_convnext_exp_test: the 5% perf-improvement assertion is noise-bound on shared runners; gated behind SUPERTONIC_ENFORCE_PERF=1, output parity stays unconditional.

Also fixes an istft tolerance edge and a Vulkan ICD discovery issue surfaced by the first run.

Verification

Full matrix green on the fork (Linux cpu + Linux vulkan + macOS + Windows, ~59 tests each):

mac-build: success
Linux Build: success
Nix Build: success
windows-build: success

Local macOS arm64: text_chunking_test, audio_dsp_test, fun_asr_nano_assets_test, supertonic_vector_convnext_exp_test all pass; suite runs in ctest with --output-on-failure and parallelism 4.

@0xShug0

0xShug0 commented Sep 4, 2026

Copy link
Copy Markdown
Owner

@gqf2008 Let’s hold off on this PR for now. The current unit tests (and, in fact, most of the test scripts) need a deep cleaning. Many of them should not belong to unittests. I did a test target leakage fix last week, and planned to do more cleanup after the next release.

@gqf2008
gqf2008 marked this pull request as draft September 4, 2026 06:07
@gqf2008

gqf2008 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Understood — that matches exactly what the first CI run surfaced: the noisy failures were all utility/perf/parity tests that probably don't belong in unittests to begin with. Marked as draft. Happy to rebase this on top of your test-target cleanup after the next release — or help with the cleanup itself if that's useful.

@0xShug0

0xShug0 commented Sep 6, 2026

Copy link
Copy Markdown
Owner

@gqf2008 This PR can be reopend. I moved random tests to a new option ENGINE_BUILD_EXTENDED_TESTS so the current unit tests are relatively clean. Also updated the thresholds in audio_dsp_test to 3.0e-5f and 5.0e-6.

ENGINE_BUILD_TESTS is OFF by default so CI never compiled or ran the
unittest suite. Configure with ENGINE_BUILD_TESTS=ON and run ctest after
the main targets on the three CMake-driven workflows; build_windows.ps1
gains a -RunTests switch (forces tests ON, builds remaining targets,
runs ctest) instead of hardcoding it into the release presets.

Rebased after the test-gate split (0xShug0#480-era): the noisy utility/perf
tests now live behind ENGINE_BUILD_EXTENDED_TESTS / MODEL_TESTS, so the
plain unit gate runs clean with no per-test exclusions.
@gqf2008
gqf2008 marked this pull request as ready for review September 7, 2026 08:19
@gqf2008

gqf2008 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on the post-split main: the unit gate now runs with no per-test exclusions — everything noisy (audio_utility, rnnoise/zipenhancer, audio_dsp istft) moved to EXTENDED_TESTS/MODEL_TESTS by your split, and the thresholds you updated cover the istft drift. PR is now just the CI wiring: ENGINE_BUILD_TESTS=ON + ctest on Linux (cpu+vulkan matrix), macOS, Windows (ps1 -RunTests).

@0xShug0
0xShug0 merged commit 0085a65 into 0xShug0:main Sep 7, 2026
6 checks passed
@0xShug0

0xShug0 commented Sep 7, 2026

Copy link
Copy Markdown
Owner

@gqf2008 Merged! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants