Bench: plan and pool modes, xruns, latency, stress and repeats with spread

.github/workflows/bench.yml

@@ -49,7 +49,10 @@ jobs:
           done
 
       - name: shellcheck
-        run: shellcheck bench/run.sh bench/validate.sh bench/lib/send.sh bench/tests/sabotage.sh
+        run: shellcheck bench/run.sh bench/validate.sh bench/table.sh bench/lib/send.sh bench/tests/sabotage.sh
+
+      - name: Unit tests of the harness
+        run: python3 -I bench/tests/test_bench.py
 
       - name: Every guard of the harness goes red when broken
         run: bench/tests/sabotage.sh --teardown
@@ -59,7 +62,7 @@ jobs:
           set -euo pipefail
           export XDG_RUNTIME_DIR=/tmp/xdg
           mkdir -p "$XDG_RUNTIME_DIR"
-          bench/run.sh --class ci --out smoke --rates 48000 --frames 256 --window 2 --scenarios lv2-1 --keep-governor
+          bench/run.sh --class ci --out smoke --rates 48000 --frames 256 --window 2 --repeats 1 --scenarios lv2-1 --keep-governor
           bench/validate.sh smoke/*.json
           python3 - smoke/*.json <<'EOF'
           import json, shutil, sys

bench/README.md

@@ -28,6 +28,8 @@ It needs the OpenMixer channel's `plugin-hostd`, `omx-clap-host`, `omx-plugins-c
 | `--scenarios a,b` | only these scenarios |
 | `--hosts H1,H2` | only these hosts |
 | `--window S` | seconds of measurement per row, 10 by default |
+| `--repeats N` | runs of each row, 3 by default; the document holds the median and the spread. 1 is a quick look; a document that is to say anything needs 3 or more |
+| `--stress none\|on\|both` | scenarios that declare stress run without it, with it, or both (the default) |
 | `--pianoteq-dir DIR` | the directory of Pianoteq's LV2 bundle; the `pianoteq` scenario runs only with it |
 | `--qualify` | also time `plugin-qualify --all --isolate` at the first rate and frame count |
 | `--keep-governor` | leave the CPU governor alone |
@@ -41,6 +43,9 @@ started is left, whatever happened, and exits 3 if one was.
 
 `bench/validate.sh <file>...` checks documents against the schema; a problem prints the file, the row and the rule,
 and the exit is 1. `bench/lib/send.sh <port> <line>...` sends protocol lines to a host by hand.
+`bench/table.sh <file>...` writes the chains and grouping rows as markdown: the figures with their spread, the ratios of
+one process per plugin to one chain, and the xruns summed per mode, with and without stress. A table of grouped against
+one process per plugin is made by running the `cvp3,cvp8,cvp16` scenarios on the machine and handing the document to it.
 
 Scenarios
 ---------
@@ -58,6 +63,7 @@ for an instrument, wired through the plugins in series into the playback ports.
 | `pianoteq` | Pianoteq 9 playing chords |
 | `kill1` | omx-comp, then `kill -9` of the process that holds it |
 | `cvp3`, `cvp8`, `cvp16` | chains vs processes: N plugins cycling omx-comp, Calf Compressor, omx-delay, omx-drive |
+| `grp8`, `grp16` | grouping vs processes: N CLAP plugins cycling omx-comp, omx-drive, omx-delay, omx-drive, omx-comp, where omx-drive is pinned `isolated` |
 
 H1 and H2 take our plugins as CLAP and Calf through the LV2 adapter. H3 and H4 take the LV2 builds of our plugins,
 and the row's note says so.
@@ -72,13 +78,33 @@ Chains vs processes runs the same N plugins in each of these modes, one row per
 | `shared-process` | H3 | one mod-host |
 | `individual` | H4 | N jalv processes |
 
+Grouping vs processes (`grp8`, `grp16`) runs one strip four ways under H2 only:
+
+| Mode | How plugin-hostd runs the strip | Processes |
+|------|---------------------------------|-----------|
+| `individual` | no chain declared, `require_pins 0` | the daemon and N workers |
+| `chain` | `chain 1 <ids>`, `require_pins 0`: the case where every plugin may share | the daemon and one worker |
+| `plan` | `grouping on`, `require_pins 1`, a verdict per plugin; `plan 1 <ids>` | the daemon and one worker per segment |
+| `pool` | the same plan, then the members it left alone that the pins let share leave for `pool:g1` | the daemon, the chain workers, the isolated ones, one pool worker |
+
+A row of `plan` or `pool` needs a plugin-hostd with the planner (plugin-hostd#185). Without it the daemon refuses the
+`grouping` setting and the row says `unsupported`, with that as its note. The runner starts a daemon once to let it learn
+each plugin's pin, then writes a verdict on each line of the daemon's own pin store: `isolated` for the plugins the
+scenario names under `unshared`, `in-process` for the rest, and starts the row's daemon on that store with the package
+pins off. A plan of one segment is a `failed` row (nothing was left for the pins to cut, so it is the `chain` row), and so
+is a pool that the daemon did not form from exactly the members the runner moved into it. One strip is one chain, so the
+pool holds that strip's leftovers; sharing a pool across chains is the planner's own later step.
 Reading the numbers
 -------------------
 
 | Field | Meaning |
 |-------|---------|
 | `dsp_load_avg`, `dsp_load_max` | JACK's DSP load in percent, one `jack_cpu_load` reading a second over the window, the first dropped |
 | `samples` | how many readings |
+| `xruns` | JACK's xrun callbacks over the window, counted by a client of the run's private server (`libjack` through ctypes, no port and no process callback, so it adds no work to the cycle). If the server's own log names xruns in the window and the probe counted none, the row is `failed`: a deaf probe reads 0 on every row |
+| `latency_frames` | the capture latency JACK reports at `system:playback_1`, the longest of its range: what the clients on the path declared, not a measurement of the audio; a host that declares none contributes none |
+| `stress` | `none`, or `busy<cores>+mem<MiB>`: one busy loop pinned to each core the host may use, and a process that keeps touching 256 MiB, running beside the host for the whole window. The same name on the same machine is the same load. The same scenario with and without stress is two rows |
+| `repeats`, `spread` | a row run `repeats` times holds the median of each measure, a count taking the higher middle run so it is never understated, and `spread`: `{measure: [min, max]}`. A repeat that fails fails the row and its note says which; a row that is `unsupported` is not repeated |
 | `rss_mb`, `rss_mb_total` | resident memory summed over the host's processes at the end of the window |
 | `pss_mb_total` | proportional memory (`Pss` of `smaps_rollup`) summed the same way: shared libraries counted once across the processes |
 | `processes` | the host's processes: itself and every descendant, the workers of plugin-hostd included |
@@ -90,8 +116,19 @@ Reading the numbers
 
 Under H1, H3 and H4 the killed process is the host itself, so `kill1` ends `failed` with the note `host process lost`.
 
+`xruns`, `latency_frames`, `stress`, `repeats` and `spread`, and the modes `plan` and `pool`, are additions to
+`omx-bench/1`: a document without them is valid, and a reader that ignores them reads the document as before.
+
 `qualification` is written with `--qualify`: the CLAP plugins on the machine, the seconds `plugin-qualify` took, and how
 many it rated suitable.
 
 The DSP load of class `ci` comes from a container on a shared machine and tells nothing about a device; it is there to
 show the format. Compare rows of one document, or of documents of one class.
+
+Tests
+-----
+
+`python3 -I bench/tests/test_bench.py` needs no JACK server and no host: the schema and validator rules, the xrun probe
+against a stand-in `libjack`, the stress processes (pinned, resident) on the machine it runs on, the median and spread,
+the plan and pool steps against a stand-in daemon on a socket, the pin verdicts, and the tables. `bench/tests/sabotage.sh`
+breaks one line behind each of those guards in a copy and requires its test to go red; `--teardown` also runs the hosts.