own-host page: an LV2 URI through omx-clap-host, run in CI on Debian bookworm and trixie (#52)

.github/workflows/ci.yml

@@ -383,7 +383,8 @@ jobs:
 
   # docs/own-host.md, command by command, as a reader would type it: a clean Debian with nothing but what the page
   # lists, the published packages of the FreeMixer channel, on both architectures (arm64 is a native runner).
-  # A step of the page that stops working fails this job.
+  # A step of the page that stops working fails this job. The page's LV2 block runs an LV2 URI with
+  # `lv2_worker omx-clap-host`, and --audio then measures the gain of a sine that went through that instance.
   own-host-page:
     name: own-host page (${{ matrix.suite }}, ${{ matrix.arch }}), run as written in a clean container
     runs-on: ${{ matrix.arch == 'arm64' && 'ubuntu-24.04-arm' || 'ubuntu-latest' }}
@@ -411,13 +412,13 @@ jobs:
       - name: The reader stops on a wrong reply
         run: python3 tests/own_host_page.py selftest
 
-      - name: The page's commands
+      - name: The page's commands, the LV2 plugin included, and its audio
         run: |
           set -euo pipefail
           export XDG_RUNTIME_DIR=/tmp/xdg XDG_STATE_HOME=/tmp/state
           mkdir -p "$XDG_RUNTIME_DIR" "$XDG_STATE_HOME" /tmp/own-host
           cd /tmp/own-host
-          python3 "$GITHUB_WORKSPACE/tests/own_host_page.py" run
+          python3 "$GITHUB_WORKSPACE/tests/own_host_page.py" run --audio
           test "$(dpkg --print-architecture)" = "${{ matrix.arch }}"
 
   # The release workflow as a tag would run it, publishing nothing: both architectures build through the

docs/own-host.md

@@ -2,8 +2,8 @@ Use plugin-hostd in your own host
 =================================
 
 This page takes a Debian machine (Raspberry Pi OS, Zynthian, any Debian 12 or 13, arm64 or amd64) from nothing to a
-plugin running under plugin-hostd, driven over its socket, in nine commands. CI runs every command below, as written, in
-a clean Debian container on both architectures; a step that stops working fails that job.
+plugin running under plugin-hostd, driven over its socket, in nine commands, and then runs an LV2 plugin the same way in eight more. CI runs every command below, as written, in
+a clean Debian container on Debian 12 and 13, on both architectures; a step that stops working fails that job.
 
 The plugin is a CLAP effect from the `omx-plugins-clap` package, so nothing else has to be installed to try it. Any
 host that already speaks mod-host's socket protocol can send the same lines.
@@ -43,6 +43,38 @@ What each step did:
 8. `param_set` sets Ratio to 8.
 9. `param_get` reads it back.
 
+LV2 plugins
+-----------
+
+Zynthian's plugins are mostly LV2. `omx-clap-host`, which you already installed, also runs an LV2 plugin through its LV2
+adapter; the LV2 build of the same plugins is the package `omx-plugins-lv2`. Continue in the same shell, with the
+`send` helper of the first block and JACK still running. `lv2_worker` names the program that runs LV2 plugins, and
+`mod-host` is the default, so the daemon is restarted with `omx-clap-host` named in its configuration file. An LV2
+plugin is added by its URI with an `lv2:` prefix, and its parameters are named by their LV2 symbol (`ratio`), not by a
+number.
+
+```sh own-host-lv2
+send "quit"; sleep 1   # resp 0
+sudo apt-get install -y omx-plugins-lv2
+echo "lv2_worker omx-clap-host" > plugin-hostd.conf
+plugin-hostd -n -p 5555 -c plugin-hostd.conf >plugin-hostd-lv2.log 2>&1 & sleep 2
+send "add lv2:urn:openmixer:comp 0"   # resp 0
+send "param_set 0 ratio 8"   # resp 0
+send "param_get 0 ratio"   # resp 0 8.0000
+send "worker_list"   # has :lv2-clap:
+```
+
+1. `quit` stops the daemon and the worker of the first block.
+2. `apt-get install` adds the LV2 bundles under `/usr/lib/lv2`, where the daemon searches.
+3. The configuration file makes `omx-clap-host` the worker of LV2 plugins.
+4. The daemon starts again with it.
+5. `add` loads the plugin by its URI as instance `0`, in a worker of its own.
+6. and 7. `param_set` and `param_get` set and read back the compressor's Ratio by its symbol.
+8. `worker_list` names the worker of each instance; `lv2-clap` is `omx-clap-host` running an LV2 plugin.
+
+CI also sends a sine into the instance's JACK ports and measures what comes out, with the Makeup gain set to 6 dB and
+Ratio to 1: the output peak must be 1.995 times the input's.
+
 Pins need nothing from you. A plugin has to be pinned (its files' hashes and its parameter layout) before the daemon
 lets it run; in the default `first-use` mode the first `add` records the pin in `$XDG_STATE_HOME/plugin-hostd/pins`, and
 a later `add` of a changed file is refused with a `pin_changed` event. A host that pins by itself sets `pin_mode strict`.
@@ -55,12 +87,12 @@ consumer can include `plugin-hostd/protocol.h` (`pkg-config --cflags plugin-host
 What a Zynthian integration still needs from us
 -----------------------------------------------
 
-Checked against the published packages (0.2.1 of the daemon, 0.8.0 of the worker and plugins) in a clean Debian
-trixie container:
+Checked in a clean Debian container against published packages. The first item against 0.4.0 of the daemon and 0.10.0 of
+the worker and plugins (the page's CI); the others against 0.2.1 of the daemon and 0.8.0 of the worker and plugins, on
+trixie, and not re-checked since:
 
-- LV2 needs mod-host, which neither Debian nor the FreeMixer channel packages. Zynthian's plugins are mostly LV2, so
-  until the channel carries an LV2 worker (mod-host, or `omx-clap-host` with `libomx-clap-lv2`, issue #45) the page
-  above only covers CLAP.
+- An LV2 plugin that the LV2 adapter of `omx-clap-host` refuses needs mod-host, which neither Debian nor the FreeMixer
+  channel packages. The add answers `-512`; `lv2_worker auto` runs such a plugin in mod-host when one is installed.
 - `param_info` answers `-511` (the worker holds no checked unit and scale) for every parameter of a CLAP instance with
   the released worker, 0.8.0; the daemon passes the answer through. A host reads ranges and defaults from
   `omx-clap-scan` (or `omx-clap-scan --json`) and not from the socket, until a worker answers `resp 0`.

tests/own_host_lv2_probe.py

@@ -0,0 +1,82 @@
+# SPDX-License-Identifier: GPL-3.0-or-later
+# Proves a JACK client processes audio: plays a sine into <client>:in_1 and in_2 and measures the peak that comes out of
+# <client>:out_1 and out_2 through libjack, with nothing but python3 and the JACK library the page installs. It exits
+# 0 only when out/in is within <tolerance> of <gain>, the linear gain the plugin was set to (a silent or unprocessed
+# output measures 0 or 1).
+#
+#   own_host_lv2_probe.py <client> <gain> <tolerance>
+import ctypes, math, sys, time
+
+PORT_IS_INPUT, PORT_IS_OUTPUT = 1, 2
+AMPLITUDE = 0.25
+HZ = 1000.0
+SETTLE_S, MEASURE_S = 1.5, 1.5
+
+CB = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_uint32, ctypes.c_void_p)
+
+
+def main():
+    if len(sys.argv) != 4:
+        sys.exit("usage: own_host_lv2_probe.py <client> <gain> <tolerance>")
+    target, want, tol = sys.argv[1], float(sys.argv[2]), float(sys.argv[3])
+    jack = ctypes.CDLL("libjack.so.0")
+    jack.jack_client_open.restype = ctypes.c_void_p
+    jack.jack_client_open.argtypes = [ctypes.c_char_p, ctypes.c_int, ctypes.c_void_p]
+    jack.jack_port_register.restype = ctypes.c_void_p
+    jack.jack_port_register.argtypes = [ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.c_ulong, ctypes.c_ulong]
+    jack.jack_port_get_buffer.restype = ctypes.c_void_p
+    jack.jack_port_get_buffer.argtypes = [ctypes.c_void_p, ctypes.c_uint32]
+    jack.jack_set_process_callback.argtypes = [ctypes.c_void_p, CB, ctypes.c_void_p]
+    jack.jack_get_sample_rate.argtypes = [ctypes.c_void_p]
+    jack.jack_activate.argtypes = [ctypes.c_void_p]
+    jack.jack_connect.argtypes = [ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p]
+
+    c = jack.jack_client_open(b"own-host-probe", 0, None)
+    if not c:
+        sys.exit("own_host_lv2_probe: cannot open a JACK client (is jackd running?)")
+    audio = b"32 bit float mono audio"
+    outs = [jack.jack_port_register(c, b"out_%d" % i, audio, PORT_IS_OUTPUT, 0) for i in (1, 2)]
+    ins = [jack.jack_port_register(c, b"in_%d" % i, audio, PORT_IS_INPUT, 0) for i in (1, 2)]
+    rate = jack.jack_get_sample_rate(c)
+    state = {"phase": 0, "measuring": False, "in_peak": 0.0, "out_peak": [0.0, 0.0], "cycles": 0}
+    step = 2.0 * math.pi * HZ / rate
+
+    def process(n, _arg):
+        sig = []
+        for k in range(n):
+            sig.append(AMPLITUDE * math.sin(state["phase"]))
+            state["phase"] = (state["phase"] + step) % (2.0 * math.pi)
+        for p in outs:
+            (ctypes.c_float * n).from_address(jack.jack_port_get_buffer(p, n))[:] = sig
+        if state["measuring"]:
+            state["cycles"] += 1
+            state["in_peak"] = max(state["in_peak"], max(abs(x) for x in sig))
+            for i, p in enumerate(ins):
+                buf = (ctypes.c_float * n).from_address(jack.jack_port_get_buffer(p, n))
+                state["out_peak"][i] = max(state["out_peak"][i], max(abs(x) for x in buf))
+        return 0
+
+    cb = CB(process)
+    jack.jack_set_process_callback(c, cb, None)
+    if jack.jack_activate(c) != 0:
+        sys.exit("own_host_lv2_probe: cannot activate the JACK client")
+    for i in (1, 2):
+        for a, b in (("own-host-probe:out_%d" % i, "%s:in_%d" % (target, i)), ("%s:out_%d" % (target, i), "own-host-probe:in_%d" % i)):
+            if jack.jack_connect(c, a.encode(), b.encode()) != 0:
+                sys.exit("own_host_lv2_probe: cannot connect %s to %s" % (a, b))
+    time.sleep(SETTLE_S)
+    state["measuring"] = True
+    time.sleep(MEASURE_S)
+    state["measuring"] = False
+    if state["cycles"] == 0:
+        sys.exit("own_host_lv2_probe: JACK ran no process cycle")
+    # the tolerance: float32 and the plugin's own smoothing leave the measured gain within about 0.005 of the set one
+    # (1.9953 measured), while the nearest wrong value, an unprocessed signal, measures 1.0 and silence 0.0
+    ratios = [p / state["in_peak"] for p in state["out_peak"]]
+    print("own_host_lv2_probe: %s in peak %.4f, out peak %s, gain %s (wanted %.4f +- %.4f)"
+          % (target, state["in_peak"], ["%.4f" % p for p in state["out_peak"]], ["%.4f" % r for r in ratios], want, tol))
+    if any(abs(r - want) > tol for r in ratios):
+        sys.exit("own_host_lv2_probe: the output does not carry the gain the plugin was set to")
+
+
+main()

tests/own_host_page.py

@@ -4,17 +4,29 @@
 #
 # A line of the block is a command. `send() ...` is a helper and is not counted. A trailing `# resp <text>` says the
 # command's output must begin with <text>; `# has <text>` says it must contain <text>. Anything else after `#` is
-# a comment. The page promises "nine commands"; the block may not grow past ten.
+# a comment. The page promises "nine commands"; the block may not grow past ten. The page has a second block,
+# `sh own-host-lv2`, that continues the first in the same shell (the `send` helper, JACK running); both run as one
+# script. `run --audio` then also proves with libjack that the LV2 instance processes audio: it sets the compressor
+# to a 6 dB makeup gain and checks the output peak of a sine against the input's.
 import os, re, subprocess, sys, tempfile
 
 PAGE = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "docs", "own-host.md")
 MAX_COMMANDS = 10
 
 
-def block(text):
-    m = re.search(r"^```sh own-host\n(.*?)^```", text, re.S | re.M)
+BLOCKS = ("own-host", "own-host-lv2")
+
+# after the page's blocks, with the LV2 instance 0 of the second one running: the sine goes in, the gain must come out
+AUDIO = """send "param_set 0 ratio 1" | grep -qx 'resp 0'
+send "param_set 0 makeupDb 6" | grep -qx 'resp 0'
+python3 "$OWN_HOST_PROBE" effect_0 1.9953 0.03
+"""
+
+
+def block(text, name="own-host"):
+    m = re.search(r"^```sh %s\n(.*?)^```" % re.escape(name), text, re.S | re.M)
     if not m:
-        sys.exit("own_host_page: no ```sh own-host block in the page")
+        sys.exit("own_host_page: no ```sh %s block in the page" % name)
     return [l for l in m.group(1).splitlines() if l.strip()]
 
 
@@ -53,6 +65,17 @@ def script(lines):
     return "\n".join(out) + "\n", n
 
 
+def page_script(text):
+    """the bash script of every block in order, and the command count of each"""
+    parts, counts = [], []
+    for name in BLOCKS:
+        sc, n = script(block(text, name))
+        commands_ok(n)
+        parts.append(sc)
+        counts.append(n)
+    return "\n".join(parts), counts
+
+
 def commands_ok(n):
     if n > MAX_COMMANDS:
         sys.exit("own_host_page: %d commands, the page promises under ten (%d at most)" % (n, MAX_COMMANDS))
@@ -70,48 +93,59 @@ def commands_ok(n):
 """
 
 
-def stubbed(sc, lines, add, setp, get, scan):
-    send = ('send() { case "$1" in add*) echo "%s";; param_set*) echo "%s";; param_get*) echo "%s";; esac; }' % (add, setp, get))
-    return (STUBS + "omx-clap-scan() { echo '%s'; }\n" % scan) + sc.replace(lines[0], send, 1)
+REPLIES = ("add_clap", "set_clap", "get_clap", "scan", "quit", "add_lv2", "set_lv2", "get_lv2", "worker_list")
+RIGHT = ("resp 0", "resp 0", "resp 0 8.0000", "    param 1 Ratio [1 .. 20] default 4 automatable",
+         "resp 0", "resp 0", "resp 0", "resp 0 8.0000", "resp 1 w1:42:lv2-clap:up:own:0")
+WRONG = {"scan": "    param 1 Wrong", "worker_list": "resp 1 w1:42:mod-host:up:own:0"}
+
+
+def stubbed(sc, lines, replies):
+    r = dict(zip(REPLIES, replies))
+    send = ('send() { case "$1" in "add clap"*) echo "%(add_clap)s";; "add lv2"*) echo "%(add_lv2)s";; '
+            '"param_set 0 ratio"*) echo "%(set_lv2)s";; param_set*) echo "%(set_clap)s";; '
+            '"param_get 0 ratio"*) echo "%(get_lv2)s";; param_get*) echo "%(get_clap)s";; '
+            'quit*) echo "%(quit)s";; worker_list*) echo "%(worker_list)s";; esac; }' % r)
+    return (STUBS + "omx-clap-scan() { echo '%s'; }\n" % r["scan"]) + sc.replace(lines[0], send, 1)
 
 
 def selftest():
-    lines = block(open(PAGE).read())
-    sc, n = script(lines)
-    commands_ok(n)
-    right = ("resp 0", "resp 0", "resp 0 8.0000", "    param 1 Ratio [1 .. 20] default 4 automatable")
-    cases = [("right replies", right, 0)]
-    for i, name in enumerate(("add", "param_set", "param_get", "scan")):
-        wrong = list(right)
-        wrong[i] = "resp -103" if i < 3 else "    param 1 Wrong"
+    text = open(PAGE).read()
+    lines = block(text)
+    sc, counts = page_script(text)
+    cases = [("right replies", RIGHT, 0)]
+    for i, name in enumerate(REPLIES):
+        wrong = list(RIGHT)
+        wrong[i] = WRONG.get(name, "resp -103")
         cases.append(("a wrong " + name, tuple(wrong), 1))
     failed = 0
     for label, vals, want in cases:
         with tempfile.NamedTemporaryFile("w", suffix=".sh", delete=False) as f:
-            f.write(stubbed(sc, lines, *vals))
+            f.write(stubbed(sc, lines, vals))
         rc = subprocess.run(["bash", f.name], capture_output=True, text=True, cwd=tempfile.gettempdir()).returncode
         os.unlink(f.name)
         if (rc != 0) != (want != 0):
             print("FAIL own_host_page selftest: %s exited %d" % (label, rc))
             failed = 1
     if not failed:
-        print("ok   own_host_page: %d commands; the right replies pass and each wrong one stops the script" % n)
+        print("ok   own_host_page: %s commands; the right replies pass and each wrong one stops the script"
+              % " + ".join(str(n) for n in counts))
     return failed
 
 
 def main():
     cmd = sys.argv[1] if len(sys.argv) > 1 else "selftest"
-    lines = block(open(PAGE).read())
-    sc, n = script(lines)
-    commands_ok(n)
+    sc, _ = page_script(open(PAGE).read())
     if cmd == "script":
         sys.stdout.write(sc)
     elif cmd == "run":
-        sys.exit(subprocess.run(["bash", "-c", sc]).returncode)
+        if "--audio" in sys.argv[2:]:
+            sc += AUDIO
+        env = dict(os.environ, OWN_HOST_PROBE=os.path.join(os.path.dirname(os.path.abspath(__file__)), "own_host_lv2_probe.py"))
+        sys.exit(subprocess.run(["bash", "-c", sc], env=env).returncode)
     elif cmd == "selftest":
         sys.exit(selftest())
     else:
-        sys.exit("usage: own_host_page.py script|run|selftest")
+        sys.exit("usage: own_host_page.py script|run [--audio]|selftest")
 
 
 main()