MOB-411: plugin self-tests — selftest manifest key, run_all/3, mix mob.selftest

CHANGELOG.md

@@ -1,3 +1,31 @@
+## [0.7.17] - 2026-10-08
+
+### Added
+
+- **Plugin self-tests: `mix mob.selftest`, `MobDev.Plugin.SelfTest.run_all/3`
+  (MOB-411).** A plugin names a `Mob.Plugin.SelfTest` implementation in its
+  manifest as `selftest: Module` (mob 0.9.15). `mix mob.selftest` attaches to
+  the deployed app on each selected device (the `mix mob.smoke` selection
+  options, honouring agent-device leases, plus `--timeout` and
+  `--no-restart`), grants the permissions the manifests declare on
+  emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy
+  grant`), relaunches the app, waits for the plugins' OTP applications to
+  be started (`:boot_timeout_ms`, 15 s), calls every activated plugin's
+  `run/1` on the device with
+  `%{platform:, device: :simulator | :emulator | :physical}` and prints a
+  table per device. A self-test that raises, exits, times out (30 s; it is
+  then killed on the device) or returns outside the contract is a `FAIL`;
+  a plugin without one is a `skip`. Non-zero exit on any `FAIL` or
+  unreachable device. `run_all/3` is the runner for callers that hold a
+  node already (mob_ci's invariant P12); `grant_permissions/4` is the
+  pre-launch grant for them.
+- **`selftest` manifest key.** `MobDev.Plugin.Manifest` accepts it (an
+  Elixir module; when the module is loadable it must export `run/1`).
+  `mix mob.validate_plugin` **warns** `missing selftest` on a manifest
+  without one; a later release turns that into an error once the first-party
+  plugins carry theirs (MOB-418). `mix mob.new_plugin` tier-1 and tier-4
+  scaffolds now generate a `SelfTest` module and declare it.
+
 ## [0.7.16] - 2026-10-06
 
 ### Fixed

README.md

@@ -37,6 +37,7 @@ end
 | `mix mob.devices` | List connected devices and their status |
 | `mix mob.attest` | Prove a device is running the code you just pushed — compares module digests, not artifacts ([see below](#did-that-deploy-actually-land-mix-mobattest)) |
 | `mix mob.smoke` | Replay recorded `agent-device` UI flows on devices and check the app's own diagnostics held up ([see below](#smoke-flows-on-devices-mix-mobsmoke)) |
+| `mix mob.selftest` | Run every activated plugin's `Mob.Plugin.SelfTest` on the app running on each device ([see below](#plugin-self-tests-on-devices-mix-mobselftest)) |
 | `mix mob.mutate` | Mutation-test the lines this branch changed: break the code on purpose and report what nothing noticed ([see below](#do-the-tests-guard-anything-mix-mobmutate)) |
 | `mix mob.push` | Hot-push only changed modules (no restart) |
 | `mix mob.enable <feature>...` | Wire up an optional Mob feature — platform-manifest entries, Elixir stubs, dep injections ([see below](#mix-mobenable-feature)) |
@@ -276,6 +277,48 @@ or a manually managed development profile for that bundle id and
 `<id>.uitests`: an Xcode-managed wildcard team profile is rejected. Simulators
 and Android need none of this.
 
+## Plugin self-tests on devices (`mix mob.selftest`)
+
+Every plugin can ship its own proof that it works on a device: a module
+implementing `Mob.Plugin.SelfTest`, named in its manifest as `selftest:
+Module` (see mob's `MOB_PLUGINS.md`, "Self-test"). `mix mob.selftest`
+attaches to the deployed app on each selected device and calls each
+activated plugin's `run/1` over distribution, one plugin at a time, with
+`%{platform: :ios | :android, device: :simulator | :emulator | :physical}`.
+
+```bash
+mix mob.selftest                       # the one connected emulator/simulator
+mix mob.selftest --device emulator-5554 --timeout 10000
+mix mob.selftest --all-devices --no-restart   # attach to the apps as they are
+```
+
+Before the tests, the permissions the manifests declare are granted on
+emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy
+grant`) so no system prompt stands between a self-test and its native code;
+then the app is relaunched as `mix mob.connect` does, because a simulator
+may terminate an app whose privacy settings changed. Each device gets a
+table:
+
+```
+Pixel 8 (emulator-5554) android emulator
+  plugin        outcome  ms   detail
+  mob_location  pass     3
+  mob_whisper   pass     1
+  mob_deliver   pass     2
+  mob_camera    skip     0    needs_hardware
+  3 passed, 0 failed, 1 skipped
+```
+
+A plugin without a self-test is a `skip` (so it is visible, not absent); a
+self-test that raises, exits, times out (30 s by default) or returns
+something outside the contract is a `FAIL` with the reason. The exit status
+is non-zero on any `FAIL`, or when a selected device's node could not be
+reached. Device selection follows `mix mob.smoke` (`--device`,
+`--all-devices`, `--all-physical`, agent-device leases honoured).
+`MobDev.Plugin.SelfTest.run_all/3` is the same runner for callers that
+already hold a node, such as mob_ci, and `grant_permissions/4` the
+pre-launch grant.
+
 ## Do the tests guard anything? (`mix mob.mutate`)
 
 A green suite says the tests ran, not that they guard anything. This changes

decisions/2026-10-08-plugin-self-tests-run-from-the-host.md

@@ -0,0 +1,94 @@
+# Plugin self-tests run from the host: `mix mob.selftest` and `MobDev.Plugin.SelfTest.run_all/3`
+
+- Date: 2026-10-08
+- Status: accepted
+- Linear: MOB-411 (epic MOB-410); the contract is mob's
+  `decisions/2026-10-08-plugin-self-test-contract.md`
+
+## Context
+
+- mob 0.9.15 gives plugins a behaviour, `Mob.Plugin.SelfTest`, with one
+  callback `run(ctx) :: :pass | {:fail, reason} | {:skip, reason}`, named in
+  the manifest as `selftest: Module`. The rule for a pass is a real answer
+  from the plugin's native code.
+- Something has to validate the key, call the tests on a device and judge
+  the answers. mob_dev already owns manifest validation, device discovery
+  and attaching to a running app over distribution (`mix mob.smoke`,
+  `mix mob.attest`), and mob_ci calls into mob_dev for builds and the
+  static gate, so the runner belongs here and mob_ci calls it (invariant
+  P12).
+- A self-test is plugin code running inside the app: it can raise, exit,
+  hang, or return `:ok` because its author misread the contract. None of
+  that may take the runner, or the other plugins' runs, down.
+
+## Decision
+
+- **`selftest` is a manifest key.** `MobDev.Plugin.Manifest` requires an
+  Elixir module; when the module is loadable at validation time (the
+  plugin is compiled: `mix mob.validate_plugin` after a compile, or a host
+  activating a dep) it must export `run/1`, otherwise only the name is
+  checked. A manifest without the key gets a **warning** from
+  `mix mob.validate_plugin` (`missing selftest`) in 0.7.17; it becomes an
+  error in a later release once the first-party plugins carry theirs
+  (MOB-418). Warning first because every published plugin would otherwise
+  stop validating the day this ships.
+- **`run_all(node, ctx, opts)` is the runner.** One remote spawn per
+  plugin, sequential (self-tests may touch shared hardware), each with its
+  own timeout (`:timeout_ms`, 30 s). The remote `run/1` is spawned the way
+  `:erpc.call/5` spawns it (`spawn_request` of `:erpc.execute_call/4`, which
+  reports the return or the exception as its exit reason) but with the pid
+  in hand, so a test that overruns the deadline is **killed** on the
+  device rather than abandoned beside the next plugin's run. Every way it
+  can go wrong becomes that plugin's `{:fail, why}`: timeout, raise
+  (`:undef` for a module the running build does not have, named as such),
+  exit, throw, a kill signal, a lost connection, a non-module `selftest:`
+  value, a manifest that failed signature verification, and a return
+  outside the contract. The contract's three shapes are restated in the runner
+  rather than calling `Mob.Plugin.SelfTest.result?/1`, so mob_dev does not
+  need mob at runtime and the runner works against an app built with an
+  older mob. Plugins with no `selftest:` (or no manifest) are entries with
+  `module: nil` and a `{:skip, _}`: a plugin without a test is visible in
+  every table, not silently absent.
+- **Permissions are pre-granted where the host can.** `grant_permissions/4`
+  runs `adb shell pm grant` for each `android.permissions` entry (only
+  runtime permissions are grantable; a normal one answers with an error
+  that is recorded, not raised) and `xcrun simctl privacy grant` for each
+  `permissions: [%{capability: _}]` whose capability has a simctl service
+  (location, microphone, photos, media, contacts, calendar, reminders,
+  motion). A physical iPhone gets nothing: there is no host-side grant.
+  `run_all/3` never grants: it holds a node, so the app is already running,
+  and `simctl privacy` warns that some changes terminate a running app.
+  Granting is the caller's job before launch: `mix mob.selftest` grants,
+  then relaunches (`--no-restart` grants nothing); mob_ci calls
+  `grant_permissions/4` before it starts the app. (The first draft had
+  `run_all/3` grant when given `:device`; the pre-merge review found that
+  it could kill the simulator app it was about to test.) Physical devices
+  get nothing granted; their self-tests skip with `:needs_user`.
+- **`mix mob.selftest` selects devices like `mix mob.smoke`**
+  (`MobDev.TaskTargets.resolve/3` with the agent-device leases: `--device`
+  / `--only`, `--all-devices`, `--all-physical`, `--ios-only`,
+  `--android-only`; a device another session holds is left alone unless
+  named) and relaunches the app by default like `mix mob.connect`
+  (`--no-restart` attaches as is). A target that `connect_all/1` no longer
+  finds is reported as unreachable, so a run never passes on zero devices. The context is
+  the device record's platform and type, not something the plugin detects:
+  a plugin deciding for itself that it is "on an emulator" is how a skip
+  hides a hole. Exit status is non-zero on any `FAIL` or on a selected
+  device whose node could not be reached (a test that could not run is not
+  a test that passed, the same rule as `mix mob.attest`).
+- **Scaffolds declare one.** `mix mob.new_plugin` tier 1 generates a
+  `SelfTest` that calls the NIF's `ping/0`; tier 4 one that round-trips the
+  supervised worker. Tiers 2 and 3 have no honest host-callable probe in
+  the template and get the warning, which is the nudge intended.
+
+## Consequences
+
+- mob_ci replaces its `device_caps.exs` probe table with `run_all/3` once
+  the plugins carry self-tests (MOB-414, MOB-418).
+- The tests (`test/mob_dev/plugin/self_test_test.exs`) use this node as the
+  device: `:erpc.call/5` to `node()` runs the fixture in a spawned local
+  process, so the raise, exit, throw, timeout, undef and off-contract paths
+  are exercised for real without a device.
+- Turning the warning into an error is a one-line change in
+  `Validator.selftest_review/1` plus moving it into the error list; record
+  the release that does it here.

lib/mix/tasks/mob.selftest.ex

@@ -0,0 +1,212 @@
+defmodule Mix.Tasks.Mob.Selftest do
+  @shortdoc "Run every activated plugin's self-test on the app running on each device"
+
+  @moduledoc """
+  Attach to the deployed app on each selected device and run every activated
+  plugin's `Mob.Plugin.SelfTest` on it, over distribution.
+
+      mix mob.selftest                       # the one connected emulator/simulator
+      mix mob.selftest --device emulator-5554
+      mix mob.selftest --all-devices --timeout 10000
+
+  The app must be deployed (`mix mob.deploy`) with the plugins activated in
+  `mob.exs`; a self-test is only on the device if the build that is running
+  included it. Each plugin's `selftest: Module` from its manifest is called
+  with `%{platform: :ios | :android, device: :simulator | :emulator |
+  :physical}` and gets one line in a table per device: `pass`, `FAIL` with
+  the reason, or `skip` with why (a plugin without a self-test is a skip,
+  so it is visible). A self-test that raises, exits, times out or returns
+  something outside the contract is a `FAIL`.
+
+  Before the tests run, the permissions the manifests declare are granted
+  on emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy
+  grant`), so no system prompt stands between a self-test and its native
+  code. The app is then relaunched (as `mix mob.connect` does), because a
+  simulator may terminate an app whose privacy settings changed.
+  `--no-restart` attaches to the app as it is and grants nothing.
+
+  ## Options
+
+    * `--device` / `--only` (`-d`) — target a device by serial/udid
+      (serial, udid or short id). Repeatable
+    * `--all-devices`  — every emulator and simulator
+    * `--all-physical` — every physical device (with `--all-devices`: everything)
+    * `--ios-only` / `--android-only` — restrict discovery to one platform
+    * `--timeout MS` — per self-test, in milliseconds (default: 30000)
+    * `--no-restart` — do not relaunch the app before testing (and do not
+      grant permissions)
+    * `--cookie C` — dist cookie (default: the app's private cookie)
+
+  With no selection flags exactly one emulator/simulator is picked, and a
+  device another `agent-device` session holds is left alone unless named.
+
+  ## Exit status
+
+  Non-zero when any self-test failed, or a selected device's node could not
+  be reached (a test that could not run is not a test that passed).
+  """
+
+  use Mix.Task
+
+  alias Mix.Tasks.Mob.Connect
+  alias Mix.Tasks.Mob.Deploy
+  alias MobDev.{Connector, Device, DeviceLeases, TaskHelp, TaskTargets}
+  alias MobDev.Plugin.SelfTest
+
+  @switches [
+    device: :keep,
+    only: :keep,
+    all_devices: :boolean,
+    all_physical: :boolean,
+    ios_only: :boolean,
+    android_only: :boolean,
+    timeout: :integer,
+    restart: :boolean,
+    cookie: :string
+  ]
+
+  @impl Mix.Task
+  def run(args), do: run(args, %{})
+
+  @doc false
+  # `deps` replaces the I/O: `:discover` (platforms -> devices, as
+  # `Mix.Tasks.Mob.Deploy.discover_devices/1`), `:leases` (-> the agent-device
+  # claims), `:grant` (device, plugins -> grants, as
+  # `SelfTest.grant_permissions/4` with the app's bundle id), `:connect`
+  # (opts -> {connected, failed} devices, as `MobDev.Connector.connect_all/1`),
+  # `:plugins` (-> activated plugins) and `:run_all` (node, ctx, opts ->
+  # entries, as `SelfTest.run_all/3`).
+  @spec run([String.t()], map()) :: :ok
+  def run(args, deps) do
+    if TaskHelp.help_requested?(args) do
+      TaskHelp.print_module_help(__MODULE__)
+    else
+      selftest(parse!(args), Map.merge(default_deps(), deps))
+    end
+  end
+
+  defp parse!(args) do
+    {opts, argv, invalid} = OptionParser.parse(args, strict: @switches, aliases: [d: :device])
+
+    unless invalid == [] and argv == [] do
+      bad = Enum.map(invalid, &elem(&1, 0)) ++ argv
+      Mix.raise("Unknown option(s) or argument(s): #{Enum.join(bad, ", ")}")
+    end
+
+    if Keyword.get(opts, :timeout, 1) < 1,
+      do: Mix.raise("--timeout must be a positive number of ms")
+
+    opts
+  end
+
+  defp selftest(opts, deps) do
+    Mix.Task.run("app.config")
+    targets = select_devices!(opts, deps)
+    plugins = deps.plugins.()
+    restart? = Keyword.get(opts, :restart, true)
+
+    # Grants come before the (re)launch: a simulator may terminate an app
+    # whose privacy settings change, and a self-test must not meet a prompt.
+    if restart?, do: Enum.each(targets, &grant(&1, plugins, deps))
+
+    {connected, failed} =
+      deps.connect.(
+        cookie: opts[:cookie],
+        only: Enum.map(targets, & &1.serial),
+        platforms: targets |> Enum.map(& &1.platform) |> Enum.uniq(),
+        restart: restart?
+      )
+
+    run_opts = [plugins: plugins, timeout_ms: Keyword.get(opts, :timeout, 30_000)]
+    results = Enum.map(connected, &run_device(&1, run_opts, deps))
+
+    seen = MapSet.new(connected ++ failed, & &1.serial)
+    missing = Enum.reject(targets, &MapSet.member?(seen, &1.serial))
+
+    unreachable =
+      Enum.map(failed, &"#{label(&1)}: node not reachable (#{&1.error || &1.status})") ++
+        Enum.map(missing, &"#{label(&1)}: not found when connecting")
+
+    Enum.each(unreachable, &IO.puts("\n" <> &1))
+
+    failures =
+      for {device, entries} <- results,
+          %{plugin: plugin, result: {:fail, reason}} <- SelfTest.failures(entries),
+          do: "#{label(device)} #{plugin}: #{reason}"
+
+    case failures ++ unreachable do
+      [] -> IO.puts("\nmob.selftest passed on #{length(results)} device(s)")
+      problems -> Mix.raise("mob.selftest failed:\n  " <> Enum.join(problems, "\n  "))
+    end
+
+    :ok
+  end
+
+  defp select_devices!(opts, deps) do
+    platforms =
+      case Connect.resolve_platforms(opts, MobDev.Config.platforms()) do
+        {:ok, platforms} -> platforms
+        {:error, message} -> Mix.raise(message)
+      end
+
+    ids = Keyword.get_values(opts, :device) ++ Keyword.get_values(opts, :only)
+    opts = Keyword.put(opts, :leases, deps.leases.())
+
+    case TaskTargets.resolve(deps.discover.(platforms), ids, opts) do
+      {:ok, devices} -> devices
+      {:error, reason, context} -> Mix.raise(Deploy.target_error(reason, context, ids))
+    end
+  end
+
+  defp grant(%Device{} = device, plugins, deps) do
+    case deps.grant.(device, plugins) do
+      [] ->
+        :ok
+
+      grants ->
+        IO.puts("\n#{label(device)} permissions:")
+
+        Enum.each(grants, fn %{plugin: plugin, permission: perm, status: status} ->
+          case status do
+            :ok -> IO.puts("  granted #{perm} (#{plugin})")
+            {:error, out} -> IO.puts("  could not grant #{perm} (#{plugin}): #{out}")
+          end
+        end)
+    end
+  end
+
+  defp run_device(%Device{} = device, run_opts, deps) do
+    ctx = %{platform: device.platform, device: device.type || :physical}
+    IO.puts("\n#{label(device)} #{ctx.platform} #{ctx.device}")
+
+    entries = deps.run_all.(device.node, ctx, run_opts)
+    Enum.each(SelfTest.table(entries), &IO.puts("  " <> &1))
+    IO.puts("  " <> SelfTest.summary(entries))
+    {device, entries}
+  end
+
+  defp label(%Device{name: name, serial: serial}), do: "#{name || serial} (#{serial})"
+
+  defp default_deps do
+    %{
+      discover: &Deploy.discover_devices/1,
+      leases: &DeviceLeases.load/0,
+      grant: &grant_permissions/2,
+      connect: &Connector.connect_all/1,
+      plugins: &MobDev.Plugin.activated_with_verify/0,
+      run_all: &SelfTest.run_all/3
+    }
+  end
+
+  defp grant_permissions(%Device{platform: platform} = device, plugins) do
+    bundle_id =
+      case platform do
+        :ios -> MobDev.Config.ios_bundle_id()
+        :android -> MobDev.Config.bundle_id()
+      end
+
+    SelfTest.grant_permissions(device, plugins, bundle_id, fn exe, argv ->
+      System.cmd(exe, argv, stderr_to_stdout: true)
+    end)
+  end
+end