From f44f48322693d329b3768809471a661e5d052497 Mon Sep 17 00:00:00 2001 From: GenericJam Date: Thu, 8 Oct 2026 18:48:28 -0600 Subject: [PATCH 1/4] =?UTF-8?q?MOB-411:=20plugin=20self-tests=20=E2=80=94?= =?UTF-8?q?=20selftest=20manifest=20key,=20MobDev.Plugin.SelfTest.run=5Fal?= =?UTF-8?q?l/3,=20mix=20mob.selftest?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Manifest: selftest names an Elixir module; a loadable one must export run/1. mix mob.validate_plugin warns 'missing selftest' (an error in a later release, MOB-418). Runner: one :erpc call per activated plugin with a per-plugin timeout; raise/exit/throw/timeout/undef/off-contract returns become that plugin's {:fail, why}; plugins without a self-test are skips. Permissions from the manifests are pre-granted on emulators/simulators (pm grant, simctl privacy). mix mob.selftest selects devices like mob.connect, grants, relaunches, prints a table per device, exits non-zero on any FAIL or unreachable node. Scaffolds (tier 1, 4) generate a SelfTest. Decision: decisions/2026-10-08-plugin-self-tests-run-from-the-host.md --- CHANGELOG.md | 24 ++ README.md | 40 +++ ...-08-plugin-self-tests-run-from-the-host.md | 82 +++++ lib/mix/tasks/mob.selftest.ex | 190 +++++++++++ lib/mob_dev/plugin/manifest.ex | 33 +- lib/mob_dev/plugin/scaffold.ex | 61 +++- lib/mob_dev/plugin/self_test.ex | 291 +++++++++++++++++ lib/mob_dev/plugin/validator.ex | 17 +- test/mix/tasks/mob_selftest_test.exs | 194 +++++++++++ test/mob_dev/plugin/manifest_test.exs | 25 ++ test/mob_dev/plugin/self_test_test.exs | 303 ++++++++++++++++++ test/mob_dev/plugin/validator_test.exs | 14 +- 12 files changed, 1269 insertions(+), 5 deletions(-) create mode 100644 decisions/2026-10-08-plugin-self-tests-run-from-the-host.md create mode 100644 lib/mix/tasks/mob.selftest.ex create mode 100644 lib/mob_dev/plugin/self_test.ex create mode 100644 test/mix/tasks/mob_selftest_test.exs create mode 100644 test/mob_dev/plugin/self_test_test.exs diff --git a/CHANGELOG.md b/CHANGELOG.md index ac6be75..2c19d9e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,27 @@ +## [0.7.17] - 2026-10-08 + +### Added + +- **Plugin self-tests: `mix mob.selftest`, `MobDev.Plugin.SelfTest.run_all/3` + (MOB-411).** A plugin names a `Mob.Plugin.SelfTest` implementation in its + manifest as `selftest: Module` (mob 0.9.15). `mix mob.selftest` attaches to + the deployed app on each selected device (the `mix mob.connect` selection + options, plus `--timeout` and `--no-restart`), grants the permissions the + manifests declare on emulators and simulators (`adb shell pm grant`, + `xcrun simctl privacy grant`), relaunches the app, calls every activated + plugin's `run/1` over `:erpc` with + `%{platform:, device: :simulator | :emulator | :physical}` and prints a + table per device. A self-test that raises, exits, times out (30 s) or + returns outside the contract is a `FAIL`; a plugin without one is a `skip`. + Non-zero exit on any `FAIL` or unreachable device. `run_all/3` is the + runner for callers that hold a node already (mob_ci's invariant P12). +- **`selftest` manifest key.** `MobDev.Plugin.Manifest` accepts it (an + Elixir module; when the module is loadable it must export `run/1`). + `mix mob.validate_plugin` **warns** `missing selftest` on a manifest + without one; a later release turns that into an error once the first-party + plugins carry theirs (MOB-418). `mix mob.new_plugin` tier-1 and tier-4 + scaffolds now generate a `SelfTest` module and declare it. + ## [0.7.16] - 2026-10-06 ### Fixed diff --git a/README.md b/README.md index 2aafe45..a0f459f 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,7 @@ end | `mix mob.devices` | List connected devices and their status | | `mix mob.attest` | Prove a device is running the code you just pushed — compares module digests, not artifacts ([see below](#did-that-deploy-actually-land-mix-mobattest)) | | `mix mob.smoke` | Replay recorded `agent-device` UI flows on devices and check the app's own diagnostics held up ([see below](#smoke-flows-on-devices-mix-mobsmoke)) | +| `mix mob.selftest` | Run every activated plugin's `Mob.Plugin.SelfTest` on the app running on each device ([see below](#plugin-self-tests-on-devices-mix-mobselftest)) | | `mix mob.mutate` | Mutation-test the lines this branch changed: break the code on purpose and report what nothing noticed ([see below](#do-the-tests-guard-anything-mix-mobmutate)) | | `mix mob.push` | Hot-push only changed modules (no restart) | | `mix mob.enable ...` | Wire up an optional Mob feature — platform-manifest entries, Elixir stubs, dep injections ([see below](#mix-mobenable-feature)) | @@ -276,6 +277,45 @@ or a manually managed development profile for that bundle id and `.uitests`: an Xcode-managed wildcard team profile is rejected. Simulators and Android need none of this. +## Plugin self-tests on devices (`mix mob.selftest`) + +Every plugin can ship its own proof that it works on a device: a module +implementing `Mob.Plugin.SelfTest`, named in its manifest as `selftest: +Module` (see mob's `MOB_PLUGINS.md`, "Self-test"). `mix mob.selftest` +attaches to the deployed app on each selected device and calls each +activated plugin's `run/1` over distribution, one plugin at a time, with +`%{platform: :ios | :android, device: :simulator | :emulator | :physical}`. + +```bash +mix mob.selftest # every connected device +mix mob.selftest --device emulator-5554 --timeout 10000 +mix mob.selftest --ios-only --no-restart # attach to the app as it is +``` + +Before the tests, the permissions the manifests declare are granted on +emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy +grant`) so no system prompt stands between a self-test and its native code; +then the app is relaunched as `mix mob.connect` does, because a simulator +may terminate an app whose privacy settings changed. Each device gets a +table: + +``` +Pixel 8 (emulator-5554) android emulator + plugin outcome ms detail + mob_location pass 3 + mob_whisper pass 1 + mob_deliver pass 2 + mob_camera skip 0 needs_hardware + 3 passed, 0 failed, 1 skipped +``` + +A plugin without a self-test is a `skip` (so it is visible, not absent); a +self-test that raises, exits, times out (30 s by default) or returns +something outside the contract is a `FAIL` with the reason. The exit status +is non-zero on any `FAIL`, or when a selected device's node could not be +reached. `MobDev.Plugin.SelfTest.run_all/3` is the same runner for callers +that already hold a node, such as mob_ci. + ## Do the tests guard anything? (`mix mob.mutate`) A green suite says the tests ran, not that they guard anything. This changes diff --git a/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md b/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md new file mode 100644 index 0000000..3793fab --- /dev/null +++ b/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md @@ -0,0 +1,82 @@ +# Plugin self-tests run from the host: `mix mob.selftest` and `MobDev.Plugin.SelfTest.run_all/3` + +- Date: 2026-10-08 +- Status: accepted +- Linear: MOB-411 (epic MOB-410); the contract is mob's + `decisions/2026-10-08-plugin-self-test-contract.md` + +## Context + +- mob 0.9.15 gives plugins a behaviour, `Mob.Plugin.SelfTest`, with one + callback `run(ctx) :: :pass | {:fail, reason} | {:skip, reason}`, named in + the manifest as `selftest: Module`. The rule for a pass is a real answer + from the plugin's native code. +- Something has to validate the key, call the tests on a device and judge + the answers. mob_dev already owns manifest validation, device discovery + and attaching to a running app over distribution (`mix mob.smoke`, + `mix mob.attest`), and mob_ci calls into mob_dev for builds and the + static gate, so the runner belongs here and mob_ci calls it (invariant + P12). +- A self-test is plugin code running inside the app: it can raise, exit, + hang, or return `:ok` because its author misread the contract. None of + that may take the runner, or the other plugins' runs, down. + +## Decision + +- **`selftest` is a manifest key.** `MobDev.Plugin.Manifest` requires an + Elixir module; when the module is loadable at validation time (the + plugin is compiled: `mix mob.validate_plugin` after a compile, or a host + activating a dep) it must export `run/1`, otherwise only the name is + checked. A manifest without the key gets a **warning** from + `mix mob.validate_plugin` (`missing selftest`) in 0.7.17; it becomes an + error in a later release once the first-party plugins carry theirs + (MOB-418). Warning first because every published plugin would otherwise + stop validating the day this ships. +- **`run_all(node, ctx, opts)` is the runner.** One `:erpc.call/5` per + plugin, sequential (self-tests may touch shared hardware), each with its + own timeout (`:timeout_ms`, 30 s). Every way a remote `run/1` can go + wrong is caught on the host and becomes that plugin's `{:fail, why}`: + `error:{erpc, timeout}`, `{exception, Reason, Stack}` (raise; `:undef` + for a module the running build does not have, named as such), + `exit:{exception, _}` and `{signal, _}`, `throw`, and a return outside + the contract. The contract's three shapes are restated in the runner + rather than calling `Mob.Plugin.SelfTest.result?/1`, so mob_dev does not + need mob at runtime and the runner works against an app built with an + older mob. Plugins with no `selftest:` (or no manifest) are entries with + `module: nil` and a `{:skip, _}`: a plugin without a test is visible in + every table, not silently absent. +- **Permissions are pre-granted where the host can.** `grant_permissions/4` + runs `adb shell pm grant` for each `android.permissions` entry (only + runtime permissions are grantable; a normal one answers with an error + that is recorded, not raised) and `xcrun simctl privacy grant` for each + `permissions: [%{capability: _}]` whose capability has a simctl service + (location, microphone, photos, media, contacts, calendar, reminders, + motion). A physical iPhone gets nothing: there is no host-side grant. + `run_all/3` grants when given `:device` (mob_ci's path); `mix mob.selftest` + grants **before** it relaunches the app, because `simctl privacy` warns + that some changes terminate a running app, and a self-test must not meet + a system prompt. +- **`mix mob.selftest` selects devices like `mix mob.connect`** (`--device` + / `--only`, `--ios-only`, `--android-only`, `--cookie`) and relaunches + the app by default like it (`--no-restart` attaches as is). The context is + the device record's platform and type, not something the plugin detects: + a plugin deciding for itself that it is "on an emulator" is how a skip + hides a hole. Exit status is non-zero on any `FAIL` or on a selected + device whose node could not be reached (a test that could not run is not + a test that passed, the same rule as `mix mob.attest`). +- **Scaffolds declare one.** `mix mob.new_plugin` tier 1 generates a + `SelfTest` that calls the NIF's `ping/0`; tier 4 one that round-trips the + supervised worker. Tiers 2 and 3 have no honest host-callable probe in + the template and get the warning, which is the nudge intended. + +## Consequences + +- mob_ci replaces its `device_caps.exs` probe table with `run_all/3` once + the plugins carry self-tests (MOB-414, MOB-418). +- The tests (`test/mob_dev/plugin/self_test_test.exs`) use this node as the + device: `:erpc.call/5` to `node()` runs the fixture in a spawned local + process, so the raise, exit, throw, timeout, undef and off-contract paths + are exercised for real without a device. +- Turning the warning into an error is a one-line change in + `Validator.selftest_review/1` plus moving it into the error list; record + the release that does it here. diff --git a/lib/mix/tasks/mob.selftest.ex b/lib/mix/tasks/mob.selftest.ex new file mode 100644 index 0000000..f6a85a6 --- /dev/null +++ b/lib/mix/tasks/mob.selftest.ex @@ -0,0 +1,190 @@ +defmodule Mix.Tasks.Mob.Selftest do + @shortdoc "Run every activated plugin's self-test on the app running on each device" + + @moduledoc """ + Attach to the deployed app on each selected device and run every activated + plugin's `Mob.Plugin.SelfTest` on it, over distribution. + + mix mob.selftest # every connected device + mix mob.selftest --device emulator-5554 + mix mob.selftest --ios-only --timeout 10000 + + The app must be deployed (`mix mob.deploy`) with the plugins activated in + `mob.exs`; a self-test is only on the device if the build that is running + included it. Each plugin's `selftest: Module` from its manifest is called + with `%{platform: :ios | :android, device: :simulator | :emulator | + :physical}` and gets one line in a table per device: `pass`, `FAIL` with + the reason, or `skip` with why (a plugin without a self-test is a skip, + so it is visible). A self-test that raises, exits, times out or returns + something outside the contract is a `FAIL`. + + Before the tests run, the permissions the manifests declare are granted + on emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy + grant`), so no system prompt stands between a self-test and its native + code. The app is then relaunched (as `mix mob.connect` does), because a + simulator may terminate an app whose privacy settings changed; + `--no-restart` attaches to the app as it is. + + ## Options + + * `--device` / `--only` (`-d`) — target a device by serial/udid + (substring match). Repeatable + * `--ios-only` / `--android-only` — restrict discovery to one platform + * `--timeout MS` — per self-test, in milliseconds (default: 30000) + * `--no-restart` — do not relaunch the app before testing + * `--cookie C` — dist cookie (default: the app's private cookie) + + ## Exit status + + Non-zero when any self-test failed, or a selected device's node could not + be reached (a test that could not run is not a test that passed). + """ + + use Mix.Task + + alias Mix.Tasks.Mob.Connect + alias MobDev.{Connector, Device, TaskHelp} + alias MobDev.Plugin.SelfTest + + @switches [ + device: :keep, + only: :keep, + ios_only: :boolean, + android_only: :boolean, + timeout: :integer, + restart: :boolean, + cookie: :string + ] + + @impl Mix.Task + def run(args), do: run(args, %{}) + @doc false + # `deps` replaces the I/O: `:discover` (platforms -> devices, as + # `Mix.Tasks.Mob.Deploy.discover_devices/1`), `:grant` (device, plugins -> + # grants, as `SelfTest.grant_permissions/4` with the app's bundle id), + # `:connect` (opts -> {connected, failed} devices, as + # `MobDev.Connector.connect_all/1`), `:plugins` (-> activated plugins) and + # `:run_all` (node, ctx, opts -> entries, as `SelfTest.run_all/3`). + @spec run([String.t()], map()) :: :ok + def run(args, deps) do + if TaskHelp.help_requested?(args) do + TaskHelp.print_module_help(__MODULE__) + else + selftest(parse!(args), Map.merge(default_deps(), deps)) + end + end + + defp parse!(args) do + {opts, argv, invalid} = OptionParser.parse(args, strict: @switches, aliases: [d: :device]) + + unless invalid == [] and argv == [] do + bad = Enum.map(invalid, &elem(&1, 0)) ++ argv + Mix.raise("Unknown option(s) or argument(s): #{Enum.join(bad, ", ")}") + end + + if Keyword.get(opts, :timeout, 1) < 1, + do: Mix.raise("--timeout must be a positive number of ms") + + opts + end + + defp selftest(opts, deps) do + Mix.Task.run("app.config") + + platforms = + case Connect.resolve_platforms(opts, MobDev.Config.platforms()) do + {:ok, platforms} -> platforms + {:error, message} -> Mix.raise(message) + end + + only = Keyword.get_values(opts, :device) ++ Keyword.get_values(opts, :only) + plugins = deps.plugins.() + + targets = platforms |> deps.discover.() |> Connector.filter_only(only) + + if targets == [] do + Mix.raise("No devices found. Deploy the app first (mix mob.deploy), then run this again.") + end + + # Grants come before the (re)launch: a simulator may terminate an app + # whose privacy settings change, and a self-test must not meet a prompt. + Enum.each(targets, &grant(&1, plugins, deps)) + + {connected, failed} = + deps.connect.( + cookie: opts[:cookie], + only: Enum.map(targets, & &1.serial), + platforms: platforms, + restart: Keyword.get(opts, :restart, true) + ) + + run_opts = [plugins: plugins, timeout_ms: Keyword.get(opts, :timeout, 30_000)] + results = Enum.map(connected, &run_device(&1, run_opts, deps)) + + unreachable = Enum.map(failed, &"#{label(&1)}: node not reachable (#{&1.error || &1.status})") + Enum.each(unreachable, &IO.puts("\n" <> &1)) + + failures = + for {device, entries} <- results, + %{plugin: plugin, result: {:fail, reason}} <- entries, + do: "#{label(device)} #{plugin}: #{reason}" + + case failures ++ unreachable do + [] -> IO.puts("\nmob.selftest passed on #{length(results)} device(s)") + problems -> Mix.raise("mob.selftest failed:\n " <> Enum.join(problems, "\n ")) + end + + :ok + end + + defp grant(%Device{} = device, plugins, deps) do + case deps.grant.(device, plugins) do + [] -> + :ok + + grants -> + IO.puts("\n#{label(device)} permissions:") + + Enum.each(grants, fn %{plugin: plugin, permission: perm, status: status} -> + case status do + :ok -> IO.puts(" granted #{perm} (#{plugin})") + {:error, out} -> IO.puts(" could not grant #{perm} (#{plugin}): #{out}") + end + end) + end + end + + defp run_device(%Device{} = device, run_opts, deps) do + ctx = %{platform: device.platform, device: device.type || :physical} + IO.puts("\n#{label(device)} #{ctx.platform} #{ctx.device}") + + entries = deps.run_all.(device.node, ctx, run_opts) + Enum.each(SelfTest.table(entries), &IO.puts(" " <> &1)) + IO.puts(" " <> SelfTest.summary(entries)) + {device, entries} + end + + defp label(%Device{name: name, serial: serial}), do: "#{name || serial} (#{serial})" + + defp default_deps do + %{ + discover: &Mix.Tasks.Mob.Deploy.discover_devices/1, + grant: &grant_permissions/2, + connect: &Connector.connect_all/1, + plugins: &MobDev.Plugin.activated/0, + run_all: &SelfTest.run_all/3 + } + end + + defp grant_permissions(%Device{platform: platform} = device, plugins) do + bundle_id = + case platform do + :ios -> MobDev.Config.ios_bundle_id() + :android -> MobDev.Config.bundle_id() + end + + SelfTest.grant_permissions(device, plugins, bundle_id, fn exe, argv -> + System.cmd(exe, argv, stderr_to_stdout: true) + end) + end +end diff --git a/lib/mob_dev/plugin/manifest.ex b/lib/mob_dev/plugin/manifest.ex index bc10196..08d25a0 100644 --- a/lib/mob_dev/plugin/manifest.ex +++ b/lib/mob_dev/plugin/manifest.ex @@ -31,7 +31,7 @@ defmodule MobDev.Plugin.Manifest do name mob_version plugin_spec_version permissions android ios nifs nifs_generator screens screens_generator migrations assets default_font lifecycle settings notifications ui_components ui_components_generator - host_requirements description version tags host_config_keys setup + host_requirements description version tags host_config_keys setup selftest )a @native_sections [ @@ -153,6 +153,7 @@ defmodule MobDev.Plugin.Manifest do |> check_notifications(manifest) |> check_ui_components(manifest) |> check_host_requirements(manifest) + |> check_selftest(manifest) warn_unknown_keys(manifest) @@ -598,6 +599,36 @@ defmodule MobDev.Plugin.Manifest do defp check_lifecycle(errors, _), do: errors + # `:selftest` names the plugin's `Mob.Plugin.SelfTest` implementation. The + # module is only loadable when the plugin is compiled (`mix + # mob.validate_plugin` after a compile, or the host activating a dep); then + # it must export run/1. Otherwise the name is all there is to check. + defp check_selftest(errors, %{selftest: mod}) when is_atom(mod) and not is_nil(mod) do + cond do + not elixir_module?(mod) -> + ["selftest must be an Elixir module, got: #{inspect(mod)}" | errors] + + Code.ensure_loaded?(mod) and not function_exported?(mod, :run, 1) -> + [ + "selftest #{inspect(mod)} does not export run/1 (implement Mob.Plugin.SelfTest)" + | errors + ] + + true -> + errors + end + end + + defp check_selftest(errors, %{selftest: other}), + do: [ + "selftest must be a module implementing Mob.Plugin.SelfTest, got: #{inspect(other)}" + | errors + ] + + defp check_selftest(errors, _), do: errors + + defp elixir_module?(mod), do: String.starts_with?(Atom.to_string(mod), "Elixir.") + defp check_optional_mfa(errors, map, key) do case Map.get(map, key) do nil -> errors diff --git a/lib/mob_dev/plugin/scaffold.ex b/lib/mob_dev/plugin/scaffold.ex index 530b19b..7dac0fb 100644 --- a/lib/mob_dev/plugin/scaffold.ex +++ b/lib/mob_dev/plugin/scaffold.ex @@ -194,6 +194,7 @@ defmodule MobDev.Plugin.Scaffold do [ {"lib/#{name}.ex", tier1_lib(name, nif_name)}, + {"lib/#{name}/self_test.ex", tier1_self_test(name, nif_name)}, {"src/#{nif_name}.erl", tier1_erl_stub(nif_name)}, {"priv/mob_plugin.exs", tier1_manifest(name, nif_name, mob_req)}, {"priv/native/jni/#{nif_name}.c", tier1_c(nif_name)}, @@ -238,6 +239,7 @@ defmodule MobDev.Plugin.Scaffold do {"lib/#{name}/worker.ex", tier4_worker(mod)}, {"lib/#{name}/notifications.ex", tier4_notifications(mod)}, {"lib/#{name}/settings_screen.ex", tier4_settings_screen(mod)}, + {"lib/#{name}/self_test.ex", tier4_self_test(mod)}, {"priv/mob_plugin.exs", tier4_manifest(name, mod, mob_req)}, {"test/test_helper.exs", test_helper()}, {"test/#{name}_test.exs", plugin_test(name)} @@ -739,6 +741,29 @@ defmodule MobDev.Plugin.Scaffold do """ end + defp tier1_self_test(name, nif_name) do + mod = module_name(name) + + """ + defmodule #{mod}.SelfTest do + @moduledoc \"\"\" + The plugin's on-device proof (`Mob.Plugin.SelfTest`), run by + `mix mob.selftest` and mob_ci for every activated plugin. A pass means a + real answer came back from the NIF; keep it side-effect free and quick. + \"\"\" + @behaviour Mob.Plugin.SelfTest + + @impl true + def run(%{platform: platform}) do + case :#{nif_name}.ping() do + :ok -> :pass + other -> {:fail, "ping/0 on \#{platform} returned \#{inspect(other)}, expected :ok"} + end + end + end + """ + end + defp tier1_manifest(name, nif_name, mob_req) do """ %{ @@ -751,7 +776,9 @@ defmodule MobDev.Plugin.Scaffold do # module — ERL_NIF_INIT uses it as both the registered module name # and the static-init C symbol prefix. %{module: :#{nif_name}, native_dir: "priv/native/jni"} - ] + ], + # On-device proof, run by `mix mob.selftest` (see Mob.Plugin.SelfTest). + selftest: #{module_name(name)}.SelfTest } """ end @@ -1104,6 +1131,33 @@ defmodule MobDev.Plugin.Scaffold do """ end + defp tier4_self_test(mod) do + """ + defmodule #{mod}.SelfTest do + @moduledoc \"\"\" + The plugin's on-device proof (`Mob.Plugin.SelfTest`), run by + `mix mob.selftest` and mob_ci for every activated plugin. A pure-Elixir + plugin proves its real API path: here, that the supervised worker is up + under the host's plugin supervisor and answers a call. + \"\"\" + @behaviour Mob.Plugin.SelfTest + + @impl true + def run(_ctx) do + case Process.whereis(#{mod}.Worker) do + nil -> + {:fail, "#{mod}.Worker is not running (lifecycle.supervised did not start it)"} + + pid -> + # A round-trip through the worker, not just a pid lookup. + %{} = :sys.get_state(pid, 5_000) + :pass + end + end + end + """ + end + defp tier4_manifest(name, mod, mob_req) do """ %{ @@ -1138,7 +1192,10 @@ defmodule MobDev.Plugin.Scaffold do handlers: [ %{match: %{type: "#{name}"}, handler: {#{mod}.Notifications, :handle, 1}} ] - } + }, + + # On-device proof, run by `mix mob.selftest` (see Mob.Plugin.SelfTest). + selftest: #{mod}.SelfTest } """ end diff --git a/lib/mob_dev/plugin/self_test.ex b/lib/mob_dev/plugin/self_test.ex new file mode 100644 index 0000000..529c6c8 --- /dev/null +++ b/lib/mob_dev/plugin/self_test.ex @@ -0,0 +1,291 @@ +defmodule MobDev.Plugin.SelfTest do + @moduledoc """ + Runs every activated plugin's `Mob.Plugin.SelfTest` on a device. + + A plugin declares `selftest: Module` in its manifest; `run_all/3` calls + `Module.run/1` on the device's node over `:erpc`, one plugin at a time, + and returns one entry per activated plugin. `mix mob.selftest` prints + those as a table; mob_ci records them per nightly cell (invariant P12). + + The entry's `:result` is always one of the contract's three shapes: + + * `:pass` + * `{:fail, reason}` — also what a self-test that raised, exited, threw, + timed out, is not on the device or returned something outside the + contract gets. The reason says which. + * `{:skip, :needs_hardware | :needs_user | reason}` — including plugins + with no `selftest:` in their manifest (`module: nil`), so a plugin + without one is visible, not silently absent. + + Before the first call, when `:device` names an emulator or simulator, + the permissions the manifests declare are granted on the device + (`adb shell pm grant`, `xcrun simctl privacy grant`), so a self-test + reaches its native code without a system prompt. See `grant_permissions/4`. + """ + + alias MobDev.Device + + @default_timeout_ms 30_000 + + @typedoc "What a self-test is told about where it runs (`Mob.Plugin.SelfTest.ctx/0`)." + @type ctx :: %{platform: :ios | :android, device: :simulator | :emulator | :physical} + + @typedoc "One plugin's outcome. `:ms` is the wall time of the call on the host." + @type entry :: %{plugin: atom(), module: module() | nil, result: term(), ms: non_neg_integer()} + + @typedoc "A granted (or attempted) permission." + @type grant :: %{plugin: atom(), permission: String.t(), status: :ok | {:error, String.t()}} + + @doc """ + Runs the self-test of every activated plugin on `node` and returns an entry per plugin. + + Options: + + * `:plugins` — `[{name_or_dir, manifest | nil}]` as `MobDev.Plugin.activated/0` + returns (the default, read from the host project's `mob.exs` and deps). + * `:timeout_ms` — per self-test (default #{@default_timeout_ms}). + * `:device` — the `MobDev.Device` the node runs on; emulators and simulators + get the manifests' permissions granted first. `nil` grants nothing. + * `:cmd` — `fn exe, argv -> {output, status} end` for the grant commands + (default `System.cmd/2`). + * `:bundle_id` — the app id to grant to (default `MobDev.Config` for the + device's platform). + """ + @spec run_all(node(), ctx(), keyword()) :: [entry()] + def run_all(node, %{platform: _, device: _} = ctx, opts \\ []) do + plugins = Keyword.get_lazy(opts, :plugins, &MobDev.Plugin.activated/0) + timeout = Keyword.get(opts, :timeout_ms, @default_timeout_ms) + + case Keyword.get(opts, :device) do + nil -> :ok + device -> grant_permissions(device, plugins, bundle_id(opts, device), cmd(opts)) + end + + for {name_or_dir, manifest} <- plugins, + plugin = plugin_name(name_or_dir, manifest), + do: run_one(node, plugin, manifest, ctx, timeout) + end + + defp run_one(_node, plugin, nil, _ctx, _timeout), + do: %{plugin: plugin, module: nil, result: {:skip, "no manifest (tier-0 plugin)"}, ms: 0} + + defp run_one(node, plugin, manifest, ctx, timeout) do + case Map.get(manifest, :selftest) do + nil -> + %{plugin: plugin, module: nil, result: {:skip, "no selftest in manifest"}, ms: 0} + + module -> + {ms, result} = timed(fn -> call(node, module, ctx, timeout) end) + %{plugin: plugin, module: module, result: result, ms: ms} + end + end + + # Everything a remote run/1 can do wrong lands here as {:fail, why}, so a + # broken self-test never takes the runner (or the other plugins' runs) down. + defp call(node, module, ctx, timeout) do + normalize(:erpc.call(node, module, :run, [ctx], timeout), module) + catch + :error, {:erpc, :timeout} -> + {:fail, "timed out after #{timeout} ms"} + + :error, {:erpc, :noconnection} -> + {:fail, "node #{node} is not reachable"} + + :error, {:exception, :undef, [{^module, :run, _, _} | _]} -> + {:fail, + "#{inspect(module)}.run/1 is not on the device (deployed before the self-test was added?)"} + + :error, {:exception, reason, stack} -> + {:fail, "raised: " <> format_exception(reason, stack)} + + :exit, {:exception, reason} -> + {:fail, "exited: #{inspect(reason, limit: 20)}"} + + :exit, {:signal, reason} -> + {:fail, "killed: #{inspect(reason, limit: 20)}"} + + :throw, value -> + {:fail, "threw: #{inspect(value, limit: 20)}"} + end + + defp format_exception(reason, stack) do + exception = Exception.normalize(:error, reason, stack) + banner = Exception.format_banner(:error, exception, stack) + + case stack do + [frame | _] -> banner <> " at " <> Exception.format_stacktrace_entry(frame) + [] -> banner + end + end + + # The contract (`Mob.Plugin.SelfTest.result?/1`), restated here so the + # runner does not need mob at runtime. + defp normalize(:pass, _module), do: :pass + defp normalize({:fail, reason} = fail, _module) when is_binary(reason), do: fail + + defp normalize({:skip, reason} = skip, _module) when reason in [:needs_hardware, :needs_user], + do: skip + + defp normalize({:skip, reason} = skip, _module) when is_binary(reason), do: skip + + defp normalize(other, module), + do: + {:fail, + "#{inspect(module)}.run/1 returned #{inspect(other, limit: 20)}, " <> + "not :pass | {:fail, reason} | {:skip, reason}"} + + defp timed(fun) do + start = System.monotonic_time(:millisecond) + result = fun.() + {System.monotonic_time(:millisecond) - start, result} + end + + defp plugin_name(_name_or_dir, %{name: name}) when is_atom(name), do: name + defp plugin_name(name, _manifest) when is_atom(name), do: name + + defp plugin_name(dir, _manifest) when is_binary(dir), + do: dir |> Path.basename() |> String.to_atom() + + # ── permissions ─────────────────────────────────────────────────────────── + + # `xcrun simctl privacy` services for the capability atoms manifests declare + # under `permissions: [%{capability: ...}]`. Capabilities with no service + # (bluetooth, notifications, speech, ...) cannot be pre-granted on a + # simulator; their self-tests skip with :needs_user if a prompt would block. + @simctl_services %{ + location: "location", + microphone: "microphone", + photo_library: "photos", + photos: "photos", + media: "media-library", + contacts: "contacts", + calendar: "calendar", + reminders: "reminders", + motion: "motion" + } + + @doc """ + Grants the permissions the plugins' manifests declare to `bundle_id` on + `device`, so self-tests do not hit a system prompt. + + * Android (emulator or physical): each `android.permissions` entry via + `adb -s shell pm grant`. Only runtime permissions are + grantable; a normal or signature permission answers with an error, + which is recorded, not raised. + * iOS simulator: each `permissions: [%{capability: cap}]` whose + capability maps to a `simctl privacy` service. + * iOS physical device: nothing can be granted from the host; returns `[]`. + + Returns one `t:grant/0` per attempt. `cmd` is `fn exe, argv -> {output, status} end`. + """ + @spec grant_permissions(Device.t(), [{term(), map() | nil}], String.t(), (String.t(), + [String.t()] -> + {String.t(), + integer()})) :: + [grant()] + def grant_permissions(%Device{platform: :android, serial: serial}, plugins, bundle_id, cmd) do + for {plugin, perm} <- android_permissions(plugins) do + {out, status} = cmd.("adb", ["-s", serial, "shell", "pm", "grant", bundle_id, perm]) + %{plugin: plugin, permission: perm, status: status(out, status)} + end + end + + def grant_permissions( + %Device{platform: :ios, type: :simulator, serial: udid}, + plugins, + bundle_id, + cmd + ) do + for {plugin, service} <- simctl_services(plugins) do + {out, status} = cmd.("xcrun", ["simctl", "privacy", udid, "grant", service, bundle_id]) + %{plugin: plugin, permission: service, status: status(out, status)} + end + end + + def grant_permissions(%Device{platform: :ios}, _plugins, _bundle_id, _cmd), do: [] + + @doc false + @spec android_permissions([{term(), map() | nil}]) :: [{atom(), String.t()}] + def android_permissions(plugins) do + for {name_or_dir, %{} = manifest} <- plugins, + perm <- get_in(manifest, [:android, :permissions]) || [], + is_binary(perm), + do: {plugin_name(name_or_dir, manifest), perm} + end + + @doc false + @spec simctl_services([{term(), map() | nil}]) :: [{atom(), String.t()}] + def simctl_services(plugins) do + for {name_or_dir, %{} = manifest} <- plugins, + %{capability: cap} <- Map.get(manifest, :permissions) || [], + service = Map.get(@simctl_services, cap), + do: {plugin_name(name_or_dir, manifest), service} + end + + defp status(_out, 0), do: :ok + defp status(out, _status), do: {:error, out |> String.trim() |> String.slice(0, 200)} + + defp bundle_id(opts, device) do + Keyword.get_lazy(opts, :bundle_id, fn -> + case device.platform do + :ios -> MobDev.Config.ios_bundle_id() + :android -> MobDev.Config.bundle_id() + end + end) + end + + defp cmd(opts) do + Keyword.get(opts, :cmd, fn exe, argv -> System.cmd(exe, argv, stderr_to_stdout: true) end) + end + + # ── reporting ───────────────────────────────────────────────────────────── + + @doc """ + The table `mix mob.selftest` prints for one device: a header plus one + line per entry, `plugin outcome ms detail`. + """ + @spec table([entry()]) :: [String.t()] + def table(entries) do + rows = Enum.map(entries, &row/1) + widths = column_widths([["plugin", "outcome", "ms", "detail"] | rows]) + + for row <- [["plugin", "outcome", "ms", "detail"] | rows] do + row + |> Enum.zip(widths) + |> Enum.map_join(" ", fn {cell, width} -> String.pad_trailing(cell, width) end) + |> String.trim_trailing() + end + end + + defp row(%{plugin: plugin, result: result, ms: ms}) do + {outcome, detail} = describe(result) + [to_string(plugin), outcome, to_string(ms), detail] + end + + defp describe(:pass), do: {"pass", ""} + defp describe({:fail, reason}), do: {"FAIL", reason} + defp describe({:skip, reason}) when is_atom(reason), do: {"skip", to_string(reason)} + defp describe({:skip, reason}), do: {"skip", reason} + + defp column_widths(rows) do + rows + |> Enum.zip_with(fn column -> column |> Enum.map(&String.length/1) |> Enum.max() end) + end + + @doc "The entries whose result is `{:fail, _}`." + @spec failures([entry()]) :: [entry()] + def failures(entries), do: Enum.filter(entries, &match?(%{result: {:fail, _}}, &1)) + + @doc "One line: `N passed, N failed, N skipped`." + @spec summary([entry()]) :: String.t() + def summary(entries) do + counts = Enum.frequencies_by(entries, &outcome/1) + + "#{Map.get(counts, :pass, 0)} passed, #{Map.get(counts, :fail, 0)} failed, " <> + "#{Map.get(counts, :skip, 0)} skipped" + end + + defp outcome(%{result: :pass}), do: :pass + defp outcome(%{result: {:fail, _}}), do: :fail + defp outcome(%{result: {:skip, _}}), do: :skip +end diff --git a/lib/mob_dev/plugin/validator.ex b/lib/mob_dev/plugin/validator.ex index 8e730e7..a7bd036 100644 --- a/lib/mob_dev/plugin/validator.ex +++ b/lib/mob_dev/plugin/validator.ex @@ -631,7 +631,22 @@ defmodule MobDev.Plugin.Validator do defp warnings(manifest) do single_platform_components(manifest) ++ permission_review(manifest) ++ - plist_review(manifest) + plist_review(manifest) ++ + selftest_review(manifest) + end + + # A warning in 0.7.17 so published plugins keep validating; an error in a + # later release once the first-party plugins all carry one (MOB-418). + defp selftest_review(manifest) do + if Map.has_key?(manifest, :selftest) do + [] + else + [ + "missing selftest — declare `selftest: Module` (a Mob.Plugin.SelfTest) so " <> + "`mix mob.selftest` and mob_ci can prove the plugin works on a device; " <> + "a later mob_dev release makes this an error" + ] + end end defp single_platform_components(manifest) do diff --git a/test/mix/tasks/mob_selftest_test.exs b/test/mix/tasks/mob_selftest_test.exs new file mode 100644 index 0000000..aebd1db --- /dev/null +++ b/test/mix/tasks/mob_selftest_test.exs @@ -0,0 +1,194 @@ +defmodule Mix.Tasks.Mob.SelftestTest do + @moduledoc """ + `mix mob.selftest` driven end to end with fake discovery, grants, connect + and runner: the exit status has to come from the entries the runner + returns, and the grants have to happen before the app is relaunched. + """ + use ExUnit.Case, async: false + + import ExUnit.CaptureIO + + alias Mix.Tasks.Mob.Selftest, as: SelftestTask + alias MobDev.Device + + @emulator %Device{ + platform: :android, + serial: "emulator-5554", + name: "Pixel 8", + type: :emulator, + node: :"app_android_emulator_5554@127.0.0.1" + } + + @simulator %Device{ + platform: :ios, + serial: "2CAF98B3-FFAF-42D0-AA93-F62694E2BFE4", + name: "iPhone 17", + type: :simulator, + node: :"app_ios@127.0.0.1" + } + + @plugins [ + {:mob_location, + %{ + name: :mob_location, + selftest: MobLocation.SelfTest, + android: %{permissions: ["android.permission.ACCESS_FINE_LOCATION"]} + }}, + {:mob_deliver, %{name: :mob_deliver}} + ] + + defp entries(:pass), + do: [ + %{plugin: :mob_location, module: MobLocation.SelfTest, result: :pass, ms: 4}, + %{plugin: :mob_deliver, module: nil, result: {:skip, "no selftest in manifest"}, ms: 0} + ] + + defp entries(:fail), + do: [ + %{ + plugin: :mob_location, + module: MobLocation.SelfTest, + result: {:fail, "location_stop/0 returned {:error, :nif_not_loaded}"}, + ms: 2 + } + ] + + # Records every step in order so the test can check what ran before what. + defp deps(devices, outcome, opts \\ []) do + test = self() + failed = Keyword.get(opts, :failed, []) + + %{ + plugins: fn -> @plugins end, + discover: fn platforms -> + send(test, {:discover, platforms}) + devices + end, + grant: fn device, plugins -> + send(test, {:grant, device.serial}) + + MobDev.Plugin.SelfTest.android_permissions(plugins) + |> Enum.map(fn {p, perm} -> %{plugin: p, permission: perm, status: :ok} end) + end, + connect: fn connect_opts -> + send(test, {:connect, connect_opts}) + + {Enum.filter(devices, &(&1.serial in Keyword.fetch!(connect_opts, :only))) -- failed, + failed} + end, + run_all: fn node, ctx, run_opts -> + send(test, {:run_all, node, ctx, run_opts}) + entries(outcome) + end + } + end + + defp run_task(args, deps) do + capture_io(fn -> send(self(), {:result, SelftestTask.run(args, deps)}) end) + end + + test "passes, printing a table per device, when every self-test passed or skipped" do + output = run_task([], deps([@emulator], :pass)) + + assert_received {:result, :ok} + assert output =~ "Pixel 8 (emulator-5554) android emulator" + assert output =~ "mob_location pass 4" + assert output =~ "mob_deliver skip 0 no selftest in manifest" + assert output =~ "1 passed, 0 failed, 1 skipped" + assert output =~ "mob.selftest passed on 1 device(s)" + end + + test "fails (non-zero exit) naming the device and plugin when a self-test failed" do + error = + assert_raise Mix.Error, fn -> run_task([], deps([@emulator], :fail)) end + + assert error.message =~ "mob.selftest failed:" + + assert error.message =~ + "Pixel 8 (emulator-5554) mob_location: location_stop/0 returned {:error, :nif_not_loaded}" + end + + test "grants the manifests' permissions before relaunching, then runs with the device's context" do + run_task(["--timeout", "5000"], deps([@emulator], :pass)) + + # Order: grant, then connect (the relaunch), then run. + {:messages, messages} = Process.info(self(), :messages) + + kinds = + messages |> Enum.map(&elem(&1, 0)) |> Enum.filter(&(&1 in [:grant, :connect, :run_all])) + + assert kinds == [:grant, :connect, :run_all] + + assert_received {:discover, [:android, :ios]} + assert_received {:grant, "emulator-5554"} + assert_received {:connect, connect_opts} + assert_received {:run_all, :"app_android_emulator_5554@127.0.0.1", ctx, run_opts} + + assert Keyword.fetch!(connect_opts, :only) == ["emulator-5554"] + assert Keyword.fetch!(connect_opts, :restart) == true + assert ctx == %{platform: :android, device: :emulator} + assert Keyword.fetch!(run_opts, :timeout_ms) == 5000 + assert Keyword.fetch!(run_opts, :plugins) == @plugins + end + + test "--no-restart attaches to the running app and --device narrows the targets" do + output = run_task(["--no-restart", "-d", "2CAF"], deps([@emulator, @simulator], :pass)) + + assert_received {:result, :ok} + assert_received {:connect, connect_opts} + assert Keyword.fetch!(connect_opts, :restart) == false + assert Keyword.fetch!(connect_opts, :only) == [@simulator.serial] + assert_received {:run_all, :"app_ios@127.0.0.1", %{platform: :ios, device: :simulator}, _} + refute_received {:grant, "emulator-5554"} + assert output =~ "iPhone 17" + refute output =~ "Pixel 8" + end + + test "a selected device whose node could not be reached fails the run" do + down = %{@simulator | node: nil, status: :error, error: "no node after 30 s"} + + error = + assert_raise Mix.Error, fn -> + run_task([], deps([@emulator, @simulator], :pass, failed: [down])) + end + + assert error.message =~ + "iPhone 17 (#{@simulator.serial}): node not reachable (no node after 30 s)" + + # The reachable device still ran. + assert_received {:run_all, :"app_android_emulator_5554@127.0.0.1", _, _} + end + + test "prints the permissions it granted" do + output = run_task([], deps([@emulator], :pass)) + assert output =~ "Pixel 8 (emulator-5554) permissions:" + assert output =~ "granted android.permission.ACCESS_FINE_LOCATION (mob_location)" + end + + test "no devices is an error, not a pass" do + error = assert_raise Mix.Error, fn -> run_task([], deps([], :pass)) end + assert error.message =~ "No devices found" + refute_received {:run_all, _, _, _} + end + + test "rejects unknown options, a bad timeout and both platform flags" do + assert_raise Mix.Error, ~r/Unknown option/, fn -> + run_task(["--flows", "x"], deps([], :pass)) + end + + assert_raise Mix.Error, ~r/--timeout must be/, fn -> + run_task(["--timeout", "0"], deps([], :pass)) + end + + assert_raise Mix.Error, ~r/Cannot combine/, fn -> + run_task(["--ios-only", "--android-only"], deps([], :pass)) + end + end + + test "--help prints the usage and runs nothing" do + output = run_task(["--help"], deps([@emulator], :fail)) + assert output =~ "mix mob.selftest" + assert_received {:result, _} + refute_received {:discover, _} + end +end diff --git a/test/mob_dev/plugin/manifest_test.exs b/test/mob_dev/plugin/manifest_test.exs index 37783ce..c6a9a24 100644 --- a/test/mob_dev/plugin/manifest_test.exs +++ b/test/mob_dev/plugin/manifest_test.exs @@ -5,6 +5,11 @@ defmodule MobDev.Plugin.ManifestTest do @valid %{name: :mob_demo, mob_version: "~> 0.6", plugin_spec_version: 1} + defmodule ManifestTestSelfTest do + @moduledoc false + def run(_ctx), do: :pass + end + # ExUnit.CaptureLog hears every process's log events while `fun` runs, and # this module is async: a sibling test's "unknown key(s)" warning lands in # the capture and trips a `refute` (MOB-290). Manifest.validate/1 logs from @@ -187,6 +192,26 @@ defmodule MobDev.Plugin.ManifestTest do assert {:error, _} = Manifest.validate("nope") end + test "selftest names a module; a compiled one must export run/1" do + # Not compiled anywhere: only the name can be checked. + m = Map.put(@valid, :selftest, MobDemo.SelfTest) + log = capture_own_log(fn -> assert {:ok, ^m} = Manifest.validate(m) end) + refute log =~ "unknown key" + + # Loadable, and a real Mob.Plugin.SelfTest shape. + assert {:ok, _} = Manifest.validate(Map.put(@valid, :selftest, ManifestTestSelfTest)) + + # Loadable but not a self-test: the module exists, run/1 does not. + assert {:error, [err]} = Manifest.validate(Map.put(@valid, :selftest, Manifest)) + assert err =~ "does not export run/1" + + assert {:error, [err]} = Manifest.validate(Map.put(@valid, :selftest, "MobDemo.SelfTest")) + assert err =~ "selftest must be a module" + + assert {:error, [err]} = Manifest.validate(Map.put(@valid, :selftest, :mob_demo_nif)) + assert err =~ "must be an Elixir module" + end + test "accepts a permissions list with capability + optional ios handler" do m = Map.put(@valid, :permissions, [ diff --git a/test/mob_dev/plugin/self_test_test.exs b/test/mob_dev/plugin/self_test_test.exs new file mode 100644 index 0000000..f16de97 --- /dev/null +++ b/test/mob_dev/plugin/self_test_test.exs @@ -0,0 +1,303 @@ +defmodule MobDev.Plugin.SelfTestTest do + # The "device" is this node: `:erpc.call/5` to `node()` runs the self-test + # in a spawned local process exactly as it would on a phone, including the + # exception, exit and timeout paths. async: false — the crash fixtures log. + use ExUnit.Case, async: false + + import ExUnit.CaptureLog + + alias MobDev.Device + alias MobDev.Plugin.SelfTest + + @ctx %{platform: :android, device: :emulator} + + defmodule Passes do + @moduledoc false + def run(%{platform: :android}), do: :pass + end + + defmodule Fails do + @moduledoc false + def run(_ctx), do: {:fail, "status/0 returned :error"} + end + + defmodule Skips do + @moduledoc false + def run(%{device: :emulator}), do: {:skip, :needs_hardware} + end + + defmodule Raises do + @moduledoc false + def run(_ctx), do: raise(ArgumentError, "nif not loaded") + end + + defmodule Exits do + @moduledoc false + def run(_ctx), do: exit(:shutdown) + end + + defmodule Throws do + @moduledoc false + def run(_ctx), do: throw(:oops) + end + + defmodule Hangs do + @moduledoc false + def run(_ctx), do: Process.sleep(:infinity) + end + + defmodule OffContract do + @moduledoc false + def run(_ctx), do: :ok + end + + defp plugin(name, module), do: {name, %{name: name, selftest: module}} + + describe "run_all/3" do + test "maps pass, fail and skip through unchanged, with the plugin and module" do + entries = + SelfTest.run_all(node(), @ctx, + plugins: [plugin(:mob_a, Passes), plugin(:mob_b, Fails), plugin(:mob_c, Skips)] + ) + + assert [ + %{plugin: :mob_a, module: Passes, result: :pass}, + %{plugin: :mob_b, module: Fails, result: {:fail, "status/0 returned :error"}}, + %{plugin: :mob_c, module: Skips, result: {:skip, :needs_hardware}} + ] = entries + + assert Enum.all?(entries, &(is_integer(&1.ms) and &1.ms >= 0)) + end + + test "a raise, exit or throw in one self-test is that plugin's failure and the rest still run" do + log = + capture_log(fn -> + entries = + SelfTest.run_all(node(), @ctx, + plugins: [ + plugin(:mob_raise, Raises), + plugin(:mob_exit, Exits), + plugin(:mob_throw, Throws), + plugin(:mob_ok, Passes) + ] + ) + + assert [ + %{plugin: :mob_raise, result: {:fail, "raised: " <> raised}}, + %{plugin: :mob_exit, result: {:fail, "exited: :shutdown"}}, + %{plugin: :mob_throw, result: {:fail, "threw: :oops"}}, + %{plugin: :mob_ok, result: :pass} + ] = entries + + assert raised =~ "ArgumentError" + assert raised =~ "nif not loaded" + send(self(), :done) + end) + + assert_received :done + # The runner's own process survived every crash (the log is the remote one's). + refute log =~ "runner" + end + + test "a self-test that overruns the timeout fails with the timeout and does not block the next" do + {ms, entries} = + :timer.tc( + fn -> + SelfTest.run_all(node(), @ctx, + plugins: [plugin(:mob_slow, Hangs), plugin(:mob_ok, Passes)], + timeout_ms: 50 + ) + end, + :millisecond + ) + + assert [ + %{plugin: :mob_slow, result: {:fail, "timed out after 50 ms"}, ms: slow_ms}, + %{plugin: :mob_ok, result: :pass} + ] = entries + + assert slow_ms >= 50 + assert ms < 5_000 + end + + test "a module that is not on the device, or a return outside the contract, is a failure" do + entries = + SelfTest.run_all(node(), @ctx, + plugins: [plugin(:mob_stale, Not.Deployed.SelfTest), plugin(:mob_bad, OffContract)] + ) + + assert [ + %{plugin: :mob_stale, result: {:fail, stale}}, + %{plugin: :mob_bad, result: {:fail, bad}} + ] = entries + + assert stale =~ "Not.Deployed.SelfTest.run/1 is not on the device" + assert bad =~ "returned :ok, not :pass | {:fail, reason} | {:skip, reason}" + end + + test "plugins without a self-test (or without a manifest) are skips, not absent" do + entries = + SelfTest.run_all(node(), @ctx, + plugins: [ + {"/deps/mob_tier0", nil}, + {:mob_old, %{name: :mob_old, nifs: []}}, + plugin(:mob_ok, Passes) + ] + ) + + assert [ + %{plugin: :mob_tier0, module: nil, result: {:skip, "no manifest (tier-0 plugin)"}}, + %{plugin: :mob_old, module: nil, result: {:skip, "no selftest in manifest"}}, + %{plugin: :mob_ok, result: :pass} + ] = entries + end + + test "an unreachable node is every plugin's failure, not a crash" do + entries = + SelfTest.run_all(:"nope@127.0.0.1", @ctx, + plugins: [plugin(:mob_a, Passes)], + timeout_ms: 500 + ) + + assert [%{plugin: :mob_a, result: {:fail, reason}}] = entries + assert reason =~ "nope@127.0.0.1 is not reachable" + end + + test "grants the manifests' permissions first when a device is given" do + test = self() + + cmd = fn exe, argv -> + send(test, {:cmd, exe, argv}) + {"", 0} + end + + plugins = [ + {:mob_location, + %{ + name: :mob_location, + selftest: Passes, + android: %{permissions: ["android.permission.ACCESS_FINE_LOCATION"]} + }} + ] + + device = %Device{platform: :android, serial: "emulator-5554", type: :emulator} + + assert [%{result: :pass}] = + SelfTest.run_all(node(), @ctx, + plugins: plugins, + device: device, + bundle_id: "com.example.host", + cmd: cmd + ) + + assert_received {:cmd, "adb", + [ + "-s", + "emulator-5554", + "shell", + "pm", + "grant", + "com.example.host", + "android.permission.ACCESS_FINE_LOCATION" + ]} + end + end + + describe "grant_permissions/4" do + @plugins [ + {:mob_location, + %{ + name: :mob_location, + permissions: [%{capability: :location, ios: %{handler: "x"}}], + android: %{ + permissions: [ + "android.permission.ACCESS_FINE_LOCATION", + "android.permission.ACCESS_COARSE_LOCATION" + ] + } + }}, + {:mob_whisper, + %{ + name: :mob_whisper, + permissions: [%{capability: :speech}], + android: %{permissions: ["android.permission.RECORD_AUDIO"]} + }}, + {"/deps/mob_deliver", %{name: :mob_deliver, nifs: []}}, + {"/deps/mob_tier0", nil} + ] + + test "android: pm grant per declared permission, recording what the device said" do + cmd = fn "adb", ["-s", "emulator-5554", "shell", "pm", "grant", "com.x.app", perm] -> + if perm =~ "COARSE", + do: {"Operation not allowed: not a changeable permission type", 255}, + else: {"", 0} + end + + device = %Device{platform: :android, serial: "emulator-5554", type: :emulator} + + assert [ + %{ + plugin: :mob_location, + permission: "android.permission.ACCESS_FINE_LOCATION", + status: :ok + }, + %{ + plugin: :mob_location, + permission: "android.permission.ACCESS_COARSE_LOCATION", + status: {:error, "Operation not allowed: not a changeable permission type"} + }, + %{plugin: :mob_whisper, permission: "android.permission.RECORD_AUDIO", status: :ok} + ] = SelfTest.grant_permissions(device, @plugins, "com.x.app", cmd) + end + + test "ios simulator: simctl privacy grant for capabilities that have a service" do + cmd = fn "xcrun", ["simctl", "privacy", "UDID-1", "grant", _service, "com.x.app"] -> + {"", 0} + end + + device = %Device{platform: :ios, serial: "UDID-1", type: :simulator} + + # :speech has no simctl service, so only location is attempted. + assert [%{plugin: :mob_location, permission: "location", status: :ok}] = + SelfTest.grant_permissions(device, @plugins, "com.x.app", cmd) + end + + test "ios physical device: nothing can be granted from the host" do + cmd = fn _exe, _argv -> flunk("no command expected") end + device = %Device{platform: :ios, serial: "00008030-ABCDEF", type: :physical} + assert SelfTest.grant_permissions(device, @plugins, "com.x.app", cmd) == [] + end + end + + describe "table/1 and summary/1" do + test "one aligned row per plugin with the outcome and its detail" do + entries = [ + %{plugin: :mob_location, module: Passes, result: :pass, ms: 12}, + %{ + plugin: :mob_whisper, + module: Fails, + result: {:fail, "nif_loaded/0 returned false"}, + ms: 3 + }, + %{plugin: :mob_deliver, module: nil, result: {:skip, "no selftest in manifest"}, ms: 0}, + %{plugin: :mob_camera, module: Skips, result: {:skip, :needs_hardware}, ms: 1} + ] + + assert SelfTest.table(entries) == [ + "plugin outcome ms detail", + "mob_location pass 12", + "mob_whisper FAIL 3 nif_loaded/0 returned false", + "mob_deliver skip 0 no selftest in manifest", + "mob_camera skip 1 needs_hardware" + ] + + assert SelfTest.summary(entries) == "1 passed, 1 failed, 2 skipped" + assert [%{plugin: :mob_whisper}] = SelfTest.failures(entries) + end + + test "an empty run is an empty table body and a zero summary" do + assert SelfTest.table([]) == ["plugin outcome ms detail"] + assert SelfTest.summary([]) == "0 passed, 0 failed, 0 skipped" + end + end +end diff --git a/test/mob_dev/plugin/validator_test.exs b/test/mob_dev/plugin/validator_test.exs index bf6c9bf..a8558db 100644 --- a/test/mob_dev/plugin/validator_test.exs +++ b/test/mob_dev/plugin/validator_test.exs @@ -3,7 +3,12 @@ defmodule MobDev.Plugin.ValidatorTest do alias MobDev.Plugin.Validator - @base %{name: :mob_demo, mob_version: "~> 0.6", plugin_spec_version: 1} + @base %{ + name: :mob_demo, + mob_version: "~> 0.6", + plugin_spec_version: 1, + selftest: MobDemo.SelfTest + } describe "referenced_paths/1" do test "nil manifest has no paths" do @@ -85,6 +90,13 @@ defmodule MobDev.Plugin.ValidatorTest do assert %{warnings: []} = Validator.validate_plugin(m, dir, "0.6.20") end + test "warns, but does not fail, a manifest without a selftest", %{dir: dir} do + m = Map.delete(@base, :selftest) + assert %{errors: [], warnings: [warning]} = Validator.validate_plugin(m, dir, "0.6.20") + assert warning =~ "missing selftest" + assert warning =~ "Mob.Plugin.SelfTest" + end + test "warns when permissions are declared", %{dir: dir} do m = Map.put(@base, :android, %{permissions: ["android.permission.CAMERA"]}) assert %{warnings: warns} = Validator.validate_plugin(m, dir, "0.6.20") From eecdf094fce93f3b3e3a11b0fcabc0975bc5e3b9 Mon Sep 17 00:00:00 2001 From: GenericJam Date: Thu, 8 Oct 2026 19:10:05 -0600 Subject: [PATCH 2/4] =?UTF-8?q?MOB-411:=20review=20fixes=20=E2=80=94=20run?= =?UTF-8?q?=5Fall=20never=20grants=20(callers=20grant=20before=20launch),?= =?UTF-8?q?=20killable=20timeouts,=20lease-aware=20selection,=20lost=20dev?= =?UTF-8?q?ices=20fail=20the=20run?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - run_all/3 drops :device/:cmd/:bundle_id: a grant on a running simulator app can terminate it. grant_permissions/4 is the pre-launch grant; the task grants only when it relaunches (--no-restart grants nothing); no grants on physical devices. - The remote run/1 is spawned via spawn_request of :erpc.execute_call/4 with the pid in hand, so a timed-out test is killed, not abandoned; a non-module selftest:, a failed manifest verification and every erpc error become that plugin's {:fail, _}; a nil name falls back to the dir. - mix mob.selftest selects like mob.smoke (TaskTargets + leases, --all-devices/--all-physical); a target connect_all no longer finds is unreachable, so a run cannot pass on zero devices. - Tests for every reverted-behaviour gap the review listed, plus scaffold self-test assertions. --- CHANGELOG.md | 21 +- README.md | 11 +- ...-08-plugin-self-tests-run-from-the-host.md | 38 ++- lib/mix/tasks/mob.selftest.ex | 88 ++++--- lib/mob_dev/plugin/self_test.ex | 238 +++++++++++------- test/mix/tasks/mob_selftest_test.exs | 90 +++++-- test/mob_dev/plugin/scaffold_test.exs | 22 ++ test/mob_dev/plugin/self_test_test.exs | 199 +++++++-------- 8 files changed, 436 insertions(+), 271 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c19d9e..d961590 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,16 +5,19 @@ - **Plugin self-tests: `mix mob.selftest`, `MobDev.Plugin.SelfTest.run_all/3` (MOB-411).** A plugin names a `Mob.Plugin.SelfTest` implementation in its manifest as `selftest: Module` (mob 0.9.15). `mix mob.selftest` attaches to - the deployed app on each selected device (the `mix mob.connect` selection - options, plus `--timeout` and `--no-restart`), grants the permissions the - manifests declare on emulators and simulators (`adb shell pm grant`, - `xcrun simctl privacy grant`), relaunches the app, calls every activated - plugin's `run/1` over `:erpc` with + the deployed app on each selected device (the `mix mob.smoke` selection + options, honouring agent-device leases, plus `--timeout` and + `--no-restart`), grants the permissions the manifests declare on + emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy + grant`), relaunches the app, calls every activated plugin's `run/1` on + the device with `%{platform:, device: :simulator | :emulator | :physical}` and prints a - table per device. A self-test that raises, exits, times out (30 s) or - returns outside the contract is a `FAIL`; a plugin without one is a `skip`. - Non-zero exit on any `FAIL` or unreachable device. `run_all/3` is the - runner for callers that hold a node already (mob_ci's invariant P12). + table per device. A self-test that raises, exits, times out (30 s; it is + then killed on the device) or returns outside the contract is a `FAIL`; + a plugin without one is a `skip`. Non-zero exit on any `FAIL` or + unreachable device. `run_all/3` is the runner for callers that hold a + node already (mob_ci's invariant P12); `grant_permissions/4` is the + pre-launch grant for them. - **`selftest` manifest key.** `MobDev.Plugin.Manifest` accepts it (an Elixir module; when the module is loadable it must export `run/1`). `mix mob.validate_plugin` **warns** `missing selftest` on a manifest diff --git a/README.md b/README.md index a0f459f..0b57a24 100644 --- a/README.md +++ b/README.md @@ -287,9 +287,9 @@ activated plugin's `run/1` over distribution, one plugin at a time, with `%{platform: :ios | :android, device: :simulator | :emulator | :physical}`. ```bash -mix mob.selftest # every connected device +mix mob.selftest # the one connected emulator/simulator mix mob.selftest --device emulator-5554 --timeout 10000 -mix mob.selftest --ios-only --no-restart # attach to the app as it is +mix mob.selftest --all-devices --no-restart # attach to the apps as they are ``` Before the tests, the permissions the manifests declare are granted on @@ -313,8 +313,11 @@ A plugin without a self-test is a `skip` (so it is visible, not absent); a self-test that raises, exits, times out (30 s by default) or returns something outside the contract is a `FAIL` with the reason. The exit status is non-zero on any `FAIL`, or when a selected device's node could not be -reached. `MobDev.Plugin.SelfTest.run_all/3` is the same runner for callers -that already hold a node, such as mob_ci. +reached. Device selection follows `mix mob.smoke` (`--device`, +`--all-devices`, `--all-physical`, agent-device leases honoured). +`MobDev.Plugin.SelfTest.run_all/3` is the same runner for callers that +already hold a node, such as mob_ci, and `grant_permissions/4` the +pre-launch grant. ## Do the tests guard anything? (`mix mob.mutate`) diff --git a/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md b/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md index 3793fab..dbe975f 100644 --- a/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md +++ b/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md @@ -34,12 +34,16 @@ stop validating the day this ships. - **`run_all(node, ctx, opts)` is the runner.** One `:erpc.call/5` per plugin, sequential (self-tests may touch shared hardware), each with its - own timeout (`:timeout_ms`, 30 s). Every way a remote `run/1` can go - wrong is caught on the host and becomes that plugin's `{:fail, why}`: - `error:{erpc, timeout}`, `{exception, Reason, Stack}` (raise; `:undef` - for a module the running build does not have, named as such), - `exit:{exception, _}` and `{signal, _}`, `throw`, and a return outside - the contract. The contract's three shapes are restated in the runner + own timeout (`:timeout_ms`, 30 s). The remote `run/1` is spawned the way + `:erpc.call/5` spawns it (`spawn_request` of `:erpc.execute_call/4`, which + reports the return or the exception as its exit reason) but with the pid + in hand, so a test that overruns the deadline is **killed** on the + device rather than abandoned beside the next plugin's run. Every way it + can go wrong becomes that plugin's `{:fail, why}`: timeout, raise + (`:undef` for a module the running build does not have, named as such), + exit, throw, a kill signal, a lost connection, a non-module `selftest:` + value, a manifest that failed signature verification, and a return + outside the contract. The contract's three shapes are restated in the runner rather than calling `Mob.Plugin.SelfTest.result?/1`, so mob_dev does not need mob at runtime and the runner works against an app built with an older mob. Plugins with no `selftest:` (or no manifest) are entries with @@ -52,13 +56,21 @@ `permissions: [%{capability: _}]` whose capability has a simctl service (location, microphone, photos, media, contacts, calendar, reminders, motion). A physical iPhone gets nothing: there is no host-side grant. - `run_all/3` grants when given `:device` (mob_ci's path); `mix mob.selftest` - grants **before** it relaunches the app, because `simctl privacy` warns - that some changes terminate a running app, and a self-test must not meet - a system prompt. -- **`mix mob.selftest` selects devices like `mix mob.connect`** (`--device` - / `--only`, `--ios-only`, `--android-only`, `--cookie`) and relaunches - the app by default like it (`--no-restart` attaches as is). The context is + `run_all/3` never grants: it holds a node, so the app is already running, + and `simctl privacy` warns that some changes terminate a running app. + Granting is the caller's job before launch: `mix mob.selftest` grants, + then relaunches (`--no-restart` grants nothing); mob_ci calls + `grant_permissions/4` before it starts the app. (The first draft had + `run_all/3` grant when given `:device`; the pre-merge review found that + it could kill the simulator app it was about to test.) Physical devices + get nothing granted; their self-tests skip with `:needs_user`. +- **`mix mob.selftest` selects devices like `mix mob.smoke`** + (`MobDev.TaskTargets.resolve/3` with the agent-device leases: `--device` + / `--only`, `--all-devices`, `--all-physical`, `--ios-only`, + `--android-only`; a device another session holds is left alone unless + named) and relaunches the app by default like `mix mob.connect` + (`--no-restart` attaches as is). A target that `connect_all/1` no longer + finds is reported as unreachable, so a run never passes on zero devices. The context is the device record's platform and type, not something the plugin detects: a plugin deciding for itself that it is "on an emulator" is how a skip hides a hole. Exit status is non-zero on any `FAIL` or on a selected diff --git a/lib/mix/tasks/mob.selftest.ex b/lib/mix/tasks/mob.selftest.ex index f6a85a6..02a748b 100644 --- a/lib/mix/tasks/mob.selftest.ex +++ b/lib/mix/tasks/mob.selftest.ex @@ -5,9 +5,9 @@ defmodule Mix.Tasks.Mob.Selftest do Attach to the deployed app on each selected device and run every activated plugin's `Mob.Plugin.SelfTest` on it, over distribution. - mix mob.selftest # every connected device + mix mob.selftest # the one connected emulator/simulator mix mob.selftest --device emulator-5554 - mix mob.selftest --ios-only --timeout 10000 + mix mob.selftest --all-devices --timeout 10000 The app must be deployed (`mix mob.deploy`) with the plugins activated in `mob.exs`; a self-test is only on the device if the build that is running @@ -22,18 +22,24 @@ defmodule Mix.Tasks.Mob.Selftest do on emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy grant`), so no system prompt stands between a self-test and its native code. The app is then relaunched (as `mix mob.connect` does), because a - simulator may terminate an app whose privacy settings changed; - `--no-restart` attaches to the app as it is. + simulator may terminate an app whose privacy settings changed. + `--no-restart` attaches to the app as it is and grants nothing. ## Options * `--device` / `--only` (`-d`) — target a device by serial/udid - (substring match). Repeatable + (serial, udid or short id). Repeatable + * `--all-devices` — every emulator and simulator + * `--all-physical` — every physical device (with `--all-devices`: everything) * `--ios-only` / `--android-only` — restrict discovery to one platform * `--timeout MS` — per self-test, in milliseconds (default: 30000) - * `--no-restart` — do not relaunch the app before testing + * `--no-restart` — do not relaunch the app before testing (and do not + grant permissions) * `--cookie C` — dist cookie (default: the app's private cookie) + With no selection flags exactly one emulator/simulator is picked, and a + device another `agent-device` session holds is left alone unless named. + ## Exit status Non-zero when any self-test failed, or a selected device's node could not @@ -43,12 +49,15 @@ defmodule Mix.Tasks.Mob.Selftest do use Mix.Task alias Mix.Tasks.Mob.Connect - alias MobDev.{Connector, Device, TaskHelp} + alias Mix.Tasks.Mob.Deploy + alias MobDev.{Connector, Device, DeviceLeases, TaskHelp, TaskTargets} alias MobDev.Plugin.SelfTest @switches [ device: :keep, only: :keep, + all_devices: :boolean, + all_physical: :boolean, ios_only: :boolean, android_only: :boolean, timeout: :integer, @@ -58,13 +67,15 @@ defmodule Mix.Tasks.Mob.Selftest do @impl Mix.Task def run(args), do: run(args, %{}) + @doc false # `deps` replaces the I/O: `:discover` (platforms -> devices, as - # `Mix.Tasks.Mob.Deploy.discover_devices/1`), `:grant` (device, plugins -> - # grants, as `SelfTest.grant_permissions/4` with the app's bundle id), - # `:connect` (opts -> {connected, failed} devices, as - # `MobDev.Connector.connect_all/1`), `:plugins` (-> activated plugins) and - # `:run_all` (node, ctx, opts -> entries, as `SelfTest.run_all/3`). + # `Mix.Tasks.Mob.Deploy.discover_devices/1`), `:leases` (-> the agent-device + # claims), `:grant` (device, plugins -> grants, as + # `SelfTest.grant_permissions/4` with the app's bundle id), `:connect` + # (opts -> {connected, failed} devices, as `MobDev.Connector.connect_all/1`), + # `:plugins` (-> activated plugins) and `:run_all` (node, ctx, opts -> + # entries, as `SelfTest.run_all/3`). @spec run([String.t()], map()) :: :ok def run(args, deps) do if TaskHelp.help_requested?(args) do @@ -90,43 +101,37 @@ defmodule Mix.Tasks.Mob.Selftest do defp selftest(opts, deps) do Mix.Task.run("app.config") - - platforms = - case Connect.resolve_platforms(opts, MobDev.Config.platforms()) do - {:ok, platforms} -> platforms - {:error, message} -> Mix.raise(message) - end - - only = Keyword.get_values(opts, :device) ++ Keyword.get_values(opts, :only) + targets = select_devices!(opts, deps) plugins = deps.plugins.() - - targets = platforms |> deps.discover.() |> Connector.filter_only(only) - - if targets == [] do - Mix.raise("No devices found. Deploy the app first (mix mob.deploy), then run this again.") - end + restart? = Keyword.get(opts, :restart, true) # Grants come before the (re)launch: a simulator may terminate an app # whose privacy settings change, and a self-test must not meet a prompt. - Enum.each(targets, &grant(&1, plugins, deps)) + if restart?, do: Enum.each(targets, &grant(&1, plugins, deps)) {connected, failed} = deps.connect.( cookie: opts[:cookie], only: Enum.map(targets, & &1.serial), - platforms: platforms, - restart: Keyword.get(opts, :restart, true) + platforms: targets |> Enum.map(& &1.platform) |> Enum.uniq(), + restart: restart? ) run_opts = [plugins: plugins, timeout_ms: Keyword.get(opts, :timeout, 30_000)] results = Enum.map(connected, &run_device(&1, run_opts, deps)) - unreachable = Enum.map(failed, &"#{label(&1)}: node not reachable (#{&1.error || &1.status})") + seen = MapSet.new(connected ++ failed, & &1.serial) + missing = Enum.reject(targets, &MapSet.member?(seen, &1.serial)) + + unreachable = + Enum.map(failed, &"#{label(&1)}: node not reachable (#{&1.error || &1.status})") ++ + Enum.map(missing, &"#{label(&1)}: not found when connecting") + Enum.each(unreachable, &IO.puts("\n" <> &1)) failures = for {device, entries} <- results, - %{plugin: plugin, result: {:fail, reason}} <- entries, + %{plugin: plugin, result: {:fail, reason}} <- SelfTest.failures(entries), do: "#{label(device)} #{plugin}: #{reason}" case failures ++ unreachable do @@ -137,6 +142,22 @@ defmodule Mix.Tasks.Mob.Selftest do :ok end + defp select_devices!(opts, deps) do + platforms = + case Connect.resolve_platforms(opts, MobDev.Config.platforms()) do + {:ok, platforms} -> platforms + {:error, message} -> Mix.raise(message) + end + + ids = Keyword.get_values(opts, :device) ++ Keyword.get_values(opts, :only) + opts = Keyword.put(opts, :leases, deps.leases.()) + + case TaskTargets.resolve(deps.discover.(platforms), ids, opts) do + {:ok, devices} -> devices + {:error, reason, context} -> Mix.raise(Deploy.target_error(reason, context, ids)) + end + end + defp grant(%Device{} = device, plugins, deps) do case deps.grant.(device, plugins) do [] -> @@ -168,10 +189,11 @@ defmodule Mix.Tasks.Mob.Selftest do defp default_deps do %{ - discover: &Mix.Tasks.Mob.Deploy.discover_devices/1, + discover: &Deploy.discover_devices/1, + leases: &DeviceLeases.load/0, grant: &grant_permissions/2, connect: &Connector.connect_all/1, - plugins: &MobDev.Plugin.activated/0, + plugins: &MobDev.Plugin.activated_with_verify/0, run_all: &SelfTest.run_all/3 } end diff --git a/lib/mob_dev/plugin/self_test.ex b/lib/mob_dev/plugin/self_test.ex index 529c6c8..535728c 100644 --- a/lib/mob_dev/plugin/self_test.ex +++ b/lib/mob_dev/plugin/self_test.ex @@ -3,24 +3,25 @@ defmodule MobDev.Plugin.SelfTest do Runs every activated plugin's `Mob.Plugin.SelfTest` on a device. A plugin declares `selftest: Module` in its manifest; `run_all/3` calls - `Module.run/1` on the device's node over `:erpc`, one plugin at a time, - and returns one entry per activated plugin. `mix mob.selftest` prints - those as a table; mob_ci records them per nightly cell (invariant P12). + `Module.run/1` on the device's node, one plugin at a time, and returns one + entry per activated plugin. `mix mob.selftest` prints those as a table; + mob_ci records them per nightly cell (invariant P12). The entry's `:result` is always one of the contract's three shapes: * `:pass` * `{:fail, reason}` — also what a self-test that raised, exited, threw, timed out, is not on the device or returned something outside the - contract gets. The reason says which. + contract gets, and what a plugin whose manifest failed signature + verification gets. The reason says which. * `{:skip, :needs_hardware | :needs_user | reason}` — including plugins with no `selftest:` in their manifest (`module: nil`), so a plugin without one is visible, not silently absent. - Before the first call, when `:device` names an emulator or simulator, - the permissions the manifests declare are granted on the device - (`adb shell pm grant`, `xcrun simctl privacy grant`), so a self-test - reaches its native code without a system prompt. See `grant_permissions/4`. + Permissions are not granted here: the node is already running, and on an + iOS simulator a privacy change can terminate the app. Callers grant + **before** launching the app with `grant_permissions/4`, as + `mix mob.selftest` does. """ alias MobDev.Device @@ -33,6 +34,13 @@ defmodule MobDev.Plugin.SelfTest do @typedoc "One plugin's outcome. `:ms` is the wall time of the call on the host." @type entry :: %{plugin: atom(), module: module() | nil, result: term(), ms: non_neg_integer()} + @typedoc """ + An activated plugin as `MobDev.Plugin.activated_with_verify/0` (3-tuple) or + `activated/0` (2-tuple) lists them; the first element is the plugin's name + or its dependency directory. + """ + @type plugin :: {atom() | Path.t(), map() | nil} | {atom() | Path.t(), map() | nil, term()} + @typedoc "A granted (or attempted) permission." @type grant :: %{plugin: atom(), permission: String.t(), status: :ok | {:error, String.t()}} @@ -41,72 +49,121 @@ defmodule MobDev.Plugin.SelfTest do Options: - * `:plugins` — `[{name_or_dir, manifest | nil}]` as `MobDev.Plugin.activated/0` - returns (the default, read from the host project's `mob.exs` and deps). - * `:timeout_ms` — per self-test (default #{@default_timeout_ms}). - * `:device` — the `MobDev.Device` the node runs on; emulators and simulators - get the manifests' permissions granted first. `nil` grants nothing. - * `:cmd` — `fn exe, argv -> {output, status} end` for the grant commands - (default `System.cmd/2`). - * `:bundle_id` — the app id to grant to (default `MobDev.Config` for the - device's platform). + * `:plugins` — `t:plugin/0` list (default: + `MobDev.Plugin.activated_with_verify/0`, read from the host project's + `mob.exs` and deps). A plugin whose manifest failed verification is a + `{:fail, _}` entry; one with no manifest (tier 0) is a skip. + * `:timeout_ms` — per self-test (default #{@default_timeout_ms}). A test + still running at the deadline is killed on the device, so the next + plugin's test never runs beside it. """ @spec run_all(node(), ctx(), keyword()) :: [entry()] def run_all(node, %{platform: _, device: _} = ctx, opts \\ []) do - plugins = Keyword.get_lazy(opts, :plugins, &MobDev.Plugin.activated/0) + plugins = Keyword.get_lazy(opts, :plugins, &MobDev.Plugin.activated_with_verify/0) timeout = Keyword.get(opts, :timeout_ms, @default_timeout_ms) - case Keyword.get(opts, :device) do - nil -> :ok - device -> grant_permissions(device, plugins, bundle_id(opts, device), cmd(opts)) - end + for plugin <- plugins, do: run_one(node, plugin, ctx, timeout) + end - for {name_or_dir, manifest} <- plugins, - plugin = plugin_name(name_or_dir, manifest), - do: run_one(node, plugin, manifest, ctx, timeout) + defp run_one(node, {name_or_dir, manifest}, ctx, timeout), + do: run_one(node, {name_or_dir, manifest, :ok}, ctx, timeout) + + defp run_one(_node, {name_or_dir, nil, {:error, reason}}, _ctx, _timeout) do + entry(name_or_dir, nil, nil, {:fail, "manifest failed verification: #{inspect(reason)}"}, 0) end - defp run_one(_node, plugin, nil, _ctx, _timeout), - do: %{plugin: plugin, module: nil, result: {:skip, "no manifest (tier-0 plugin)"}, ms: 0} + defp run_one(_node, {name_or_dir, nil, _status}, _ctx, _timeout), + do: entry(name_or_dir, nil, nil, {:skip, "no manifest (tier-0 plugin)"}, 0) - defp run_one(node, plugin, manifest, ctx, timeout) do + defp run_one(node, {name_or_dir, manifest, _status}, ctx, timeout) do case Map.get(manifest, :selftest) do nil -> - %{plugin: plugin, module: nil, result: {:skip, "no selftest in manifest"}, ms: 0} + entry(name_or_dir, manifest, nil, {:skip, "no selftest in manifest"}, 0) - module -> + module when is_atom(module) -> {ms, result} = timed(fn -> call(node, module, ctx, timeout) end) - %{plugin: plugin, module: module, result: result, ms: ms} + entry(name_or_dir, manifest, module, result, ms) + + other -> + entry( + name_or_dir, + manifest, + nil, + {:fail, "selftest is not a module: #{inspect(other)}"}, + 0 + ) end end - # Everything a remote run/1 can do wrong lands here as {:fail, why}, so a - # broken self-test never takes the runner (or the other plugins' runs) down. + defp entry(name_or_dir, manifest, module, result, ms), + do: %{plugin: plugin_name(name_or_dir, manifest), module: module, result: result, ms: ms} + + # The remote run/1 is spawned the way `:erpc.call/5` spawns it (its + # `execute_call/4` reports the return or the exception as the exit reason) + # but with the pid in hand, so a test that overruns the deadline is killed + # instead of abandoned. Every way it can go wrong lands here as {:fail, why}, + # so a broken self-test never takes the runner, or the next plugin's run, down. defp call(node, module, ctx, timeout) do - normalize(:erpc.call(node, module, :run, [ctx], timeout), module) - catch - :error, {:erpc, :timeout} -> - {:fail, "timed out after #{timeout} ms"} + ref = make_ref() + + req = + :erlang.spawn_request(node, :erpc, :execute_call, [ref, module, :run, [ctx]], [ + :monitor, + {:reply, :yes} + ]) + + receive do + {:spawn_reply, ^req, :ok, pid} -> + await(req, pid, ref, module, node, timeout) + + {:spawn_reply, ^req, :error, reason} -> + {:fail, "could not spawn on #{node}: #{inspect(reason)}"} + after + timeout -> {:fail, "no answer from #{node} in #{timeout} ms"} + end + end - :error, {:erpc, :noconnection} -> - {:fail, "node #{node} is not reachable"} + defp await(req, pid, ref, module, node, timeout) do + receive do + {:DOWN, ^req, :process, ^pid, reason} -> normalize(reason, ref, module, node) + after + timeout -> + Process.exit(pid, :kill) - :error, {:exception, :undef, [{^module, :run, _, _} | _]} -> + receive do + {:DOWN, ^req, :process, ^pid, _} -> :ok + after + 1_000 -> :ok + end + + {:fail, "timed out after #{timeout} ms"} + end + end + + # The exit reasons `:erpc.execute_call/4` produces, plus the signals. + defp normalize({ref, :return, value}, ref, module, _node), do: contract(value, module) + + defp normalize({ref, :throw, value}, ref, _module, _node), + do: {:fail, "threw: #{inspect(value, limit: 20)}"} + + defp normalize({ref, :exit, reason}, ref, _module, _node), + do: {:fail, "exited: #{inspect(reason, limit: 20)}"} + + defp normalize({ref, :error, :undef, [{module, :run, [_], _} | _]}, ref, module, _node), + do: {:fail, "#{inspect(module)}.run/1 is not on the device (deployed before the self-test was added?)"} - :error, {:exception, reason, stack} -> - {:fail, "raised: " <> format_exception(reason, stack)} + defp normalize({ref, :error, reason, stack}, ref, _module, _node), + do: {:fail, "raised: " <> format_exception(reason, stack)} - :exit, {:exception, reason} -> - {:fail, "exited: #{inspect(reason, limit: 20)}"} + defp normalize({ref, :error, {:erpc, reason}}, ref, _module, _node), + do: {:fail, "erpc failed: #{inspect(reason)}"} - :exit, {:signal, reason} -> - {:fail, "killed: #{inspect(reason, limit: 20)}"} + defp normalize(:noconnection, _ref, _module, node), do: {:fail, "node #{node} is not reachable"} - :throw, value -> - {:fail, "threw: #{inspect(value, limit: 20)}"} - end + defp normalize(reason, _ref, _module, _node), + do: {:fail, "killed: #{inspect(reason, limit: 20)}"} defp format_exception(reason, stack) do exception = Exception.normalize(:error, reason, stack) @@ -120,15 +177,15 @@ defmodule MobDev.Plugin.SelfTest do # The contract (`Mob.Plugin.SelfTest.result?/1`), restated here so the # runner does not need mob at runtime. - defp normalize(:pass, _module), do: :pass - defp normalize({:fail, reason} = fail, _module) when is_binary(reason), do: fail + defp contract(:pass, _module), do: :pass + defp contract({:fail, reason} = fail, _module) when is_binary(reason), do: fail - defp normalize({:skip, reason} = skip, _module) when reason in [:needs_hardware, :needs_user], + defp contract({:skip, reason} = skip, _module) when reason in [:needs_hardware, :needs_user], do: skip - defp normalize({:skip, reason} = skip, _module) when is_binary(reason), do: skip + defp contract({:skip, reason} = skip, _module) when is_binary(reason), do: skip - defp normalize(other, module), + defp contract(other, module), do: {:fail, "#{inspect(module)}.run/1 returned #{inspect(other, limit: 20)}, " <> @@ -140,7 +197,10 @@ defmodule MobDev.Plugin.SelfTest do {System.monotonic_time(:millisecond) - start, result} end - defp plugin_name(_name_or_dir, %{name: name}) when is_atom(name), do: name + defp plugin_name(_name_or_dir, %{name: name}) + when is_atom(name) and not is_nil(name) and name != false, + do: name + defp plugin_name(name, _manifest) when is_atom(name), do: name defp plugin_name(dir, _manifest) when is_binary(dir), @@ -166,24 +226,33 @@ defmodule MobDev.Plugin.SelfTest do @doc """ Grants the permissions the plugins' manifests declare to `bundle_id` on - `device`, so self-tests do not hit a system prompt. + `device`, so self-tests do not hit a system prompt. Call it **before** + launching the app: a simulator may terminate a running app whose privacy + settings change. - * Android (emulator or physical): each `android.permissions` entry via + * Android emulator: each `android.permissions` entry via `adb -s shell pm grant`. Only runtime permissions are grantable; a normal or signature permission answers with an error, which is recorded, not raised. * iOS simulator: each `permissions: [%{capability: cap}]` whose capability maps to a `simctl privacy` service. - * iOS physical device: nothing can be granted from the host; returns `[]`. + * Physical devices: nothing is granted (returns `[]`); a self-test that + needs a permission the user has not given skips with `:needs_user`. Returns one `t:grant/0` per attempt. `cmd` is `fn exe, argv -> {output, status} end`. """ - @spec grant_permissions(Device.t(), [{term(), map() | nil}], String.t(), (String.t(), - [String.t()] -> - {String.t(), - integer()})) :: - [grant()] - def grant_permissions(%Device{platform: :android, serial: serial}, plugins, bundle_id, cmd) do + @spec grant_permissions( + Device.t(), + [plugin()], + String.t(), + (String.t(), [String.t()] -> {String.t(), integer()}) + ) :: [grant()] + def grant_permissions( + %Device{platform: :android, type: :emulator, serial: serial}, + plugins, + bundle_id, + cmd + ) do for {plugin, perm} <- android_permissions(plugins) do {out, status} = cmd.("adb", ["-s", serial, "shell", "pm", "grant", bundle_id, perm]) %{plugin: plugin, permission: perm, status: status(out, status)} @@ -202,41 +271,35 @@ defmodule MobDev.Plugin.SelfTest do end end - def grant_permissions(%Device{platform: :ios}, _plugins, _bundle_id, _cmd), do: [] + def grant_permissions(%Device{}, _plugins, _bundle_id, _cmd), do: [] @doc false - @spec android_permissions([{term(), map() | nil}]) :: [{atom(), String.t()}] + @spec android_permissions([plugin()]) :: [{atom(), String.t()}] def android_permissions(plugins) do - for {name_or_dir, %{} = manifest} <- plugins, + for {name_or_dir, %{} = manifest} <- manifests(plugins), perm <- get_in(manifest, [:android, :permissions]) || [], is_binary(perm), do: {plugin_name(name_or_dir, manifest), perm} end @doc false - @spec simctl_services([{term(), map() | nil}]) :: [{atom(), String.t()}] + @spec simctl_services([plugin()]) :: [{atom(), String.t()}] def simctl_services(plugins) do - for {name_or_dir, %{} = manifest} <- plugins, + for {name_or_dir, %{} = manifest} <- manifests(plugins), %{capability: cap} <- Map.get(manifest, :permissions) || [], service = Map.get(@simctl_services, cap), do: {plugin_name(name_or_dir, manifest), service} end - defp status(_out, 0), do: :ok - defp status(out, _status), do: {:error, out |> String.trim() |> String.slice(0, 200)} - - defp bundle_id(opts, device) do - Keyword.get_lazy(opts, :bundle_id, fn -> - case device.platform do - :ios -> MobDev.Config.ios_bundle_id() - :android -> MobDev.Config.bundle_id() - end + defp manifests(plugins) do + Enum.map(plugins, fn + {name_or_dir, manifest} -> {name_or_dir, manifest} + {name_or_dir, manifest, _status} -> {name_or_dir, manifest} end) end - defp cmd(opts) do - Keyword.get(opts, :cmd, fn exe, argv -> System.cmd(exe, argv, stderr_to_stdout: true) end) - end + defp status(_out, 0), do: :ok + defp status(out, _status), do: {:error, out |> String.trim() |> String.slice(0, 200)} # ── reporting ───────────────────────────────────────────────────────────── @@ -246,10 +309,12 @@ defmodule MobDev.Plugin.SelfTest do """ @spec table([entry()]) :: [String.t()] def table(entries) do - rows = Enum.map(entries, &row/1) - widths = column_widths([["plugin", "outcome", "ms", "detail"] | rows]) + rows = [["plugin", "outcome", "ms", "detail"] | Enum.map(entries, &row/1)] + + widths = + Enum.zip_with(rows, fn column -> column |> Enum.map(&String.length/1) |> Enum.max() end) - for row <- [["plugin", "outcome", "ms", "detail"] | rows] do + for row <- rows do row |> Enum.zip(widths) |> Enum.map_join(" ", fn {cell, width} -> String.pad_trailing(cell, width) end) @@ -267,11 +332,6 @@ defmodule MobDev.Plugin.SelfTest do defp describe({:skip, reason}) when is_atom(reason), do: {"skip", to_string(reason)} defp describe({:skip, reason}), do: {"skip", reason} - defp column_widths(rows) do - rows - |> Enum.zip_with(fn column -> column |> Enum.map(&String.length/1) |> Enum.max() end) - end - @doc "The entries whose result is `{:fail, _}`." @spec failures([entry()]) :: [entry()] def failures(entries), do: Enum.filter(entries, &match?(%{result: {:fail, _}}, &1)) diff --git a/test/mix/tasks/mob_selftest_test.exs b/test/mix/tasks/mob_selftest_test.exs index aebd1db..4f68ce0 100644 --- a/test/mix/tasks/mob_selftest_test.exs +++ b/test/mix/tasks/mob_selftest_test.exs @@ -1,15 +1,15 @@ defmodule Mix.Tasks.Mob.SelftestTest do @moduledoc """ - `mix mob.selftest` driven end to end with fake discovery, grants, connect - and runner: the exit status has to come from the entries the runner - returns, and the grants have to happen before the app is relaunched. + `mix mob.selftest` driven end to end with fake discovery, leases, grants, + connect and runner: the exit status has to come from the entries the + runner returns, and the grants have to happen before the app is relaunched. """ use ExUnit.Case, async: false import ExUnit.CaptureIO alias Mix.Tasks.Mob.Selftest, as: SelftestTask - alias MobDev.Device + alias MobDev.{Device, DeviceLeases} @emulator %Device{ platform: :android, @@ -33,8 +33,8 @@ defmodule Mix.Tasks.Mob.SelftestTest do name: :mob_location, selftest: MobLocation.SelfTest, android: %{permissions: ["android.permission.ACCESS_FINE_LOCATION"]} - }}, - {:mob_deliver, %{name: :mob_deliver}} + }, :ok}, + {:mob_deliver, %{name: :mob_deliver}, :ok} ] defp entries(:pass), @@ -53,13 +53,16 @@ defmodule Mix.Tasks.Mob.SelftestTest do } ] - # Records every step in order so the test can check what ran before what. + # Records every step so the test can check what ran, and in which order. defp deps(devices, outcome, opts \\ []) do test = self() failed = Keyword.get(opts, :failed, []) + lost = Keyword.get(opts, :lost, []) + leases = Keyword.get(opts, :leases, %DeviceLeases{}) %{ plugins: fn -> @plugins end, + leases: fn -> leases end, discover: fn platforms -> send(test, {:discover, platforms}) devices @@ -72,9 +75,9 @@ defmodule Mix.Tasks.Mob.SelftestTest do end, connect: fn connect_opts -> send(test, {:connect, connect_opts}) - - {Enum.filter(devices, &(&1.serial in Keyword.fetch!(connect_opts, :only))) -- failed, - failed} + only = Keyword.fetch!(connect_opts, :only) + connected = (Enum.filter(devices, &(&1.serial in only)) -- failed) -- lost + {connected, failed} end, run_all: fn node, ctx, run_opts -> send(test, {:run_all, node, ctx, run_opts}) @@ -87,6 +90,11 @@ defmodule Mix.Tasks.Mob.SelftestTest do capture_io(fn -> send(self(), {:result, SelftestTask.run(args, deps)}) end) end + defp steps do + {:messages, messages} = Process.info(self(), :messages) + messages |> Enum.map(&elem(&1, 0)) |> Enum.filter(&(&1 in [:grant, :connect, :run_all])) + end + test "passes, printing a table per device, when every self-test passed or skipped" do output = run_task([], deps([@emulator], :pass)) @@ -99,8 +107,7 @@ defmodule Mix.Tasks.Mob.SelftestTest do end test "fails (non-zero exit) naming the device and plugin when a self-test failed" do - error = - assert_raise Mix.Error, fn -> run_task([], deps([@emulator], :fail)) end + error = assert_raise Mix.Error, fn -> run_task([], deps([@emulator], :fail)) end assert error.message =~ "mob.selftest failed:" @@ -111,13 +118,7 @@ defmodule Mix.Tasks.Mob.SelftestTest do test "grants the manifests' permissions before relaunching, then runs with the device's context" do run_task(["--timeout", "5000"], deps([@emulator], :pass)) - # Order: grant, then connect (the relaunch), then run. - {:messages, messages} = Process.info(self(), :messages) - - kinds = - messages |> Enum.map(&elem(&1, 0)) |> Enum.filter(&(&1 in [:grant, :connect, :run_all])) - - assert kinds == [:grant, :connect, :run_all] + assert steps() == [:grant, :connect, :run_all] assert_received {:discover, [:android, :ios]} assert_received {:grant, "emulator-5554"} @@ -126,30 +127,61 @@ defmodule Mix.Tasks.Mob.SelftestTest do assert Keyword.fetch!(connect_opts, :only) == ["emulator-5554"] assert Keyword.fetch!(connect_opts, :restart) == true + assert Keyword.fetch!(connect_opts, :platforms) == [:android] assert ctx == %{platform: :android, device: :emulator} assert Keyword.fetch!(run_opts, :timeout_ms) == 5000 assert Keyword.fetch!(run_opts, :plugins) == @plugins end - test "--no-restart attaches to the running app and --device narrows the targets" do - output = run_task(["--no-restart", "-d", "2CAF"], deps([@emulator, @simulator], :pass)) + test "--no-restart attaches to the running app without granting, --device narrows the targets" do + output = run_task(["--no-restart", "-d", "2CAF98B3"], deps([@emulator, @simulator], :pass)) assert_received {:result, :ok} + assert steps() == [:connect, :run_all] assert_received {:connect, connect_opts} assert Keyword.fetch!(connect_opts, :restart) == false assert Keyword.fetch!(connect_opts, :only) == [@simulator.serial] assert_received {:run_all, :"app_ios@127.0.0.1", %{platform: :ios, device: :simulator}, _} - refute_received {:grant, "emulator-5554"} assert output =~ "iPhone 17" refute output =~ "Pixel 8" end + test "two development devices need --all-devices; with it both run" do + assert_raise Mix.Error, fn -> run_task([], deps([@emulator, @simulator], :pass)) end + refute_received {:connect, _} + + run_task(["--all-devices"], deps([@emulator, @simulator], :pass)) + assert_received {:result, :ok} + assert_received {:run_all, :"app_android_emulator_5554@127.0.0.1", _, _} + assert_received {:run_all, :"app_ios@127.0.0.1", _, _} + end + + test "a device another agent-device session holds is left alone unless named" do + claimed = %DeviceLeases{ + claims: [ + %{ + id: "emulator-5554", + platform: :android, + kind: :emulator, + session: "other", + workspace: "/w" + } + ] + } + + run_task([], deps([@emulator, @simulator], :pass, leases: claimed)) + assert_received {:result, :ok} + refute_received {:grant, "emulator-5554"} + assert_received {:connect, connect_opts} + assert Keyword.fetch!(connect_opts, :only) == [@simulator.serial] + end + test "a selected device whose node could not be reached fails the run" do down = %{@simulator | node: nil, status: :error, error: "no node after 30 s"} error = assert_raise Mix.Error, fn -> - run_task([], deps([@emulator, @simulator], :pass, failed: [down])) + run_task(["--all-devices"], deps([@emulator, @simulator], :pass, failed: [down])) end assert error.message =~ @@ -159,6 +191,16 @@ defmodule Mix.Tasks.Mob.SelftestTest do assert_received {:run_all, :"app_android_emulator_5554@127.0.0.1", _, _} end + test "a device lost between discovery and connect fails the run instead of passing on 0 devices" do + error = + assert_raise Mix.Error, fn -> + run_task([], deps([@emulator], :pass, lost: [@emulator])) + end + + assert error.message =~ "Pixel 8 (emulator-5554): not found when connecting" + refute_received {:run_all, _, _, _} + end + test "prints the permissions it granted" do output = run_task([], deps([@emulator], :pass)) assert output =~ "Pixel 8 (emulator-5554) permissions:" @@ -167,7 +209,7 @@ defmodule Mix.Tasks.Mob.SelftestTest do test "no devices is an error, not a pass" do error = assert_raise Mix.Error, fn -> run_task([], deps([], :pass)) end - assert error.message =~ "No devices found" + assert error.message =~ "No connected devices" refute_received {:run_all, _, _, _} end diff --git a/test/mob_dev/plugin/scaffold_test.exs b/test/mob_dev/plugin/scaffold_test.exs index e338ebe..ae2b066 100644 --- a/test/mob_dev/plugin/scaffold_test.exs +++ b/test/mob_dev/plugin/scaffold_test.exs @@ -115,6 +115,28 @@ defmodule MobDev.Plugin.ScaffoldTest do end end + describe "self-tests in the scaffolds" do + test "tier 1 and tier 4 generate a Mob.Plugin.SelfTest and declare it in the manifest" do + for tier <- [1, 4] do + files = Map.new(Scaffold.files_for(tier, "mob_demo_widget")) + self_test = Map.fetch!(files, "lib/mob_demo_widget/self_test.ex") + assert self_test =~ "defmodule MobDemoWidget.SelfTest do" + assert self_test =~ "@behaviour Mob.Plugin.SelfTest" + assert self_test =~ "def run(" + assert Map.fetch!(files, "priv/mob_plugin.exs") =~ "selftest: MobDemoWidget.SelfTest" + end + end + + test "the tier-1 self-test expects what the generated C NIF returns from ping/0" do + files = Map.new(Scaffold.files_for(1, "mob_demo_widget")) + + assert Map.fetch!(files, "priv/native/jni/mob_demo_widget_nif.c") =~ + ~S|enif_make_atom(env, "ok")| + + assert Map.fetch!(files, "lib/mob_demo_widget/self_test.ex") =~ ":ok -> :pass" + end + end + describe "files_for/2 — tier 0" do setup do {:ok, files: Scaffold.files_for(0, "mob_demo_widget")} diff --git a/test/mob_dev/plugin/self_test_test.exs b/test/mob_dev/plugin/self_test_test.exs index f16de97..c1ee041 100644 --- a/test/mob_dev/plugin/self_test_test.exs +++ b/test/mob_dev/plugin/self_test_test.exs @@ -1,10 +1,7 @@ defmodule MobDev.Plugin.SelfTestTest do - # The "device" is this node: `:erpc.call/5` to `node()` runs the self-test - # in a spawned local process exactly as it would on a phone, including the - # exception, exit and timeout paths. async: false — the crash fixtures log. - use ExUnit.Case, async: false - - import ExUnit.CaptureLog + # The "device" is this node: the self-test is spawned here exactly as it + # would be on a phone, including the exception, exit, kill and timeout paths. + use ExUnit.Case, async: true alias MobDev.Device alias MobDev.Plugin.SelfTest @@ -26,6 +23,11 @@ defmodule MobDev.Plugin.SelfTestTest do def run(%{device: :emulator}), do: {:skip, :needs_hardware} end + defmodule SkipsWithReason do + @moduledoc false + def run(_ctx), do: {:skip, "no camera on this device"} + end + defmodule Raises do @moduledoc false def run(_ctx), do: raise(ArgumentError, "nif not loaded") @@ -41,9 +43,17 @@ defmodule MobDev.Plugin.SelfTestTest do def run(_ctx), do: throw(:oops) end + defmodule Killed do + @moduledoc false + def run(_ctx), do: Process.exit(self(), :kill) + end + defmodule Hangs do @moduledoc false - def run(_ctx), do: Process.sleep(:infinity) + def run(_ctx) do + Process.register(self(), :mob_selftest_hangs) + Process.sleep(:infinity) + end end defmodule OffContract do @@ -51,64 +61,69 @@ defmodule MobDev.Plugin.SelfTestTest do def run(_ctx), do: :ok end + defmodule FailAtom do + @moduledoc false + def run(_ctx), do: {:fail, :nope} + end + + defmodule SkipAtom do + @moduledoc false + def run(_ctx), do: {:skip, :no_reason} + end + defp plugin(name, module), do: {name, %{name: name, selftest: module}} describe "run_all/3" do - test "maps pass, fail and skip through unchanged, with the plugin and module" do + test "maps pass, fail and both skip shapes through unchanged, with the plugin and module" do entries = SelfTest.run_all(node(), @ctx, - plugins: [plugin(:mob_a, Passes), plugin(:mob_b, Fails), plugin(:mob_c, Skips)] + plugins: [ + plugin(:mob_a, Passes), + plugin(:mob_b, Fails), + plugin(:mob_c, Skips), + plugin(:mob_d, SkipsWithReason) + ] ) assert [ %{plugin: :mob_a, module: Passes, result: :pass}, %{plugin: :mob_b, module: Fails, result: {:fail, "status/0 returned :error"}}, - %{plugin: :mob_c, module: Skips, result: {:skip, :needs_hardware}} + %{plugin: :mob_c, module: Skips, result: {:skip, :needs_hardware}}, + %{plugin: :mob_d, result: {:skip, "no camera on this device"}} ] = entries assert Enum.all?(entries, &(is_integer(&1.ms) and &1.ms >= 0)) end - test "a raise, exit or throw in one self-test is that plugin's failure and the rest still run" do - log = - capture_log(fn -> - entries = - SelfTest.run_all(node(), @ctx, - plugins: [ - plugin(:mob_raise, Raises), - plugin(:mob_exit, Exits), - plugin(:mob_throw, Throws), - plugin(:mob_ok, Passes) - ] - ) - - assert [ - %{plugin: :mob_raise, result: {:fail, "raised: " <> raised}}, - %{plugin: :mob_exit, result: {:fail, "exited: :shutdown"}}, - %{plugin: :mob_throw, result: {:fail, "threw: :oops"}}, - %{plugin: :mob_ok, result: :pass} - ] = entries - - assert raised =~ "ArgumentError" - assert raised =~ "nif not loaded" - send(self(), :done) - end) - - assert_received :done - # The runner's own process survived every crash (the log is the remote one's). - refute log =~ "runner" + test "a raise, exit, throw or kill in one self-test is that plugin's failure and the rest still run" do + entries = + SelfTest.run_all(node(), @ctx, + plugins: [ + plugin(:mob_raise, Raises), + plugin(:mob_exit, Exits), + plugin(:mob_throw, Throws), + plugin(:mob_kill, Killed), + plugin(:mob_ok, Passes) + ] + ) + + assert [ + %{plugin: :mob_raise, result: {:fail, "raised: " <> raised}}, + %{plugin: :mob_exit, result: {:fail, "exited: :shutdown"}}, + %{plugin: :mob_throw, result: {:fail, "threw: :oops"}}, + %{plugin: :mob_kill, result: {:fail, "killed: :killed"}}, + %{plugin: :mob_ok, result: :pass} + ] = entries + + assert raised =~ "ArgumentError" + assert raised =~ "nif not loaded" end - test "a self-test that overruns the timeout fails with the timeout and does not block the next" do - {ms, entries} = - :timer.tc( - fn -> - SelfTest.run_all(node(), @ctx, - plugins: [plugin(:mob_slow, Hangs), plugin(:mob_ok, Passes)], - timeout_ms: 50 - ) - end, - :millisecond + test "a self-test that overruns the timeout is killed on the device, fails, and the next one runs" do + entries = + SelfTest.run_all(node(), @ctx, + plugins: [plugin(:mob_slow, Hangs), plugin(:mob_ok, Passes)], + timeout_ms: 50 ) assert [ @@ -117,37 +132,56 @@ defmodule MobDev.Plugin.SelfTestTest do ] = entries assert slow_ms >= 50 - assert ms < 5_000 + assert Process.whereis(:mob_selftest_hangs) == nil end - test "a module that is not on the device, or a return outside the contract, is a failure" do + test "a module not on the device, or a return outside the contract, is a failure" do entries = SelfTest.run_all(node(), @ctx, - plugins: [plugin(:mob_stale, Not.Deployed.SelfTest), plugin(:mob_bad, OffContract)] + plugins: [ + plugin(:mob_stale, Not.Deployed.SelfTest), + plugin(:mob_bad, OffContract), + plugin(:mob_fail_atom, FailAtom), + plugin(:mob_skip_atom, SkipAtom), + {:mob_string, %{name: :mob_string, selftest: "MobString.SelfTest"}} + ] ) assert [ %{plugin: :mob_stale, result: {:fail, stale}}, - %{plugin: :mob_bad, result: {:fail, bad}} + %{plugin: :mob_bad, result: {:fail, bad}}, + %{plugin: :mob_fail_atom, result: {:fail, fail_atom}}, + %{plugin: :mob_skip_atom, result: {:fail, skip_atom}}, + %{plugin: :mob_string, module: nil, result: {:fail, string}} ] = entries assert stale =~ "Not.Deployed.SelfTest.run/1 is not on the device" assert bad =~ "returned :ok, not :pass | {:fail, reason} | {:skip, reason}" + assert fail_atom =~ "returned {:fail, :nope}, not" + assert skip_atom =~ "returned {:skip, :no_reason}, not" + assert string == ~s(selftest is not a module: "MobString.SelfTest") end - test "plugins without a self-test (or without a manifest) are skips, not absent" do + test "plugins without a self-test or a manifest are skips, a failed verification is a failure" do entries = SelfTest.run_all(node(), @ctx, plugins: [ - {"/deps/mob_tier0", nil}, + {"/deps/mob_tier0", nil, :unsigned}, + {"/deps/mob_tampered", nil, {:error, :signature_mismatch}}, {:mob_old, %{name: :mob_old, nifs: []}}, + {"/deps/mob_nameless", %{name: nil, selftest: Passes}}, plugin(:mob_ok, Passes) ] ) assert [ %{plugin: :mob_tier0, module: nil, result: {:skip, "no manifest (tier-0 plugin)"}}, + %{ + plugin: :mob_tampered, + result: {:fail, "manifest failed verification: :signature_mismatch"} + }, %{plugin: :mob_old, module: nil, result: {:skip, "no selftest in manifest"}}, + %{plugin: :mob_nameless, result: :pass}, %{plugin: :mob_ok, result: :pass} ] = entries end @@ -160,46 +194,8 @@ defmodule MobDev.Plugin.SelfTestTest do ) assert [%{plugin: :mob_a, result: {:fail, reason}}] = entries - assert reason =~ "nope@127.0.0.1 is not reachable" - end - - test "grants the manifests' permissions first when a device is given" do - test = self() - - cmd = fn exe, argv -> - send(test, {:cmd, exe, argv}) - {"", 0} - end - - plugins = [ - {:mob_location, - %{ - name: :mob_location, - selftest: Passes, - android: %{permissions: ["android.permission.ACCESS_FINE_LOCATION"]} - }} - ] - - device = %Device{platform: :android, serial: "emulator-5554", type: :emulator} - - assert [%{result: :pass}] = - SelfTest.run_all(node(), @ctx, - plugins: plugins, - device: device, - bundle_id: "com.example.host", - cmd: cmd - ) - - assert_received {:cmd, "adb", - [ - "-s", - "emulator-5554", - "shell", - "pm", - "grant", - "com.example.host", - "android.permission.ACCESS_FINE_LOCATION" - ]} + assert reason =~ "nope@127.0.0.1" + assert reason =~ "noconnection" end end @@ -215,7 +211,7 @@ defmodule MobDev.Plugin.SelfTestTest do "android.permission.ACCESS_COARSE_LOCATION" ] } - }}, + }, :ok}, {:mob_whisper, %{ name: :mob_whisper, @@ -226,7 +222,7 @@ defmodule MobDev.Plugin.SelfTestTest do {"/deps/mob_tier0", nil} ] - test "android: pm grant per declared permission, recording what the device said" do + test "android emulator: pm grant per declared permission, recording what the device said" do cmd = fn "adb", ["-s", "emulator-5554", "shell", "pm", "grant", "com.x.app", perm] -> if perm =~ "COARSE", do: {"Operation not allowed: not a changeable permission type", 255}, @@ -262,10 +258,15 @@ defmodule MobDev.Plugin.SelfTestTest do SelfTest.grant_permissions(device, @plugins, "com.x.app", cmd) end - test "ios physical device: nothing can be granted from the host" do + test "physical devices get nothing granted from the host" do cmd = fn _exe, _argv -> flunk("no command expected") end - device = %Device{platform: :ios, serial: "00008030-ABCDEF", type: :physical} - assert SelfTest.grant_permissions(device, @plugins, "com.x.app", cmd) == [] + + for device <- [ + %Device{platform: :ios, serial: "00008030-ABCDEF", type: :physical}, + %Device{platform: :android, serial: "ZY22DP6HFL", type: :physical} + ] do + assert SelfTest.grant_permissions(device, @plugins, "com.x.app", cmd) == [] + end end end From 40a55b40e6942612f4dcb2baf8554a3697790713 Mon Sep 17 00:00:00 2001 From: GenericJam Date: Thu, 8 Oct 2026 19:14:02 -0600 Subject: [PATCH 3/4] MOB-411: run_all waits for the plugins' OTP applications before the first test (seen: mob_deliver's supervisor not up yet right after a relaunch on the iPhone 17 sim) --- CHANGELOG.md | 5 ++-- lib/mob_dev/plugin/self_test.ex | 29 ++++++++++++++++++++++ test/mob_dev/plugin/self_test_test.exs | 34 ++++++++++++++++++++++++++ 3 files changed, 66 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d961590..f308ed6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,9 @@ options, honouring agent-device leases, plus `--timeout` and `--no-restart`), grants the permissions the manifests declare on emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy - grant`), relaunches the app, calls every activated plugin's `run/1` on - the device with + grant`), relaunches the app, waits for the plugins' OTP applications to + be started (`:boot_timeout_ms`, 15 s), calls every activated plugin's + `run/1` on the device with `%{platform:, device: :simulator | :emulator | :physical}` and prints a table per device. A self-test that raises, exits, times out (30 s; it is then killed on the device) or returns outside the contract is a `FAIL`; diff --git a/lib/mob_dev/plugin/self_test.ex b/lib/mob_dev/plugin/self_test.ex index 535728c..c0cf6aa 100644 --- a/lib/mob_dev/plugin/self_test.ex +++ b/lib/mob_dev/plugin/self_test.ex @@ -56,15 +56,44 @@ defmodule MobDev.Plugin.SelfTest do * `:timeout_ms` — per self-test (default #{@default_timeout_ms}). A test still running at the deadline is killed on the device, so the next plugin's test never runs beside it. + * `:boot_timeout_ms` — how long to wait for the plugins' OTP applications + to be started on the node before the first test (default 15000). mob + starts them after the node is up, so a run right after a relaunch + would otherwise test a plugin whose supervisor is not there yet. A + plugin whose application never starts is tested anyway and reports it. """ @spec run_all(node(), ctx(), keyword()) :: [entry()] def run_all(node, %{platform: _, device: _} = ctx, opts \\ []) do plugins = Keyword.get_lazy(opts, :plugins, &MobDev.Plugin.activated_with_verify/0) timeout = Keyword.get(opts, :timeout_ms, @default_timeout_ms) + await_applications(node, plugins, Keyword.get(opts, :boot_timeout_ms, 15_000)) for plugin <- plugins, do: run_one(node, plugin, ctx, timeout) end + defp await_applications(node, plugins, remaining) do + wanted = for {_, %{name: name}} <- manifests(plugins), is_atom(name), do: name + + started = + try do + :erpc.call(node, Application, :started_applications, [], 5_000) |> Enum.map(&elem(&1, 0)) + catch + _, _ -> wanted + end + + cond do + wanted -- started == [] -> + :ok + + remaining <= 0 -> + :ok + + true -> + Process.sleep(250) + await_applications(node, plugins, remaining - 250) + end + end + defp run_one(node, {name_or_dir, manifest}, ctx, timeout), do: run_one(node, {name_or_dir, manifest, :ok}, ctx, timeout) diff --git a/test/mob_dev/plugin/self_test_test.exs b/test/mob_dev/plugin/self_test_test.exs index c1ee041..11ba1a0 100644 --- a/test/mob_dev/plugin/self_test_test.exs +++ b/test/mob_dev/plugin/self_test_test.exs @@ -186,6 +186,40 @@ defmodule MobDev.Plugin.SelfTestTest do ] = entries end + test "waits for the plugins' applications to be started before the first test, then gives up" do + # :kernel is started; :mob_never_starts is not, so the wait runs out. + {ms, entries} = + :timer.tc( + fn -> + SelfTest.run_all(node(), @ctx, + plugins: [ + {:kernel, %{name: :kernel, selftest: Passes}}, + {:mob_never_starts, %{name: :mob_never_starts, selftest: Passes}} + ], + boot_timeout_ms: 300 + ) + end, + :millisecond + ) + + assert [%{plugin: :kernel, result: :pass}, %{plugin: :mob_never_starts, result: :pass}] = + entries + + assert ms >= 300 and ms < 3_000 + + {ms, _} = + :timer.tc( + fn -> + SelfTest.run_all(node(), @ctx, + plugins: [{:kernel, %{name: :kernel, selftest: Passes}}] + ) + end, + :millisecond + ) + + assert ms < 250 + end + test "an unreachable node is every plugin's failure, not a crash" do entries = SelfTest.run_all(:"nope@127.0.0.1", @ctx, From 958eaad4bd58b67ce7bab4236b8c3af005968087 Mon Sep 17 00:00:00 2001 From: GenericJam Date: Thu, 8 Oct 2026 19:17:55 -0600 Subject: [PATCH 4/4] =?UTF-8?q?MOB-411:=20review=20nitpicks=20=E2=80=94=20?= =?UTF-8?q?abandon=20a=20late=20spawn,=20flush=20monitors,=20decision=20wo?= =?UTF-8?q?rding?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...-08-plugin-self-tests-run-from-the-host.md | 2 +- lib/mob_dev/plugin/self_test.ex | 25 +++++++++++++++---- 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md b/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md index dbe975f..d766469 100644 --- a/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md +++ b/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md @@ -32,7 +32,7 @@ error in a later release once the first-party plugins carry theirs (MOB-418). Warning first because every published plugin would otherwise stop validating the day this ships. -- **`run_all(node, ctx, opts)` is the runner.** One `:erpc.call/5` per +- **`run_all(node, ctx, opts)` is the runner.** One remote spawn per plugin, sequential (self-tests may touch shared hardware), each with its own timeout (`:timeout_ms`, 30 s). The remote `run/1` is spawned the way `:erpc.call/5` spawns it (`spawn_request` of `:erpc.execute_call/4`, which diff --git a/lib/mob_dev/plugin/self_test.ex b/lib/mob_dev/plugin/self_test.ex index c0cf6aa..afeac3f 100644 --- a/lib/mob_dev/plugin/self_test.ex +++ b/lib/mob_dev/plugin/self_test.ex @@ -141,28 +141,43 @@ defmodule MobDev.Plugin.SelfTest do {:reply, :yes} ]) + deadline = System.monotonic_time(:millisecond) + timeout + receive do {:spawn_reply, ^req, :ok, pid} -> - await(req, pid, ref, module, node, timeout) + await(req, pid, ref, module, node, deadline, timeout) {:spawn_reply, ^req, :error, reason} -> {:fail, "could not spawn on #{node}: #{inspect(reason)}"} after - timeout -> {:fail, "no answer from #{node} in #{timeout} ms"} + timeout -> + # A spawn that lands after the deadline must not run beside the + # next plugin's test, nor leave its reply and :DOWN in the mailbox. + unless :erlang.spawn_request_abandon(req) do + receive do + {:spawn_reply, ^req, :ok, pid} -> Process.exit(pid, :kill) + {:spawn_reply, ^req, :error, _} -> :ok + after + 0 -> :ok + end + end + + Process.demonitor(req, [:flush]) + {:fail, "no answer from #{node} in #{timeout} ms"} end end - defp await(req, pid, ref, module, node, timeout) do + defp await(req, pid, ref, module, node, deadline, timeout) do receive do {:DOWN, ^req, :process, ^pid, reason} -> normalize(reason, ref, module, node) after - timeout -> + max(deadline - System.monotonic_time(:millisecond), 0) -> Process.exit(pid, :kill) receive do {:DOWN, ^req, :process, ^pid, _} -> :ok after - 1_000 -> :ok + 1_000 -> Process.demonitor(req, [:flush]) end {:fail, "timed out after #{timeout} ms"}