diff --git a/CHANGELOG.md b/CHANGELOG.md index ac6be75..f308ed6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,31 @@ +## [0.7.17] - 2026-10-08 + +### Added + +- **Plugin self-tests: `mix mob.selftest`, `MobDev.Plugin.SelfTest.run_all/3` + (MOB-411).** A plugin names a `Mob.Plugin.SelfTest` implementation in its + manifest as `selftest: Module` (mob 0.9.15). `mix mob.selftest` attaches to + the deployed app on each selected device (the `mix mob.smoke` selection + options, honouring agent-device leases, plus `--timeout` and + `--no-restart`), grants the permissions the manifests declare on + emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy + grant`), relaunches the app, waits for the plugins' OTP applications to + be started (`:boot_timeout_ms`, 15 s), calls every activated plugin's + `run/1` on the device with + `%{platform:, device: :simulator | :emulator | :physical}` and prints a + table per device. A self-test that raises, exits, times out (30 s; it is + then killed on the device) or returns outside the contract is a `FAIL`; + a plugin without one is a `skip`. Non-zero exit on any `FAIL` or + unreachable device. `run_all/3` is the runner for callers that hold a + node already (mob_ci's invariant P12); `grant_permissions/4` is the + pre-launch grant for them. +- **`selftest` manifest key.** `MobDev.Plugin.Manifest` accepts it (an + Elixir module; when the module is loadable it must export `run/1`). + `mix mob.validate_plugin` **warns** `missing selftest` on a manifest + without one; a later release turns that into an error once the first-party + plugins carry theirs (MOB-418). `mix mob.new_plugin` tier-1 and tier-4 + scaffolds now generate a `SelfTest` module and declare it. + ## [0.7.16] - 2026-10-06 ### Fixed diff --git a/README.md b/README.md index 2aafe45..0b57a24 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,7 @@ end | `mix mob.devices` | List connected devices and their status | | `mix mob.attest` | Prove a device is running the code you just pushed — compares module digests, not artifacts ([see below](#did-that-deploy-actually-land-mix-mobattest)) | | `mix mob.smoke` | Replay recorded `agent-device` UI flows on devices and check the app's own diagnostics held up ([see below](#smoke-flows-on-devices-mix-mobsmoke)) | +| `mix mob.selftest` | Run every activated plugin's `Mob.Plugin.SelfTest` on the app running on each device ([see below](#plugin-self-tests-on-devices-mix-mobselftest)) | | `mix mob.mutate` | Mutation-test the lines this branch changed: break the code on purpose and report what nothing noticed ([see below](#do-the-tests-guard-anything-mix-mobmutate)) | | `mix mob.push` | Hot-push only changed modules (no restart) | | `mix mob.enable ...` | Wire up an optional Mob feature — platform-manifest entries, Elixir stubs, dep injections ([see below](#mix-mobenable-feature)) | @@ -276,6 +277,48 @@ or a manually managed development profile for that bundle id and `.uitests`: an Xcode-managed wildcard team profile is rejected. Simulators and Android need none of this. +## Plugin self-tests on devices (`mix mob.selftest`) + +Every plugin can ship its own proof that it works on a device: a module +implementing `Mob.Plugin.SelfTest`, named in its manifest as `selftest: +Module` (see mob's `MOB_PLUGINS.md`, "Self-test"). `mix mob.selftest` +attaches to the deployed app on each selected device and calls each +activated plugin's `run/1` over distribution, one plugin at a time, with +`%{platform: :ios | :android, device: :simulator | :emulator | :physical}`. + +```bash +mix mob.selftest # the one connected emulator/simulator +mix mob.selftest --device emulator-5554 --timeout 10000 +mix mob.selftest --all-devices --no-restart # attach to the apps as they are +``` + +Before the tests, the permissions the manifests declare are granted on +emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy +grant`) so no system prompt stands between a self-test and its native code; +then the app is relaunched as `mix mob.connect` does, because a simulator +may terminate an app whose privacy settings changed. Each device gets a +table: + +``` +Pixel 8 (emulator-5554) android emulator + plugin outcome ms detail + mob_location pass 3 + mob_whisper pass 1 + mob_deliver pass 2 + mob_camera skip 0 needs_hardware + 3 passed, 0 failed, 1 skipped +``` + +A plugin without a self-test is a `skip` (so it is visible, not absent); a +self-test that raises, exits, times out (30 s by default) or returns +something outside the contract is a `FAIL` with the reason. The exit status +is non-zero on any `FAIL`, or when a selected device's node could not be +reached. Device selection follows `mix mob.smoke` (`--device`, +`--all-devices`, `--all-physical`, agent-device leases honoured). +`MobDev.Plugin.SelfTest.run_all/3` is the same runner for callers that +already hold a node, such as mob_ci, and `grant_permissions/4` the +pre-launch grant. + ## Do the tests guard anything? (`mix mob.mutate`) A green suite says the tests ran, not that they guard anything. This changes diff --git a/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md b/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md new file mode 100644 index 0000000..d766469 --- /dev/null +++ b/decisions/2026-10-08-plugin-self-tests-run-from-the-host.md @@ -0,0 +1,94 @@ +# Plugin self-tests run from the host: `mix mob.selftest` and `MobDev.Plugin.SelfTest.run_all/3` + +- Date: 2026-10-08 +- Status: accepted +- Linear: MOB-411 (epic MOB-410); the contract is mob's + `decisions/2026-10-08-plugin-self-test-contract.md` + +## Context + +- mob 0.9.15 gives plugins a behaviour, `Mob.Plugin.SelfTest`, with one + callback `run(ctx) :: :pass | {:fail, reason} | {:skip, reason}`, named in + the manifest as `selftest: Module`. The rule for a pass is a real answer + from the plugin's native code. +- Something has to validate the key, call the tests on a device and judge + the answers. mob_dev already owns manifest validation, device discovery + and attaching to a running app over distribution (`mix mob.smoke`, + `mix mob.attest`), and mob_ci calls into mob_dev for builds and the + static gate, so the runner belongs here and mob_ci calls it (invariant + P12). +- A self-test is plugin code running inside the app: it can raise, exit, + hang, or return `:ok` because its author misread the contract. None of + that may take the runner, or the other plugins' runs, down. + +## Decision + +- **`selftest` is a manifest key.** `MobDev.Plugin.Manifest` requires an + Elixir module; when the module is loadable at validation time (the + plugin is compiled: `mix mob.validate_plugin` after a compile, or a host + activating a dep) it must export `run/1`, otherwise only the name is + checked. A manifest without the key gets a **warning** from + `mix mob.validate_plugin` (`missing selftest`) in 0.7.17; it becomes an + error in a later release once the first-party plugins carry theirs + (MOB-418). Warning first because every published plugin would otherwise + stop validating the day this ships. +- **`run_all(node, ctx, opts)` is the runner.** One remote spawn per + plugin, sequential (self-tests may touch shared hardware), each with its + own timeout (`:timeout_ms`, 30 s). The remote `run/1` is spawned the way + `:erpc.call/5` spawns it (`spawn_request` of `:erpc.execute_call/4`, which + reports the return or the exception as its exit reason) but with the pid + in hand, so a test that overruns the deadline is **killed** on the + device rather than abandoned beside the next plugin's run. Every way it + can go wrong becomes that plugin's `{:fail, why}`: timeout, raise + (`:undef` for a module the running build does not have, named as such), + exit, throw, a kill signal, a lost connection, a non-module `selftest:` + value, a manifest that failed signature verification, and a return + outside the contract. The contract's three shapes are restated in the runner + rather than calling `Mob.Plugin.SelfTest.result?/1`, so mob_dev does not + need mob at runtime and the runner works against an app built with an + older mob. Plugins with no `selftest:` (or no manifest) are entries with + `module: nil` and a `{:skip, _}`: a plugin without a test is visible in + every table, not silently absent. +- **Permissions are pre-granted where the host can.** `grant_permissions/4` + runs `adb shell pm grant` for each `android.permissions` entry (only + runtime permissions are grantable; a normal one answers with an error + that is recorded, not raised) and `xcrun simctl privacy grant` for each + `permissions: [%{capability: _}]` whose capability has a simctl service + (location, microphone, photos, media, contacts, calendar, reminders, + motion). A physical iPhone gets nothing: there is no host-side grant. + `run_all/3` never grants: it holds a node, so the app is already running, + and `simctl privacy` warns that some changes terminate a running app. + Granting is the caller's job before launch: `mix mob.selftest` grants, + then relaunches (`--no-restart` grants nothing); mob_ci calls + `grant_permissions/4` before it starts the app. (The first draft had + `run_all/3` grant when given `:device`; the pre-merge review found that + it could kill the simulator app it was about to test.) Physical devices + get nothing granted; their self-tests skip with `:needs_user`. +- **`mix mob.selftest` selects devices like `mix mob.smoke`** + (`MobDev.TaskTargets.resolve/3` with the agent-device leases: `--device` + / `--only`, `--all-devices`, `--all-physical`, `--ios-only`, + `--android-only`; a device another session holds is left alone unless + named) and relaunches the app by default like `mix mob.connect` + (`--no-restart` attaches as is). A target that `connect_all/1` no longer + finds is reported as unreachable, so a run never passes on zero devices. The context is + the device record's platform and type, not something the plugin detects: + a plugin deciding for itself that it is "on an emulator" is how a skip + hides a hole. Exit status is non-zero on any `FAIL` or on a selected + device whose node could not be reached (a test that could not run is not + a test that passed, the same rule as `mix mob.attest`). +- **Scaffolds declare one.** `mix mob.new_plugin` tier 1 generates a + `SelfTest` that calls the NIF's `ping/0`; tier 4 one that round-trips the + supervised worker. Tiers 2 and 3 have no honest host-callable probe in + the template and get the warning, which is the nudge intended. + +## Consequences + +- mob_ci replaces its `device_caps.exs` probe table with `run_all/3` once + the plugins carry self-tests (MOB-414, MOB-418). +- The tests (`test/mob_dev/plugin/self_test_test.exs`) use this node as the + device: `:erpc.call/5` to `node()` runs the fixture in a spawned local + process, so the raise, exit, throw, timeout, undef and off-contract paths + are exercised for real without a device. +- Turning the warning into an error is a one-line change in + `Validator.selftest_review/1` plus moving it into the error list; record + the release that does it here. diff --git a/lib/mix/tasks/mob.selftest.ex b/lib/mix/tasks/mob.selftest.ex new file mode 100644 index 0000000..02a748b --- /dev/null +++ b/lib/mix/tasks/mob.selftest.ex @@ -0,0 +1,212 @@ +defmodule Mix.Tasks.Mob.Selftest do + @shortdoc "Run every activated plugin's self-test on the app running on each device" + + @moduledoc """ + Attach to the deployed app on each selected device and run every activated + plugin's `Mob.Plugin.SelfTest` on it, over distribution. + + mix mob.selftest # the one connected emulator/simulator + mix mob.selftest --device emulator-5554 + mix mob.selftest --all-devices --timeout 10000 + + The app must be deployed (`mix mob.deploy`) with the plugins activated in + `mob.exs`; a self-test is only on the device if the build that is running + included it. Each plugin's `selftest: Module` from its manifest is called + with `%{platform: :ios | :android, device: :simulator | :emulator | + :physical}` and gets one line in a table per device: `pass`, `FAIL` with + the reason, or `skip` with why (a plugin without a self-test is a skip, + so it is visible). A self-test that raises, exits, times out or returns + something outside the contract is a `FAIL`. + + Before the tests run, the permissions the manifests declare are granted + on emulators and simulators (`adb shell pm grant`, `xcrun simctl privacy + grant`), so no system prompt stands between a self-test and its native + code. The app is then relaunched (as `mix mob.connect` does), because a + simulator may terminate an app whose privacy settings changed. + `--no-restart` attaches to the app as it is and grants nothing. + + ## Options + + * `--device` / `--only` (`-d`) — target a device by serial/udid + (serial, udid or short id). Repeatable + * `--all-devices` — every emulator and simulator + * `--all-physical` — every physical device (with `--all-devices`: everything) + * `--ios-only` / `--android-only` — restrict discovery to one platform + * `--timeout MS` — per self-test, in milliseconds (default: 30000) + * `--no-restart` — do not relaunch the app before testing (and do not + grant permissions) + * `--cookie C` — dist cookie (default: the app's private cookie) + + With no selection flags exactly one emulator/simulator is picked, and a + device another `agent-device` session holds is left alone unless named. + + ## Exit status + + Non-zero when any self-test failed, or a selected device's node could not + be reached (a test that could not run is not a test that passed). + """ + + use Mix.Task + + alias Mix.Tasks.Mob.Connect + alias Mix.Tasks.Mob.Deploy + alias MobDev.{Connector, Device, DeviceLeases, TaskHelp, TaskTargets} + alias MobDev.Plugin.SelfTest + + @switches [ + device: :keep, + only: :keep, + all_devices: :boolean, + all_physical: :boolean, + ios_only: :boolean, + android_only: :boolean, + timeout: :integer, + restart: :boolean, + cookie: :string + ] + + @impl Mix.Task + def run(args), do: run(args, %{}) + + @doc false + # `deps` replaces the I/O: `:discover` (platforms -> devices, as + # `Mix.Tasks.Mob.Deploy.discover_devices/1`), `:leases` (-> the agent-device + # claims), `:grant` (device, plugins -> grants, as + # `SelfTest.grant_permissions/4` with the app's bundle id), `:connect` + # (opts -> {connected, failed} devices, as `MobDev.Connector.connect_all/1`), + # `:plugins` (-> activated plugins) and `:run_all` (node, ctx, opts -> + # entries, as `SelfTest.run_all/3`). + @spec run([String.t()], map()) :: :ok + def run(args, deps) do + if TaskHelp.help_requested?(args) do + TaskHelp.print_module_help(__MODULE__) + else + selftest(parse!(args), Map.merge(default_deps(), deps)) + end + end + + defp parse!(args) do + {opts, argv, invalid} = OptionParser.parse(args, strict: @switches, aliases: [d: :device]) + + unless invalid == [] and argv == [] do + bad = Enum.map(invalid, &elem(&1, 0)) ++ argv + Mix.raise("Unknown option(s) or argument(s): #{Enum.join(bad, ", ")}") + end + + if Keyword.get(opts, :timeout, 1) < 1, + do: Mix.raise("--timeout must be a positive number of ms") + + opts + end + + defp selftest(opts, deps) do + Mix.Task.run("app.config") + targets = select_devices!(opts, deps) + plugins = deps.plugins.() + restart? = Keyword.get(opts, :restart, true) + + # Grants come before the (re)launch: a simulator may terminate an app + # whose privacy settings change, and a self-test must not meet a prompt. + if restart?, do: Enum.each(targets, &grant(&1, plugins, deps)) + + {connected, failed} = + deps.connect.( + cookie: opts[:cookie], + only: Enum.map(targets, & &1.serial), + platforms: targets |> Enum.map(& &1.platform) |> Enum.uniq(), + restart: restart? + ) + + run_opts = [plugins: plugins, timeout_ms: Keyword.get(opts, :timeout, 30_000)] + results = Enum.map(connected, &run_device(&1, run_opts, deps)) + + seen = MapSet.new(connected ++ failed, & &1.serial) + missing = Enum.reject(targets, &MapSet.member?(seen, &1.serial)) + + unreachable = + Enum.map(failed, &"#{label(&1)}: node not reachable (#{&1.error || &1.status})") ++ + Enum.map(missing, &"#{label(&1)}: not found when connecting") + + Enum.each(unreachable, &IO.puts("\n" <> &1)) + + failures = + for {device, entries} <- results, + %{plugin: plugin, result: {:fail, reason}} <- SelfTest.failures(entries), + do: "#{label(device)} #{plugin}: #{reason}" + + case failures ++ unreachable do + [] -> IO.puts("\nmob.selftest passed on #{length(results)} device(s)") + problems -> Mix.raise("mob.selftest failed:\n " <> Enum.join(problems, "\n ")) + end + + :ok + end + + defp select_devices!(opts, deps) do + platforms = + case Connect.resolve_platforms(opts, MobDev.Config.platforms()) do + {:ok, platforms} -> platforms + {:error, message} -> Mix.raise(message) + end + + ids = Keyword.get_values(opts, :device) ++ Keyword.get_values(opts, :only) + opts = Keyword.put(opts, :leases, deps.leases.()) + + case TaskTargets.resolve(deps.discover.(platforms), ids, opts) do + {:ok, devices} -> devices + {:error, reason, context} -> Mix.raise(Deploy.target_error(reason, context, ids)) + end + end + + defp grant(%Device{} = device, plugins, deps) do + case deps.grant.(device, plugins) do + [] -> + :ok + + grants -> + IO.puts("\n#{label(device)} permissions:") + + Enum.each(grants, fn %{plugin: plugin, permission: perm, status: status} -> + case status do + :ok -> IO.puts(" granted #{perm} (#{plugin})") + {:error, out} -> IO.puts(" could not grant #{perm} (#{plugin}): #{out}") + end + end) + end + end + + defp run_device(%Device{} = device, run_opts, deps) do + ctx = %{platform: device.platform, device: device.type || :physical} + IO.puts("\n#{label(device)} #{ctx.platform} #{ctx.device}") + + entries = deps.run_all.(device.node, ctx, run_opts) + Enum.each(SelfTest.table(entries), &IO.puts(" " <> &1)) + IO.puts(" " <> SelfTest.summary(entries)) + {device, entries} + end + + defp label(%Device{name: name, serial: serial}), do: "#{name || serial} (#{serial})" + + defp default_deps do + %{ + discover: &Deploy.discover_devices/1, + leases: &DeviceLeases.load/0, + grant: &grant_permissions/2, + connect: &Connector.connect_all/1, + plugins: &MobDev.Plugin.activated_with_verify/0, + run_all: &SelfTest.run_all/3 + } + end + + defp grant_permissions(%Device{platform: platform} = device, plugins) do + bundle_id = + case platform do + :ios -> MobDev.Config.ios_bundle_id() + :android -> MobDev.Config.bundle_id() + end + + SelfTest.grant_permissions(device, plugins, bundle_id, fn exe, argv -> + System.cmd(exe, argv, stderr_to_stdout: true) + end) + end +end diff --git a/lib/mob_dev/plugin/manifest.ex b/lib/mob_dev/plugin/manifest.ex index bc10196..08d25a0 100644 --- a/lib/mob_dev/plugin/manifest.ex +++ b/lib/mob_dev/plugin/manifest.ex @@ -31,7 +31,7 @@ defmodule MobDev.Plugin.Manifest do name mob_version plugin_spec_version permissions android ios nifs nifs_generator screens screens_generator migrations assets default_font lifecycle settings notifications ui_components ui_components_generator - host_requirements description version tags host_config_keys setup + host_requirements description version tags host_config_keys setup selftest )a @native_sections [ @@ -153,6 +153,7 @@ defmodule MobDev.Plugin.Manifest do |> check_notifications(manifest) |> check_ui_components(manifest) |> check_host_requirements(manifest) + |> check_selftest(manifest) warn_unknown_keys(manifest) @@ -598,6 +599,36 @@ defmodule MobDev.Plugin.Manifest do defp check_lifecycle(errors, _), do: errors + # `:selftest` names the plugin's `Mob.Plugin.SelfTest` implementation. The + # module is only loadable when the plugin is compiled (`mix + # mob.validate_plugin` after a compile, or the host activating a dep); then + # it must export run/1. Otherwise the name is all there is to check. + defp check_selftest(errors, %{selftest: mod}) when is_atom(mod) and not is_nil(mod) do + cond do + not elixir_module?(mod) -> + ["selftest must be an Elixir module, got: #{inspect(mod)}" | errors] + + Code.ensure_loaded?(mod) and not function_exported?(mod, :run, 1) -> + [ + "selftest #{inspect(mod)} does not export run/1 (implement Mob.Plugin.SelfTest)" + | errors + ] + + true -> + errors + end + end + + defp check_selftest(errors, %{selftest: other}), + do: [ + "selftest must be a module implementing Mob.Plugin.SelfTest, got: #{inspect(other)}" + | errors + ] + + defp check_selftest(errors, _), do: errors + + defp elixir_module?(mod), do: String.starts_with?(Atom.to_string(mod), "Elixir.") + defp check_optional_mfa(errors, map, key) do case Map.get(map, key) do nil -> errors diff --git a/lib/mob_dev/plugin/scaffold.ex b/lib/mob_dev/plugin/scaffold.ex index 530b19b..7dac0fb 100644 --- a/lib/mob_dev/plugin/scaffold.ex +++ b/lib/mob_dev/plugin/scaffold.ex @@ -194,6 +194,7 @@ defmodule MobDev.Plugin.Scaffold do [ {"lib/#{name}.ex", tier1_lib(name, nif_name)}, + {"lib/#{name}/self_test.ex", tier1_self_test(name, nif_name)}, {"src/#{nif_name}.erl", tier1_erl_stub(nif_name)}, {"priv/mob_plugin.exs", tier1_manifest(name, nif_name, mob_req)}, {"priv/native/jni/#{nif_name}.c", tier1_c(nif_name)}, @@ -238,6 +239,7 @@ defmodule MobDev.Plugin.Scaffold do {"lib/#{name}/worker.ex", tier4_worker(mod)}, {"lib/#{name}/notifications.ex", tier4_notifications(mod)}, {"lib/#{name}/settings_screen.ex", tier4_settings_screen(mod)}, + {"lib/#{name}/self_test.ex", tier4_self_test(mod)}, {"priv/mob_plugin.exs", tier4_manifest(name, mod, mob_req)}, {"test/test_helper.exs", test_helper()}, {"test/#{name}_test.exs", plugin_test(name)} @@ -739,6 +741,29 @@ defmodule MobDev.Plugin.Scaffold do """ end + defp tier1_self_test(name, nif_name) do + mod = module_name(name) + + """ + defmodule #{mod}.SelfTest do + @moduledoc \"\"\" + The plugin's on-device proof (`Mob.Plugin.SelfTest`), run by + `mix mob.selftest` and mob_ci for every activated plugin. A pass means a + real answer came back from the NIF; keep it side-effect free and quick. + \"\"\" + @behaviour Mob.Plugin.SelfTest + + @impl true + def run(%{platform: platform}) do + case :#{nif_name}.ping() do + :ok -> :pass + other -> {:fail, "ping/0 on \#{platform} returned \#{inspect(other)}, expected :ok"} + end + end + end + """ + end + defp tier1_manifest(name, nif_name, mob_req) do """ %{ @@ -751,7 +776,9 @@ defmodule MobDev.Plugin.Scaffold do # module — ERL_NIF_INIT uses it as both the registered module name # and the static-init C symbol prefix. %{module: :#{nif_name}, native_dir: "priv/native/jni"} - ] + ], + # On-device proof, run by `mix mob.selftest` (see Mob.Plugin.SelfTest). + selftest: #{module_name(name)}.SelfTest } """ end @@ -1104,6 +1131,33 @@ defmodule MobDev.Plugin.Scaffold do """ end + defp tier4_self_test(mod) do + """ + defmodule #{mod}.SelfTest do + @moduledoc \"\"\" + The plugin's on-device proof (`Mob.Plugin.SelfTest`), run by + `mix mob.selftest` and mob_ci for every activated plugin. A pure-Elixir + plugin proves its real API path: here, that the supervised worker is up + under the host's plugin supervisor and answers a call. + \"\"\" + @behaviour Mob.Plugin.SelfTest + + @impl true + def run(_ctx) do + case Process.whereis(#{mod}.Worker) do + nil -> + {:fail, "#{mod}.Worker is not running (lifecycle.supervised did not start it)"} + + pid -> + # A round-trip through the worker, not just a pid lookup. + %{} = :sys.get_state(pid, 5_000) + :pass + end + end + end + """ + end + defp tier4_manifest(name, mod, mob_req) do """ %{ @@ -1138,7 +1192,10 @@ defmodule MobDev.Plugin.Scaffold do handlers: [ %{match: %{type: "#{name}"}, handler: {#{mod}.Notifications, :handle, 1}} ] - } + }, + + # On-device proof, run by `mix mob.selftest` (see Mob.Plugin.SelfTest). + selftest: #{mod}.SelfTest } """ end diff --git a/lib/mob_dev/plugin/self_test.ex b/lib/mob_dev/plugin/self_test.ex new file mode 100644 index 0000000..afeac3f --- /dev/null +++ b/lib/mob_dev/plugin/self_test.ex @@ -0,0 +1,395 @@ +defmodule MobDev.Plugin.SelfTest do + @moduledoc """ + Runs every activated plugin's `Mob.Plugin.SelfTest` on a device. + + A plugin declares `selftest: Module` in its manifest; `run_all/3` calls + `Module.run/1` on the device's node, one plugin at a time, and returns one + entry per activated plugin. `mix mob.selftest` prints those as a table; + mob_ci records them per nightly cell (invariant P12). + + The entry's `:result` is always one of the contract's three shapes: + + * `:pass` + * `{:fail, reason}` — also what a self-test that raised, exited, threw, + timed out, is not on the device or returned something outside the + contract gets, and what a plugin whose manifest failed signature + verification gets. The reason says which. + * `{:skip, :needs_hardware | :needs_user | reason}` — including plugins + with no `selftest:` in their manifest (`module: nil`), so a plugin + without one is visible, not silently absent. + + Permissions are not granted here: the node is already running, and on an + iOS simulator a privacy change can terminate the app. Callers grant + **before** launching the app with `grant_permissions/4`, as + `mix mob.selftest` does. + """ + + alias MobDev.Device + + @default_timeout_ms 30_000 + + @typedoc "What a self-test is told about where it runs (`Mob.Plugin.SelfTest.ctx/0`)." + @type ctx :: %{platform: :ios | :android, device: :simulator | :emulator | :physical} + + @typedoc "One plugin's outcome. `:ms` is the wall time of the call on the host." + @type entry :: %{plugin: atom(), module: module() | nil, result: term(), ms: non_neg_integer()} + + @typedoc """ + An activated plugin as `MobDev.Plugin.activated_with_verify/0` (3-tuple) or + `activated/0` (2-tuple) lists them; the first element is the plugin's name + or its dependency directory. + """ + @type plugin :: {atom() | Path.t(), map() | nil} | {atom() | Path.t(), map() | nil, term()} + + @typedoc "A granted (or attempted) permission." + @type grant :: %{plugin: atom(), permission: String.t(), status: :ok | {:error, String.t()}} + + @doc """ + Runs the self-test of every activated plugin on `node` and returns an entry per plugin. + + Options: + + * `:plugins` — `t:plugin/0` list (default: + `MobDev.Plugin.activated_with_verify/0`, read from the host project's + `mob.exs` and deps). A plugin whose manifest failed verification is a + `{:fail, _}` entry; one with no manifest (tier 0) is a skip. + * `:timeout_ms` — per self-test (default #{@default_timeout_ms}). A test + still running at the deadline is killed on the device, so the next + plugin's test never runs beside it. + * `:boot_timeout_ms` — how long to wait for the plugins' OTP applications + to be started on the node before the first test (default 15000). mob + starts them after the node is up, so a run right after a relaunch + would otherwise test a plugin whose supervisor is not there yet. A + plugin whose application never starts is tested anyway and reports it. + """ + @spec run_all(node(), ctx(), keyword()) :: [entry()] + def run_all(node, %{platform: _, device: _} = ctx, opts \\ []) do + plugins = Keyword.get_lazy(opts, :plugins, &MobDev.Plugin.activated_with_verify/0) + timeout = Keyword.get(opts, :timeout_ms, @default_timeout_ms) + + await_applications(node, plugins, Keyword.get(opts, :boot_timeout_ms, 15_000)) + for plugin <- plugins, do: run_one(node, plugin, ctx, timeout) + end + + defp await_applications(node, plugins, remaining) do + wanted = for {_, %{name: name}} <- manifests(plugins), is_atom(name), do: name + + started = + try do + :erpc.call(node, Application, :started_applications, [], 5_000) |> Enum.map(&elem(&1, 0)) + catch + _, _ -> wanted + end + + cond do + wanted -- started == [] -> + :ok + + remaining <= 0 -> + :ok + + true -> + Process.sleep(250) + await_applications(node, plugins, remaining - 250) + end + end + + defp run_one(node, {name_or_dir, manifest}, ctx, timeout), + do: run_one(node, {name_or_dir, manifest, :ok}, ctx, timeout) + + defp run_one(_node, {name_or_dir, nil, {:error, reason}}, _ctx, _timeout) do + entry(name_or_dir, nil, nil, {:fail, "manifest failed verification: #{inspect(reason)}"}, 0) + end + + defp run_one(_node, {name_or_dir, nil, _status}, _ctx, _timeout), + do: entry(name_or_dir, nil, nil, {:skip, "no manifest (tier-0 plugin)"}, 0) + + defp run_one(node, {name_or_dir, manifest, _status}, ctx, timeout) do + case Map.get(manifest, :selftest) do + nil -> + entry(name_or_dir, manifest, nil, {:skip, "no selftest in manifest"}, 0) + + module when is_atom(module) -> + {ms, result} = timed(fn -> call(node, module, ctx, timeout) end) + entry(name_or_dir, manifest, module, result, ms) + + other -> + entry( + name_or_dir, + manifest, + nil, + {:fail, "selftest is not a module: #{inspect(other)}"}, + 0 + ) + end + end + + defp entry(name_or_dir, manifest, module, result, ms), + do: %{plugin: plugin_name(name_or_dir, manifest), module: module, result: result, ms: ms} + + # The remote run/1 is spawned the way `:erpc.call/5` spawns it (its + # `execute_call/4` reports the return or the exception as the exit reason) + # but with the pid in hand, so a test that overruns the deadline is killed + # instead of abandoned. Every way it can go wrong lands here as {:fail, why}, + # so a broken self-test never takes the runner, or the next plugin's run, down. + defp call(node, module, ctx, timeout) do + ref = make_ref() + + req = + :erlang.spawn_request(node, :erpc, :execute_call, [ref, module, :run, [ctx]], [ + :monitor, + {:reply, :yes} + ]) + + deadline = System.monotonic_time(:millisecond) + timeout + + receive do + {:spawn_reply, ^req, :ok, pid} -> + await(req, pid, ref, module, node, deadline, timeout) + + {:spawn_reply, ^req, :error, reason} -> + {:fail, "could not spawn on #{node}: #{inspect(reason)}"} + after + timeout -> + # A spawn that lands after the deadline must not run beside the + # next plugin's test, nor leave its reply and :DOWN in the mailbox. + unless :erlang.spawn_request_abandon(req) do + receive do + {:spawn_reply, ^req, :ok, pid} -> Process.exit(pid, :kill) + {:spawn_reply, ^req, :error, _} -> :ok + after + 0 -> :ok + end + end + + Process.demonitor(req, [:flush]) + {:fail, "no answer from #{node} in #{timeout} ms"} + end + end + + defp await(req, pid, ref, module, node, deadline, timeout) do + receive do + {:DOWN, ^req, :process, ^pid, reason} -> normalize(reason, ref, module, node) + after + max(deadline - System.monotonic_time(:millisecond), 0) -> + Process.exit(pid, :kill) + + receive do + {:DOWN, ^req, :process, ^pid, _} -> :ok + after + 1_000 -> Process.demonitor(req, [:flush]) + end + + {:fail, "timed out after #{timeout} ms"} + end + end + + # The exit reasons `:erpc.execute_call/4` produces, plus the signals. + defp normalize({ref, :return, value}, ref, module, _node), do: contract(value, module) + + defp normalize({ref, :throw, value}, ref, _module, _node), + do: {:fail, "threw: #{inspect(value, limit: 20)}"} + + defp normalize({ref, :exit, reason}, ref, _module, _node), + do: {:fail, "exited: #{inspect(reason, limit: 20)}"} + + defp normalize({ref, :error, :undef, [{module, :run, [_], _} | _]}, ref, module, _node), + do: + {:fail, + "#{inspect(module)}.run/1 is not on the device (deployed before the self-test was added?)"} + + defp normalize({ref, :error, reason, stack}, ref, _module, _node), + do: {:fail, "raised: " <> format_exception(reason, stack)} + + defp normalize({ref, :error, {:erpc, reason}}, ref, _module, _node), + do: {:fail, "erpc failed: #{inspect(reason)}"} + + defp normalize(:noconnection, _ref, _module, node), do: {:fail, "node #{node} is not reachable"} + + defp normalize(reason, _ref, _module, _node), + do: {:fail, "killed: #{inspect(reason, limit: 20)}"} + + defp format_exception(reason, stack) do + exception = Exception.normalize(:error, reason, stack) + banner = Exception.format_banner(:error, exception, stack) + + case stack do + [frame | _] -> banner <> " at " <> Exception.format_stacktrace_entry(frame) + [] -> banner + end + end + + # The contract (`Mob.Plugin.SelfTest.result?/1`), restated here so the + # runner does not need mob at runtime. + defp contract(:pass, _module), do: :pass + defp contract({:fail, reason} = fail, _module) when is_binary(reason), do: fail + + defp contract({:skip, reason} = skip, _module) when reason in [:needs_hardware, :needs_user], + do: skip + + defp contract({:skip, reason} = skip, _module) when is_binary(reason), do: skip + + defp contract(other, module), + do: + {:fail, + "#{inspect(module)}.run/1 returned #{inspect(other, limit: 20)}, " <> + "not :pass | {:fail, reason} | {:skip, reason}"} + + defp timed(fun) do + start = System.monotonic_time(:millisecond) + result = fun.() + {System.monotonic_time(:millisecond) - start, result} + end + + defp plugin_name(_name_or_dir, %{name: name}) + when is_atom(name) and not is_nil(name) and name != false, + do: name + + defp plugin_name(name, _manifest) when is_atom(name), do: name + + defp plugin_name(dir, _manifest) when is_binary(dir), + do: dir |> Path.basename() |> String.to_atom() + + # ── permissions ─────────────────────────────────────────────────────────── + + # `xcrun simctl privacy` services for the capability atoms manifests declare + # under `permissions: [%{capability: ...}]`. Capabilities with no service + # (bluetooth, notifications, speech, ...) cannot be pre-granted on a + # simulator; their self-tests skip with :needs_user if a prompt would block. + @simctl_services %{ + location: "location", + microphone: "microphone", + photo_library: "photos", + photos: "photos", + media: "media-library", + contacts: "contacts", + calendar: "calendar", + reminders: "reminders", + motion: "motion" + } + + @doc """ + Grants the permissions the plugins' manifests declare to `bundle_id` on + `device`, so self-tests do not hit a system prompt. Call it **before** + launching the app: a simulator may terminate a running app whose privacy + settings change. + + * Android emulator: each `android.permissions` entry via + `adb -s shell pm grant`. Only runtime permissions are + grantable; a normal or signature permission answers with an error, + which is recorded, not raised. + * iOS simulator: each `permissions: [%{capability: cap}]` whose + capability maps to a `simctl privacy` service. + * Physical devices: nothing is granted (returns `[]`); a self-test that + needs a permission the user has not given skips with `:needs_user`. + + Returns one `t:grant/0` per attempt. `cmd` is `fn exe, argv -> {output, status} end`. + """ + @spec grant_permissions( + Device.t(), + [plugin()], + String.t(), + (String.t(), [String.t()] -> {String.t(), integer()}) + ) :: [grant()] + def grant_permissions( + %Device{platform: :android, type: :emulator, serial: serial}, + plugins, + bundle_id, + cmd + ) do + for {plugin, perm} <- android_permissions(plugins) do + {out, status} = cmd.("adb", ["-s", serial, "shell", "pm", "grant", bundle_id, perm]) + %{plugin: plugin, permission: perm, status: status(out, status)} + end + end + + def grant_permissions( + %Device{platform: :ios, type: :simulator, serial: udid}, + plugins, + bundle_id, + cmd + ) do + for {plugin, service} <- simctl_services(plugins) do + {out, status} = cmd.("xcrun", ["simctl", "privacy", udid, "grant", service, bundle_id]) + %{plugin: plugin, permission: service, status: status(out, status)} + end + end + + def grant_permissions(%Device{}, _plugins, _bundle_id, _cmd), do: [] + + @doc false + @spec android_permissions([plugin()]) :: [{atom(), String.t()}] + def android_permissions(plugins) do + for {name_or_dir, %{} = manifest} <- manifests(plugins), + perm <- get_in(manifest, [:android, :permissions]) || [], + is_binary(perm), + do: {plugin_name(name_or_dir, manifest), perm} + end + + @doc false + @spec simctl_services([plugin()]) :: [{atom(), String.t()}] + def simctl_services(plugins) do + for {name_or_dir, %{} = manifest} <- manifests(plugins), + %{capability: cap} <- Map.get(manifest, :permissions) || [], + service = Map.get(@simctl_services, cap), + do: {plugin_name(name_or_dir, manifest), service} + end + + defp manifests(plugins) do + Enum.map(plugins, fn + {name_or_dir, manifest} -> {name_or_dir, manifest} + {name_or_dir, manifest, _status} -> {name_or_dir, manifest} + end) + end + + defp status(_out, 0), do: :ok + defp status(out, _status), do: {:error, out |> String.trim() |> String.slice(0, 200)} + + # ── reporting ───────────────────────────────────────────────────────────── + + @doc """ + The table `mix mob.selftest` prints for one device: a header plus one + line per entry, `plugin outcome ms detail`. + """ + @spec table([entry()]) :: [String.t()] + def table(entries) do + rows = [["plugin", "outcome", "ms", "detail"] | Enum.map(entries, &row/1)] + + widths = + Enum.zip_with(rows, fn column -> column |> Enum.map(&String.length/1) |> Enum.max() end) + + for row <- rows do + row + |> Enum.zip(widths) + |> Enum.map_join(" ", fn {cell, width} -> String.pad_trailing(cell, width) end) + |> String.trim_trailing() + end + end + + defp row(%{plugin: plugin, result: result, ms: ms}) do + {outcome, detail} = describe(result) + [to_string(plugin), outcome, to_string(ms), detail] + end + + defp describe(:pass), do: {"pass", ""} + defp describe({:fail, reason}), do: {"FAIL", reason} + defp describe({:skip, reason}) when is_atom(reason), do: {"skip", to_string(reason)} + defp describe({:skip, reason}), do: {"skip", reason} + + @doc "The entries whose result is `{:fail, _}`." + @spec failures([entry()]) :: [entry()] + def failures(entries), do: Enum.filter(entries, &match?(%{result: {:fail, _}}, &1)) + + @doc "One line: `N passed, N failed, N skipped`." + @spec summary([entry()]) :: String.t() + def summary(entries) do + counts = Enum.frequencies_by(entries, &outcome/1) + + "#{Map.get(counts, :pass, 0)} passed, #{Map.get(counts, :fail, 0)} failed, " <> + "#{Map.get(counts, :skip, 0)} skipped" + end + + defp outcome(%{result: :pass}), do: :pass + defp outcome(%{result: {:fail, _}}), do: :fail + defp outcome(%{result: {:skip, _}}), do: :skip +end diff --git a/lib/mob_dev/plugin/validator.ex b/lib/mob_dev/plugin/validator.ex index 8e730e7..a7bd036 100644 --- a/lib/mob_dev/plugin/validator.ex +++ b/lib/mob_dev/plugin/validator.ex @@ -631,7 +631,22 @@ defmodule MobDev.Plugin.Validator do defp warnings(manifest) do single_platform_components(manifest) ++ permission_review(manifest) ++ - plist_review(manifest) + plist_review(manifest) ++ + selftest_review(manifest) + end + + # A warning in 0.7.17 so published plugins keep validating; an error in a + # later release once the first-party plugins all carry one (MOB-418). + defp selftest_review(manifest) do + if Map.has_key?(manifest, :selftest) do + [] + else + [ + "missing selftest — declare `selftest: Module` (a Mob.Plugin.SelfTest) so " <> + "`mix mob.selftest` and mob_ci can prove the plugin works on a device; " <> + "a later mob_dev release makes this an error" + ] + end end defp single_platform_components(manifest) do diff --git a/test/mix/tasks/mob_selftest_test.exs b/test/mix/tasks/mob_selftest_test.exs new file mode 100644 index 0000000..4f68ce0 --- /dev/null +++ b/test/mix/tasks/mob_selftest_test.exs @@ -0,0 +1,236 @@ +defmodule Mix.Tasks.Mob.SelftestTest do + @moduledoc """ + `mix mob.selftest` driven end to end with fake discovery, leases, grants, + connect and runner: the exit status has to come from the entries the + runner returns, and the grants have to happen before the app is relaunched. + """ + use ExUnit.Case, async: false + + import ExUnit.CaptureIO + + alias Mix.Tasks.Mob.Selftest, as: SelftestTask + alias MobDev.{Device, DeviceLeases} + + @emulator %Device{ + platform: :android, + serial: "emulator-5554", + name: "Pixel 8", + type: :emulator, + node: :"app_android_emulator_5554@127.0.0.1" + } + + @simulator %Device{ + platform: :ios, + serial: "2CAF98B3-FFAF-42D0-AA93-F62694E2BFE4", + name: "iPhone 17", + type: :simulator, + node: :"app_ios@127.0.0.1" + } + + @plugins [ + {:mob_location, + %{ + name: :mob_location, + selftest: MobLocation.SelfTest, + android: %{permissions: ["android.permission.ACCESS_FINE_LOCATION"]} + }, :ok}, + {:mob_deliver, %{name: :mob_deliver}, :ok} + ] + + defp entries(:pass), + do: [ + %{plugin: :mob_location, module: MobLocation.SelfTest, result: :pass, ms: 4}, + %{plugin: :mob_deliver, module: nil, result: {:skip, "no selftest in manifest"}, ms: 0} + ] + + defp entries(:fail), + do: [ + %{ + plugin: :mob_location, + module: MobLocation.SelfTest, + result: {:fail, "location_stop/0 returned {:error, :nif_not_loaded}"}, + ms: 2 + } + ] + + # Records every step so the test can check what ran, and in which order. + defp deps(devices, outcome, opts \\ []) do + test = self() + failed = Keyword.get(opts, :failed, []) + lost = Keyword.get(opts, :lost, []) + leases = Keyword.get(opts, :leases, %DeviceLeases{}) + + %{ + plugins: fn -> @plugins end, + leases: fn -> leases end, + discover: fn platforms -> + send(test, {:discover, platforms}) + devices + end, + grant: fn device, plugins -> + send(test, {:grant, device.serial}) + + MobDev.Plugin.SelfTest.android_permissions(plugins) + |> Enum.map(fn {p, perm} -> %{plugin: p, permission: perm, status: :ok} end) + end, + connect: fn connect_opts -> + send(test, {:connect, connect_opts}) + only = Keyword.fetch!(connect_opts, :only) + connected = (Enum.filter(devices, &(&1.serial in only)) -- failed) -- lost + {connected, failed} + end, + run_all: fn node, ctx, run_opts -> + send(test, {:run_all, node, ctx, run_opts}) + entries(outcome) + end + } + end + + defp run_task(args, deps) do + capture_io(fn -> send(self(), {:result, SelftestTask.run(args, deps)}) end) + end + + defp steps do + {:messages, messages} = Process.info(self(), :messages) + messages |> Enum.map(&elem(&1, 0)) |> Enum.filter(&(&1 in [:grant, :connect, :run_all])) + end + + test "passes, printing a table per device, when every self-test passed or skipped" do + output = run_task([], deps([@emulator], :pass)) + + assert_received {:result, :ok} + assert output =~ "Pixel 8 (emulator-5554) android emulator" + assert output =~ "mob_location pass 4" + assert output =~ "mob_deliver skip 0 no selftest in manifest" + assert output =~ "1 passed, 0 failed, 1 skipped" + assert output =~ "mob.selftest passed on 1 device(s)" + end + + test "fails (non-zero exit) naming the device and plugin when a self-test failed" do + error = assert_raise Mix.Error, fn -> run_task([], deps([@emulator], :fail)) end + + assert error.message =~ "mob.selftest failed:" + + assert error.message =~ + "Pixel 8 (emulator-5554) mob_location: location_stop/0 returned {:error, :nif_not_loaded}" + end + + test "grants the manifests' permissions before relaunching, then runs with the device's context" do + run_task(["--timeout", "5000"], deps([@emulator], :pass)) + + assert steps() == [:grant, :connect, :run_all] + + assert_received {:discover, [:android, :ios]} + assert_received {:grant, "emulator-5554"} + assert_received {:connect, connect_opts} + assert_received {:run_all, :"app_android_emulator_5554@127.0.0.1", ctx, run_opts} + + assert Keyword.fetch!(connect_opts, :only) == ["emulator-5554"] + assert Keyword.fetch!(connect_opts, :restart) == true + assert Keyword.fetch!(connect_opts, :platforms) == [:android] + assert ctx == %{platform: :android, device: :emulator} + assert Keyword.fetch!(run_opts, :timeout_ms) == 5000 + assert Keyword.fetch!(run_opts, :plugins) == @plugins + end + + test "--no-restart attaches to the running app without granting, --device narrows the targets" do + output = run_task(["--no-restart", "-d", "2CAF98B3"], deps([@emulator, @simulator], :pass)) + + assert_received {:result, :ok} + assert steps() == [:connect, :run_all] + assert_received {:connect, connect_opts} + assert Keyword.fetch!(connect_opts, :restart) == false + assert Keyword.fetch!(connect_opts, :only) == [@simulator.serial] + assert_received {:run_all, :"app_ios@127.0.0.1", %{platform: :ios, device: :simulator}, _} + assert output =~ "iPhone 17" + refute output =~ "Pixel 8" + end + + test "two development devices need --all-devices; with it both run" do + assert_raise Mix.Error, fn -> run_task([], deps([@emulator, @simulator], :pass)) end + refute_received {:connect, _} + + run_task(["--all-devices"], deps([@emulator, @simulator], :pass)) + assert_received {:result, :ok} + assert_received {:run_all, :"app_android_emulator_5554@127.0.0.1", _, _} + assert_received {:run_all, :"app_ios@127.0.0.1", _, _} + end + + test "a device another agent-device session holds is left alone unless named" do + claimed = %DeviceLeases{ + claims: [ + %{ + id: "emulator-5554", + platform: :android, + kind: :emulator, + session: "other", + workspace: "/w" + } + ] + } + + run_task([], deps([@emulator, @simulator], :pass, leases: claimed)) + assert_received {:result, :ok} + refute_received {:grant, "emulator-5554"} + assert_received {:connect, connect_opts} + assert Keyword.fetch!(connect_opts, :only) == [@simulator.serial] + end + + test "a selected device whose node could not be reached fails the run" do + down = %{@simulator | node: nil, status: :error, error: "no node after 30 s"} + + error = + assert_raise Mix.Error, fn -> + run_task(["--all-devices"], deps([@emulator, @simulator], :pass, failed: [down])) + end + + assert error.message =~ + "iPhone 17 (#{@simulator.serial}): node not reachable (no node after 30 s)" + + # The reachable device still ran. + assert_received {:run_all, :"app_android_emulator_5554@127.0.0.1", _, _} + end + + test "a device lost between discovery and connect fails the run instead of passing on 0 devices" do + error = + assert_raise Mix.Error, fn -> + run_task([], deps([@emulator], :pass, lost: [@emulator])) + end + + assert error.message =~ "Pixel 8 (emulator-5554): not found when connecting" + refute_received {:run_all, _, _, _} + end + + test "prints the permissions it granted" do + output = run_task([], deps([@emulator], :pass)) + assert output =~ "Pixel 8 (emulator-5554) permissions:" + assert output =~ "granted android.permission.ACCESS_FINE_LOCATION (mob_location)" + end + + test "no devices is an error, not a pass" do + error = assert_raise Mix.Error, fn -> run_task([], deps([], :pass)) end + assert error.message =~ "No connected devices" + refute_received {:run_all, _, _, _} + end + + test "rejects unknown options, a bad timeout and both platform flags" do + assert_raise Mix.Error, ~r/Unknown option/, fn -> + run_task(["--flows", "x"], deps([], :pass)) + end + + assert_raise Mix.Error, ~r/--timeout must be/, fn -> + run_task(["--timeout", "0"], deps([], :pass)) + end + + assert_raise Mix.Error, ~r/Cannot combine/, fn -> + run_task(["--ios-only", "--android-only"], deps([], :pass)) + end + end + + test "--help prints the usage and runs nothing" do + output = run_task(["--help"], deps([@emulator], :fail)) + assert output =~ "mix mob.selftest" + assert_received {:result, _} + refute_received {:discover, _} + end +end diff --git a/test/mob_dev/plugin/manifest_test.exs b/test/mob_dev/plugin/manifest_test.exs index 37783ce..c6a9a24 100644 --- a/test/mob_dev/plugin/manifest_test.exs +++ b/test/mob_dev/plugin/manifest_test.exs @@ -5,6 +5,11 @@ defmodule MobDev.Plugin.ManifestTest do @valid %{name: :mob_demo, mob_version: "~> 0.6", plugin_spec_version: 1} + defmodule ManifestTestSelfTest do + @moduledoc false + def run(_ctx), do: :pass + end + # ExUnit.CaptureLog hears every process's log events while `fun` runs, and # this module is async: a sibling test's "unknown key(s)" warning lands in # the capture and trips a `refute` (MOB-290). Manifest.validate/1 logs from @@ -187,6 +192,26 @@ defmodule MobDev.Plugin.ManifestTest do assert {:error, _} = Manifest.validate("nope") end + test "selftest names a module; a compiled one must export run/1" do + # Not compiled anywhere: only the name can be checked. + m = Map.put(@valid, :selftest, MobDemo.SelfTest) + log = capture_own_log(fn -> assert {:ok, ^m} = Manifest.validate(m) end) + refute log =~ "unknown key" + + # Loadable, and a real Mob.Plugin.SelfTest shape. + assert {:ok, _} = Manifest.validate(Map.put(@valid, :selftest, ManifestTestSelfTest)) + + # Loadable but not a self-test: the module exists, run/1 does not. + assert {:error, [err]} = Manifest.validate(Map.put(@valid, :selftest, Manifest)) + assert err =~ "does not export run/1" + + assert {:error, [err]} = Manifest.validate(Map.put(@valid, :selftest, "MobDemo.SelfTest")) + assert err =~ "selftest must be a module" + + assert {:error, [err]} = Manifest.validate(Map.put(@valid, :selftest, :mob_demo_nif)) + assert err =~ "must be an Elixir module" + end + test "accepts a permissions list with capability + optional ios handler" do m = Map.put(@valid, :permissions, [ diff --git a/test/mob_dev/plugin/scaffold_test.exs b/test/mob_dev/plugin/scaffold_test.exs index e338ebe..ae2b066 100644 --- a/test/mob_dev/plugin/scaffold_test.exs +++ b/test/mob_dev/plugin/scaffold_test.exs @@ -115,6 +115,28 @@ defmodule MobDev.Plugin.ScaffoldTest do end end + describe "self-tests in the scaffolds" do + test "tier 1 and tier 4 generate a Mob.Plugin.SelfTest and declare it in the manifest" do + for tier <- [1, 4] do + files = Map.new(Scaffold.files_for(tier, "mob_demo_widget")) + self_test = Map.fetch!(files, "lib/mob_demo_widget/self_test.ex") + assert self_test =~ "defmodule MobDemoWidget.SelfTest do" + assert self_test =~ "@behaviour Mob.Plugin.SelfTest" + assert self_test =~ "def run(" + assert Map.fetch!(files, "priv/mob_plugin.exs") =~ "selftest: MobDemoWidget.SelfTest" + end + end + + test "the tier-1 self-test expects what the generated C NIF returns from ping/0" do + files = Map.new(Scaffold.files_for(1, "mob_demo_widget")) + + assert Map.fetch!(files, "priv/native/jni/mob_demo_widget_nif.c") =~ + ~S|enif_make_atom(env, "ok")| + + assert Map.fetch!(files, "lib/mob_demo_widget/self_test.ex") =~ ":ok -> :pass" + end + end + describe "files_for/2 — tier 0" do setup do {:ok, files: Scaffold.files_for(0, "mob_demo_widget")} diff --git a/test/mob_dev/plugin/self_test_test.exs b/test/mob_dev/plugin/self_test_test.exs new file mode 100644 index 0000000..11ba1a0 --- /dev/null +++ b/test/mob_dev/plugin/self_test_test.exs @@ -0,0 +1,338 @@ +defmodule MobDev.Plugin.SelfTestTest do + # The "device" is this node: the self-test is spawned here exactly as it + # would be on a phone, including the exception, exit, kill and timeout paths. + use ExUnit.Case, async: true + + alias MobDev.Device + alias MobDev.Plugin.SelfTest + + @ctx %{platform: :android, device: :emulator} + + defmodule Passes do + @moduledoc false + def run(%{platform: :android}), do: :pass + end + + defmodule Fails do + @moduledoc false + def run(_ctx), do: {:fail, "status/0 returned :error"} + end + + defmodule Skips do + @moduledoc false + def run(%{device: :emulator}), do: {:skip, :needs_hardware} + end + + defmodule SkipsWithReason do + @moduledoc false + def run(_ctx), do: {:skip, "no camera on this device"} + end + + defmodule Raises do + @moduledoc false + def run(_ctx), do: raise(ArgumentError, "nif not loaded") + end + + defmodule Exits do + @moduledoc false + def run(_ctx), do: exit(:shutdown) + end + + defmodule Throws do + @moduledoc false + def run(_ctx), do: throw(:oops) + end + + defmodule Killed do + @moduledoc false + def run(_ctx), do: Process.exit(self(), :kill) + end + + defmodule Hangs do + @moduledoc false + def run(_ctx) do + Process.register(self(), :mob_selftest_hangs) + Process.sleep(:infinity) + end + end + + defmodule OffContract do + @moduledoc false + def run(_ctx), do: :ok + end + + defmodule FailAtom do + @moduledoc false + def run(_ctx), do: {:fail, :nope} + end + + defmodule SkipAtom do + @moduledoc false + def run(_ctx), do: {:skip, :no_reason} + end + + defp plugin(name, module), do: {name, %{name: name, selftest: module}} + + describe "run_all/3" do + test "maps pass, fail and both skip shapes through unchanged, with the plugin and module" do + entries = + SelfTest.run_all(node(), @ctx, + plugins: [ + plugin(:mob_a, Passes), + plugin(:mob_b, Fails), + plugin(:mob_c, Skips), + plugin(:mob_d, SkipsWithReason) + ] + ) + + assert [ + %{plugin: :mob_a, module: Passes, result: :pass}, + %{plugin: :mob_b, module: Fails, result: {:fail, "status/0 returned :error"}}, + %{plugin: :mob_c, module: Skips, result: {:skip, :needs_hardware}}, + %{plugin: :mob_d, result: {:skip, "no camera on this device"}} + ] = entries + + assert Enum.all?(entries, &(is_integer(&1.ms) and &1.ms >= 0)) + end + + test "a raise, exit, throw or kill in one self-test is that plugin's failure and the rest still run" do + entries = + SelfTest.run_all(node(), @ctx, + plugins: [ + plugin(:mob_raise, Raises), + plugin(:mob_exit, Exits), + plugin(:mob_throw, Throws), + plugin(:mob_kill, Killed), + plugin(:mob_ok, Passes) + ] + ) + + assert [ + %{plugin: :mob_raise, result: {:fail, "raised: " <> raised}}, + %{plugin: :mob_exit, result: {:fail, "exited: :shutdown"}}, + %{plugin: :mob_throw, result: {:fail, "threw: :oops"}}, + %{plugin: :mob_kill, result: {:fail, "killed: :killed"}}, + %{plugin: :mob_ok, result: :pass} + ] = entries + + assert raised =~ "ArgumentError" + assert raised =~ "nif not loaded" + end + + test "a self-test that overruns the timeout is killed on the device, fails, and the next one runs" do + entries = + SelfTest.run_all(node(), @ctx, + plugins: [plugin(:mob_slow, Hangs), plugin(:mob_ok, Passes)], + timeout_ms: 50 + ) + + assert [ + %{plugin: :mob_slow, result: {:fail, "timed out after 50 ms"}, ms: slow_ms}, + %{plugin: :mob_ok, result: :pass} + ] = entries + + assert slow_ms >= 50 + assert Process.whereis(:mob_selftest_hangs) == nil + end + + test "a module not on the device, or a return outside the contract, is a failure" do + entries = + SelfTest.run_all(node(), @ctx, + plugins: [ + plugin(:mob_stale, Not.Deployed.SelfTest), + plugin(:mob_bad, OffContract), + plugin(:mob_fail_atom, FailAtom), + plugin(:mob_skip_atom, SkipAtom), + {:mob_string, %{name: :mob_string, selftest: "MobString.SelfTest"}} + ] + ) + + assert [ + %{plugin: :mob_stale, result: {:fail, stale}}, + %{plugin: :mob_bad, result: {:fail, bad}}, + %{plugin: :mob_fail_atom, result: {:fail, fail_atom}}, + %{plugin: :mob_skip_atom, result: {:fail, skip_atom}}, + %{plugin: :mob_string, module: nil, result: {:fail, string}} + ] = entries + + assert stale =~ "Not.Deployed.SelfTest.run/1 is not on the device" + assert bad =~ "returned :ok, not :pass | {:fail, reason} | {:skip, reason}" + assert fail_atom =~ "returned {:fail, :nope}, not" + assert skip_atom =~ "returned {:skip, :no_reason}, not" + assert string == ~s(selftest is not a module: "MobString.SelfTest") + end + + test "plugins without a self-test or a manifest are skips, a failed verification is a failure" do + entries = + SelfTest.run_all(node(), @ctx, + plugins: [ + {"/deps/mob_tier0", nil, :unsigned}, + {"/deps/mob_tampered", nil, {:error, :signature_mismatch}}, + {:mob_old, %{name: :mob_old, nifs: []}}, + {"/deps/mob_nameless", %{name: nil, selftest: Passes}}, + plugin(:mob_ok, Passes) + ] + ) + + assert [ + %{plugin: :mob_tier0, module: nil, result: {:skip, "no manifest (tier-0 plugin)"}}, + %{ + plugin: :mob_tampered, + result: {:fail, "manifest failed verification: :signature_mismatch"} + }, + %{plugin: :mob_old, module: nil, result: {:skip, "no selftest in manifest"}}, + %{plugin: :mob_nameless, result: :pass}, + %{plugin: :mob_ok, result: :pass} + ] = entries + end + + test "waits for the plugins' applications to be started before the first test, then gives up" do + # :kernel is started; :mob_never_starts is not, so the wait runs out. + {ms, entries} = + :timer.tc( + fn -> + SelfTest.run_all(node(), @ctx, + plugins: [ + {:kernel, %{name: :kernel, selftest: Passes}}, + {:mob_never_starts, %{name: :mob_never_starts, selftest: Passes}} + ], + boot_timeout_ms: 300 + ) + end, + :millisecond + ) + + assert [%{plugin: :kernel, result: :pass}, %{plugin: :mob_never_starts, result: :pass}] = + entries + + assert ms >= 300 and ms < 3_000 + + {ms, _} = + :timer.tc( + fn -> + SelfTest.run_all(node(), @ctx, + plugins: [{:kernel, %{name: :kernel, selftest: Passes}}] + ) + end, + :millisecond + ) + + assert ms < 250 + end + + test "an unreachable node is every plugin's failure, not a crash" do + entries = + SelfTest.run_all(:"nope@127.0.0.1", @ctx, + plugins: [plugin(:mob_a, Passes)], + timeout_ms: 500 + ) + + assert [%{plugin: :mob_a, result: {:fail, reason}}] = entries + assert reason =~ "nope@127.0.0.1" + assert reason =~ "noconnection" + end + end + + describe "grant_permissions/4" do + @plugins [ + {:mob_location, + %{ + name: :mob_location, + permissions: [%{capability: :location, ios: %{handler: "x"}}], + android: %{ + permissions: [ + "android.permission.ACCESS_FINE_LOCATION", + "android.permission.ACCESS_COARSE_LOCATION" + ] + } + }, :ok}, + {:mob_whisper, + %{ + name: :mob_whisper, + permissions: [%{capability: :speech}], + android: %{permissions: ["android.permission.RECORD_AUDIO"]} + }}, + {"/deps/mob_deliver", %{name: :mob_deliver, nifs: []}}, + {"/deps/mob_tier0", nil} + ] + + test "android emulator: pm grant per declared permission, recording what the device said" do + cmd = fn "adb", ["-s", "emulator-5554", "shell", "pm", "grant", "com.x.app", perm] -> + if perm =~ "COARSE", + do: {"Operation not allowed: not a changeable permission type", 255}, + else: {"", 0} + end + + device = %Device{platform: :android, serial: "emulator-5554", type: :emulator} + + assert [ + %{ + plugin: :mob_location, + permission: "android.permission.ACCESS_FINE_LOCATION", + status: :ok + }, + %{ + plugin: :mob_location, + permission: "android.permission.ACCESS_COARSE_LOCATION", + status: {:error, "Operation not allowed: not a changeable permission type"} + }, + %{plugin: :mob_whisper, permission: "android.permission.RECORD_AUDIO", status: :ok} + ] = SelfTest.grant_permissions(device, @plugins, "com.x.app", cmd) + end + + test "ios simulator: simctl privacy grant for capabilities that have a service" do + cmd = fn "xcrun", ["simctl", "privacy", "UDID-1", "grant", _service, "com.x.app"] -> + {"", 0} + end + + device = %Device{platform: :ios, serial: "UDID-1", type: :simulator} + + # :speech has no simctl service, so only location is attempted. + assert [%{plugin: :mob_location, permission: "location", status: :ok}] = + SelfTest.grant_permissions(device, @plugins, "com.x.app", cmd) + end + + test "physical devices get nothing granted from the host" do + cmd = fn _exe, _argv -> flunk("no command expected") end + + for device <- [ + %Device{platform: :ios, serial: "00008030-ABCDEF", type: :physical}, + %Device{platform: :android, serial: "ZY22DP6HFL", type: :physical} + ] do + assert SelfTest.grant_permissions(device, @plugins, "com.x.app", cmd) == [] + end + end + end + + describe "table/1 and summary/1" do + test "one aligned row per plugin with the outcome and its detail" do + entries = [ + %{plugin: :mob_location, module: Passes, result: :pass, ms: 12}, + %{ + plugin: :mob_whisper, + module: Fails, + result: {:fail, "nif_loaded/0 returned false"}, + ms: 3 + }, + %{plugin: :mob_deliver, module: nil, result: {:skip, "no selftest in manifest"}, ms: 0}, + %{plugin: :mob_camera, module: Skips, result: {:skip, :needs_hardware}, ms: 1} + ] + + assert SelfTest.table(entries) == [ + "plugin outcome ms detail", + "mob_location pass 12", + "mob_whisper FAIL 3 nif_loaded/0 returned false", + "mob_deliver skip 0 no selftest in manifest", + "mob_camera skip 1 needs_hardware" + ] + + assert SelfTest.summary(entries) == "1 passed, 1 failed, 2 skipped" + assert [%{plugin: :mob_whisper}] = SelfTest.failures(entries) + end + + test "an empty run is an empty table body and a zero summary" do + assert SelfTest.table([]) == ["plugin outcome ms detail"] + assert SelfTest.summary([]) == "0 passed, 0 failed, 0 skipped" + end + end +end diff --git a/test/mob_dev/plugin/validator_test.exs b/test/mob_dev/plugin/validator_test.exs index bf6c9bf..a8558db 100644 --- a/test/mob_dev/plugin/validator_test.exs +++ b/test/mob_dev/plugin/validator_test.exs @@ -3,7 +3,12 @@ defmodule MobDev.Plugin.ValidatorTest do alias MobDev.Plugin.Validator - @base %{name: :mob_demo, mob_version: "~> 0.6", plugin_spec_version: 1} + @base %{ + name: :mob_demo, + mob_version: "~> 0.6", + plugin_spec_version: 1, + selftest: MobDemo.SelfTest + } describe "referenced_paths/1" do test "nil manifest has no paths" do @@ -85,6 +90,13 @@ defmodule MobDev.Plugin.ValidatorTest do assert %{warnings: []} = Validator.validate_plugin(m, dir, "0.6.20") end + test "warns, but does not fail, a manifest without a selftest", %{dir: dir} do + m = Map.delete(@base, :selftest) + assert %{errors: [], warnings: [warning]} = Validator.validate_plugin(m, dir, "0.6.20") + assert warning =~ "missing selftest" + assert warning =~ "Mob.Plugin.SelfTest" + end + test "warns when permissions are declared", %{dir: dir} do m = Map.put(@base, :android, %{permissions: ["android.permission.CAMERA"]}) assert %{warnings: warns} = Validator.validate_plugin(m, dir, "0.6.20")