diff --git a/CHANGELOG.md b/CHANGELOG.md index c7f2490..87e99ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,18 @@ Full module documentation: [hexdocs.pm/mob](https://hexdocs.pm/mob). ## [Unreleased] +### Added +- **Plugin self-test contract** (MOB-411). `Mob.Plugin.SelfTest` is a + behaviour with one callback, `run(%{platform:, device:}) :: :pass | + {:fail, reason} | {:skip, :needs_hardware | :needs_user | reason}`. A + plugin names its implementation in the manifest as `selftest: Module`; + `mix mob.selftest` (mob_dev 0.7.17) and mob_ci call it on the device + over distribution for every activated plugin. A pass means the test got + a real answer back from the plugin's native code, not that a module + loaded. `Mob.Plugin.SelfTest.result?/1` says whether a term is a valid + result. See `MOB_PLUGINS.md` ("Self-test") and + `decisions/2026-10-08-plugin-self-test-contract.md`. + ## [0.9.14] - 2026-10-08 ### Added diff --git a/MOB_PLUGINS.md b/MOB_PLUGINS.md index 02a0d70..42a7b30 100644 --- a/MOB_PLUGINS.md +++ b/MOB_PLUGINS.md @@ -775,6 +775,13 @@ Setup section (tier 3+): Optional; mostly for tier-3/4 plugins. Reserved: mob_dev accepts the key, but nothing reads it yet. +Self-test (any manifest, tiers 1–4; a tier-0 package has no manifest to +declare one in, and adding a manifest moves it into this model): + +- `:selftest` — a module implementing `Mob.Plugin.SelfTest`. See + "Self-test" below. Missing: mob_dev warns today and will refuse the + plugin in a later release. + ## Validation rules `mix mob.validate_plugin` (run from a plugin project) checks: @@ -788,6 +795,9 @@ Setup section (tier 3+): - `permissions` and `plist_keys` declared (warning + manual review recommended before publishing) - `mob_version` satisfied by the version of `:mob` in deps +- `selftest` names a module; when the plugin is compiled, that module + exports `run/1`. A manifest without `selftest` is a warning (an error + in a later mob_dev) Compile-time validation (run by mob_dev when activating plugins): @@ -797,6 +807,58 @@ Compile-time validation (run by mob_dev when activating plugins): Both stages fail loud — never silent. +### Self-test + +Static validation checks manifests on paper. It cannot see a NIF that +links but aborts in `nif_init`, a Kotlin bridge that compiles alone but +not next to another plugin's, or an Objective-C delegate the release link +dropped. So every plugin ships its own proof: a module implementing +`Mob.Plugin.SelfTest`, named in the manifest: + +```elixir +%{ + name: :mob_location, + # ... + selftest: MobLocation.SelfTest +} +``` + +```elixir +defmodule MobLocation.SelfTest do + @behaviour Mob.Plugin.SelfTest + + @impl true + def run(%{platform: platform, device: _}) do + # A no-op while nothing runs, but it goes through the NIF into + # CLLocationManager / the Kotlin bridge: :ok proves the native side + # is linked and initialised. + case :mob_location_nif.location_stop() do + :ok -> :pass + other -> {:fail, "location_stop/0 on #{platform} returned #{inspect(other)}"} + end + end +end +``` + +`run/1` gets `%{platform: :ios | :android, device: :simulator | :emulator +| :physical}` and answers `:pass`, `{:fail, "why"}` or `{:skip, +:needs_hardware | :needs_user | "why"}`. The rule for `:pass`: the test +got a real answer back from native code (or, for a pure-Elixir plugin, +from its real API path). Loading a module or reading config is not a +pass. A skip is for a resource the device does not have, after the test +has proved everything it can without it; base `:needs_hardware` on what +the native side reports is absent, not on the device being an emulator. + +The host runs them with `mix mob.selftest` (mob_dev; attaches to the +running app on each selected device, pre-grants the manifest's declared +permissions on emulators and simulators, calls every activated plugin's +`run/1` over distribution with a 30 s timeout, prints a table and exits +non-zero on any failure). mob_ci runs the same per nightly cell and +attributes a failure to the plugin when it fails alone, or to the plugin +set when it fails only in company. A raise, timeout or a return outside +the contract counts as a failure, so a test can assert with pattern +matches instead of rescuing. + ### Cross-plugin conflict detection Anyone can ship a plugin, and a host can activate any combination — so when two diff --git a/decisions/2026-10-08-plugin-self-test-contract.md b/decisions/2026-10-08-plugin-self-test-contract.md new file mode 100644 index 0000000..cded192 --- /dev/null +++ b/decisions/2026-10-08-plugin-self-test-contract.md @@ -0,0 +1,66 @@ +# Plugins carry their own on-device proof: `Mob.Plugin.SelfTest` + +- Date: 2026-10-08 +- Status: accepted +- Linear: MOB-411 (MOB-410 epic; MOB-418 for the remaining plugins) + +## Context + +Every first-party plugin has ubuntu-only CI that never compiles its native +code. mob_ci (the nightly device CI, `~/code/mob_ci`) proves a NIF +initialised with a "safe probe" per plugin that it keeps in its own +`priv/device_caps.exs`: `{:location_stop, []}` for mob_location, +`{:video_probe, ["/nonexistent"]}` for mob_video, `nil` for plugins whose +only exports open UI. That table is knowledge about plugin internals held +by a repo that does not own them, it went stale as soon as it was written +(26 plugins, 10 rows), and `nil` rows mean "we checked the module loaded", +which MOB-372/373/376 showed says nothing: those were "dev build works, +release doesn't" bugs that only a real native call sees. + +## Decision + +The proof moves into the plugin. `mob` gains a behaviour with one callback: + +```elixir +@callback run(%{platform: :ios | :android, device: :simulator | :emulator | :physical}) :: + :pass | {:fail, String.t()} | {:skip, :needs_hardware | :needs_user | String.t()} +``` + +declared in the manifest as `selftest: Module`. The rule for `:pass` is +that the test got a real answer back from native code (or, for a +pure-Elixir plugin, from its real API path); loading a module is not a +pass. `{:skip, _}` is for a resource the device lacks, after the test has +proved what it can without it. The runner (mob_dev's +`MobDev.Plugin.SelfTest.run_all/3`, `mix mob.selftest`; mob_ci's P12) +treats a raise, exit, timeout or off-contract return as a failure, so +tests assert with pattern matches instead of rescuing. + +`mob` owns only the contract: the behaviour, its types, `result?/1` and the +documentation in `MOB_PLUGINS.md`. mob_dev owns validation (the manifest +key, a warning when missing that becomes an error one release later) and +the runner; mob_ci owns attribution (failing alone is `plugin:

`, failing +only in company is `conflict:`). The runner passes the context rather +than the plugin detecting it, so a plugin cannot mistake a simulator for a +phone and skip where it should prove. + +## Consequences + +- `Mob.Plugin.*` is a new namespace for behaviours a plugin implements; + `Mob.Plugins` (plural) stays the on-device registry. +- mob_ci's `device_caps.exs` shrinks to what the plugin cannot know (which + fixture host can build it) once every plugin has a self-test (MOB-418). +- A plugin without a self-test is a warning from mob_dev 0.7.17 and an + error from a later release; pilots are mob_location, mob_whisper and + mob_deliver. + +## Alternatives considered + +- **Keep the probe table in mob_ci.** Rejected: it is knowledge about + plugin internals held outside the plugin, and it cannot express "call + this and expect that" without becoming a second test framework. +- **A lifecycle hook (`lifecycle.selftest`) instead of a behaviour.** + Rejected: lifecycle MFAs are tier-4 and run at boot; a self-test runs on + demand from the host, with a context, and wants a typed result. +- **Detect the context on the device (`Mob.Device`).** Rejected: the + runner already knows the device it attached to, and a plugin deciding + for itself that it is "on an emulator" is how a skip hides a real hole. diff --git a/lib/mob/plugin/self_test.ex b/lib/mob/plugin/self_test.ex new file mode 100644 index 0000000..3e75a9a --- /dev/null +++ b/lib/mob/plugin/self_test.ex @@ -0,0 +1,123 @@ +defmodule Mob.Plugin.SelfTest do + @moduledoc """ + A plugin's own on-device proof that it works in the host it was built into. + + A plugin declares the module in its manifest (`priv/mob_plugin.exs`): + + %{ + name: :mob_location, + # ... + selftest: MobLocation.SelfTest + } + + and mob_dev's `mix mob.selftest` (or mob_ci, invariant P12) calls + `run/1` on the device over distribution for every activated plugin. The + result is attributed to that plugin: a plugin whose self-test fails on its + own has a bug; one that fails only when other plugins are active has a + conflict. Without a self-test the ecosystem's CI can only prove that the + plugin's NIF module loaded, which says nothing about whether `nif_init` + ran, the Kotlin bridge registered or the Objective-C delegate exists. + + ## The three outcomes + + * `:pass` — the plugin did its job on this device. The test MUST get a + real answer back from the plugin's native code (a NIF export, a JNI + bridge call, an Objective-C or Swift entry point) or, for a pure-Elixir + plugin, from its real API path (a store, a verifier, a client). Loading + a module, checking `function_exported?/3` or reading config proves + nothing about the build and does not count. Prefer a side-effect-free + export: a status query, a version string, a stop/cancel that is a no-op + while nothing runs. `erlang:nif_error(nif_not_loaded)` from the stub + means the native library was not linked; that is a `{:fail, _}`. + + * `{:fail, reason}` — the plugin is broken here. `reason` is a sentence + for a human reading a CI report: say what was called, what came back + and what was expected. A raise, exit, throw, timeout or a return value + outside this contract is also counted as a failure by the runner, so a + test does not need to rescue: `{:ok, _} = :my_nif.status()` is a fine + assertion. + + * `{:skip, why}` — the test cannot run on this device, not a verdict on + the plugin. `:needs_hardware` for a sensor, radio or camera this + device does not have; `:needs_user` when a person must tap a system + dialog or unlock something; a string for anything else (say what is + missing). Reach for a skip only after the test has done everything it + can without that resource: a plugin that can prove its NIF initialised + and *then* finds no GPS passes; a GPS fix is a feature, not the proof. + Base `:needs_hardware` on what the native side reports is absent (no + NFC controller, no LiDAR, no cellular radio), not on + `device: :emulator` alone: a skip that is really "I did not look" + hides a hole, and one that assumes a phone has everything fails on + the tablet without a radio. + + ## The context + + `run/1` receives `%{platform: :ios | :android, device: :simulator | + :emulator | :physical}` so the test can choose what to prove: a simulator + has no camera, an emulator has a fake GPS, a physical device usually has + the hardware but needs the user for permissions. The runner pre-grants the + permissions the manifest declares on emulators and simulators before + calling the test, so `:needs_user` is for prompts that cannot be granted + from the host. + + ## Writing one + + defmodule MobLocation.SelfTest do + @behaviour Mob.Plugin.SelfTest + + @impl true + def run(%{platform: platform}) do + # location_stop/0 is a no-op while no updates run, but it goes + # through the NIF into CLLocationManager / the Kotlin bridge, so + # :ok proves the native side is linked and initialised. + case :mob_location_nif.location_stop() do + :ok -> :pass + other -> {:fail, "location_stop/0 on \#{platform} returned \#{inspect(other)}, expected :ok"} + end + end + end + + Keep it under a few seconds: the runner's default timeout is 30 s and the + test runs once per activated plugin, per device, per nightly cell. Leave + the device as you found it (stop what you start, delete what you write). + """ + + @typedoc "Where the test is running." + @type ctx :: %{platform: :ios | :android, device: :simulator | :emulator | :physical} + + @typedoc "Why the test could not run on this device." + @type skip_reason :: :needs_hardware | :needs_user | String.t() + + @typedoc "What `run/1` returns." + @type result :: :pass | {:fail, String.t()} | {:skip, skip_reason()} + + @doc """ + Prove the plugin works on this device, or say why that cannot be checked. + See the module documentation for what each outcome means. + """ + @callback run(ctx()) :: result() + + @doc """ + Whether `term` is a result `run/1` may return. + + The runner treats anything else as a failure: a test that returns `:ok` or + `true` has not said whether the plugin works. + + iex> Mob.Plugin.SelfTest.result?(:pass) + true + iex> Mob.Plugin.SelfTest.result?({:skip, :needs_hardware}) + true + iex> Mob.Plugin.SelfTest.result?({:fail, "status/0 returned :error"}) + true + iex> Mob.Plugin.SelfTest.result?(:ok) + false + iex> Mob.Plugin.SelfTest.result?({:skip, :no_reason}) + false + """ + @spec result?(term()) :: boolean() + def result?(:pass), do: true + def result?({:fail, reason}) when is_binary(reason), do: true + def result?({:skip, reason}) when reason in [:needs_hardware, :needs_user], do: true + def result?({:skip, reason}) when is_binary(reason), do: true + def result?(_), do: false +end diff --git a/mix.exs b/mix.exs index a540fdf..0f9c445 100644 --- a/mix.exs +++ b/mix.exs @@ -190,7 +190,7 @@ defmodule Mob.MixProject do ], Navigation: [Mob.Nav, Mob.Nav.Registry, Mob.Router], "Runtime Processes": [Mob.Screen.Server, Mob.Listener, Mob.Sender], - Plugins: [Mob.Plugins, Mob.Plugins.Supervisor, Mob.Plugins.Lifecycle], + Plugins: [Mob.Plugins, Mob.Plugins.Supervisor, Mob.Plugins.Lifecycle, Mob.Plugin.SelfTest], "Device APIs": [ Mob.Haptic, Mob.Clipboard, diff --git a/test/mob/plugin/self_test_test.exs b/test/mob/plugin/self_test_test.exs new file mode 100644 index 0000000..625d818 --- /dev/null +++ b/test/mob/plugin/self_test_test.exs @@ -0,0 +1,37 @@ +defmodule Mob.Plugin.SelfTestTest do + use ExUnit.Case, async: true + + alias Mob.Plugin.SelfTest + + doctest SelfTest + + defmodule Probe do + @moduledoc false + @behaviour Mob.Plugin.SelfTest + + @impl true + def run(%{platform: :ios, device: :simulator}), do: {:skip, :needs_hardware} + def run(%{platform: platform}), do: {:fail, "no native code on #{platform}"} + end + + test "the behaviour asks for run/1 only" do + assert SelfTest.behaviour_info(:callbacks) == [run: 1] + assert SelfTest.behaviour_info(:optional_callbacks) == [] + end + + test "an implementation answers per context with a result the runner accepts" do + sim = Probe.run(%{platform: :ios, device: :simulator}) + phone = Probe.run(%{platform: :android, device: :physical}) + + assert sim == {:skip, :needs_hardware} + assert phone == {:fail, "no native code on android"} + assert SelfTest.result?(sim) and SelfTest.result?(phone) + end + + test "result?/1 rejects shapes outside the contract" do + refute SelfTest.result?({:fail, :nif_not_loaded}) + refute SelfTest.result?({:skip, nil}) + refute SelfTest.result?({:pass, "extra"}) + refute SelfTest.result?(true) + end +end