From e425fac95904a82428635de98c5ab5f0b1d44ef4 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Sun, 23 Aug 2026 22:10:19 +0800 Subject: [PATCH 01/20] Call every adapter, with a stub grown from the callee's own contract The MCP handler table and the AC_* dispatch table are the same layer twice: a thousand short functions that take a client's arguments, call one headless function, and return something json.dumps can encode. They were the two least-covered files in the package because each is two to eight lines, so the per-feature test that touches one is testing the feature, not the wiring -- and wiring fails only when a client calls it. Sweep both registries with arguments taken from the schemas that describe them, never from the adapter source, so neither sweep can pass by restating the code it checks. The callee is replaced by a stub built from its return annotation, which only became possible once the typing contract's exemption list was emptied: an adapter now runs against exactly what its callee promises, with no mouse, display or network. Three MCP adapters need more than the contract offers; each is named with a reason, and a further test deletes the entry when one starts passing. One convention becomes a rule in passing: a mandatory executor parameter with no Script Builder field must be a shape no scalar field could express, or the editor emits an action that raises on first run. --- Progress.md | 76 ++- WHATS_NEW.md | 65 ++ .../headless/test_adapter_registry_sweep.py | 591 ++++++++++++++++++ 3 files changed, 719 insertions(+), 13 deletions(-) create mode 100644 test/unit_test/headless/test_adapter_registry_sweep.py diff --git a/Progress.md b/Progress.md index 153e4c4d..95143ddf 100644 --- a/Progress.md +++ b/Progress.md @@ -220,22 +220,22 @@ capability enum 值與 variadic `ei_seat_bind_capabilities`、event-type enum --- -## 兩個門檻：mypy 那半到終點了，覆蓋率那半是量錯了 +## 兩個門檻：mypy 那半到終點了，覆蓋率那半在往 80 爬 -`DECIDE` — 兩半都做完了，只剩「下一個覆蓋率目標是多少」要維護者拍板 +`TODO` — 型別那半 2026-08-22 收工；覆蓋率那半有目標了（80），還沒到 原本這一條記的是兩個只存在於 `pyproject.toml` 註解裡、沒有任何機制的承諾。 -2026-08-21 把**機制**補上了（做法見 [WHATS_NEW.md](WHATS_NEW.md)），兩半也都走完了： +2026-08-21 把**機制**補上了（做法見 [WHATS_NEW.md](WHATS_NEW.md)）： 型別契約的豁免清單 2026-08-22 清空，平台縫最後兩個名稱（`keyboard`／`mouse`） 2026-08-23 拿到合約；覆蓋率那半發現不是爬得不夠，是量測起點錯了，修正後地板 從 50 提到 69。 -**這一條還留著，是因為只剩一個問題要維護者回答：下一個覆蓋率目標值是多少。** -另外兩節留著是因為它們記的那幾個坑之後還會踩到。 +**2026-08-23 維護者拍板：下一個覆蓋率目標是 80。** 這一條留到那時候。 +型別那一節留著是因為它記的那幾個坑之後還會踩到。 -### 覆蓋率：目標 70 其實早就到了，是量錯了 +### 覆蓋率：地板 69，下一站 80 -`DECIDE` — 地板已經設成修正後矩陣的最低那一格（69）；**下一個目標值要維護者定** +`TODO` — 目標 **80**（2026-08-23 拍板）。地板本身照舊只跟著實測的最低那一格走。 `fail_under` 一度從 35 提到 50，理由寫在 `pyproject.toml`。**那兩個數字都低了大約 24 點**，而原因不在測試，在量測的起點： @@ -277,12 +277,62 @@ import 期程式碼（`def` 行、類別本體、常數、兩張大分派表） Windows 是高的那一角，因為門面 import 進來的是**它自己那個平台的後端**； 換句話說剩下的那 30 點裡，有一部分是任何單一平台都拿不到的。 -**還要決定的**：70 是舊的目的地，現在等於已經到了，**下一個目標值該由維護者定**。 -真正還低的是哪幾塊，現在有實測（本機 Windows／3.14，修正後）： -`utils/remote_desktop` 35%、`utils/mcp_server` 34%（`_handlers.py` 自己 10%）、 -`utils/executor` 41%、`utils/accessibility` 29%、`wrapper/window_backends` 10%。 -這四塊的共同形狀是「一大堆薄轉接函式包著已經測過的無頭函式」，所以往上爬的方式 -是走註冊表逐一驅動，而不是一支一支手寫測試。 +#### 往 80 怎麼爬：走註冊表，而且替身要從型別合約長出來 + +第一批 2026-08-23 落地（`test/unit_test/headless/test_adapter_registry_sweep.py`， +做法見 [WHATS_NEW.md](WHATS_NEW.md)）。兩個註冊表、一支測試檔： + +| 掃什麼 | 參數從哪來 | 換掉什麼 | +| --- | --- | --- | +| 657 個 MCP 工具 | 工具自己宣告的 JSON schema（`_factories.py`） | 被呼叫者 | +| 773 個 `AC_*` 命令 | Script Builder 的 `command_schema.py`；沒有 spec 的用參數標注 | 被呼叫者 | + +**關鍵不是「掃」，是替身怎麼來的**：替身的回傳值是從**被呼叫者自己的回傳標注** +造出來的（`Optional[X]` → `None`、容器 → 空容器、純量 → 零值）。這件事 +2026-08-22 之前做不到——那時候還有 136 個模組不在型別契約裡，沒有東西可讀。 +契約清空之後，「這個轉接函式有權假設什麼」變成程式讀得出來的東西，於是 460 個 +MCP adapter 與 372 個執行器 adapter 可以在沒有滑鼠、沒有螢幕、沒有網路的情況下 +整段跑完，而且跑的是**剛好等於合約承諾的東西，不多也不少**。 + +實測（本機 Windows／3.14）： + +| | 之前 | 之後 | +| --- | ---: | ---: | +| `_handlers.py` | 37.47% | **75.21%** | +| `action_executor.py` | 55.78% | **73.52%** | +| 全專案 | 71.91% | **74.91%** | + +掃不到的是兩種形狀，而且是刻意的：adapter 伸手進**兩個**專案模組（那是組合， +不是轉接），以及 adapter 自己 `import` 第三方套件（那是它在挑後端，回傳取決於 +機器）。三個 MCP adapter 需要的比合約承諾的更多，用名字列在測試檔裡，各附一行 +理由；**其中任何一個哪天開始通過，測試會要求把它刪掉**，清單不會爛在那裡。 + +#### 剩下的在哪裡（2026-08-23 實測，本機 Windows／3.14） + +兩支 sweep 之後的數字（`utils/executor` 與 `utils/mcp_server` 已經含進去了）： + +| 子系統 | 沒蓋到 / 總 statement | 覆蓋率 | +| --- | ---: | ---: | +| `utils/remote_desktop` | 2,923 / 6,622 | 55.9% | +| `utils/accessibility` | 822 / 1,397 | 41.2% | +| `utils/executor` | 797 / 3,906 | 79.6% | +| `utils/mcp_server` | 761 / 4,618 | 83.5% | +| `utils/usb` | 455 / 2,137 | 78.7% | +| `wrapper/window_backends` | 397 / 477 | 16.8% | +| `utils/hotkey` | 221 / 426 | 48.1% | +| `utils/rest_api` | 220 / 808 | 72.8% | + +**一件之前沒量過、但決定 80 到底可不可能的事**：其他平台的程式碼 +（X11／Wayland／macOS／Android／iOS）在這一格是 3,291 個 statement， +其中 1,854 個沒被蓋到。就算那些永遠蓋不到，這一格的上限仍有 96%， +所以**80 不是被「單一平台拿不到」卡住的**——純粹是量的問題： +可攜程式碼還有 9,539 個 statement 沒被執行過。 + +**最大的那一塊有個前提要先決定**：`utils/remote_desktop` 佔了將近三分之一， +但 `quality.yml` 不裝 `[webrtc]` extra，所以 webrtc 那一族在 CI 上**永遠是 0%**， +在那裡補的測試會整批 skip，對地板一個點也不會動。要嘛先讓 CI 裝那個 extra， +要嘛承認那 2,900 個 statement 是分母裡的死重、把 80 算在其餘部分上。 +**兩條路都可以，但得先選一條再動手。** ### mypy：整包把關，**豁免清單已經清空** diff --git a/WHATS_NEW.md b/WHATS_NEW.md index dcc58e16..fc21eea9 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,70 @@ # What's New — AutoControl +## What's new (2026-08-23) + +### A Thousand Adapters Now Get Called, Because the Type Contract Can Build Their Stubs + +`utils/mcp_server/tools/_handlers.py` and the `AC_*` dispatch table in +`utils/executor/action_executor.py` are the same layer twice: about a thousand +short functions whose whole job is to take a client's arguments, call one +headless function, and hand back something that survives `json.dumps`. They were +also the two least-covered files in the project — `_handlers.py` at 37.47%, +`action_executor.py` at 55.78% — and not through neglect. Each adapter is two to +eight lines, so the per-feature test that touches one is testing the feature; the +adapter itself, which is where the wiring lives, was checked by nobody. Wiring +fails only when a client calls it. + +Both registries are now swept, in one file: +`test/unit_test/headless/test_adapter_registry_sweep.py`. 657 MCP tools driven +with the arguments their own JSON schema declares, and 773 `AC_*` commands driven +with the arguments the Script Builder's `command_schema.py` says a client sends. +Neither sweep reads an adapter's source to decide what to pass it, so neither can +pass by restating the code it is checking. + +**What made this possible was the other gate that landed last week.** The +problem with calling a thousand adapters is what to do about the thing each one +calls: a real call moves a mouse, and hand-writing a fake per callee is a +thousand guesses at what each returns. Since the typing contract's exemption +list was emptied on 2026-08-22, there is a third option — every callee's return +annotation is machine-readable, so the stub can be *derived* from it. +`Optional[X]` becomes None, containers become empty containers, scalars become +their zero. Each adapter then runs against exactly what its callee promises, no +more and no less, with no mouse, no display and no network in the picture. + +That is the sharper test as well as the cheaper one: an adapter that needs *more* +than the contract offers now says so here instead of in front of a client. Three +do, and each is named in the file with its reason — one uses the return value as +a context manager, one imports a class rather than a function, one indexes a key +out of a `Dict[str, Any]`. A fourth test fails if any of the three starts +passing, so the list cannot quietly rot. + +| | before | after | +| --- | ---: | ---: | +| `_handlers.py` | 37.47% | **75.21%** | +| `action_executor.py` | 55.78% | **73.52%** | +| whole package | 71.91% | **74.91%** | + +Everything was green on the first run, so these are guards rather than a bug +report — and a guard is worth only what it catches, so it was checked by breaking +it: swapping two arguments inside one adapter (`_open_path(verb, target)`) fails +the forwarding sweep by name, pointing at the tool and both values. + +**One invariant did fall out of building it.** 31 mandatory executor parameters +have no field in the visual editor, and every one of them is annotated `Any`, +`List[...]` or `Dict[...]` — a shape the editor's scalar field types cannot +express, filled from its raw JSON view instead. That was true by convention and +enforced by nobody. A mandatory *scalar* with no field is a different thing +entirely: it means the editor emits an action that raises `TypeError` the first +time it is run. That is a test now, alongside two more of the same kind — every +schema field names a parameter its command accepts, and every command the editor +can emit is a command the dispatch table resolves. + +Out of scope by design, and stated in the file rather than left to be +rediscovered: an adapter that reaches into two project modules composes them +rather than normalising one, and an adapter that imports a third-party module is +picking its own backend, so what it returns depends on the machine — the +opposite of what a sweep can assert. + ## What's new (2026-08-21) ### Two Quality Gates That Had Been Standing Still diff --git a/test/unit_test/headless/test_adapter_registry_sweep.py b/test/unit_test/headless/test_adapter_registry_sweep.py new file mode 100644 index 00000000..9f278436 --- /dev/null +++ b/test/unit_test/headless/test_adapter_registry_sweep.py @@ -0,0 +1,591 @@ +"""The two adapter registries, swept for what only a real call would reveal. + +``utils/mcp_server/tools/_handlers.py`` and the ``AC_*`` dispatch table in +``utils/executor/action_executor.py`` are the same layer twice: about a thousand +short functions whose whole job is to take a client's arguments, call one +headless function, and hand back something that survives ``json.dumps``. Each is +two to eight lines, so the per-feature tests that touch them are testing the +feature; the adapter itself -- the wiring -- is checked by nobody, and it fails +only when a client calls it. + +The technique both sweeps share is what makes them possible at all: **the callee +is replaced by a stub built from its own return annotation**. Before the typing +contract was emptied of exemptions on 2026-08-22 that could not be done -- 136 +modules had nothing to read. Now "what is this adapter entitled to assume?" is a +question a program can answer, so an adapter can be run against exactly what its +callee promises, no more and no less, with no mouse, no display and no network +in the picture. An adapter that needs *more* than the promise shows up here +rather than in front of a client. + +Arguments never come from the adapter's source. The MCP sweep takes them from +the tool's declared JSON schema in ``_factories.py``; the executor sweep takes +them from the Script Builder's ``command_schema.py`` where a command has an +entry, and from the adapter's own parameter annotations where it does not. Both +sources are maintained in different files from the adapters, so neither sweep +can pass by restating the code it checks. + +What this catches: a required property with no matching parameter, a declared +property the adapter rejects, a mandatory parameter the schema never fills, a +swapped or dropped argument, a "pass-through" that is not one, a command name +the table does not resolve, and a return value the JSON boundary cannot encode. + +Out of scope by design: an adapter that reaches into two project modules +(it composes them rather than normalising one), and one that imports a +third-party module (it picks its own backend, so what it returns depends on the +machine -- the opposite of what a sweep can assert). +""" +import ast +import importlib +import inspect +import json +import sys +import textwrap +import typing +from typing import Any, Callable, Dict, List, Optional, Tuple + +import pytest + +from je_auto_control.gui.script_builder.command_schema import ( + FieldSpec, FieldType, _build_specs, +) +from je_auto_control.utils.executor.action_executor import executor +from je_auto_control.utils.mcp_server.tools import ( + MCPContent, MCPTool, build_default_tool_registry, +) + +_NONE_TYPE = type(None) + +# MCP adapters that need more than their callee's return annotation promises. +# Each is a real coupling the type contract does not express, not a stub defect: +_NEEDS_MORE_THAN_THE_CONTRACT = { + # Uses the return value as a context manager; the annotation is untyped. + "ac_list_monitors", + # Its single import is a class, so there is no return annotation to build. + "ac_anchor_click", + # Indexes a key out of a Dict[str, Any] the annotation cannot promise. + "ac_tween_drag", +} + + +# === Reading a value out of a declared type ================================= + +# Sample values by JSON-Schema type. Strings carry their property name so a +# swapped pair of same-typed arguments shows as a mismatch, not as two equal +# placeholders. +_SCALARS: Dict[str, Any] = { + "integer": 3, "number": 1.5, "boolean": True, + "array": [], "object": {}, "null": None, +} + +# One sample per Script Builder field type; a field's own default wins where it +# has one, so enums and paths stay inside the values the editor would offer. +_FIELD_SAMPLES: Dict[FieldType, Callable[[FieldSpec], Any]] = { + FieldType.STRING: lambda field: f"value-for-{field.name}", + FieldType.INT: lambda field: 3, + FieldType.FLOAT: lambda field: 1.5, + FieldType.BOOL: lambda field: True, + FieldType.ENUM: lambda field: field.choices[0] if field.choices else "", + FieldType.FILE_PATH: lambda field: "sample.txt", + FieldType.RGB: lambda field: [1, 2, 3], +} + +# The emptiest value each concrete annotation allows, as factories so two +# adapters never share one mutable container. +_ZEROS: Dict[Any, Callable[[], Any]] = { + int: lambda: 0, float: lambda: 0.0, bool: lambda: False, + str: lambda: "", bytes: lambda: b"", + list: list, dict: dict, set: set, tuple: tuple, +} + + +def _sample_value(spec: Dict[str, Any], name: str) -> Any: + """Return a value satisfying one JSON-Schema property node.""" + enum = spec.get("enum") + if isinstance(enum, list) and enum: + return enum[0] + kind = spec.get("type", "string") + if isinstance(kind, list): + kind = next((entry for entry in kind if entry != "null"), "string") + if kind == "array": + item = spec.get("items") + return [_sample_value(item, name)] if item else [] + if kind == "string": + return f"value-for-{name}" + return _SCALARS.get(kind, f"value-for-{name}") + + +def _field_value(field: FieldSpec) -> Any: + """Return a value the visual editor could have produced for one field.""" + if field.default is not None and field.field_type is not FieldType.RGB: + return field.default + return _FIELD_SAMPLES[field.field_type](field) + + +def _annotated_value(annotation: Any, name: str) -> Any: + """Return an argument value for a parameter with no schema field. + + Structured parameters (the editor leaves those to its raw JSON view) get an + empty container; scalars get a sample of their annotated type. ``Any`` and + unannotated parameters get None, which is why a command carrying one is only + swept when the schema names it. + """ + origin = typing.get_origin(annotation) + arguments = typing.get_args(annotation) + if origin is typing.Union: + if _NONE_TYPE in arguments: + return None + return _annotated_value(arguments[0], name) + if origin in (list, tuple, set): + return [] + if origin is dict: + return {} + scalars = {int: 3, float: 1.5, bool: True, str: f"value-for-{name}"} + return scalars.get(annotation) + + +def _value_for_generic(origin: Any, arguments: Tuple[Any, ...]) -> Any: + """Build the emptiest value a parameterised annotation allows.""" + if origin is typing.Union: + return None if _NONE_TYPE in arguments else _value_for(arguments[0]) + if origin is tuple: + if not arguments or arguments[-1] is Ellipsis: + return () + return tuple(_value_for(entry) for entry in arguments) + if origin in (list, set, frozenset, dict): + return origin() + raise ValueError(f"unmodelled container {origin!r}") + + +def _value_for(annotation: Any) -> Any: + """Build the emptiest value a return annotation allows. + + ``Optional[X]`` yields None, containers yield empty containers, scalars + yield their zero. Raises :class:`ValueError` for an annotation this cannot + model, which leaves the adapter out of the sweep rather than testing it + against a value its callee never promised. + """ + if annotation is inspect.Signature.empty: + raise ValueError("no return annotation") + if annotation in (None, _NONE_TYPE, typing.Any): + return None + origin = typing.get_origin(annotation) + if origin is not None: + return _value_for_generic(origin, typing.get_args(annotation)) + if annotation in _ZEROS: + return _ZEROS[annotation]() + raise ValueError(f"unmodelled return annotation {annotation!r}") + + +# === Finding the callee an adapter stands in front of ======================= + +def _adapter_source(adapter: Any) -> Optional[ast.FunctionDef]: + """Return the parsed definition of ``adapter``, or None if unavailable.""" + try: + source = textwrap.dedent(inspect.getsource(adapter)) + except (OSError, TypeError): + return None + node = ast.parse(source).body[0] + return node if isinstance(node, ast.FunctionDef) else None + + +def _is_stdlib(module_path: str) -> bool: + """Return True for a standard-library module path.""" + return module_path.split(".")[0] in sys.stdlib_module_names + + +def _is_foreign_import(node: ast.stmt) -> bool: + """Return True for an import that is neither stdlib nor first-party-from. + + A plain ``import`` of a third-party package means the adapter picks its own + backend; a relative import means the source could not be resolved to a + module path. Either way the adapter is out of scope. + """ + if isinstance(node, ast.Import): + return any(not _is_stdlib(alias.name) for alias in node.names) + return not node.module or bool(node.level) + + +def _first_party_from_imports(imports: List[ast.stmt] + ) -> List[Tuple[str, List[str]]]: + """Return ``(module, names)`` for each from-import outside the stdlib.""" + return [(child.module, [alias.name for alias in child.names]) + for child in imports + if isinstance(child, ast.ImportFrom) + and not _is_stdlib(child.module)] + + +def _project_import(adapter: Any) -> Optional[Tuple[str, List[str]]]: + """Return ``(module, names)`` for the one project callee an adapter imports. + + Standard-library imports are ignored: ``import json`` inside an adapter + parses a field the visual editor passed as text, and ``import base64`` + encodes what the callee returned -- neither is the callee. + """ + node = _adapter_source(adapter) + if node is None: + return None + imports = [child for child in ast.walk(node) + if isinstance(child, (ast.Import, ast.ImportFrom))] + if any(_is_foreign_import(child) for child in imports): + return None + found = _first_party_from_imports(imports) + return found[0] if len(found) == 1 else None + + +def _sole_imported_name(node: ast.stmt) -> Optional[Tuple[str, str, str]]: + """Return ``(module, attribute, local_name)`` for ``from X import y as z``.""" + if not isinstance(node, ast.ImportFrom) or not node.module or node.level: + return None + if len(node.names) != 1: + return None + alias = node.names[0] + return node.module, alias.name, alias.asname or alias.name + + +def _returned_callee(node: ast.stmt) -> Optional[str]: + """Return the plain name a ``return name(...)`` statement calls.""" + if not isinstance(node, ast.Return): + return None + call = node.value + if not isinstance(call, ast.Call) or not isinstance(call.func, ast.Name): + return None + return call.func.id + + +def _delegated_call(node: ast.FunctionDef) -> Optional[Tuple[str, str]]: + """Return ``(module, attribute)`` for a body that is import-then-return.""" + body = [statement for statement in node.body + if not (isinstance(statement, ast.Expr) + and isinstance(statement.value, ast.Constant))] + if len(body) != 2: + return None + imported = _sole_imported_name(body[0]) + called = _returned_callee(body[1]) + if imported is None or called is None or called != imported[2]: + return None + return imported[0], imported[1] + + +def _delegation(adapter: Any) -> Optional[Tuple[str, str]]: + """Return ``(module, attribute)`` when ``adapter`` is a pure delegator. + + A pure delegator's body is exactly one ``from ... import name`` followed by + ``return name(...)``. Anything else -- a temporary, a branch, a second + import -- means the adapter does work of its own; the output sweeps cover + those instead. + """ + node = _adapter_source(adapter) + return None if node is None else _delegated_call(node) + + +def _contract_stubs(adapter: Any) -> Optional[Tuple[Any, Dict[str, Any]]]: + """Return ``(module, {name: value})`` stubbing one adapter's callees. + + Each value comes from that callee's own return annotation, so the adapter + runs against exactly what the typing contract promises it. + """ + found = _project_import(adapter) + if found is None: + return None + module_path, names = found + try: + module = importlib.import_module(module_path) + except ImportError: + return None + values: Dict[str, Any] = {} + for name in names: + target = getattr(module, name, None) + if not callable(target): + return None + try: + hints = typing.get_type_hints(target) + values[name] = _value_for(hints.get("return", + inspect.Signature.empty)) + except (ValueError, TypeError, NameError): + return None + return module, values + + +def _install_stubs(monkeypatch: pytest.MonkeyPatch, module: Any, + values: Dict[str, Any]) -> None: + """Replace each named callee with a stub returning its contract value.""" + for name, value in values.items(): + monkeypatch.setattr(module, name, lambda *a, _v=value, **k: _v) + + +def _is_serialisable(value: Any) -> bool: + """Return True when a result can cross the JSON boundary.""" + if isinstance(value, MCPContent): + return True + if isinstance(value, list) and value and all( + isinstance(entry, MCPContent) for entry in value): + return True + try: + json.dumps(value) + except (TypeError, ValueError): + return False + return True + + +# === The MCP tool registry ================================================== + +def _tool_arguments(schema: Dict[str, Any], + required_only: bool) -> Dict[str, Any]: + """Build a call payload from a tool's input schema.""" + properties = schema.get("properties") or {} + names = (schema.get("required") or []) if required_only else list(properties) + return {name: _sample_value(properties[name], name) + for name in names if name in properties} + + +def _unique_by_handler(tools: List[MCPTool]) -> List[MCPTool]: + """Drop the short aliases, which are copies pointing at the same handler.""" + seen = set() + unique = [] + for tool in tools: + if tool.handler in seen: + continue + seen.add(tool.handler) + unique.append(tool) + return unique + + +REGISTRY = build_default_tool_registry(read_only=False, aliases=False) +_TOOLS = _unique_by_handler(REGISTRY) +DELEGATING = [(tool, *_delegation(tool.handler)) for tool in _TOOLS + if _delegation(tool.handler) is not None] +STUBBABLE = [tool for tool in _TOOLS + if _contract_stubs(tool.handler) is not None] + + +def _install_recorder(monkeypatch: pytest.MonkeyPatch, module_path: str, + attribute: str) -> Tuple[Dict[str, Any], Any, Callable]: + """Replace ``module_path.attribute`` with a call recorder. + + Returns the record dict, the sentinel the recorder returns, and the real + callable, whose signature says what the recorded arguments are named. + """ + module = importlib.import_module(module_path) + original = getattr(module, attribute) + record: Dict[str, Any] = {} + sentinel = object() + + def _recorder(*args: Any, **kwargs: Any) -> Any: + record["args"] = args + record["kwargs"] = kwargs + return sentinel + + monkeypatch.setattr(module, attribute, _recorder) + return record, sentinel, original + + +@pytest.mark.parametrize("case", DELEGATING, ids=lambda case: case[0].name) +def test_required_arguments_alone_make_the_tool_callable(case, monkeypatch): + """A client sending exactly the schema's required properties succeeds.""" + tool, module_path, attribute = case + record, sentinel, _original = _install_recorder( + monkeypatch, module_path, attribute) + payload = _tool_arguments(tool.input_schema, required_only=True) + assert tool.invoke(payload) is sentinel + assert record, f"{tool.name} never reached {module_path}.{attribute}" + + +@pytest.mark.parametrize("case", DELEGATING, ids=lambda case: case[0].name) +def test_declared_arguments_reach_the_delegate_unchanged(case, monkeypatch): + """Every property the schema declares is accepted and forwarded intact.""" + tool, module_path, attribute = case + record, sentinel, original = _install_recorder( + monkeypatch, module_path, attribute) + payload = _tool_arguments(tool.input_schema, required_only=False) + assert tool.invoke(payload) is sentinel + try: + bound = inspect.signature(original).bind( + *record["args"], **record["kwargs"]).arguments + except TypeError as error: # the real delegate would have rejected it + pytest.fail(f"{tool.name} calls {attribute} wrongly: {error}") + for name, value in payload.items(): + if name in bound: + assert bound[name] == value, ( + f"{tool.name} sent {value!r} but {attribute} bound " + f"{bound[name]!r} to {name}") + else: + # The delegate renames the parameter; the value must still arrive. + assert any(seen == value for seen in bound.values()), ( + f"{tool.name} dropped {name}={value!r} on the way to " + f"{module_path}.{attribute}") + + +@pytest.mark.parametrize("tool", STUBBABLE, ids=lambda tool: tool.name) +def test_tool_result_survives_json(tool, monkeypatch): + """An adapter fed exactly what its callee promises returns JSON.""" + if tool.name in _NEEDS_MORE_THAN_THE_CONTRACT: + pytest.skip("documented: needs more than the callee's annotation") + module, values = _contract_stubs(tool.handler) + _install_stubs(monkeypatch, module, values) + result = tool.invoke(_tool_arguments(tool.input_schema, + required_only=False)) + assert _is_serialisable(result), ( + f"{tool.name} returned {type(result).__name__}, which json.dumps " + "cannot encode") + + +def test_the_documented_exceptions_are_still_needed(monkeypatch): + """A named exception that starts passing is stale and must be removed.""" + still_failing = set() + for tool in STUBBABLE: + if tool.name not in _NEEDS_MORE_THAN_THE_CONTRACT: + continue + module, values = _contract_stubs(tool.handler) + with monkeypatch.context() as patch: + _install_stubs(patch, module, values) + try: + result = tool.invoke( + _tool_arguments(tool.input_schema, required_only=False)) + except Exception: # noqa: BLE001 # reason: any failure keeps it + still_failing.add(tool.name) + else: + if not _is_serialisable(result): + still_failing.add(tool.name) + assert still_failing == _NEEDS_MORE_THAN_THE_CONTRACT, ( + "these entries now pass and should be deleted: " + f"{sorted(_NEEDS_MORE_THAN_THE_CONTRACT - still_failing)}") + + +# === The AC_* dispatch table ================================================ + +SPECS = _build_specs() +SPECS_BY_COMMAND = {spec.command: spec for spec in SPECS} + + +def _mandatory_parameters(adapter: Any) -> Dict[str, Any]: + """Return the parameters a caller must supply, mapped to their annotation.""" + parameters = inspect.signature(adapter).parameters + return {name: parameter.annotation + for name, parameter in parameters.items() + if parameter.default is parameter.empty + and parameter.kind in (parameter.POSITIONAL_OR_KEYWORD, + parameter.KEYWORD_ONLY)} + + +def _command_arguments(command: str, adapter: Any) -> Dict[str, Any]: + """Build the action payload a client would send for one command.""" + spec = SPECS_BY_COMMAND.get(command) + payload = ({field.name: _field_value(field) for field in spec.fields} + if spec is not None else {}) + for name, annotation in _mandatory_parameters(adapter).items(): + if name not in payload: + payload[name] = _annotated_value(annotation, name) + return payload + + +def _is_drivable(command: str, adapter: Any) -> bool: + """Return True when the sweep can supply every argument a command needs. + + Without a schema entry, a mandatory parameter annotated ``Any`` (or not + annotated at all) leaves nothing to build a value from, and passing None + would test the adapter's None-handling rather than its wiring. + """ + if _contract_stubs(adapter) is None: + return False + if any(parameter.kind is parameter.VAR_KEYWORD + for parameter in inspect.signature(adapter).parameters.values()): + return False + if command in SPECS_BY_COMMAND: + return True + return all(annotation not in (inspect.Parameter.empty, typing.Any) + for annotation in _mandatory_parameters(adapter).values()) + + +SWEEPABLE = sorted(command for command, adapter in executor.event_dict.items() + if _is_drivable(command, adapter)) + + +@pytest.mark.parametrize("command", SWEEPABLE) +def test_command_dispatches_and_records_json(command, monkeypatch): + """The command runs from a client's arguments and lands JSON in the record.""" + adapter = executor.event_dict[command] + module, values = _contract_stubs(adapter) + _install_stubs(monkeypatch, module, values) + payload = _command_arguments(command, adapter) + record = executor.execute_action([[command, payload]]) + assert record, f"{command} produced no execution record" + try: + json.dumps(record) + except (TypeError, ValueError) as error: + pytest.fail(f"{command} put an unencodable value in the record: " + f"{error}") + + +def test_every_schema_field_names_a_parameter_of_its_command(): + """The editor cannot offer a field the executor would reject.""" + unknown = [] + for spec in SPECS: + adapter = executor.event_dict.get(spec.command) + if adapter is None: + continue + parameters = inspect.signature(adapter).parameters + if any(parameter.kind is parameter.VAR_KEYWORD + for parameter in parameters.values()): + continue + unknown.extend(f"{spec.command}.{field.name}" for field in spec.fields + if field.name not in parameters) + assert not unknown, f"fields no adapter accepts: {unknown}" + + +def test_every_schema_command_is_in_the_dispatch_table(): + """A command the editor can emit must be a command the executor knows.""" + known = executor.known_commands() + missing = [spec.command for spec in SPECS if spec.command not in known] + assert not missing, f"schema commands the executor cannot run: {missing}" + + +def _unfielded_mandatory(spec: Any, adapter: Any) -> List[str]: + """Return this command's mandatory scalar parameters that have no field.""" + parameters = inspect.signature(adapter).parameters + if any(parameter.kind is parameter.VAR_KEYWORD + for parameter in parameters.values()): + return [] + fielded = {field.name for field in spec.fields} + gaps = [] + for name, annotation in _mandatory_parameters(adapter).items(): + if name in fielded: + continue + structured = (typing.get_origin(annotation) is not None + or annotation is typing.Any) + if not structured: + gaps.append(f"{spec.command}.{name}: {annotation}") + return gaps + + +def test_unfielded_mandatory_parameters_are_all_structured(): + """A mandatory parameter with no field must be one no field could express. + + The editor's field types are all scalars, so list- and dict-shaped + parameters are filled from its raw JSON view instead. A mandatory *scalar* + with no field is different: it means the editor emits an action that raises + ``TypeError`` the first time it runs. + """ + scalar_gaps = [] + for spec in SPECS: + adapter = executor.event_dict.get(spec.command) + if adapter is not None: + scalar_gaps.extend(_unfielded_mandatory(spec, adapter)) + assert not scalar_gaps, ( + f"mandatory scalar parameters the editor never fills: {scalar_gaps}") + + +# === The sweeps have to keep finding things ================================= + +def test_both_registries_are_swept_in_bulk(): + """Guard the discovery itself: a broken matcher would sweep nothing.""" + tools = len(REGISTRY) + assert len(DELEGATING) > tools // 3, ( + f"only {len(DELEGATING)} of {tools} tools matched the pure-delegation " + "shape -- the AST matcher has probably drifted") + assert len(STUBBABLE) > tools // 2, ( + f"only {len(STUBBABLE)} of {tools} tools could be stubbed from their " + "callee's annotations -- the typing contract or the matcher has drifted") + commands = len(executor.event_dict) + assert len(SWEEPABLE) > commands // 5, ( + f"only {len(SWEEPABLE)} of {commands} commands matched the " + "single-project-import shape -- the matcher has probably drifted") From 4ec0c03e02629e87d82f625a3d93267a9b69f5cc Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Sun, 23 Aug 2026 22:22:09 +0800 Subject: [PATCH 02/20] Resolve the swept module path against the package, not just "not stdlib" The sweep called the parsed path first-party when it merely was not in the standard library, so a `from PySide6... import` would have been imported and stubbed like a project callee. Requiring the package prefix makes the filter mean what its name says and makes the dynamic import provably local; today all 280 resolved modules already match, so nothing swept changes. --- .../headless/test_adapter_registry_sweep.py | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/test/unit_test/headless/test_adapter_registry_sweep.py b/test/unit_test/headless/test_adapter_registry_sweep.py index 9f278436..b48cb3f8 100644 --- a/test/unit_test/headless/test_adapter_registry_sweep.py +++ b/test/unit_test/headless/test_adapter_registry_sweep.py @@ -54,6 +54,7 @@ ) _NONE_TYPE = type(None) +_PACKAGE = "je_auto_control" # MCP adapters that need more than their callee's return annotation promises. # Each is a real coupling the type contract does not express, not a stub defect: @@ -207,11 +208,17 @@ def _is_foreign_import(node: ast.stmt) -> bool: def _first_party_from_imports(imports: List[ast.stmt] ) -> List[Tuple[str, List[str]]]: - """Return ``(module, names)`` for each from-import outside the stdlib.""" + """Return ``(module, names)`` for each from-import inside this package. + + "First party" is the package prefix, not merely "not the stdlib": a + ``from PySide6... import`` is a backend choice like a plain third-party + import, and it is the prefix that lets the resolved path be imported below + without trusting whatever the parsed source happened to say. + """ return [(child.module, [alias.name for alias in child.names]) for child in imports if isinstance(child, ast.ImportFrom) - and not _is_stdlib(child.module)] + and child.module.startswith(_PACKAGE + ".")] def _project_import(adapter: Any) -> Optional[Tuple[str, List[str]]]: @@ -289,7 +296,9 @@ def _contract_stubs(adapter: Any) -> Optional[Tuple[Any, Dict[str, Any]]]: return None module_path, names = found try: - module = importlib.import_module(module_path) + # ``module_path`` is a ``je_auto_control.*`` path parsed out of this + # repository's own source; no caller supplies it. + module = importlib.import_module(module_path) # nosemgrep # reason: prefix-checked at the parse site except ImportError: return None values: Dict[str, Any] = {} From 79dc259c5f9d32df02c3f6c1e04223d6a576d173 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Sun, 23 Aug 2026 23:09:21 +0800 Subject: [PATCH 03/20] Measure coverage with the WebRTC extra, so the subsystem is measured at all Eleven modules under `utils/remote_desktop` raise ImportError at module level without aiortc/av, so 2,090 statements were a hard 0% in CI no matter what anyone wrote. The number is not the point: the tests covering the WebRTC host's auth, TLS, resume tokens and file transfer were already written and `importorskip`ped straight past on every square, so they ran on developer machines and nowhere else. Measured on this tree with one variable changed: 513 of those statements are covered by tests that exist today (+1.23 points on Windows/3.14 end to end). The extra resolves on every square in the matrix -- aiortc 1.15.0 and av 17.1.0 have wheels for win_amd64, manylinux x86_64 and macos-14 arm64 on both ends of the supported Python range. Deliberately not added to `typing-stable-api`: that gate must not depend on what is installed. --- .github/workflows/quality.yml | 15 +++++++++++++-- CLAUDE.md | 9 ++++++++- dev_requirements.txt | 6 ++++++ 3 files changed, 27 insertions(+), 3 deletions(-) diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml index 4d0c0814..f03f8051 100644 --- a/.github/workflows/quality.yml +++ b/.github/workflows/quality.yml @@ -120,9 +120,20 @@ jobs: # snapshot of `je_auto_control/` straight into site-packages and masks # the editable install for any sub-package the snapshot doesn't include # (admin, usb, remote_desktop, vision, …). - - name: Install the project itself + # + # `[webrtc]` is part of the install and is load-bearing for what this job + # measures. Eleven modules under `utils/remote_desktop` raise ImportError + # at module level without `aiortc`/`av` — 2,090 statements that were a + # hard 0% here no matter what anyone wrote. Worse than the number: the + # tests that cover the WebRTC host's auth, TLS, tokens and file transfer + # were already written and `importorskip`ped straight past on every + # square, so they ran on developer machines and nowhere else. Measured on + # this tree, one variable changed: 513 of those statements are covered by + # tests that exist today. The extra is NOT added to `typing-stable-api` + # below — that gate must not depend on what is installed. + - name: Install the project itself, with the WebRTC extra shell: bash - run: pip install -e . # NOSONAR githubactions:S8544 githubactions:S8541 # reason: installs the checked-out project itself, so there is no upstream version to lock and no third-party setup script to run + run: pip install -e ".[webrtc]" # NOSONAR githubactions:S8544 githubactions:S8541 # reason: installs the checked-out project itself, so there is no upstream version to lock and no third-party setup script to run - name: Install the test tooling shell: bash diff --git a/CLAUDE.md b/CLAUDE.md index d8e97752..5815281e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -24,7 +24,7 @@ Layering: entry points (`cli.py`, `gui/`, socket / REST / MCP servers) → execu ```bash pip install -r dev_requirements.txt # dev deps -pip install -e .[gui] # + GUI extra +pip install -e .[gui,webrtc] # + GUI and WebRTC extras python -m pytest test/unit_test/headless # headless unit tests python -m pytest test/integrated_test/ # cross-module workflows python -m coverage run -m pytest # the suite WITH coverage (see below) @@ -41,6 +41,13 @@ have their import-time lines recorded as never executed: measured, that is suite). `test/unit_test/headless/test_coverage_measurement.py` holds CI to the correct spelling. +**Measure it with the `[webrtc]` extra installed**, which is why it is in the +line above. Eleven modules under `utils/remote_desktop` raise `ImportError` at +module level without `aiortc`/`av` — 2,090 statements, about 4 points — and the +tests covering the WebRTC host's auth, TLS, tokens and file transfer +`importorskip` straight past. `quality.yml` installs the extra so the floor is +measured against the same tree a developer sees. + `pyproject.toml` pins `python_files = ["test_*.py"]` on purpose: the `*_test.py` files under `test/unit_test/` are manual demo scripts whose module bodies drive the real mouse and keyboard on import. Never loosen that setting. ## Feature Delivery Rules diff --git a/dev_requirements.txt b/dev_requirements.txt index d64208db..0f26e449 100644 --- a/dev_requirements.txt +++ b/dev_requirements.txt @@ -9,6 +9,12 @@ qt-material==2.17 mss==10.2.0 defusedxml==0.7.1 +# WebRTC ([webrtc] extra) — without these, eleven `utils/remote_desktop` +# modules raise ImportError at import and their tests skip, so the coverage +# figure a developer measures is ~4 points under the one CI enforces. +aiortc>=1.14.0 +av>=14.0.0 + # Office I/O ([office] extra) — exercised by the headless Office tests. openpyxl==3.1.5 python-docx==1.2.0 From e2a7a58d2ab98752748c94d72f92cdfe54a94e6b Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Sun, 23 Aug 2026 23:42:14 +0800 Subject: [PATCH 04/20] Sweep a third registry, and reach the callees the contract hid behind Three changes, all the same idea one step further: an adapter can be run against exactly what its callee promises, and the promise is machine-readable. * A dataclass return annotation is not the end of the contract, it is one more level of it. `_value_for` now builds an instance from the dataclass's own fields, so the 92 adapters whose callee returns `HealOutcome`, `PollResult`, `AXTreeNode` and friends run for real -- and their `.to_dict()` runs with them, against the declared shape rather than a mock that answers everything. * Several dozen adapters import a module-level singleton rather than a function -- `default_observer`, `default_scheduler`, `registry` -- and call one method on it. That is the same wiring shape one indirection along, so the callee is the method and its annotation is the contract. Two different methods mean the adapter orchestrates the object, and it stays out. * The REST route table is the third registry of this shape, and the one whose handlers nothing exercised: the existing tests go through the HTTP layer, so on a headless runner they reach a handler only to watch it fall into its own `except`. `test_rest_route_sweep.py` calls all 31 routes with the arguments their own OpenAPI document declares, and asserts what the dispatcher relies on -- (status, dict), a real HTTP code, a payload json.dumps accepts -- for a documented request, an empty one, and one of the wrong shape. The machinery the three sweeps share now lives in `_contract_sweep.py`; each keeps its own argument source, which is what stops a sweep from passing by restating the code it checks. `ac_rrule_next` and its neighbours declare `"format": "date-time"` on the properties they parse, which their descriptions already said in prose. Without it a client generating values from the schema alone gets a ValueError out of `datetime.fromisoformat`, and the sweep got one too. --- CHANGELOG.md | 7 + architecture_explore.md | 22 +- .../utils/mcp_server/tools/_factories.py | 12 +- test/unit_test/headless/_contract_sweep.py | 401 ++++++++++++++++++ .../headless/test_adapter_registry_sweep.py | 286 +------------ .../headless/test_rest_route_sweep.py | 211 +++++++++ 6 files changed, 658 insertions(+), 281 deletions(-) create mode 100644 test/unit_test/headless/_contract_sweep.py create mode 100644 test/unit_test/headless/test_rest_route_sweep.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 18911307..f534c58e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,13 @@ only when documented here with a migration path. ### Added +- **`ac_rrule_next`, `ac_rrule_occurrences` and `ac_format_date` now declare + the string format they parse.** Their `dtstart` / `now` / `value` properties + carry `"format": "date-time"` (or `"date"`) in the tool's input schema, which + the descriptions already said in prose and the schema did not. A client + generating values from the schema alone used to produce a plain string and + get a `ValueError` out of `datetime.fromisoformat`. + - **Windows on arm64 installs.** `opencv-python`, `cryptography` and `je_open_cv` now carry the environment marker `sys_platform != 'win32' or platform_machine != 'ARM64'`, because none of diff --git a/architecture_explore.md b/architecture_explore.md index b3438b8a..ce325a37 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -20,7 +20,7 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | | Python 模組總數（含周邊子專案） | 1,032 | -| 程式碼總行數 | 141,316 | +| 程式碼總行數 | 141,324 | | `je_auto_control/utils/` 子套件數 | 310 | | `AC_*` 動作指令數（`known_commands()` 實測） | 773 | | 套件門面 `__all__` 公開名稱數 | 1,238 | @@ -490,7 +490,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.9 AI / Agent / LLM -> 13 個套件、約 20,643 行。 +> 13 個套件、約 20,645 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -503,21 +503,21 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/cua_action/` | 127 | 標準化 computer-use 動作結構（Anthropic／OpenAI → `AC_*`） | | `utils/llm/` | 357 | 自然語言 → action list 規劃器 + Anthropic／null 後端 | | `utils/mcp_registry/` | 92 | MCP registry `server.json` 資訊清單產生（可被發現） | -| `utils/mcp_server/` | 17,354 | **無頭 MCP 伺服器**（16K LOC，預設註冊 676 個工具＝657 個 `ac_*` + 19 個別名）：stdio + HTTP 傳輸、工具工廠與處理器、資源、prompt、稽核、限流、外掛熱重載 | +| `utils/mcp_server/` | 17,356 | **無頭 MCP 伺服器**（16K LOC，預設註冊 676 個工具＝657 個 `ac_*` + 19 個別名）：stdio + HTTP 傳輸、工具工廠與處理器、資源、prompt、稽核、限流、外掛熱重載 | | `utils/tool_use_schema/` | 180 | 把 `AC_*` 指令匯出成 Claude／OpenAI 的 tool-use schema | | `utils/trajectory_eval/` | 106 | agent 軌跡評估：依評分規準為一次執行打分 | | `utils/vision/` | 448 | VLM 元素定位器（依描述找元素）+ Anthropic／OpenAI／null 後端 | ### 5.4.10 遠端桌面與 USB -> 6 個套件、約 17,903 行。 +> 6 個套件、約 17,909 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | | `utils/admin/` | 328 | 多主機管理主控台：平行輪詢 N 個 AutoControl REST 端點 | | `utils/config_sync/` | 246 | 透過訊令伺服器做跨機器設定同步 | | `utils/device_matrix/` | 138 | 行動裝置矩陣：同一 action list 於多台裝置平行執行 | -| `utils/remote_desktop/` | 11,990 | **遠端桌面子系統**（56 檔／11.7K LOC）：TCP／WebSocket／WebRTC 三條傳輸路徑、主機與檢視端、訊令伺服器、TURN／中繼、多檢視者、錄影、信任清單、TOTP、稽核鏈 | +| `utils/remote_desktop/` | 11,996 | **遠端桌面子系統**（56 檔／11.7K LOC）：TCP／WebSocket／WebRTC 三條傳輸路徑、主機與檢視端、訊令伺服器、TURN／中繼、多檢視者、錄影、信任清單、TOTP、稽核鏈 | | `utils/usb/` | 4,281 | 跨平台 USB 列舉／熱插拔／裝置直通（WinUSB、IOKit、libusb 後端 + ACL + WebRTC DataChannel 通道） | | `utils/usbip/` | 920 | USB/IP 線路協定主機端（協定封包、TCP 伺服器、libusb URB 後端） | @@ -702,7 +702,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `action_schema.py` | 128 | action list 的結構驗證：形狀、參數型別、未知指令拒絕。單一走訪同時支援兩種消費方式：`validate_actions()` 遇到第一個問題就拋、`unknown_command_names()` 收齊全部不認得的名字（REST `/execute` 用它回 400）。 | | `mouse_aliases.py` | 39 | 單鍵點擊別名（`AC_click_left` 等），executor 與 callback executor 共用。 | -#### `utils/mcp_server/`（17,354 行，676 個工具）— 最大子系統 +#### `utils/mcp_server/`（17,356 行，676 個工具）— 最大子系統 | 檔案 | 行數 | 職責 | | --- | ---: | --- | @@ -726,7 +726,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `rate_limit.py` | 48 | 工具呼叫的 token bucket 限流。 | | `__main__.py` | 88 | `je_auto_control_mcp` console script 進入點。 | -#### `utils/remote_desktop/`（11,990 行／56 檔） +#### `utils/remote_desktop/`（11,996 行／56 檔） 三條傳輸路徑並存：**TCP**（JPEG 影格）、**WebSocket**（同協定換傳輸）、**WebRTC**（aiortc 視訊 + DataChannel）。 @@ -751,7 +751,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `turn_config.py` | 234 | coturn 設定產生器。 | | `presence.py` | 221 | 多檢視者的執行緒安全在場註冊表。 | | `jpeg_recorder_encrypted.py` | 223 | AES-GCM 加密版 session 錄影。 | -| `address_book.py` | 209 | 檢視端的主機通訊錄。 | +| `address_book.py` | 215 | 檢視端的主機通訊錄。 | | `audio.py` / `webrtc_audio.py` / `webrtc_mic.py` | 206 / 190 / 152 | 音訊擷取播放、音訊軌、麥克風上行。 | | `webrtc_files.py` | 205 | 專屬 DataChannel 的分塊檔案傳輸。 | | `webrtc_host_auth.py` | 222 | 檢視端認證與核准：token 檢查、信任清單／IP 白名單自動放行、手動接受／拒絕、SAS、逾時關閉。 | @@ -1026,8 +1026,8 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | 層／子系統 | 檔案數 | 行數 | | --- | ---: | ---: | | `gui/` | 90 | 26,699 | -| `utils/mcp_server/` | 21 | 17,354 | -| `utils/remote_desktop/` | 56 | 11,990 | +| `utils/mcp_server/` | 21 | 17,356 | +| `utils/remote_desktop/` | 56 | 11,996 | | `utils/executor/` | 6 | 9,081 | | `utils/usb/` | 17 | 4,281 | | `je_auto_control/`（頂層 3 檔） | 3 | 2,367 | @@ -1046,5 +1046,5 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 727 | | 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 673 | 47,683 | -| **總計** | **1,026** | **141,251** | +| **總計** | **1,026** | **141,259** | diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index ee2bfc39..82de33ca 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -6035,8 +6035,8 @@ def locale_tools() -> List[MCPTool]: description=("Format an ISO (YYYY-MM-DD) date for a locale. 'fmt' " "is short/medium/long/full. Returns {text}."), input_schema=schema( - {"value": {"type": "string"}, "locale": _LOC, - "fmt": {"type": "string"}}, ["value"]), + {"value": {"type": "string", "format": "date"}, + "locale": _LOC, "fmt": {"type": "string"}}, ["value"]), handler=h.format_date, annotations=READ_ONLY, ), @@ -7533,7 +7533,8 @@ def recurrence_tools() -> List[MCPTool]: "'dtstart' into the next 'count' ISO datetimes. " "Returns {occurrences}."), input_schema=schema( - {"rule": {"type": "string"}, "dtstart": {"type": "string"}, + {"rule": {"type": "string"}, + "dtstart": {"type": "string", "format": "date-time"}, "count": {"type": "integer"}}, ["rule", "dtstart"]), handler=h.rrule_occurrences, @@ -7545,8 +7546,9 @@ def recurrence_tools() -> List[MCPTool]: "(ISO; defaults to current time), anchored at ISO " "'dtstart'. Returns {next}."), input_schema=schema( - {"rule": {"type": "string"}, "dtstart": {"type": "string"}, - "now": {"type": "string"}}, + {"rule": {"type": "string"}, + "dtstart": {"type": "string", "format": "date-time"}, + "now": {"type": "string", "format": "date-time"}}, ["rule", "dtstart"]), handler=h.rrule_next, annotations=READ_ONLY, diff --git a/test/unit_test/headless/_contract_sweep.py b/test/unit_test/headless/_contract_sweep.py new file mode 100644 index 00000000..37abf6f9 --- /dev/null +++ b/test/unit_test/headless/_contract_sweep.py @@ -0,0 +1,401 @@ +"""The machinery three sweeps share: what a callee promises, and how to fake it. + +``test_adapter_registry_sweep`` (MCP tools and the ``AC_*`` table) and +``test_rest_route_sweep`` (the REST route table) all face the same problem. +Each registry is a few hundred short functions whose whole job is to take a +client's arguments, call one headless function, and hand back something that +survives ``json.dumps``. Calling one for real needs a mouse, a screen or a +network; not calling it at all leaves the wiring checked by nobody. + +The way out is the same in every case: **replace the callee with a stub built +from its own return annotation**. That became possible when the typing +contract's exemption list was emptied on 2026-08-22 -- before that, 136 modules +had nothing to read. "What is this adapter entitled to assume?" is now a +question a program can answer, so an adapter can be run against exactly what +its callee promises, no more and no less. + +This module holds only that shared half: reading a value out of a declared +type, finding the callee an adapter stands in front of, and installing the +stub. Each sweep keeps its own argument source -- a JSON schema, the Script +Builder's field specs, an HTTP request context -- because that is what stops a +sweep from passing by restating the code it checks. + +Nothing here is a test; the file is named so pytest does not collect it. +""" +import ast +import dataclasses +import importlib +import inspect +import json +import sys +import textwrap +import typing +from typing import Any, Callable, Dict, FrozenSet, List, Optional, Tuple + +import pytest + +_NONE_TYPE = type(None) +_PACKAGE = "je_auto_control" + + +# === Building a value from a JSON Schema node ================================ + +_SCALARS: Dict[str, Any] = { + "integer": 3, "number": 1.5, "boolean": True, + "array": [], "object": {}, "null": None, +} + +# The emptiest value each concrete annotation allows, as factories so two +# adapters never share one mutable container. + + +# A declared `format` narrows what "string" means, and an adapter that parses +# before delegating enforces it. Without this the sweep hands `ac_rrule_next` +# the generic sample and watches `datetime.fromisoformat` reject it -- which +# says nothing about the adapter and everything about the sample. +_STRING_FORMATS: Dict[str, str] = { + "date-time": "2026-01-02T03:04:05", + "date": "2026-01-02", + "time": "03:04:05", +} + + +def _sample_value(spec: Dict[str, Any], name: str) -> Any: + """Return a value satisfying one JSON-Schema property node.""" + enum = spec.get("enum") + if isinstance(enum, list) and enum: + return enum[0] + kind = spec.get("type", "string") + if isinstance(kind, list): + kind = next((entry for entry in kind if entry != "null"), "string") + if kind == "array": + item = spec.get("items") + return [_sample_value(item, name)] if item else [] + if kind == "string": + return _STRING_FORMATS.get(spec.get("format"), f"value-for-{name}") + return _SCALARS.get(kind, f"value-for-{name}") + + +# === Reading a value out of a declared type ================================= + +_ZEROS: Dict[Any, Callable[[], Any]] = { + int: lambda: 0, float: lambda: 0.0, bool: lambda: False, + str: lambda: "", bytes: lambda: b"", + list: list, dict: dict, set: set, tuple: tuple, +} + + + +def _value_for_generic(origin: Any, arguments: Tuple[Any, ...], + seen: FrozenSet[type] = frozenset()) -> Any: + """Build the emptiest value a parameterised annotation allows.""" + if origin is typing.Union: + return None if _NONE_TYPE in arguments else _value_for(arguments[0], + seen) + if origin is tuple: + if not arguments or arguments[-1] is Ellipsis: + return () + return tuple(_value_for(entry, seen) for entry in arguments) + if origin in (list, set, frozenset, dict): + return origin() + raise ValueError(f"unmodelled container {origin!r}") + + +def _dataclass_value(cls: type, seen: FrozenSet[type]) -> Any: + """Build an instance of ``cls`` with every field at its emptiest value. + + A dataclass return annotation is not the end of the contract, it is one + more level of it: each field carries an annotation of its own, so "what + may the caller assume?" stays a question the program can answer. The + instance is real -- the adapter's ``.to_dict()``, its attribute reads and + its comparisons all run against the declared shape rather than a mock that + answers everything. + + ``seen`` breaks a cycle of dataclasses that reach each other by a bare + field; a self-reference through ``Optional`` or a container terminates on + its own, at None and at the empty container. + """ + if cls in seen: + raise ValueError(f"self-referential dataclass field on {cls!r}") + hints = typing.get_type_hints(cls) + arguments = {field.name: _value_for(hints.get(field.name, field.type), + seen | {cls}) + for field in dataclasses.fields(cls) if field.init} + return cls(**arguments) + + +def _value_for(annotation: Any, seen: FrozenSet[type] = frozenset()) -> Any: + """Build the emptiest value a return annotation allows. + + ``Optional[X]`` yields None, containers yield empty containers, scalars + yield their zero, and a dataclass yields an instance built the same way + from its own fields. Raises :class:`ValueError` for an annotation this + cannot model, which leaves the adapter out of the sweep rather than + testing it against a value its callee never promised. + """ + if annotation is inspect.Signature.empty: + raise ValueError("no return annotation") + if annotation in (None, _NONE_TYPE, typing.Any): + return None + origin = typing.get_origin(annotation) + if origin is not None: + return _value_for_generic(origin, typing.get_args(annotation), seen) + if annotation in _ZEROS: + return _ZEROS[annotation]() + if isinstance(annotation, type) and dataclasses.is_dataclass(annotation): + return _dataclass_value(annotation, seen) + raise ValueError(f"unmodelled return annotation {annotation!r}") + + +# === Finding the callee an adapter stands in front of ======================= + +def _adapter_source(adapter: Any) -> Optional[ast.FunctionDef]: + """Return the parsed definition of ``adapter``, or None if unavailable.""" + try: + source = textwrap.dedent(inspect.getsource(adapter)) + except (OSError, TypeError): + return None + node = ast.parse(source).body[0] + return node if isinstance(node, ast.FunctionDef) else None + + +def _is_stdlib(module_path: str) -> bool: + """Return True for a standard-library module path.""" + return module_path.split(".")[0] in sys.stdlib_module_names + + +def _is_foreign_import(node: ast.stmt) -> bool: + """Return True for an import that is neither stdlib nor first-party-from. + + A plain ``import`` of a third-party package means the adapter picks its own + backend; a relative import means the source could not be resolved to a + module path. Either way the adapter is out of scope. + """ + if isinstance(node, ast.Import): + return any(not _is_stdlib(alias.name) for alias in node.names) + return not node.module or bool(node.level) + + +def _first_party_from_imports(imports: List[ast.stmt] + ) -> List[Tuple[str, List[str]]]: + """Return ``(module, names)`` for each from-import inside this package. + + "First party" is the package prefix, not merely "not the stdlib": a + ``from PySide6... import`` is a backend choice like a plain third-party + import, and it is the prefix that lets the resolved path be imported below + without trusting whatever the parsed source happened to say. + """ + return [(child.module, [alias.name for alias in child.names]) + for child in imports + if isinstance(child, ast.ImportFrom) + and child.module.startswith(_PACKAGE + ".")] + + +def _project_import(adapter: Any) -> Optional[Tuple[str, List[str]]]: + """Return ``(module, names)`` for the one project callee an adapter imports. + + Standard-library imports are ignored: ``import json`` inside an adapter + parses a field the visual editor passed as text, and ``import base64`` + encodes what the callee returned -- neither is the callee. + """ + node = _adapter_source(adapter) + if node is None: + return None + imports = [child for child in ast.walk(node) + if isinstance(child, (ast.Import, ast.ImportFrom))] + if any(_is_foreign_import(child) for child in imports): + return None + found = _first_party_from_imports(imports) + return found[0] if len(found) == 1 else None + + +def _sole_imported_name(node: ast.stmt) -> Optional[Tuple[str, str, str]]: + """Return ``(module, attribute, local_name)`` for ``from X import y as z``.""" + if not isinstance(node, ast.ImportFrom) or not node.module or node.level: + return None + if len(node.names) != 1: + return None + alias = node.names[0] + return node.module, alias.name, alias.asname or alias.name + + +def _returned_callee(node: ast.stmt) -> Optional[str]: + """Return the plain name a ``return name(...)`` statement calls.""" + if not isinstance(node, ast.Return): + return None + call = node.value + if not isinstance(call, ast.Call) or not isinstance(call.func, ast.Name): + return None + return call.func.id + + +def _delegated_call(node: ast.FunctionDef) -> Optional[Tuple[str, str]]: + """Return ``(module, attribute)`` for a body that is import-then-return.""" + body = [statement for statement in node.body + if not (isinstance(statement, ast.Expr) + and isinstance(statement.value, ast.Constant))] + if len(body) != 2: + return None + imported = _sole_imported_name(body[0]) + called = _returned_callee(body[1]) + if imported is None or called is None or called != imported[2]: + return None + return imported[0], imported[1] + + +def _delegation(adapter: Any) -> Optional[Tuple[str, str]]: + """Return ``(module, attribute)`` when ``adapter`` is a pure delegator. + + A pure delegator's body is exactly one ``from ... import name`` followed by + ``return name(...)``. Anything else -- a temporary, a branch, a second + import -- means the adapter does work of its own; the output sweeps cover + those instead. + """ + node = _adapter_source(adapter) + return None if node is None else _delegated_call(node) + + +def _sole_attribute_on(node: ast.FunctionDef, name: str) -> Optional[str]: + """Return the one attribute an adapter reaches for on the local ``name``. + + Several dozen adapters import a module-level singleton rather than a + function -- ``default_observer``, ``default_scheduler``, ``registry`` -- + and call one method on it. That is the same wiring shape one indirection + along, so the callee is the method and its annotation is the contract. + Two different methods mean the adapter orchestrates the object instead of + standing in front of one call, and it stays out of the sweep. + """ + attributes = {child.attr for child in ast.walk(node) + if isinstance(child, ast.Attribute) + and isinstance(child.value, ast.Name) + and child.value.id == name} + return attributes.pop() if len(attributes) == 1 else None + + +def _submodule(module: Any, name: str) -> Any: + """Return ``module.name`` when the imported name is a submodule. + + ``from ...usb.passthrough import commands`` binds a module, and a package + only grows that attribute once something has imported it -- which, for a + lazily-imported adapter, may not have happened yet. + """ + # Both halves are ``je_auto_control.*`` paths parsed out of this + # repository's own source; no caller supplies either. + path = f"{module.__name__}.{name}" + try: + return importlib.import_module(path) # nosemgrep # reason: prefix-checked at the parse site + except ImportError: + return None + + +def _callee_of(adapter: Any, module: Any, name: str + ) -> Optional[Tuple[Any, str]]: + """Return the ``(owner, attribute)`` pair an adapter actually calls.""" + target = getattr(module, name, None) + if callable(target): + return module, name + if target is None: + target = _submodule(module, name) + if target is None: + return None + node = _adapter_source(adapter) + attribute = None if node is None else _sole_attribute_on(node, name) + if attribute is None or not callable(getattr(target, attribute, None)): + return None + return target, attribute + + +def _contract_stubs(adapter: Any) -> Optional[List[Tuple[Any, str, Any]]]: + """Return ``(owner, attribute, value)`` for each of an adapter's callees. + + Each value comes from that callee's own return annotation, so the adapter + runs against exactly what the typing contract promises it. The owner is + the callee's module for a plain function and the singleton itself for a + method on one. + """ + found = _project_import(adapter) + if found is None: + return None + module_path, names = found + try: + # ``module_path`` is a ``je_auto_control.*`` path parsed out of this + # repository's own source; no caller supplies it. + module = importlib.import_module(module_path) # nosemgrep # reason: prefix-checked at the parse site + except ImportError: + return None + stubs: List[Tuple[Any, str, Any]] = [] + for name in names: + callee = _callee_of(adapter, module, name) + if callee is None: + return None + owner, attribute = callee + try: + hints = typing.get_type_hints(getattr(owner, attribute)) + stubs.append((owner, attribute, + _value_for(hints.get("return", + inspect.Signature.empty)))) + except (ValueError, TypeError, NameError): + return None + return stubs + + +def _install_stubs(monkeypatch: pytest.MonkeyPatch, + stubs: List[Tuple[Any, str, Any]]) -> None: + """Replace each named callee with a stub returning its contract value.""" + for owner, attribute, value in stubs: + monkeypatch.setattr(owner, attribute, lambda *a, _v=value, **k: _v) + + +def _is_serialisable(value: Any, native: Tuple[type, ...] = ()) -> bool: + """Return True when a result can cross the JSON boundary. + + ``native`` names the types a registry encodes itself -- the MCP registry + turns its own content objects into JSON on the way out, so one of those, + or a non-empty list of them, is already at the boundary. + """ + if native and isinstance(value, native): + return True + if native and isinstance(value, list) and value and all( + isinstance(entry, native) for entry in value): + return True + try: + json.dumps(value) + except (TypeError, ValueError): + return False + return True + + +def _install_recorder(monkeypatch: pytest.MonkeyPatch, module_path: str, + attribute: str) -> Tuple[Dict[str, Any], Any, Callable]: + """Replace ``module_path.attribute`` with a call recorder. + + Returns the record dict, the sentinel the recorder returns, and the real + callable, whose signature says what the recorded arguments are named. + """ + module = importlib.import_module(module_path) + original = getattr(module, attribute) + record: Dict[str, Any] = {} + sentinel = object() + + def _recorder(*args: Any, **kwargs: Any) -> Any: + record["args"] = args + record["kwargs"] = kwargs + return sentinel + + monkeypatch.setattr(module, attribute, _recorder) + return record, sentinel, original + + +# Public names for the sweeps; the underscored ones above stay private to the +# implementation so a future change here does not read as a change there. +value_for = _value_for +adapter_source = _adapter_source +project_import = _project_import +delegation = _delegation +contract_stubs = _contract_stubs +install_stubs = _install_stubs +sample_value = _sample_value +NONE_TYPE = _NONE_TYPE +install_recorder = _install_recorder +is_serialisable = _is_serialisable +is_stdlib = _is_stdlib diff --git a/test/unit_test/headless/test_adapter_registry_sweep.py b/test/unit_test/headless/test_adapter_registry_sweep.py index b48cb3f8..091a150c 100644 --- a/test/unit_test/headless/test_adapter_registry_sweep.py +++ b/test/unit_test/headless/test_adapter_registry_sweep.py @@ -34,17 +34,18 @@ third-party module (it picks its own backend, so what it returns depends on the machine -- the opposite of what a sweep can assert). """ -import ast -import importlib import inspect import json -import sys -import textwrap import typing -from typing import Any, Callable, Dict, List, Optional, Tuple +from typing import Any, Callable, Dict, List import pytest +from headless._contract_sweep import ( + NONE_TYPE, contract_stubs, delegation, install_recorder, install_stubs, + is_serialisable, sample_value, +) + from je_auto_control.gui.script_builder.command_schema import ( FieldSpec, FieldType, _build_specs, ) @@ -53,8 +54,6 @@ MCPContent, MCPTool, build_default_tool_registry, ) -_NONE_TYPE = type(None) -_PACKAGE = "je_auto_control" # MCP adapters that need more than their callee's return annotation promises. # Each is a real coupling the type contract does not express, not a stub defect: @@ -65,6 +64,7 @@ "ac_anchor_click", # Indexes a key out of a Dict[str, Any] the annotation cannot promise. "ac_tween_drag", + "ac_voice_dispatch", } @@ -73,11 +73,6 @@ # Sample values by JSON-Schema type. Strings carry their property name so a # swapped pair of same-typed arguments shows as a mismatch, not as two equal # placeholders. -_SCALARS: Dict[str, Any] = { - "integer": 3, "number": 1.5, "boolean": True, - "array": [], "object": {}, "null": None, -} - # One sample per Script Builder field type; a field's own default wins where it # has one, so enums and paths stay inside the values the editor would offer. _FIELD_SAMPLES: Dict[FieldType, Callable[[FieldSpec], Any]] = { @@ -90,30 +85,6 @@ FieldType.RGB: lambda field: [1, 2, 3], } -# The emptiest value each concrete annotation allows, as factories so two -# adapters never share one mutable container. -_ZEROS: Dict[Any, Callable[[], Any]] = { - int: lambda: 0, float: lambda: 0.0, bool: lambda: False, - str: lambda: "", bytes: lambda: b"", - list: list, dict: dict, set: set, tuple: tuple, -} - - -def _sample_value(spec: Dict[str, Any], name: str) -> Any: - """Return a value satisfying one JSON-Schema property node.""" - enum = spec.get("enum") - if isinstance(enum, list) and enum: - return enum[0] - kind = spec.get("type", "string") - if isinstance(kind, list): - kind = next((entry for entry in kind if entry != "null"), "string") - if kind == "array": - item = spec.get("items") - return [_sample_value(item, name)] if item else [] - if kind == "string": - return f"value-for-{name}" - return _SCALARS.get(kind, f"value-for-{name}") - def _field_value(field: FieldSpec) -> Any: """Return a value the visual editor could have produced for one field.""" @@ -133,7 +104,7 @@ def _annotated_value(annotation: Any, name: str) -> Any: origin = typing.get_origin(annotation) arguments = typing.get_args(annotation) if origin is typing.Union: - if _NONE_TYPE in arguments: + if NONE_TYPE in arguments: return None return _annotated_value(arguments[0], name) if origin in (list, tuple, set): @@ -144,198 +115,6 @@ def _annotated_value(annotation: Any, name: str) -> Any: return scalars.get(annotation) -def _value_for_generic(origin: Any, arguments: Tuple[Any, ...]) -> Any: - """Build the emptiest value a parameterised annotation allows.""" - if origin is typing.Union: - return None if _NONE_TYPE in arguments else _value_for(arguments[0]) - if origin is tuple: - if not arguments or arguments[-1] is Ellipsis: - return () - return tuple(_value_for(entry) for entry in arguments) - if origin in (list, set, frozenset, dict): - return origin() - raise ValueError(f"unmodelled container {origin!r}") - - -def _value_for(annotation: Any) -> Any: - """Build the emptiest value a return annotation allows. - - ``Optional[X]`` yields None, containers yield empty containers, scalars - yield their zero. Raises :class:`ValueError` for an annotation this cannot - model, which leaves the adapter out of the sweep rather than testing it - against a value its callee never promised. - """ - if annotation is inspect.Signature.empty: - raise ValueError("no return annotation") - if annotation in (None, _NONE_TYPE, typing.Any): - return None - origin = typing.get_origin(annotation) - if origin is not None: - return _value_for_generic(origin, typing.get_args(annotation)) - if annotation in _ZEROS: - return _ZEROS[annotation]() - raise ValueError(f"unmodelled return annotation {annotation!r}") - - -# === Finding the callee an adapter stands in front of ======================= - -def _adapter_source(adapter: Any) -> Optional[ast.FunctionDef]: - """Return the parsed definition of ``adapter``, or None if unavailable.""" - try: - source = textwrap.dedent(inspect.getsource(adapter)) - except (OSError, TypeError): - return None - node = ast.parse(source).body[0] - return node if isinstance(node, ast.FunctionDef) else None - - -def _is_stdlib(module_path: str) -> bool: - """Return True for a standard-library module path.""" - return module_path.split(".")[0] in sys.stdlib_module_names - - -def _is_foreign_import(node: ast.stmt) -> bool: - """Return True for an import that is neither stdlib nor first-party-from. - - A plain ``import`` of a third-party package means the adapter picks its own - backend; a relative import means the source could not be resolved to a - module path. Either way the adapter is out of scope. - """ - if isinstance(node, ast.Import): - return any(not _is_stdlib(alias.name) for alias in node.names) - return not node.module or bool(node.level) - - -def _first_party_from_imports(imports: List[ast.stmt] - ) -> List[Tuple[str, List[str]]]: - """Return ``(module, names)`` for each from-import inside this package. - - "First party" is the package prefix, not merely "not the stdlib": a - ``from PySide6... import`` is a backend choice like a plain third-party - import, and it is the prefix that lets the resolved path be imported below - without trusting whatever the parsed source happened to say. - """ - return [(child.module, [alias.name for alias in child.names]) - for child in imports - if isinstance(child, ast.ImportFrom) - and child.module.startswith(_PACKAGE + ".")] - - -def _project_import(adapter: Any) -> Optional[Tuple[str, List[str]]]: - """Return ``(module, names)`` for the one project callee an adapter imports. - - Standard-library imports are ignored: ``import json`` inside an adapter - parses a field the visual editor passed as text, and ``import base64`` - encodes what the callee returned -- neither is the callee. - """ - node = _adapter_source(adapter) - if node is None: - return None - imports = [child for child in ast.walk(node) - if isinstance(child, (ast.Import, ast.ImportFrom))] - if any(_is_foreign_import(child) for child in imports): - return None - found = _first_party_from_imports(imports) - return found[0] if len(found) == 1 else None - - -def _sole_imported_name(node: ast.stmt) -> Optional[Tuple[str, str, str]]: - """Return ``(module, attribute, local_name)`` for ``from X import y as z``.""" - if not isinstance(node, ast.ImportFrom) or not node.module or node.level: - return None - if len(node.names) != 1: - return None - alias = node.names[0] - return node.module, alias.name, alias.asname or alias.name - - -def _returned_callee(node: ast.stmt) -> Optional[str]: - """Return the plain name a ``return name(...)`` statement calls.""" - if not isinstance(node, ast.Return): - return None - call = node.value - if not isinstance(call, ast.Call) or not isinstance(call.func, ast.Name): - return None - return call.func.id - - -def _delegated_call(node: ast.FunctionDef) -> Optional[Tuple[str, str]]: - """Return ``(module, attribute)`` for a body that is import-then-return.""" - body = [statement for statement in node.body - if not (isinstance(statement, ast.Expr) - and isinstance(statement.value, ast.Constant))] - if len(body) != 2: - return None - imported = _sole_imported_name(body[0]) - called = _returned_callee(body[1]) - if imported is None or called is None or called != imported[2]: - return None - return imported[0], imported[1] - - -def _delegation(adapter: Any) -> Optional[Tuple[str, str]]: - """Return ``(module, attribute)`` when ``adapter`` is a pure delegator. - - A pure delegator's body is exactly one ``from ... import name`` followed by - ``return name(...)``. Anything else -- a temporary, a branch, a second - import -- means the adapter does work of its own; the output sweeps cover - those instead. - """ - node = _adapter_source(adapter) - return None if node is None else _delegated_call(node) - - -def _contract_stubs(adapter: Any) -> Optional[Tuple[Any, Dict[str, Any]]]: - """Return ``(module, {name: value})`` stubbing one adapter's callees. - - Each value comes from that callee's own return annotation, so the adapter - runs against exactly what the typing contract promises it. - """ - found = _project_import(adapter) - if found is None: - return None - module_path, names = found - try: - # ``module_path`` is a ``je_auto_control.*`` path parsed out of this - # repository's own source; no caller supplies it. - module = importlib.import_module(module_path) # nosemgrep # reason: prefix-checked at the parse site - except ImportError: - return None - values: Dict[str, Any] = {} - for name in names: - target = getattr(module, name, None) - if not callable(target): - return None - try: - hints = typing.get_type_hints(target) - values[name] = _value_for(hints.get("return", - inspect.Signature.empty)) - except (ValueError, TypeError, NameError): - return None - return module, values - - -def _install_stubs(monkeypatch: pytest.MonkeyPatch, module: Any, - values: Dict[str, Any]) -> None: - """Replace each named callee with a stub returning its contract value.""" - for name, value in values.items(): - monkeypatch.setattr(module, name, lambda *a, _v=value, **k: _v) - - -def _is_serialisable(value: Any) -> bool: - """Return True when a result can cross the JSON boundary.""" - if isinstance(value, MCPContent): - return True - if isinstance(value, list) and value and all( - isinstance(entry, MCPContent) for entry in value): - return True - try: - json.dumps(value) - except (TypeError, ValueError): - return False - return True - - # === The MCP tool registry ================================================== def _tool_arguments(schema: Dict[str, Any], @@ -343,7 +122,7 @@ def _tool_arguments(schema: Dict[str, Any], """Build a call payload from a tool's input schema.""" properties = schema.get("properties") or {} names = (schema.get("required") or []) if required_only else list(properties) - return {name: _sample_value(properties[name], name) + return {name: sample_value(properties[name], name) for name in names if name in properties} @@ -361,38 +140,17 @@ def _unique_by_handler(tools: List[MCPTool]) -> List[MCPTool]: REGISTRY = build_default_tool_registry(read_only=False, aliases=False) _TOOLS = _unique_by_handler(REGISTRY) -DELEGATING = [(tool, *_delegation(tool.handler)) for tool in _TOOLS - if _delegation(tool.handler) is not None] +DELEGATING = [(tool, *delegation(tool.handler)) for tool in _TOOLS + if delegation(tool.handler) is not None] STUBBABLE = [tool for tool in _TOOLS - if _contract_stubs(tool.handler) is not None] - - -def _install_recorder(monkeypatch: pytest.MonkeyPatch, module_path: str, - attribute: str) -> Tuple[Dict[str, Any], Any, Callable]: - """Replace ``module_path.attribute`` with a call recorder. - - Returns the record dict, the sentinel the recorder returns, and the real - callable, whose signature says what the recorded arguments are named. - """ - module = importlib.import_module(module_path) - original = getattr(module, attribute) - record: Dict[str, Any] = {} - sentinel = object() - - def _recorder(*args: Any, **kwargs: Any) -> Any: - record["args"] = args - record["kwargs"] = kwargs - return sentinel - - monkeypatch.setattr(module, attribute, _recorder) - return record, sentinel, original + if contract_stubs(tool.handler) is not None] @pytest.mark.parametrize("case", DELEGATING, ids=lambda case: case[0].name) def test_required_arguments_alone_make_the_tool_callable(case, monkeypatch): """A client sending exactly the schema's required properties succeeds.""" tool, module_path, attribute = case - record, sentinel, _original = _install_recorder( + record, sentinel, _original = install_recorder( monkeypatch, module_path, attribute) payload = _tool_arguments(tool.input_schema, required_only=True) assert tool.invoke(payload) is sentinel @@ -403,7 +161,7 @@ def test_required_arguments_alone_make_the_tool_callable(case, monkeypatch): def test_declared_arguments_reach_the_delegate_unchanged(case, monkeypatch): """Every property the schema declares is accepted and forwarded intact.""" tool, module_path, attribute = case - record, sentinel, original = _install_recorder( + record, sentinel, original = install_recorder( monkeypatch, module_path, attribute) payload = _tool_arguments(tool.input_schema, required_only=False) assert tool.invoke(payload) is sentinel @@ -429,11 +187,10 @@ def test_tool_result_survives_json(tool, monkeypatch): """An adapter fed exactly what its callee promises returns JSON.""" if tool.name in _NEEDS_MORE_THAN_THE_CONTRACT: pytest.skip("documented: needs more than the callee's annotation") - module, values = _contract_stubs(tool.handler) - _install_stubs(monkeypatch, module, values) + install_stubs(monkeypatch, contract_stubs(tool.handler)) result = tool.invoke(_tool_arguments(tool.input_schema, required_only=False)) - assert _is_serialisable(result), ( + assert is_serialisable(result, (MCPContent,)), ( f"{tool.name} returned {type(result).__name__}, which json.dumps " "cannot encode") @@ -444,16 +201,16 @@ def test_the_documented_exceptions_are_still_needed(monkeypatch): for tool in STUBBABLE: if tool.name not in _NEEDS_MORE_THAN_THE_CONTRACT: continue - module, values = _contract_stubs(tool.handler) + stubs = contract_stubs(tool.handler) with monkeypatch.context() as patch: - _install_stubs(patch, module, values) + install_stubs(patch, stubs) try: result = tool.invoke( _tool_arguments(tool.input_schema, required_only=False)) except Exception: # noqa: BLE001 # reason: any failure keeps it still_failing.add(tool.name) else: - if not _is_serialisable(result): + if not is_serialisable(result, (MCPContent,)): still_failing.add(tool.name) assert still_failing == _NEEDS_MORE_THAN_THE_CONTRACT, ( "these entries now pass and should be deleted: " @@ -494,7 +251,7 @@ def _is_drivable(command: str, adapter: Any) -> bool: annotated at all) leaves nothing to build a value from, and passing None would test the adapter's None-handling rather than its wiring. """ - if _contract_stubs(adapter) is None: + if contract_stubs(adapter) is None: return False if any(parameter.kind is parameter.VAR_KEYWORD for parameter in inspect.signature(adapter).parameters.values()): @@ -513,8 +270,7 @@ def _is_drivable(command: str, adapter: Any) -> bool: def test_command_dispatches_and_records_json(command, monkeypatch): """The command runs from a client's arguments and lands JSON in the record.""" adapter = executor.event_dict[command] - module, values = _contract_stubs(adapter) - _install_stubs(monkeypatch, module, values) + install_stubs(monkeypatch, contract_stubs(adapter)) payload = _command_arguments(command, adapter) record = executor.execute_action([[command, payload]]) assert record, f"{command} produced no execution record" diff --git a/test/unit_test/headless/test_rest_route_sweep.py b/test/unit_test/headless/test_rest_route_sweep.py new file mode 100644 index 00000000..3af0361c --- /dev/null +++ b/test/unit_test/headless/test_rest_route_sweep.py @@ -0,0 +1,211 @@ +"""Every REST route, called the way its own OpenAPI document says to call it. + +`rest_handlers` is the third registry of the same shape as the MCP tool table +and the `AC_*` dispatch table: thirty-odd functions that take a decoded request, +call one headless function, and return `(status, payload)` for the dispatcher to +write out as JSON. Its module docstring says the handlers are "pure ... trivial +to unit-test without an HTTP layer", and the existing REST tests go through the +HTTP layer instead, so on a headless runner most of them reach a handler only to +watch it fall into its own `except` and answer 500. The happy path -- the branch +a real client actually gets -- was checked by nobody. + +The arguments come from `rest_openapi.build_openapi_spec()`, which is built from +`_ENDPOINT_METADATA` in a different module from the handlers. That matters for +the same reason the MCP sweep reads `_factories.py`: a sweep that took its +arguments from the handler it is testing could not fail. Here it also buys a +contract test for free -- a route the document does not describe, or a documented +route nothing serves, is a defect either way, and `test_the_route_table_and_the_ +document_agree` says so by name. + +Two things are asserted about every route: + +* **It answers.** No route may raise: the dispatcher writes the handler's return + value straight into the socket, so an exception is a dropped HTTP response and + a client that hangs until it times out. This holds for a well-formed request + and for the empty one an unhelpful client sends. +* **The answer is JSON with an HTTP status.** `(int, dict)`, a status in the + 100-599 range, and a payload `json.dumps` accepts. + +The callee is replaced by a stub grown from its own return annotation, exactly +as in the other two sweeps -- see `_contract_sweep`. +""" +import json +from typing import Any, Dict, List, Optional, Tuple +from urllib.parse import urlencode + +import pytest + +from headless._contract_sweep import ( + contract_stubs, install_stubs, is_serialisable, sample_value, +) +from je_auto_control.utils.rest_api import rest_server +from je_auto_control.utils.rest_api.rest_handlers import RouteContext +from je_auto_control.utils.rest_api.rest_openapi import build_openapi_spec + +_SPEC = build_openapi_spec() +_JSON = "application/json" + +# Served by the dispatcher itself rather than from the two route tables, +# because none of them answers with JSON: Prometheus text, the dashboard HTML +# and the Swagger UI page. They are in the document because a client still has +# to be told they exist. +_NOT_JSON_ROUTES = {("GET", "/dashboard"), ("GET", "/docs"), ("GET", "/metrics")} + + +def _routes() -> List[Tuple[str, str, Any]]: + """Every `(method, path, handler)` the dispatcher can reach.""" + return ([("GET", path, handler) + for path, handler in rest_server._GET_ROUTES.items()] + + [("POST", path, handler) + for path, handler in rest_server._POST_ROUTES.items()]) + + +ROUTES = sorted(_routes(), key=lambda route: (route[0], route[1])) +_IDS = [f"{method} {path}" for method, path, _ in ROUTES] + + +def _operation(method: str, path: str) -> Dict[str, Any]: + return _SPEC["paths"].get(path, {}).get(method.lower(), {}) + + +def _query_for(operation: Dict[str, Any]) -> str: + """Build a query string from the operation's declared parameters.""" + pairs = [(parameter["name"], + sample_value(parameter.get("schema") or {}, parameter["name"])) + for parameter in operation.get("parameters", []) + if parameter.get("in") == "query"] + return urlencode([(name, str(value)) for name, value in pairs]) + + +def _body_for(operation: Dict[str, Any]) -> Optional[Any]: + """Build a request body from the operation's declared requestBody.""" + content = (operation.get("requestBody") or {}).get("content") or {} + schema = (content.get(_JSON) or {}).get("schema") + if not schema: + return None + return {name: sample_value(spec, name) + for name, spec in (schema.get("properties") or {}).items()} + + +def _assert_answers_json(result: Any, label: str) -> None: + """A handler's return value is what the dispatcher writes to the socket.""" + assert isinstance(result, tuple) and len(result) == 2, ( + f"{label} returned {result!r}, not (status, payload)") + status, payload = result + assert isinstance(status, int) and 100 <= status <= 599, ( + f"{label} returned {status!r}, which is not an HTTP status") + assert isinstance(payload, dict), ( + f"{label} returned a {type(payload).__name__} payload; the dispatcher " + "writes a JSON object") + assert is_serialisable(payload), ( + f"{label} returned a payload json.dumps cannot encode") + + +# === The document and the table have to describe the same API =============== + +def test_the_route_table_and_the_document_agree(): + """A route nobody documents, or a documented route nobody serves.""" + documented = {(method.upper(), path) + for path, item in _SPEC["paths"].items() for method in item} + routed = {(method, path) for method, path, _ in ROUTES} + assert routed - documented == set(), ( + "routes the OpenAPI document does not describe: " + f"{sorted(routed - documented)}") + assert documented - routed == _NOT_JSON_ROUTES, ( + "documented routes nothing serves, or a non-JSON route that grew a " + f"JSON handler: {sorted(documented - routed - _NOT_JSON_ROUTES)}") + + +def test_every_route_is_reachable_from_the_dispatcher(): + """The two tables are what `handle_request` looks in; neither may be empty.""" + assert len(rest_server._GET_ROUTES) > 10 + assert len(rest_server._POST_ROUTES) > 5 + assert not set(rest_server._GET_ROUTES) & set(rest_server._POST_ROUTES), ( + "a path in both tables would resolve by method alone; nothing here " + "expects that today") + + +# === Every route answers, twice ============================================== + +@pytest.mark.parametrize("method, path, handler", ROUTES, ids=_IDS) +def test_a_documented_request_gets_a_json_answer(method, path, handler, + monkeypatch): + """Called as the document says, every route answers in JSON.""" + operation = _operation(method, path) + stubs = contract_stubs(handler) + if stubs is not None: + install_stubs(monkeypatch, stubs) + context = RouteContext(query=_query_for(operation), + body=_body_for(operation), + client_ip="127.0.0.1") + _assert_answers_json(handler(context), f"{method} {path}") + + +@pytest.mark.parametrize("method, path, handler", ROUTES, ids=_IDS) +def test_an_empty_request_gets_a_json_answer(method, path, handler, + monkeypatch): + """A client that sends nothing gets a status, not a dropped connection.""" + stubs = contract_stubs(handler) + if stubs is not None: + install_stubs(monkeypatch, stubs) + context = RouteContext(query="", body=None, client_ip="127.0.0.1") + _assert_answers_json(handler(context), f"{method} {path}") + + +@pytest.mark.parametrize("method, path, handler", ROUTES, ids=_IDS) +def test_a_body_of_the_wrong_shape_is_rejected_not_raised(method, path, + handler, + monkeypatch): + """Bodies arrive from the network; a list where a dict was expected is + the client's mistake to be told about, not the server's to crash on.""" + stubs = contract_stubs(handler) + if stubs is not None: + install_stubs(monkeypatch, stubs) + context = RouteContext(query="limit=not-a-number&n=not-a-number", + body=["not", "an", "object"], + client_ip="127.0.0.1") + _assert_answers_json(handler(context), f"{method} {path}") + + +# === The sweep has to keep finding things ==================================== + +def test_the_sweep_covers_the_whole_table(): + """A broken matcher would sweep nothing and still look green.""" + assert len(ROUTES) >= 30, f"only {len(ROUTES)} routes discovered" + stubbable = [route for route in ROUTES if contract_stubs(route[2])] + assert len(stubbable) > len(ROUTES) // 3, ( + f"only {len(stubbable)} of {len(ROUTES)} handlers could be stubbed " + "from their callee's annotations -- the typing contract or the " + "matcher has drifted") + + +# === The pieces the routes are built from ==================================== + +def test_query_parsing_takes_the_first_value_and_falls_back(): + context = RouteContext(query="limit=5&limit=9&other=x", body=None, + client_ip="127.0.0.1") + assert context.query_first("limit") == "5" + assert context.query_first("missing") is None + assert context.query_first("missing", "fallback") == "fallback" + assert context.query_params()["limit"] == ["5", "9"] + + +def test_a_route_without_a_query_string_parses_to_nothing(): + context = RouteContext(query="", body=None, client_ip="127.0.0.1") + assert context.query_params() == {} + assert context.query_first("limit", "100") == "100" + + +def test_health_is_the_one_route_that_needs_no_token(): + """A liveness probe must not need the bearer token the server minted.""" + assert rest_server._PUBLIC_PATHS == frozenset({"/health"}) + status, payload = rest_server.handle_health( + RouteContext(query="", body=None, client_ip="127.0.0.1")) + assert (status, payload) == (200, {"status": "ok"}) + + +def test_the_published_document_is_json_and_describes_this_api(): + spec = json.loads(json.dumps(_SPEC)) + assert spec["openapi"].startswith("3.") + assert spec["info"]["title"] + assert spec["paths"] From 0498a861b505d4eb14c029500fae50165dc34fc9 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Sun, 23 Aug 2026 23:42:34 +0800 Subject: [PATCH 05/20] Test what the trust store decides while nobody is watching `trust_list`, `fingerprint`, `address_book` and `adaptive_bitrate` decide who may drive this machine unattended, whether the host answering is the one that answered last time, what the viewer offers to reconnect to, and how hard the encoder is pushed when the link degrades. All four are ordinary Python -- a JSON file, a lock and some arithmetic -- and none of them was imported by any test on any CI square, because the subsystem raised ImportError at module level without the WebRTC extra. 61 tests over the decisions they make in the operator's absence: a trust entry that must not lose its label on re-add, a store that must open empty rather than throw on a truncated file, a fingerprint comparison that must survive an SDP spelling the same certificate in a different case, and a downscale that must need two bad samples rather than one. One is a fix rather than a guard. `AddressBook.set_tags()` cleaned its input with `str(t).strip()`, and `str(None)` is the non-empty string "None", so a JSON `null` -- what a client sends for an omitted tag -- was stored as a tag and then listed by `all_tags()` next to the real ones. --- CHANGELOG.md | 7 + .../utils/remote_desktop/address_book.py | 10 +- .../test_remote_desktop_trust_and_quality.py | 459 ++++++++++++++++++ 3 files changed, 474 insertions(+), 2 deletions(-) create mode 100644 test/unit_test/headless/test_remote_desktop_trust_and_quality.py diff --git a/CHANGELOG.md b/CHANGELOG.md index f534c58e..2c94bec1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -387,6 +387,13 @@ only when documented here with a migration path. ### Fixed +- **A `null` in a remote-desktop entry's `tags` became a tag named `"None"`.** + `AddressBook.set_tags()` cleaned its input with `str(t).strip()`, and + `str(None)` is the non-empty string `"None"`, so a JSON `null` in the array — + what a client sends for an omitted tag — was stored as a tag and then listed + by `all_tags()` alongside the real ones. Nulls are now dropped. Tags that + were already stored this way stay until the entry's tags are set again. + - **Typing text through the key-event route raised `AttributeError` on the three platforms that cannot do it.** `type_unicode_keys()` (and `AC_type_unicode_keys` / `ac_type_unicode_keys`) called the backend's diff --git a/je_auto_control/utils/remote_desktop/address_book.py b/je_auto_control/utils/remote_desktop/address_book.py index 279dbfd1..8727adfd 100644 --- a/je_auto_control/utils/remote_desktop/address_book.py +++ b/je_auto_control/utils/remote_desktop/address_book.py @@ -142,8 +142,14 @@ def _build_entry(*, host_id: str, server_url: str, def set_tags(self, *, host_id: str, server_url: str, tags: list) -> None: - """Replace ``tags`` on the matching entry.""" - clean = [str(t).strip() for t in tags if str(t).strip()] + """Replace ``tags`` on the matching entry. + + A JSON ``null`` reaches here as None, and ``str(None)`` is the + non-empty string ``"None"`` -- so an omitted tag used to be stored as + one, and `all_tags` then listed it alongside the real ones. + """ + clean = [str(t).strip() for t in tags + if t is not None and str(t).strip()] with self._lock: for entry in self._entries: if (entry.get("host_id") == host_id diff --git a/test/unit_test/headless/test_remote_desktop_trust_and_quality.py b/test/unit_test/headless/test_remote_desktop_trust_and_quality.py new file mode 100644 index 00000000..7d153ac4 --- /dev/null +++ b/test/unit_test/headless/test_remote_desktop_trust_and_quality.py @@ -0,0 +1,459 @@ +"""The remote-desktop trust store, address book and quality controller. + +These four modules decide who is allowed to drive this machine unattended +(`trust_list`), whether the host answering is the host that answered last time +(`fingerprint`), what the viewer offers to reconnect to (`address_book`), and +how hard the encoder is pushed when the link degrades (`adaptive_bitrate`). +They are ordinary Python -- a JSON file, a lock and some arithmetic -- but +until the `[webrtc]` extra became part of the measured install they were +imported by nothing on any CI square, because `adaptive_bitrate` reaches +`webrtc_stats` and the whole subsystem raised ImportError at module level. + +What these tests are for: every one of them describes a decision the code makes +in the operator's absence. A trust entry that silently loses its label on +re-add, a known-hosts file that throws on a truncated write, a fingerprint +comparison that is case-sensitive against an SDP that is not, a downscale that +fires on a single dropped packet -- each is invisible until someone is +connected from another building. +""" +import json + +import pytest + +from je_auto_control.utils.remote_desktop.address_book import AddressBook +from je_auto_control.utils.remote_desktop.adaptive_bitrate import ( + AdaptiveBitrateController, +) +from je_auto_control.utils.remote_desktop.fingerprint import ( + FingerprintMismatchError, KnownHosts, extract_dtls_fingerprint, + fingerprint_for_display, load_or_create_host_fingerprint, + verify_dtls_fingerprint, +) +from je_auto_control.utils.remote_desktop.trust_list import TrustList +from je_auto_control.utils.remote_desktop.webrtc_stats import StatsSnapshot + + +# === The host's own fingerprint ============================================= + +def test_host_fingerprint_is_created_once_and_then_reused(tmp_path): + """First call mints 64 hex chars; later calls return the same string.""" + target = tmp_path / "nested" / "host_fingerprint" + first = load_or_create_host_fingerprint(target) + assert len(first) == 64 + assert int(first, 16) >= 0 # hex, not arbitrary text + assert load_or_create_host_fingerprint(target) == first + assert target.read_text(encoding="utf-8").strip() == first + + +def test_a_truncated_fingerprint_file_is_replaced_not_trusted(tmp_path): + """A half-written file must not become this host's identity.""" + target = tmp_path / "host_fingerprint" + target.write_text("deadbeef", encoding="utf-8") + minted = load_or_create_host_fingerprint(target) + assert len(minted) == 64 + assert minted != "deadbeef" + + +def test_an_unwritable_fingerprint_path_still_returns_one(tmp_path): + """Persistence is best-effort: the session gets an identity regardless.""" + blocker = tmp_path / "blocker" + blocker.write_text("not a directory", encoding="utf-8") + minted = load_or_create_host_fingerprint(blocker / "sub" / "fp") + assert len(minted) == 64 + + +# === The viewer's known-hosts map =========================================== + +def test_known_hosts_survives_a_restart(tmp_path): + path = tmp_path / "known_hosts.json" + first = KnownHosts(path) + first.remember("host-a", "a" * 64) + first.remember_dtls_fingerprint("host-a", "AB:CD") + first.touch("host-a") + + second = KnownHosts(path) + assert second.fingerprint_for("host-a") == "a" * 64 + assert second.dtls_fingerprint_for("host-a") == "AB:CD" + assert second.last_seen("host-a") is not None + + +def test_remembering_one_fingerprint_keeps_the_other(tmp_path): + """The app-layer and DTLS fingerprints are stored side by side.""" + hosts = KnownHosts(tmp_path / "known_hosts.json") + hosts.remember_dtls_fingerprint("host-a", "AB:CD") + hosts.remember("host-a", "a" * 64) + assert hosts.dtls_fingerprint_for("host-a") == "AB:CD" + hosts.remember_dtls_fingerprint("host-a", "EF:01") + assert hosts.fingerprint_for("host-a") == "a" * 64 + assert hosts.dtls_fingerprint_for("host-a") == "EF:01" + + +def test_a_legacy_plain_string_entry_is_migrated_on_load(tmp_path): + """Files written before the DTLS fingerprint existed still open.""" + path = tmp_path / "known_hosts.json" + path.write_text(json.dumps({"host-a": "a" * 64}), encoding="utf-8") + hosts = KnownHosts(path) + assert hosts.fingerprint_for("host-a") == "a" * 64 + assert hosts.dtls_fingerprint_for("host-a") is None + + +@pytest.mark.parametrize("payload", ["{not json", json.dumps(["a", "b"]), + json.dumps({"host-a": 7})]) +def test_an_unreadable_known_hosts_file_opens_empty(tmp_path, payload): + """A corrupt store must not stop the viewer from connecting at all.""" + path = tmp_path / "known_hosts.json" + path.write_text(payload, encoding="utf-8") + assert KnownHosts(path).list_entries() == {} + + +def test_forget_reports_whether_anything_was_removed(tmp_path): + hosts = KnownHosts(tmp_path / "known_hosts.json") + hosts.remember("host-a", "a" * 64) + assert hosts.forget("host-a") is True + assert hosts.forget("host-a") is False + assert hosts.fingerprint_for("host-a") is None + + +def test_list_entries_hands_back_copies(tmp_path): + """A caller mutating the report must not rewrite the trust store.""" + hosts = KnownHosts(tmp_path / "known_hosts.json") + hosts.remember("host-a", "a" * 64) + hosts.list_entries()["host-a"]["app_fp"] = "tampered" + assert hosts.fingerprint_for("host-a") == "a" * 64 + + +# === Showing and comparing fingerprints ===================================== + +def test_display_form_groups_a_full_fingerprint_into_fours(): + grouped = fingerprint_for_display("ab" * 32) + assert grouped.count(":") == 15 + assert grouped.replace(":", "") == "ab" * 32 + + +@pytest.mark.parametrize("value, expected", [("", ""), ("short", "short")]) +def test_display_form_passes_through_what_it_cannot_group(value, expected): + assert fingerprint_for_display(value) == expected + + +_SDP = ( + "v=0\r\n" + "a=fingerprint:sha-1 11:22:33\r\n" + "m=video 9 UDP/TLS/RTP/SAVPF 96\r\n" + "a=fingerprint:SHA-256 ab:cd:ef:01\r\n" +) + + +def test_the_requested_algorithm_is_the_one_extracted(): + """An offer carries several; picking the wrong line pins the wrong cert.""" + assert extract_dtls_fingerprint(_SDP, "sha-256") == "AB:CD:EF:01" + assert extract_dtls_fingerprint(_SDP, "sha-1") == "11:22:33" + assert extract_dtls_fingerprint(_SDP, "sha-384") is None + + +def test_a_non_string_offer_yields_no_fingerprint(): + assert extract_dtls_fingerprint(None) is None + + +@pytest.mark.parametrize("expected", ["ab:cd:ef:01", "ABCDEF01", + "AB:CD:EF:01"]) +def test_verification_accepts_either_spelling_of_the_same_value(expected): + verify_dtls_fingerprint(_SDP, expected) + + +def test_verification_rejects_a_different_certificate(): + with pytest.raises(FingerprintMismatchError, match="mismatch"): + verify_dtls_fingerprint(_SDP, "00:11:22:33") + + +def test_verification_fails_closed_when_the_offer_pins_nothing(): + with pytest.raises(FingerprintMismatchError, match="no sha-256"): + verify_dtls_fingerprint("v=0\r\n", "AB:CD") + + +# === The unattended-access trust list ======================================= + +def test_a_trusted_viewer_is_still_trusted_after_a_restart(tmp_path): + path = tmp_path / "trusted_viewers.json" + TrustList(path).add("viewer-1", label="office laptop") + reopened = TrustList(path) + assert reopened.is_trusted("viewer-1") is True + assert reopened.list_entries()[0]["label"] == "office laptop" + + +def test_re_adding_keeps_the_original_label_and_first_seen_time(tmp_path): + """Re-authenticating must not quietly erase what the operator typed.""" + trust = TrustList(tmp_path / "trusted_viewers.json") + trust.add("viewer-1", label="office laptop") + added_at = trust.list_entries()[0]["added_at"] + trust.add("viewer-1") + entry = trust.list_entries()[0] + assert entry["label"] == "office laptop" + assert entry["added_at"] == added_at + assert len(trust.list_entries()) == 1 + + +def test_a_new_label_replaces_the_old_one(tmp_path): + trust = TrustList(tmp_path / "trusted_viewers.json") + trust.add("viewer-1", label="old") + trust.add("viewer-1", label="new") + assert trust.list_entries()[0]["label"] == "new" + + +def test_touch_records_use_only_for_a_viewer_already_trusted(tmp_path): + trust = TrustList(tmp_path / "trusted_viewers.json") + trust.touch("stranger") + assert trust.list_entries() == [] + trust.add("viewer-1") + assert trust.list_entries()[0]["last_used"] is None + trust.touch("viewer-1") + assert trust.list_entries()[0]["last_used"] is not None + + +def test_touch_preserves_last_used_across_a_re_add(tmp_path): + trust = TrustList(tmp_path / "trusted_viewers.json") + trust.add("viewer-1") + trust.touch("viewer-1") + last_used = trust.list_entries()[0]["last_used"] + trust.add("viewer-1") + assert trust.list_entries()[0]["last_used"] == last_used + + +@pytest.mark.parametrize("viewer_id", ["", None, 7]) +def test_an_empty_or_non_string_viewer_id_is_refused(tmp_path, viewer_id): + """A blank id would trust every viewer that fails to send one.""" + trust = TrustList(tmp_path / "trusted_viewers.json") + with pytest.raises(ValueError): + trust.add(viewer_id) + + +@pytest.mark.parametrize("viewer_id", [None, 7, b"viewer-1"]) +def test_a_non_string_id_is_never_trusted(tmp_path, viewer_id): + assert TrustList(tmp_path / "trusted_viewers.json").is_trusted( + viewer_id) is False + + +def test_remove_and_clear_report_and_persist(tmp_path): + path = tmp_path / "trusted_viewers.json" + trust = TrustList(path) + trust.add("viewer-1") + trust.add("viewer-2") + assert trust.remove("viewer-1") is True + assert trust.remove("viewer-1") is False + trust.clear() + assert TrustList(path).list_entries() == [] + + +@pytest.mark.parametrize("payload", ["{not json", json.dumps(["a"]), + json.dumps({"viewers": "not a list"}), + json.dumps({"viewers": [1, {}]})]) +def test_an_unreadable_trust_list_opens_empty(tmp_path, payload): + """Fail closed: a damaged file trusts nobody rather than everybody.""" + path = tmp_path / "trusted_viewers.json" + path.write_text(payload, encoding="utf-8") + assert TrustList(path).list_entries() == [] + + +# === The viewer's address book ============================================== + +def test_upsert_refreshes_the_matching_entry_instead_of_appending(tmp_path): + book = AddressBook(tmp_path / "book.json") + book.upsert(host_id="h1", server_url="ws://a", label="desk") + first_used = book.list_entries()[0]["last_used"] + book.upsert(host_id="h1", server_url="ws://a", mac_address="00:11:22") + entries = book.list_entries() + assert len(entries) == 1 + assert entries[0]["label"] == "desk" # blank label does not erase + assert entries[0]["mac_address"] == "00:11:22" + assert entries[0]["last_used"] >= first_used + + +def test_the_same_host_on_a_different_url_is_a_different_entry(tmp_path): + book = AddressBook(tmp_path / "book.json") + book.upsert(host_id="h1", server_url="ws://a") + book.upsert(host_id="h1", server_url="ws://b") + assert len(book.list_entries()) == 2 + + +@pytest.mark.parametrize("host_id, server_url", [("", "ws://a"), ("h1", "")]) +def test_an_incomplete_target_is_refused(tmp_path, host_id, server_url): + book = AddressBook(tmp_path / "book.json") + with pytest.raises(ValueError): + book.upsert(host_id=host_id, server_url=server_url) + + +def test_tags_are_stripped_deduplicated_and_sorted(tmp_path): + book = AddressBook(tmp_path / "book.json") + book.upsert(host_id="h1", server_url="ws://a") + book.upsert(host_id="h2", server_url="ws://b") + book.set_tags(host_id="h1", server_url="ws://a", tags=[" lab ", "", "prod"]) + book.set_tags(host_id="h2", server_url="ws://b", tags=["prod", None]) + assert book.list_entries()[0]["tags"] == ["lab", "prod"] + # A JSON `null` is dropped, not stringified into a tag named "None". + assert book.all_tags() == ["lab", "prod"] + + +def test_setting_tags_on_an_unknown_target_changes_nothing(tmp_path): + book = AddressBook(tmp_path / "book.json") + book.upsert(host_id="h1", server_url="ws://a") + book.set_tags(host_id="nope", server_url="ws://z", tags=["x"]) + assert book.all_tags() == [] + + +def test_toggle_favorite_reports_the_new_state_and_persists(tmp_path): + path = tmp_path / "book.json" + book = AddressBook(path) + book.upsert(host_id="h1", server_url="ws://a") + assert book.toggle_favorite(host_id="h1", server_url="ws://a") is True + assert book.toggle_favorite(host_id="h1", server_url="ws://a") is False + assert book.toggle_favorite(host_id="nope", server_url="ws://z") is False + assert AddressBook(path).list_entries()[0]["favorite"] is False + + +def test_remove_and_clear_report_and_persist_for_the_book(tmp_path): + path = tmp_path / "book.json" + book = AddressBook(path) + book.upsert(host_id="h1", server_url="ws://a") + assert book.remove(host_id="h1", server_url="ws://a") is True + assert book.remove(host_id="h1", server_url="ws://a") is False + book.upsert(host_id="h2", server_url="ws://b") + book.clear() + assert AddressBook(path).list_entries() == [] + + +@pytest.mark.parametrize("payload", ["{not json", json.dumps(["a"]), + json.dumps({"entries": [1, {}, { + "host_id": 7}]})]) +def test_an_unreadable_address_book_opens_empty(tmp_path, payload): + path = tmp_path / "book.json" + path.write_text(payload, encoding="utf-8") + assert AddressBook(path).list_entries() == [] + + +# === The adaptive quality controller ======================================== + +class _FakeTrack: + """The one thing the controller drives: a track with a target FPS.""" + + def __init__(self, fps: int = 20) -> None: + self.fps = fps + self.calls = [] + + def set_target_fps(self, value: int) -> None: + self.calls.append(value) + self.fps = value + + +def _feed(controller, count, **fields): + for _ in range(count): + controller.on_stats(StatsSnapshot(**fields)) + + +def test_a_single_bad_sample_does_not_move_the_encoder(): + """Hysteresis: one dropped packet is noise, not a degraded link.""" + track = _FakeTrack(20) + controller = AdaptiveBitrateController(track) + _feed(controller, 1, packet_loss_pct=9.0) + assert track.calls == [] + assert controller.current_fps == 20 + + +def test_two_consecutive_lossy_samples_step_the_rate_down(): + track = _FakeTrack(20) + controller = AdaptiveBitrateController(track) + _feed(controller, 2, packet_loss_pct=9.0) + assert track.calls == [16] + + +def test_a_latency_spike_downscales_the_same_way_loss_does(): + track = _FakeTrack(20) + controller = AdaptiveBitrateController(track) + _feed(controller, 2, rtt_ms=400.0) + assert track.calls == [16] + + +def test_a_good_sample_between_two_bad_ones_resets_the_streak(): + track = _FakeTrack(20) + controller = AdaptiveBitrateController(track) + _feed(controller, 1, packet_loss_pct=9.0) + _feed(controller, 1, packet_loss_pct=3.0) # neither down nor up + _feed(controller, 1, packet_loss_pct=9.0) + assert track.calls == [] + + +def test_the_rate_never_falls_through_the_floor(): + track = _FakeTrack(7) + controller = AdaptiveBitrateController(track, floor_fps=6) + _feed(controller, 2, packet_loss_pct=9.0) + assert track.fps == 6 + _feed(controller, 2, packet_loss_pct=9.0) + assert track.calls == [6] # already at the floor: no second call + + +def test_climbing_back_takes_four_clean_samples_and_stops_at_the_ceiling(): + track = _FakeTrack(12) + controller = AdaptiveBitrateController(track, max_fps=14) + _feed(controller, 3, packet_loss_pct=0.1) + assert track.calls == [] + _feed(controller, 1, packet_loss_pct=0.1) + assert track.calls == [14] # +4 would be 16; the ceiling wins + _feed(controller, 4, packet_loss_pct=0.1) + assert track.calls == [14] # at the ceiling: nothing further + + +def test_sustained_latency_keeps_stepping_down_however_clean_the_link(): + """Loss near zero does not license a climb while RTT stays bad.""" + track = _FakeTrack(12) + controller = AdaptiveBitrateController(track, max_fps=20) + _feed(controller, 8, packet_loss_pct=0.1, rtt_ms=400.0) + assert track.calls == [8, 5] # the RTT rule downscales every two samples + + +def test_exceeding_the_hard_bitrate_cap_steps_down_on_the_first_sample(): + """A configured cap is a promise to the network, not a trend to confirm.""" + track = _FakeTrack(20) + controller = AdaptiveBitrateController(track, max_bitrate_kbps=1000) + _feed(controller, 1, bitrate_kbps=4000.0, packet_loss_pct=0.1) + assert track.calls == [16] + + +def test_the_hard_cap_resets_the_quality_streaks(): + """A cap breach mid-climb must not count towards the next upscale.""" + track = _FakeTrack(12) + controller = AdaptiveBitrateController(track, max_fps=20, + max_bitrate_kbps=1000) + _feed(controller, 3, packet_loss_pct=0.1) + _feed(controller, 1, bitrate_kbps=4000.0) + assert track.calls == [8] + _feed(controller, 3, packet_loss_pct=0.1) + assert track.calls == [8] # the streak restarted from zero + + +def test_a_bitrate_under_the_cap_leaves_the_hard_path_alone(): + track = _FakeTrack(20) + controller = AdaptiveBitrateController(track, max_bitrate_kbps=5000) + _feed(controller, 1, bitrate_kbps=1000.0, packet_loss_pct=9.0) + assert track.calls == [] # quality path only, and it needs two + + +def test_disabling_the_controller_freezes_the_rate(): + track = _FakeTrack(20) + controller = AdaptiveBitrateController(track) + controller.set_enabled(False) + _feed(controller, 6, packet_loss_pct=9.0) + assert track.calls == [] + controller.set_enabled(True) + _feed(controller, 2, packet_loss_pct=9.0) + assert track.calls == [16] + + +def test_a_controller_with_no_track_reports_no_rate(): + controller = AdaptiveBitrateController(_FakeTrack(20)) + controller._track = None # the teardown state the host leaves behind + controller.on_stats(StatsSnapshot(packet_loss_pct=9.0)) + assert controller.current_fps == 0 + + +def test_a_stats_snapshot_crosses_the_json_boundary(): + snapshot = StatsSnapshot(rtt_ms=12.5, fps=30.0, bitrate_kbps=900.0, + packet_loss_pct=0.2, jitter_ms=3.0) + assert json.loads(json.dumps(snapshot.to_dict()))["rtt_ms"] == 12.5 From b0d6d5f1587d65c1cea64a5b61cdda16950d3785 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Sun, 23 Aug 2026 23:57:18 +0800 Subject: [PATCH 06/20] Test the auth boundary and the link arithmetic, and raise the floor to 74 `ViewerAuthMixin` decides whether a peer that just connected may drive this machine, and `StatsPoller` produces the numbers the adaptive controller acts on. Both are reachable only through `webrtc_transport`, which raised ImportError at module level without the extra, so neither was imported by any test on any CI square until this week. 56 tests over what they actually decide: a token that is only accepted when it is an equal string, a trust list that fails closed when it cannot be read, an IP whitelist that matches by network rather than by string, a grace period that closes a peer which never authenticated, and five derived rates that must refuse to invent a number from one sample, a zero interval or a counter that went backwards. The auth host double supplies exactly the attribute list the mixin's own docstring asks for, so a mixin that starts reaching for something else fails here rather than leaning on whatever the real host happens to own. `_sole_imported_name` now requires the package prefix, the same tightening the other matcher got: a delegator standing in front of `json.dumps` is not wiring under test, and it makes both dynamic imports provably local. The floor moves 69 -> 74, read from the nine-way matrix at 74.99% (ubuntu-22.04 / 3.14) to 76.19% (windows-2022). The comment now also says which number that is: `coverage report`'s total includes branches and the `coverage.xml` artifact's `line-rate` does not, so the artifact reads ~1.8 points higher and a floor set from it would be one the suite cannot clear. --- pyproject.toml | 21 +- test/unit_test/headless/_contract_sweep.py | 13 +- .../headless/test_webrtc_host_auth.py | 381 ++++++++++++++++++ .../headless/test_webrtc_stats_poller.py | 240 +++++++++++ 4 files changed, 649 insertions(+), 6 deletions(-) create mode 100644 test/unit_test/headless/test_webrtc_host_auth.py create mode 100644 test/unit_test/headless/test_webrtc_stats_poller.py diff --git a/pyproject.toml b/pyproject.toml index e4150f14..0b02bc81 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -172,9 +172,24 @@ precision = 2 # Floored to the integer below the lowest square, the same convention that # turned a 50.26% minimum into 50. # -# 70 was the destination this was climbing towards and the suite is level with -# it; Progress.md records that the next destination is the maintainer's call. -fail_under = 69 +# 70 was the destination that number was climbing towards; the maintainer set +# the next one at 80 on 2026-08-23, and Progress.md tracks the climb. +# +# Measured 2026-08-23 on the run for this PR, after `quality.yml` began +# installing the `[webrtc]` extra and two more registries were swept: the +# nine-way matrix runs 74.99% (ubuntu-22.04 / 3.14) to 76.19% (windows-2022, +# 3.10 through 3.13). Windows is still the high corner for the same reason as +# before — the facade imports the backend of the platform it is running on. +# Floored to the integer below the lowest square: 74. +# +# Read that matrix from the "Enforce the coverage floor" step, which is this +# report and therefore the number `fail_under` is compared against. The +# `coverage.xml` artifact is NOT interchangeable with it: `branch = true` +# above puts branches in this total and Cobertura's `line-rate` attribute +# leaves them out, so the XML reads about 1.8 points higher (78.33% against +# 76.14% on windows-2022 / 3.14, same run). A floor set from the artifact +# would be a floor the suite cannot clear. +fail_under = 74 [tool.mypy] python_version = "3.10" diff --git a/test/unit_test/headless/_contract_sweep.py b/test/unit_test/headless/_contract_sweep.py index 37abf6f9..69484e96 100644 --- a/test/unit_test/headless/_contract_sweep.py +++ b/test/unit_test/headless/_contract_sweep.py @@ -210,10 +210,15 @@ def _project_import(adapter: Any) -> Optional[Tuple[str, List[str]]]: def _sole_imported_name(node: ast.stmt) -> Optional[Tuple[str, str, str]]: - """Return ``(module, attribute, local_name)`` for ``from X import y as z``.""" + """Return ``(module, attribute, local_name)`` for ``from X import y as z``. + + ``X`` has to be inside this package: a delegator standing in front of + ``json.dumps`` is not wiring under test, and it is the prefix that lets the + parsed path be imported below without trusting what the source said. + """ if not isinstance(node, ast.ImportFrom) or not node.module or node.level: return None - if len(node.names) != 1: + if len(node.names) != 1 or not node.module.startswith(_PACKAGE + "."): return None alias = node.names[0] return node.module, alias.name, alias.asname or alias.name @@ -372,7 +377,9 @@ def _install_recorder(monkeypatch: pytest.MonkeyPatch, module_path: str, Returns the record dict, the sentinel the recorder returns, and the real callable, whose signature says what the recorded arguments are named. """ - module = importlib.import_module(module_path) + # ``module_path`` is the ``je_auto_control.*`` path `_sole_imported_name` + # parsed and prefix-checked; no caller supplies it. + module = importlib.import_module(module_path) # nosemgrep # reason: prefix-checked at the parse site original = getattr(module, attribute) record: Dict[str, Any] = {} sentinel = object() diff --git a/test/unit_test/headless/test_webrtc_host_auth.py b/test/unit_test/headless/test_webrtc_host_auth.py new file mode 100644 index 00000000..eb6e947d --- /dev/null +++ b/test/unit_test/headless/test_webrtc_host_auth.py @@ -0,0 +1,381 @@ +"""Everything between a viewer's first `auth` message and its first keystroke. + +`ViewerAuthMixin` is the half of the WebRTC host that decides whether a peer +that just connected may drive this machine: the token check, the two +auto-approve paths (trust list, IP whitelist), the manual accept/reject the GUI +drives, and the grace period that closes a peer which never authenticates. It +is the security boundary of the whole remote-desktop subsystem, and until the +`[webrtc]` extra became part of the measured install nothing imported it on any +CI square -- `webrtc_transport` raises ImportError at module level without +aiortc, so the module could not even be loaded to be tested. + +The mixin's own docstring lists what it needs from the host it is mixed into +(`_token`, `_trust_list`, `_ip_whitelist`, `_authenticated`, `_send_ctrl`, +`_spawn_bg`, `_async_stop`, ...). `_Host` below supplies exactly that list and +nothing else, so a mixin that starts reaching for something new fails here +rather than depending on whatever `WebRTCDesktopHost` happens to also own. + +The two collaborators that reach outside the process are replaced: the asyncio +bridge, whose `call_soon` would otherwise queue the work onto a loop no test is +running, and the audit log, which writes to the user's real +`~/.je_auto_control`. +""" +import asyncio + +import pytest + +from je_auto_control.utils.remote_desktop import webrtc_host_auth as auth_module +from je_auto_control.utils.remote_desktop.permissions import SessionPermissions +from je_auto_control.utils.remote_desktop.trust_list import TrustList +from je_auto_control.utils.remote_desktop.webrtc_host_auth import ViewerAuthMixin + + +class _ImmediateBridge: + """The asyncio bridge, minus the loop: run the callback where it is made.""" + + def __init__(self) -> None: + self.deferred = [] + + def call_soon(self, callback) -> None: + self.deferred.append(callback) + callback() + + +class _AuditLog: + def __init__(self) -> None: + self.events = [] + + def log(self, event_type, **fields) -> None: + self.events.append((event_type, fields)) + + +class _Host(ViewerAuthMixin): + """A host with exactly the attributes the mixin's docstring asks for.""" + + def __init__(self, *, token="secret", trust_list=None, ip_whitelist=None, + remote_ip=None, permissions=None, on_pending_viewer=None): + self._token = token + self._trust_list = trust_list + self._ip_whitelist = list(ip_whitelist or []) + self._permissions = permissions or SessionPermissions.full_control() + self._remote_ip = remote_ip + self._authenticated = False + self._has_pending_viewer = False + self._pending_viewer_id = None + self._auth_deadline_handle = None + self._on_authenticated = None + self._on_pending_viewer = on_pending_viewer + self.sent = [] + self.stopped = 0 + self.spawned = [] + + def _send_ctrl(self, message): + self.sent.append(message) + + def _spawn_bg(self, coroutine): + coroutine.close() # never awaited here; do not leak the frame + self.spawned.append(coroutine) + return coroutine + + async def _async_stop(self): + self.stopped += 1 + + def types_sent(self): + return [message["type"] for message in self.sent] + + +@pytest.fixture +def bridge(monkeypatch): + """Replace the asyncio bridge and the audit log for the whole module. + + A loop is installed but never run, which is the shape the real bridge + presents to this code: ``call_soon`` work happens on the loop thread, so + ``asyncio.get_event_loop()`` resolves there, while anything handed to + ``call_later`` is still only scheduled when the test looks at it. + """ + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + immediate = _ImmediateBridge() + log = _AuditLog() + monkeypatch.setattr(auth_module, "get_bridge", lambda: immediate) + monkeypatch.setattr(auth_module, "default_audit_log", lambda: log) + monkeypatch.setattr(auth_module, "load_or_create_host_fingerprint", + lambda: "f" * 64) + immediate.audit = log + immediate.loop = loop + yield immediate + asyncio.set_event_loop(None) + loop.close() + + +# === The token is the gate ================================================== + +@pytest.mark.parametrize("token", ["wrong", "", None, 7, ["secret"]]) +def test_a_wrong_or_malformed_token_never_authenticates(bridge, token): + """The token arrives from the network; only an equal string may pass.""" + host = _Host(token="secret") + host._handle_auth({"token": token, "viewer_id": "v1"}) + assert host._authenticated is False + assert host.types_sent() == ["auth_fail"] + + +def test_a_missing_token_field_is_a_wrong_token(bridge): + host = _Host(token="secret") + host._handle_auth({"viewer_id": "v1"}) + assert host.types_sent() == ["auth_fail"] + + +def test_a_rejected_viewer_is_audited_and_then_disconnected(bridge): + host = _Host(token="secret", remote_ip="10.0.0.9") + host._handle_auth({"token": "wrong", "viewer_id": "v1"}) + assert [event for event, _ in bridge.audit.events] == ["auth_fail"] + fields = bridge.audit.events[0][1] + assert fields["viewer_id"] == "v1" + assert "10.0.0.9" in fields["detail"] + # `_schedule_close_after_fail` was handed to the bridge, so the peer is + # on its way out rather than left holding an unauthenticated channel. + assert bridge.deferred + + +def test_the_right_token_with_no_prompt_hook_approves_immediately(bridge): + """Headless use has no GUI to ask; the token is then the whole check.""" + host = _Host(token="secret") + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + assert host._authenticated is True + assert host.types_sent() == ["auth_ok"] + + +def test_the_right_token_with_a_prompt_hook_waits_for_a_person(bridge): + asked = [] + host = _Host(token="secret", on_pending_viewer=lambda: asked.append(True)) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + assert asked == [True] + assert host.has_pending_viewer is True + assert host._authenticated is False + assert host.types_sent() == [] + + +def test_a_prompt_hook_that_throws_leaves_the_viewer_pending(bridge): + """A broken GUI callback must not authenticate the peer by accident.""" + def explode(): + raise RuntimeError("no window") + + host = _Host(token="secret", on_pending_viewer=explode) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + assert host._authenticated is False + assert host.has_pending_viewer is True + + +def test_a_non_string_viewer_id_is_recorded_as_absent(bridge): + host = _Host(token="secret", on_pending_viewer=lambda: None) + host._handle_auth({"token": "secret", "viewer_id": 7}) + assert host.pending_viewer_id is None + + +# === Auto-approve by trust list ============================================= + +def test_a_trusted_viewer_skips_the_prompt(bridge, tmp_path): + trust = TrustList(tmp_path / "trusted.json") + trust.add("v1", label="office laptop") + asked = [] + host = _Host(token="secret", trust_list=trust, + on_pending_viewer=lambda: asked.append(True)) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + assert asked == [] + assert host._authenticated is True + assert trust.list_entries()[0]["last_used"] is not None + + +def test_an_untrusted_viewer_still_gets_the_prompt(bridge, tmp_path): + trust = TrustList(tmp_path / "trusted.json") + trust.add("someone-else") + asked = [] + host = _Host(token="secret", trust_list=trust, + on_pending_viewer=lambda: asked.append(True)) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + assert asked == [True] + assert host._authenticated is False + + +def test_a_trust_list_that_throws_does_not_auto_approve(bridge): + """Fail closed: an unreadable trust store trusts nobody.""" + class _Broken: + def is_trusted(self, viewer_id): + raise OSError("disk gone") + + asked = [] + host = _Host(token="secret", trust_list=_Broken(), + on_pending_viewer=lambda: asked.append(True)) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + assert host._authenticated is False + assert asked == [True] + + +def test_trusting_the_pending_viewer_adds_it_and_approves(bridge, tmp_path): + trust = TrustList(tmp_path / "trusted.json") + host = _Host(token="secret", trust_list=trust, + on_pending_viewer=lambda: None) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + host.trust_pending_viewer(label="lab machine") + assert trust.is_trusted("v1") is True + assert trust.list_entries()[0]["label"] == "lab machine" + assert host._authenticated is True + + +def test_trusting_with_no_trust_list_still_approves(bridge): + host = _Host(token="secret", on_pending_viewer=lambda: None) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + host.trust_pending_viewer() + assert host._authenticated is True + + +# === Auto-approve by IP whitelist =========================================== + +@pytest.mark.parametrize("remote_ip, allowed", [ + ("10.0.0.5", True), + ("10.0.1.5", False), + ("192.168.1.7", True), + ("::1", False), +]) +def test_the_whitelist_matches_by_network_not_by_string(bridge, remote_ip, + allowed): + asked = [] + host = _Host(token="secret", remote_ip=remote_ip, + ip_whitelist=["10.0.0.0/24", " 192.168.1.7 "], + on_pending_viewer=lambda: asked.append(True)) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + assert host._authenticated is allowed + assert asked == ([] if allowed else [True]) + + +def test_an_ipv6_peer_matches_an_ipv6_whitelist(bridge): + host = _Host(token="secret", remote_ip="fd00::5", + ip_whitelist=["fd00::/8"], on_pending_viewer=lambda: None) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + assert host._authenticated is True + + +@pytest.mark.parametrize("whitelist", [["not-a-cidr"], ["10.0.0.0/99"], []]) +def test_an_unparseable_whitelist_entry_is_skipped_not_fatal(bridge, + whitelist): + host = _Host(token="secret", remote_ip="10.0.0.5", + ip_whitelist=whitelist, on_pending_viewer=lambda: None) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + assert host._authenticated is False + + +@pytest.mark.parametrize("remote_ip", [None, "", "not-an-address"]) +def test_a_peer_with_no_usable_address_never_matches(bridge, remote_ip): + host = _Host(token="secret", remote_ip=remote_ip, + ip_whitelist=["0.0.0.0/0"], on_pending_viewer=lambda: None) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + assert host._authenticated is False + + +# === What approval and rejection actually send ============================== + +def test_approval_tells_the_viewer_what_it_may_do(bridge): + host = _Host(token="secret", + permissions=SessionPermissions.view_only(), + on_pending_viewer=lambda: None) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + host.approve_pending_viewer() + message = host.sent[-1] + assert message["type"] == "auth_ok" + assert message["read_only"] is True + assert message["permissions"]["allow_input"] is False + assert message["fingerprint"] == "f" * 64 + + +def test_approval_is_audited_and_cancels_the_grace_period(bridge): + class _Handle: + cancelled = False + + def cancel(self): + self.cancelled = True + + handle = _Handle() + host = _Host(token="secret", remote_ip="10.0.0.9", + on_pending_viewer=lambda: None) + host._auth_deadline_handle = handle + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + host.approve_pending_viewer() + assert handle.cancelled is True + assert host._auth_deadline_handle is None + assert [event for event, _ in bridge.audit.events] == ["auth_ok"] + + +def test_the_authenticated_callback_runs_and_its_failure_is_contained(bridge): + called = [] + host = _Host(token="secret", on_pending_viewer=lambda: None) + host._on_authenticated = lambda: called.append(True) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + host.approve_pending_viewer() + assert called == [True] + + def explode(): + raise RuntimeError("gui gone") + + other = _Host(token="secret", on_pending_viewer=lambda: None) + other._on_authenticated = explode + other._handle_auth({"token": "secret", "viewer_id": "v2"}) + other.approve_pending_viewer() + assert other._authenticated is True + + +def test_approving_twice_does_not_re_send_auth_ok(bridge): + """The GUI can double-click; the viewer must not see two sessions open.""" + host = _Host(token="secret", on_pending_viewer=lambda: None) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + host.approve_pending_viewer() + host.approve_pending_viewer() + assert host.types_sent() == ["auth_ok"] + + +def test_rejecting_the_pending_viewer_fails_and_closes(bridge): + host = _Host(token="secret", on_pending_viewer=lambda: None) + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + host.reject_pending_viewer() + assert host.types_sent() == ["auth_fail"] + assert host.has_pending_viewer is False + assert host._authenticated is False + + +# === The grace period ======================================================= + +def test_the_deadline_closes_a_peer_that_never_authenticated(bridge): + host = _Host(token="secret", on_pending_viewer=lambda: None) + host._enforce_auth_deadline() + assert host.spawned, "an unauthenticated peer must be stopped" + + +def test_the_deadline_leaves_an_authenticated_peer_alone(bridge): + host = _Host(token="secret") + host._handle_auth({"token": "secret", "viewer_id": "v1"}) + host._enforce_auth_deadline() + assert host.spawned == [] + + +# === The secure attention sequence ========================================== + +def test_a_successful_sas_is_reported_to_the_viewer(bridge, monkeypatch): + import je_auto_control.utils.remote_desktop.session_actions as actions + monkeypatch.setattr(actions, "send_secure_attention_sequence", + lambda: None) + host = _Host(token="secret") + host._handle_send_sas() + assert host.types_sent() == ["sas_ok"] + + +def test_a_platform_that_cannot_send_sas_says_so_rather_than_raising( + bridge, monkeypatch): + import je_auto_control.utils.remote_desktop.session_actions as actions + + def unsupported(): + raise RuntimeError("SendSAS is Windows-only") + + monkeypatch.setattr(actions, "send_secure_attention_sequence", unsupported) + host = _Host(token="secret") + host._handle_send_sas() + assert host.types_sent() == ["sas_fail"] + assert "Windows-only" in host.sent[-1]["error"] diff --git a/test/unit_test/headless/test_webrtc_stats_poller.py b/test/unit_test/headless/test_webrtc_stats_poller.py new file mode 100644 index 00000000..a4c27423 --- /dev/null +++ b/test/unit_test/headless/test_webrtc_stats_poller.py @@ -0,0 +1,240 @@ +"""The arithmetic that decides the remote-desktop link is going bad. + +`StatsPoller` turns aiortc's cumulative `getStats()` counters into the per-sample +rates `AdaptiveBitrateController` acts on: bitrate, fps, packet loss, RTT and +jitter. Every one of those is a *delta* between two samples, which is where this +kind of code goes wrong -- a first sample with no predecessor, a counter that +resets when a track is replaced, a report that names no candidate pair. Get one +wrong and the controller drops the frame rate on a link that is fine, or holds a +high rate on one that is not. + +Nothing imported this module on any CI square before the `[webrtc]` extra joined +the measured install: it is reachable only through `webrtc_transport`, which +raises ImportError at module level without aiortc. + +The poller is driven here through `_sample()` on a bare event loop rather than +through `start()`, because `start()` submits to the shared asyncio bridge and +what is under test is the arithmetic, not the bridge. +""" +import asyncio + +import pytest + +from je_auto_control.utils.remote_desktop import webrtc_stats +from je_auto_control.utils.remote_desktop.webrtc_stats import ( + StatsPoller, StatsSnapshot, +) + + +class _Entry: + """One `RTCStats` row: aiortc exposes these as attributes.""" + + def __init__(self, **fields): + self.__dict__.update(fields) + + +class _PeerConnection: + """Hands back one prepared report per `getStats()` call.""" + + def __init__(self, *reports): + self._reports = list(reports) + self.calls = 0 + + async def getStats(self): # noqa: N802 # reason: the aiortc name + self.calls += 1 + report = self._reports.pop(0) if self._reports else {} + return {str(index): entry for index, entry in enumerate(report)} + + +def _inbound(**fields): + return _Entry(type="inbound-rtp", kind="video", **fields) + + +class _Clock: + """A monotonic clock the test drives. + + Replaces the module's own ``time`` binding rather than ``time.monotonic`` + itself: asyncio reads the real clock while these coroutines run, and a + fake installed globally makes the event loop read the fake too. + """ + + def __init__(self, *ticks): + self._ticks = list(ticks) + + def monotonic(self): + return self._ticks.pop(0) if len(self._ticks) > 1 else self._ticks[0] + + +def _pin_clock(monkeypatch, *ticks): + monkeypatch.setattr(webrtc_stats, "time", _Clock(*ticks)) + + +def _sample(poller): + """Take one sample.""" + return asyncio.run(poller._sample()) + + +# === A poller with nothing to poll ========================================== + +def test_a_poller_without_a_peer_connection_samples_nothing(): + assert asyncio.run(StatsPoller(None, lambda snap: None)._sample()) is None + + +def test_an_empty_report_yields_an_empty_snapshot(): + poller = StatsPoller(_PeerConnection([]), lambda snap: None) + snapshot = _sample(poller) + assert snapshot.to_dict() == StatsSnapshot().to_dict() + + +def test_the_poll_interval_has_a_floor(): + """A caller asking for a 1 ms poll would spin the loop, not measure it.""" + assert StatsPoller(None, lambda snap: None, interval_s=0.001)._interval == 0.25 + assert StatsPoller(None, lambda snap: None, interval_s=5)._interval == 5.0 + + +# === Rates need two samples ================================================= + +def test_the_first_sample_reports_no_rate(): + """There is nothing to subtract from yet; a rate would be invented.""" + poller = StatsPoller( + _PeerConnection([_inbound(bytesReceived=10_000, framesDecoded=30)]), + lambda snap: None) + snapshot = _sample(poller) + assert snapshot.bitrate_kbps is None + assert snapshot.fps is None + + +def test_the_second_sample_reports_the_rate_between_them(monkeypatch): + reports = [[_inbound(bytesReceived=0, framesDecoded=0)], + [_inbound(bytesReceived=125_000, framesDecoded=60)]] + poller = StatsPoller(_PeerConnection(*reports), lambda snap: None) + _pin_clock(monkeypatch, 100.0, 102.0) + _sample(poller) + snapshot = _sample(poller) + # 125,000 bytes over 2 s = 500 kbit/s; 60 frames over 2 s = 30 fps. + assert snapshot.bitrate_kbps == pytest.approx(500.0) + assert snapshot.fps == pytest.approx(30.0) + + +def test_a_counter_that_went_backwards_reports_no_rate(monkeypatch): + """A replaced track restarts the counters; a negative delta is not a rate.""" + reports = [[_inbound(bytesReceived=125_000, framesDecoded=60)], + [_inbound(bytesReceived=10, framesDecoded=1)]] + poller = StatsPoller(_PeerConnection(*reports), lambda snap: None) + _pin_clock(monkeypatch, 100.0, 102.0) + _sample(poller) + snapshot = _sample(poller) + assert snapshot.bitrate_kbps is None + assert snapshot.fps is None + + +def test_two_samples_at_the_same_instant_report_no_rate(monkeypatch): + """Dividing by a zero interval is the other way to invent a number.""" + reports = [[_inbound(bytesReceived=0)], [_inbound(bytesReceived=125_000)]] + poller = StatsPoller(_PeerConnection(*reports), lambda snap: None) + _pin_clock(monkeypatch, 100.0) + _sample(poller) + assert _sample(poller).bitrate_kbps is None + + +# === Packet loss is measured over the interval, not since the connection ==== + +def test_loss_is_the_recent_ratio_not_the_lifetime_one(monkeypatch): + """A link that lost 50 packets an hour ago is not a link losing them now.""" + reports = [[_inbound(packetsReceived=1_000, packetsLost=50)], + [_inbound(packetsReceived=1_100, packetsLost=50)]] + poller = StatsPoller(_PeerConnection(*reports), lambda snap: None) + _pin_clock(monkeypatch, 100.0, 101.0) + _sample(poller) + assert _sample(poller).packet_loss_pct == pytest.approx(0.0) + + +def test_loss_in_the_last_interval_is_reported(monkeypatch): + reports = [[_inbound(packetsReceived=1_000, packetsLost=0)], + [_inbound(packetsReceived=1_090, packetsLost=10)]] + poller = StatsPoller(_PeerConnection(*reports), lambda snap: None) + _pin_clock(monkeypatch, 100.0, 101.0) + _sample(poller) + # 10 lost out of 100 sent in the interval. + assert _sample(poller).packet_loss_pct == pytest.approx(10.0) + + +def test_a_report_with_no_packets_at_all_reports_no_loss(): + poller = StatsPoller( + _PeerConnection([_inbound(packetsReceived=0, packetsLost=0)]), + lambda snap: None) + assert _sample(poller).packet_loss_pct is None + + +def test_missing_counters_read_as_zero_rather_than_raising(): + """aiortc omits attributes it has no value for.""" + poller = StatsPoller(_PeerConnection([_inbound()]), lambda snap: None) + assert _sample(poller).packet_loss_pct is None + + +def test_a_null_counter_reads_as_zero(): + poller = StatsPoller( + _PeerConnection([_inbound(packetsReceived=None, packetsLost=None)]), + lambda snap: None) + assert _sample(poller).packet_loss_pct is None + + +# === Where RTT and jitter come from ========================================= + +def test_the_remote_report_supplies_rtt_and_jitter_in_milliseconds(): + entry = _Entry(type="remote-inbound-rtp", roundTripTime=0.125, jitter=0.004) + poller = StatsPoller(_PeerConnection([entry]), lambda snap: None) + snapshot = _sample(poller) + assert snapshot.rtt_ms == pytest.approx(125.0) + assert snapshot.jitter_ms == pytest.approx(4.0) + + +def test_the_candidate_pair_supplies_rtt_when_the_remote_report_does_not(): + entry = _Entry(type="candidate-pair", currentRoundTripTime=0.05) + poller = StatsPoller(_PeerConnection([entry]), lambda snap: None) + assert _sample(poller).rtt_ms == pytest.approx(50.0) + + +def test_the_remote_report_wins_over_the_candidate_pair(): + """Both are in the report; the one measured end to end is the real one.""" + report = [_Entry(type="remote-inbound-rtp", roundTripTime=0.125), + _Entry(type="candidate-pair", currentRoundTripTime=0.05)] + poller = StatsPoller(_PeerConnection(report), lambda snap: None) + assert _sample(poller).rtt_ms == pytest.approx(125.0) + + +def test_a_candidate_pair_with_no_measurement_is_ignored(): + entry = _Entry(type="candidate-pair", currentRoundTripTime=None) + poller = StatsPoller(_PeerConnection([entry]), lambda snap: None) + assert _sample(poller).rtt_ms is None + + +def test_audio_inbound_rows_do_not_drive_the_video_rate(monkeypatch): + """`kind` is what separates the two; only video feeds the controller.""" + audio = _Entry(type="inbound-rtp", kind="audio", + bytesReceived=1_000_000, framesDecoded=0) + poller = StatsPoller(_PeerConnection([audio], [audio]), lambda snap: None) + _pin_clock(monkeypatch, 100.0, 101.0) + _sample(poller) + assert _sample(poller).bitrate_kbps is None + + +def test_an_unknown_stat_type_is_ignored(): + poller = StatsPoller(_PeerConnection([_Entry(type="transport")]), + lambda snap: None) + assert _sample(poller).to_dict() == StatsSnapshot().to_dict() + + +def test_a_row_with_no_type_at_all_is_ignored(): + poller = StatsPoller(_PeerConnection([_Entry(bytesReceived=5)]), + lambda snap: None) + assert _sample(poller).to_dict() == StatsSnapshot().to_dict() + + +# === Stopping ================================================================ + +def test_stopping_a_poller_that_never_started_is_harmless(): + poller = StatsPoller(_PeerConnection(), lambda snap: None) + poller.stop() + assert poller._stopped is True + assert poller._task is None From 9cfbebf64e743bb9e8dc47c747989f2125ee3f11 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 00:04:09 +0800 Subject: [PATCH 07/20] Record the WebRTC decision, the new floor, and where the last five points are MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Progress.md's open decision — install the extra in CI, or write those 2,090 statements off as dead weight in the denominator — is closed with what it was waiting on measured, and replaced by the roadmap the decision unblocks: which files are still dark on *every* square, which of those can move the floor (only the portable ones do), and which two must not be touched until CI installs their extra, because tests written there would skip and move nothing. It also corrects a trap of its own making. The entry said the floor "had to be dug out of the XML artifact"; `coverage report` is what enforces `fail_under` and it includes branches, while Cobertura's `line-rate` does not — about 1.8 points apart on the same square. A floor set from the artifact is one the suite cannot clear. --- Progress.md | 120 ++++++++++++++++++++++++++++++++++++++++----------- WHATS_NEW.md | 102 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 198 insertions(+), 24 deletions(-) diff --git a/Progress.md b/Progress.md index 95143ddf..c0eb4f1a 100644 --- a/Progress.md +++ b/Progress.md @@ -228,12 +228,12 @@ capability enum 值與 variadic `ei_seat_bind_capabilities`、event-type enum 2026-08-21 把**機制**補上了（做法見 [WHATS_NEW.md](WHATS_NEW.md)）： 型別契約的豁免清單 2026-08-22 清空，平台縫最後兩個名稱（`keyboard`／`mouse`） 2026-08-23 拿到合約；覆蓋率那半發現不是爬得不夠，是量測起點錯了，修正後地板 -從 50 提到 69。 +從 50 提到 69，同日再提到 74。 **2026-08-23 維護者拍板：下一個覆蓋率目標是 80。** 這一條留到那時候。 型別那一節留著是因為它記的那幾個坑之後還會踩到。 -### 覆蓋率：地板 69，下一站 80 +### 覆蓋率：地板 74，下一站 80 `TODO` — 目標 **80**（2026-08-23 拍板）。地板本身照舊只跟著實測的最低那一格走。 @@ -270,10 +270,17 @@ import 期程式碼（`def` 行、類別本體、常數、兩張大分派表） 地板因此設成 **69**——取最低那一格往下取整，與當初 50 取自 50.26% 是同一個慣例。 `[tool.coverage.report]` 的 `precision` 也從預設的 0 提到 2：預設精度下九格全部印 -「70%」，而它們其實是 69.67 到 70.97，害得這次的地板得去 XML artifact 裡撈。 +「70%」，而它們其實是 69.67 到 70.97，分不出最低的是哪一格。 順帶把 `fail_under` 的容差從一整個百分點縮到 0.01。 地板只有一個家（`pyproject.toml` 的 `fail_under`），`quality.yml` 不再另外抄一份。 +**這一段原本寫「地板得去 XML artifact 裡撈」，那是錯的，2026-08-23 實測更正。** +上面那九個數字是 `coverage report` 印的，也就是 `fail_under` 真正比對的那個數字， +而它**含分支**（`[tool.coverage.run]` 的 `branch = true`）；`coverage.xml` 的 +`line-rate` 屬性**不含分支**，所以同一格會高出約 1.8 點——a585e65 那一格 +report 印 69.67%，artifact 是 71.07%。照 artifact 設地板，設出來的會是這套測試 +過不了的地板。要看單一子系統的缺口用 artifact，要設地板只能用 report。 + Windows 是高的那一角，因為門面 import 進來的是**它自己那個平台的後端**； 換句話說剩下的那 30 點裡，有一部分是任何單一平台都拿不到的。 @@ -307,32 +314,97 @@ MCP adapter 與 372 個執行器 adapter 可以在沒有滑鼠、沒有螢幕、 機器）。三個 MCP adapter 需要的比合約承諾的更多，用名字列在測試檔裡，各附一行 理由；**其中任何一個哪天開始通過，測試會要求把它刪掉**，清單不會爛在那裡。 -#### 剩下的在哪裡（2026-08-23 實測，本機 Windows／3.14） +#### 2026-08-23 拍板：`quality.yml` 裝 `[webrtc]` extra + +原本這裡是一條 `DECIDE`，寫的是「要嘛讓 CI 裝那個 extra，要嘛承認那 2,900 個 +statement 是分母裡的死重」。**維護者選了裝**，理由不是那一點覆蓋率： + +`utils/remote_desktop` 底下有 **11 個模組在沒有 `aiortc`／`av` 時於模組層拋 +ImportError**，共 2,090 個 statement——在 CI 上是硬性的 0%，寫什麼測試都動不了。 +比數字嚴重的是另一件事：**涵蓋 WebRTC host 的 auth、TLS、resume token、檔案傳輸的 +測試早就寫好了**，只是每一格都 `importorskip` 略過，等於只在開發機上跑過。 + +只換這一個變數實測（同一台機器、同一套測試）：那 2,090 個裡有 **513 個**是現有測試 +就會蓋到的，端到端 +1.23 點（windows-2022／3.14）。相依在九宮格上都解得開 +（`aiortc` 1.15.0 + `av` 17.1.0 對 win_amd64／manylinux x86_64／macos-14 arm64 的 +3.10 與 3.14 都有 wheel）。`typing-stable-api` **刻意不裝**——那個閘門的判定不能隨 +環境浮動，理由在下一節。 + +`dev_requirements.txt` 與 `CLAUDE.md` 的開發指令也跟著加了那個 extra：少裝它的人 +量到的數字會比 CI 執行的地板低約 4 點。 + +#### 三個註冊表都掃過了，替身再往合約深一層 + +`test_adapter_registry_sweep.py` 之後又走了三步（做法見 [WHATS_NEW.md](WHATS_NEW.md)）： + +| 這次多掃到的 | 為什麼原本掃不到 | +| --- | --- | +| 92 個 adapter：被呼叫者回傳專案 dataclass | `_value_for` 只認純量與容器，dataclass 直接被判成「模不出來」。現在從 dataclass **自己的欄位標注**造實例，於是 adapter 的 `.to_dict()` 也一起跑起來 | +| 101 個 adapter：被呼叫者是模組層單例的方法 | 匯入的名字不是 function 而是 `default_observer`／`default_scheduler`／`registry` 這種物件。呼叫**單一**方法就是同一個轉接形狀往下一層，方法的標注就是合約；呼叫兩個以上算編排，仍然排除 | +| 31 條 REST 路由 | `rest_handlers` 是同一形狀的第三個註冊表，而現有的 REST 測試走 HTTP 層，在無頭 runner 上多半只是看著 handler 掉進自己的 `except` 回 500 | + +REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handler 不同檔， +所以掃不出「拿被測程式當答案」的循環；順帶白拿一條契約測試：**路由表與 OpenAPI +文件必須描述同一組 API**，多一條少一條都當場紅。 -兩支 sweep 之後的數字（`utils/executor` 與 `utils/mcp_server` 已經含進去了）： +三支共用的機器搬進 `test/unit_test/headless/_contract_sweep.py`，各自只留自己的 +參數來源。 + +#### 現在的九宮格（2026-08-23 實測，本 PR 的 run） + +| | 最低 | 最高 | +| --- | --- | --- | +| 上一版（PR #486 首輪） | 69.67%（ubuntu-22.04／3.14） | 70.97%（windows-2022／3.12） | +| 這一版 | **74.99%**（ubuntu-22.04／3.14） | 76.19%（windows-2022，3.10–3.13） | + +地板隨之從 69 提到 **74**。 + +#### 剩下的 5 點在哪裡（2026-08-23，ubuntu-22.04／3.14 的 artifact） + +以下是 statement 數（artifact 的口徑，不含分支），拿來看缺口分佈： | 子系統 | 沒蓋到 / 總 statement | 覆蓋率 | | --- | ---: | ---: | -| `utils/remote_desktop` | 2,923 / 6,622 | 55.9% | -| `utils/accessibility` | 822 / 1,397 | 41.2% | -| `utils/executor` | 797 / 3,906 | 79.6% | -| `utils/mcp_server` | 761 / 4,618 | 83.5% | -| `utils/usb` | 455 / 2,137 | 78.7% | -| `wrapper/window_backends` | 397 / 477 | 16.8% | +| `utils/remote_desktop` | 2,653 / 6,622 | 59.9% | +| `utils/accessibility` | 824 / 1,397 | 41.0% | +| `utils/executor` | 653 / 3,906 | 83.3% | +| `utils/usb` | 573 / 2,137 | 73.2% | +| `utils/mcp_server` | 573 / 4,618 | 87.6% | +| `wrapper/window_backends` | 361 / 477 | 24.3% | | `utils/hotkey` | 221 / 426 | 48.1% | -| `utils/rest_api` | 220 / 808 | 72.8% | - -**一件之前沒量過、但決定 80 到底可不可能的事**：其他平台的程式碼 -（X11／Wayland／macOS／Android／iOS）在這一格是 3,291 個 statement， -其中 1,854 個沒被蓋到。就算那些永遠蓋不到，這一格的上限仍有 96%， -所以**80 不是被「單一平台拿不到」卡住的**——純粹是量的問題： -可攜程式碼還有 9,539 個 statement 沒被執行過。 - -**最大的那一塊有個前提要先決定**：`utils/remote_desktop` 佔了將近三分之一， -但 `quality.yml` 不裝 `[webrtc]` extra，所以 webrtc 那一族在 CI 上**永遠是 0%**， -在那裡補的測試會整批 skip，對地板一個點也不會動。要嘛先讓 CI 裝那個 extra， -要嘛承認那 2,900 個 statement 是分母裡的死重、把 80 算在其餘部分上。 -**兩條路都可以，但得先選一條再動手。** +| `utils/rest_api` | 146 / 808 | 81.9% | + +**要動地板，只能補在每一格都跑得到的程式碼上。** 地板取的是最低那一格， +所以只在 Windows 跑得到的東西補再多也不會動它。把九格的未覆蓋行取交集， +2026-08-23 量到 **9,697 個 statement 在每一格都沒被執行過**——那就是可攜的缺口， +也是唯一會抬地板的地方。最大的幾塊： + +| 檔案 | 每一格都沒蓋到 | 備註 | +| --- | ---: | --- | +| `utils/executor/action_executor.py` | 589 | 掃不到的那批 adapter：被呼叫者是 class（沒有回傳標注可讀）、或 adapter 伸手進兩個模組 | +| `utils/accessibility/backends/windows_backend.py` | 446 | UIA COM，要一套夠像的替身 | +| `utils/remote_desktop/webrtc_viewer.py` | 354 | 現在 import 得到了（17.9%），但沒有測試驅動 | +| `utils/remote_desktop/webrtc_host.py` | 351 | 同上（24.8%） | +| `utils/mcp_server/tools/_handlers.py` | 348 | 同 `action_executor.py` | +| `utils/remote_desktop/signaling_server.py` | 155 | **CI 動不了**：要 `[signaling]` extra（fastapi／uvicorn），沒裝 | +| `utils/remote_desktop/multi_viewer.py` | 146 | | +| `wrapper/window_backends/x11_backend.py` | 133 | 只有 Linux 那兩格跑得到，抬得動地板 | +| `utils/accessibility/backends/linux_backend.py` | 132 | 同上 | + +**下一步的順序**（都還沒做）： + +1. `webrtc_host`／`webrtc_viewer`／`multi_viewer`／`webrtc_files`：`[webrtc]` 裝了 + 之後這一族才第一次可測，合計 960 個 statement 在每一格都沒跑過。`webrtc_host_auth` + 與 `webrtc_stats` 已經照這個做法補完（56 個測試），其餘照抄。 +2. 掃不到的那批 adapter：目前卡在「被呼叫者是 class」。要嘛從 class 自己的方法標注 + 長出一個替身物件，要嘛承認那批不掃——**得先決定，因為前者會讓「跑起來了」和 + 「驗到了東西」分家**。 +3. `utils/accessibility` 與 `wrapper/window_backends` 的 Linux 後端：抬得動地板， + 但需要一套 Xlib／AT-SPI 的替身。 + +`utils/office`（77）與 `signaling_server`（155）**不要碰**：`quality.yml` 沒裝 +`[office]`／`[signaling]`，補的測試會整批 skip，對地板一個點都不動。要補之前 +先照 `[webrtc]` 的先例把 extra 加進 CI。 ### mypy：整包把關，**豁免清單已經清空** diff --git a/WHATS_NEW.md b/WHATS_NEW.md index fc21eea9..16eae787 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,107 @@ # What's New — AutoControl +## What's new (2026-08-24) + +### A Whole Subsystem Was Being Measured At Zero, And Its Tests Were Skipping + +`quality.yml` now installs the `[webrtc]` extra. The coverage number was the +smaller half of the reason. Eleven modules under `utils/remote_desktop` raise +`ImportError` at module level without `aiortc`/`av` — 2,090 statements that were +a hard 0% on every square no matter what anyone wrote — but the part that +mattered is that the tests covering the WebRTC host's auth, TLS, resume tokens +and file transfer *were already written*. They `importorskip`ped straight past on +all nine squares, so they ran on developer machines and nowhere else. + +Measured with one variable changed, same machine and same suite: 513 of those +statements are covered by tests that exist today, worth +1.23 points end to end. +The extra resolves on every square (`aiortc` 1.15.0 and `av` 17.1.0 have wheels +for win_amd64, manylinux x86_64 and macos-14 arm64 on both ends of the supported +Python range). `typing-stable-api` deliberately does not get it: that gate's +verdict must not depend on what happens to be installed. + +`dev_requirements.txt` and the `CLAUDE.md` setup line carry it too, because a +developer without it measures about 4 points below the floor CI enforces. + +### The Stub Grows One Level Deeper, And A Third Registry Gets Swept + +Yesterday's sweep replaced each adapter's callee with a value built from its +return annotation. Three things it could not reach, and now does: + +**A dataclass return is not the end of the contract, it is one more level of +it.** `Optional[X]` → None and containers → empty containers stopped at the +first `-> HealOutcome`, so 92 adapters sat out. The stub is now an instance +built from the dataclass's own field annotations, which means the adapter's +`.to_dict()` runs too — against the declared shape rather than a mock that +answers everything. + +**Some adapters import a singleton, not a function.** `default_observer`, +`default_scheduler`, `registry` — the adapter calls one method on the object. +That is the same wiring shape one indirection along, so the callee is the method +and its annotation is the contract; 101 more adapters. Calling *two* methods +means the adapter orchestrates the object rather than standing in front of one +call, and those stay out. + +**The REST route table is the third registry of this shape.** Its handlers' +own module docstring says they are "pure … trivial to unit-test without an HTTP +layer", and the existing REST tests go through the HTTP layer instead — so on a +headless runner most of them reached a handler only to watch it fall into its +own `except` and answer 500. `test_rest_route_sweep.py` calls all 31 routes with +the arguments their own OpenAPI document declares, and asserts what the +dispatcher relies on: `(status, dict)`, a real HTTP code, a payload `json.dumps` +accepts — for a documented request, for the empty one an unhelpful client sends, +and for a body of the wrong shape. It also compares the route table against the +document, so a route nobody describes or a documented route nobody serves is +now a named failure. + +The machinery the three sweeps share moved to `test/unit_test/headless/ +_contract_sweep.py`. Each keeps its own argument source — a JSON schema, the +Script Builder's field specs, an OpenAPI document — which is what stops a sweep +from passing by restating the code it checks. + +Two things fell out of building it. `ac_rrule_next` and its neighbours now +declare `"format": "date-time"` on the properties they parse, which their +descriptions already said in prose and their schemas did not; a client +generating values from the schema alone used to get a `ValueError` out of +`datetime.fromisoformat`. And `AddressBook.set_tags()` cleaned its input with +`str(t).strip()`, so a JSON `null` — what a client sends for an omitted tag — +became a tag literally named `"None"`, which `all_tags()` then listed next to +the real ones. + +### The Trust Store And The Auth Boundary Get Tests, Because Now They Can + +Four modules decide who may drive this machine unattended, whether the host +answering is the one that answered last time, what the viewer reconnects to, and +how hard the encoder is pushed when the link degrades. All are ordinary Python — +a JSON file, a lock and some arithmetic — and none was imported by any test on +any square, because the subsystem could not be loaded without the extra. + +117 tests over the decisions they make in the operator's absence: a trust entry +that must not lose its label on re-add, a store that opens empty rather than +throwing on a truncated file, a fingerprint comparison that survives an SDP +spelling the same certificate in a different case, a token accepted only when it +is an equal string, an IP whitelist that matches by network rather than by +string, a grace period that closes a peer which never authenticated, and five +derived rates that must refuse to invent a number from one sample, a zero +interval, or a counter that went backwards. + +The auth host double supplies exactly the attribute list `ViewerAuthMixin`'s own +docstring asks for, so a mixin that starts reaching for something else fails +there rather than leaning on whatever the real host happens to own. + +### The Floor Is 74, And The Comment Now Says Which Number That Is + +The nine-way matrix runs 74.99% (ubuntu-22.04 / 3.14) to 76.19% (windows-2022), +up from 69.67–70.97, so `fail_under` goes 69 → 74 on the usual convention: floor +of the lowest square. + +`Progress.md` had recorded that the floor "had to be dug out of the XML +artifact". That is wrong, and following it would set a floor the suite cannot +clear. `coverage report` — the step that enforces `fail_under` — includes branch +coverage, because `branch = true`; Cobertura's `line-rate` attribute does not. +On the same square and the same run those differ by about 1.8 points (76.14% +against 78.33%). The artifact is the right thing to read for a single +subsystem's gap and the wrong thing to set a floor from. + ## What's new (2026-08-23) ### A Thousand Adapters Now Get Called, Because the Type Contract Can Build Their Stubs From 5ba623c13246905435d882384936ac8db02a60bb Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 00:14:38 +0800 Subject: [PATCH 08/20] Test the file channel and the media renegotiation the extra just unlocked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `webrtc_files` is a filesystem write driven by remote input — the peer names the file — and its own docstring states the defence: "incoming filenames are stripped of any directory components to defeat path traversal". Nothing tested it. 39 tests do now, including the round trip, where the sender's output is fed straight into the receiver: neither half's view of the protocol is asserted from the other's code. `webrtc_host_media` exists because aiortc has no `removeTransceiver`, so enabling and disabling a viewer stream are not symmetric and each reaches for its slot *by position* — the first video transceiver is the host's own outbound screen track, so the viewer's is the second. 24 tests, one of which exists purely to fail if that off-by-one ever mutes the screen share. Local: 77.04% -> the run before this was 76.60%. --- .../headless/test_webrtc_file_transfer.py | 329 +++++++++++++++++ .../headless/test_webrtc_host_media.py | 347 ++++++++++++++++++ 2 files changed, 676 insertions(+) create mode 100644 test/unit_test/headless/test_webrtc_file_transfer.py create mode 100644 test/unit_test/headless/test_webrtc_host_media.py diff --git a/test/unit_test/headless/test_webrtc_file_transfer.py b/test/unit_test/headless/test_webrtc_file_transfer.py new file mode 100644 index 00000000..911e036b --- /dev/null +++ b/test/unit_test/headless/test_webrtc_file_transfer.py @@ -0,0 +1,329 @@ +"""The file channel: what a remote peer can and cannot write to this disk. + +`webrtc_files` reassembles a viewer's upload into a file under the host's inbox. +The name in the envelope is chosen by the peer, so this module is a filesystem +write driven by remote input, and its docstring says the defence out loud: +"incoming filenames are stripped of any directory components to defeat path +traversal". Nothing tested it -- the module reaches `webrtc_transport`, which +raises ImportError at module level without aiortc, so it could not be imported +on any CI square until the `[webrtc]` extra joined the measured install. + +Both halves are exercised here, and then wired to each other: the sender's +output is fed straight into the receiver, which is the only way to check that +the two agree about the protocol their shared docstring describes. +""" +import json + +import pytest + +from je_auto_control.utils.remote_desktop import webrtc_files +from je_auto_control.utils.remote_desktop.webrtc_files import ( + FileTransferError, FileTransferReceiver, FileTransferSender, +) + + +class _Bridge: + """`call_soon` on the real bridge runs the call on the loop thread.""" + + def call_soon(self, callback, *args): + callback(*args) + + +class _Channel: + """A DataChannel that keeps what was sent.""" + + def __init__(self): + self.messages = [] + + def send(self, message): + self.messages.append(message) + + +@pytest.fixture +def bridge(monkeypatch): + monkeypatch.setattr(webrtc_files, "get_bridge", _Bridge) + + +def _begin(name, size, transfer_id="t1"): + return json.dumps({"type": "file_begin", "name": name, "size": size, + "transfer_id": transfer_id}) + + +def _end(transfer_id="t1"): + return json.dumps({"type": "file_end", "transfer_id": transfer_id}) + + +def _receiver(tmp_path): + return FileTransferReceiver(tmp_path / "inbox") + + +# === What the peer may name the file ======================================== + +@pytest.mark.parametrize("name, written_as", [ + ("report.txt", "report.txt"), + ("../../../etc/passwd", "passwd"), + ("a/b/c/nested.bin", "nested.bin"), + ("./visible.txt", "visible.txt"), + ("trailing/", "trailing"), +]) +def test_a_directory_in_the_name_never_escapes_the_inbox(bridge, tmp_path, + name, written_as): + receiver = _receiver(tmp_path) + receiver.handle_message(_begin(name, 2)) + receiver.handle_message(b"hi") + receiver.handle_message(_end()) + inbox = tmp_path / "inbox" + assert [entry.name for entry in inbox.iterdir()] == [written_as] + assert (inbox / written_as).read_bytes() == b"hi" + + +@pytest.mark.parametrize("name", ["", ".", "..", "bad\x00name", + "pipe|name", "star*name", "quest?name", + "ltname", 'quote"name']) +def test_a_name_that_cannot_be_made_safe_is_refused(bridge, tmp_path, name): + receiver = _receiver(tmp_path) + errors = [] + receiver.handle_message(_begin(name, 2), on_error=errors.append) + assert errors, f"{name!r} was accepted" + assert list((tmp_path / "inbox").iterdir()) == [] + + +def test_a_non_string_name_is_refused(bridge, tmp_path): + receiver = _receiver(tmp_path) + errors = [] + receiver.handle_message( + json.dumps({"type": "file_begin", "name": None, "size": 1}), + on_error=errors.append) + assert errors + + +@pytest.mark.parametrize("size", [-1, 5 * 1024 * 1024 * 1024]) +def test_an_impossible_size_is_refused_before_any_file_is_opened( + bridge, tmp_path, size): + receiver = _receiver(tmp_path) + errors = [] + receiver.handle_message(_begin("big.bin", size), on_error=errors.append) + assert errors + assert list((tmp_path / "inbox").iterdir()) == [] + + +# === The protocol =========================================================== + +def test_a_transfer_arrives_in_pieces_and_reports_progress(bridge, tmp_path): + receiver = _receiver(tmp_path) + progress, done = [], [] + receiver.handle_message(_begin("split.bin", 6)) + receiver.handle_message(b"abc", on_progress=lambda n, total: + progress.append((n, total))) + receiver.handle_message(b"def", on_progress=lambda n, total: + progress.append((n, total))) + receiver.handle_message(_end(), on_done=done.append) + assert progress == [(3, 6), (6, 6)] + assert done and done[0].read_bytes() == b"abcdef" + + +@pytest.mark.parametrize("chunk", [b"raw", bytearray(b"raw"), + memoryview(b"raw")]) +def test_a_chunk_arrives_however_the_transport_spells_bytes(bridge, tmp_path, + chunk): + receiver = _receiver(tmp_path) + receiver.handle_message(_begin("x.bin", 3)) + receiver.handle_message(chunk) + receiver.handle_message(_end()) + assert (tmp_path / "inbox" / "x.bin").read_bytes() == b"raw" + + +def test_a_chunk_with_no_transfer_open_is_dropped(bridge, tmp_path): + """A late chunk from an aborted transfer must not create a file.""" + receiver = _receiver(tmp_path) + receiver.handle_message(b"orphan") + assert list((tmp_path / "inbox").iterdir()) == [] + + +def test_an_end_with_no_transfer_open_is_ignored(bridge, tmp_path): + receiver = _receiver(tmp_path) + done = [] + receiver.handle_message(_end(), on_done=done.append) + assert done == [] + + +def test_a_second_begin_is_refused_and_takes_the_first_with_it(bridge, + tmp_path): + """One transfer per channel: the module says so, and it fails closed.""" + receiver = _receiver(tmp_path) + errors = [] + receiver.handle_message(_begin("first.bin", 10)) + receiver.handle_message(b"partial") + receiver.handle_message(_begin("second.bin", 10), on_error=errors.append) + assert errors == ["transfer already in progress"] + # The partial first file is removed rather than left as a truncated one. + assert list((tmp_path / "inbox").iterdir()) == [] + + +def test_an_abort_from_the_sender_removes_the_partial_file(bridge, tmp_path): + receiver = _receiver(tmp_path) + errors = [] + receiver.handle_message(_begin("partial.bin", 100)) + receiver.handle_message(b"half") + receiver.handle_message(json.dumps({"type": "file_abort"}), + on_error=errors.append) + assert errors == ["aborted by sender"] + assert list((tmp_path / "inbox").iterdir()) == [] + + +def test_an_abort_with_nothing_in_flight_is_harmless(bridge, tmp_path): + receiver = _receiver(tmp_path) + receiver.handle_message(json.dumps({"type": "file_abort"})) + assert list((tmp_path / "inbox").iterdir()) == [] + + +def test_a_malformed_envelope_is_reported_not_raised(bridge, tmp_path): + receiver = _receiver(tmp_path) + errors = [] + receiver.handle_message("{not json", on_error=errors.append) + assert errors and "bad envelope" in errors[0] + + +def test_an_unknown_envelope_type_is_ignored(bridge, tmp_path): + receiver = _receiver(tmp_path) + receiver.handle_message(json.dumps({"type": "file_pause"})) + assert list((tmp_path / "inbox").iterdir()) == [] + + +def test_a_message_that_is_neither_text_nor_bytes_is_ignored(bridge, tmp_path): + receiver = _receiver(tmp_path) + receiver.handle_message(42) + assert list((tmp_path / "inbox").iterdir()) == [] + + +def test_the_inbox_is_created_on_construction(bridge, tmp_path): + inbox = tmp_path / "deep" / "inbox" + FileTransferReceiver(inbox) + assert inbox.is_dir() + + +# === The sending half ======================================================= + +def test_a_sender_needs_a_channel(): + with pytest.raises(ValueError): + FileTransferSender(None) + + +def test_sending_a_path_that_is_not_a_file_is_refused(bridge, tmp_path): + sender = FileTransferSender(_Channel()) + with pytest.raises(FileTransferError, match="not a file"): + sender.send(tmp_path / "missing.txt") + with pytest.raises(FileTransferError, match="not a file"): + sender.send(tmp_path) + + +def test_a_send_is_a_begin_then_chunks_then_an_end(bridge, tmp_path): + source = tmp_path / "payload.bin" + source.write_bytes(b"0123456789") + channel = _Channel() + progress = [] + FileTransferSender(channel).send(source, chunk_size=4, + on_progress=lambda n, total: + progress.append((n, total))) + begin = json.loads(channel.messages[0]) + assert begin["type"] == "file_begin" + assert begin["name"] == "payload.bin" + assert begin["size"] == 10 + assert channel.messages[1:-1] == [b"0123", b"4567", b"89"] + end = json.loads(channel.messages[-1]) + assert end["type"] == "file_end" + assert end["transfer_id"] == begin["transfer_id"] + assert progress == [(4, 10), (8, 10), (10, 10)] + + +def test_an_empty_file_is_a_begin_and_an_end_with_no_chunks(bridge, tmp_path): + source = tmp_path / "empty.bin" + source.write_bytes(b"") + channel = _Channel() + FileTransferSender(channel).send(source) + assert [json.loads(message)["type"] for message in channel.messages] == [ + "file_begin", "file_end"] + + +def test_the_remote_name_is_sanitised_before_it_leaves(bridge, tmp_path): + """The sender is the other side's remote peer; it does not get to say + where its file lands either.""" + source = tmp_path / "payload.bin" + source.write_bytes(b"x") + channel = _Channel() + FileTransferSender(channel).send(source, remote_name="a/b/../evil.txt") + assert json.loads(channel.messages[0])["name"] == "evil.txt" + + +def test_a_read_failure_mid_transfer_aborts_the_receiver_too(bridge, tmp_path): + source = tmp_path / "payload.bin" + source.write_bytes(b"0123456789") + + class _Failing: + def __init__(self, real): + self._real = real + self._reads = 0 + + def read(self, size): + self._reads += 1 + if self._reads > 1: + raise OSError("device disappeared") + return self._real.read(size) + + def __enter__(self): + return self + + def __exit__(self, *exc): + self._real.close() + return False + + real_open = type(source).open + + def _open(self, *args, **kwargs): + handle = real_open(self, *args, **kwargs) + return _Failing(handle) if self == source else handle + + channel = _Channel() + monkey = pytest.MonkeyPatch() + monkey.setattr(type(source), "open", _open) + try: + with pytest.raises(FileTransferError, match="read failed"): + FileTransferSender(channel).send(source, chunk_size=4) + finally: + monkey.undo() + assert json.loads(channel.messages[-1])["type"] == "file_abort" + + +# === The two halves agree =================================================== + +def test_a_file_survives_a_round_trip_through_the_protocol(bridge, tmp_path): + """Neither side's view of the protocol is asserted from the other's code.""" + source = tmp_path / "round.bin" + source.write_bytes(bytes(range(256)) * 40) # 10,240 bytes + receiver = _receiver(tmp_path) + done = [] + + class _Wired: + def send(self, message): + receiver.handle_message(message, on_done=done.append) + + FileTransferSender(_Wired()).send(source, remote_name="landed.bin", + chunk_size=1024) + assert done, "the receiver never completed the transfer" + assert done[0].name == "landed.bin" + assert done[0].read_bytes() == source.read_bytes() + + +def test_an_aborted_send_leaves_nothing_behind_on_the_receiver(bridge, + tmp_path): + receiver = _receiver(tmp_path) + + class _Wired: + def send(self, message): + receiver.handle_message(message) + + channel = _Wired() + receiver.handle_message(_begin("stale.bin", 100)) + receiver.handle_message(b"partial") + channel.send(json.dumps({"type": "file_abort"})) + assert list((tmp_path / "inbox").iterdir()) == [] diff --git a/test/unit_test/headless/test_webrtc_host_media.py b/test/unit_test/headless/test_webrtc_host_media.py new file mode 100644 index 00000000..c91afc91 --- /dev/null +++ b/test/unit_test/headless/test_webrtc_host_media.py @@ -0,0 +1,347 @@ +"""Turning the viewer's camera and microphone on and off, asymmetrically. + +`MediaNegotiationMixin` exists because of one upstream fact its docstring +states: aiortc has no `removeTransceiver`. Enabling a viewer stream adds a +recvonly transceiver and re-offers; disabling can only mark the existing one +inactive and stop the receiver. The two directions therefore do different +things, and the slot each one reaches for is identified *by position* -- the +first video transceiver is the host's own outbound screen track, so the viewer's +is the second. Off-by-one there mutes the host's own screen share. + +Nothing imported this module on any CI square before the `[webrtc]` extra joined +the measured install; it reaches `webrtc_transport`, which raises ImportError at +module level without aiortc. + +`_pc`, the config and the spawn hook come from the host the mixin is mixed into, +and `_Host` supplies exactly the list the mixin's own docstring asks for. +""" +import asyncio + +import pytest + +from je_auto_control.utils.remote_desktop import webrtc_host_media as media +from je_auto_control.utils.remote_desktop.webrtc_host_media import ( + MediaNegotiationMixin, +) + + +class _Bridge: + def call_soon(self, callback): + callback() + + +class _Receiver: + def __init__(self, track=None): + self.track = track + + +class _Transceiver: + def __init__(self, kind, track=None, direction="sendrecv"): + self.kind = kind + self.receiver = _Receiver(track) if track is not None else None + self.direction = direction + + +class _PeerConnection: + def __init__(self, *transceivers): + self._transceivers = list(transceivers) + self.added = [] + + def getTransceivers(self): # noqa: N802 # reason: the aiortc name + return list(self._transceivers) + + def addTransceiver(self, kind, direction): # noqa: N802 # the aiortc name + self.added.append((kind, direction)) + self._transceivers.append(_Transceiver(kind, direction=direction)) + + +class _Config: + def __init__(self, *, accept_viewer_video=False, + accept_viewer_audio_opus=False): + self.accept_viewer_video = accept_viewer_video + self.accept_viewer_audio_opus = accept_viewer_audio_opus + + +class _Task: + def __init__(self): + self.cancelled = False + + def cancel(self): + self.cancelled = True + + +class _Host(MediaNegotiationMixin): + """A host with exactly the attributes the mixin's docstring asks for.""" + + def __init__(self, pc=None, config=None): + self._pc = pc + self._config = config or _Config() + self._viewer_video_task = None + self._opus_audio_receiver = None + self.sent = [] + self.spawned = [] + self.consumed = [] + self.opus_started = [] + + def _send_ctrl(self, message): + self.sent.append(message) + + def _spawn_bg(self, coroutine): + if asyncio.iscoroutine(coroutine): + coroutine.close() # never awaited here; do not leak the frame + self.spawned.append(coroutine) + return coroutine + + def _consume_viewer_video(self, track): + # The mixin calls this and hands the result to `_spawn_bg`, so the + # double records at call time and returns something awaitable — a + # coroutine body would not run until the task it never becomes. + self.consumed.append(track) + return self._noop() + + async def _noop(self): + return None + + def _start_opus_audio_receive(self, track): + self.opus_started.append(track) + + +@pytest.fixture(autouse=True) +def bridge(monkeypatch): + monkeypatch.setattr(media, "get_bridge", _Bridge) + + +# === Nothing happens without a peer connection ============================== + +@pytest.mark.parametrize("method", [ + "request_renegotiation", "enable_accept_viewer_video", + "enable_accept_viewer_audio_opus", "disable_accept_viewer_video", + "disable_accept_viewer_audio_opus", +]) +def test_every_entry_point_is_inert_before_a_connection_exists(method): + """These are GUI buttons; clicking one before connecting must not throw.""" + host = _Host(pc=None) + getattr(host, method)() + assert host.spawned == [] + assert host.sent == [] + + +# === Enabling adds capacity ================================================= + +def test_enabling_viewer_video_adds_a_recvonly_slot_and_renegotiates(): + pc = _PeerConnection(_Transceiver("video")) # the outbound screen + host = _Host(pc, _Config()) + host.enable_accept_viewer_video() + assert pc.added == [("video", "recvonly")] + assert host._config.accept_viewer_video is True + assert len(host.spawned) == 1 # the renegotiation + + +def test_enabling_twice_does_not_add_a_second_slot(): + """Two recvonly video slots would be two SDP m-lines nothing fills.""" + pc = _PeerConnection(_Transceiver("video")) + host = _Host(pc, _Config()) + host.enable_accept_viewer_video() + host.enable_accept_viewer_video() + assert pc.added == [("video", "recvonly")] + assert len(host.spawned) == 2 # but it re-offers again + + +def test_enabling_viewer_audio_adds_the_first_audio_slot(): + """There is no outbound audio slot to skip, so the first one is theirs.""" + pc = _PeerConnection(_Transceiver("video")) + host = _Host(pc, _Config()) + host.enable_accept_viewer_audio_opus() + assert pc.added == [("audio", "recvonly")] + assert host._config.accept_viewer_audio_opus is True + + +def test_enabling_audio_twice_does_not_add_a_second_slot(): + pc = _PeerConnection(_Transceiver("video")) + host = _Host(pc, _Config()) + host.enable_accept_viewer_audio_opus() + host.enable_accept_viewer_audio_opus() + assert pc.added == [("audio", "recvonly")] + + +# === Disabling can only deactivate ========================================== + +def test_disabling_viewer_video_leaves_the_hosts_own_track_alone(): + """The first video transceiver is the screen share; muting it is the bug + this test exists for.""" + outbound = _Transceiver("video") + inbound = _Transceiver("video") + host = _Host(_PeerConnection(outbound, inbound), + _Config(accept_viewer_video=True)) + host.disable_accept_viewer_video() + assert inbound.direction == "inactive" + assert outbound.direction == "sendrecv" + assert host._config.accept_viewer_video is False + + +def test_disabling_viewer_video_cancels_the_consume_task(): + task = _Task() + host = _Host(_PeerConnection(_Transceiver("video"), _Transceiver("video")), + _Config(accept_viewer_video=True)) + host._viewer_video_task = task + host.disable_accept_viewer_video() + assert task.cancelled is True + assert host._viewer_video_task is None + + +def test_disabling_viewer_video_with_no_inbound_slot_still_renegotiates(): + host = _Host(_PeerConnection(_Transceiver("video")), + _Config(accept_viewer_video=True)) + host.disable_accept_viewer_video() + assert len(host.spawned) == 1 + + +def test_a_transceiver_that_refuses_to_go_inactive_does_not_stop_the_rest(): + class _Stubborn: + kind = "video" + receiver = None + + @property + def direction(self): + return "sendrecv" + + @direction.setter + def direction(self, value): + raise RuntimeError("closed") + + task = _Task() + host = _Host(_PeerConnection(_Transceiver("video"), _Stubborn()), + _Config(accept_viewer_video=True)) + host._viewer_video_task = task + host.disable_accept_viewer_video() + assert task.cancelled is True + assert len(host.spawned) == 1 + + +def test_disabling_viewer_audio_deactivates_and_stops_the_receiver(): + class _Opus: + stopped = False + + def stop(self): + self.stopped = True + + audio = _Transceiver("audio") + receiver = _Opus() + host = _Host(_PeerConnection(_Transceiver("video"), audio), + _Config(accept_viewer_audio_opus=True)) + host._opus_audio_receiver = receiver + host.disable_accept_viewer_audio_opus() + assert audio.direction == "inactive" + assert receiver.stopped is True + assert host._opus_audio_receiver is None + + +def test_a_receiver_that_throws_on_stop_is_still_dropped(): + class _Opus: + def stop(self): + raise OSError("already gone") + + host = _Host(_PeerConnection(_Transceiver("audio")), + _Config(accept_viewer_audio_opus=True)) + host._opus_audio_receiver = _Opus() + host.disable_accept_viewer_audio_opus() + assert host._opus_audio_receiver is None + + +# === Re-subscribing after a renegotiation =================================== + +def test_the_viewer_video_track_is_picked_up_from_the_second_slot(): + track = object() + pc = _PeerConnection(_Transceiver("video", track=object()), + _Transceiver("video", track=track)) + host = _Host(pc, _Config(accept_viewer_video=True)) + host._maybe_resubscribe_viewer_video() + assert host.consumed == [track] + assert host._viewer_video_task is not None + + +def test_no_resubscribe_when_the_feature_is_off(): + pc = _PeerConnection(_Transceiver("video"), _Transceiver("video", + track=object())) + host = _Host(pc, _Config(accept_viewer_video=False)) + host._maybe_resubscribe_viewer_video() + assert host.consumed == [] + + +def test_no_resubscribe_when_a_task_is_already_running(): + pc = _PeerConnection(_Transceiver("video"), _Transceiver("video", + track=object())) + host = _Host(pc, _Config(accept_viewer_video=True)) + host._viewer_video_task = _Task() + host._maybe_resubscribe_viewer_video() + assert host.consumed == [] + + +def test_a_slot_with_no_track_yet_is_skipped(): + """A transceiver exists as soon as it is negotiated; the track arrives + later.""" + pc = _PeerConnection(_Transceiver("video"), _Transceiver("video")) + host = _Host(pc, _Config(accept_viewer_video=True)) + host._maybe_resubscribe_viewer_video() + assert host.consumed == [] + assert host._viewer_video_task is None + + +def test_the_viewer_audio_track_is_picked_up_from_any_audio_slot(): + track = object() + pc = _PeerConnection(_Transceiver("video", track=object()), + _Transceiver("audio", track=track)) + host = _Host(pc, _Config(accept_viewer_audio_opus=True)) + host._maybe_resubscribe_viewer_audio() + assert host.opus_started == [track] + + +def test_no_audio_resubscribe_when_a_receiver_is_already_running(): + pc = _PeerConnection(_Transceiver("audio", track=object())) + host = _Host(pc, _Config(accept_viewer_audio_opus=True)) + host._opus_audio_receiver = object() + host._maybe_resubscribe_viewer_audio() + assert host.opus_started == [] + + +# === The host-initiated offer =============================================== + +def test_renegotiation_sends_the_new_offer_over_the_control_channel( + monkeypatch): + class _Description: + sdp = "v=0\r\nfake offer\r\n" + + class _Negotiating(_PeerConnection): + localDescription = _Description() + + async def createOffer(self): # noqa: N802 # reason: the aiortc name + return _Description() + + async def setLocalDescription(self, offer): # noqa: N802 + self.local = offer + + async def _gathered(pc): + return None + + monkeypatch.setattr(media, "wait_for_ice_gathering", _gathered) + host = _Host(_Negotiating(), _Config()) + asyncio.run(host._async_renegotiate()) + assert host.sent == [{"type": "renegotiate_offer", + "sdp": "v=0\r\nfake offer\r\n"}] + + +def test_a_failed_offer_sends_nothing_rather_than_a_half_negotiation( + monkeypatch): + class _Broken(_PeerConnection): + async def createOffer(self): # noqa: N802 # reason: the aiortc name + raise RuntimeError("connection closed") + + host = _Host(_Broken(), _Config()) + asyncio.run(host._async_renegotiate()) + assert host.sent == [] + + +def test_renegotiating_without_a_connection_is_a_no_op(): + host = _Host(pc=None) + asyncio.run(host._async_renegotiate()) + assert host.sent == [] From abb29fa24109e357bd50268b8fe86f4bf64220f7 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 00:29:36 +0800 Subject: [PATCH 09/20] Ratchet the floor to 75 and re-read the matrix after the WebRTC tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured on the last run for this PR: 75.79% (ubuntu-22.04 / 3.14) to 76.99% (windows-2022, 3.11 and 3.12), so the floor follows the lowest square as usual. Progress.md's subsystem table and portable-gap figures are re-measured from the same run rather than carried forward, and the WebRTC follow-up list now names what each remaining module costs — `webrtc_host` and `webrtc_viewer` are the two that need a bigger double than the four already done, and the two mixins that came out of them are the precedent for splitting first. --- Progress.md | 27 ++++++++++++++++----------- WHATS_NEW.md | 10 +++++----- pyproject.toml | 18 +++++++++--------- 3 files changed, 30 insertions(+), 25 deletions(-) diff --git a/Progress.md b/Progress.md index c0eb4f1a..551d825a 100644 --- a/Progress.md +++ b/Progress.md @@ -228,12 +228,12 @@ capability enum 值與 variadic `ei_seat_bind_capabilities`、event-type enum 2026-08-21 把**機制**補上了（做法見 [WHATS_NEW.md](WHATS_NEW.md)）： 型別契約的豁免清單 2026-08-22 清空，平台縫最後兩個名稱（`keyboard`／`mouse`） 2026-08-23 拿到合約；覆蓋率那半發現不是爬得不夠，是量測起點錯了，修正後地板 -從 50 提到 69，同日再提到 74。 +從 50 提到 69，隔日再提到 75。 **2026-08-23 維護者拍板：下一個覆蓋率目標是 80。** 這一條留到那時候。 型別那一節留著是因為它記的那幾個坑之後還會踩到。 -### 覆蓋率：地板 74，下一站 80 +### 覆蓋率：地板 75，下一站 80 `TODO` — 目標 **80**（2026-08-23 拍板）。地板本身照舊只跟著實測的最低那一格走。 @@ -350,22 +350,23 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl 三支共用的機器搬進 `test/unit_test/headless/_contract_sweep.py`，各自只留自己的 參數來源。 -#### 現在的九宮格（2026-08-23 實測，本 PR 的 run） +#### 現在的九宮格（2026-08-24 實測，本 PR 最後一次 run） | | 最低 | 最高 | | --- | --- | --- | | 上一版（PR #486 首輪） | 69.67%（ubuntu-22.04／3.14） | 70.97%（windows-2022／3.12） | -| 這一版 | **74.99%**（ubuntu-22.04／3.14） | 76.19%（windows-2022，3.10–3.13） | +| 這一版 | **75.79%**（ubuntu-22.04／3.14） | 76.99%（windows-2022，3.11／3.12） | -地板隨之從 69 提到 **74**。 +地板隨之從 69 提到 **75**。這九個數字讀的是 `coverage report`（也就是 +`fail_under` 真正比對的那個），不是 artifact 的 `line-rate`——理由見上一節。 -#### 剩下的 5 點在哪裡（2026-08-23，ubuntu-22.04／3.14 的 artifact） +#### 剩下的 4 點在哪裡（2026-08-24，ubuntu-22.04／3.14 的 artifact） 以下是 statement 數（artifact 的口徑，不含分支），拿來看缺口分佈： | 子系統 | 沒蓋到 / 總 statement | 覆蓋率 | | --- | ---: | ---: | -| `utils/remote_desktop` | 2,653 / 6,622 | 59.9% | +| `utils/remote_desktop` | 2,297 / 6,622 | 65.3% | | `utils/accessibility` | 824 / 1,397 | 41.0% | | `utils/executor` | 653 / 3,906 | 83.3% | | `utils/usb` | 573 / 2,137 | 73.2% | @@ -376,7 +377,7 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl **要動地板，只能補在每一格都跑得到的程式碼上。** 地板取的是最低那一格， 所以只在 Windows 跑得到的東西補再多也不會動它。把九格的未覆蓋行取交集， -2026-08-23 量到 **9,697 個 statement 在每一格都沒被執行過**——那就是可攜的缺口， +2026-08-24 量到 **9,343 個 statement 在每一格都沒被執行過**——那就是可攜的缺口， 也是唯一會抬地板的地方。最大的幾塊： | 檔案 | 每一格都沒蓋到 | 備註 | @@ -393,9 +394,13 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl **下一步的順序**（都還沒做）： -1. `webrtc_host`／`webrtc_viewer`／`multi_viewer`／`webrtc_files`：`[webrtc]` 裝了 - 之後這一族才第一次可測，合計 960 個 statement 在每一格都沒跑過。`webrtc_host_auth` - 與 `webrtc_stats` 已經照這個做法補完（56 個測試），其餘照抄。 +1. `webrtc_host`（351）／`webrtc_viewer`（354）／`multi_viewer`（146）／ + `webrtc_transport`（139）／`webrtc_audio`（114）：`[webrtc]` 裝了之後這一族才 + 第一次可測。`webrtc_host_auth`、`webrtc_stats`、`webrtc_files`、 + `webrtc_host_media` 已經照這個做法補完（119 個測試，做法見 + [WHATS_NEW.md](WHATS_NEW.md)），其餘照抄。`webrtc_host` 與 `webrtc_viewer` + 是兩個大類別，要的替身比前四個大——**先看能不能拆出可測的那一半**， + `webrtc_host_auth`／`webrtc_host_media` 這兩個 mixin 已經是拆出來的先例。 2. 掃不到的那批 adapter：目前卡在「被呼叫者是 class」。要嘛從 class 自己的方法標注 長出一個替身物件，要嘛承認那批不掃——**得先決定，因為前者會讓「跑起來了」和 「驗到了東西」分家**。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index 16eae787..96dc5fc9 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -88,18 +88,18 @@ The auth host double supplies exactly the attribute list `ViewerAuthMixin`'s own docstring asks for, so a mixin that starts reaching for something else fails there rather than leaning on whatever the real host happens to own. -### The Floor Is 74, And The Comment Now Says Which Number That Is +### The Floor Is 75, And The Comment Now Says Which Number That Is -The nine-way matrix runs 74.99% (ubuntu-22.04 / 3.14) to 76.19% (windows-2022), -up from 69.67–70.97, so `fail_under` goes 69 → 74 on the usual convention: floor +The nine-way matrix runs 75.79% (ubuntu-22.04 / 3.14) to 76.99% (windows-2022), +up from 69.67–70.97, so `fail_under` goes 69 → 75 on the usual convention: floor of the lowest square. `Progress.md` had recorded that the floor "had to be dug out of the XML artifact". That is wrong, and following it would set a floor the suite cannot clear. `coverage report` — the step that enforces `fail_under` — includes branch coverage, because `branch = true`; Cobertura's `line-rate` attribute does not. -On the same square and the same run those differ by about 1.8 points (76.14% -against 78.33%). The artifact is the right thing to read for a single +On the same square and the same run those differ by about 2 points (75.79% +against 77.78%). The artifact is the right thing to read for a single subsystem's gap and the wrong thing to set a floor from. ## What's new (2026-08-23) diff --git a/pyproject.toml b/pyproject.toml index 0b02bc81..16648211 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -175,21 +175,21 @@ precision = 2 # 70 was the destination that number was climbing towards; the maintainer set # the next one at 80 on 2026-08-23, and Progress.md tracks the climb. # -# Measured 2026-08-23 on the run for this PR, after `quality.yml` began -# installing the `[webrtc]` extra and two more registries were swept: the -# nine-way matrix runs 74.99% (ubuntu-22.04 / 3.14) to 76.19% (windows-2022, -# 3.10 through 3.13). Windows is still the high corner for the same reason as -# before — the facade imports the backend of the platform it is running on. -# Floored to the integer below the lowest square: 74. +# Measured 2026-08-24 on the last run for this PR, after `quality.yml` began +# installing the `[webrtc]` extra and the WebRTC subsystem got its first +# tests: the nine-way matrix runs 75.79% (ubuntu-22.04 / 3.14) to 76.99% +# (windows-2022, 3.11 and 3.12). Windows is still the high corner for the same +# reason as before — the facade imports the backend of the platform it is +# running on. Floored to the integer below the lowest square: 75. # # Read that matrix from the "Enforce the coverage floor" step, which is this # report and therefore the number `fail_under` is compared against. The # `coverage.xml` artifact is NOT interchangeable with it: `branch = true` # above puts branches in this total and Cobertura's `line-rate` attribute -# leaves them out, so the XML reads about 1.8 points higher (78.33% against -# 76.14% on windows-2022 / 3.14, same run). A floor set from the artifact +# leaves them out, so the XML reads about 2 points higher (77.78% against +# 75.79% on ubuntu-22.04 / 3.14, same run). A floor set from the artifact # would be a floor the suite cannot clear. -fail_under = 74 +fail_under = 75 [tool.mypy] python_version = "3.10" From 81ca550cfd4bc10ee53eaada6429b6340e0e4767 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 06:57:13 +0800 Subject: [PATCH 10/20] Test the last five WebRTC modules, which never needed the split Progress.md had webrtc_host and webrtc_viewer down as "first see whether a testable half can be split out", on the precedent of the two mixins that had already been carved off them. Going and looking says that was the wrong question: neither constructor touches aiortc, and every collaborator arrives as a keyword argument or a module-level name. What held them at 19.83% and 14.08% was that reaching line one meant standing up an RTCPeerConnection, a screen grabber and a background event loop. 364 tests against doubles instead, and the classes stay where they are. webrtc_host, webrtc_viewer and multi_viewer reach 100%, webrtc_audio 96.27% from zero, webrtc_transport 88.84%; 77.04% -> 79.40% on one machine. The floor stays at 75 until the matrix says otherwise. Most of it is refusal, because that is what these classes do with what arrives from the wire: each of the host's four channels against a peer that never authenticated, one the operator has since put in read-only, and one that is flooding; the three inbox verbs against the _safe_basename that stands between "../secret.txt" and the host's own files; the viewer's slot arithmetic, which has to find the host's recvonly slot by m-line position because aiortc gives every answerer transceiver the same direction. The doubles found a real one. _consume_video caught (OSError, RuntimeError), and aiortc ends a track by raising MediaStreamError, which derives straight from Exception. Nothing awaits that task, so the ordinary end of a session reached the console as an un-retrieved task exception instead of the line the host's own drain loop has always logged. What did have to be split was the test files, against the 750-line limit the project applies to every new one. --- CHANGELOG.md | 10 + Progress.md | 56 +- WHATS_NEW.md | 70 ++ architecture_explore.md | 16 +- .../utils/remote_desktop/webrtc_viewer.py | 10 + test/unit_test/headless/_webrtc_doubles.py | 307 ++++++++ .../test_remote_desktop_multi_viewer.py | 522 +++++++++++++ .../headless/test_webrtc_host_channels.py | 739 ++++++++++++++++++ .../headless/test_webrtc_host_session.py | 558 +++++++++++++ .../headless/test_webrtc_opus_audio.py | 395 ++++++++++ .../headless/test_webrtc_transport.py | 424 ++++++++++ .../headless/test_webrtc_viewer_control.py | 520 ++++++++++++ .../headless/test_webrtc_viewer_media.py | 385 +++++++++ .../headless/test_webrtc_viewer_session.py | 465 +++++++++++ 14 files changed, 4458 insertions(+), 19 deletions(-) create mode 100644 test/unit_test/headless/_webrtc_doubles.py create mode 100644 test/unit_test/headless/test_remote_desktop_multi_viewer.py create mode 100644 test/unit_test/headless/test_webrtc_host_channels.py create mode 100644 test/unit_test/headless/test_webrtc_host_session.py create mode 100644 test/unit_test/headless/test_webrtc_opus_audio.py create mode 100644 test/unit_test/headless/test_webrtc_transport.py create mode 100644 test/unit_test/headless/test_webrtc_viewer_control.py create mode 100644 test/unit_test/headless/test_webrtc_viewer_media.py create mode 100644 test/unit_test/headless/test_webrtc_viewer_session.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c94bec1..c77bb4fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -387,6 +387,16 @@ only when documented here with a migration path. ### Fixed +- **The WebRTC viewer ended every clean disconnect with an unhandled task + exception.** `WebRTCDesktopViewer._consume_video` caught + `(OSError, RuntimeError)`, but aiortc signals the end of a track by raising + `MediaStreamError`, which derives straight from `Exception` and so matched + neither. Nothing awaits that task, so the normal end of a session — the host + stopping its screen share, or the connection closing — reached the console as + asyncio's "Task exception was never retrieved" traceback instead of the + "video stream ended" line the host's own drain loop already logged. The + stream is unaffected either way; only the logging changes. + - **A `null` in a remote-desktop entry's `tags` became a tag named `"None"`.** `AddressBook.set_tags()` cleaned its input with `str(t).strip()`, and `str(None)` is the non-empty string `"None"`, so a JSON `null` in the array — diff --git a/Progress.md b/Progress.md index 551d825a..6a751ab6 100644 --- a/Progress.md +++ b/Progress.md @@ -384,23 +384,18 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl | --- | ---: | --- | | `utils/executor/action_executor.py` | 589 | 掃不到的那批 adapter：被呼叫者是 class（沒有回傳標注可讀）、或 adapter 伸手進兩個模組 | | `utils/accessibility/backends/windows_backend.py` | 446 | UIA COM，要一套夠像的替身 | -| `utils/remote_desktop/webrtc_viewer.py` | 354 | 現在 import 得到了（17.9%），但沒有測試驅動 | -| `utils/remote_desktop/webrtc_host.py` | 351 | 同上（24.8%） | +| ~~`utils/remote_desktop/webrtc_viewer.py`~~ | ~~354~~ | **2026-08-24 補完（100%）** | +| ~~`utils/remote_desktop/webrtc_host.py`~~ | ~~351~~ | **2026-08-24 補完（100%）** | | `utils/mcp_server/tools/_handlers.py` | 348 | 同 `action_executor.py` | | `utils/remote_desktop/signaling_server.py` | 155 | **CI 動不了**：要 `[signaling]` extra（fastapi／uvicorn），沒裝 | -| `utils/remote_desktop/multi_viewer.py` | 146 | | +| ~~`utils/remote_desktop/multi_viewer.py`~~ | ~~146~~ | **2026-08-24 補完（100%）** | | `wrapper/window_backends/x11_backend.py` | 133 | 只有 Linux 那兩格跑得到，抬得動地板 | | `utils/accessibility/backends/linux_backend.py` | 132 | 同上 | -**下一步的順序**（都還沒做）： +**下一步的順序**： -1. `webrtc_host`（351）／`webrtc_viewer`（354）／`multi_viewer`（146）／ - `webrtc_transport`（139）／`webrtc_audio`（114）：`[webrtc]` 裝了之後這一族才 - 第一次可測。`webrtc_host_auth`、`webrtc_stats`、`webrtc_files`、 - `webrtc_host_media` 已經照這個做法補完（119 個測試，做法見 - [WHATS_NEW.md](WHATS_NEW.md)），其餘照抄。`webrtc_host` 與 `webrtc_viewer` - 是兩個大類別，要的替身比前四個大——**先看能不能拆出可測的那一半**， - `webrtc_host_auth`／`webrtc_host_media` 這兩個 mixin 已經是拆出來的先例。 +1. ~~`webrtc_host`／`webrtc_viewer`／`multi_viewer`／`webrtc_transport`／ + `webrtc_audio`~~ **2026-08-24 做完**，見下一節。 2. 掃不到的那批 adapter：目前卡在「被呼叫者是 class」。要嘛從 class 自己的方法標注 長出一個替身物件，要嘛承認那批不掃——**得先決定，因為前者會讓「跑起來了」和 「驗到了東西」分家**。 @@ -411,6 +406,45 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl `[office]`／`[signaling]`，補的測試會整批 skip，對地板一個點都不動。要補之前 先照 `[webrtc]` 的先例把 extra 加進 CI。 +#### 2026-08-24：WebRTC 那一族補完了，而「先拆一半」是問錯了問題 + +上面第 1 項原本寫著「`webrtc_host` 與 `webrtc_viewer` 是兩個大類別， +**先看能不能拆出可測的那一半**」，理由是那兩個 mixin +（`webrtc_host_auth`／`webrtc_host_media`）是拆出來才測得到的先例。 +**實際去看之後發現不必拆**：兩個類別的建構子都不碰 aiortc——主機只存下 config、 +RateLimiter 與 SessionPermissions，檢視端只存下 callback——而每一個協作對象 +（`RTCPeerConnection`、`ScreenVideoTrack`、asyncio 橋接、稽核記錄）都是 +模組層名稱或建構參數，換掉就好。擋在 19.83%／14.08% 的從來不是類別的形狀， +是「要跑到第一行得先站起一個 PeerConnection、一個螢幕擷取器和一條背景事件迴圈」。 + +| 模組 | 之前 | 之後 | +| --- | ---: | ---: | +| `webrtc_host.py` | 19.83% | **100%** | +| `webrtc_viewer.py` | 14.08% | **100%** | +| `multi_viewer.py` | 21.24% | **100%** | +| `webrtc_audio.py` | 0.00% | **96.27%** | +| `webrtc_transport.py` | 27.89% | **88.84%** | + +沒補完的兩塊是刻意的：`webrtc_transport._get_cursor_position` 是三條平台分支， +任何一格只跑得到自己那條；`webrtc_audio._enqueue` 裡兩個吞掉的 queue 競態 +只有「謊報自己滿了的 queue」造得出來，那是在測替身不是在測程式。 + +替身集中在 `test/unit_test/headless/_webrtc_doubles.py`（形狀比照 +`_contract_sweep.py`），六個測試檔共用；`FakePeerConnection` 刻意把主機端與 +檢視端的介面放在同一個類別裡——它替的是同一個 aiortc 型別，照方向拆成兩個 +只會讓兩份替身各自漂走。 + +**要拆的是測試檔，不是被測的類別**，因為 `CLAUDE.md` 的 750 行上限對新檔案 +沒有例外。主機拆成 `test_webrtc_host_session.py`（557，offer／answer／狀態／拆除） +與 `test_webrtc_host_channels.py`（739，四條 DataChannel、權限、限流、收件匣）； +檢視端拆成 `test_webrtc_viewer_session.py`（466）／ +`test_webrtc_viewer_media.py`（385，m-line 位置與開關）／ +`test_webrtc_viewer_control.py`（519）。 + +**地板還沒動。** 本機（Windows／3.14）全專案 77.04% → 79.40%，363 個新測試， +但地板取的是九宮格最低那一格，**要等 CI 的 `coverage report` 印出來才能改**—— +這條規則在上面那節寫過，不從單機的數字推。 + ### mypy：整包把關，**豁免清單已經清空** `TODO` → **完成（2026-08-22）** diff --git a/WHATS_NEW.md b/WHATS_NEW.md index 96dc5fc9..e8fd271a 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -88,6 +88,76 @@ The auth host double supplies exactly the attribute list `ViewerAuthMixin`'s own docstring asks for, so a mixin that starts reaching for something else fails there rather than leaning on whatever the real host happens to own. +### The Two Big WebRTC Classes Did Not Need Splitting, They Needed Doubles + +`Progress.md` had the next step down as "first see whether a testable half can +be split out of `webrtc_host` and `webrtc_viewer`", on the precedent of the two +mixins that had already been carved off them. Going and looking says that was +the wrong question. Neither constructor touches aiortc — the host stores a +config, a rate limiter and a permission set, the viewer stores callbacks — and +every collaborator either arrives as a keyword argument or is a module-level +name. What held them at 19.83% and 14.08% was never the shape of the class; it +was that reaching line one meant standing up an `RTCPeerConnection`, a screen +grabber and a background event loop. + +So the five remaining WebRTC modules got 363 tests against doubles, and the +classes were left where they are: + +| module | before | after | +| --- | ---: | ---: | +| `webrtc_host.py` | 19.83% | **100%** | +| `webrtc_viewer.py` | 14.08% | **100%** | +| `multi_viewer.py` | 21.24% | **100%** | +| `webrtc_audio.py` | 0.00% | **96.27%** | +| `webrtc_transport.py` | 27.89% | **88.84%** | + +Two remainders are deliberate. `_get_cursor_position` is three platform +branches of which any one square runs one, and the two swallowed queue races in +`_enqueue` need a queue that lies about being full — that would be a test of the +double, not of the code. + +Most of what the tests pin down is refusal, because that is most of what these +classes do with what arrives from the wire. The host's four channels each open +with the PeerConnection, which is *before* the token has been checked, so each +one is tested for what it does with traffic from a peer that never +authenticated, from one the operator has since put in read-only, and from one +that is simply flooding. The three inbox verbs — list, fetch, delete — are +tested for the thing they all route through: `_safe_basename`, which is what +stands between a viewer sending `../secret.txt` and the host's own files. + +On the viewer side the load-bearing detail is that slots are found by m-line +order rather than by direction: aiortc gives every answerer transceiver the +default `recvonly` regardless of what the offer asked for, so "my screen goes in +the second video transceiver" is arithmetic, and off by one there replaces the +picture the viewer is watching. Same for the toggles, which are asymmetric on +purpose — off is `replaceTrack(None)` in place, on always renegotiates, because +the host needs a fresh `track` event to restart its consume task. + +The doubles live in `test/unit_test/headless/_webrtc_doubles.py`, shaped like +`_contract_sweep.py` and shared by six test modules. `FakePeerConnection` +carries both ends' surface in one class deliberately: it stands in for one +aiortc type, and splitting it by direction would be two doubles drifting apart +from the same original. + +The thing that had to be split, in the end, was the test files — `CLAUDE.md`'s +750-line limit has no exception for new ones. Host: session (557) and channels +(739). Viewer: session (466), media (385), control (519). + +### The Viewer Logged Every Clean Disconnect As An Unhandled Task Exception + +`WebRTCDesktopViewer._consume_video` caught `(OSError, RuntimeError)`. aiortc +ends a track by raising `MediaStreamError` out of `recv()`, and that derives +straight from `Exception`, so it matched neither arm. Nothing awaits that task, +which means the ordinary end of a session — the host stopping its screen share, +or the connection closing — surfaced as asyncio's "Task exception was never +retrieved" instead of the "video stream ended" line the host's own drain loop +and the Opus receiver have always logged. It now catches it, and +`CancelledError` still propagates as the comment there requires. + +The doubles found it: the shared `FrameTrack` ends a stream the way aiortc does, +and the viewer's own test had been written against a local double that ended it +with `OSError`. + ### The Floor Is 75, And The Comment Now Says Which Number That Is The nine-way matrix runs 75.79% (ubuntu-22.04 / 3.14) to 76.99% (windows-2022), diff --git a/architecture_explore.md b/architecture_explore.md index ce325a37..358bc7d3 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -6,7 +6,7 @@ > 擷取每個模組的 docstring 與頂層公開名稱；統計數字取自實際檔案，非估算。 > 指令數與公開 API 數以 `executor.known_commands()` 與 `je_auto_control.__all__` 在工作樹上實測取得。 > -> **掃描時間**：2026-08-21　**版本**：`pyproject.toml` version `0.0.220`　**分支**：`feat/typing-contract-and-coverage-ratchet` +> **掃描時間**：2026-08-24　**版本**：`pyproject.toml` version `0.0.221`　**分支**：`feat/coverage-to-80` --- @@ -20,7 +20,7 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | | Python 模組總數（含周邊子專案） | 1,032 | -| 程式碼總行數 | 141,324 | +| 程式碼總行數 | 141,334 | | `je_auto_control/utils/` 子套件數 | 310 | | `AC_*` 動作指令數（`known_commands()` 實測） | 773 | | 套件門面 `__all__` 公開名稱數 | 1,238 | @@ -510,14 +510,14 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.10 遠端桌面與 USB -> 6 個套件、約 17,909 行。 +> 6 個套件、約 17,919 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | | `utils/admin/` | 328 | 多主機管理主控台：平行輪詢 N 個 AutoControl REST 端點 | | `utils/config_sync/` | 246 | 透過訊令伺服器做跨機器設定同步 | | `utils/device_matrix/` | 138 | 行動裝置矩陣：同一 action list 於多台裝置平行執行 | -| `utils/remote_desktop/` | 11,996 | **遠端桌面子系統**（56 檔／11.7K LOC）：TCP／WebSocket／WebRTC 三條傳輸路徑、主機與檢視端、訊令伺服器、TURN／中繼、多檢視者、錄影、信任清單、TOTP、稽核鏈 | +| `utils/remote_desktop/` | 12,006 | **遠端桌面子系統**（56 檔／11.7K LOC）：TCP／WebSocket／WebRTC 三條傳輸路徑、主機與檢視端、訊令伺服器、TURN／中繼、多檢視者、錄影、信任清單、TOTP、稽核鏈 | | `utils/usb/` | 4,281 | 跨平台 USB 列舉／熱插拔／裝置直通（WinUSB、IOKit、libusb 後端 + ACL + WebRTC DataChannel 通道） | | `utils/usbip/` | 920 | USB/IP 線路協定主機端（協定封包、TCP 伺服器、libusb URB 後端） | @@ -726,14 +726,14 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `rate_limit.py` | 48 | 工具呼叫的 token bucket 限流。 | | `__main__.py` | 88 | `je_auto_control_mcp` console script 進入點。 | -#### `utils/remote_desktop/`（11,996 行／56 檔） +#### `utils/remote_desktop/`（12,006 行／56 檔） 三條傳輸路徑並存：**TCP**（JPEG 影格）、**WebSocket**（同協定換傳輸）、**WebRTC**（aiortc 視訊 + DataChannel）。 | 檔案 | 行數 | 職責 | | --- | ---: | --- | | `webrtc_host.py` | 702 | WebRTC 主機：串流螢幕視訊並接受檢視端輸入;session 生命週期、DataChannel 接線、檔案收發。 | -| `webrtc_viewer.py` | 662 | WebRTC 檢視端：接收視訊並送出輸入。 | +| `webrtc_viewer.py` | 672 | WebRTC 檢視端：接收視訊並送出輸入。 | | `host.py` | 625 | TCP 主機：接受迴圈、TLS 包裝、連線／認證握手、音訊與剪貼簿廣播、檔案推送、單次 token。 | | `viewer.py` | 623 | TCP 檢視端。 | | `host_service.py` | 542 | 無頭 WebRTC 主機執行器 + 多平台服務安裝器。 | @@ -1027,7 +1027,7 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | --- | ---: | ---: | | `gui/` | 90 | 26,699 | | `utils/mcp_server/` | 21 | 17,356 | -| `utils/remote_desktop/` | 56 | 11,996 | +| `utils/remote_desktop/` | 56 | 12,006 | | `utils/executor/` | 6 | 9,081 | | `utils/usb/` | 17 | 4,281 | | `je_auto_control/`（頂層 3 檔） | 3 | 2,367 | @@ -1046,5 +1046,5 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 727 | | 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 673 | 47,683 | -| **總計** | **1,026** | **141,259** | +| **總計** | **1,026** | **141,269** | diff --git a/je_auto_control/utils/remote_desktop/webrtc_viewer.py b/je_auto_control/utils/remote_desktop/webrtc_viewer.py index 72ee7331..dbdd1795 100644 --- a/je_auto_control/utils/remote_desktop/webrtc_viewer.py +++ b/je_auto_control/utils/remote_desktop/webrtc_viewer.py @@ -540,6 +540,14 @@ async def _consume_video(self, track) -> None: # CancelledError is intentionally not caught — it must propagate # so the awaiter knows the consumer ended via cancellation # rather than a stream error (S7497). + # + # MediaStreamError is the *normal* end: aiortc raises it from recv() + # when the host stops sharing or the connection closes. It derives + # straight from Exception, so it is not covered by the OSError / + # RuntimeError arm below, and nobody awaits this task — letting it + # escape turns every clean disconnect into an un-retrieved task + # exception. The host's own drain loop has always caught it. + from aiortc.mediastreams import MediaStreamError try: while not self._closed.is_set(): frame = await track.recv() @@ -548,6 +556,8 @@ async def _consume_video(self, track) -> None: self._on_frame(frame) except (RuntimeError, OSError) as error: autocontrol_logger.debug("frame cb: %r", error) + except MediaStreamError: + autocontrol_logger.info("webrtc viewer: video stream ended") except (OSError, RuntimeError) as error: autocontrol_logger.info("webrtc viewer: video stream ended: %r", error) diff --git a/test/unit_test/headless/_webrtc_doubles.py b/test/unit_test/headless/_webrtc_doubles.py new file mode 100644 index 00000000..eb663b7d --- /dev/null +++ b/test/unit_test/headless/_webrtc_doubles.py @@ -0,0 +1,307 @@ +"""The aiortc-shaped doubles the WebRTC tests share. + +Six test modules -- host session, host channels, host inbox, viewer session, +viewer media, viewer control -- all stand the same two classes up against the +same three collaborators, and every one of those collaborators is a boundary +we deliberately do not cross in a unit test: + +* **`RTCPeerConnection`** would start ICE traffic against public STUN + servers from a CI runner. +* **`ScreenVideoTrack`** would open a screen grabber, which on a headless + runner is either absent or a black rectangle. +* **The asyncio bridge** would queue work onto a background event loop that + no test is running, so nothing would ever be observed. + +What is faked is only the aiortc surface: the doubles record what they were +handed and hand back what the real objects hand back. The code under test -- +which transceiver the host adds, which slot the viewer replaces, which +envelope goes out on which channel -- is never replaced. + +`FakePeerConnection` carries both ends' surface in one class on purpose. It +stands in for one aiortc type, and splitting it per direction would mean two +doubles drifting apart from the same original. + +Nothing here is a test; the file is named so pytest does not collect it. +""" +from __future__ import annotations + +import asyncio +from concurrent.futures import Future + + +class Channel: + """A DataChannel: records its handlers, and what was sent on it.""" + + def __init__(self, label: str = "ctrl", + ready_state: str = "connecting") -> None: + self.label = label + self.readyState = ready_state # noqa: N815 # reason: the aiortc name + self.handlers = {} + self.sent = [] + self.send_error = None + + def on(self, event): + def _register(func): + self.handlers[event] = func + return func + return _register + + def fire(self, event, *args): + return self.handlers[event](*args) + + def send(self, payload): + if self.send_error is not None: + raise self.send_error + self.sent.append(payload) + + +class Track: + """A MediaStreamTrack: a kind, and whether anybody stopped it.""" + + def __init__(self, kind: str = "video", **kwargs) -> None: + self.kind = kind + self.kwargs = kwargs + self.stopped = False + + def stop(self) -> None: + self.stopped = True + + +class FrameTrack: + """Yields a scripted sequence of frames, then raises to end the stream.""" + + def __init__(self, *frames, ending=None) -> None: + from aiortc.mediastreams import MediaStreamError + self.frames = list(frames) + self._ending = ending if ending is not None else MediaStreamError() + + async def recv(self): + if not self.frames: + raise self._ending + return self.frames.pop(0) + + +class Sender: + def __init__(self, track=None) -> None: + self.track = track + self.replaced = [] + self.replace_error = None + + def replaceTrack(self, track): # noqa: N802 # reason: the aiortc name + if self.replace_error is not None: + raise self.replace_error + self.replaced.append(track) + self.track = track + + +class Transceiver: + def __init__(self, kind: str, track=None, + direction: str = "recvonly") -> None: + self.kind = kind + self.sender = Sender(track) + self.direction = direction + + +class FakePeerConnection: + """The aiortc surface both ends touch, and nothing beyond it.""" + + instances: list = [] + + def __init__(self, configuration=None) -> None: + self.configuration = configuration + self.connectionState = "new" # noqa: N815 # reason: the aiortc name + self.iceGatheringState = "new" # noqa: N815 # reason: the aiortc name + self.tracks = [] + self.transceivers = [] + self.channels = [] + self.handlers = {} + self.local_sdp = "v=0 local-sdp" + self.remote_descriptions = [] + self.closed = False + self.stats = {} + self.stats_error = None + self.answer_error = None + self.remote_description_error = None + FakePeerConnection.instances.append(self) + + # --- the aiortc surface -------------------------------------------------- + + def addTrack(self, track): # noqa: N802 # reason: the aiortc name + self.tracks.append(track) + + def addTransceiver(self, kind, direction): # noqa: N802 # the aiortc name + self.transceivers.append((kind, direction)) + + def getTransceivers(self): # noqa: N802 # reason: the aiortc name + return list(self.transceivers) + + def createDataChannel(self, label): # noqa: N802 # reason: the aiortc name + channel = Channel(label) + self.channels.append(channel) + return channel + + def on(self, event): + def _register(func): + self.handlers[event] = func + return func + return _register + + async def createOffer(self): # noqa: N802 # reason: the aiortc name + return "offer-object" + + async def createAnswer(self): # noqa: N802 # reason: the aiortc name + if self.answer_error is not None: + raise self.answer_error + return "answer-object" + + async def setLocalDescription(self, description): # noqa: N802 + self.local_description_arg = description + + async def setRemoteDescription(self, description): # noqa: N802 + if self.remote_description_error is not None: + raise self.remote_description_error + self.remote_descriptions.append(description) + + async def getStats(self): # noqa: N802 # reason: the aiortc name + if self.stats_error is not None: + raise self.stats_error + return self.stats + + async def close(self): + self.closed = True + + @property + def localDescription(self): # noqa: N802 # reason: the aiortc name + return type("_Desc", (), {"sdp": self.local_sdp})() + + # --- test helpers -------------------------------------------------------- + + def channel(self, label: str) -> Channel: + return next(c for c in self.channels if c.label == label) + + def fire(self, event, *args): + return self.handlers[event](*args) + + def video_slots(self, count: int = 2): + """Offer `count` video transceivers, as a host's offer would.""" + self.transceivers = [Transceiver("video") for _ in range(count)] + return self.transceivers + + def complete_ice_gathering(self) -> None: + """Finish gathering and notify whoever subscribed to the change.""" + self.iceGatheringState = "complete" + self.handlers["icegatheringstatechange"]() + + +class Bridge: + """The asyncio bridge, minus the loop: run the work here and now. + + `submit` mirrors `run_coroutine_threadsafe` in parking any exception on + the future rather than raising into the caller, because that is where + `create_offer`'s and `stop`'s error handling reads it from. + """ + + def __init__(self) -> None: + self.deferred = [] + + def submit(self, coro) -> Future: + future: Future = Future() + try: + future.set_result(asyncio.run(coro)) + except BaseException as error: # noqa: BLE001 # reason: mirrors run_coroutine_threadsafe, which parks any exception on the future rather than raising into the submitting thread + future.set_exception(error) + return future + + def call_soon(self, callback, *args) -> None: + self.deferred.append((callback, args)) + callback(*args) + + +class HangingBridge: + """A bridge whose work never lands: every future times out.""" + + def submit(self, coro) -> Future: + coro.close() + future: Future = Future() + future.set_exception(asyncio.TimeoutError()) + return future + + def call_soon(self, callback, *args) -> None: + return None + + +class AuditLog: + """The audit log, without the write to the user's real home directory.""" + + def __init__(self) -> None: + self.events = [] + self.error = None + + def log(self, event_type, **fields) -> None: + if self.error is not None: + raise self.error + self.events.append((event_type, fields)) + + @property + def event_types(self) -> list: + return [event for event, _ in self.events] + + +class MicReceiver: + """Host side of the raw-PCM mic uplink: what arrived, and was it closed.""" + + def __init__(self) -> None: + self.chunks = [] + self.started = False + self.stopped = False + self.stop_error = None + + def start(self) -> None: + self.started = True + + def on_chunk(self, chunk) -> None: + self.chunks.append(chunk) + + def stop(self) -> None: + if self.stop_error is not None: + raise self.stop_error + self.stopped = True + + +class MicSender: + """Viewer side of the same uplink; it is handed the channel to write to.""" + + def __init__(self, channel) -> None: + self.channel = channel + self.started = False + self.running = True + self.stop_error = None + + def start(self) -> None: + self.started = True + + def is_running(self) -> bool: + return self.running + + def stop(self) -> None: + if self.stop_error is not None: + raise self.stop_error + self.running = False + + +class Stoppable: + """Something the teardown path is expected to stop, and may fail to.""" + + def __init__(self, error=None) -> None: + self.stopped = False + self._error = error + + def stop(self) -> None: + if self._error is not None: + raise self._error + self.stopped = True + + +async def noop_ice_gathering(pc, timeout=None): + """Stand in for `wait_for_ice_gathering`: there are no candidates here.""" + return None diff --git a/test/unit_test/headless/test_remote_desktop_multi_viewer.py b/test/unit_test/headless/test_remote_desktop_multi_viewer.py new file mode 100644 index 00000000..a75f64af --- /dev/null +++ b/test/unit_test/headless/test_remote_desktop_multi_viewer.py @@ -0,0 +1,522 @@ +"""One capture, N viewers: what the coordinator owns and what it delegates. + +`MultiViewerHost` is deliberately thin -- it runs one `WebRTCDesktopHost` +per viewer and forwards almost everything -- but the four things it does +own are the ones that go wrong quietly: + +* **The screen source is shared and reference-counted by hand.** It is + built on the first session and stopped when the last one goes, so the + capture thread outliving every viewer, or being torn down while one is + still watching, are both bugs the tests below would catch. +* **Every callback is re-wrapped to carry a `session_id`.** The GUI gets + `(session_id, state)` where the single-viewer host gives it `(state)`, + and the pending-viewer wrapper has to look the host up *at fire time* -- + the viewer id it reports does not exist when the wrapper is built. +* **The connection timestamp is minted here, not in the host.** It is + written from the auth wrapper, which is the only moment the coordinator + is told a viewer got through. +* **A broadcast must not be stopped by one bad recipient.** `broadcast_file` + skips viewers that never authenticated and keeps going past a viewer + whose channel has died; the count it returns is what the GUI reports. + +`WebRTCDesktopHost`, `ScreenVideoTrack` and `MediaRelay` are all replaced +here: this module is a coordinator, and every one of those three would drag +in a real PeerConnection or a real screen grab. What is *not* faked is the +session bookkeeping, which is the code under test. +""" +from __future__ import annotations + +import pytest + +from headless._webrtc_doubles import Track +from je_auto_control.utils.remote_desktop import multi_viewer as mv +from je_auto_control.utils.remote_desktop.multi_viewer import MultiViewerHost +from je_auto_control.utils.remote_desktop.permissions import SessionPermissions + + +class _FakeRelay: + def __init__(self) -> None: + self.subscribed = [] + + def subscribe(self, track): + proxy = ("proxy", len(self.subscribed)) + self.subscribed.append(track) + return proxy + + +class _FakeHost: + """A `WebRTCDesktopHost` stand-in that records what it was told.""" + + instances = [] + + def __init__(self, **kwargs) -> None: + self.kwargs = kwargs + self.authenticated = False + self.connection_state = "new" + self.pending_viewer_id = None + self.offers = [] + self.answers = [] + self.pushed = [] + self.stopped = False + self.permissions = None + self.calls = [] + self.raise_on = set() + self._pc = None + _FakeHost.instances.append(self) + + def _maybe_raise(self, name: str) -> None: + if name in self.raise_on: + raise RuntimeError(f"{name} failed") + + def create_offer(self, peer_label="remote viewer"): + self.offers.append(peer_label) + return f"sdp-for-{peer_label}" + + def accept_answer(self, answer_sdp): + self.answers.append(answer_sdp) + + def stop(self): + self._maybe_raise("stop") + self.stopped = True + + def approve_pending_viewer(self): + self.calls.append("approve") + + def reject_pending_viewer(self): + self.calls.append("reject") + + def trust_pending_viewer(self, label=""): + self.calls.append(("trust", label)) + + def set_permissions(self, permissions): + self._maybe_raise("set_permissions") + self.permissions = permissions + + def disable_accept_viewer_video(self): + self._maybe_raise("disable_video") + self.calls.append("disable_video") + + def disable_accept_viewer_audio_opus(self): + self._maybe_raise("disable_audio") + self.calls.append("disable_audio") + + def push_file(self, local_path, remote_name=None): + self._maybe_raise("push_file") + self.pushed.append((local_path, remote_name)) + + +@pytest.fixture(autouse=True) +def fake_webrtc(monkeypatch): + """Replace the three things that would touch a screen or a network.""" + _FakeHost.instances = [] + monkeypatch.setattr(mv, "WebRTCDesktopHost", _FakeHost) + monkeypatch.setattr(mv, "ScreenVideoTrack", Track) + monkeypatch.setattr(mv, "MediaRelay", _FakeRelay) + yield + _FakeHost.instances = [] + + +def _host(**kwargs) -> MultiViewerHost: + kwargs.setdefault("token", "shared-secret") + return MultiViewerHost(**kwargs) + + +# --- construction ------------------------------------------------------------- + +def test_a_host_without_a_token_is_refused(): + with pytest.raises(ValueError, match="non-empty token"): + MultiViewerHost(token="") + + +def test_read_only_shorthand_becomes_a_permission_set(): + assert _host(read_only=True).permissions.allow_input is False + assert _host(read_only=False).permissions.allow_input is True + + +def test_explicit_permissions_win_over_the_shorthand(): + # Both arguments reach the GUI's constructor call; the granular one is + # the newer surface and must not be overridden by a stale bool. + permissions = SessionPermissions.from_read_only(False) + host = _host(read_only=True, permissions=permissions) + assert host.permissions is permissions + + +# --- session lifecycle -------------------------------------------------------- + +def test_first_session_builds_the_capture_and_labels_the_peer(): + host = _host() + session_id, offer = host.create_session_offer() + assert len(session_id) == 16, "secrets.token_hex(8)" + assert offer == f"sdp-for-viewer-{session_id[:6]}" + assert host.session_count() == 1 + assert isinstance(host.screen_track(), Track) + + +def test_every_session_subscribes_to_the_same_capture(): + host = _host() + host.create_session_offer() + host.create_session_offer() + track = host.screen_track() + tracks = [inst.kwargs["external_video_track"] + for inst in _FakeHost.instances] + assert len(set(tracks)) == 2, "each viewer gets its own relay proxy" + assert all(t is not track for t in tracks) + assert host.session_count() == 2 + + +def test_the_capture_track_is_built_from_the_shared_config(): + from je_auto_control.utils.remote_desktop.webrtc_transport import ( + WebRTCConfig, + ) + config = WebRTCConfig(monitor_index=2, fps=15, region=(1, 2, 3, 4), + show_cursor=False) + host = _host(config=config) + host.create_session_offer() + assert host.screen_track().kwargs == { + "monitor_index": 2, "fps": 15, "region": (1, 2, 3, 4), + "show_cursor": False, + } + + +def test_sessions_inherit_the_coordinator_settings(): + trust = object() + host = _host(trust_list=trust, ip_whitelist=["10.0.0.1"]) + host.create_session_offer() + kwargs = _FakeHost.instances[0].kwargs + assert kwargs["token"] == "shared-secret" + assert kwargs["trust_list"] is trust + assert kwargs["ip_whitelist"] == ["10.0.0.1"] + + +def test_answers_go_to_the_session_they_belong_to(): + host = _host() + first, _ = host.create_session_offer() + second, _ = host.create_session_offer() + host.accept_session_answer(second, "answer-sdp") + by_id = dict(zip([first, second], _FakeHost.instances)) + assert by_id[second].answers == ["answer-sdp"] + assert by_id[first].answers == [] + + +def test_an_unknown_session_id_is_a_key_error(): + host = _host() + with pytest.raises(KeyError, match="unknown session_id"): + host.accept_session_answer("deadbeef", "answer") + + +def test_the_capture_survives_until_the_last_viewer_leaves(): + host = _host() + first, _ = host.create_session_offer() + second, _ = host.create_session_offer() + track = host.screen_track() + + host.stop_session(first) + assert not track.stopped, "one viewer is still watching" + assert host.session_count() == 1 + + host.stop_session(second) + assert track.stopped + assert host.screen_track() is None + + +def test_stopping_an_unknown_session_is_a_no_op(): + host = _host() + host.create_session_offer() + track = host.screen_track() + host.stop_session("not-a-session") + assert not track.stopped + assert host.session_count() == 1 + + +def test_a_session_that_fails_to_stop_is_still_forgotten(): + # The PeerConnection may already be dead when the GUI asks to close the + # tab; the session must leave the table anyway, or the capture it holds + # a reference to never gets released. + host = _host() + session_id, _ = host.create_session_offer() + track = host.screen_track() + _FakeHost.instances[0].raise_on.add("stop") + host.stop_session(session_id) + assert host.session_count() == 0 + assert track.stopped + + +def test_stop_all_tears_every_session_and_the_capture_down(): + host = _host() + host.create_session_offer() + host.create_session_offer() + track = host.screen_track() + host.stop_all() + assert all(inst.stopped for inst in _FakeHost.instances) + assert track.stopped + assert host.session_count() == 0 + + +def test_a_session_that_fails_to_stop_does_not_strand_the_others(): + host = _host() + host.create_session_offer() + host.create_session_offer() + _FakeHost.instances[0].raise_on.add("stop") + host.stop_all() + assert _FakeHost.instances[1].stopped + assert host.screen_track() is None, "the capture is still released" + + +def test_a_new_session_after_stop_all_builds_a_fresh_capture(): + host = _host() + host.create_session_offer() + first_track = host.screen_track() + host.stop_all() + host.create_session_offer() + assert host.screen_track() is not first_track + + +# --- per-session controls ----------------------------------------------------- + +@pytest.mark.parametrize("method,recorded", [ + ("approve_pending_viewer", "approve"), + ("reject_pending_viewer", "reject"), +]) +def test_pending_viewer_decisions_reach_only_that_session(method, recorded): + host = _host() + first, _ = host.create_session_offer() + host.create_session_offer() + getattr(host, method)(first) + assert _FakeHost.instances[0].calls == [recorded] + assert _FakeHost.instances[1].calls == [] + + +def test_trusting_a_viewer_carries_the_label_through(): + host = _host() + session_id, _ = host.create_session_offer() + host.trust_pending_viewer(session_id, label="Ops laptop") + assert _FakeHost.instances[0].calls == [("trust", "Ops laptop")] + + +def test_pending_viewer_id_is_read_from_the_live_session(): + host = _host() + session_id, _ = host.create_session_offer() + _FakeHost.instances[0].pending_viewer_id = "viewer-7" + assert host.pending_viewer_id(session_id) == "viewer-7" + + +# --- permissions -------------------------------------------------------------- + +def test_permissions_propagate_to_live_sessions_and_to_the_next_one(): + host = _host() + host.create_session_offer() + permissions = SessionPermissions.from_read_only(True) + host.set_permissions(permissions) + assert _FakeHost.instances[0].permissions is permissions + host.create_session_offer() + assert _FakeHost.instances[1].kwargs["permissions"] is permissions + + +def test_set_read_only_is_the_shorthand_for_the_same_thing(): + host = _host() + host.create_session_offer() + host.set_read_only(True) + assert host.permissions.allow_input is False + assert _FakeHost.instances[0].permissions.allow_input is False + + +def test_one_dead_session_does_not_block_the_permission_broadcast(): + host = _host() + host.create_session_offer() + host.create_session_offer() + _FakeHost.instances[0].raise_on.add("set_permissions") + host.set_read_only(True) + assert _FakeHost.instances[1].permissions is not None + + +@pytest.mark.parametrize("method,recorded", [ + ("disable_accept_viewer_video", "disable_video"), + ("disable_accept_viewer_audio_opus", "disable_audio"), +]) +def test_disabling_an_inbound_slot_hits_every_session(method, recorded): + host = _host() + host.create_session_offer() + host.create_session_offer() + getattr(host, method)() + assert all(inst.calls == [recorded] for inst in _FakeHost.instances) + + +@pytest.mark.parametrize("method,failure", [ + ("disable_accept_viewer_video", "disable_video"), + ("disable_accept_viewer_audio_opus", "disable_audio"), +]) +def test_disabling_a_slot_survives_a_session_that_is_already_gone(method, + failure): + host = _host() + host.create_session_offer() + host.create_session_offer() + _FakeHost.instances[0].raise_on.add(failure) + getattr(host, method)() + assert _FakeHost.instances[1].calls, "the live session still got it" + + +# --- broadcast ---------------------------------------------------------------- + +def test_broadcast_reaches_only_authenticated_viewers(): + host = _host() + host.create_session_offer() + host.create_session_offer() + _FakeHost.instances[0].authenticated = True + assert host.broadcast_file("C:/report.txt", remote_name="r.txt") == 1 + assert _FakeHost.instances[0].pushed == [("C:/report.txt", "r.txt")] + assert _FakeHost.instances[1].pushed == [] + + +def test_broadcast_counts_the_viewers_that_actually_took_the_file(): + host = _host() + host.create_session_offer() + host.create_session_offer() + for inst in _FakeHost.instances: + inst.authenticated = True + _FakeHost.instances[0].raise_on.add("push_file") + assert host.broadcast_file("C:/report.txt") == 1 + + +def test_broadcast_with_no_sessions_sends_nothing(): + assert _host().broadcast_file("C:/report.txt") == 0 + + +# --- introspection ------------------------------------------------------------ + +def test_list_sessions_reports_live_state_per_viewer(): + host = _host() + session_id, _ = host.create_session_offer() + inst = _FakeHost.instances[0] + inst.authenticated = True + inst.connection_state = "connected" + inst.pending_viewer_id = "viewer-1" + [row] = host.list_sessions() + assert row["session_id"] == session_id + assert row["authenticated"] is True + assert row["state"] == "connected" + assert row["pending_viewer_id"] == "viewer-1" + assert row["connected_at"] is None, "nobody has authenticated yet" + + +def test_screen_track_is_none_before_any_viewer_arrives(): + assert _host().screen_track() is None + + +def test_first_session_pc_skips_sessions_that_have_no_connection_yet(): + host = _host() + host.create_session_offer() + host.create_session_offer() + pc = object() + _FakeHost.instances[1]._pc = pc + assert host.first_session_pc() is pc + + +def test_first_session_pc_is_none_when_nothing_is_connected(): + host = _host() + host.create_session_offer() + assert host.first_session_pc() is None + + +def test_session_pc_addresses_one_named_session(): + host = _host() + first, _ = host.create_session_offer() + second, _ = host.create_session_offer() + pc = object() + _FakeHost.instances[1]._pc = pc + assert host.session_pc(second) is pc + assert host.session_pc(first) is None + + +def test_session_pc_of_a_closed_session_is_none_rather_than_an_error(): + # The GUI polls this on a timer; a viewer that left between two ticks + # must not raise out of the timer slot. + assert _host().session_pc("gone") is None + + +# --- callback wrappers -------------------------------------------------------- + +def _fire(host_instance, key): + """Invoke the wrapper the coordinator handed to one fake session.""" + host_instance.kwargs[key]() + + +def test_state_changes_are_reported_with_their_session_id(): + seen = [] + host = _host(on_session_state=lambda sid, state: seen.append((sid, state))) + session_id, _ = host.create_session_offer() + _FakeHost.instances[0].kwargs["on_state_change"]("connected") + assert seen == [(session_id, "connected")] + + +def test_no_state_callback_means_no_wrapper_is_installed(): + # The single-viewer host checks this for None before calling it, so + # handing it a wrapper that calls nothing would only cost work. + host = _host() + host.create_session_offer() + assert _FakeHost.instances[0].kwargs["on_state_change"] is None + + +def test_authentication_stamps_the_connection_time_and_notifies(): + seen = [] + host = _host(on_session_authenticated=seen.append) + session_id, _ = host.create_session_offer() + _fire(_FakeHost.instances[0], "on_authenticated") + assert seen == [session_id] + [row] = host.list_sessions() + assert row["connected_at"], "an ISO timestamp minted at auth time" + + +def test_the_connection_time_is_stamped_even_without_a_listener(): + # `list_sessions` reports it to the GUI table regardless of whether + # anyone subscribed to the event, so the wrapper is always installed. + host = _host() + host.create_session_offer() + _fire(_FakeHost.instances[0], "on_authenticated") + assert host.list_sessions()[0]["connected_at"] + + +def test_a_pending_viewer_is_reported_with_the_id_read_at_fire_time(): + seen = [] + host = _host(on_pending_viewer=lambda sid, vid: seen.append((sid, vid))) + session_id, _ = host.create_session_offer() + # The id does not exist when the wrapper is built -- only when a viewer + # actually knocks -- so the wrapper has to look the host up on each call. + _FakeHost.instances[0].pending_viewer_id = "viewer-9" + _fire(_FakeHost.instances[0], "on_pending_viewer") + assert seen == [(session_id, "viewer-9")] + + +def test_a_pending_viewer_from_a_session_that_just_left_reports_none(): + seen = [] + host = _host(on_pending_viewer=lambda sid, vid: seen.append((sid, vid))) + session_id, _ = host.create_session_offer() + wrapper = _FakeHost.instances[0].kwargs["on_pending_viewer"] + host.stop_session(session_id) + wrapper() + assert seen == [(session_id, None)] + + +def test_no_pending_callback_means_no_wrapper_is_installed(): + host = _host() + host.create_session_offer() + assert _FakeHost.instances[0].kwargs["on_pending_viewer"] is None + + +@pytest.mark.parametrize("kwarg,wrapper_key", [ + ("on_session_state", "on_state_change"), + ("on_session_authenticated", "on_authenticated"), + ("on_pending_viewer", "on_pending_viewer"), +]) +def test_a_raising_gui_callback_never_reaches_the_session(kwarg, wrapper_key): + # These fire on the asyncio thread; an exception escaping one of them + # would kill the event loop that every other session shares. + def _boom(*_args): + raise RuntimeError("Qt widget already deleted") + + host = _host(**{kwarg: _boom}) + host.create_session_offer() + wrapper = _FakeHost.instances[0].kwargs[wrapper_key] + if wrapper_key == "on_state_change": + wrapper("connected") + else: + wrapper() diff --git a/test/unit_test/headless/test_webrtc_host_channels.py b/test/unit_test/headless/test_webrtc_host_channels.py new file mode 100644 index 00000000..81a8de50 --- /dev/null +++ b/test/unit_test/headless/test_webrtc_host_channels.py @@ -0,0 +1,739 @@ +"""What a connected viewer can make the host do, and what it cannot. + +The session setup is in `test_webrtc_host_session.py`; this file is about +the four DataChannels that ride on it, and it is mostly about refusal. Every +message here arrives from the other end of the wire, so each handler is a +boundary with the same three questions behind it: + +* **Has this viewer authenticated?** The channels open with the + PeerConnection, which is *before* the token is checked -- so a peer that + never authenticates can still push bytes at every one of them. +* **Do the session permissions allow it?** `read_only` is a shorthand over + five flags the operator can flip mid-session from the GUI, and each + channel consults a different one: input, files, audio. +* **Is it flooding?** The token buckets exist for a viewer that is + authenticated and permitted and still sending 10,000 events a second. + +The inbox handlers get the most attention because they are the ones that +take a *name* from the viewer and turn it into a path on the host's disk. +`_safe_basename` is what stands between `../../.ssh/authorized_keys` and the +host's home directory, and these tests pin down that the host actually +routes through it -- on the listing, the fetch and the delete alike. + +`get_bridge` and the audit log are replaced (from +`headless._webrtc_doubles`): the first would queue work onto a loop no test +is running, the second writes to the real `~/.je_auto_control`. +""" +from __future__ import annotations + +import asyncio +import json + +import pytest + +from headless._webrtc_doubles import ( + AuditLog, Bridge, Channel, FakePeerConnection, MicReceiver, +) +from je_auto_control.utils.remote_desktop import webrtc_host as host_module +from je_auto_control.utils.remote_desktop.permissions import SessionPermissions +from je_auto_control.utils.remote_desktop.rate_limit import RateLimitConfig +from je_auto_control.utils.remote_desktop.webrtc_host import WebRTCDesktopHost + + +@pytest.fixture(autouse=True) +def bridge(monkeypatch): + fake = Bridge() + monkeypatch.setattr(host_module, "get_bridge", lambda: fake) + return fake + + +@pytest.fixture(autouse=True) +def audit_log(monkeypatch): + log = AuditLog() + monkeypatch.setattr(host_module, "default_audit_log", lambda: log) + return log + + +@pytest.fixture +def host(tmp_path): + """An authenticated host with a control channel, and an empty inbox.""" + instance = WebRTCDesktopHost(token="secret", inbox_dir=tmp_path / "inbox") + channel = Channel() + instance._control_channel = channel + instance._wire_control_channel(channel) + instance._authenticated = True + return instance + + +def _sent(host_instance): + return [json.loads(text) for text in host_instance._control_channel.sent] + + +def _deliver(host_instance, payload): + host_instance._control_channel.fire("message", json.dumps(payload)) + + +def _revoke_files(host_instance): + """Drop to read-only, then forget the permissions envelope that sends.""" + host_instance.set_read_only(True) + host_instance._control_channel.sent.clear() + + +# --- the control envelope ----------------------------------------------------- + +@pytest.mark.parametrize("message", [ + b"\x00\x01", 42, None, ["input"], +]) +def test_a_non_text_control_message_is_ignored(host, message): + # The ctrl channel is text-only; binary on it is either a bug at the + # other end or someone probing. + host._control_channel.fire("message", message) + assert _sent(host) == [] + + +def test_a_malformed_envelope_is_dropped(host): + host._control_channel.fire("message", "{not json") + assert _sent(host) == [] + + +def test_a_json_scalar_is_not_an_envelope(host): + host._control_channel.fire("message", '"input"') + assert _sent(host) == [] + + +def test_an_unknown_message_type_is_ignored(host): + _deliver(host, {"type": "reboot_the_machine"}) + assert _sent(host) == [] + + +def test_an_envelope_with_no_type_is_ignored(host): + _deliver(host, {"payload": {"kind": "mouse"}}) + assert _sent(host) == [] + + +def test_an_unauthenticated_peer_can_only_send_auth(): + host = WebRTCDesktopHost(token="secret") + channel = Channel() + host._control_channel = channel + host._wire_control_channel(channel) + dispatched = [] + host._dispatch = dispatched.append + _deliver(host, {"type": "input", "payload": {"kind": "mouse"}}) + assert dispatched == [], "input before auth is not input, it is noise" + + +def test_an_unauthenticated_peer_reaches_the_token_check(): + host = WebRTCDesktopHost(token="secret") + channel = Channel() + host._control_channel = channel + host._wire_control_channel(channel) + _deliver(host, {"type": "auth", "token": "secret"}) + assert host.authenticated is True + + +def test_the_control_channel_open_event_is_survivable(host): + host._control_channel.fire("open") + + +def test_closing_the_control_channel_ends_the_session(host): + host._control_channel.fire("close") + assert host.authenticated is False + + +# --- input -------------------------------------------------------------------- + +def test_input_reaches_the_dispatcher(host): + dispatched = [] + host._dispatch = dispatched.append + _deliver(host, {"type": "input", "payload": {"kind": "mouse", "x": 3}}) + assert dispatched == [{"kind": "mouse", "x": 3}] + + +def test_input_is_refused_in_read_only_mode(host): + dispatched = [] + host._dispatch = dispatched.append + host.set_read_only(True) + _deliver(host, {"type": "input", "payload": {"kind": "mouse"}}) + assert dispatched == [] + + +def test_a_payload_that_is_not_a_dict_never_reaches_the_dispatcher(host): + dispatched = [] + host._dispatch = dispatched.append + _deliver(host, {"type": "input", "payload": "click"}) + assert dispatched == [] + + +def test_a_failing_dispatch_does_not_kill_the_channel(host): + # `dispatch_input` raises the whole AutoControl exception family plus + # OSError; anything escaping here would take the DataChannel bridge with + # it and end the session over one bad event. + def _boom(_payload): + raise ValueError("unknown key name") + + host._dispatch = _boom + _deliver(host, {"type": "input", "payload": {"kind": "key"}}) + _deliver(host, {"type": "input", "payload": {"kind": "key"}}) + + +def test_a_flood_of_input_is_dropped_and_audited(audit_log): + host = WebRTCDesktopHost( + token="secret", rate_limit=RateLimitConfig(input_burst=0), + ) + channel = Channel() + host._control_channel = channel + host._wire_control_channel(channel) + host._authenticated = True + dispatched = [] + host._dispatch = dispatched.append + _deliver(host, {"type": "input", "payload": {"kind": "mouse"}}) + assert dispatched == [] + assert [event for event, _ in audit_log.events] == ["rate_limit_input"] + + +def test_the_flood_audit_entry_is_written_once_per_window(audit_log): + # One log line per five-second window, not one per dropped event -- + # otherwise the flood is mirrored straight into the audit log. + host = WebRTCDesktopHost( + token="secret", rate_limit=RateLimitConfig(input_burst=0), + ) + channel = Channel() + host._control_channel = channel + host._wire_control_channel(channel) + host._authenticated = True + host._dispatch = lambda payload: None + for _ in range(5): + _deliver(host, {"type": "input", "payload": {"kind": "mouse"}}) + assert len(audit_log.events) == 1 + + +def test_an_audit_log_that_cannot_be_written_does_not_break_the_drop( + audit_log): + audit_log.error = OSError("disk full") + host = WebRTCDesktopHost( + token="secret", rate_limit=RateLimitConfig(input_burst=0), + ) + channel = Channel() + host._control_channel = channel + host._wire_control_channel(channel) + host._authenticated = True + host._dispatch = lambda payload: None + _deliver(host, {"type": "input", "payload": {"kind": "mouse"}}) + + +# --- the secure attention sequence -------------------------------------------- + +def test_send_sas_is_refused_in_read_only_mode(host, monkeypatch): + called = [] + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.session_actions" + ".send_secure_attention_sequence", + lambda: called.append(1), + ) + host.set_read_only(True) + _deliver(host, {"type": "send_sas"}) + assert called == [] + + +def test_send_sas_reaches_the_platform_call_when_permitted(host, monkeypatch): + called = [] + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.session_actions" + ".send_secure_attention_sequence", + lambda: called.append(1), + ) + _deliver(host, {"type": "send_sas"}) + assert called == [1] + assert _sent(host)[-1]["type"] == "sas_ok" + + +# --- annotations -------------------------------------------------------------- + +def test_an_annotation_reaches_the_gui_callback(): + seen = [] + host = WebRTCDesktopHost(token="secret", on_annotation=seen.append) + channel = Channel() + host._control_channel = channel + host._wire_control_channel(channel) + host._authenticated = True + _deliver(host, {"type": "annotate", "shape": "arrow"}) + assert seen == [{"type": "annotate", "shape": "arrow"}] + + +def test_an_annotation_with_no_listener_is_dropped(host): + _deliver(host, {"type": "annotate", "shape": "arrow"}) + + +def test_a_raising_annotation_callback_is_contained(): + def _boom(_data): + raise RuntimeError("Qt widget already deleted") + + host = WebRTCDesktopHost(token="secret", on_annotation=_boom) + channel = Channel() + host._control_channel = channel + host._wire_control_channel(channel) + host._authenticated = True + _deliver(host, {"type": "annotate"}) + + +# --- renegotiation ------------------------------------------------------------ + +def test_a_renegotiate_answer_without_a_connection_is_ignored(host): + _deliver(host, {"type": "renegotiate_answer", "sdp": "v=0\r\nanswer"}) + assert host._background_tasks == set() + + +def test_a_renegotiate_answer_with_no_sdp_is_ignored(host): + host._pc = object() + _deliver(host, {"type": "renegotiate_answer", "sdp": None}) + assert host._background_tasks == set() + + +def test_a_renegotiate_answer_is_applied_on_the_event_loop(host): + pc = FakePeerConnection() + + async def _drive(): + host._pc = pc + _deliver(host, {"type": "renegotiate_answer", "sdp": "v=0 answer"}) + # The handler runs on the DataChannel callback; applying the answer + # is awaited work, so it is spawned rather than run inline. + assert len(host._background_tasks) == 1 + await asyncio.gather(*host._background_tasks) + + asyncio.run(_drive()) + assert [d.sdp for d in pc.remote_descriptions] == ["v=0 answer"] + + +def test_applying_a_renegotiate_answer_resubscribes_the_viewer_media(host): + pc = FakePeerConnection() + host._pc = pc + asyncio.run(host._async_apply_renegotiate_answer("v=0\r\nanswer")) + assert [d.sdp for d in pc.remote_descriptions] == ["v=0\r\nanswer"] + + +def test_a_renegotiate_answer_that_aiortc_rejects_is_logged_not_raised(host): + pc = FakePeerConnection() + pc.remote_description_error = RuntimeError("invalid SDP") + host._pc = pc + asyncio.run(host._async_apply_renegotiate_answer("v=0\r\nnonsense")) + + +def test_applying_an_answer_without_a_connection_is_a_no_op(host): + asyncio.run(host._async_apply_renegotiate_answer("v=0\r\nanswer")) + + +# --- the mic channel ---------------------------------------------------------- + +def _wire_mic(host_instance): + channel = Channel("mic") + host_instance._wire_mic_channel(channel) + return channel + + +def test_mic_chunks_reach_the_receiver(host): + receiver = MicReceiver() + host._mic_receiver = receiver + channel = _wire_mic(host) + channel.fire("message", b"pcm") + assert receiver.chunks == [b"pcm"] + + +def test_mic_chunks_are_dropped_before_authentication(host): + receiver = MicReceiver() + host._mic_receiver = receiver + host._authenticated = False + _wire_mic(host).fire("message", b"pcm") + assert receiver.chunks == [] + + +def test_mic_chunks_are_dropped_when_nobody_is_listening(host): + # The channel exists from the moment the PeerConnection does; the + # receiver only exists once the operator turned the mic on. + _wire_mic(host).fire("message", b"pcm") + + +def test_mic_chunks_are_dropped_when_audio_is_not_permitted(host): + receiver = MicReceiver() + host._mic_receiver = receiver + host.set_permissions(SessionPermissions.none()) + _wire_mic(host).fire("message", b"pcm") + assert receiver.chunks == [] + + +def test_enabling_mic_receive_starts_one_receiver(host, monkeypatch): + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_mic.MicUplinkReceiver", + MicReceiver, + ) + host.enable_mic_receive() + first = host._mic_receiver + host.enable_mic_receive() + assert host._mic_receiver is first + assert first.started + + +def test_disabling_mic_receive_stops_it(host): + receiver = MicReceiver() + host._mic_receiver = receiver + host.disable_mic_receive() + assert receiver.stopped + assert host._mic_receiver is None + + +def test_disabling_a_mic_that_was_never_enabled_is_a_no_op(host): + host.disable_mic_receive() + assert host._mic_receiver is None + + +def test_a_mic_that_fails_to_close_is_still_forgotten(host): + receiver = MicReceiver() + receiver.stop_error = OSError("stream already closed") + host._mic_receiver = receiver + host.disable_mic_receive() + assert host._mic_receiver is None + + +# --- the files channel -------------------------------------------------------- + +def _file_begin(name="report.txt", size=4): + return json.dumps({"type": "file_begin", "name": name, "size": size}) + + +def test_an_incoming_file_lands_in_the_inbox(host): + channel = Channel("files") + host._wire_files_channel(channel) + channel.fire("message", _file_begin()) + channel.fire("message", b"data") + channel.fire("message", json.dumps({"type": "file_end"})) + assert (host._files_receiver._inbox / "report.txt").read_bytes() == b"data" + + +def test_an_unauthenticated_peer_cannot_write_to_the_inbox(host): + host._authenticated = False + channel = Channel("files") + host._wire_files_channel(channel) + channel.fire("message", _file_begin()) + channel.fire("message", b"data") + assert list(host._files_receiver._inbox.iterdir()) == [] + + +def test_a_viewer_without_file_permission_cannot_write_to_the_inbox(host): + host.set_read_only(True) + channel = Channel("files") + host._wire_files_channel(channel) + channel.fire("message", _file_begin()) + assert list(host._files_receiver._inbox.iterdir()) == [] + + +def test_a_flood_of_transfers_is_dropped_and_audited(audit_log, tmp_path): + host = WebRTCDesktopHost( + token="secret", inbox_dir=tmp_path / "inbox", + rate_limit=RateLimitConfig(files_burst=0), + ) + host._authenticated = True + channel = Channel("files") + host._wire_files_channel(channel) + channel.fire("message", _file_begin()) + assert list(host._files_receiver._inbox.iterdir()) == [] + assert [event for event, _ in audit_log.events] == ["rate_limit_files"] + + +def test_chunks_of_an_allowed_transfer_are_not_rate_limited(tmp_path): + # The bucket counts transfers, not bytes: a file already accepted must + # not stall halfway through because its chunks exhausted the same + # bucket its envelope came out of. + host = WebRTCDesktopHost( + token="secret", inbox_dir=tmp_path / "inbox", + rate_limit=RateLimitConfig(files_per_minute=60.0, files_burst=1), + ) + host._authenticated = True + channel = Channel("files") + host._wire_files_channel(channel) + channel.fire("message", _file_begin(size=6)) + for chunk in (b"ab", b"cd", b"ef"): + channel.fire("message", chunk) + channel.fire("message", json.dumps({"type": "file_end"})) + assert (host._files_receiver._inbox / "report.txt").read_bytes() == b"abcdef" + + +def test_the_transfer_flood_audit_entry_is_written_once_per_window(audit_log, + tmp_path): + # Same rule as the input bucket: one line per five-second window, not + # one per refused transfer. + host = WebRTCDesktopHost( + token="secret", inbox_dir=tmp_path / "inbox", + rate_limit=RateLimitConfig(files_burst=0), + ) + host._authenticated = True + channel = Channel("files") + host._wire_files_channel(channel) + for _ in range(4): + channel.fire("message", _file_begin()) + assert len(audit_log.events) == 1 + + +def test_a_completed_transfer_is_audited_and_announced(host, audit_log): + seen = [] + host.set_file_received_callback(seen.append) + channel = Channel("files") + host._wire_files_channel(channel) + channel.fire("message", _file_begin()) + channel.fire("message", b"data") + channel.fire("message", json.dumps({"type": "file_end"})) + assert [event for event, _ in audit_log.events] == ["file_received"] + assert seen and seen[0].name == "report.txt" + + +def test_a_raising_file_callback_does_not_lose_the_file(host): + def _boom(_path): + raise RuntimeError("Qt widget already deleted") + + host.set_file_received_callback(_boom) + host._on_file_done(host._inbox_dir) + + +def test_an_audit_failure_still_lets_the_callback_run(host, audit_log): + audit_log.error = OSError("disk full") + seen = [] + host.set_file_received_callback(seen.append) + host._on_file_done("C:/inbox/report.txt") + assert seen == ["C:/inbox/report.txt"] + + +def test_pushing_a_file_before_a_viewer_connects_is_refused(host): + host._files_channel = None + with pytest.raises(RuntimeError, match="not connected"): + host.push_file("C:/report.txt") + + +def test_pushing_a_file_to_an_unauthenticated_peer_is_refused(host): + host._files_channel = Channel("files") + host._authenticated = False + with pytest.raises(RuntimeError, match="not connected"): + host.push_file("C:/report.txt") + + +def test_pushing_a_file_streams_it_over_the_files_channel(host, tmp_path): + source = tmp_path / "notes.txt" + source.write_bytes(b"hello") + host._files_channel = Channel("files") + host.push_file(str(source), remote_name="renamed.txt") + envelope = json.loads(host._files_channel.sent[0]) + assert envelope["type"] == "file_begin" + assert envelope["name"] == "renamed.txt" + + +# --- the usb channel ---------------------------------------------------------- + +def test_the_usb_channel_is_gated_on_auth_and_the_global_opt_in(host, + monkeypatch): + channel = Channel("usb") + host._wire_usb_channel(channel) + gate = host._usb_host._enabled + + monkeypatch.setattr( + "je_auto_control.utils.usb.passthrough.is_usb_passthrough_enabled", + lambda: False, + ) + assert gate() is False, "authenticated is not enough; it is opt-in" + + monkeypatch.setattr( + "je_auto_control.utils.usb.passthrough.is_usb_passthrough_enabled", + lambda: True, + ) + assert gate() is True + + host._authenticated = False + assert gate() is False, "the opt-in is not enough either" + + +def test_the_usb_session_is_built_with_a_default_deny_acl(host, monkeypatch): + import je_auto_control.utils.usb.passthrough as passthrough + + built = {} + + def _session(backend, acl=None, viewer_id=None): + built.update(backend=backend, acl=acl, viewer_id=viewer_id) + return "session" + + monkeypatch.setattr(passthrough, "UsbPassthroughSession", _session) + monkeypatch.setattr(passthrough, "default_passthrough_backend", + lambda: "backend") + channel = Channel("usb") + host._viewer_id = "viewer-3" + host._wire_usb_channel(channel) + assert host._usb_host._factory() == "session" + assert built["backend"] == "backend" + assert built["viewer_id"] == "viewer-3" + assert built["acl"] is not None, "never an unrestricted session" + + +# --- the inbox listing -------------------------------------------------------- + +def _inbox(host_instance): + return host_instance._ensure_files_receiver()._inbox + + +def test_listing_the_inbox_reports_name_size_and_mtime(host): + (_inbox(host) / "a.txt").write_bytes(b"12345") + _deliver(host, {"type": "list_inbox"}) + [response] = _sent(host) + assert response["type"] == "list_inbox_response" + [entry] = response["files"] + assert entry["name"] == "a.txt" + assert entry["size"] == 5 + assert "mtime" in entry + + +def test_listing_the_inbox_skips_directories(host): + (_inbox(host) / "sub").mkdir() + (_inbox(host) / "a.txt").write_bytes(b"1") + _deliver(host, {"type": "list_inbox"}) + assert [f["name"] for f in _sent(host)[0]["files"]] == ["a.txt"] + + +def test_listing_the_inbox_without_file_permission_is_refused(host): + (_inbox(host) / "a.txt").write_bytes(b"1") + _revoke_files(host) + _deliver(host, {"type": "list_inbox"}) + [response] = _sent(host) + assert response["files"] == [] + assert response["error"] == "files not permitted" + + +def test_an_unreadable_inbox_is_reported_as_an_error(host, monkeypatch): + receiver = host._ensure_files_receiver() + monkeypatch.setattr( + type(receiver._inbox), "iterdir", + lambda self: (_ for _ in ()).throw(OSError("permission denied")), + ) + _deliver(host, {"type": "list_inbox"}) + [response] = _sent(host) + assert response["files"] == [] + assert "permission denied" in response["error"] + + +def test_the_inbox_receiver_is_built_once_and_reused(host): + assert host._ensure_files_receiver() is host._ensure_files_receiver() + + +# --- fetching a file back ----------------------------------------------------- + +def test_requesting_a_file_pushes_it_over_the_files_channel(host): + (_inbox(host) / "a.txt").write_bytes(b"12345") + host._files_channel = Channel("files") + _deliver(host, {"type": "request_file", "name": "a.txt"}) + envelope = json.loads(host._files_channel.sent[0]) + assert (envelope["type"], envelope["name"]) == ("file_begin", "a.txt") + + +def test_requesting_a_missing_file_is_answered_not_ignored(host): + _deliver(host, {"type": "request_file", "name": "gone.txt"}) + [response] = _sent(host) + assert response == {"type": "request_file_response", "name": "gone.txt", + "ok": False, "error": "not found"} + + +def test_requesting_a_path_cannot_escape_the_inbox(host, tmp_path): + # The name arrives from the viewer; `..` in it must resolve to a + # basename inside the inbox rather than to the host's own files. + outside = tmp_path / "secret.txt" + outside.write_bytes(b"private") + _deliver(host, {"type": "request_file", "name": "../secret.txt"}) + [response] = _sent(host) + assert response["ok"] is False + assert response["name"] == "secret.txt", "sanitized to a bare basename" + + +def test_requesting_a_file_without_permission_is_silently_refused(host): + (_inbox(host) / "a.txt").write_bytes(b"1") + _revoke_files(host) + _deliver(host, {"type": "request_file", "name": "a.txt"}) + assert _sent(host) == [] + + +def test_a_request_with_a_non_string_name_is_ignored(host): + _deliver(host, {"type": "request_file", "name": {"path": "a.txt"}}) + assert _sent(host) == [] + + +def test_an_unusable_filename_is_reported_rather_than_raised(host): + _deliver(host, {"type": "request_file", "name": "co:n<>|.txt"}) + [response] = _sent(host) + assert response["type"] == "request_file_response" + assert response["ok"] is False + + +# --- deleting a file ---------------------------------------------------------- + +def test_deleting_an_inbox_file_removes_it(host): + target = _inbox(host) / "a.txt" + target.write_bytes(b"1") + _deliver(host, {"type": "delete_inbox_file", "name": "a.txt"}) + assert not target.exists() + assert _sent(host) == [{"type": "delete_inbox_response", "name": "a.txt", + "ok": True}] + + +def test_deleting_a_missing_file_is_answered_with_the_reason(host): + _deliver(host, {"type": "delete_inbox_file", "name": "gone.txt"}) + [response] = _sent(host) + assert response["ok"] is False + assert response["error"] + + +def test_deleting_without_permission_is_refused_with_a_reason(host): + target = _inbox(host) / "a.txt" + target.write_bytes(b"1") + host.set_read_only(True) + _deliver(host, {"type": "delete_inbox_file", "name": "a.txt"}) + assert target.exists() + assert _sent(host)[-1]["error"] == "files not permitted" + + +def test_a_delete_with_a_non_string_name_is_ignored(host): + _deliver(host, {"type": "delete_inbox_file", "name": 7}) + assert _sent(host) == [] + + +def test_deleting_a_path_cannot_escape_the_inbox(host, tmp_path): + outside = tmp_path / "secret.txt" + outside.write_bytes(b"private") + _deliver(host, {"type": "delete_inbox_file", "name": "../secret.txt"}) + assert outside.exists(), "the delete stayed inside the inbox" + assert _sent(host)[-1]["ok"] is False + + +# --- permissions and outbound sends ------------------------------------------- + +def test_changing_permissions_tells_the_viewer(host): + host.set_permissions(SessionPermissions.view_only()) + assert _sent(host)[-1] == { + "type": "permissions", + "value": SessionPermissions.view_only().to_dict(), + } + + +def test_read_only_is_derived_from_the_input_flag(host): + host.set_permissions(SessionPermissions.view_only()) + assert host.read_only is True + host.set_permissions(SessionPermissions.full_control()) + assert host.read_only is False + + +def test_sending_before_the_channel_exists_is_a_no_op(bridge): + host = WebRTCDesktopHost(token="secret") + host._send_ctrl({"type": "permissions"}) + assert bridge.deferred == [] + + +def test_a_channel_that_fails_mid_send_does_not_raise(host): + host._control_channel.send_error = OSError("channel closed") + host.set_permissions(SessionPermissions.view_only()) + + +def test_a_channel_closed_between_queue_and_send_is_a_no_op(host): + # `_send_ctrl` hops onto the asyncio loop, so the channel can go away + # between the two halves; `_safe_channel_send` re-checks for exactly that. + host._control_channel = None + host._safe_channel_send("{}") diff --git a/test/unit_test/headless/test_webrtc_host_session.py b/test/unit_test/headless/test_webrtc_host_session.py new file mode 100644 index 00000000..46cc5d68 --- /dev/null +++ b/test/unit_test/headless/test_webrtc_host_session.py @@ -0,0 +1,558 @@ +"""Building, renegotiating and tearing down a host session. + +`WebRTCDesktopHost` is the largest thing under `utils/remote_desktop` and, +until the `[webrtc]` extra joined the measured install, one of the least +covered: it reaches `webrtc_transport`, which raises ImportError at module +level without aiortc. Its two mixins already have tests +(`test_webrtc_host_auth.py`, `test_webrtc_host_media.py`); this file covers +the class those are mixed into. + +It splits along the seam the class itself has. Here: the session -- offer +construction, answer application, the connection-state handlers, and +teardown. The DataChannel traffic that rides on top is in +`test_webrtc_host_channels.py`. + +`RTCPeerConnection` and `ScreenVideoTrack` are replaced, from +`headless._webrtc_doubles`. A real one of either would open a screen +grabber and start STUN traffic on a CI runner, +and neither is what these tests are about: what matters is the *shape* of +the offer the host builds -- which transceivers it adds for which config, +which four DataChannels it opens, and in what order -- because that shape +is what the viewer's `_attach_viewer_screen_track` counts m-lines in. + +Teardown gets the same attention as setup for one reason: the host does +not own everything it holds. A relayed track belongs to `MultiViewerHost`, +and stopping it there would blank the screen for every *other* viewer. +""" +from __future__ import annotations + +import asyncio + +import pytest + +from headless._webrtc_doubles import ( + AuditLog, Bridge, FakePeerConnection, FrameTrack, HangingBridge, + Stoppable, Track, noop_ice_gathering, +) +from je_auto_control.utils.remote_desktop import webrtc_host as host_module +from je_auto_control.utils.remote_desktop.permissions import SessionPermissions +from je_auto_control.utils.remote_desktop.webrtc_host import WebRTCDesktopHost +from je_auto_control.utils.remote_desktop.webrtc_transport import WebRTCConfig + + +@pytest.fixture +def bridge(monkeypatch): + fake = Bridge() + monkeypatch.setattr(host_module, "get_bridge", lambda: fake) + return fake + + +@pytest.fixture(autouse=True) +def fake_peer_connection(monkeypatch): + """Replace the two collaborators that would touch a screen or a socket.""" + FakePeerConnection.instances = [] + monkeypatch.setattr(host_module, "RTCPeerConnection", FakePeerConnection) + monkeypatch.setattr(host_module, "ScreenVideoTrack", + lambda **kwargs: Track("video")) + monkeypatch.setattr(host_module, "wait_for_ice_gathering", + noop_ice_gathering) + yield + FakePeerConnection.instances = [] + + +@pytest.fixture(autouse=True) +def fake_audit_log(monkeypatch): + log = AuditLog() + monkeypatch.setattr(host_module, "default_audit_log", lambda: log) + return log + + +def _host(**kwargs) -> WebRTCDesktopHost: + kwargs.setdefault("token", "secret") + return WebRTCDesktopHost(**kwargs) + + +# --- construction ------------------------------------------------------------- + +def test_a_host_without_a_token_is_refused(): + with pytest.raises(ValueError, match="non-empty token"): + WebRTCDesktopHost(token="") + + +def test_read_only_shorthand_becomes_a_permission_set(): + assert _host(read_only=True).read_only is True + assert _host(read_only=True).permissions.allow_files is False + assert _host().read_only is False + + +def test_explicit_permissions_win_over_the_shorthand(): + permissions = SessionPermissions.view_only() + assert _host(read_only=False, permissions=permissions).permissions is ( + permissions + ) + + +def test_a_fresh_host_is_neither_authenticated_nor_connected(): + host = _host() + assert host.authenticated is False + assert host.connection_state == "closed", "no PeerConnection yet" + + +# --- building the offer ------------------------------------------------------- + +def test_create_offer_returns_the_local_sdp(bridge): + host = _host() + assert host.create_offer() == "v=0 local-sdp" + assert host.connection_state == "new" + + +def test_create_offer_asks_the_consent_callback_first(bridge): + seen = [] + + def _consent(peer_label): + seen.append(peer_label) + return True + + _host(offer_consent=_consent).create_offer(peer_label="Ops laptop") + assert seen == ["Ops laptop"] + + +def test_a_rejected_offer_never_builds_a_peer_connection(bridge): + host = _host(offer_consent=lambda peer: False) + with pytest.raises(PermissionError, match="rejected by consent"): + host.create_offer() + assert not FakePeerConnection.instances, "no capture, no ICE, nothing started" + + +def test_the_offer_carries_the_configured_ice_servers(bridge): + config = WebRTCConfig(ice_servers=["stun:example:3478"]) + _host(config=config).create_offer() + [pc] = FakePeerConnection.instances + assert [s.urls for s in pc.configuration.iceServers] == [ + "stun:example:3478", + ] + + +def test_the_offer_opens_the_four_data_channels_the_viewer_expects(bridge): + _host().create_offer() + [pc] = FakePeerConnection.instances + assert [c.label for c in pc.channels] == ["ctrl", "mic", "files", "usb"] + + +def test_the_screen_track_is_the_first_media_line(bridge): + # The viewer identifies the slot for its own screen by m-line order, so + # the host's outbound track has to be added before any recvonly slot. + config = WebRTCConfig(accept_viewer_video=True) + _host(config=config).create_offer() + [pc] = FakePeerConnection.instances + assert len(pc.tracks) == 1 + assert pc.transceivers == [("video", "recvonly")] + + +def test_no_inbound_slots_are_advertised_by_default(bridge): + _host().create_offer() + assert FakePeerConnection.instances[0].transceivers == [] + + +def test_accepting_viewer_audio_advertises_a_recvonly_audio_slot(bridge): + config = WebRTCConfig(accept_viewer_audio_opus=True) + _host(config=config).create_offer() + assert FakePeerConnection.instances[0].transceivers == [("audio", "recvonly")] + + +def test_an_external_track_is_used_as_is(bridge): + # `MultiViewerHost` hands each session a relay proxy of one shared + # capture; building a second grabber here would be a second screen read + # per viewer. + relayed = Track("video") + _host(external_video_track=relayed).create_offer() + assert FakePeerConnection.instances[0].tracks == [relayed] + + +def test_host_voice_attaches_a_microphone_track(bridge, monkeypatch): + mic = Track("audio") + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicAudioTrack", + lambda: mic, + ) + _host(config=WebRTCConfig(host_voice=True)).create_offer() + assert mic in FakePeerConnection.instances[0].tracks + + +def test_a_host_with_no_microphone_still_gets_an_offer(bridge, monkeypatch): + # No input device is the normal state of a server; the screen share must + # not be lost because the mic could not be opened. + def _no_device(): + raise OSError("no input device") + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicAudioTrack", + _no_device, + ) + host = _host(config=WebRTCConfig(host_voice=True)) + assert host.create_offer() == "v=0 local-sdp" + assert host._host_voice_track is None + + +def test_a_second_offer_closes_the_first_peer_connection(bridge): + host = _host() + host.create_offer() + host.create_offer() + first, second = FakePeerConnection.instances + assert first.closed + assert not second.closed + + +# --- applying the answer ------------------------------------------------------ + +@pytest.mark.parametrize("answer", ["", " "]) +def test_an_empty_answer_is_refused_before_it_reaches_aiortc(bridge, answer): + host = _host() + host.create_offer() + with pytest.raises(ValueError, match="answer_sdp is empty"): + host.accept_answer(answer) + + +def test_accept_answer_before_create_offer_is_a_runtime_error(bridge): + with pytest.raises(RuntimeError, match="create_offer"): + _host().accept_answer("v=0 answer") + + +def test_accept_answer_applies_it_and_arms_the_auth_deadline(bridge): + host = _host() + host.create_offer() + host.accept_answer("v=0 answer") + [pc] = FakePeerConnection.instances + [description] = pc.remote_descriptions + assert (description.sdp, description.type) == ("v=0 answer", "answer") + assert host._auth_deadline_handle is not None, "the grace period is armed" + + +# --- connection state --------------------------------------------------------- + +def test_the_state_callback_sees_every_transition(bridge): + seen = [] + host = _host(on_state_change=seen.append) + host.create_offer() + pc = FakePeerConnection.instances[0] + for state in ("connecting", "connected", "failed"): + pc.connectionState = state + asyncio.run(pc.fire("connectionstatechange")) + assert seen == ["connecting", "connected", "failed"] + + +@pytest.mark.parametrize("state", ["failed", "closed", "disconnected"]) +def test_a_lost_connection_drops_the_authenticated_flag(bridge, state): + # Anything that reconnects starts a new session and must authenticate + # again; leaving the flag set would let a reused channel skip the token. + host = _host() + host.create_offer() + host._authenticated = True + pc = FakePeerConnection.instances[0] + pc.connectionState = state + asyncio.run(pc.fire("connectionstatechange")) + assert host.authenticated is False + + +def test_a_raising_state_callback_does_not_break_the_handler(bridge): + def _boom(_state): + raise RuntimeError("Qt widget already deleted") + + host = _host(on_state_change=_boom) + host.create_offer() + pc = FakePeerConnection.instances[0] + pc.connectionState = "connected" + asyncio.run(pc.fire("connectionstatechange")) + + +def test_connecting_snapshots_the_remote_ip_from_the_selected_pair(bridge): + host = _host() + host.create_offer() + pc = FakePeerConnection.instances[0] + pc.stats = _stats_with_selected_pair(ip="203.0.113.9") + pc.connectionState = "connected" + asyncio.run(pc.fire("connectionstatechange")) + assert host._remote_ip == "203.0.113.9" + + +def _stat(**fields): + return type("_Stat", (), fields)() + + +def _stats_with_selected_pair(*, ip=None, address=None, remote_id="remote-1", + include_remote=True): + stats = {"pair-1": _stat(type="candidate-pair", selected=True, + remoteCandidateId=remote_id)} + if include_remote: + fields = {} + if ip is not None: + fields["ip"] = ip + if address is not None: + fields["address"] = address + stats[remote_id] = _stat(**fields) + return stats + + +def test_the_remote_ip_falls_back_to_the_address_field(): + # aiortc renamed `ip` to `address` following the spec; both spellings + # turn up depending on the version installed. + report = _stats_with_selected_pair(address="198.51.100.4") + assert WebRTCDesktopHost._extract_remote_ip(report) == "198.51.100.4" + + +def test_an_unselected_candidate_pair_is_not_the_remote_peer(): + report = {"pair-1": _stat(type="candidate-pair", selected=False, + remoteCandidateId="remote-1")} + assert WebRTCDesktopHost._extract_remote_ip(report) is None + + +def test_a_dangling_candidate_reference_yields_no_ip(): + report = _stats_with_selected_pair(include_remote=False) + assert WebRTCDesktopHost._extract_remote_ip(report) is None + + +def test_a_candidate_with_neither_ip_nor_address_yields_no_ip(): + report = _stats_with_selected_pair() + assert WebRTCDesktopHost._extract_remote_ip(report) is None + + +def test_stats_that_cannot_be_read_leave_the_ip_unknown(bridge): + # The IP feeds the whitelist check and the audit log; failing to read it + # must not fail the connection. + host = _host() + host.create_offer() + FakePeerConnection.instances[0].stats_error = RuntimeError("pc already closed") + asyncio.run(host._snapshot_remote_ip()) + assert host._remote_ip is None + + +def test_snapshotting_without_a_peer_connection_is_a_no_op(): + asyncio.run(_host()._snapshot_remote_ip()) + + +# --- inbound viewer media ----------------------------------------------------- + +def test_an_inbound_video_track_starts_a_consume_task(bridge): + host = _host() + host.create_offer() + + async def _drive(): + FakePeerConnection.instances[0].fire("track", Track("video")) + assert host._viewer_video_task is not None + host._viewer_video_task.cancel() + + asyncio.run(_drive()) + + +def test_an_inbound_audio_track_is_ignored_unless_it_was_advertised(bridge): + host = _host() + host.create_offer() + FakePeerConnection.instances[0].fire("track", Track("audio")) + assert host._opus_audio_receiver is None + + +def test_an_advertised_audio_track_starts_an_opus_receiver(bridge, + monkeypatch): + receivers = [] + + class _Receiver: + def __init__(self) -> None: + self.consumed = [] + receivers.append(self) + + def consume(self, track): + self.consumed.append(track) + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicReceiver", + _Receiver, + ) + host = _host(config=WebRTCConfig(accept_viewer_audio_opus=True)) + host.create_offer() + track = Track("audio") + FakePeerConnection.instances[0].fire("track", track) + assert receivers[0].consumed == [track] + + +def test_a_second_audio_track_does_not_open_a_second_receiver(bridge, + monkeypatch): + class _Receiver: + def __init__(self) -> None: + self.consumed = [] + + def consume(self, track): + self.consumed.append(track) + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicReceiver", + _Receiver, + ) + host = _host(config=WebRTCConfig(accept_viewer_audio_opus=True)) + host.create_offer() + pc = FakePeerConnection.instances[0] + pc.fire("track", Track("audio")) + first = host._opus_audio_receiver + pc.fire("track", Track("audio")) + assert host._opus_audio_receiver is first + + +def test_a_missing_speaker_does_not_break_the_session(bridge, monkeypatch): + def _no_device(): + raise OSError("no output device") + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicReceiver", + _no_device, + ) + host = _host(config=WebRTCConfig(accept_viewer_audio_opus=True)) + host.create_offer() + FakePeerConnection.instances[0].fire("track", Track("audio")) + assert host._opus_audio_receiver is None + + +def test_viewer_frames_are_dropped_until_the_viewer_authenticates(): + # The video slot opens with the PeerConnection, which is before the + # token has been checked: frames arriving in that window are a stream + # from a peer we have not accepted yet. + host = _host() + seen = [] + host.set_viewer_video_callback(seen.append) + asyncio.run(host._consume_viewer_video(FrameTrack("f1", "f2"))) + assert seen == [] + + +def test_viewer_frames_reach_the_callback_once_authenticated(): + host = _host() + seen = [] + host.set_viewer_video_callback(seen.append) + host._authenticated = True + asyncio.run(host._consume_viewer_video(FrameTrack("f1", "f2"))) + assert seen == ["f1", "f2"] + + +def test_viewer_frames_with_no_listener_are_drained_and_discarded(): + # Nothing is registered until the GUI opens the viewer-screen window; + # the frames still have to be pulled or the receiver backs up. + host = _host() + host._authenticated = True + track = FrameTrack("f1", "f2") + asyncio.run(host._consume_viewer_video(track)) + assert track.frames == [] + + +def test_a_track_that_is_neither_audio_nor_video_is_ignored(bridge): + host = _host() + host.create_offer() + FakePeerConnection.instances[0].fire("track", Track("application")) + assert host._viewer_video_task is None + assert host._opus_audio_receiver is None + + +def test_a_raising_frame_callback_does_not_end_the_stream(): + host = _host() + host._authenticated = True + seen = [] + + def _cb(frame): + seen.append(frame) + raise RuntimeError("paint failed") + + host.set_viewer_video_callback(_cb) + asyncio.run(host._consume_viewer_video(FrameTrack("f1", "f2"))) + assert seen == ["f1", "f2"], "the second frame was still delivered" + + +def test_the_consume_task_clears_itself_when_the_stream_ends(): + host = _host() + host._viewer_video_task = "placeholder" + asyncio.run(host._consume_viewer_video(FrameTrack(ending=OSError("gone")))) + assert host._viewer_video_task is None + + +# --- teardown ----------------------------------------------------------------- + +def test_stopping_a_host_that_never_connected_is_a_no_op(bridge): + _host().stop() + assert not bridge.deferred + + +def test_stop_closes_the_peer_connection_and_forgets_the_channels(bridge): + host = _host() + host.create_offer() + host._authenticated = True + host.stop() + assert FakePeerConnection.instances[0].closed + assert host._pc is None + assert host._control_channel is None + assert host._files_channel is None + assert host.authenticated is False + assert host.connection_state == "closed" + + +def test_stop_stops_a_capture_the_host_created(): + host = _host() + track = Track("video") + host._video_track = track + asyncio.run(host._async_stop()) + assert track.stopped + + +def test_stop_leaves_a_relayed_track_alone(): + # `MultiViewerHost` owns the shared capture; stopping it from one + # session would blank the screen for every other viewer. + relayed = Track("video") + host = _host(external_video_track=relayed) + host._video_track = relayed + asyncio.run(host._async_stop()) + assert not relayed.stopped + + +def test_stop_releases_the_audio_and_mic_helpers(): + host = _host() + voice, receiver, mic = Stoppable(), Stoppable(), Stoppable() + host._host_voice_track = voice + host._opus_audio_receiver = receiver + host._mic_receiver = mic + asyncio.run(host._async_stop()) + assert (voice.stopped, receiver.stopped, mic.stopped) == (True, True, True) + assert host._host_voice_track is None + assert host._opus_audio_receiver is None + assert host._mic_receiver is None + + +def test_stop_cancels_the_viewer_video_task_and_the_auth_deadline(): + host = _host() + + async def _drive(): + task = asyncio.ensure_future(asyncio.sleep(10)) + handle = asyncio.get_event_loop().call_later(10, lambda: None) + host._viewer_video_task = task + host._auth_deadline_handle = handle + await host._async_stop() + return task, handle + + task, handle = asyncio.run(_drive()) + assert task.cancelled() + assert handle.cancelled() + assert host._auth_deadline_handle is None + + +def test_a_teardown_failure_does_not_abort_the_rest_of_the_teardown(): + host = _host() + voice = Stoppable(error=OSError("device gone")) + receiver = Stoppable() + host._host_voice_track = voice + host._opus_audio_receiver = receiver + asyncio.run(host._async_stop()) + assert receiver.stopped, "teardown continued past the failure" + + +def test_stop_quietly_ignores_a_collaborator_that_was_never_built(): + WebRTCDesktopHost._stop_quietly(None, "nothing") + + +def test_a_stop_that_times_out_is_reported_rather_than_raised(monkeypatch): + monkeypatch.setattr(host_module, "get_bridge", HangingBridge) + host = _host() + host._pc = object() + host.stop() # the GUI's close button must not raise diff --git a/test/unit_test/headless/test_webrtc_opus_audio.py b/test/unit_test/headless/test_webrtc_opus_audio.py new file mode 100644 index 00000000..0425c996 --- /dev/null +++ b/test/unit_test/headless/test_webrtc_opus_audio.py @@ -0,0 +1,395 @@ +"""The Opus mic uplink: a sounddevice thread on one end, asyncio on the other. + +`webrtc_audio` sits on a thread boundary in both directions and had no test +at all -- it needs aiortc *and* av to import, so it read 0% on every CI +square until the `[webrtc]` extra joined the measured install. + +What the tests below are actually about: + +* **The capture callback runs on the sounddevice thread.** It may not touch + the asyncio queue directly, so it hops through `call_soon_threadsafe` -- + and when the loop has already closed under it (the viewer disconnected + while a block was in flight) the `RuntimeError` that raises is the normal + case, not an error to report. +* **The queue drops the oldest block, not the newest.** A bounded queue is + what keeps mic latency from growing without limit when the encoder falls + behind; dropping the *newest* block would bound the queue just as well + and make the audio permanently stale. +* **`recv` mints the presentation timestamps itself.** aiortc packetises + what it is handed, so a pts that does not advance by exactly the sample + count is a stream that drifts against its own clock. +* **The receiver's drain loop is a containment boundary.** It runs as a + fire-and-forget task on the shared bridge loop; every way a track can end + -- cancellation, `MediaStreamError`, a decode failure, the player being + stopped underneath it -- has to end the loop quietly rather than take the + loop down with it. + +`AudioCapture` and `AudioPlayer` are replaced with recorders: they are the +sounddevice boundary, and sounddevice is not installed on a CI runner. The +frames are real `av.AudioFrame`s, because their layout is the contract +between this module and aiortc. +""" +from __future__ import annotations + +import asyncio + +import numpy as np +import pytest + +from headless._webrtc_doubles import FrameTrack +from je_auto_control.utils.remote_desktop import webrtc_audio as audio_mod +from je_auto_control.utils.remote_desktop.audio import AudioBackendError +from je_auto_control.utils.remote_desktop.webrtc_audio import ( + OpusMicAudioTrack, OpusMicReceiver, +) + + +class _FakeCapture: + instances = [] + + def __init__(self, *, on_block, device, sample_rate, channels, + block_frames) -> None: + self.on_block = on_block + self.device = device + self.sample_rate = sample_rate + self.channels = channels + self.block_frames = block_frames + self.started = False + self.stopped = False + self.stop_error = None + _FakeCapture.instances.append(self) + + def start(self) -> None: + self.started = True + + def stop(self) -> None: + if self.stop_error is not None: + raise self.stop_error + self.stopped = True + + +class _FakePlayer: + instances = [] + + def __init__(self, *, device, sample_rate, channels) -> None: + self.device = device + self.sample_rate = sample_rate + self.channels = channels + self.is_running = False + self.played = [] + self.stopped = False + self.stop_error = None + _FakePlayer.instances.append(self) + + def start(self) -> None: + self.is_running = True + + def play(self, pcm_bytes) -> None: + self.played.append(pcm_bytes) + + def stop(self) -> None: + if self.stop_error is not None: + raise self.stop_error + self.stopped = True + + +class _Frame: + """Stands in for an ``av.AudioFrame`` the decoder handed us.""" + + def __init__(self, array=None, error=None) -> None: + self._array = array + self._error = error + + def to_ndarray(self): + if self._error is not None: + raise self._error + return self._array + + +@pytest.fixture(autouse=True) +def fake_audio_backend(monkeypatch): + """Swap the sounddevice boundary out; keep everything above it real.""" + _FakeCapture.instances = [] + _FakePlayer.instances = [] + monkeypatch.setattr(audio_mod, "is_audio_backend_available", lambda: True) + monkeypatch.setattr(audio_mod, "AudioCapture", _FakeCapture) + monkeypatch.setattr(audio_mod, "AudioPlayer", _FakePlayer) + yield + _FakeCapture.instances = [] + _FakePlayer.instances = [] + + +# --- the viewer's outbound track ---------------------------------------------- + +def test_track_refuses_to_start_without_a_sounddevice_backend(monkeypatch): + monkeypatch.setattr(audio_mod, "is_audio_backend_available", lambda: False) + + async def _build(): + OpusMicAudioTrack() + + with pytest.raises(AudioBackendError, match="sounddevice"): + asyncio.run(_build()) + + +def test_track_starts_capture_at_the_rate_opus_wants(): + async def _build(): + return OpusMicAudioTrack() + + track = asyncio.run(_build()) + [capture] = _FakeCapture.instances + assert capture.started + assert (capture.sample_rate, capture.channels) == (48000, 1) + assert capture.block_frames == 960, "20 ms at 48 kHz" + assert track.kind == "audio" + + +def test_track_passes_the_chosen_input_device_through(): + async def _build(): + return OpusMicAudioTrack(sample_rate=16000, channels=2, device=3) + + asyncio.run(_build()) + [capture] = _FakeCapture.instances + assert (capture.device, capture.sample_rate, capture.channels) == ( + 3, 16000, 2, + ) + + +def test_a_second_start_does_not_open_a_second_stream(): + async def _build(): + track = OpusMicAudioTrack() + track._start_capture() + return track + + asyncio.run(_build()) + assert len(_FakeCapture.instances) == 1 + + +def test_a_captured_block_reaches_the_queue_from_the_capture_thread(): + async def _drive(): + track = OpusMicAudioTrack() + capture = _FakeCapture.instances[0] + capture.on_block(b"\x01\x02") + await asyncio.sleep(0) # let call_soon_threadsafe land + return track._queue.get_nowait() + + assert asyncio.run(_drive()) == b"\x01\x02" + + +def test_a_block_arriving_after_the_loop_closed_is_dropped_silently(): + # The sounddevice thread outlives the loop by design: it is stopped from + # `stop()`, which itself runs after the viewer has already gone away. + async def _build(): + return OpusMicAudioTrack() + + track = asyncio.run(_build()) + track._on_block(b"\x00\x00") # the loop from _build is closed now + + +def test_a_full_queue_drops_the_oldest_block(): + async def _drive(): + track = OpusMicAudioTrack() + for index in range(track._queue.maxsize): + track._enqueue(bytes([index])) + track._enqueue(b"\xff") + first = track._queue.get_nowait() + drained = [first] + while not track._queue.empty(): + drained.append(track._queue.get_nowait()) + return drained + + drained = asyncio.run(_drive()) + assert drained[0] == b"\x01", "the oldest block made room" + assert drained[-1] == b"\xff", "the newest one is kept" + + +def test_recv_turns_pcm_into_an_interleaved_s16_frame(): + pcm = np.arange(960, dtype=np.int16).tobytes() + + async def _drive(): + track = OpusMicAudioTrack() + track._enqueue(pcm) + return await track.recv() + + frame = asyncio.run(_drive()) + assert frame.sample_rate == 48000 + assert frame.samples == 960 + assert frame.time_base.denominator == 48000 + assert frame.pts == 0 + + +def test_recv_advances_the_timestamp_by_the_samples_it_sent(): + pcm = np.zeros(960, dtype=np.int16).tobytes() + + async def _drive(): + track = OpusMicAudioTrack() + track._enqueue(pcm) + track._enqueue(pcm) + first = await track.recv() + second = await track.recv() + return first.pts, second.pts + + # A pts that does not advance by exactly one block's worth of samples is + # a stream that drifts against the clock aiortc packetises it with. + assert asyncio.run(_drive()) == (0, 960) + + +def test_recv_counts_samples_per_channel_on_a_stereo_capture(): + pcm = np.zeros(960 * 2, dtype=np.int16).tobytes() + + async def _drive(): + track = OpusMicAudioTrack(channels=2) + track._enqueue(pcm) + track._enqueue(pcm) + await track.recv() + return (await track.recv()).pts + + assert asyncio.run(_drive()) == 960 + + +def test_stopping_the_track_stops_the_capture_once(): + async def _drive(): + track = OpusMicAudioTrack() + track.stop() + track.stop() + return track + + asyncio.run(_drive()) + [capture] = _FakeCapture.instances + assert capture.stopped + + +def test_a_capture_that_fails_to_close_still_leaves_the_track_stopped(): + async def _drive(): + track = OpusMicAudioTrack() + _FakeCapture.instances[0].stop_error = OSError("device unplugged") + track.stop() + return track + + track = asyncio.run(_drive()) + assert track._capture is None + + +# --- the host's inbound receiver ---------------------------------------------- + +def test_receiver_refuses_to_start_without_a_sounddevice_backend(monkeypatch): + monkeypatch.setattr(audio_mod, "is_audio_backend_available", lambda: False) + with pytest.raises(AudioBackendError, match="sounddevice"): + OpusMicReceiver() + + +def test_receiver_opens_the_player_at_the_requested_rate(): + OpusMicReceiver(sample_rate=16000, channels=2, device=4) + [player] = _FakePlayer.instances + assert (player.device, player.sample_rate, player.channels) == ( + 4, 16000, 2, + ) + assert player.is_running + + +def test_receiver_plays_the_decoded_frames_it_drains(): + samples = np.array([[1, 2, 3, 4]], dtype=np.int16) + + async def _drive(): + receiver = OpusMicReceiver() + track = FrameTrack(_Frame(samples)) + receiver.consume(track) + await receiver._task + return receiver + + asyncio.run(_drive()) + [player] = _FakePlayer.instances + assert player.played == [samples.tobytes()] + + +def test_receiver_converts_a_float_frame_before_playing_it(): + # av hands back whatever the decoder produced; the player only speaks + # int16 PCM, so a float layout has to be narrowed rather than passed on. + async def _drive(): + receiver = OpusMicReceiver() + receiver.consume(FrameTrack(_Frame(np.array([[1.0, 2.0]], + dtype=np.float32)))) + await receiver._task + + asyncio.run(_drive()) + [player] = _FakePlayer.instances + assert player.played == [np.array([[1, 2]], dtype=np.int16).tobytes()] + + +def test_a_second_consume_does_not_start_a_second_drain(): + async def _drive(): + receiver = OpusMicReceiver() + track = FrameTrack(_Frame(np.array([[1]], dtype=np.int16))) + receiver.consume(track) + first = receiver._task + receiver.consume(FrameTrack()) + assert receiver._task is first + await first + + asyncio.run(_drive()) + + +def test_an_undecodable_frame_is_skipped_rather_than_ending_the_stream(): + good = np.array([[5, 6]], dtype=np.int16) + + async def _drive(): + receiver = OpusMicReceiver() + receiver.consume(FrameTrack(_Frame(error=ValueError("bad plane")), + _Frame(good))) + await receiver._task + + asyncio.run(_drive()) + [player] = _FakePlayer.instances + assert player.played == [good.tobytes()], "the good frame still played" + + +def test_the_drain_stops_when_the_player_is_no_longer_running(): + async def _drive(): + receiver = OpusMicReceiver() + _FakePlayer.instances[0].is_running = False + track = FrameTrack(_Frame(np.array([[1]], dtype=np.int16)), + _Frame(np.array([[2]], dtype=np.int16))) + receiver.consume(track) + await receiver._task + return track + + track = asyncio.run(_drive()) + assert len(track.frames) == 1, "it did not keep pulling from the track" + assert _FakePlayer.instances[0].played == [] + + +def test_the_drain_ends_quietly_when_the_track_dies(): + # The viewer closing its tab surfaces here as an OSError out of recv; + # this task runs on the shared bridge loop, so it may not propagate. + async def _drive(): + receiver = OpusMicReceiver() + receiver.consume(FrameTrack(ending=OSError("connection reset"))) + await receiver._task + + asyncio.run(_drive()) + + +def test_stopping_the_receiver_cancels_the_drain_and_closes_the_player(): + async def _drive(): + receiver = OpusMicReceiver() + receiver.consume(FrameTrack(_Frame(np.array([[1]], dtype=np.int16)))) + task = receiver._task + receiver.stop() + assert receiver._task is None + await asyncio.sleep(0) + return task + + task = asyncio.run(_drive()) + assert task.cancelled() or task.done() + assert _FakePlayer.instances[0].stopped + + +def test_stopping_a_receiver_that_never_consumed_still_closes_the_player(): + receiver = OpusMicReceiver() + receiver.stop() + assert _FakePlayer.instances[0].stopped + + +def test_a_player_that_fails_to_close_does_not_escape_stop(): + receiver = OpusMicReceiver() + _FakePlayer.instances[0].stop_error = OSError("stream already closed") + receiver.stop() diff --git a/test/unit_test/headless/test_webrtc_transport.py b/test/unit_test/headless/test_webrtc_transport.py new file mode 100644 index 00000000..716185d1 --- /dev/null +++ b/test/unit_test/headless/test_webrtc_transport.py @@ -0,0 +1,424 @@ +"""The plumbing both WebRTC ends stand on: bridge, capture, cursor, ICE. + +`webrtc_transport` is the module every other WebRTC module imports, and the +one that raises ImportError at module level without aiortc -- so before the +`[webrtc]` extra joined the measured install, nothing here ran on any CI +square at all. + +Four things in it are worth pinning down, and none of them need a peer: + +* **The bridge is a singleton around one background loop.** `get_bridge()` + hands the same object to host, viewer and every GUI panel, so `start()` + has to be idempotent under a lock -- a second loop would silently split + the DataChannel sends from the PeerConnection they belong to. +* **`_draw_cursor_overlay` writes into a slice of the captured frame.** The + slice is clamped at the array edges but the circle arithmetic is not, so + the ring is computed in absolute coordinates and applied to a window that + may start anywhere. A cursor at the very corner is the case that would + raise if the two disagreed. +* **`_resolve_monitor` is fed an index that comes from the GUI**, where + monitors are listed per mss numbering (1-based, 0 = "all"). An index the + user picked before unplugging a screen must land somewhere real rather + than raise out of the capture thread. +* **`wait_for_ice_gathering` deliberately gives up.** Its timeout branch is + the one that ships a half-gathered SDP rather than hanging the offer, and + it is only reachable when nothing ever completes. + +The capture path is exercised against a fake grabber: `_capture_frame` +caches one mss instance per thread in a module-level `threading.local()`, +which is worth a test of its own because a leak there is one screen grabber +per capture rather than one per thread. +""" +from __future__ import annotations + +import asyncio +import threading +import time + +import numpy as np +import pytest + +from headless._webrtc_doubles import FakePeerConnection +from je_auto_control.utils.remote_desktop import webrtc_transport as transport +from je_auto_control.utils.remote_desktop.webrtc_transport import ( + BANDWIDTH_PRESETS, ScreenVideoTrack, WebRTCConfig, _AsyncioBridge, + _capture_frame, _draw_cursor_overlay, _resolve_monitor, fps_for_preset, + get_bridge, wait_for_ice_gathering, +) + + +# --- fakes -------------------------------------------------------------------- + +class _Grab: + """What ``mss.grab()`` returns: a BGRA buffer plus its dimensions.""" + + def __init__(self, width: int, height: int, fill: int = 7) -> None: + self.width = width + self.height = height + self.bgra = bytes([fill, fill + 1, fill + 2, 255]) * (width * height) + + +class _FakeSct: + def __init__(self, monitors=None) -> None: + self.monitors = monitors if monitors is not None else [ + {"left": 0, "top": 0, "width": 8, "height": 4}, + {"left": 0, "top": 0, "width": 4, "height": 2}, + ] + self.grabs = [] + + def grab(self, monitor): + self.grabs.append(monitor) + return _Grab(monitor["width"], monitor["height"]) + + +@pytest.fixture +def fake_grabber(monkeypatch): + """Install a fake mss grabber and clear the per-thread capture cache.""" + sct = _FakeSct() + calls = [] + + def _grabber(): + calls.append(1) + return sct + + monkeypatch.setattr(transport, "mss_grabber", _grabber) + if hasattr(transport._capture_local, "sct"): + del transport._capture_local.sct + yield sct, calls + if hasattr(transport._capture_local, "sct"): + del transport._capture_local.sct + + +# --- presets and config ------------------------------------------------------- + +def test_fps_for_preset_is_case_insensitive_and_falls_back_to_auto(): + assert fps_for_preset("LOW") == BANDWIDTH_PRESETS["low"]["fps"] + assert fps_for_preset("high") == 30 + # The GUI persists the preset name; a config written by a newer build + # must not make an older one raise out of the capture setup. + assert fps_for_preset("ludicrous") == BANDWIDTH_PRESETS["auto"]["fps"] + + +def test_config_turns_stun_list_into_ice_servers(): + config = WebRTCConfig(ice_servers=["stun:a:1", "stun:b:2"]) + rtc = config.to_rtc_configuration() + assert [server.urls for server in rtc.iceServers] == ["stun:a:1", "stun:b:2"] + + +def test_config_appends_turn_server_with_its_credentials(): + config = WebRTCConfig( + ice_servers=["stun:a:1"], turn_url="turn:relay:3478", + turn_username="user", turn_credential="secret", + ) + turn = config.to_rtc_configuration().iceServers[-1] + assert (turn.urls, turn.username, turn.credential) == ( + "turn:relay:3478", "user", "secret", + ) + + +def test_config_without_turn_url_adds_no_extra_server(): + config = WebRTCConfig(ice_servers=["stun:a:1"], turn_username="user") + # Credentials with no URL are half-filled GUI state, not a server. + assert len(config.to_rtc_configuration().iceServers) == 1 + + +# --- the shared asyncio bridge ------------------------------------------------ + +def test_bridge_start_is_idempotent_and_runs_submitted_coroutines(): + bridge = _AsyncioBridge() + try: + loop = bridge.start() + assert bridge.start() is loop, "a second loop would split the session" + assert loop.is_running() + + async def _answer(): + return 42 + + assert bridge.submit(_answer()).result(timeout=5.0) == 42 + finally: + bridge.stop() + + +def test_bridge_call_soon_runs_the_callable_on_the_loop_thread(): + bridge = _AsyncioBridge() + seen = {} + done = threading.Event() + + def _record(value): + seen["thread"] = threading.current_thread().name + seen["value"] = value + done.set() + + try: + bridge.call_soon(_record, "payload") + assert done.wait(timeout=5.0) + finally: + bridge.stop() + assert seen["value"] == "payload" + assert seen["thread"] == "webrtc-loop" + + +def test_bridge_stop_is_safe_before_start_and_after_stop(): + bridge = _AsyncioBridge() + bridge.stop() # never started: nothing to join + bridge.start() + bridge.stop() + bridge.stop() # already torn down + + +def test_get_bridge_returns_the_process_wide_instance(): + assert get_bridge() is get_bridge() + + +# --- cursor overlay ----------------------------------------------------------- + +def _white(height=48, width=48): + return np.full((height, width, 3), 255, dtype=np.uint8) + + +def test_cursor_overlay_draws_a_yellow_ring_around_a_black_core(): + frame = _white() + _draw_cursor_overlay(frame, 24, 24) + assert tuple(frame[24, 24]) == (0, 0, 0), "core" + assert tuple(frame[24, 24 + 8]) == (0, 255, 255), "ring at radius 8" + assert tuple(frame[0, 0]) == (255, 255, 255), "far corner untouched" + + +@pytest.mark.parametrize("x,y", [(-1, 10), (10, -1), (48, 10), (10, 48)]) +def test_cursor_overlay_ignores_positions_outside_the_frame(x, y): + frame = _white() + _draw_cursor_overlay(frame, x, y) + assert (frame == 255).all(), "an off-screen cursor must not paint" + + +def test_cursor_overlay_clamps_its_window_at_the_frame_corner(): + # The ring is computed in absolute coordinates and written into a slice + # that stops at the edge; at (0, 0) three quarters of it are off-frame. + frame = _white() + _draw_cursor_overlay(frame, 0, 0) + assert tuple(frame[0, 0]) == (0, 0, 0) + assert tuple(frame[0, 8]) == (0, 255, 255) + + +# --- monitor resolution and capture ------------------------------------------- + +def test_resolve_monitor_returns_the_requested_entry(): + sct = _FakeSct() + assert _resolve_monitor(sct, 1) is sct.monitors[1] + + +@pytest.mark.parametrize("index", [-1, 2, 99]) +def test_resolve_monitor_falls_back_to_the_first_screen(index): + # A stale index (the user unplugged the screen they had picked) must + # land on a real monitor rather than raise inside the capture thread. + sct = _FakeSct() + assert _resolve_monitor(sct, index) is sct.monitors[1] + + +def test_resolve_monitor_falls_back_to_the_only_entry_when_alone(): + sct = _FakeSct(monitors=[{"left": 0, "top": 0, "width": 2, "height": 2}]) + assert _resolve_monitor(sct, 5) is sct.monitors[0] + + +def test_resolve_monitor_raises_when_mss_reports_nothing(): + with pytest.raises(RuntimeError, match="no monitors"): + _resolve_monitor(_FakeSct(monitors=[]), 1) + + +def test_capture_frame_drops_alpha_and_returns_a_contiguous_bgr_array( + fake_grabber): + monitor = {"left": 0, "top": 0, "width": 4, "height": 2} + arr = _capture_frame(monitor) + assert arr.shape == (2, 4, 3) + assert arr.dtype == np.uint8 + # av.VideoFrame.from_ndarray requires contiguity; the BGRA slice is not. + assert arr.flags["C_CONTIGUOUS"] + assert tuple(arr[0, 0]) == (7, 8, 9), "the alpha byte is dropped, not read" + + +def test_capture_frame_reuses_one_grabber_per_thread(fake_grabber): + sct, calls = fake_grabber + monitor = {"left": 0, "top": 0, "width": 4, "height": 2} + _capture_frame(monitor) + _capture_frame(monitor) + assert len(calls) == 1, "one mss instance per thread, not per frame" + assert len(sct.grabs) == 2 + + +# --- ScreenVideoTrack --------------------------------------------------------- + +@pytest.fixture +def track_factory(): + """Build ScreenVideoTracks and shut their capture executors down.""" + made = [] + + def _make(**kwargs): + track = ScreenVideoTrack(**kwargs) + made.append(track) + return track + + yield _make + for track in made: + track.stop() + + +@pytest.mark.parametrize("requested,expected", [ + (0, 1), (-5, 1), (24, 24), (61, 60), (1000, 60), +]) +def test_track_clamps_its_frame_rate(track_factory, requested, expected): + assert track_factory(fps=requested).fps == expected + + +def test_set_target_fps_updates_the_period_it_sleeps_on(track_factory): + track = track_factory(fps=10) + track.set_target_fps(20) + assert track.fps == 20 + assert track._period == pytest.approx(0.05) + + +def test_set_target_fps_ignores_a_repeat_of_the_current_rate(track_factory): + track = track_factory(fps=24) + period = track._period + track.set_target_fps(24) + assert track._period is period + + +def test_set_target_fps_clamps_the_adaptive_controller_too(track_factory): + # The bandwidth controller feeds this from observed RTT, so the clamp + # is the only thing between a bad measurement and a 1/0 period. + track = track_factory(fps=24) + track.set_target_fps(0) + assert track.fps == 1 + + +def test_track_resolves_a_region_without_asking_mss(track_factory, + fake_grabber): + _, calls = fake_grabber + track = track_factory(region=(10, 20, 30, 40)) + assert track._resolve() == {"left": 10, "top": 20, + "width": 30, "height": 40} + assert not calls, "a fixed region needs no monitor enumeration" + + +def test_track_resolves_and_caches_the_monitor(track_factory, fake_grabber): + sct, _ = fake_grabber + track = track_factory(monitor_index=1) + assert track._resolve() is sct.monitors[1] + assert track._resolve() is sct.monitors[1] + + +def test_set_target_monitor_invalidates_the_cached_lookup(track_factory, + fake_grabber): + sct, _ = fake_grabber + track = track_factory(monitor_index=1) + track._resolve() + track.set_target_monitor(0) + assert track._resolve() is sct.monitors[0] + + +def test_track_recv_returns_a_bgr_video_frame_with_a_timestamp( + track_factory, monkeypatch): + captured = np.full((4, 6, 3), 255, dtype=np.uint8) + monkeypatch.setattr(transport, "_capture_frame", lambda monitor: captured) + monkeypatch.setattr(transport, "_get_cursor_position", lambda: None) + track = track_factory(region=(0, 0, 6, 4), fps=60, show_cursor=True) + frame = asyncio.run(track.recv()) + assert (frame.width, frame.height) == (6, 4) + assert frame.time_base is not None + assert (frame.to_ndarray(format="bgr24") == 255).all() + + +def test_track_recv_overlays_the_cursor_in_monitor_local_coordinates( + track_factory, monkeypatch): + captured = np.full((48, 48, 3), 255, dtype=np.uint8) + monkeypatch.setattr(transport, "_capture_frame", lambda monitor: captured) + # Absolute (124, 224) on a monitor pinned at (100, 200) is local (24, 24). + monkeypatch.setattr(transport, "_get_cursor_position", lambda: (124, 224)) + track = track_factory(region=(100, 200, 48, 48), fps=60) + frame = asyncio.run(track.recv()) + arr = frame.to_ndarray(format="bgr24") + assert tuple(arr[24, 24]) == (0, 0, 0) + + +def test_track_recv_skips_the_overlay_when_the_cursor_is_unknown( + track_factory, monkeypatch): + captured = np.full((48, 48, 3), 255, dtype=np.uint8) + monkeypatch.setattr(transport, "_capture_frame", lambda monitor: captured) + # Wayland and headless X both leave the position unavailable. + monkeypatch.setattr(transport, "_get_cursor_position", lambda: None) + track = track_factory(region=(0, 0, 48, 48), fps=60, show_cursor=True) + frame = asyncio.run(track.recv()) + assert (frame.to_ndarray(format="bgr24") == 255).all() + + +def test_track_recv_paces_itself_at_the_target_rate(track_factory, + monkeypatch): + # Real time, not a patched `asyncio.sleep`: `transport.asyncio` IS the + # asyncio module, so patching through it also rewires aiortc's own + # `next_timestamp`, and the recorded delays stop being ours. 10 fps + # keeps the whole test inside a tenth of a second. + captured = np.full((4, 4, 3), 255, dtype=np.uint8) + monkeypatch.setattr(transport, "_capture_frame", lambda monitor: captured) + monkeypatch.setattr(transport, "_get_cursor_position", lambda: None) + track = track_factory(region=(0, 0, 4, 4), fps=10, show_cursor=False) + + started = time.monotonic() + asyncio.run(track.recv()) + first = time.monotonic() - started + asyncio.run(track.recv()) + second = time.monotonic() - started - first + + # The first frame goes out immediately; the second waits out the period. + assert first < 0.05 + assert second >= 0.05 + + +def test_track_stop_shuts_the_capture_executor_down(): + track = ScreenVideoTrack(region=(0, 0, 4, 4)) + track.stop() + with pytest.raises(RuntimeError): + track._executor.submit(lambda: None) + + +# --- ICE gathering ------------------------------------------------------------ + +def test_wait_for_ice_gathering_returns_at_once_when_already_complete(): + pc = FakePeerConnection() + pc.iceGatheringState = "complete" + asyncio.run(wait_for_ice_gathering(pc)) + assert not pc.handlers, "no need to subscribe to a finished gather" + + +def test_wait_for_ice_gathering_resolves_on_the_state_change(): + pc = FakePeerConnection() + + async def _drive(): + waiter = asyncio.ensure_future(wait_for_ice_gathering(pc, timeout=5.0)) + await asyncio.sleep(0) + pc.complete_ice_gathering() + await waiter + + asyncio.run(_drive()) + + +def test_wait_for_ice_gathering_ignores_intermediate_states(): + pc = FakePeerConnection() + + async def _drive(): + waiter = asyncio.ensure_future(wait_for_ice_gathering(pc, timeout=5.0)) + await asyncio.sleep(0) + pc.iceGatheringState = "gathering" + pc.handlers["icegatheringstatechange"]() + assert not waiter.done() + pc.complete_ice_gathering() + await waiter + + asyncio.run(_drive()) + + +def test_wait_for_ice_gathering_gives_up_and_sends_what_it_has(): + # The timeout is the branch that ships a half-gathered SDP instead of + # hanging create_offer() until its own 12 s future times out. + pc = FakePeerConnection() + asyncio.run(wait_for_ice_gathering(pc, timeout=0.05)) + assert pc.iceGatheringState == "new" diff --git a/test/unit_test/headless/test_webrtc_viewer_control.py b/test/unit_test/headless/test_webrtc_viewer_control.py new file mode 100644 index 00000000..d3640780 --- /dev/null +++ b/test/unit_test/headless/test_webrtc_viewer_control.py @@ -0,0 +1,520 @@ +"""What the viewer says to the host, and what it believes coming back. + +The session half is in `test_webrtc_viewer_session.py`. This file covers the +control channel: the request verbs the GUI drives, and the dispatch table +that decides what an inbound envelope means. + +The direction matters. On the host every inbound message is a boundary to be +refused; here every inbound message is a *claim by the host about this +session* -- that the token was accepted, that the session is read-only, that +the inbox holds these files -- and the viewer acts on it. So the tests below +are about the viewer believing exactly what it was told and no more: + +* **`auth_ok` is what flips the session live**, and it carries both the + read-only flag and the host's stable fingerprint in the same envelope. + A fingerprint that arrives empty must not overwrite one already shown to + the user, because that is the string they compared out-of-band. +* **`permissions` and `read_only` are two spellings of the same state.** + The newer envelope carries five flags; the viewer only mirrors one of them + (input), and a missing flag has to default to permitted, not to denied -- + the GUI greys out its own controls from this. +* **Every callback here is a Qt slot** reached from the asyncio thread, so + each one is wrapped: a raising GUI must not take the DataChannel down. + +`get_bridge` is replaced (from `headless._webrtc_doubles`) with one that +runs the callback inline; otherwise every `_send` would queue onto a loop no +test is running. +""" +from __future__ import annotations + +import asyncio +import json + +import pytest + +from headless._webrtc_doubles import Bridge, Channel, MicSender +from je_auto_control.utils.remote_desktop import webrtc_viewer as viewer_module +from je_auto_control.utils.remote_desktop.webrtc_viewer import ( + WebRTCDesktopViewer, +) + + +@pytest.fixture(autouse=True) +def bridge(monkeypatch): + fake = Bridge() + monkeypatch.setattr(viewer_module, "get_bridge", lambda: fake) + return fake + + +@pytest.fixture +def viewer(): + """A viewer with an open control channel, as after `_attach_datachannel`.""" + instance = WebRTCDesktopViewer(token="secret") + channel = Channel() + instance._control_channel = channel + instance._wire_control_channel(channel) + channel.sent.clear() # drop the auth envelope the wiring may send + return instance + + +def _sent(viewer_instance): + return [json.loads(text) for text in viewer_instance._control_channel.sent] + + +def _deliver(viewer_instance, payload): + viewer_instance._control_channel.fire("message", json.dumps(payload)) + + +# --- outbound verbs ----------------------------------------------------------- + +def test_input_is_sent_as_a_payload_envelope(viewer): + viewer.send_input({"kind": "mouse", "x": 3}) + assert _sent(viewer) == [{"type": "input", + "payload": {"kind": "mouse", "x": 3}}] + + +def test_the_input_payload_is_copied_not_referenced(viewer): + payload = {"kind": "mouse"} + viewer.send_input(payload) + payload["kind"] = "mutated" + assert _sent(viewer)[0]["payload"] == {"kind": "mouse"} + + +@pytest.mark.parametrize("method,expected", [ + ("request_send_sas", "send_sas"), + ("request_inbox_listing", "list_inbox"), + ("request_renegotiation", "renegotiate_request"), +]) +def test_the_parameterless_verbs_each_send_their_envelope(viewer, method, + expected): + getattr(viewer, method)() + assert _sent(viewer) == [{"type": expected}] + + +def test_requesting_a_file_names_it(viewer): + viewer.request_inbox_file("report.txt") + assert _sent(viewer) == [{"type": "request_file", "name": "report.txt"}] + + +def test_deleting_a_file_names_it(viewer): + viewer.delete_inbox_file("report.txt") + assert _sent(viewer) == [{"type": "delete_inbox_file", + "name": "report.txt"}] + + +@pytest.mark.parametrize("method", ["request_inbox_file", "delete_inbox_file"]) +def test_an_empty_name_is_refused_before_it_reaches_the_wire(viewer, method): + with pytest.raises(ValueError, match="name required"): + getattr(viewer, method)("") + assert _sent(viewer) == [] + + +def test_sending_before_the_channel_opens_is_dropped_not_raised(bridge): + # The GUI can wire buttons up before the host's channel arrives; a click + # in that window is a no-op, not a traceback in a Qt slot. + WebRTCDesktopViewer(token="secret").send_input({"kind": "mouse"}) + assert bridge.deferred == [] + + +def test_a_channel_that_fails_mid_send_does_not_raise(viewer): + viewer._control_channel.send_error = OSError("channel closed") + viewer.send_input({"kind": "mouse"}) + + +def test_a_channel_closed_between_queue_and_send_is_a_no_op(viewer): + viewer._control_channel = None + viewer._safe_channel_send("{}") + + +def test_the_auth_envelope_omits_a_viewer_id_that_was_never_set(viewer): + viewer._send_auth() + assert _sent(viewer) == [{"type": "auth", "token": "secret"}] + + +# --- inbound envelopes: shape ------------------------------------------------- + +@pytest.mark.parametrize("message", [b"\x00", 42, None, ["auth_ok"]]) +def test_a_non_text_message_is_ignored(viewer, message): + viewer._control_channel.fire("message", message) + assert viewer.authenticated is False + + +def test_a_malformed_envelope_is_dropped(viewer): + viewer._control_channel.fire("message", "{not json") + assert viewer.authenticated is False + + +def test_a_json_scalar_is_not_an_envelope(viewer): + viewer._control_channel.fire("message", '"auth_ok"') + assert viewer.authenticated is False + + +def test_an_unknown_message_type_is_ignored(viewer): + _deliver(viewer, {"type": "shutdown"}) + assert viewer.authenticated is False + + +def test_closing_the_control_channel_ends_the_session(viewer): + viewer._authenticated = True + viewer._control_channel.fire("close") + assert viewer.authenticated is False + + +# --- authentication result ---------------------------------------------------- + +def test_auth_ok_makes_the_session_live(): + seen = [] + viewer = WebRTCDesktopViewer(token="secret", on_auth_result=seen.append) + viewer._control_channel = Channel() + viewer._wire_control_channel(viewer._control_channel) + _deliver(viewer, {"type": "auth_ok"}) + assert viewer.authenticated is True + assert seen == [True] + + +def test_auth_ok_carries_the_read_only_flag(viewer): + _deliver(viewer, {"type": "auth_ok", "read_only": True}) + assert viewer.read_only is True + + +def test_auth_ok_publishes_the_host_fingerprint(): + seen = [] + viewer = WebRTCDesktopViewer(token="secret", on_fingerprint=seen.append) + viewer._control_channel = Channel() + viewer._wire_control_channel(viewer._control_channel) + _deliver(viewer, {"type": "auth_ok", "fingerprint": "AB:CD:EF"}) + assert viewer.host_fingerprint == "AB:CD:EF" + assert seen == ["AB:CD:EF"] + + +def test_the_fingerprint_is_recorded_even_with_nobody_subscribed(viewer): + # `host_fingerprint` is read by the GUI on demand as well as pushed, so + # it has to be stored whether or not a callback was registered. + _deliver(viewer, {"type": "auth_ok", "fingerprint": "AB:CD:EF"}) + assert viewer.host_fingerprint == "AB:CD:EF" + + +@pytest.mark.parametrize("fingerprint", [None, "", 42]) +def test_an_absent_fingerprint_does_not_overwrite_the_one_on_screen( + viewer, fingerprint): + # It is the string the user compared out-of-band; a later envelope + # without one must leave it standing rather than blank the field. + viewer._host_fingerprint = "AB:CD:EF" + _deliver(viewer, {"type": "auth_ok", "fingerprint": fingerprint}) + assert viewer.host_fingerprint == "AB:CD:EF" + + +def test_a_raising_fingerprint_callback_still_authenticates(): + def _boom(_value): + raise RuntimeError("Qt widget already deleted") + + viewer = WebRTCDesktopViewer(token="secret", on_fingerprint=_boom) + viewer._control_channel = Channel() + viewer._wire_control_channel(viewer._control_channel) + _deliver(viewer, {"type": "auth_ok", "fingerprint": "AB"}) + assert viewer.authenticated is True + + +def test_auth_fail_leaves_the_session_closed(): + seen = [] + viewer = WebRTCDesktopViewer(token="wrong", on_auth_result=seen.append) + viewer._control_channel = Channel() + viewer._wire_control_channel(viewer._control_channel) + viewer._authenticated = True + _deliver(viewer, {"type": "auth_fail"}) + assert viewer.authenticated is False + assert seen == [False] + + +def test_an_auth_result_with_no_listener_is_survivable(viewer): + _deliver(viewer, {"type": "auth_ok"}) + assert viewer.authenticated is True + + +def test_a_raising_auth_callback_does_not_break_the_channel(): + def _boom(_ok): + raise RuntimeError("Qt widget already deleted") + + viewer = WebRTCDesktopViewer(token="secret", on_auth_result=_boom) + viewer._control_channel = Channel() + viewer._wire_control_channel(viewer._control_channel) + _deliver(viewer, {"type": "auth_ok"}) + assert viewer.authenticated is True + + +# --- permission updates ------------------------------------------------------- + +def test_the_legacy_read_only_envelope_is_still_understood(viewer): + _deliver(viewer, {"type": "read_only", "value": True}) + assert viewer.read_only is True + _deliver(viewer, {"type": "read_only", "value": False}) + assert viewer.read_only is False + + +def test_a_read_only_envelope_with_no_value_means_not_read_only(viewer): + viewer._read_only = True + _deliver(viewer, {"type": "read_only"}) + assert viewer.read_only is False + + +def test_the_permissions_envelope_mirrors_the_input_flag(viewer): + _deliver(viewer, {"type": "permissions", "value": {"allow_input": False}}) + assert viewer.read_only is True + _deliver(viewer, {"type": "permissions", "value": {"allow_input": True}}) + assert viewer.read_only is False + + +def test_a_permissions_envelope_without_the_input_flag_assumes_permitted( + viewer): + # The GUI greys out its own controls from this; defaulting to denied + # would lock the user out because a newer host sent a shorter dict. + viewer._read_only = True + _deliver(viewer, {"type": "permissions", "value": {"allow_files": True}}) + assert viewer.read_only is False + + +def test_a_permissions_envelope_that_is_not_a_dict_is_ignored(viewer): + viewer._read_only = True + _deliver(viewer, {"type": "permissions", "value": "read_only"}) + assert viewer.read_only is True + + +# --- the inbox listing -------------------------------------------------------- + +def test_an_inbox_listing_reaches_its_callback(viewer): + seen = [] + viewer.set_inbox_listing_callback(seen.append) + _deliver(viewer, {"type": "list_inbox_response", + "files": [{"name": "a.txt", "size": 1}]}) + assert seen == [[{"name": "a.txt", "size": 1}]] + + +def test_a_listing_with_no_files_key_becomes_an_empty_list(viewer): + seen = [] + viewer.set_inbox_listing_callback(seen.append) + _deliver(viewer, {"type": "list_inbox_response"}) + assert seen == [[]] + + +def test_a_listing_with_no_listener_is_dropped(viewer): + _deliver(viewer, {"type": "list_inbox_response", "files": []}) + + +def test_a_raising_listing_callback_is_contained(viewer): + def _boom(_files): + raise RuntimeError("Qt model already deleted") + + viewer.set_inbox_listing_callback(_boom) + _deliver(viewer, {"type": "list_inbox_response", "files": []}) + + +@pytest.mark.parametrize("msg_type", ["delete_inbox_response", + "request_file_response"]) +def test_an_inbox_operation_result_reaches_its_callback(viewer, msg_type): + seen = [] + viewer.set_inbox_op_result_callback( + lambda name, ok, error: seen.append((name, ok, error)), + ) + _deliver(viewer, {"type": msg_type, "name": "a.txt", "ok": False, + "error": "not found"}) + assert seen == [("a.txt", False, "not found")] + + +def test_an_operation_result_defaults_to_a_failure(viewer): + # A response missing its fields is not a success; the GUI would + # otherwise report a delete that never happened. + seen = [] + viewer.set_inbox_op_result_callback( + lambda name, ok, error: seen.append((name, ok, error)), + ) + _deliver(viewer, {"type": "delete_inbox_response"}) + assert seen == [("", False, None)] + + +def test_an_operation_result_with_no_listener_is_dropped(viewer): + _deliver(viewer, {"type": "delete_inbox_response", "name": "a.txt"}) + + +def test_a_raising_operation_callback_is_contained(viewer): + def _boom(_name, _ok, _error): + raise RuntimeError("Qt widget already deleted") + + viewer.set_inbox_op_result_callback(_boom) + _deliver(viewer, {"type": "delete_inbox_response", "name": "a.txt"}) + + +# --- host-initiated renegotiation --------------------------------------------- + +def test_a_renegotiate_offer_without_a_connection_is_ignored(viewer): + _deliver(viewer, {"type": "renegotiate_offer", "sdp": "v=0 offer"}) + assert viewer._background_tasks == set() + + +def test_a_renegotiate_offer_with_no_sdp_is_ignored(viewer): + viewer._pc = object() + _deliver(viewer, {"type": "renegotiate_offer", "sdp": None}) + assert viewer._background_tasks == set() + + +def test_a_renegotiate_offer_is_handled_on_the_event_loop(viewer, monkeypatch): + applied = [] + + class _Pc: + transceivers = [] + + async def setRemoteDescription(self, description): # noqa: N802 + applied.append(description.sdp) + + def getTransceivers(self): # noqa: N802 # reason: the aiortc name + return [] + + async def createAnswer(self): # noqa: N802 # reason: the aiortc name + return "answer" + + async def setLocalDescription(self, description): # noqa: N802 + pass + + @property + def localDescription(self): # noqa: N802 # reason: the aiortc name + return type("_Desc", (), {"sdp": "v=0 answer"})() + + async def _drive(): + viewer._pc = _Pc() + _deliver(viewer, {"type": "renegotiate_offer", "sdp": "v=0 offer"}) + assert len(viewer._background_tasks) == 1 + await asyncio.gather(*viewer._background_tasks) + + async def _noop(pc, timeout=None): + return None + + monkeypatch.setattr(viewer_module, "wait_for_ice_gathering", _noop) + asyncio.run(_drive()) + assert applied == ["v=0 offer"] + + +# --- microphone uplink -------------------------------------------------------- + +@pytest.fixture +def mic_sender(monkeypatch): + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_mic.MicUplinkSender", + MicSender, + ) + + +def test_enabling_the_microphone_before_connecting_is_refused(viewer): + with pytest.raises(RuntimeError, match="mic channel not open"): + viewer.enable_mic_send() + + +def test_enabling_the_microphone_starts_one_sender(viewer, mic_sender): + channel = Channel("mic") + viewer._mic_channel = channel + viewer.enable_mic_send() + first = viewer._mic_sender + assert first.started + assert first.channel is channel + assert viewer.mic_active is True + viewer.enable_mic_send() + assert viewer._mic_sender is first + + +def test_disabling_the_microphone_stops_it(viewer, mic_sender): + viewer._mic_channel = Channel("mic") + viewer.enable_mic_send() + viewer.disable_mic_send() + assert viewer._mic_sender is None + assert viewer.mic_active is False + + +def test_disabling_a_microphone_that_was_never_enabled_is_a_no_op(viewer): + viewer.disable_mic_send() + assert viewer.mic_active is False + + +def test_a_microphone_that_fails_to_close_is_still_forgotten(viewer, + mic_sender): + viewer._mic_channel = Channel("mic") + viewer.enable_mic_send() + viewer._mic_sender.stop_error = OSError("stream already closed") + viewer.disable_mic_send() + assert viewer._mic_sender is None + + +def test_a_sender_that_stopped_on_its_own_is_not_active(viewer, mic_sender): + viewer._mic_channel = Channel("mic") + viewer.enable_mic_send() + viewer._mic_sender.running = False + assert viewer.mic_active is False + + +# --- files -------------------------------------------------------------------- + +def test_sending_a_file_before_the_channel_opens_is_refused(viewer): + with pytest.raises(RuntimeError, match="files channel not open"): + viewer.send_file("C:/report.txt") + + +def test_sending_a_file_streams_it_over_the_files_channel(viewer, tmp_path): + source = tmp_path / "notes.txt" + source.write_bytes(b"hello") + viewer._files_channel = Channel("files") + viewer.send_file(str(source), remote_name="renamed.txt") + envelope = json.loads(viewer._files_channel.sent[0]) + assert (envelope["type"], envelope["name"]) == ("file_begin", + "renamed.txt") + + +@pytest.fixture +def viewer_inbox(tmp_path, monkeypatch): + """Point the viewer's inbox at tmp_path. + + The viewer takes no `inbox_dir` -- unlike the host it always uses + `webrtc_files._DEFAULT_INBOX`, which is resolved at import time under + the real `~/.je_auto_control`. Redirecting HOME would be too late. + """ + from je_auto_control.utils.remote_desktop import webrtc_files + inbox = tmp_path / "inbox" + monkeypatch.setattr(webrtc_files, "_DEFAULT_INBOX", inbox) + return inbox + + +def test_a_file_pushed_by_the_host_lands_and_is_announced(viewer, + viewer_inbox): + seen = [] + viewer.set_file_received_callback(seen.append) + channel = Channel("files") + viewer._wire_files_channel(channel) + inbox = viewer_inbox + channel.fire("message", json.dumps({"type": "file_begin", + "name": "pushed.txt", "size": 4})) + channel.fire("message", b"data") + channel.fire("message", json.dumps({"type": "file_end"})) + assert (inbox / "pushed.txt").read_bytes() == b"data" + assert seen and seen[0].name == "pushed.txt" + + +def test_a_file_arriving_with_no_listener_is_still_written(viewer, + viewer_inbox): + channel = Channel("files") + viewer._wire_files_channel(channel) + channel.fire("message", json.dumps({"type": "file_begin", + "name": "pushed.txt", "size": 1})) + channel.fire("message", b"x") + channel.fire("message", json.dumps({"type": "file_end"})) + assert (viewer._files_receiver._inbox / "pushed.txt").exists() + + +def test_a_raising_file_callback_does_not_lose_the_file(viewer): + def _boom(_path): + raise RuntimeError("Qt widget already deleted") + + viewer.set_file_received_callback(_boom) + viewer._on_viewer_file_done("C:/inbox/pushed.txt") + + +def test_the_files_receiver_is_built_once_and_reused(viewer, viewer_inbox): + viewer._wire_files_channel(Channel("files")) + first = viewer._files_receiver + viewer._wire_files_channel(Channel("files")) + assert viewer._files_receiver is first diff --git a/test/unit_test/headless/test_webrtc_viewer_media.py b/test/unit_test/headless/test_webrtc_viewer_media.py new file mode 100644 index 00000000..1fcbfebe --- /dev/null +++ b/test/unit_test/headless/test_webrtc_viewer_media.py @@ -0,0 +1,385 @@ +"""Attaching the viewer's own screen and microphone to the host's slots. + +Split out of `test_webrtc_viewer_session.py`, which covers the offer/answer +exchange this rides on. Everything here turns on one upstream fact and one +design decision: + +* **Slots are identified by m-line order, not by direction.** aiortc gives + every answerer transceiver the default `recvonly` direction regardless of + what the offer asked for, so the viewer cannot filter by direction and + counts instead: the *second* video transceiver is the host's recvonly + slot, because the first is the host's own outbound screen. Off by one + there and the viewer replaces the picture it is here to watch. +* **OFF is in-place; ON renegotiates.** Turning a stream off is + `replaceTrack(None)` plus a stop, keeping the SDP direction so the slot + survives -- the host simply sees its consume task end. Turning one on + always needs a fresh `track` event at the host, so it costs a + renegotiation round trip. Getting that backwards costs one on every mute. + +The doubles come from `headless._webrtc_doubles`; what is not faked is the +slot arithmetic, which is the thing under test. +""" +from __future__ import annotations + +import asyncio + +import pytest + +from headless._webrtc_doubles import ( + Bridge, Channel, FakePeerConnection, Track, Transceiver, + noop_ice_gathering, +) +from je_auto_control.utils.remote_desktop import webrtc_viewer as viewer_module +from je_auto_control.utils.remote_desktop.webrtc_transport import WebRTCConfig +from je_auto_control.utils.remote_desktop.webrtc_viewer import ( + WebRTCDesktopViewer, +) + + +@pytest.fixture(autouse=True) +def fake_peer_connection(monkeypatch): + FakePeerConnection.instances = [] + monkeypatch.setattr(viewer_module, "RTCPeerConnection", FakePeerConnection) + monkeypatch.setattr(viewer_module, "wait_for_ice_gathering", + noop_ice_gathering) + yield + FakePeerConnection.instances = [] + + +@pytest.fixture(autouse=True) +def bridge(monkeypatch): + fake = Bridge() + monkeypatch.setattr(viewer_module, "get_bridge", lambda: fake) + return fake + + +@pytest.fixture +def screen_track(monkeypatch): + """Replace the capture the viewer attaches when it shares its screen.""" + made = [] + + def _factory(**kwargs): + track = Track("video") + track.kwargs = kwargs + made.append(track) + return track + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_transport" + ".ScreenVideoTrack", _factory, + ) + return made + + +@pytest.fixture +def opus_track(monkeypatch): + made = [] + + def _factory(): + track = Track("audio") + made.append(track) + return track + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicAudioTrack", + _factory, + ) + return made + + +def _viewer(**kwargs) -> WebRTCDesktopViewer: + kwargs.setdefault("token", "secret") + return WebRTCDesktopViewer(**kwargs) + + +def _connected(**kwargs): + """A viewer with a peer connection already in place.""" + viewer = _viewer(**kwargs) + viewer._pc = FakePeerConnection() + return viewer, viewer._pc + + +# --- attaching the viewer's own media ----------------------------------------- + +def _video_slots(pc, count=2): + pc.transceivers = [Transceiver("video") for _ in range(count)] + return pc.transceivers + + +def test_sharing_a_screen_takes_the_second_video_slot(screen_track): + # The first video m-line is the host's outbound screen; taking it would + # replace the picture the viewer is here to watch. + viewer = _viewer(config=WebRTCConfig(share_my_screen=True)) + viewer.process_offer("v=0 host-offer") + pc = FakePeerConnection.instances[0] + assert pc.transceivers == [] # no slots offered + viewer._pc = pc + first, second = _video_slots(pc) + viewer._attach_viewer_screen_track() + assert second.sender.track is screen_track[0] + assert second.direction == "sendonly" + assert first.sender.track is None + + +def test_sharing_a_screen_uses_the_viewers_own_capture_settings(screen_track): + config = WebRTCConfig(share_my_screen=True, monitor_index=2, fps=12, + region=(1, 2, 3, 4), show_cursor=False) + viewer, pc = _connected(config=config) + _video_slots(pc) + viewer._attach_viewer_screen_track() + assert screen_track[0].kwargs == { + "monitor_index": 2, "fps": 12, "region": (1, 2, 3, 4), + "show_cursor": False, + } + + +def test_sharing_a_screen_the_host_never_offered_is_a_warning_not_a_crash( + screen_track): + # `accept_viewer_video=False` on the host means there is no second slot; + # the viewer must keep watching rather than raise out of the answer. + viewer, pc = _connected(config=WebRTCConfig(share_my_screen=True)) + _video_slots(pc, count=1) + viewer._attach_viewer_screen_track() + assert viewer._viewer_screen_track is None + assert screen_track == [] + + +def test_sharing_a_microphone_takes_the_only_audio_slot(opus_track): + viewer, pc = _connected(config=WebRTCConfig(share_my_audio_opus=True)) + slot = Transceiver("audio") + pc.transceivers = [Transceiver("video"), slot] + viewer._attach_opus_audio_track() + assert slot.sender.track is opus_track[0] + assert slot.direction == "sendonly" + + +def test_sharing_a_microphone_the_host_never_offered_is_survivable(opus_track): + viewer, pc = _connected(config=WebRTCConfig(share_my_audio_opus=True)) + pc.transceivers = [Transceiver("video")] + viewer._attach_opus_audio_track() + assert viewer._opus_audio_track is None + + +def test_a_viewer_with_no_microphone_still_answers(monkeypatch): + def _no_device(): + raise OSError("no input device") + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicAudioTrack", + _no_device, + ) + viewer, pc = _connected(config=WebRTCConfig(share_my_audio_opus=True)) + pc.transceivers = [Transceiver("audio")] + viewer._attach_opus_audio_track() + assert viewer._opus_audio_track is None + + +def test_the_answer_attaches_both_streams_the_config_asks_for(screen_track, + opus_track): + config = WebRTCConfig(share_my_screen=True, share_my_audio_opus=True) + viewer = _viewer(config=config) + + class _PcWithSlots(FakePeerConnection): + def __init__(self, configuration=None) -> None: + super().__init__(configuration) + self.transceivers = [Transceiver("video"), Transceiver("video"), + Transceiver("audio")] + + viewer_module.RTCPeerConnection = _PcWithSlots + try: + viewer.process_offer("v=0 host-offer") + finally: + viewer_module.RTCPeerConnection = FakePeerConnection + assert viewer._viewer_screen_track is screen_track[0] + assert viewer._opus_audio_track is opus_track[0] + + +# --- live toggles ------------------------------------------------------------- + +def test_turning_screen_share_on_asks_the_host_to_renegotiate(): + viewer, _ = _connected() + channel = Channel() + viewer._control_channel = channel + viewer.toggle_share_screen(True) + assert viewer._config.share_my_screen is True + assert '"renegotiate_request"' in channel.sent[0] + + +def test_turning_screen_share_on_twice_costs_one_renegotiation(): + viewer, _ = _connected() + channel = Channel() + viewer._control_channel = channel + viewer._viewer_screen_track = Track("video") + viewer.toggle_share_screen(True) + assert channel.sent == [], "already sharing" + + +def test_turning_screen_share_off_detaches_in_place(): + # OFF costs nothing: the SDP direction stays, so the slot survives and + # the host simply sees its consume task end. + viewer, pc = _connected() + channel = Channel() + viewer._control_channel = channel + track = Track("video") + viewer._viewer_screen_track = track + pc.transceivers = [Transceiver("video", track=track)] + viewer.toggle_share_screen(False) + assert track.stopped + assert pc.transceivers[0].sender.replaced == [None] + assert viewer._viewer_screen_track is None + assert channel.sent == [], "no renegotiation on the way down" + + +def test_turning_the_microphone_off_detaches_in_place(): + viewer, pc = _connected() + track = Track("audio") + viewer._opus_audio_track = track + pc.transceivers = [Transceiver("audio", track=track)] + viewer.toggle_opus_mic(False) + assert track.stopped + assert viewer._opus_audio_track is None + + +def test_turning_the_microphone_on_asks_the_host_to_renegotiate(): + viewer, _ = _connected() + channel = Channel() + viewer._control_channel = channel + viewer.toggle_opus_mic(True) + assert viewer._config.share_my_audio_opus is True + assert '"renegotiate_request"' in channel.sent[0] + + +def test_turning_the_microphone_on_twice_costs_one_renegotiation(): + viewer, _ = _connected() + channel = Channel() + viewer._control_channel = channel + viewer._opus_audio_track = Track("audio") + viewer.toggle_opus_mic(True) + assert channel.sent == [] + + +def test_detaching_skips_transceivers_that_hold_a_different_track(): + # Two video slots and only one of them is ours; the other is the host's + # screen, and detaching it would blank the window. + viewer, pc = _connected() + ours = Track("video") + theirs = Transceiver("video", track=Track("video")) + mine = Transceiver("video", track=ours) + pc.transceivers = [theirs, mine] + viewer._viewer_screen_track = ours + viewer.toggle_share_screen(False) + assert theirs.sender.replaced == [] + assert mine.sender.replaced == [None] + + +def test_detaching_ignores_transceivers_of_the_other_kind(): + viewer, pc = _connected() + track = Track("audio") + audio_slot = Transceiver("audio", track=track) + pc.transceivers = [Transceiver("video", track=track), audio_slot] + viewer._opus_audio_track = track + viewer.toggle_opus_mic(False) + assert audio_slot.sender.replaced == [None] + + +def test_a_track_the_connection_no_longer_carries_is_still_stopped(): + # Renegotiation can replace the transceiver set underneath us; the + # capture thread has to be stopped even when its slot has gone. + viewer, pc = _connected() + ours = Track("video") + pc.transceivers = [Transceiver("video", track=Track("video"))] + viewer._viewer_screen_track = ours + viewer.toggle_share_screen(False) + assert ours.stopped + assert viewer._viewer_screen_track is None + + +def test_detaching_a_track_that_was_never_attached_is_a_no_op(): + viewer, pc = _connected() + pc.transceivers = [Transceiver("video")] + viewer.toggle_share_screen(False) + assert pc.transceivers[0].sender.replaced == [] + + +def test_detaching_without_a_peer_connection_is_a_no_op(): + viewer = _viewer() + viewer._viewer_screen_track = Track("video") + viewer.toggle_share_screen(False) + assert viewer._viewer_screen_track is not None, "nothing to detach from" + + +def test_a_sender_that_rejects_the_detach_still_stops_the_capture(): + # A PeerConnection torn down underneath us fails `replaceTrack`; the + # capture thread must still be stopped or it runs until the process ends. + viewer, pc = _connected() + track = Track("video") + slot = Transceiver("video", track=track) + slot.sender.replace_error = RuntimeError("connection closed") + pc.transceivers = [slot] + viewer._viewer_screen_track = track + viewer.toggle_share_screen(False) + assert track.stopped + assert viewer._viewer_screen_track is None + + +def test_a_capture_that_fails_to_stop_is_still_forgotten(): + viewer, pc = _connected() + + class _StubbornTrack(Track): + def stop(self): + raise OSError("device gone") + + track = _StubbornTrack("video") + pc.transceivers = [Transceiver("video", track=track)] + viewer._viewer_screen_track = track + viewer.toggle_share_screen(False) + assert viewer._viewer_screen_track is None + + +# --- host-initiated renegotiation --------------------------------------------- + +def test_a_renegotiation_offer_is_answered_over_the_control_channel(): + viewer, pc = _connected() + channel = Channel() + viewer._control_channel = channel + asyncio.run(viewer._async_handle_renegotiate("v=0 new-offer")) + assert pc.remote_descriptions[0].type == "offer" + assert '"renegotiate_answer"' in channel.sent[0] + assert "v=0 local-sdp" in channel.sent[0] + + +def test_a_renegotiation_attaches_the_media_the_toggles_asked_for( + screen_track, opus_track): + config = WebRTCConfig(share_my_screen=True, share_my_audio_opus=True) + viewer, pc = _connected(config=config) + viewer._control_channel = Channel() + pc.transceivers = [Transceiver("video"), Transceiver("video"), + Transceiver("audio")] + asyncio.run(viewer._async_handle_renegotiate("v=0 new-offer")) + assert viewer._viewer_screen_track is screen_track[0] + assert viewer._opus_audio_track is opus_track[0] + + +def test_a_renegotiation_does_not_re_attach_media_already_sending( + screen_track): + viewer, pc = _connected(config=WebRTCConfig(share_my_screen=True)) + viewer._control_channel = Channel() + existing = Track("video") + viewer._viewer_screen_track = existing + pc.transceivers = [Transceiver("video"), Transceiver("video")] + asyncio.run(viewer._async_handle_renegotiate("v=0 new-offer")) + assert viewer._viewer_screen_track is existing + assert screen_track == [] + + +def test_a_renegotiation_aiortc_rejects_sends_no_answer(): + viewer, pc = _connected() + channel = Channel() + viewer._control_channel = channel + pc.answer_error = RuntimeError("invalid SDP") + asyncio.run(viewer._async_handle_renegotiate("v=0 nonsense")) + assert channel.sent == [], "a half-built answer is worse than none" + + +def test_a_renegotiation_without_a_peer_connection_is_a_no_op(): + asyncio.run(_viewer()._async_handle_renegotiate("v=0 new-offer")) diff --git a/test/unit_test/headless/test_webrtc_viewer_session.py b/test/unit_test/headless/test_webrtc_viewer_session.py new file mode 100644 index 00000000..4e8cad5c --- /dev/null +++ b/test/unit_test/headless/test_webrtc_viewer_session.py @@ -0,0 +1,465 @@ +"""Answering a host's offer, watching what arrives, and letting go. + +`WebRTCDesktopViewer` is the offer-consumer half of the pair. Like the host +it needs aiortc to import at all, so nothing here ran on a CI square before +the `[webrtc]` extra joined the measured install. It is split three ways: +the media it attaches to the host's slots is in +`test_webrtc_viewer_media.py`, the control channel in +`test_webrtc_viewer_control.py`, and this file has the session around them. + +Three things here are load-bearing well beyond their size: + +* **The fingerprint is checked before any DTLS handshake.** That check is + the whole value of the pinning feature -- a signaling slot that has been + hijacked has to be caught while the offer is still text, not after + encrypted bytes are flowing, so a mismatch must not even build a + PeerConnection. +* **Inbound channels are routed by label**, and the control channel may + arrive already open. aiortc fires "datachannel" whenever it likes; if the + channel is open by then, "open" never fires again and an authentication + handshake that waited for it would wait forever. +* **The frame pump has to end quietly.** Nobody awaits it, so every way a + track can stop -- the host ending its share, the transport failing, the + viewer closing -- is a path that must not leave an exception behind. + +The doubles come from `headless._webrtc_doubles`. +""" +from __future__ import annotations + +import asyncio + +import pytest + +from headless._webrtc_doubles import ( + Bridge, Channel, FakePeerConnection, FrameTrack, HangingBridge, + Stoppable, Track, noop_ice_gathering, +) +from je_auto_control.utils.remote_desktop import webrtc_viewer as viewer_module +from je_auto_control.utils.remote_desktop.fingerprint import ( + FingerprintMismatchError, +) +from je_auto_control.utils.remote_desktop.webrtc_transport import WebRTCConfig +from je_auto_control.utils.remote_desktop.webrtc_viewer import ( + WebRTCDesktopViewer, +) + + +@pytest.fixture(autouse=True) +def fake_peer_connection(monkeypatch): + FakePeerConnection.instances = [] + monkeypatch.setattr(viewer_module, "RTCPeerConnection", FakePeerConnection) + monkeypatch.setattr(viewer_module, "wait_for_ice_gathering", + noop_ice_gathering) + yield + FakePeerConnection.instances = [] + + +@pytest.fixture(autouse=True) +def bridge(monkeypatch): + fake = Bridge() + monkeypatch.setattr(viewer_module, "get_bridge", lambda: fake) + return fake + + +@pytest.fixture +def screen_track(monkeypatch): + """Replace the capture the viewer attaches when it shares its screen.""" + made = [] + + def _factory(**kwargs): + track = Track("video") + track.kwargs = kwargs + made.append(track) + return track + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_transport" + ".ScreenVideoTrack", _factory, + ) + return made + + +@pytest.fixture +def opus_track(monkeypatch): + made = [] + + def _factory(): + track = Track("audio") + made.append(track) + return track + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicAudioTrack", + _factory, + ) + return made + + +def _viewer(**kwargs) -> WebRTCDesktopViewer: + kwargs.setdefault("token", "secret") + return WebRTCDesktopViewer(**kwargs) + + +def _connected(**kwargs): + """A viewer with a peer connection already in place.""" + viewer = _viewer(**kwargs) + viewer._pc = FakePeerConnection() + return viewer, viewer._pc + + +# --- construction ------------------------------------------------------------- + +def test_a_viewer_without_a_token_is_refused(): + with pytest.raises(ValueError, match="non-empty token"): + WebRTCDesktopViewer(token="") + + +def test_a_fresh_viewer_reports_no_session(): + viewer = _viewer() + assert viewer.authenticated is False + assert viewer.read_only is False + assert viewer.host_fingerprint is None + assert viewer.connection_state == "closed" + + +def test_asking_for_the_peer_connection_before_connecting_is_an_error(): + with pytest.raises(RuntimeError, match="connect first"): + _viewer()._require_pc() + + +# --- processing the offer ----------------------------------------------------- + +@pytest.mark.parametrize("offer", ["", " "]) +def test_an_empty_offer_is_refused_before_it_reaches_aiortc(offer): + with pytest.raises(ValueError, match="offer_sdp is empty"): + _viewer().process_offer(offer) + + +def test_processing_an_offer_returns_the_answer_sdp(): + assert _viewer().process_offer("v=0 host-offer") == "v=0 local-sdp" + + +def test_the_offer_is_applied_as_the_remote_description(): + viewer = _viewer() + viewer.process_offer("v=0 host-offer") + [pc] = FakePeerConnection.instances + [description] = pc.remote_descriptions + assert (description.sdp, description.type) == ("v=0 host-offer", "offer") + + +def test_the_answer_uses_the_viewers_own_ice_servers(): + config = WebRTCConfig(ice_servers=["stun:example:3478"]) + _viewer(config=config).process_offer("v=0 host-offer") + [pc] = FakePeerConnection.instances + assert [s.urls for s in pc.configuration.iceServers] == [ + "stun:example:3478", + ] + + +def test_a_second_offer_closes_the_first_peer_connection(): + viewer = _viewer() + viewer.process_offer("v=0 host-offer") + viewer.process_offer("v=0 host-offer-2") + first, second = FakePeerConnection.instances + assert first.closed + assert not second.closed + + +def test_a_pinned_fingerprint_that_matches_lets_the_offer_through(): + sdp = "v=0\r\na=fingerprint:sha-256 AB:CD\r\n" + assert _viewer().process_offer(sdp, "abcd") == "v=0 local-sdp" + + +def test_a_pinned_fingerprint_that_does_not_match_stops_the_handshake(): + # The point of pinning is catching a hijacked signaling slot while the + # offer is still text -- so nothing may be built before the check. + sdp = "v=0\r\na=fingerprint:sha-256 AB:CD\r\n" + with pytest.raises(FingerprintMismatchError): + _viewer().process_offer(sdp, "ffff") + assert not FakePeerConnection.instances + + +def test_an_offer_with_no_fingerprint_at_all_is_refused_when_pinned(): + with pytest.raises(FingerprintMismatchError): + _viewer().process_offer("v=0\r\nm=video 9 UDP/TLS/RTP/SAVPF\r\n", "ab") + + +# --- peer connection events --------------------------------------------------- + +def test_the_state_callback_sees_every_transition(): + seen = [] + viewer = _viewer(on_state_change=seen.append) + viewer.process_offer("v=0 host-offer") + pc = FakePeerConnection.instances[0] + for state in ("connecting", "connected", "closed"): + pc.connectionState = state + asyncio.run(pc.fire("connectionstatechange")) + assert seen == ["connecting", "connected", "closed"] + assert viewer.connection_state == "closed" + + +@pytest.mark.parametrize("state", ["failed", "closed", "disconnected"]) +def test_a_lost_connection_drops_the_authenticated_flag(state): + viewer = _viewer() + viewer.process_offer("v=0 host-offer") + viewer._authenticated = True + pc = FakePeerConnection.instances[0] + pc.connectionState = state + asyncio.run(pc.fire("connectionstatechange")) + assert viewer.authenticated is False + + +def test_a_raising_state_callback_does_not_break_the_handler(): + def _boom(_state): + raise RuntimeError("Qt widget already deleted") + + viewer = _viewer(on_state_change=_boom) + viewer.process_offer("v=0 host-offer") + pc = FakePeerConnection.instances[0] + pc.connectionState = "connected" + asyncio.run(pc.fire("connectionstatechange")) + + +def test_an_inbound_video_track_starts_the_frame_pump(): + viewer = _viewer() + viewer.process_offer("v=0 host-offer") + + async def _drive(): + FakePeerConnection.instances[0].fire("track", Track("video")) + assert viewer._receive_task is not None + viewer._receive_task.cancel() + + asyncio.run(_drive()) + + +def test_an_inbound_audio_track_starts_host_voice_playback(monkeypatch): + receivers = [] + + class _Receiver: + def __init__(self) -> None: + self.consumed = [] + receivers.append(self) + + def consume(self, track): + self.consumed.append(track) + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicReceiver", + _Receiver, + ) + viewer = _viewer() + viewer.process_offer("v=0 host-offer") + track = Track("audio") + FakePeerConnection.instances[0].fire("track", track) + assert receivers[0].consumed == [track] + + +def test_a_second_audio_track_does_not_open_a_second_player(monkeypatch): + class _Receiver: + def consume(self, track): + pass + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicReceiver", + _Receiver, + ) + viewer = _viewer() + viewer.process_offer("v=0 host-offer") + pc = FakePeerConnection.instances[0] + pc.fire("track", Track("audio")) + first = viewer._host_voice_receiver + pc.fire("track", Track("audio")) + assert viewer._host_voice_receiver is first + + +def test_a_viewer_with_no_speaker_keeps_the_video(monkeypatch): + def _no_device(): + raise OSError("no output device") + + monkeypatch.setattr( + "je_auto_control.utils.remote_desktop.webrtc_audio.OpusMicReceiver", + _no_device, + ) + viewer = _viewer() + viewer.process_offer("v=0 host-offer") + FakePeerConnection.instances[0].fire("track", Track("audio")) + assert viewer._host_voice_receiver is None + + +def test_a_track_that_is_neither_audio_nor_video_is_ignored(): + viewer = _viewer() + viewer.process_offer("v=0 host-offer") + FakePeerConnection.instances[0].fire("track", Track("application")) + assert viewer._receive_task is None + assert viewer._host_voice_receiver is None + + +def test_frames_reach_the_paint_callback(): + seen = [] + viewer = _viewer(on_frame=seen.append) + asyncio.run(viewer._consume_video(FrameTrack("f1", "f2"))) + assert seen == ["f1", "f2"] + + +def test_the_end_of_the_stream_is_an_ending_not_an_escaping_exception(): + # `MediaStreamError` is how aiortc says "the host stopped sharing" -- the + # ordinary way a session ends. It derives straight from Exception, so it + # is not covered by the OSError / RuntimeError arm, and nobody awaits + # this task: letting it out turns every clean disconnect into an + # un-retrieved task exception. The host's drain loop always caught it; + # this one did not until 2026-08-24. + from aiortc.mediastreams import MediaStreamError + viewer = _viewer(on_frame=lambda frame: None) + asyncio.run(viewer._consume_video( + FrameTrack("f1", ending=MediaStreamError()), + )) + + +def test_a_transport_failure_also_ends_the_stream_quietly(): + viewer = _viewer(on_frame=lambda frame: None) + asyncio.run(viewer._consume_video(FrameTrack(ending=OSError("reset")))) + + +def test_a_raising_paint_callback_does_not_end_the_stream(): + seen = [] + + def _cb(frame): + seen.append(frame) + raise RuntimeError("QImage conversion failed") + + viewer = _viewer(on_frame=_cb) + asyncio.run(viewer._consume_video(FrameTrack("f1", "f2"))) + assert seen == ["f1", "f2"] + + +def test_frames_with_no_listener_are_still_drained(): + track = FrameTrack("f1", "f2") + asyncio.run(_viewer()._consume_video(track)) + assert track.frames == [] + + +def test_the_frame_pump_stops_when_the_viewer_is_closing(): + viewer = _viewer(on_frame=lambda frame: None) + viewer._closed.set() + track = FrameTrack("f1") + asyncio.run(viewer._consume_video(track)) + assert track.frames == ["f1"], "it never pulled a frame" + + +# --- data channel routing ----------------------------------------------------- + +@pytest.mark.parametrize("label,attribute", [ + ("mic", "_mic_channel"), + ("files", "_files_channel"), + ("usb", "_usb_channel"), + ("ctrl", "_control_channel"), +]) +def test_each_inbound_channel_is_routed_by_its_label(label, attribute): + viewer = _viewer() + viewer.process_offer("v=0 host-offer") + channel = Channel(label) + FakePeerConnection.instances[0].fire("datachannel", channel) + assert getattr(viewer, attribute) is channel + + +def test_an_unknown_label_is_treated_as_the_control_channel(): + # The host names it "ctrl"; anything else arriving is still the channel + # the auth handshake has to go out on, so it must not be dropped. + viewer = _viewer() + channel = Channel("control") + viewer._attach_datachannel(channel) + assert viewer._control_channel is channel + + +def test_the_control_channel_sends_auth_as_soon_as_it_opens(): + viewer = _viewer(viewer_id="viewer-3") + channel = Channel("ctrl") + viewer._attach_datachannel(channel) + assert channel.sent == [], "not open yet" + channel.fire("open") + assert '"auth"' in channel.sent[0] + assert '"viewer-3"' in channel.sent[0] + + +def test_a_channel_already_open_on_arrival_still_authenticates(): + # aiortc may fire "datachannel" after the channel is open, in which case + # the "open" event never fires again and the handshake would never start. + viewer = _viewer() + viewer._attach_datachannel(Channel("ctrl", ready_state="open")) + assert '"auth"' in viewer._control_channel.sent[0] + + +def test_the_usb_channel_gets_a_passthrough_client(): + viewer = _viewer() + viewer._attach_datachannel(Channel("usb")) + assert viewer.usb_client() is not None + + +def test_a_viewer_with_no_usb_channel_has_no_client(): + assert _viewer().usb_client() is None + + +# --- teardown ----------------------------------------------------------------- + +def test_stopping_a_viewer_that_never_connected_is_a_no_op(bridge): + _viewer().stop() + assert bridge.deferred == [] + + +def test_stop_closes_the_connection_and_forgets_the_channels(): + viewer = _viewer() + viewer.process_offer("v=0 host-offer") + viewer._control_channel = Channel() + viewer._files_channel = Channel("files") + viewer._authenticated = True + viewer.stop() + assert FakePeerConnection.instances[0].closed + assert viewer._pc is None + assert viewer._control_channel is None + assert viewer._files_channel is None + assert viewer.authenticated is False + + +def test_stop_releases_every_stream_the_viewer_owns(): + viewer = _viewer() + voice, opus, screen, mic = (Stoppable() for _ in range(4)) + viewer._host_voice_receiver = voice + viewer._opus_audio_track = opus + viewer._viewer_screen_track = screen + viewer._mic_sender = mic + asyncio.run(viewer._async_stop()) + assert all(s.stopped for s in (voice, opus, screen, mic)) + assert viewer._viewer_screen_track is None + + +def test_stop_cancels_the_frame_pump(): + viewer = _viewer() + + async def _drive(): + task = asyncio.ensure_future(asyncio.sleep(10)) + viewer._receive_task = task + await viewer._async_stop() + return task + + task = asyncio.run(_drive()) + assert task.cancelled() + assert viewer._receive_task is None + + +def test_a_teardown_failure_does_not_abort_the_rest_of_the_teardown(): + viewer = _viewer() + failing = Stoppable(error=OSError("device gone")) + rest = Stoppable() + viewer._host_voice_receiver = failing + viewer._opus_audio_track = rest + asyncio.run(viewer._async_stop()) + assert rest.stopped + + +def test_a_stop_that_times_out_is_reported_rather_than_raised(monkeypatch): + monkeypatch.setattr(viewer_module, "get_bridge", HangingBridge) + viewer = _viewer() + viewer._pc = object() + viewer.stop() + + From dc3c61d1fd3ddb5a6caec1552c69f3defd479dd6 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 07:15:35 +0800 Subject: [PATCH 11/20] Test the window backends, which were never Linux-only to test Progress.md priced x11_backend and its neighbours as reachable by "only the two Linux squares", which is work that lifts two of nine when the floor is the lowest one. One command says otherwise: python -c "import je_auto_control.wrapper.window_backends.x11_backend" That passes on Windows with no python-Xlib installed. Every import of Xlib, Quartz, AppKit, ApplicationServices and comtypes under window_backends/ is inside a method -- the seam was built that way so a platform without a backend still imports. What kept them at 0% was that nobody had written the double. Stubbed into sys.modules, the package goes 14.24% -> 100% on all nine squares: x11 and macos from 0%, the selector from 46.34%, the base from 63.64%, the Win32 adapter from 89.19%. 184 tests, whole project 79.40% -> 80.22% on one machine. The doubles are pinned differently because their constants differ in kind. Xlib's carry their real X.h values, since those numbers go on the wire, and test_xlib_stub_values.py compares all twelve against the installed library wherever there is one -- both Linux squares on CI, and a throwaway pip --target here, twelve for twelve. pyobjc's are sentinels: each is a key into a dict the stub builds or a token handed back to a function the stub provides, so no value reaches any arithmetic, and the names are pinned against the real frameworks on the macOS squares instead. Worth knowing for the next one: `from package import name` resolves by attribute on the package before it consults sys.modules, so registering a double under its own dotted name does nothing where the real module is importable. The package has to be the double. --- Progress.md | 60 +- WHATS_NEW.md | 64 ++ test/unit_test/headless/_pyobjc_stub.py | 228 ++++++++ test/unit_test/headless/_xlib_stub.py | 233 ++++++++ .../headless/test_pyobjc_stub_names.py | 52 ++ .../headless/test_window_backend_macos.py | 492 ++++++++++++++++ .../headless/test_window_backend_selection.py | 299 ++++++++++ .../headless/test_window_backend_x11.py | 553 ++++++++++++++++++ .../headless/test_xlib_stub_values.py | 58 ++ 9 files changed, 2034 insertions(+), 5 deletions(-) create mode 100644 test/unit_test/headless/_pyobjc_stub.py create mode 100644 test/unit_test/headless/_xlib_stub.py create mode 100644 test/unit_test/headless/test_pyobjc_stub_names.py create mode 100644 test/unit_test/headless/test_window_backend_macos.py create mode 100644 test/unit_test/headless/test_window_backend_selection.py create mode 100644 test/unit_test/headless/test_window_backend_x11.py create mode 100644 test/unit_test/headless/test_xlib_stub_values.py diff --git a/Progress.md b/Progress.md index 6a751ab6..59d93597 100644 --- a/Progress.md +++ b/Progress.md @@ -371,7 +371,7 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl | `utils/executor` | 653 / 3,906 | 83.3% | | `utils/usb` | 573 / 2,137 | 73.2% | | `utils/mcp_server` | 573 / 4,618 | 87.6% | -| `wrapper/window_backends` | 361 / 477 | 24.3% | +| ~~`wrapper/window_backends`~~ | ~~361 / 477~~ | **2026-08-24 補完（100%）** | | `utils/hotkey` | 221 / 426 | 48.1% | | `utils/rest_api` | 146 / 808 | 81.9% | @@ -389,8 +389,8 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl | `utils/mcp_server/tools/_handlers.py` | 348 | 同 `action_executor.py` | | `utils/remote_desktop/signaling_server.py` | 155 | **CI 動不了**：要 `[signaling]` extra（fastapi／uvicorn），沒裝 | | ~~`utils/remote_desktop/multi_viewer.py`~~ | ~~146~~ | **2026-08-24 補完（100%）** | -| `wrapper/window_backends/x11_backend.py` | 133 | 只有 Linux 那兩格跑得到，抬得動地板 | -| `utils/accessibility/backends/linux_backend.py` | 132 | 同上 | +| ~~`wrapper/window_backends/x11_backend.py`~~ | ~~133~~ | **2026-08-24 補完（100%）**。「只有 Linux 那兩格跑得到」是錯的，見下 | +| `utils/accessibility/backends/linux_backend.py` | 132 | 同上，尚未補 | **下一步的順序**： @@ -399,8 +399,10 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl 2. 掃不到的那批 adapter：目前卡在「被呼叫者是 class」。要嘛從 class 自己的方法標注 長出一個替身物件，要嘛承認那批不掃——**得先決定，因為前者會讓「跑起來了」和 「驗到了東西」分家**。 -3. `utils/accessibility` 與 `wrapper/window_backends` 的 Linux 後端：抬得動地板， - 但需要一套 Xlib／AT-SPI 的替身。 +3. ~~`wrapper/window_backends`~~ **2026-08-24 整包補完（100%）**，見下下節； + `utils/accessibility` 還沒補，做法照抄——AT-SPI 那一層要的是 `SessionBus` + 的替身，不是 `_AtspiConnection` 的（現有的 `test_accessibility_linux.py` + 換掉的是後者，所以整個 D-Bus 呼叫層還沒被跑過）。 `utils/office`（77）與 `signaling_server`（155）**不要碰**：`quality.yml` 沒裝 `[office]`／`[signaling]`，補的測試會整批 skip，對地板一個點都不動。要補之前 @@ -445,6 +447,54 @@ RateLimiter 與 SessionPermissions，檢視端只存下 callback——而每一 但地板取的是九宮格最低那一格，**要等 CI 的 `coverage report` 印出來才能改**—— 這條規則在上面那節寫過，不從單機的數字推。 +#### 2026-08-24：平台後端不是「只有那一格跑得到」，是沒人給過替身 + +上面第 3 項與缺口表都寫著 `x11_backend.py`／`linux_backend.py` +「只有 Linux 那兩格跑得到」。**實測是錯的**：`wrapper/window_backends/` 底下 +八個平台模組，每一句 `import Xlib`／`import Quartz`／`import AppKit`／ +`import ApplicationServices`／`import comtypes` **全部在函式內**，所以在一台 +沒裝任何一個的 Windows 上這八個模組都 import 得起來—— + +```bash +python -c "import je_auto_control.wrapper.window_backends.x11_backend" # 在 Windows 上就過 +``` + +擋住它們的從來不是平台，是**沒有替身**。把套件塞進 `sys.modules` 之後， +同一份測試在九格都跑得到，而不是只有兩格；地板取最低那一格，所以 +「九格都漲」比「兩格漲」更有意義。 + +| 檔案 | 之前 | 之後 | +| --- | ---: | ---: | +| `window_backends/x11_backend.py` | 0.00% | **100%** | +| `window_backends/macos_backend.py` | 0.00% | **100%** | +| `window_backends/__init__.py`（後端選擇） | 46.34% | **100%** | +| `window_backends/base.py` | 63.64% | **100%** | +| `window_backends/windows_backend.py` | 89.19% | **100%** | +| `window_backends/null_backend.py` | 100% | 100% | + +**替身要對得上真貨，否則只是自己跟自己同意。** 兩支 stub 的處理不一樣， +因為兩邊的常數性質不同： + +- `_xlib_stub.py` 的常數**帶真值**（`SubstructureRedirectMask` 就是 `1 << 20`）， + 因為那些數字會真的上線；`test_xlib_stub_values.py` 在有裝 python-Xlib 的地方 + （CI 的兩格 Linux）逐一比對，對不上就當場紅。本機另外用 + `pip install --target` 拉 0.33 實測過一輪，12 個常數全中。 +- `_pyobjc_stub.py` 的常數**是哨兵**，因為它們不是 dict key 就是原封不動傳回 + 同一支 stub 函式的 token，數值到不了任何算術；`test_pyobjc_stub_names.py` + 改成在 macOS 那兩格比對**名字存在**，另外只釘那三個真的會做位元運算的 + window-list 旗標。 + +**一個踩到的坑**：`from je_auto_control.windows.window import windows_window_manage` +這種 `from 套件 import 名字`，Python **先用套件的屬性解析**，解析不到才回頭查 +`sys.modules`。只把替身放進 `sys.modules["...windows_window_manage"]` 在 +Windows 上完全沒效果——真的 Win32 模組會被呼叫。要換掉的是**那個套件**。 + +本機（Windows／3.14）全專案 79.40% → **80.22%**，184 個新測試。 +地板一樣要等九宮格。**下一個照抄的對象是 `utils/accessibility`**：那裡的 +`windows_backend.py`（446 行沒蓋到）用 comtypes，`macos_backend.py`（73，0%） +用 pyobjc，`linux_backend.py` 的 D-Bus 呼叫層用 `SessionBus`——三個都是 +同樣的 lazy import，同樣塞得進 `sys.modules`。 + ### mypy：整包把關，**豁免清單已經清空** `TODO` → **完成（2026-08-22）** diff --git a/WHATS_NEW.md b/WHATS_NEW.md index e8fd271a..d5c2fbd9 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -143,6 +143,70 @@ The thing that had to be split, in the end, was the test files — `CLAUDE.md`'s 750-line limit has no exception for new ones. Host: session (557) and channels (739). Viewer: session (466), media (385), control (519). +### The Platform Backends Were Never Linux-Only To Test, They Just Had No Double + +`Progress.md` listed `x11_backend.py` and its accessibility neighbour as +reachable by "only the two Linux squares", and priced them accordingly: work +that lifts two squares out of nine, when the floor is the lowest one. + +That was wrong, and one command says so: + +```bash +python -c "import je_auto_control.wrapper.window_backends.x11_backend" +``` + +That passes on Windows, with no python-Xlib installed. Every `import Xlib`, +`import Quartz`, `import AppKit`, `import ApplicationServices` and +`import comtypes` under `wrapper/window_backends/` is *inside* a method — the +seam was built that way on purpose, so a platform without a backend still +imports. What kept those modules at 0% was never the platform. It was that +nobody had written the double. + +With the frameworks stubbed into `sys.modules`, the whole package goes from +14.24% to 100% and does it on all nine squares: + +| module | before | after | +| --- | ---: | ---: | +| `x11_backend.py` | 0.00% | **100%** | +| `macos_backend.py` | 0.00% | **100%** | +| `__init__.py` (backend selection) | 46.34% | **100%** | +| `base.py` | 63.64% | **100%** | +| `windows_backend.py` | 89.19% | **100%** | + +184 tests, and what they are about is protocol arithmetic that fails quietly. +EWMH requests are addressed to the *root* window with `SubstructureRedirect`, +because that is what routes them to the window manager — sent to the window +itself they reach nobody. `_NET_CLIENT_LIST_STACKING` is bottom-to-top, so it +is reversed and `_NET_CLIENT_LIST`, which carries no order at all, is not. +`move()` sizes the client while `window_rect()` reads the frame, so the +decorations come off the requested size — get that wrong and a window shrinks +by a title bar every time a script round-trips it. On macOS an AX call +returns an error code where zero means success, and `close()` returns +`not error`: read backwards, that is a cheerful "yes" for every action that +failed. + +**A double that nobody checks is a test agreeing with itself**, so the two +stubs are pinned differently, because their constants are different in kind: + +* The Xlib stub's constants carry their real `X.h` values — + `SubstructureRedirectMask` is `1 << 20` — because those numbers go on the + wire. `test_xlib_stub_values.py` compares all twelve against the installed + library wherever there is one, which on CI is both Linux squares. (They + were also checked here against python-Xlib 0.33 pulled into a throwaway + `--target` directory: twelve for twelve.) +* The pyobjc stub's constants are sentinels, because every one of them is + either a key into an info dictionary the stub itself builds or a token + handed straight back to a function the stub itself provides — no number + reaches any arithmetic. `test_pyobjc_stub_names.py` pins the *names* + against the real frameworks on the macOS squares instead, plus the three + window-list flags that genuinely get OR-ed together. + +One thing worth knowing before writing the next one of these: `from +je_auto_control.windows.window import windows_window_manage` resolves by +**attribute on the package** before it consults `sys.modules`. Registering the +double under its own dotted name does nothing on a machine where the real +module is importable — the package it is read off has to be the double. + ### The Viewer Logged Every Clean Disconnect As An Unhandled Task Exception `WebRTCDesktopViewer._consume_video` caught `(OSError, RuntimeError)`. aiortc diff --git a/test/unit_test/headless/_pyobjc_stub.py b/test/unit_test/headless/_pyobjc_stub.py new file mode 100644 index 00000000..8685a786 --- /dev/null +++ b/test/unit_test/headless/_pyobjc_stub.py @@ -0,0 +1,228 @@ +"""A pyobjc stand-in for the macOS window backend, usable off macOS. + +`macos_backend.py` imports `Quartz`, `AppKit` and `ApplicationServices` +*inside* its methods, so the module loads on every platform and only the +calls need a Mac. Stubs in `sys.modules` therefore reach all of it from all +nine CI squares rather than the two Darwin ones -- which matters because the +coverage floor is the lowest square. + +Unlike the Xlib stub, the constants here are sentinels rather than real +values, and deliberately so: every one of them is either a dictionary key +into an info dict this file also creates, or an opaque token handed straight +back to a function this file also provides. Their numeric values never reach +any arithmetic in the code under test, so pinning them would pin nothing. +What *is* worth pinning is that the names exist at all -- +`test_pyobjc_stub_names.py` checks that against the installed frameworks on +the macOS squares. + +The accessibility half models one thing carefully: **AX functions return an +error code, where zero means success.** `close()` and `minimize()` return +`not ax.AXUIElementPerformAction(...)`, so a stub that returned a truthy +"success" would invert every one of those tests. + +Nothing here is a test; the file is named so pytest does not collect it. +""" +from __future__ import annotations + +import sys +import types + +#: Quartz's window-info dictionary keys, spelled as pyobjc spells them. +WINDOW_KEYS = ( + "kCGWindowNumber", "kCGWindowName", "kCGWindowLayer", + "kCGWindowOwnerPID", "kCGWindowBounds", +) + +#: The rest of the surface the backend names, by module. +QUARTZ_NAMES = WINDOW_KEYS + ( + "kCGWindowListOptionOnScreenOnly", "kCGWindowListExcludeDesktopElements", + "kCGNullWindowID", "kAXValueCGPointType", "kAXValueCGSizeType", + "CGWindowListCopyWindowInfo", "CGPoint", "CGSize", +) +APPKIT_NAMES = ( + "NSWorkspace", "NSRunningApplication", + "NSApplicationActivateIgnoringOtherApps", +) +AX_NAMES = ( + "AXUIElementCreateApplication", "AXUIElementCopyAttributeValue", + "AXUIElementSetAttributeValue", "AXUIElementPerformAction", + "AXValueCreate", "AXValueGetValue", +) + +#: What an AX call returns when it worked. Zero, and the backend reads it as +#: `not error`, so this is load-bearing in every action test. +AX_SUCCESS = 0 +AX_FAILURE = -25200 # kAXErrorCannotComplete, in spirit + + +def window_info(number: int, *, name: str = "", layer: int = 0, + pid: int = 0, bounds=None) -> dict: + """One entry of what `CGWindowListCopyWindowInfo` returns.""" + info = { + "kCGWindowNumber": number, + "kCGWindowName": name, + "kCGWindowLayer": layer, + "kCGWindowOwnerPID": pid, + } + if bounds is not None: + left, top, width, height = bounds + info["kCGWindowBounds"] = {"X": left, "Y": top, + "Width": width, "Height": height} + return info + + +class AXElement: + """An accessibility element: attributes, and what was done to it.""" + + def __init__(self, **attributes) -> None: + self.attributes = dict(attributes) + self.actions = [] + self.assignments = [] + self.set_error = AX_SUCCESS + self.action_error = AX_SUCCESS + self.read_error = AX_SUCCESS + + +class AXPoint: + """What `AXValueCreate(kAXValueCGPointType, ...)` hands back.""" + + def __init__(self, x, y) -> None: + self.x = x + self.y = y + + +class AXSize: + def __init__(self, width, height) -> None: + self.width = width + self.height = height + + +class World: + """The Mac the backend thinks it is talking to.""" + + def __init__(self, windows=None, ax_windows=None, frontmost_pid=None, + running_pids=None): + self.windows = list(windows or []) + self.ax_windows = dict(ax_windows or {}) # pid -> [AXElement] + self.frontmost_pid = frontmost_pid + # Whether `NSRunningApplication` still answers for a pid. Separate + # from `ax_windows` on purpose: an application can quit between the + # pid lookup and the activation, which is its own branch. + self.running_pids = (set() if running_pids is None + else set(running_pids)) + self.list_options = [] + self.activated = [] + self.ax_list_error = AX_SUCCESS + + # -- Quartz -- + def copy_window_info(self, options, relative_to): + self.list_options.append((options, relative_to)) + return list(self.windows) + + # -- ApplicationServices -- + def ax_application(self, pid): + return ("application", int(pid)) + + def ax_copy_attribute(self, element, attribute, _placeholder): + if isinstance(element, tuple) and element[0] == "application": + if attribute == "AXWindows": + if self.ax_list_error: + return (self.ax_list_error, None) + return (AX_SUCCESS, self.ax_windows.get(element[1], [])) + return (AX_FAILURE, None) + if element.read_error: + return (element.read_error, None) + if attribute not in element.attributes: + return (AX_FAILURE, None) + return (AX_SUCCESS, element.attributes[attribute]) + + def ax_set_attribute(self, element, attribute, value): + element.assignments.append((attribute, value)) + if element.set_error: + return element.set_error + element.attributes[attribute] = value + return AX_SUCCESS + + def ax_perform_action(self, element, action): + element.actions.append(action) + return element.action_error + + def ax_value_create(self, kind, value): + return (kind, value) + + def ax_value_get(self, value, kind, _placeholder): + if not isinstance(value, tuple) or value[0] != kind: + return (False, None) + return (True, value[1]) + + +def install(monkeypatch, world: World) -> World: + """Put Quartz, AppKit and ApplicationServices in `sys.modules`.""" + quartz = types.ModuleType("Quartz") + for name in WINDOW_KEYS: + setattr(quartz, name, name) + quartz.kCGWindowListOptionOnScreenOnly = 1 + quartz.kCGWindowListExcludeDesktopElements = 16 + quartz.kCGNullWindowID = 0 + quartz.kAXValueCGPointType = "point" + quartz.kAXValueCGSizeType = "size" + quartz.CGWindowListCopyWindowInfo = world.copy_window_info + quartz.CGPoint = AXPoint + quartz.CGSize = AXSize + + appkit = types.ModuleType("AppKit") + appkit.NSApplicationActivateIgnoringOtherApps = 2 + appkit.NSWorkspace = _Workspace(world) + appkit.NSRunningApplication = _RunningApplication(world) + + services = types.ModuleType("ApplicationServices") + services.AXUIElementCreateApplication = world.ax_application + services.AXUIElementCopyAttributeValue = world.ax_copy_attribute + services.AXUIElementSetAttributeValue = world.ax_set_attribute + services.AXUIElementPerformAction = world.ax_perform_action + services.AXValueCreate = world.ax_value_create + services.AXValueGetValue = world.ax_value_get + + for name, module in (("Quartz", quartz), ("AppKit", appkit), + ("ApplicationServices", services)): + monkeypatch.setitem(sys.modules, name, module) + return world + + +def install_missing(monkeypatch, name: str = "Quartz") -> None: + """Make one framework import fail, as a Mac without pyobjc does. + + `None` in `sys.modules` is the import system's own way of recording + "this one is not there": `import Quartz` then raises ImportError without + going near the filesystem, which is exactly what the probe is written to + survive. + """ + monkeypatch.setitem(sys.modules, name, None) + + +class _Workspace: + def __init__(self, world: World) -> None: + self._world = world + + def sharedWorkspace(self): # noqa: N802 # reason: the AppKit name + return self + + def frontmostApplication(self): # noqa: N802 # reason: the AppKit name + if self._world.frontmost_pid is None: + return None + return types.SimpleNamespace( + processIdentifier=lambda: self._world.frontmost_pid) + + +class _RunningApplication: + def __init__(self, world: World) -> None: + self._world = world + + def runningApplicationWithProcessIdentifier_(self, pid): # noqa: N802 + if pid not in self._world.running_pids: + return None + world = self._world + + return types.SimpleNamespace( + activateWithOptions_=lambda options: world.activated.append( + (pid, options))) diff --git a/test/unit_test/headless/_xlib_stub.py b/test/unit_test/headless/_xlib_stub.py new file mode 100644 index 00000000..2ca08d2c --- /dev/null +++ b/test/unit_test/headless/_xlib_stub.py @@ -0,0 +1,233 @@ +"""A python-Xlib stand-in, so the X11 backends can be tested off Linux. + +`python-Xlib` is a Linux/BSD-only dependency, and both X11 backends import it +*inside* their methods — so the modules load anywhere and only the calls need +a display. That is what makes them testable on all nine CI squares rather +than the two that have X: put a stub in `sys.modules` and the lazy import +finds it, on Linux as well as on Windows and macOS. + +Two things make this a stub rather than a mock. + +**The constants carry their real values.** `SubstructureRedirectMask` here is +`1 << 20` because that is what `X.h` says, so a test asserting an event mask +is asserting the number that goes on the wire. `test_xlib_stub_values.py` +compares every constant below against the real module wherever it is +installed, which on CI is both Linux squares — so a wrong value here is a +named failure there rather than a test that agrees with itself. + +**The display is a real object graph, not a recorder.** Windows have +properties, properties are keyed by interned atom, and `query_tree` returns a +parent — because the code under test walks up to the frame, interns atoms +once per connection, and reads properties by id. A recorder would let a +backend that never interned an atom pass. + +Nothing here is a test; the file is named so pytest does not collect it. +""" +from __future__ import annotations + +import sys +import types + +# --- X.h ----------------------------------------------------------------- + +#: `X.h` event masks and sentinels the window backend actually names. +X_CONSTANTS = { + "NONE": 0, + "CurrentTime": 0, + "AnyPropertyType": 0, + "KeyPressMask": 1 << 0, + "KeyReleaseMask": 1 << 1, + "ButtonPressMask": 1 << 2, + "ButtonReleaseMask": 1 << 3, + "SubstructureNotifyMask": 1 << 19, + "SubstructureRedirectMask": 1 << 20, +} + +#: `Xatom.h` predefined atoms, by their fixed protocol numbers. +XATOM_CONSTANTS = { + "CARDINAL": 6, + "STRING": 31, + "WINDOW": 33, +} + + +# --- the object graph ---------------------------------------------------- + +class Event: + """One X event, recording the keywords the backend built it from.""" + + def __init__(self, kind: str, **fields) -> None: + self.kind = kind + self.fields = fields + + def __repr__(self) -> str: # pragma: no cover - debugging aid + return f"Event({self.kind!r}, {self.fields!r})" + + +class _Property: + def __init__(self, value) -> None: + self.value = value + + +class Window: + """A window with properties, a parent, and a geometry.""" + + def __init__(self, display, window_id: int, *, properties=None, + geometry=None, wm_name=None, parent_id=None) -> None: + self._display = display + self.id = int(window_id) + self.properties = dict(properties or {}) + self.geometry = geometry or (0, 0, 0, 0) + self.wm_name = wm_name + self.parent_id = parent_id + self.sent = [] + self.mapped = None + self.property_error = None + self.tree_error = None + self.geometry_error = None + self.send_error = None + + # -- the python-Xlib surface -- + def get_full_property(self, atom: int, kind): + if self.property_error is not None: + raise self.property_error + name = self._display.atom_name(atom) + if name not in self.properties: + return None + return _Property(self.properties[name]) + + def get_wm_name(self): + return self.wm_name + + def query_tree(self): + if self.tree_error is not None: + raise self.tree_error + parent = (None if self.parent_id is None + else self._display.window(self.parent_id)) + return types.SimpleNamespace(parent=parent) + + def get_geometry(self): + if self.geometry_error is not None: + raise self.geometry_error + x, y, width, height = self.geometry + return types.SimpleNamespace(x=x, y=y, width=width, height=height) + + def send_event(self, event, event_mask=0, propagate=False): + if self.send_error is not None: + raise self.send_error + self.sent.append((event, event_mask, propagate)) + + def map(self): + self.mapped = True + + def unmap(self): + self.mapped = False + + +class Display: + """One X connection: an atom table, a root, and a window table.""" + + #: Interned atoms start above the predefined range so a test can tell an + #: interned atom from `Xatom.WINDOW` at a glance. + FIRST_INTERNED_ATOM = 100 + + def __init__(self) -> None: + self.atoms = {} + self.flushes = 0 + self._windows = {} + self.root = Window(self, 1) + self._windows[1] = self.root + + # -- the python-Xlib surface -- + def intern_atom(self, name: str) -> int: + if name not in self.atoms: + self.atoms[name] = self.FIRST_INTERNED_ATOM + len(self.atoms) + return self.atoms[name] + + def screen(self): + return types.SimpleNamespace(root=self.root) + + def create_resource_object(self, kind: str, window_id: int) -> Window: + assert kind == "window" + return self.window(window_id) + + def flush(self) -> None: + self.flushes += 1 + + # -- test helpers -- + def atom_name(self, atom: int): + for name, value in self.atoms.items(): + if value == atom: + return name + return None + + def window(self, window_id: int, **kwargs) -> Window: + window_id = int(window_id) + if window_id not in self._windows: + self._windows[window_id] = Window(self, window_id, **kwargs) + elif kwargs: + existing = self._windows[window_id] + for key, value in kwargs.items(): + setattr(existing, key, value) + return self._windows[window_id] + + def set_root_property(self, name: str, value) -> None: + """Publish an EWMH property on the root, interning its atom.""" + self.intern_atom(name) + self.root.properties[name] = value + + +def install(monkeypatch, display=None) -> Display: + """Put the stub in `sys.modules` and hand back the display it opens. + + Shadows the real `Xlib` for the duration wherever one is installed, so + the same test reads the same on every square. + """ + display = display if display is not None else Display() + + x_module = types.ModuleType("Xlib.X") + for name, value in X_CONSTANTS.items(): + setattr(x_module, name, value) + + xatom_module = types.ModuleType("Xlib.Xatom") + for name, value in XATOM_CONSTANTS.items(): + setattr(xatom_module, name, value) + + event_module = types.SimpleNamespace( + ClientMessage=lambda **kwargs: Event("ClientMessage", **kwargs), + KeyPress=lambda **kwargs: Event("KeyPress", **kwargs), + KeyRelease=lambda **kwargs: Event("KeyRelease", **kwargs), + ButtonPress=lambda **kwargs: Event("ButtonPress", **kwargs), + ButtonRelease=lambda **kwargs: Event("ButtonRelease", **kwargs), + ) + protocol_module = types.ModuleType("Xlib.protocol") + protocol_module.event = event_module + + display_module = types.ModuleType("Xlib.display") + display_module.Display = lambda *args, **kwargs: display + + package = types.ModuleType("Xlib") + package.X = x_module + package.Xatom = xatom_module + package.protocol = protocol_module + package.display = display_module + + for name, module in (("Xlib", package), ("Xlib.X", x_module), + ("Xlib.Xatom", xatom_module), + ("Xlib.protocol", protocol_module), + ("Xlib.display", display_module)): + monkeypatch.setitem(sys.modules, name, module) + return display + + +def install_failing(monkeypatch, error) -> None: + """Install a stub whose `Display()` raises, as a headless session does.""" + def _raise(*args, **kwargs): + raise error + + display_module = types.ModuleType("Xlib.display") + display_module.Display = _raise + package = types.ModuleType("Xlib") + package.display = display_module + monkeypatch.setitem(sys.modules, "Xlib", package) + monkeypatch.setitem(sys.modules, "Xlib.display", display_module) diff --git a/test/unit_test/headless/test_pyobjc_stub_names.py b/test/unit_test/headless/test_pyobjc_stub_names.py new file mode 100644 index 00000000..6688c03e --- /dev/null +++ b/test/unit_test/headless/test_pyobjc_stub_names.py @@ -0,0 +1,52 @@ +"""Hold the pyobjc stub to the frameworks it stands in for. + +`_pyobjc_stub.py` lets the macOS backends be tested on all nine CI squares by +shadowing `Quartz`, `AppKit` and `ApplicationServices` in `sys.modules`. The +risk that buys is a stub that answers for a name the real framework does not +have -- a test that passes everywhere and a backend that raises +`AttributeError` on the one platform it exists for. + +pyobjc is a hard dependency on Darwin, so on the macOS squares the genuine +frameworks are installed and every name the stub declares is looked up on +them. Elsewhere there is nothing to compare against and these skip. + +Names, not values. Every constant in the stub is either a key into an info +dictionary the stub itself builds or an opaque token handed back to a +function the stub itself provides, so none of their numbers reaches any +arithmetic in the code under test. What matters is that the backend is +spelling real API. +""" +from __future__ import annotations + +import pytest + +from headless import _pyobjc_stub as objc_stub + +pytest.importorskip("Quartz", reason="pyobjc is a macOS-only dependency") + + +@pytest.mark.parametrize("name", sorted(objc_stub.QUARTZ_NAMES)) +def test_every_quartz_name_the_stub_answers_for_exists(name): + import Quartz + assert hasattr(Quartz, name), f"Quartz.{name}" + + +@pytest.mark.parametrize("name", sorted(objc_stub.APPKIT_NAMES)) +def test_every_appkit_name_the_stub_answers_for_exists(name): + import AppKit + assert hasattr(AppKit, name), f"AppKit.{name}" + + +@pytest.mark.parametrize("name", sorted(objc_stub.AX_NAMES)) +def test_every_accessibility_name_the_stub_answers_for_exists(name): + import ApplicationServices + assert hasattr(ApplicationServices, name), f"ApplicationServices.{name}" + + +def test_the_window_list_options_are_the_flags_the_backend_composes(): + # These two are the only Quartz numbers the backend does arithmetic on: + # it ORs them into the argument of CGWindowListCopyWindowInfo. + import Quartz + assert Quartz.kCGWindowListOptionOnScreenOnly == 1 + assert Quartz.kCGWindowListExcludeDesktopElements == 16 + assert Quartz.kCGNullWindowID == 0 diff --git a/test/unit_test/headless/test_window_backend_macos.py b/test/unit_test/headless/test_window_backend_macos.py new file mode 100644 index 00000000..f5c7e621 --- /dev/null +++ b/test/unit_test/headless/test_window_backend_macos.py @@ -0,0 +1,492 @@ +"""macOS window management, where reading and acting are two APIs. + +`macos_backend.py` was at 0% on every square. Like the X11 one it imports its +platform libraries inside its methods, so stubs in `sys.modules` +(`_pyobjc_stub.py`) exercise it from all nine rather than the two Darwin +squares -- and the coverage floor is the lowest square, so that difference is +the whole point. + +The split down the middle of this backend is what the tests are about: + +* **Quartz reads, the accessibility API acts**, and they have different + permission stories. Listing, rectangles and ownership need no grant; + moving, closing, minimising and raising are gated behind TCC, and until the + user grants Accessibility every one of them silently does nothing. So each + action reports refusal rather than a false success. +* **The two APIs do not share a handle.** A `CGWindowID` has no public bridge + to an `AXUIElement`, so the window is found again inside its owning + application by origin and then by title. Origin is the stronger signal -- + two windows cannot share one at the same moment, while titles are empty or + duplicated all the time -- and getting that preference backwards moves the + wrong window of an application that has several. +* **An AX call returns an error code, and zero is success.** `close()` and + `minimize()` return `not error`. Reading that backwards is a "yes" for + every failed action, which is exactly the answer a caller cannot recover + from. +* **Layer 0 is the application layer.** Menu bars, the Dock and status items + live above it and are not what anyone means by "the Safari window". + +`test_pyobjc_stub_names.py` holds the stub's surface to the real frameworks +wherever they are installed, which on CI is the macOS squares. +""" +from __future__ import annotations + +import pytest + +from headless import _pyobjc_stub as objc_stub +from headless._pyobjc_stub import AX_FAILURE, AXElement, World, window_info +from je_auto_control.utils.exception.exceptions import ( + AutoControlUnsupportedOperationException, +) +from je_auto_control.wrapper.window_backends import macos_backend as macos +from je_auto_control.wrapper.window_backends.macos_backend import ( + MacOSWindowBackend, _point, +) + + +@pytest.fixture +def on_darwin(monkeypatch): + monkeypatch.setattr(macos.sys, "platform", "darwin") + + +def _build(monkeypatch, world: World) -> MacOSWindowBackend: + objc_stub.install(monkeypatch, world) + backend = MacOSWindowBackend() + assert backend.available, "the stubbed frameworks import" + return backend + + +@pytest.fixture +def world(): + return World() + + +@pytest.fixture +def backend(monkeypatch, on_darwin, world): + return _build(monkeypatch, world) + + +# --- availability ------------------------------------------------------------- + +def test_the_backend_is_unavailable_off_macos(monkeypatch, world): + monkeypatch.setattr(macos.sys, "platform", "linux") + objc_stub.install(monkeypatch, world) + assert MacOSWindowBackend().available is False + + +def test_a_mac_without_pyobjc_is_unavailable_rather_than_fatal(monkeypatch, + on_darwin): + # pyobjc is a hard dependency on Darwin, but a broken or partial install + # must degrade to the null backend, not to an ImportError at start-up. + objc_stub.install_missing(monkeypatch, "Quartz") + assert MacOSWindowBackend().available is False + + +def test_the_backend_names_both_apis_it_uses(backend): + assert backend.name == "macos-quartz-ax" + + +# --- listing ------------------------------------------------------------------ + +def test_listing_asks_quartz_for_on_screen_windows_only(backend, world): + backend.list_windows() + [(options, relative_to)] = world.list_options + assert options == 1 | 16, "on-screen only, excluding desktop elements" + assert relative_to == 0, "kCGNullWindowID" + + +def test_listing_keeps_the_order_quartz_gives(backend, world): + # Quartz returns front-to-back, which is already what a caller means by + # "the first window that matches". + world.windows = [window_info(3, name="front"), window_info(1, name="back")] + assert backend.list_windows() == [(3, "front"), (1, "back")] + + +def test_listing_skips_everything_above_the_application_layer(backend, world): + world.windows = [window_info(1, name="Menubar", layer=25), + window_info(2, name="Editor", layer=0)] + assert backend.list_windows() == [(2, "Editor")] + + +def test_listing_skips_a_window_with_no_id(backend, world): + # Quartz omits the number for windows a caller has no business addressing. + world.windows = [window_info(0, name="anonymous")] + assert backend.list_windows() == [] + + +def test_a_window_with_no_title_lists_with_an_empty_one(backend, world): + # Screen-recording permission is what puts titles in this list; without + # it Quartz still reports the windows, with the names left out. + world.windows = [{"kCGWindowNumber": 7, "kCGWindowLayer": 0}] + assert backend.list_windows() == [(7, "")] + + +def test_a_desktop_quartz_answers_nothing_for_lists_nothing(backend, world): + world.windows = [] + assert backend.list_windows() == [] + + +# --- the focused window ------------------------------------------------------- + +def test_the_foreground_window_is_the_front_one_of_the_front_app(backend, + world): + world.frontmost_pid = 501 + world.windows = [window_info(1, pid=99), window_info(2, pid=501), + window_info(3, pid=501)] + assert backend.foreground_window() == 2, "front-to-back, so the first" + + +def test_no_frontmost_application_reads_as_no_window(backend, world): + world.frontmost_pid = None + assert backend.foreground_window() == 0 + + +def test_a_frontmost_app_with_no_ordinary_window_reads_as_none(backend, + world): + world.frontmost_pid = 501 + world.windows = [window_info(1, pid=501, layer=25)] + assert backend.foreground_window() == 0 + + +# --- rectangles and ownership ------------------------------------------------- + +def test_the_rectangle_is_the_bounds_quartz_reports(backend, world): + world.windows = [window_info(7, bounds=(100, 200, 640, 480))] + assert backend.window_rect(7) == (100, 200, 740, 680) + + +def test_a_rectangle_for_a_window_that_is_gone_is_none(backend, world): + # The lookup walks the whole list before giving up, so an unrelated + # window in front of it is the ordinary case, not an edge one. + world.windows = [window_info(9, bounds=(0, 0, 1, 1))] + assert backend.window_rect(7) is None + + +def test_a_window_quartz_reports_without_bounds_has_no_rectangle(backend, + world): + world.windows = [window_info(7)] + assert backend.window_rect(7) is None + + +def test_the_owning_pid_comes_from_the_window_info(backend, world): + world.windows = [window_info(7, pid=501)] + assert backend.window_process_id(7) == 501 + + +def test_a_window_that_is_gone_has_no_owner(backend, world): + assert backend.window_process_id(7) == 0 + + +# --- matching a Quartz window to an accessibility element --------------------- + +def _mac_with_window(number=7, *, pid=501, name="Editor", + bounds=(10, 20, 300, 400), ax_windows=None, + running=True): + world = World(windows=[window_info(number, name=name, pid=pid, + bounds=bounds)], + running_pids={pid} if running else set()) + world.ax_windows = {pid: list(ax_windows or [])} + return world + + +def _ax_window(origin=None, title=None): + attributes = {} + if origin is not None: + attributes["AXPosition"] = ("point", objc_stub.AXPoint(*origin)) + if title is not None: + attributes["AXTitle"] = title + return AXElement(**attributes) + + +def test_the_element_is_matched_by_its_origin(monkeypatch, on_darwin): + wanted = _ax_window(origin=(10, 20)) + world = _mac_with_window(ax_windows=[_ax_window(origin=(99, 99)), wanted]) + backend = _build(monkeypatch, world) + assert backend._ax_window(7) is wanted + + +def test_the_title_is_only_the_fallback(monkeypatch, on_darwin): + # Two windows cannot share an origin at one moment; they share titles all + # the time, so the frame is tried first and the title only after. + by_origin = _ax_window(origin=(10, 20), title="Other") + by_title = _ax_window(origin=(0, 0), title="Editor") + world = _mac_with_window(ax_windows=[by_title, by_origin]) + backend = _build(monkeypatch, world) + assert backend._ax_window(7) is by_origin + + +def test_a_title_match_is_used_when_no_origin_matches(monkeypatch, on_darwin): + by_title = _ax_window(origin=(0, 0), title="Editor") + world = _mac_with_window(ax_windows=[by_title]) + backend = _build(monkeypatch, world) + assert backend._ax_window(7) is by_title + + +def test_an_untitled_quartz_window_is_not_matched_by_an_empty_title( + monkeypatch, on_darwin): + # Matching "" against "" would pick an arbitrary window of the app. + candidate = _ax_window(origin=(0, 0), title="") + world = _mac_with_window(name="", ax_windows=[candidate]) + backend = _build(monkeypatch, world) + assert backend._ax_window(7) is None + + +def test_an_element_with_no_readable_position_is_skipped(monkeypatch, + on_darwin): + unreadable = _ax_window(title="Editor") + world = _mac_with_window(ax_windows=[unreadable]) + backend = _build(monkeypatch, world) + assert backend._ax_window(7) is unreadable, "found by title instead" + + +def test_a_window_quartz_does_not_know_has_no_element(backend): + assert backend._ax_window(7) is None + + +def test_a_window_with_no_owner_has_no_element(monkeypatch, on_darwin): + world = World(windows=[window_info(7, pid=0)]) + backend = _build(monkeypatch, world) + assert backend._ax_window(7) is None + + +def test_an_application_that_refuses_to_list_windows_yields_nothing( + monkeypatch, on_darwin): + world = _mac_with_window(ax_windows=[_ax_window(origin=(10, 20))]) + world.ax_list_error = AX_FAILURE + backend = _build(monkeypatch, world) + assert backend._ax_window(7) is None + + +@pytest.mark.parametrize("value,expected", [ + (None, (-1, -1)), + (("size", objc_stub.AXSize(1, 2)), (-1, -1)), + (("point", objc_stub.AXPoint(3, 4)), (3, 4)), +]) +def test_an_ax_point_reads_as_a_pair_or_as_nothing(monkeypatch, on_darwin, + world, value, expected): + objc_stub.install(monkeypatch, world) + assert _point(value) == expected + + +# --- minimised state ---------------------------------------------------------- + +def test_a_window_the_element_says_is_minimised_is_minimised(monkeypatch, + on_darwin): + element = _ax_window(origin=(10, 20)) + element.attributes["AXMinimized"] = True + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + assert backend.is_minimized(7) is True + + +def test_a_window_the_element_says_is_not_minimised_is_not(monkeypatch, + on_darwin): + element = _ax_window(origin=(10, 20)) + element.attributes["AXMinimized"] = False + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + assert backend.is_minimized(7) is False + + +def test_a_window_that_is_not_on_screen_at_all_is_minimised(backend): + # A minimised window is absent from the on-screen list, so failing to + # find it is itself the answer rather than an error. + assert backend.is_minimized(7) is True + + +def test_a_window_on_screen_with_no_element_is_not_minimised(monkeypatch, + on_darwin): + world = _mac_with_window(ax_windows=[]) + backend = _build(monkeypatch, world) + assert backend.is_minimized(7) is False + + +# --- raising ------------------------------------------------------------------ + +def test_raising_activates_the_app_and_then_the_window(monkeypatch, + on_darwin): + element = _ax_window(origin=(10, 20)) + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + backend.set_foreground(7) + assert world.activated == [(501, 2)], "ignoring other apps" + assert element.actions == ["AXRaise"], ( + "activating brings the app's front window forward, not this one" + ) + + +def test_raising_a_window_with_no_owner_is_refused(monkeypatch, on_darwin): + world = World(windows=[window_info(7, pid=0)]) + backend = _build(monkeypatch, world) + with pytest.raises(AutoControlUnsupportedOperationException, + match="set_foreground"): + backend.set_foreground(7) + + +def test_raising_still_raises_when_the_app_object_is_gone(monkeypatch, + on_darwin): + # The application can quit between the pid lookup and the activation, + # which leaves nothing to activate but still an element to raise. + element = _ax_window(origin=(10, 20)) + world = _mac_with_window(ax_windows=[element], running=False) + backend = _build(monkeypatch, world) + backend.set_foreground(7) + assert world.activated == [] + assert element.actions == ["AXRaise"] + + +def test_raising_a_window_the_accessibility_api_cannot_find_still_activates( + monkeypatch, on_darwin): + # Without Accessibility there is no element, but the application can + # still be brought forward -- which is most of what the user asked for. + world = _mac_with_window(ax_windows=[]) + backend = _build(monkeypatch, world) + backend.set_foreground(7) + assert world.activated == [(501, 2)] + + +# --- restoring and show-state codes ------------------------------------------- + +def test_restoring_clears_the_minimised_attribute(monkeypatch, on_darwin): + element = _ax_window(origin=(10, 20)) + element.attributes["AXMinimized"] = True + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + backend.restore(7) + assert element.attributes["AXMinimized"] is False + + +def test_restoring_a_window_with_no_element_says_why(monkeypatch, on_darwin): + # This is the TCC case: without Accessibility there is no element, and a + # silent no-op would look like a window that refused to restore. + world = _mac_with_window(ax_windows=[]) + backend = _build(monkeypatch, world) + with pytest.raises(AutoControlUnsupportedOperationException, + match="Accessibility"): + backend.restore(7) + + +@pytest.mark.parametrize("code", [macos.SW_SHOWNORMAL, macos.SW_RESTORE]) +def test_the_normal_and_restore_codes_both_restore(monkeypatch, on_darwin, + code): + element = _ax_window(origin=(10, 20)) + element.attributes["AXMinimized"] = True + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + backend.show(7, code) + assert element.attributes["AXMinimized"] is False + + +@pytest.mark.parametrize("code", [macos.SW_MINIMIZE, macos.SW_SHOWMINIMIZED]) +def test_both_minimise_codes_minimise(monkeypatch, on_darwin, code): + element = _ax_window(origin=(10, 20)) + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + backend.show(7, code) + assert element.attributes["AXMinimized"] is True + + +@pytest.mark.parametrize("code", [0, 3, 8]) +def test_a_show_code_with_no_macos_meaning_is_refused(backend, code): + # macOS has no hide-this-window and no maximise-this-window; zoom is not + # maximise, and the difference matters to the caller. + with pytest.raises(AutoControlUnsupportedOperationException, + match="show"): + backend.show(7, code) + + +# --- closing and minimising --------------------------------------------------- + +def test_closing_presses_the_windows_own_close_button(monkeypatch, on_darwin): + button = AXElement() + element = _ax_window(origin=(10, 20)) + element.attributes["AXCloseButton"] = button + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + assert backend.close(7) is True + assert button.actions == ["AXPress"] + + +def test_a_close_the_accessibility_api_refuses_reports_failure(monkeypatch, + on_darwin): + # AX answers with an error code where zero is success, so this is the + # test that would fail if that were read the other way round. + button = AXElement() + button.action_error = AX_FAILURE + element = _ax_window(origin=(10, 20)) + element.attributes["AXCloseButton"] = button + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + assert backend.close(7) is False + + +def test_a_window_with_no_close_button_cannot_be_closed(monkeypatch, + on_darwin): + world = _mac_with_window(ax_windows=[_ax_window(origin=(10, 20))]) + backend = _build(monkeypatch, world) + assert backend.close(7) is False + + +def test_closing_a_window_with_no_element_reports_failure(backend): + assert backend.close(7) is False + + +def test_minimising_sets_the_attribute(monkeypatch, on_darwin): + element = _ax_window(origin=(10, 20)) + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + assert backend.minimize(7) is True + assert element.attributes["AXMinimized"] is True + + +def test_a_minimise_the_accessibility_api_refuses_reports_failure(monkeypatch, + on_darwin): + element = _ax_window(origin=(10, 20)) + element.set_error = AX_FAILURE + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + assert backend.minimize(7) is False + + +def test_minimising_a_window_with_no_element_reports_failure(backend): + assert backend.minimize(7) is False + + +# --- moving ------------------------------------------------------------------- + +def test_moving_sets_position_and_size_as_ax_values(monkeypatch, on_darwin): + element = _ax_window(origin=(10, 20)) + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + assert backend.move(7, 100, 200, 640, 480) is True + written = dict(element.assignments) + kind, point = written["AXPosition"] + assert (kind, point.x, point.y) == ("point", 100.0, 200.0) + kind, size = written["AXSize"] + assert (kind, size.width, size.height) == ("size", 640.0, 480.0) + + +def test_a_move_the_accessibility_api_refuses_reports_failure(monkeypatch, + on_darwin): + element = _ax_window(origin=(10, 20)) + element.set_error = AX_FAILURE + world = _mac_with_window(ax_windows=[element]) + backend = _build(monkeypatch, world) + assert backend.move(7, 0, 0, 10, 10) is False + + +def test_moving_a_window_with_no_element_reports_failure(backend): + assert backend.move(7, 0, 0, 10, 10) is False + + +# --- what macOS deliberately will not do -------------------------------------- + +@pytest.mark.parametrize("call", [ + lambda b: b.post_key(7, 38), + lambda b: b.post_click(7, "left", 0, 0), +]) +def test_input_to_an_unfocused_window_is_refused_not_faked(backend, call): + # macOS has no PostMessage and no XSendEvent: an event goes wherever the + # focus is. A "success" that clicked somewhere else is worse than a no. + with pytest.raises(AutoControlUnsupportedOperationException): + call(backend) diff --git a/test/unit_test/headless/test_window_backend_selection.py b/test/unit_test/headless/test_window_backend_selection.py new file mode 100644 index 00000000..dc18cb7d --- /dev/null +++ b/test/unit_test/headless/test_window_backend_selection.py @@ -0,0 +1,299 @@ +"""Which window backend a platform gets, and what the others refuse. + +Window management was Windows-only for the project's whole life -- the facade +branched on `sys.platform` and raised `NotImplementedError` everywhere else, +leaving 23 `AC_*` commands dead on macOS and Linux. This package is the seam +that replaced that branch, and the seam itself was covered by nothing: the +selector only ever ran its own platform's arm, and every square skipped the +other two. + +Three things here are worth stating in a test: + +* **A platform without a working backend gets a refusal that says why.** Not + an ImportError at start-up and not a backend that answers falsely -- a null + backend carrying the reason, so `AC_list_window` on a Wayland session says + "Wayland does not expose other windows to a client" rather than failing + with a stack trace about Xlib. +* **Selection is cached, because probing costs a connection.** It opens an X + display or runs a Quartz query, and the answer cannot change inside a + process. +* **"Cannot" and "did not work this time" are different answers.** The base + class raises for the first and every backend returns a falsy value for the + second; a caller that cannot tell them apart retries forever. + +Both stubbed platforms are driven from every square, so this file tests all +three arms wherever it runs -- which is the point, since the coverage floor +is the lowest square. +""" +from __future__ import annotations + +import sys +import types + +import pytest + +from headless import _pyobjc_stub as objc_stub +from headless import _xlib_stub +from je_auto_control.utils.exception.exceptions import ( + AutoControlUnsupportedOperationException, +) +from je_auto_control.wrapper import window_backends +from je_auto_control.wrapper.window_backends import ( + NullWindowBackend, WindowManageBackend, get_backend, reset_backend_cache, +) +from je_auto_control.wrapper.window_backends.windows_backend import ( + WindowsWindowBackend, +) + + +@pytest.fixture(autouse=True) +def clean_cache(): + """The selector caches; a test that leaves one behind poisons the next.""" + reset_backend_cache() + yield + reset_backend_cache() + + +@pytest.fixture +def on_platform(monkeypatch): + def _set(name: str): + monkeypatch.setattr(window_backends.sys, "platform", name) + return _set + + +# --- selection ---------------------------------------------------------------- + +@pytest.mark.parametrize("platform", ["win32", "cygwin", "msys"]) +def test_every_windows_spelling_selects_the_win32_backend(on_platform, + platform): + on_platform(platform) + assert isinstance(get_backend(), WindowsWindowBackend) + + +def test_a_mac_with_pyobjc_selects_the_quartz_backend(on_platform, monkeypatch): + on_platform("darwin") + objc_stub.install(monkeypatch, objc_stub.World()) + assert get_backend().name == "macos-quartz-ax" + + +def test_a_mac_without_pyobjc_gets_a_refusal_naming_it(on_platform, + monkeypatch): + on_platform("darwin") + objc_stub.install_missing(monkeypatch, "Quartz") + backend = get_backend() + assert isinstance(backend, NullWindowBackend) + assert "pyobjc" in backend.reason + + +@pytest.mark.parametrize("platform", ["linux", "linux2"]) +def test_a_linux_session_with_x_selects_the_ewmh_backend(on_platform, + monkeypatch, + platform): + on_platform(platform) + _xlib_stub.install(monkeypatch) + assert get_backend().name == "x11-ewmh" + + +def test_a_session_with_no_x_display_gets_a_refusal_naming_wayland( + on_platform, monkeypatch): + # Wayland deliberately does not let a client enumerate or move other + # applications' windows. That is a protocol decision, so the message has + # to point at XWayland rather than read as a missing dependency. + on_platform("linux") + _xlib_stub.install_failing(monkeypatch, RuntimeError("no DISPLAY")) + backend = get_backend() + assert isinstance(backend, NullWindowBackend) + assert "Wayland" in backend.reason + assert "XWayland" in backend.reason + + +def test_an_unknown_platform_gets_a_refusal_naming_it(on_platform): + on_platform("sunos5") + backend = get_backend() + assert isinstance(backend, NullWindowBackend) + assert "sunos5" in backend.reason + + +def test_the_choice_is_made_once_and_cached(on_platform): + # Probing opens an X connection or runs a Quartz query; the answer cannot + # change inside a process, so paying for it twice is pure cost. + on_platform("win32") + assert get_backend() is get_backend() + + +def test_resetting_the_cache_re_detects(on_platform): + on_platform("win32") + first = get_backend() + reset_backend_cache() + assert get_backend() is not first + + +# --- the null backend --------------------------------------------------------- + +def test_the_null_backend_lists_nothing_rather_than_refusing(): + # "There are no windows I can see" is a truthful answer a caller can + # iterate; raising here would make every listing a special case. + assert NullWindowBackend().list_windows() == [] + + +def test_the_null_backend_carries_its_reason_in_its_name(): + backend = NullWindowBackend("no X display") + assert backend.reason == "no X display" + assert "no X display" in backend.name + assert backend.available is False + + +def test_a_null_backend_with_no_reason_still_says_something(): + assert NullWindowBackend().reason + + +@pytest.mark.parametrize("call", [ + lambda b: b.foreground_window(), + lambda b: b.window_rect(1), + lambda b: b.window_process_id(1), + lambda b: b.is_minimized(1), + lambda b: b.set_foreground(1), + lambda b: b.restore(1), + lambda b: b.show(1, 9), + lambda b: b.close(1), + lambda b: b.minimize(1), + lambda b: b.move(1, 0, 0, 1, 1), + lambda b: b.post_key(1, 38), + lambda b: b.post_click(1, "left", 0, 0), +]) +def test_every_action_on_a_null_backend_refuses_loudly(call): + with pytest.raises(AutoControlUnsupportedOperationException): + call(NullWindowBackend("no X display")) + + +# --- the abstract base -------------------------------------------------------- + +def test_the_base_class_has_no_listing_of_its_own(): + # Every backend must answer this one; there is no sensible default, so + # the base leaves it abstract rather than returning an empty list. + with pytest.raises(NotImplementedError): + WindowManageBackend().list_windows() + + +def test_a_refusal_names_the_operation_and_the_backend(): + backend = WindowManageBackend() + with pytest.raises(AutoControlUnsupportedOperationException) as caught: + backend.restore(1) + assert "restore" in str(caught.value) + assert "abstract" in str(caught.value) + + +# --- the Windows backend's delegation ------------------------------------------ + +class _WinManager(types.ModuleType): + """The Win32 module, which stays the single home of the real calls.""" + + SW_RESTORE = 9 + + def __init__(self) -> None: + super().__init__("je_auto_control.windows.window.windows_window_manage") + self.calls = [] + + def _record(self, name, *args): + self.calls.append((name, args)) + return f"{name}-result" + + def get_all_window_hwnd(self): + return self._record("get_all_window_hwnd") + + def get_foreground_window(self): + return self._record("get_foreground_window") + + def get_window_rect(self, window_id): + return self._record("get_window_rect", window_id) + + def get_window_process_id(self, window_id): + return self._record("get_window_process_id", window_id) + + def is_window_minimized(self, window_id): + return self._record("is_window_minimized", window_id) + + def set_foreground_window(self, window_id): + return self._record("set_foreground_window", window_id) + + def show_window(self, window_id, cmd_show): + return self._record("show_window", window_id, cmd_show) + + def close_window(self, window_id): + return self._record("close_window", window_id) + + def minimize_window(self, window_id): + return self._record("minimize_window", window_id) + + def move_window(self, window_id, x, y, width, height): + return self._record("move_window", window_id, x, y, width, height) + + def post_key(self, window_id, keycode, character): + return self._record("post_key", window_id, keycode, character) + + def post_click(self, window_id, button, x, y): + return self._record("post_click", window_id, button, x, y) + + +@pytest.fixture +def win_manager(monkeypatch): + """Stand in for the Win32 module, on every platform including Windows. + + The backend reaches it with `from je_auto_control.windows.window import + windows_window_manage`, and `from package import name` resolves by + *attribute* on the package before it looks in `sys.modules` -- so putting + the double under its own dotted name is not enough on a machine where the + real one is importable. The package it is read off is replaced instead. + """ + module = _WinManager() + package = types.ModuleType("je_auto_control.windows.window") + package.windows_window_manage = module + monkeypatch.setitem(sys.modules, "je_auto_control.windows.window", package) + monkeypatch.setitem( + sys.modules, "je_auto_control.windows.window.windows_window_manage", + module) + return module + + +@pytest.mark.parametrize("call,expected", [ + (lambda b: b.list_windows(), ("get_all_window_hwnd", ())), + (lambda b: b.foreground_window(), ("get_foreground_window", ())), + (lambda b: b.window_rect(5), ("get_window_rect", (5,))), + (lambda b: b.window_process_id(5), ("get_window_process_id", (5,))), + (lambda b: b.is_minimized(5), ("is_window_minimized", (5,))), + (lambda b: b.set_foreground(5), ("set_foreground_window", (5,))), + (lambda b: b.show(5, 3), ("show_window", (5, 3))), + (lambda b: b.close(5), ("close_window", (5,))), + (lambda b: b.minimize(5), ("minimize_window", (5,))), + (lambda b: b.move(5, 1, 2, 3, 4), ("move_window", (5, 1, 2, 3, 4))), + (lambda b: b.post_key(5, 38, "a"), ("post_key", (5, 38, "a"))), + (lambda b: b.post_click(5, "left", 1, 2), ("post_click", + (5, "left", 1, 2))), +]) +def test_the_windows_backend_forwards_to_the_win32_module(win_manager, call, + expected): + call(WindowsWindowBackend()) + assert win_manager.calls == [expected] + + +def test_restoring_goes_through_the_win32_show_state_constant(win_manager): + # `restore` is deliberately not "show it however": it un-minimises + # without un-maximising, which is what SW_RESTORE means. + WindowsWindowBackend().restore(5) + assert win_manager.calls == [("show_window", (5, _WinManager.SW_RESTORE))] + + +def test_a_show_code_is_passed_through_as_an_int(win_manager): + WindowsWindowBackend().show(5, "3") + assert win_manager.calls == [("show_window", (5, 3))] + + +@pytest.mark.parametrize("platform,available", [ + ("win32", True), ("cygwin", True), ("msys", True), + ("linux", False), ("darwin", False), +]) +def test_the_windows_backend_knows_where_it_can_run(monkeypatch, platform, + available): + from je_auto_control.wrapper.window_backends import windows_backend + monkeypatch.setattr(windows_backend.sys, "platform", platform) + assert WindowsWindowBackend().available is available diff --git a/test/unit_test/headless/test_window_backend_x11.py b/test/unit_test/headless/test_window_backend_x11.py new file mode 100644 index 00000000..3c4fa23a --- /dev/null +++ b/test/unit_test/headless/test_window_backend_x11.py @@ -0,0 +1,553 @@ +"""Window management through EWMH, from any square rather than only Linux. + +`x11_backend.py` was at 0% everywhere. `Progress.md` had it down as "only the +two Linux squares can run it" -- but every `Xlib` import in the module is +*inside* a method, so the module loads on Windows and macOS too and only the +calls need a display. A stub in `sys.modules` (`_xlib_stub.py`) reaches all of +it from every square, which is what makes it worth writing: the coverage floor +is the lowest square, so a test that runs on two of nine moves it least. + +What is actually being checked is protocol arithmetic, and it is the kind +that fails silently: + +* **EWMH requests go to the *root*, not to the window.** They are addressed + there with `SubstructureRedirect`, which is what routes them to the window + manager -- the only party allowed to act on them. Sending to the window + instead is a message nobody answers. +* **Stacking order is bottom-to-top.** `list_windows` reverses it so the + front-most window is first, which is what makes "the first title that + matches" the one the user meant. `_NET_CLIENT_LIST` carries no order at + all, so it is only a fallback and must *not* be reversed. +* **`move()` sizes the client while `window_rect()` reads the frame.** EWMH + and Win32 disagree about which rectangle they mean, so the decorations + come off the requested size; getting that wrong is a window that shrinks a + title bar's worth every time a script round-trips it. +* **Source indication 2 means "pager".** A window manager honours it without + the focus-stealing prevention it applies to source 1, so a raise sent as an + application is a raise that quietly does not happen. + +The stub's constants carry their real `X.h` values, and +`test_xlib_stub_values.py` holds them to the installed library wherever there +is one -- so the masks asserted below are the numbers that go on the wire. +""" +from __future__ import annotations + +import pytest + +from headless import _xlib_stub +from je_auto_control.utils.exception.exceptions import ( + AutoControlUnsupportedOperationException, +) +from je_auto_control.wrapper.window_backends import x11_backend as x11 +from je_auto_control.wrapper.window_backends.x11_backend import ( + X11WindowBackend, _as_text, +) + +_X = _xlib_stub.X_CONSTANTS +_REDIRECT = (_X["SubstructureRedirectMask"] | _X["SubstructureNotifyMask"]) + + +@pytest.fixture +def on_linux(monkeypatch): + """Answer `_is_linux()` the way an X session would.""" + monkeypatch.setattr(x11.sys, "platform", "linux") + + +@pytest.fixture +def display(monkeypatch, on_linux): + return _xlib_stub.install(monkeypatch) + + +@pytest.fixture +def backend(display): + instance = X11WindowBackend() + assert instance.available, "the stub display opens" + return instance + + +def _client_messages(display): + """Every EWMH request the backend addressed to the root window.""" + return [(event.fields["client_type"], event.fields["data"], mask) + for event, mask, _propagate in display.root.sent] + + +def _named_messages(display): + """The same, with atoms resolved back to the names they were interned as.""" + return [(display.atom_name(atom), data[1], mask) + for atom, data, mask in _client_messages(display)] + + +# --- availability ------------------------------------------------------------- + +def test_the_backend_is_unavailable_off_linux(monkeypatch): + monkeypatch.setattr(x11.sys, "platform", "win32") + _xlib_stub.install(monkeypatch) + backend = X11WindowBackend() + assert backend.available is False + assert backend._connection is None, "it did not even try to connect" + + +@pytest.mark.parametrize("platform", ["linux", "linux2"]) +def test_both_spellings_of_linux_are_linux(monkeypatch, platform): + monkeypatch.setattr(x11.sys, "platform", platform) + _xlib_stub.install(monkeypatch) + assert X11WindowBackend().available is True + + +def test_a_session_with_no_display_is_unavailable_rather_than_fatal( + monkeypatch, on_linux): + # No `DISPLAY` is the ordinary state of a Wayland session or an ssh + # login; the selector turns this into a null backend with a reason. + _xlib_stub.install_failing(monkeypatch, RuntimeError("no DISPLAY")) + assert X11WindowBackend().available is False + + +def test_the_backend_names_the_protocol_it_speaks(backend): + assert backend.name == "x11-ewmh" + + +# --- the connection and its atoms --------------------------------------------- + +def test_the_display_is_opened_once_and_kept(backend, display): + assert backend._display() is display + assert backend._display() is display + + +def test_an_atom_is_interned_once_per_connection(backend, display): + first = backend._atom("_NET_WM_STATE") + assert backend._atom("_NET_WM_STATE") == first + assert list(display.atoms) == ["_NET_WM_STATE"] + + +def test_a_missing_property_reads_as_none(backend, display): + assert backend._property(display.root, "_NET_CLIENT_LIST") is None + + +# --- listing ------------------------------------------------------------------ + +def test_listing_puts_the_front_most_window_first(backend, display): + # `_NET_CLIENT_LIST_STACKING` is bottom-to-top, so 30 is in front. + display.set_root_property("_NET_CLIENT_LIST_STACKING", [10, 20, 30]) + for window_id, title in ((10, "back"), (20, "middle"), (30, "front")): + _titled(display, window_id, title) + assert backend.list_windows() == [(30, "front"), (20, "middle"), + (10, "back")] + + +def test_listing_falls_back_to_the_unordered_list(backend, display): + # `_NET_CLIENT_LIST` has no stacking meaning, so reversing it would + # invent an order the window manager never stated. + display.set_root_property("_NET_CLIENT_LIST", [10, 20]) + _titled(display, 10, "first") + _titled(display, 20, "second") + assert backend.list_windows() == [(10, "first"), (20, "second")] + + +def test_a_desktop_with_no_windows_lists_nothing(backend, display): + assert backend.list_windows() == [] + + +def test_a_title_prefers_the_utf8_property(backend, display): + display.set_root_property("_NET_CLIENT_LIST_STACKING", [10]) + display.intern_atom("_NET_WM_NAME") + display.window(10, properties={"_NET_WM_NAME": b"Editor \xe2\x80\x94 file"}, + wm_name="legacy") + assert backend.list_windows() == [(10, "Editor — file")] + + +def test_a_title_falls_back_to_the_legacy_property(backend, display): + display.set_root_property("_NET_CLIENT_LIST_STACKING", [10]) + display.intern_atom("_NET_WM_NAME") + display.window(10, wm_name="Old Toolkit") + assert backend.list_windows() == [(10, "Old Toolkit")] + + +def test_a_window_that_vanishes_mid_walk_lists_with_an_empty_title( + backend, display): + # The list is a snapshot; by the time a title is read the window may be + # gone, and one closing window must not fail the whole enumeration. + display.set_root_property("_NET_CLIENT_LIST_STACKING", [10, 20]) + _titled(display, 10, "still here") + display.window(20).property_error = RuntimeError("BadWindow") + assert backend.list_windows() == [(20, ""), (10, "still here")] + + +def _titled(display, window_id: int, title: str): + display.intern_atom("_NET_WM_NAME") + return display.window(window_id, + properties={"_NET_WM_NAME": title.encode("utf-8")}) + + +# --- the focused window ------------------------------------------------------- + +def test_the_foreground_window_is_the_first_id_the_property_names(backend, + display): + display.set_root_property("_NET_ACTIVE_WINDOW", [42]) + assert backend.foreground_window() == 42 + + +@pytest.mark.parametrize("value", [None, []]) +def test_no_focused_window_reads_as_zero(backend, display, value): + if value is not None: + display.set_root_property("_NET_ACTIVE_WINDOW", value) + assert backend.foreground_window() == 0 + + +# --- rectangles --------------------------------------------------------------- + +def test_the_rectangle_is_the_frame_not_the_client(backend, display): + # A reparenting window manager makes the client a grandchild of the root + # inside a frame that carries the border and title bar; the frame is what + # the user sees and drags, and what Win32's GetWindowRect would report. + display.window(50, parent_id=99, geometry=(0, 0, 10, 10)) + display.window(99, parent_id=1, geometry=(100, 200, 640, 480)) + assert backend.window_rect(50) == (100, 200, 740, 680) + + +def test_an_undecorated_window_is_its_own_frame(backend, display): + display.window(50, parent_id=1, geometry=(5, 6, 70, 80)) + assert backend.window_rect(50) == (5, 6, 75, 86) + + +def test_a_window_whose_parent_is_gone_is_its_own_frame(backend, display): + display.window(50, parent_id=None, geometry=(1, 2, 3, 4)) + assert backend.window_rect(50) == (1, 2, 4, 6) + + +def test_a_frame_walk_cannot_loop_forever(backend, display): + # A cycle in the tree would hang the caller; the walk is bounded instead. + display.window(50, parent_id=51, geometry=(0, 0, 1, 1)) + display.window(51, parent_id=50, geometry=(7, 7, 2, 2)) + assert backend.window_rect(50) is not None + + +def test_a_rectangle_for_a_window_that_is_gone_is_none(backend, display): + display.window(50, parent_id=1) + display.window(50).geometry_error = RuntimeError("BadWindow") + assert backend.window_rect(50) is None + + +# --- ownership and state ------------------------------------------------------ + +def test_the_owning_pid_comes_from_the_ewmh_property(backend, display): + display.intern_atom("_NET_WM_PID") + display.window(50, properties={"_NET_WM_PID": [4321]}) + assert backend.window_process_id(50) == 4321 + + +def test_a_window_with_no_pid_property_reports_zero(backend, display): + assert backend.window_process_id(50) == 0 + + +def test_a_pid_read_that_fails_reports_zero(backend, display): + display.window(50).property_error = RuntimeError("BadWindow") + assert backend.window_process_id(50) == 0 + + +def test_a_hidden_window_is_minimised(backend, display): + hidden = backend._atom("_NET_WM_STATE_HIDDEN") + display.intern_atom("_NET_WM_STATE") + display.window(50, properties={"_NET_WM_STATE": [hidden]}) + assert backend.is_minimized(50) is True + + +def test_a_window_in_another_state_is_not_minimised(backend, display): + display.intern_atom("_NET_WM_STATE") + other = display.intern_atom("_NET_WM_STATE_MAXIMIZED_VERT") + display.window(50, properties={"_NET_WM_STATE": [other]}) + assert backend.is_minimized(50) is False + + +def test_a_window_with_no_state_property_is_not_minimised(backend, display): + assert backend.is_minimized(50) is False + + +def test_a_state_read_that_fails_is_not_minimised(backend, display): + display.window(50).property_error = RuntimeError("BadWindow") + assert backend.is_minimized(50) is False + + +# --- raising and restoring ---------------------------------------------------- + +def test_raising_a_window_addresses_the_root_as_a_pager(backend, display): + backend.set_foreground(50) + [(name, source, mask)] = _named_messages(display) + assert name == "_NET_ACTIVE_WINDOW" + assert source[0] == 2, "source 2 is 'pager'; source 1 gets refused" + assert mask == _REDIRECT + assert display.root.sent[0][0].fields["window"].id == 50 + + +def test_the_request_carries_the_window_it_is_about(backend, display): + backend.set_foreground(50) + event = display.root.sent[0][0] + assert event.fields["window"].id == 50, "addressed to root, about 50" + + +def test_a_client_message_is_padded_to_five_values(backend, display): + backend.set_foreground(50) + _format, data = display.root.sent[0][0].fields["data"] + assert _format == 32 + assert len(data) == 5 + + +def test_restoring_maps_the_window_before_raising_it(backend, display): + backend.restore(50) + assert display.window(50).mapped is True + assert [name for name, _, _ in _named_messages(display)] == [ + "_NET_ACTIVE_WINDOW", + ] + + +# --- show-state codes --------------------------------------------------------- + +def test_hiding_unmaps_the_window(backend, display): + backend.show(50, x11.SW_HIDE) + assert display.window(50).mapped is False + + +@pytest.mark.parametrize("code", [x11.SW_SHOWNORMAL, x11.SW_RESTORE]) +def test_the_normal_and_restore_codes_both_restore(backend, display, code): + backend.show(50, code) + assert display.window(50).mapped is True + + +@pytest.mark.parametrize("code", [x11.SW_MINIMIZE, x11.SW_SHOWMINIMIZED]) +def test_both_minimise_codes_send_the_icccm_request(backend, display, code): + backend.show(50, code) + assert [name for name, _, _ in _named_messages(display)] == [ + "WM_CHANGE_STATE", + ] + + +def test_maximising_names_both_axes(backend, display): + # `_NET_WM_STATE_ADD` is 1, and a request that names one axis grows the + # window one way only. + backend.show(50, x11.SW_MAXIMIZE) + [(name, data, _mask)] = _named_messages(display) + assert name == "_NET_WM_STATE" + assert data[0] == 1, "_NET_WM_STATE_ADD" + assert {data[1], data[2]} == { + backend._atom("_NET_WM_STATE_MAXIMIZED_HORZ"), + backend._atom("_NET_WM_STATE_MAXIMIZED_VERT"), + } + + +def test_a_show_code_with_no_x11_meaning_is_refused(backend, display): + # SW_SHOWNA, SW_FORCEMINIMIZE and friends have no X11 equivalent, and + # guessing at one would move a window the caller did not ask to move. + with pytest.raises(AutoControlUnsupportedOperationException, + match="show"): + backend.show(50, 8) + assert display.root.sent == [] + + +# --- closing and minimising --------------------------------------------------- + +def test_closing_sends_the_ewmh_close_request(backend, display): + assert backend.close(50) is True + [(name, data, _mask)] = _named_messages(display) + assert name == "_NET_CLOSE_WINDOW" + assert data[1] == 2, "source 2, as with every other request here" + + +def test_a_close_that_the_connection_refuses_reports_failure(backend, display): + display.root.send_error = RuntimeError("connection lost") + assert backend.close(50) is False + + +def test_minimising_uses_the_icccm_request(backend, display): + # There is no `_NET_` message for iconify; `WM_CHANGE_STATE` with the + # ICCCM iconic state is what every window manager implements. + assert backend.minimize(50) is True + [(name, data, _mask)] = _named_messages(display) + assert name == "WM_CHANGE_STATE" + assert data[0] == 3, "IconicState" + + +def test_a_minimise_that_the_connection_refuses_reports_failure(backend, + display): + display.root.send_error = RuntimeError("connection lost") + assert backend.minimize(50) is False + + +# --- moving ------------------------------------------------------------------- + +def test_moving_takes_the_decorations_off_the_requested_size(backend, display): + # EWMH sizes the client; Win32's MoveWindow sizes the frame. Subtracting + # the extents is what makes move() round-trip with window_rect(). + display.intern_atom("_NET_FRAME_EXTENTS") + display.window(50, properties={"_NET_FRAME_EXTENTS": [4, 4, 30, 2]}) + assert backend.move(50, 100, 200, 640, 480) is True + [(name, data, _mask)] = _named_messages(display) + assert name == "_NET_MOVERESIZE_WINDOW" + assert data[1:] == [100, 200, 640 - 8, 480 - 32] + + +def test_moving_a_window_with_no_extents_uses_the_size_as_given(backend, + display): + backend.move(50, 10, 20, 300, 400) + [(_name, data, _mask)] = _named_messages(display) + assert data[1:] == [10, 20, 300, 400] + + +def test_an_unreadable_extents_property_is_treated_as_no_decorations(backend, + display): + # Not knowing the border thickness is not a reason to refuse the move: + # the window ends up a border's worth larger, which is recoverable, while + # refusing leaves the caller with a window that never moved at all. + display.window(50).property_error = RuntimeError("BadWindow") + assert backend.move(50, 0, 0, 100, 100) is True + [(_name, data, _mask)] = _named_messages(display) + assert data[3:] == [100, 100] + + +def test_a_short_extents_property_is_treated_as_no_decorations(backend, + display): + display.intern_atom("_NET_FRAME_EXTENTS") + display.window(50, properties={"_NET_FRAME_EXTENTS": [4, 4]}) + backend.move(50, 0, 0, 100, 100) + [(_name, data, _mask)] = _named_messages(display) + assert data[3:] == [100, 100] + + +def test_a_window_smaller_than_its_own_decorations_still_gets_a_size(backend, + display): + # A client size of zero or less is not a request any window manager can + # honour, so it floors at one pixel rather than going negative. + display.intern_atom("_NET_FRAME_EXTENTS") + display.window(50, properties={"_NET_FRAME_EXTENTS": [40, 40, 40, 40]}) + backend.move(50, 0, 0, 10, 10) + [(_name, data, _mask)] = _named_messages(display) + assert data[3:] == [1, 1] + + +def test_the_move_flags_name_all_four_fields_and_the_pager_source(backend, + display): + backend.move(50, 0, 0, 100, 100) + [(_name, data, _mask)] = _named_messages(display) + flags = data[0] + for bit in (8, 9, 10, 11): + assert flags & (1 << bit), f"bit {bit} says one of x/y/w/h is supplied" + assert (flags >> 12) & 0b11 == 2, "source indication 2 is 'pager'" + + +def test_a_move_the_connection_refuses_reports_failure(backend, display): + display.root.send_error = RuntimeError("connection lost") + assert backend.move(50, 0, 0, 100, 100) is False + + +# --- posting input to an unfocused window ------------------------------------- + +def test_posting_a_key_sends_a_press_and_a_release(backend, display): + assert backend.post_key(50, keycode=38, character="a") is True + sent = display.window(50).sent + assert [event.kind for event, _mask, _p in sent] == ["KeyPress", + "KeyRelease"] + assert [mask for _e, mask, _p in sent] == [_X["KeyPressMask"], + _X["KeyReleaseMask"]] + + +def test_a_posted_key_is_addressed_by_keycode_not_by_character(backend, + display): + # X11 has no "type this character" event; the character is Win32's idea + # and is dropped rather than guessed at. + backend.post_key(50, keycode=38, character="Z") + press = display.window(50).sent[0][0] + assert press.fields["detail"] == 38 + assert "character" not in press.fields + + +def test_a_posted_key_is_marked_as_propagating(backend, display): + # Synthetic events are what XSendEvent delivers; propagate lets the + # toolkit that does accept them see it on an ancestor. + backend.post_key(50, keycode=38) + assert display.window(50).sent[0][2] is True + + +def test_a_post_key_that_the_connection_refuses_reports_failure(backend, + display): + display.window(50).send_error = RuntimeError("connection lost") + assert backend.post_key(50, keycode=38) is False + + +@pytest.mark.parametrize("button,number", [ + ("left", 1), ("middle", 2), ("right", 3), + ("LEFT", 1), ("mouse_right", 3), +]) +def test_posting_a_click_maps_the_button_name_to_x11_numbering( + backend, display, button, number): + assert backend.post_click(50, button, x=7, y=9) is True + press = display.window(50).sent[0][0] + assert press.fields["detail"] == number + + +def test_a_posted_click_carries_window_relative_coordinates(backend, display): + backend.post_click(50, "left", x=7, y=9) + press = display.window(50).sent[0][0] + assert (press.fields["event_x"], press.fields["event_y"]) == (7, 9) + assert (press.fields["root_x"], press.fields["root_y"]) == (0, 0) + + +def test_posting_a_click_sends_a_press_and_a_release(backend, display): + backend.post_click(50, "left", x=0, y=0) + sent = display.window(50).sent + assert [event.kind for event, _mask, _p in sent] == ["ButtonPress", + "ButtonRelease"] + assert [mask for _e, mask, _p in sent] == [_X["ButtonPressMask"], + _X["ButtonReleaseMask"]] + + +def test_an_unknown_button_is_refused_rather_than_guessed(backend, display): + with pytest.raises(AutoControlUnsupportedOperationException, + match="post_click"): + backend.post_click(50, "thumb", x=0, y=0) + assert display.window(50).sent == [] + + +def test_a_post_click_that_the_connection_refuses_reports_failure(backend, + display): + display.window(50).send_error = RuntimeError("connection lost") + assert backend.post_click(50, "left", x=0, y=0) is False + + +# --- every request is flushed ------------------------------------------------- + +@pytest.mark.parametrize("action", [ + lambda b: b.set_foreground(50), + lambda b: b.restore(50), + lambda b: b.show(50, x11.SW_HIDE), + lambda b: b.close(50), + lambda b: b.minimize(50), + lambda b: b.move(50, 0, 0, 10, 10), + lambda b: b.post_key(50, 38), + lambda b: b.post_click(50, "left", 0, 0), +]) +def test_every_action_flushes_the_connection(backend, display, action): + # X buffers requests; one that is never flushed is one the window manager + # does not see until something else happens to flush it. + before = display.flushes + action(backend) + assert display.flushes > before + + +# --- property decoding -------------------------------------------------------- + +@pytest.mark.parametrize("value,expected", [ + (None, ""), + (b"plain", "plain"), + ("already text", "already text"), + (b"caf\xc3\xa9", "café"), + ([104, 105], "hi"), + ([104, 105, 0, 0], "hi"), + (b"\xff\xfe", "��"), +]) +def test_a_property_value_decodes_to_text(value, expected): + # X properties arrive as bytes, as str, or as an array of byte-sized + # ints depending on the property and the Xlib version, and a title that + # is not valid UTF-8 is a title to show badly, not an exception. + assert _as_text(value) == expected + + +def test_an_undecodable_property_falls_back_to_its_repr(): + assert _as_text(object) == str(object) diff --git a/test/unit_test/headless/test_xlib_stub_values.py b/test/unit_test/headless/test_xlib_stub_values.py new file mode 100644 index 00000000..65c8a47b --- /dev/null +++ b/test/unit_test/headless/test_xlib_stub_values.py @@ -0,0 +1,58 @@ +"""Hold the Xlib stub to the library it stands in for. + +`_xlib_stub.py` lets `test_window_backend_x11.py` and its accessibility +counterpart run on all nine CI squares rather than the two with an X server, +by shadowing `Xlib` in `sys.modules`. That buys portability at one cost: a +stub can agree with a test about a number that is wrong on the wire. + +This file is what stops that. `python-Xlib` is a real dependency on Linux and +BSD, so on both Linux squares the genuine module is installed and every +constant the stub declares is compared against it. Elsewhere there is nothing +to compare with and these skip -- which is the right shape for the check, +because a wrong constant is a Linux-only bug and Linux is where it is caught. + +If a test here fails, the stub is wrong, not the library: fix +`_xlib_stub.py`'s value and re-read whichever assertion in +`test_window_backend_x11.py` depended on it. +""" +from __future__ import annotations + +import pytest + +from headless import _xlib_stub + +# python-Xlib ships no wheel-less platforms: it is pinned in pyproject for +# Linux, FreeBSD, OpenBSD and NetBSD, and absent everywhere else. +xlib = pytest.importorskip( + "Xlib", reason="python-Xlib is a Linux/BSD-only dependency", +) + + +@pytest.mark.parametrize("name", sorted(_xlib_stub.X_CONSTANTS)) +def test_every_x_constant_matches_the_installed_library(name): + from Xlib import X + assert _xlib_stub.X_CONSTANTS[name] == getattr(X, name), ( + f"the stub's X.{name} is not what python-Xlib says it is" + ) + + +@pytest.mark.parametrize("name", sorted(_xlib_stub.XATOM_CONSTANTS)) +def test_every_predefined_atom_matches_the_installed_library(name): + from Xlib import Xatom + assert _xlib_stub.XATOM_CONSTANTS[name] == getattr(Xatom, name), ( + f"the stub's Xatom.{name} is not what python-Xlib says it is" + ) + + +def test_the_event_classes_the_stub_fakes_all_exist(): + # A stub that answers for an event class the library does not have would + # let a backend build a message nothing can send. + from Xlib.protocol import event + for name in ("ClientMessage", "KeyPress", "KeyRelease", + "ButtonPress", "ButtonRelease"): + assert hasattr(event, name), f"Xlib.protocol.event.{name}" + + +def test_the_display_entry_point_the_stub_fakes_exists(): + from Xlib import display + assert hasattr(display, "Display") From 090c5c0e96e7e6f2b60afeb40c89e65a76ad40ad Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 07:46:31 +0800 Subject: [PATCH 12/20] Test the accessibility backends, and fix 37 guards that caught nothing Same fact as the window backends: every comtypes, pyobjc and D-Bus import in utils/accessibility is inside a method, so doubles in sys.modules drive all three platform backends from every square rather than from the one they were written for. windows_backend goes 17.72% -> 99.42%, linux_backend 42.41% -> 100%, macos_backend 0% -> 100%, windows_query 24.75% -> 99.01%, windows_state 18.75% -> 100%, base and the selector to 100%. Whole project 80.22% -> 81.89% on one machine. Two of the doubles earn their keep by refusing things. The UIA one models the indirection every control pattern goes through -- a pattern id, then an interface name queried off the generated module -- and rejects a mismatched pair, because those are two independent constants with nothing checking them and the failure is a None that reads exactly like "no such control". It also raises AttributeError for any Current* property a test did not supply, so `str(pattern.CurrentValue or "")` cannot pass against the repr of a function. The AT-SPI one replaces SessionBus rather than _AtspiConnection. The existing Linux test replaces the connection, which is right for the walk and leaves the whole D-Bus call layer beneath it unexecuted -- and that is where the protocol is: the accessibility bus is not the session bus, an accessible is a (sender, path) pair, and the state bitfield arrives as two 32-bit words, so reading only the first drops every state above bit 31. Writing them turned up a real one, the same shape as the WebRTC viewer's last commit: a catch tuple missing the type that actually gets raised. comtypes reports a provider failure as COMError, which derives straight from Exception. windows_query._uia_errors exists to say so and its docstring names the case, but only the two tree-walking guards in windows_backend used it -- the other 37, every control pattern in the file, named (OSError, AttributeError, ...) and contained none of them. A window closing between the search that found an element and the call that read it raised COMError out of the ac_* tool, past the executor's AutoControlException boundary. All 37 now share one tuple, which only widens what is caught. That fix costs 5 lines in a file already on the over-750 list, so Progress.md carries the reason rather than a split: the right seam there is the UIA COM lifetime management, which has nothing to do with this. --- CHANGELOG.md | 14 + Progress.md | 68 +- WHATS_NEW.md | 65 ++ architecture_explore.md | 10 +- .../accessibility/backends/windows_backend.py | 81 +- test/unit_test/headless/_uia_doubles.py | 246 ++++++ .../test_accessibility_backend_selection.py | 216 ++++++ .../headless/test_accessibility_linux.py | 277 ++++++- .../headless/test_accessibility_linux_bus.py | 390 ++++++++++ .../headless/test_accessibility_macos.py | 314 ++++++++ .../test_accessibility_windows_content.py | 731 ++++++++++++++++++ .../test_accessibility_windows_patterns.py | 450 +++++++++++ .../test_accessibility_windows_query.py | 447 +++++++++++ .../test_accessibility_windows_uia.py | 527 +++++++++++++ 14 files changed, 3779 insertions(+), 57 deletions(-) create mode 100644 test/unit_test/headless/_uia_doubles.py create mode 100644 test/unit_test/headless/test_accessibility_backend_selection.py create mode 100644 test/unit_test/headless/test_accessibility_linux_bus.py create mode 100644 test/unit_test/headless/test_accessibility_macos.py create mode 100644 test/unit_test/headless/test_accessibility_windows_content.py create mode 100644 test/unit_test/headless/test_accessibility_windows_patterns.py create mode 100644 test/unit_test/headless/test_accessibility_windows_query.py create mode 100644 test/unit_test/headless/test_accessibility_windows_uia.py diff --git a/CHANGELOG.md b/CHANGELOG.md index c77bb4fd..27275e85 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -387,6 +387,20 @@ only when documented here with a migration path. ### Fixed +- **A window closing mid-call let a COM error escape every Windows + accessibility read.** `comtypes` reports a provider failure as `COMError`, + which derives straight from `Exception` — the reason + `windows_query._uia_errors()` exists — but only the two tree-walking guards + in `backends/windows_backend.py` used that tuple. The other 37, covering + every control pattern (`get_value`, `invoke`, `toggle`, `read_table`, the + text and grid reads, …), named `(OSError, AttributeError, …)` and therefore + contained none of them. An application that stopped responding, or a window + that closed between the search that found an element and the call that read + it, raised `COMError` out of the `ac_*` tool or `AC_*` command instead of + answering `None` / `False` / `[]`, and past the executor's + `AutoControlException` boundary. All 37 now use the same tuple. This only + widens what is caught: no call that used to succeed behaves differently. + - **The WebRTC viewer ended every clean disconnect with an unhandled task exception.** `WebRTCDesktopViewer._consume_video` caught `(OSError, RuntimeError)`, but aiortc signals the end of a track by raising diff --git a/Progress.md b/Progress.md index 59d93597..957c0d58 100644 --- a/Progress.md +++ b/Progress.md @@ -25,7 +25,7 @@ | --- | ---: | --- | | `utils/mcp_server/tools/_handlers.py` | 4,789 | 676 個 MCP 工具的處理函式本體。與 `_factories.py`（表）不同,這裡是邏輯,應該依主題拆成 `_handlers/` 套件（input／screen／window／file／agent…）。拆點清楚,純粹是量大。 | | `gui/remote_desktop/webrtc_panel.py` | 2,545 | 單一 Qt 面板,但已含連線、監視器選擇、頻寬自適應、麥克風、錄影五組互動狀態。應拆成 panel + 各控制器。 | -| `utils/accessibility/backends/windows_backend.py` | 915 | 已拆出 `windows_query.py`（170）與 `windows_state.py`（98）。剩下的是同一套 UIA COM 生命週期管理,再拆會把 `CoInitialize`／介面釋放的配對邏輯切散。 | +| `utils/accessibility/backends/windows_backend.py` | 923 | 已拆出 `windows_query.py`（170）與 `windows_state.py`（98）。剩下的是同一套 UIA COM 生命週期管理,再拆會把 `CoInitialize`／介面釋放的配對邏輯切散。**2026-08-24 從 918 長到 923**:見下面的說明。 | **本質豁免（依 `CLAUDE.md` 的「flat data tables」條款,不算既有豁免）**: `utils/mcp_server/tools/_factories.py`（8,972,MCP 工具註冊表）、 @@ -49,6 +49,17 @@ 行數沒有任何 CI 在把關（`quality.yml` 只跑 ruff 與 bandit,而 ruff 只管行寬), 所以這張表只會在有人手動實測時才會被發現對不上——上次就是。 +### 2026-08-24:`windows_backend.py` 從 918 長到 923，理由記在這裡 + +表上原本寫 915，2026-08-24 實測時工作樹已經是 **918**（表本身就過期了， +正是上一段講的那件事）。這次又 +5，是為了改掉一個真的錯：檔內 37 個 +`except (OSError, AttributeError, …)` 攔不到 comtypes 的 `COMError` +（細節見下面覆蓋率那一節與 [CHANGELOG.md](CHANGELOG.md)）。5 行是一個模組層 +常數加兩行註解——`CLAUDE.md` 允許「超標檔案再變長」的兩條路是**先拆**或 +**在這裡寫明為什麼不拆**，這是後者：拆這個檔的正確切點是 UIA COM 的生命週期 +管理，和這次的修正無關，綁在一起會讓一個三行的正確性修補變成大面積 diff。 +**新上限是 923**，規則不變。 + --- ## Windows arm64:裝得起來了，但少了影像與加密 @@ -367,7 +378,7 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl | 子系統 | 沒蓋到 / 總 statement | 覆蓋率 | | --- | ---: | ---: | | `utils/remote_desktop` | 2,297 / 6,622 | 65.3% | -| `utils/accessibility` | 824 / 1,397 | 41.0% | +| ~~`utils/accessibility`~~ | ~~824 / 1,397~~ | **2026-08-24 補完（backends 全數 99–100%）** | | `utils/executor` | 653 / 3,906 | 83.3% | | `utils/usb` | 573 / 2,137 | 73.2% | | `utils/mcp_server` | 573 / 4,618 | 87.6% | @@ -383,14 +394,14 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl | 檔案 | 每一格都沒蓋到 | 備註 | | --- | ---: | --- | | `utils/executor/action_executor.py` | 589 | 掃不到的那批 adapter：被呼叫者是 class（沒有回傳標注可讀）、或 adapter 伸手進兩個模組 | -| `utils/accessibility/backends/windows_backend.py` | 446 | UIA COM，要一套夠像的替身 | +| ~~`utils/accessibility/backends/windows_backend.py`~~ | ~~446~~ | **2026-08-24 補完（99.42%）**，順便修掉 37 個攔不到 `COMError` 的 except | | ~~`utils/remote_desktop/webrtc_viewer.py`~~ | ~~354~~ | **2026-08-24 補完（100%）** | | ~~`utils/remote_desktop/webrtc_host.py`~~ | ~~351~~ | **2026-08-24 補完（100%）** | | `utils/mcp_server/tools/_handlers.py` | 348 | 同 `action_executor.py` | | `utils/remote_desktop/signaling_server.py` | 155 | **CI 動不了**：要 `[signaling]` extra（fastapi／uvicorn），沒裝 | | ~~`utils/remote_desktop/multi_viewer.py`~~ | ~~146~~ | **2026-08-24 補完（100%）** | | ~~`wrapper/window_backends/x11_backend.py`~~ | ~~133~~ | **2026-08-24 補完（100%）**。「只有 Linux 那兩格跑得到」是錯的，見下 | -| `utils/accessibility/backends/linux_backend.py` | 132 | 同上，尚未補 | +| ~~`utils/accessibility/backends/linux_backend.py`~~ | ~~132~~ | **2026-08-24 補完（100%）** | **下一步的順序**： @@ -399,10 +410,8 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl 2. 掃不到的那批 adapter：目前卡在「被呼叫者是 class」。要嘛從 class 自己的方法標注 長出一個替身物件，要嘛承認那批不掃——**得先決定，因為前者會讓「跑起來了」和 「驗到了東西」分家**。 -3. ~~`wrapper/window_backends`~~ **2026-08-24 整包補完（100%）**，見下下節； - `utils/accessibility` 還沒補，做法照抄——AT-SPI 那一層要的是 `SessionBus` - 的替身，不是 `_AtspiConnection` 的（現有的 `test_accessibility_linux.py` - 換掉的是後者，所以整個 D-Bus 呼叫層還沒被跑過）。 +3. ~~`wrapper/window_backends` 與 `utils/accessibility`~~ + **2026-08-24 兩個都整包補完**，見下下節。 `utils/office`（77）與 `signaling_server`（155）**不要碰**：`quality.yml` 沒裝 `[office]`／`[signaling]`，補的測試會整批 skip，對地板一個點都不動。要補之前 @@ -490,10 +499,45 @@ python -c "import je_auto_control.wrapper.window_backends.x11_backend" # 在 W Windows 上完全沒效果——真的 Win32 模組會被呼叫。要換掉的是**那個套件**。 本機（Windows／3.14）全專案 79.40% → **80.22%**，184 個新測試。 -地板一樣要等九宮格。**下一個照抄的對象是 `utils/accessibility`**：那裡的 -`windows_backend.py`（446 行沒蓋到）用 comtypes，`macos_backend.py`（73，0%） -用 pyobjc，`linux_backend.py` 的 D-Bus 呼叫層用 `SessionBus`——三個都是 -同樣的 lazy import，同樣塞得進 `sys.modules`。 +地板一樣要等九宮格。 + +#### 2026-08-24：`utils/accessibility` 照抄同一招，並抓到 37 個攔不到的 except + +同一個事實在這裡也成立——三個平台後端的 comtypes／pyobjc／D-Bus import +全在函式內，所以塞 `sys.modules` 就能在九格都跑： + +| 模組 | 之前 | 之後 | +| --- | ---: | ---: | +| `backends/windows_backend.py` | 17.72% | **99.42%** | +| `backends/linux_backend.py` | 42.41% | **100%** | +| `backends/macos_backend.py` | 0.00% | **100%** | +| `backends/windows_query.py` | 24.75% | **99.01%** | +| `backends/windows_state.py` | 18.75% | **100%** | +| `backends/base.py` | 74.03% | **100%** | +| `backends/__init__.py` | 48.94% | **100%** | + +剩的三行在 `_process_name` 的 Win32 失敗路徑（`OpenProcess` 成功但 +`QueryFullProcessImageNameW` 失敗），要一個「開得到卻查不到」的行程才踩得到。 + +**AT-SPI 那一層要換的是 `SessionBus`，不是 `_AtspiConnection`。** 現有的 +`test_accessibility_linux.py` 換掉後者，那對測「走訪」是對的，但底下整個 +D-Bus 呼叫層一行都沒跑過——而協定就住在那裡（無障礙匯流排不是 session bus、 +accessible 是 `(sender, path)` **配對**、狀態位元是**兩個 32-bit word**， +只讀第一個會靜靜丟掉第 31 位以上的每一個狀態）。 + +**寫測試時抓到一個真的錯，形狀和 WebRTC 那個一模一樣：攔截 tuple 漏了型別。** +comtypes 把 provider 失敗報成 `COMError`，而它直接繼承 `Exception`： + +```python +issubclass(COMError, (OSError, AttributeError, ValueError, TypeError)) # False +``` + +`windows_query._uia_errors()` 存在就是為了講這件事，docstring 也點名了情境 +（「視窗在走訪途中關掉，或應用程式停止回應」）。但 `windows_backend.py` 裡 +只有**兩個**走訪用的 except 用了那個 tuple，**另外 37 個**——也就是每一個 +control pattern——寫的是 `(OSError, AttributeError, …)`，一個都攔不到。 +race 很小但真實：`_find_raw` 自己會攔，所以曝露的是「找到之後、讀之前」 +那一段。已全部改用同一個 tuple；這只會**放寬**攔截範圍。 ### mypy：整包把關，**豁免清單已經清空** diff --git a/WHATS_NEW.md b/WHATS_NEW.md index d5c2fbd9..851b9ea2 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -207,6 +207,71 @@ je_auto_control.windows.window import windows_window_manage` resolves by double under its own dotted name does nothing on a machine where the real module is importable — the package it is read off has to be the double. +### The Accessibility Backends, Including The One Nobody Could Reach + +`utils/accessibility` was the second-largest gap in the project and the one +that looked hardest: a UIAutomation provider, an AT-SPI bus and a granted +macOS Accessibility permission are three things no CI runner has, and the +Windows backend alone is 569 statements at 17%. + +The same fact that unlocked the window backends applies here — every +platform import is inside a method — so the three backends and everything +under them now run on all nine squares: + +| module | before | after | +| --- | ---: | ---: | +| `backends/windows_backend.py` | 17.72% | **99.42%** | +| `backends/linux_backend.py` | 42.41% | **100%** | +| `backends/macos_backend.py` | 0.00% | **100%** | +| `backends/windows_query.py` | 24.75% | **99.01%** | +| `backends/windows_state.py` | 18.75% | **100%** | +| `backends/base.py` | 74.03% | **100%** | +| `backends/__init__.py` | 48.94% | **100%** | + +Two of the doubles are worth describing because of what they refuse. + +The UIA one models the indirection every control pattern goes through — ask +an element for a pattern id, then query an interface name off the generated +module — and **refuses a mismatched pair**. Those are two independent +constants with nothing checking them at runtime, and asking for the +ValuePattern id while querying the RangeValuePattern interface fails as a +`None` that reads exactly like "no such control". It also raises +`AttributeError` for any `Current…` property the test did not supply, because +answering with a callable would let `str(pattern.CurrentValue or "")` pass +against the repr of a function. + +The AT-SPI one replaces `SessionBus` rather than `_AtspiConnection`. The +existing Linux test replaces the connection, which is right for testing the +*walk* and leaves the entire D-Bus call layer beneath it unexecuted — and +that is where the protocol lives: the accessibility bus is not the session +bus, an accessible is addressed by a `(sender, path)` *pair*, and the state +bitfield arrives as two 32-bit words, so reading only the first drops every +state above bit 31. + +### 37 Guards That Did Not Contain The Failure They Were Written For + +Writing those tests turned one up. `comtypes` reports a provider failure as +`COMError`, which derives straight from `Exception`: + +```python +>>> issubclass(COMError, (OSError, AttributeError, ValueError, TypeError)) +False +``` + +`windows_query._uia_errors()` exists to say exactly that, and its docstring +names the case: "a window that closes mid-walk, or an application that stops +responding, surfaces exactly that way". Two guards in +`backends/windows_backend.py` used it. The other **37** — every control +pattern in the file — spelled `(OSError, AttributeError, …)` and contained +none of them. + +The race is real and small: `_find_raw` contains a dead provider and answers +`None`, so the found path is the exposed one. Between the search that returns +an element and the `GetCurrentPattern` call that reads it, an application can +go away — and the answer was a `COMError` out of the `ac_*` tool, past the +executor's `AutoControlException` boundary, instead of the `None` the method +promised. All 37 now share one tuple. It only widens what is caught. + ### The Viewer Logged Every Clean Disconnect As An Unhandled Task Exception `WebRTCDesktopViewer._consume_video` caught `(OSError, RuntimeError)`. aiortc diff --git a/architecture_explore.md b/architecture_explore.md index 358bc7d3..5f1a4a63 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -20,7 +20,7 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | | Python 模組總數（含周邊子專案） | 1,032 | -| 程式碼總行數 | 141,334 | +| 程式碼總行數 | 141,339 | | `je_auto_control/utils/` 子套件數 | 310 | | `AC_*` 動作指令數（`known_commands()` 實測） | 773 | | 套件門面 `__all__` 公開名稱數 | 1,238 | @@ -437,12 +437,12 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.7 無障礙樹與原生控制項 -> 16 個套件、約 4,313 行。 +> 16 個套件、約 4,318 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | | `utils/a11y_audit/` | 355 | 以無障礙樹 + OCR 進行無障礙與 i18n 稽核 | -| `utils/accessibility/` | 2,835 | 跨平台無障礙樹定位與錄製；Windows UIA／macOS AX／null 三後端。支援限定視窗（換搜尋起點，不是過濾）、逐節點可中斷走訪、`IUIAutomation2` 連線逾時、名稱子字串比對與排序、`control_get_state` 一次讀完值／勾選／選取／數值（密碼欄位不回內容） | +| `utils/accessibility/` | 2,840 | 跨平台無障礙樹定位與錄製；Windows UIA／macOS AX／null 三後端。支援限定視窗（換搜尋起點，不是過濾）、逐節點可中斷走訪、`IUIAutomation2` 連線逾時、名稱子字串比對與排序、`control_get_state` 一次讀完值／勾選／選取／數值（密碼欄位不回內容） | | `utils/ax_events/` | 29 | 反應式 UIA 事件等待（focus-changed） | | `utils/ax_props/` | 44 | 讀取豐富 UIA 屬性（enabled／offscreen／help／status／快捷鍵） | | `utils/ax_text/` | 102 | 透過 UIA TextPattern 取得原生文字（讀取／尋找／選取／屬性） | @@ -1031,7 +1031,7 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `utils/executor/` | 6 | 9,081 | | `utils/usb/` | 17 | 4,281 | | `je_auto_control/`（頂層 3 檔） | 3 | 2,367 | -| `utils/accessibility/` | 13 | 2,835 | +| `utils/accessibility/` | 13 | 2,840 | | `wrapper/` | 19 | 3,514 | | `windows/` | 23 | 1,906 | | `utils/rest_api/` | 8 | 1,751 | @@ -1046,5 +1046,5 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 727 | | 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 673 | 47,683 | -| **總計** | **1,026** | **141,269** | +| **總計** | **1,026** | **141,274** | diff --git a/je_auto_control/utils/accessibility/backends/windows_backend.py b/je_auto_control/utils/accessibility/backends/windows_backend.py index 0d2783f4..a7833263 100644 --- a/je_auto_control/utils/accessibility/backends/windows_backend.py +++ b/je_auto_control/utils/accessibility/backends/windows_backend.py @@ -10,7 +10,7 @@ """ import functools import sys -from typing import Any, Dict, List, Optional +from typing import Any, Dict, List, Optional, Tuple, Type from je_auto_control.utils.accessibility.backends.base import ( AccessibilityBackend, @@ -26,6 +26,11 @@ ) from je_auto_control.utils.logging.logging_instance import autocontrol_logger +#: Every guard below. ``COMError`` is why a narrower tuple would contain +#: nothing — see ``windows_query._uia_errors``; ``TypeError`` joins it because +#: several of these reads coerce whatever the provider returned. +_UIA_ERRORS: Tuple[Type[BaseException], ...] = UIA_ERRORS + (TypeError,) + _UIA_IS_CONTROL_ELEMENT_PROPERTY = 30016 _UIA_NAME_PROPERTY = 30005 _UIA_VALUE_PATTERN_ID = 10002 @@ -98,7 +103,7 @@ def _create_automation(uia_module): interface=interface) automation.ConnectionTimeout = _CONNECTION_TIMEOUT_MS return automation - except (OSError, AttributeError, ValueError) as error: + except _UIA_ERRORS as error: autocontrol_logger.info( "UIAutomation2 unavailable, provider waits are unbounded: %r", error) @@ -258,7 +263,7 @@ def _pattern(self, raw, pattern_id, interface_name): return None interface = getattr(self._uia_module, interface_name) return unknown.QueryInterface(interface) - except (OSError, AttributeError, ValueError): + except _UIA_ERRORS: return None def get_value(self, name=None, role=None, app_name=None, @@ -278,7 +283,7 @@ def get_value(self, name=None, role=None, app_name=None, return None try: return str(pattern.CurrentValue or "") - except (OSError, AttributeError): + except _UIA_ERRORS: return None def set_value(self, value, name=None, role=None, app_name=None, @@ -291,7 +296,7 @@ def set_value(self, value, name=None, role=None, app_name=None, try: pattern.SetValue(str(value)) return True - except (OSError, AttributeError): + except _UIA_ERRORS: return False def invoke(self, name=None, role=None, app_name=None, @@ -304,7 +309,7 @@ def invoke(self, name=None, role=None, app_name=None, try: pattern.Invoke() return True - except (OSError, AttributeError): + except _UIA_ERRORS: return False def toggle(self, name=None, role=None, app_name=None, @@ -317,7 +322,7 @@ def toggle(self, name=None, role=None, app_name=None, try: pattern.Toggle() return True - except (OSError, AttributeError): + except _UIA_ERRORS: return False def read_table(self, name=None, role=None, app_name=None, @@ -330,7 +335,7 @@ def read_table(self, name=None, role=None, app_name=None, try: rows = int(pattern.CurrentRowCount or 0) cols = int(pattern.CurrentColumnCount or 0) - except (OSError, AttributeError): + except _UIA_ERRORS: return [] return [self._read_row(pattern, r, cols) for r in range(rows)] @@ -344,7 +349,7 @@ def _invoke_pattern_method(self, name, role, app_name, automation_id, try: action(pattern) return True - except (OSError, AttributeError): + except _UIA_ERRORS: return False def expand(self, name=None, role=None, app_name=None, automation_id=None): @@ -366,7 +371,7 @@ def expand_state(self, name=None, role=None, app_name=None, return None try: return _EXPAND_STATES.get(int(pattern.CurrentExpandCollapseState)) - except (OSError, AttributeError, ValueError, TypeError): + except _UIA_ERRORS: return None def select_item(self, name=None, role=None, app_name=None, automation_id=None): @@ -397,7 +402,7 @@ def get_range(self, name=None, role=None, app_name=None, return {"value": float(pattern.CurrentValue), "minimum": float(pattern.CurrentMinimum), "maximum": float(pattern.CurrentMaximum)} - except (OSError, AttributeError, ValueError, TypeError): + except _UIA_ERRORS: return None def _realize(self, raw) -> None: @@ -408,7 +413,7 @@ def _realize(self, raw) -> None: return try: pattern.Realize() - except (OSError, AttributeError): + except _UIA_ERRORS: pass def find_virtual_item(self, item_name=None, by="name", container_name=None, @@ -424,7 +429,7 @@ def find_virtual_item(self, item_name=None, by="name", container_name=None, else _UIA_NAME_PROPERTY) try: found = pattern.FindItemByProperty(None, property_id, item_name) - except (OSError, AttributeError, ValueError): + except _UIA_ERRORS: return None if not found: return None @@ -479,7 +484,7 @@ def window_interaction_state(self, name=None, role=None, app_name=None, try: return _WINDOW_INTERACTION_STATES.get( int(pattern.CurrentWindowInteractionState)) - except (OSError, AttributeError, ValueError, TypeError): + except _UIA_ERRORS: return None def legacy_info(self, name=None, role=None, app_name=None, @@ -511,7 +516,7 @@ def get_selection(self, name=None, role=None, app_name=None, items = _header_names(pattern.GetCurrentSelection()) can_multiple = bool(pattern.CurrentCanSelectMultiple) required = bool(pattern.CurrentIsSelectionRequired) - except (OSError, AttributeError): + except _UIA_ERRORS: return None return {"items": items, "can_select_multiple": can_multiple, "is_required": required} @@ -529,7 +534,7 @@ def list_views(self, name=None, role=None, app_name=None, try: view_ids = list(pattern.GetCurrentSupportedViews()) current = int(pattern.CurrentCurrentView) - except (OSError, AttributeError, ValueError, TypeError): + except _UIA_ERRORS: return None return {"current": _view_name(pattern, current), "views": [_view_name(pattern, view_id) for view_id in view_ids]} @@ -544,7 +549,7 @@ def set_view(self, view="", name=None, role=None, app_name=None, if _view_name(pattern, view_id) == str(view): pattern.SetCurrentView(int(view_id)) return True - except (OSError, AttributeError, ValueError, TypeError): + except _UIA_ERRORS: return False return False @@ -577,7 +582,7 @@ def wait_for_focus_change(self, timeout=5.0) -> Optional[Dict[str, Any]]: try: with self._event_lock: automation.AddFocusChangedEventHandler(None, handler) - except (OSError, AttributeError): + except _UIA_ERRORS: return None try: return events.get(timeout=float(timeout)) @@ -587,7 +592,7 @@ def wait_for_focus_change(self, timeout=5.0) -> Optional[Dict[str, Any]]: with self._event_lock: try: automation.RemoveFocusChangedEventHandler(handler) - except (OSError, AttributeError): + except _UIA_ERRORS: pass def get_table_headers(self, name=None, role=None, app_name=None, @@ -600,7 +605,7 @@ def get_table_headers(self, name=None, role=None, app_name=None, try: columns = pattern.GetCurrentColumnHeaders() rows = pattern.GetCurrentRowHeaders() - except (OSError, AttributeError): + except _UIA_ERRORS: return None return {"columns": _header_names(columns), "rows": _header_names(rows)} @@ -613,7 +618,7 @@ def get_grid_cell(self, row=0, column=0, name=None, role=None, return None try: cell = grid.GetItem(int(row), int(column)) - except (OSError, AttributeError): + except _UIA_ERRORS: return None if not cell: return None @@ -636,7 +641,7 @@ def document_text(self, name=None, role=None, app_name=None, return None try: return str(pattern.DocumentRange.GetText(-1) or "") - except (OSError, AttributeError): + except _UIA_ERRORS: return None def selected_text(self, name=None, role=None, app_name=None, @@ -649,7 +654,7 @@ def selected_text(self, name=None, role=None, app_name=None, if not selection or int(selection.Length or 0) == 0: return "" return str(selection.GetElement(0).GetText(-1) or "") - except (OSError, AttributeError): + except _UIA_ERRORS: return None def visible_text(self, name=None, role=None, app_name=None, @@ -662,7 +667,7 @@ def visible_text(self, name=None, role=None, app_name=None, count = int(ranges.Length or 0) return "".join(str(ranges.GetElement(i).GetText(-1) or "") for i in range(count)) - except (OSError, AttributeError): + except _UIA_ERRORS: return None def _find_range(self, text, ignore_case, name, role, app_name, automation_id): @@ -673,7 +678,7 @@ def _find_range(self, text, ignore_case, name, role, app_name, automation_id): try: return pattern.DocumentRange.FindText(str(text), False, bool(ignore_case)) - except (OSError, AttributeError): + except _UIA_ERRORS: return None def find_text(self, text="", ignore_case=True, name=None, role=None, @@ -690,7 +695,7 @@ def select_text(self, text="", ignore_case=True, name=None, role=None, try: found.Select() return True - except (OSError, AttributeError): + except _UIA_ERRORS: return False def text_attributes(self, name=None, role=None, app_name=None, @@ -703,7 +708,7 @@ def text_attributes(self, name=None, role=None, app_name=None, text_range = (selection.GetElement(0) if selection and int(selection.Length or 0) > 0 else pattern.DocumentRange) - except (OSError, AttributeError): + except _UIA_ERRORS: return None return _read_text_attributes(text_range) @@ -715,7 +720,7 @@ def set_focus(self, name=None, role=None, app_name=None, try: raw.SetFocus() return True - except (OSError, AttributeError): + except _UIA_ERRORS: return False @staticmethod @@ -726,7 +731,7 @@ def _read_row(pattern, row: int, cols: int): try: cell = pattern.GetItem(row, col) cells.append(str(cell.CurrentName or "") if cell else "") - except (OSError, AttributeError): + except _UIA_ERRORS: cells.append("") return cells @@ -735,7 +740,7 @@ def _view_name(pattern, view_id) -> str: """Return a MultipleViewPattern view's name, or '' on failure.""" try: return str(pattern.GetViewName(int(view_id)) or "") - except (OSError, AttributeError, ValueError, TypeError): + except _UIA_ERRORS: return "" @@ -744,12 +749,12 @@ def _header_names(array) -> List[str]: names: List[str] = [] try: count = int(array.Length or 0) - except (OSError, AttributeError): + except _UIA_ERRORS: return names for index in range(count): try: names.append(str(array.GetElement(index).CurrentName or "")) - except (OSError, AttributeError): + except _UIA_ERRORS: names.append("") return names @@ -766,7 +771,7 @@ def _read_cell(item_pattern, cell, row: int, column: int) -> Dict[str, Any]: ("column_span", "CurrentColumnSpan")): try: info[key] = int(getattr(item_pattern, attr)) - except (OSError, AttributeError, ValueError, TypeError): + except _UIA_ERRORS: pass return info @@ -774,7 +779,7 @@ def _read_cell(item_pattern, cell, row: int, column: int) -> Dict[str, Any]: def _safe_name(raw) -> str: try: return str(raw.CurrentName or "") - except (OSError, AttributeError): + except _UIA_ERRORS: return "" @@ -793,7 +798,7 @@ def _as_text(value) -> str: def _attr(text_range, attribute_id, cast): try: return cast(text_range.GetAttributeValue(attribute_id)) - except (OSError, AttributeError, ValueError, TypeError): + except _UIA_ERRORS: return None @@ -826,7 +831,7 @@ def _read_legacy(pattern) -> Dict[str, Any]: for key, attribute, cast in _LEGACY_READS: try: info[key] = cast(getattr(pattern, attribute)) - except (OSError, AttributeError, ValueError, TypeError): + except _UIA_ERRORS: info[key] = None return info @@ -849,7 +854,7 @@ def _read_properties(raw) -> Dict[str, Any]: for key, attribute, cast in _PROPERTY_READS: try: properties[key] = cast(getattr(raw, attribute)) - except (OSError, AttributeError, ValueError, TypeError): + except _UIA_ERRORS: properties[key] = None return properties @@ -871,7 +876,7 @@ def _convert_uia(raw, cached: bool = False) -> Optional[AccessibilityElement]: process_id = int(getattr(raw, prefix + "ProcessId") or 0) automation_id = str(getattr(raw, prefix + "AutomationId") or "") enabled = bool(getattr(raw, prefix + "IsEnabled")) - except (OSError, AttributeError): + except _UIA_ERRORS: return None width = max(0, int(rect.right - rect.left)) height = max(0, int(rect.bottom - rect.top)) diff --git a/test/unit_test/headless/_uia_doubles.py b/test/unit_test/headless/_uia_doubles.py new file mode 100644 index 00000000..dcc79da5 --- /dev/null +++ b/test/unit_test/headless/_uia_doubles.py @@ -0,0 +1,246 @@ +"""A UIAutomation stand-in, so the Windows accessibility backend is testable. + +`backends/windows_backend.py` is the largest single gap in the project's +coverage -- 568 statements at 17% -- and it stayed there on the Windows +squares too, because reaching it needs a UIAutomation provider, a desktop +with windows on it, and applications willing to answer. None of that exists +on a CI runner of any platform. + +It does not need any of it. Every `comtypes` import in the module is inside a +function, the automation object is held on the instance, and each control +pattern is reached the same way: find a raw element, ask it for a pattern id, +query an interface off the generated module. All three are values a test can +supply. + +The shape modelled here is that indirection, because it is where a mistake +hides: + + unknown = raw.GetCurrentPattern(pattern_id) # None if unsupported + interface = getattr(uia_module, interface_name) # generated at import + pattern = unknown.QueryInterface(interface) + +A control that does not support a pattern answers with nothing, and every +caller has to turn that into its own "no" -- `None`, `False`, or `[]` +depending on what it promised to return. `Unknown` below therefore refuses to +hand back a pattern for the wrong interface name, so a method that asks for +`IUIAutomationValuePattern` where it meant `IUIAutomationRangeValuePattern` +fails here rather than silently working against a double that answers +everything. + +Nothing here is a test; the file is named so pytest does not collect it. +""" +from __future__ import annotations + +import sys +import types + + +class Rect: + """What UIA's BoundingRectangle property hands back.""" + + def __init__(self, left=0, top=0, right=0, bottom=0) -> None: + self.left = left + self.top = top + self.right = right + self.bottom = bottom + + +class Pattern: + """One control pattern: attributes to read, and calls to record.""" + + def __init__(self, **attributes) -> None: + self.__dict__.update(attributes) + self.calls = [] + self.errors = {} # method name -> exception to raise + + def _record(self, name, *args): + self.calls.append((name, args)) + if name in self.errors: + raise self.errors[name] + return None + + def __getattr__(self, name): + if name.startswith("_"): + raise AttributeError(name) + # A UIA *property* is spelled `Current…` / `Cached…`; one that was + # not supplied is one the provider will not answer, which reaches the + # backend as AttributeError and is a path it handles. Answering with + # a callable instead would let `str(pattern.CurrentValue or "")` pass + # against the repr of a function. + if name.startswith(("Current", "Cached")): + raise AttributeError(name) + + def _call(*args): + return self._record(name, *args) + return _call + + +class Unknown: + """What `GetCurrentPattern` returns: an IUnknown to query an interface off.""" + + def __init__(self, interface_name: str, pattern: Pattern) -> None: + self.interface_name = interface_name + self.pattern = pattern + self.queried = [] + + def QueryInterface(self, interface): # noqa: N802 # reason: COM name + self.queried.append(interface) + if interface != self.interface_name: + raise TypeError( + f"asked for {interface!r}, this pattern is " + f"{self.interface_name!r}") + return self.pattern + + +class RawElement: + """A raw UIA element: properties, patterns, and what was done to it.""" + + def __init__(self, name="", control_type=0, rect=None, process_id=0, + automation_id="", enabled=True, patterns=None, + properties=None, cached=False) -> None: + self.patterns = dict(patterns or {}) # pattern id -> Unknown + self.properties = dict(properties or {}) # property id -> value + self.focused = 0 + self.pattern_error = None + self.property_error = None + self.focus_error = None + prefix = "Cached" if cached else "Current" + setattr(self, prefix + "Name", name) + setattr(self, prefix + "ControlType", control_type) + setattr(self, prefix + "BoundingRectangle", rect or Rect()) + setattr(self, prefix + "ProcessId", process_id) + setattr(self, prefix + "AutomationId", automation_id) + setattr(self, prefix + "IsEnabled", enabled) + # `_convert_uia` reads one prefix or the other; a raw element that + # only ever appears cached must not answer to `Current*`. + self._prefix = prefix + + def GetCurrentPattern(self, pattern_id): # noqa: N802 # reason: UIA name + if self.pattern_error is not None: + raise self.pattern_error + return self.patterns.get(pattern_id) + + def GetCurrentPropertyValue(self, property_id): # noqa: N802 # UIA name + if self.property_error is not None: + raise self.property_error + return self.properties.get(property_id) + + def SetFocus(self): # noqa: N802 # reason: the UIA name + if self.focus_error is not None: + raise self.focus_error + self.focused += 1 + + +class ElementArray: + """An `IUIAutomationElementArray`: a length and indexed access.""" + + def __init__(self, elements=None, length_error=None, + element_error=None) -> None: + self.elements = list(elements or []) + self.length_error = length_error + self.element_error = element_error + + @property + def Length(self): # noqa: N802 # reason: the UIA name + if self.length_error is not None: + raise self.length_error + return len(self.elements) + + def GetElement(self, index): # noqa: N802 # reason: the UIA name + if self.element_error is not None: + raise self.element_error + return self.elements[index] + + +class UiaModule: + """The comtypes-generated `UIAutomationClient` module. + + Its interfaces are generated at import time, so a name is all there is to + hold on to; `getattr` therefore answers with the name itself and + `Unknown` compares against it. + """ + + def __init__(self, has_iuiautomation2: bool = True) -> None: + self._has_2 = has_iuiautomation2 + self.IUIAutomation = "IUIAutomation" + + def __getattr__(self, name): + # An older Windows generates a module without IUIAutomation2 at all, + # which is the fallback the backend is written to take -- so the + # absence has to be a real AttributeError, not a name that answers. + if name.startswith("_"): + raise AttributeError(name) + if name == "IUIAutomation2" and not self._has_2: + raise AttributeError(name) + return name + + +class Automation: + """The UIAutomation object: roots, a walker, and event handler bookkeeping.""" + + def __init__(self, root=None) -> None: + self.root = root + self.focus_handlers = [] + self.add_error = None + self.remove_error = None + self.ConnectionTimeout = None # noqa: N815 # reason: the UIA name + + def GetRootElement(self): # noqa: N802 # reason: the UIA name + return self.root + + def AddFocusChangedEventHandler(self, cache_request, handler): # noqa: N802 + if self.add_error is not None: + raise self.add_error + self.focus_handlers.append(handler) + + def RemoveFocusChangedEventHandler(self, handler): # noqa: N802 + if self.remove_error is not None: + raise self.remove_error + if handler in self.focus_handlers: + self.focus_handlers.remove(handler) + + +def with_pattern(pattern_id: int, interface_name: str, + pattern: Pattern = None) -> dict: + """A `patterns` mapping carrying one pattern, ready for `RawElement`.""" + return {pattern_id: Unknown(interface_name, pattern or Pattern())} + + +def install_comtypes(monkeypatch, uia_module=None, module_error=None, + create=None): + """Put a `comtypes` package in `sys.modules` and return what it hands out. + + Only the four names the backend reaches for: `client.GetModule`, which + generates the UIAutomationClient module, and `CoCreateInstance` / `GUID` / + `COMObject`, which build the automation object and the event handler. + """ + created = [] + + def _co_create_instance(clsid, interface=None): + created.append((clsid, interface)) + if create is not None: + return create(clsid, interface) + return Automation() + + comtypes = types.ModuleType("comtypes") + comtypes.CoCreateInstance = _co_create_instance + comtypes.GUID = lambda text: text + + class _ComObject: + """The base comtypes gives a Python-implemented COM interface.""" + + comtypes.COMObject = _ComObject + + client = types.ModuleType("comtypes.client") + + def _get_module(name): + if module_error is not None: + raise module_error + return uia_module if uia_module is not None else UiaModule() + + client.GetModule = _get_module + comtypes.client = client + + monkeypatch.setitem(sys.modules, "comtypes", comtypes) + monkeypatch.setitem(sys.modules, "comtypes.client", client) + return created diff --git a/test/unit_test/headless/test_accessibility_backend_selection.py b/test/unit_test/headless/test_accessibility_backend_selection.py new file mode 100644 index 00000000..fe9584e9 --- /dev/null +++ b/test/unit_test/headless/test_accessibility_backend_selection.py @@ -0,0 +1,216 @@ +"""Which accessibility backend a platform gets, and what the rest refuse. + +The seam is the same shape as the window-management one -- abstract base, +three implementations, a null fallback carrying a reason -- and it had the +same hole: the selector only ever ran its own platform's arm, so two of the +three branches were dead on every square, along with the base class's whole +refusal surface. + +Two things are worth stating in a test rather than in a comment: + +* **The reason has to name the *right* missing thing.** These are the + messages an operator reads when `ac_list_accessibility_elements` comes back + empty, and each platform fails for a different reason: a missing pip + package on Windows, a missing framework on macOS, and on Linux a *bus* that + may be absent or merely unbridged -- which is why that one names + at-spi2-core and the toolkit bridge in the same breath. +* **The base refuses rather than answering falsely.** Thirty-odd control + patterns exist because UIA has them; AT-SPI and AX have a fraction. A + backend that returned `None` or `False` for the rest would be indis- + tinguishable from "the control is not there", and a caller cannot recover + from that. Every one of them raises instead. + +The macOS backend is exercised here too, through the pyobjc stub, which is +what lets all of this run on every square rather than only on Darwin. +""" +from __future__ import annotations + +import sys +import types + +import pytest + +from headless import _pyobjc_stub as objc_stub +from je_auto_control.utils.accessibility import backends as backends_mod +from je_auto_control.utils.accessibility.backends import ( + NullAccessibilityBackend, get_backend, reset_backend_cache, +) +from je_auto_control.utils.accessibility.backends.base import ( + AccessibilityBackend, +) +from je_auto_control.utils.accessibility.element import ( + AccessibilityNotAvailableError, +) + + +@pytest.fixture(autouse=True) +def clean_cache(): + reset_backend_cache() + yield + reset_backend_cache() + + +@pytest.fixture +def on_platform(monkeypatch): + def _set(name: str): + monkeypatch.setattr(backends_mod.sys, "platform", name) + return _set + + +def _stub_module(monkeypatch, name: str, **attributes): + module = types.ModuleType(name) + for key, value in attributes.items(): + setattr(module, key, value) + monkeypatch.setitem(sys.modules, name, module) + return module + + +# --- selection ---------------------------------------------------------------- + +def test_windows_with_comtypes_selects_the_uia_backend(on_platform, + monkeypatch): + on_platform("win32") + monkeypatch.setattr( + "je_auto_control.utils.accessibility.backends.windows_backend" + "._is_available", lambda: True) + assert get_backend().name == "windows-uia" + + +def test_windows_without_comtypes_gets_a_refusal_naming_it(on_platform, + monkeypatch): + on_platform("win32") + monkeypatch.setattr( + "je_auto_control.utils.accessibility.backends.windows_backend" + "._is_available", lambda: False) + backend = get_backend() + assert isinstance(backend, NullAccessibilityBackend) + with pytest.raises(AccessibilityNotAvailableError, match="comtypes"): + backend.list_elements() + + +def test_a_mac_with_pyobjc_selects_the_ax_backend(on_platform, monkeypatch): + on_platform("darwin") + objc_stub.install(monkeypatch, objc_stub.World()) + _stub_module(monkeypatch, "ApplicationServices") + assert get_backend().name == "macos-ax" + + +def test_a_mac_without_pyobjc_gets_a_refusal_naming_it(on_platform, + monkeypatch): + on_platform("darwin") + monkeypatch.setitem(sys.modules, "ApplicationServices", None) + backend = get_backend() + with pytest.raises(AccessibilityNotAvailableError, match="pyobjc"): + backend.list_elements() + + +def test_a_linux_session_with_a_bus_selects_the_atspi_backend(on_platform, + monkeypatch): + on_platform("linux") + monkeypatch.setattr( + "je_auto_control.utils.accessibility.backends.linux_backend" + "._is_available", lambda: True) + assert get_backend().name == "linux-atspi" + + +def test_a_linux_session_with_no_bus_names_both_ways_it_can_be_missing( + on_platform, monkeypatch): + # AT-SPI is a bus, not a library, so "not installed" and "installed but + # nothing is bridged to it" look identical from here. + on_platform("linux") + monkeypatch.setattr( + "je_auto_control.utils.accessibility.backends.linux_backend" + "._is_available", lambda: False) + backend = get_backend() + with pytest.raises(AccessibilityNotAvailableError) as caught: + backend.list_elements() + message = str(caught.value) + assert "at-spi2-core" in message + assert "atk-bridge" in message + + +def test_an_unknown_platform_gets_a_refusal_naming_it(on_platform): + on_platform("sunos5") + with pytest.raises(AccessibilityNotAvailableError, match="sunos5"): + get_backend().list_elements() + + +def test_the_choice_is_made_once_and_cached(on_platform): + on_platform("sunos5") + assert get_backend() is get_backend() + + +def test_resetting_the_cache_re_detects(on_platform): + on_platform("sunos5") + first = get_backend() + reset_backend_cache() + assert get_backend() is not first + + +# --- what the base class refuses ---------------------------------------------- + +_REFUSALS = [ + ("get_value", lambda b: b.get_value()), + ("set_value", lambda b: b.set_value("x")), + ("invoke", lambda b: b.invoke()), + ("toggle", lambda b: b.toggle()), + ("read_table", lambda b: b.read_table()), + ("expand", lambda b: b.expand()), + ("collapse", lambda b: b.collapse()), + ("expand_state", lambda b: b.expand_state()), + ("select_item", lambda b: b.select_item()), + ("get_range", lambda b: b.get_range()), + ("set_range_value", lambda b: b.set_range_value(1.0)), + ("scroll_into_view", lambda b: b.scroll_into_view()), + ("document_text", lambda b: b.document_text()), + ("selected_text", lambda b: b.selected_text()), + ("visible_text", lambda b: b.visible_text()), + ("find_text", lambda b: b.find_text("x")), + ("select_text", lambda b: b.select_text("x")), + ("text_attributes", lambda b: b.text_attributes()), + ("set_focus", lambda b: b.set_focus()), + ("find_virtual_item", lambda b: b.find_virtual_item("x")), + ("get_properties", lambda b: b.get_properties()), + ("get_state", lambda b: b.get_state()), + ("get_table_headers", lambda b: b.get_table_headers()), + ("get_grid_cell", lambda b: b.get_grid_cell(0, 0)), + ("move_element", lambda b: b.move_element(1.0, 2.0)), + ("resize_element", lambda b: b.resize_element(1.0, 2.0)), + ("set_window_state", lambda b: b.set_window_state("normal")), + ("window_interaction_state", lambda b: b.window_interaction_state()), + ("legacy_info", lambda b: b.legacy_info()), + ("legacy_default_action", lambda b: b.legacy_default_action()), + ("get_selection", lambda b: b.get_selection()), + ("list_views", lambda b: b.list_views()), + ("set_view", lambda b: b.set_view("Details")), + ("wait_for_focus_change", lambda b: b.wait_for_focus_change(0.1)), +] + + +@pytest.mark.parametrize("operation,call", + _REFUSALS, ids=[name for name, _ in _REFUSALS]) +def test_an_unimplemented_pattern_says_which_one_and_whose(operation, call): + # "This backend cannot do it" and "the control is not there" are + # different answers, and a caller that cannot tell them apart retries + # forever. The message carries both halves. + backend = AccessibilityBackend() + with pytest.raises(AccessibilityNotAvailableError) as caught: + call(backend) + assert operation in str(caught.value) + assert backend.name in str(caught.value) + + +def test_the_base_class_has_no_listing_of_its_own(): + with pytest.raises(NotImplementedError): + AccessibilityBackend().list_elements() + + +def test_the_null_backend_refuses_to_list_with_its_reason(): + with pytest.raises(AccessibilityNotAvailableError, match="no display"): + NullAccessibilityBackend("no display").list_elements() + + +def test_a_null_backend_with_no_reason_still_says_something(): + with pytest.raises(AccessibilityNotAvailableError) as caught: + NullAccessibilityBackend().list_elements() + assert str(caught.value) diff --git a/test/unit_test/headless/test_accessibility_linux.py b/test/unit_test/headless/test_accessibility_linux.py index c8340500..ea63a9e9 100644 --- a/test/unit_test/headless/test_accessibility_linux.py +++ b/test/unit_test/headless/test_accessibility_linux.py @@ -33,6 +33,13 @@ def __init__(self, tree=None, names=None, roles=None, extents=None, self.roles = roles or {} self.extents_map = extents or {} self.states = states or {} + self.texts = {} + self.numbers = {} + self.errors = {} # reference -> DBusError to raise on a write + self.actions = [] + self.written = [] + self.focused = [] + self.write_result = True self.entered = 0 self.exited = 0 @@ -65,10 +72,32 @@ def extents(self, reference): return self.extents_map.get(reference, (0, 0, 0, 0)) def text(self, reference): - return None + if reference in self.errors: + raise self.errors[reference] + return self.texts.get(reference) def number(self, reference): - return None + return self.numbers.get(reference) + + # --- writes, recorded rather than sent --------------------------------- + + def do_action(self, reference, index=0): + if reference in self.errors: + raise self.errors[reference] + self.actions.append((reference, index)) + return self.write_result + + def set_text(self, reference, value): + if reference in self.errors: + raise self.errors[reference] + self.written.append((reference, value)) + return self.write_result + + def grab_focus(self, reference): + if reference in self.errors: + raise self.errors[reference] + self.focused.append(reference) + return self.write_result APP = ("app", "/app") @@ -231,3 +260,247 @@ def call(self, *_args, **_kwargs): connection = atspi._AtspiConnection() connection._bus = TwoWordBus() assert connection.state(BUTTON) == (1 << 8) | (1 << 32) + + +# --- control patterns ------------------------------------------------------ +# +# The five object-level actions all share one shape: find the reference the +# caller described, then make exactly one AT-SPI call on it. What is worth +# pinning is what happens when the find comes back empty -- every one of them +# has to answer "no", because the caller cannot tell a control that refused +# from a control that was never there, and will otherwise retry forever. + + +def test_get_value_prefers_the_text_interface(backend): + backend.connection.texts[BUTTON] = "typed" + backend.connection.numbers[BUTTON] = 0.5 + assert backend.get_value(name="OK") == "typed" + + +def test_get_value_falls_back_to_a_numeric_value(backend): + # A slider has no text; its value is a double on the Value interface. + backend.connection.numbers[BUTTON] = 0.75 + assert backend.get_value(name="OK") == "0.75" + + +def test_get_value_of_a_control_with_neither_is_none(backend): + assert backend.get_value(name="OK") is None + + +def test_get_value_of_a_control_that_is_not_there_is_none(backend): + assert backend.get_value(name="Cancel") is None + + +def test_get_value_can_be_scoped_to_one_application(backend): + backend.connection.texts[BUTTON] = "typed" + assert backend.get_value(name="OK", app_name="zenity") == "typed" + assert backend.get_value(name="OK", app_name="gedit") is None + + +def test_get_value_matches_a_substring_when_asked(backend): + # Real interfaces label controls "Save(&S)" and "OK "; exact stays + # the default, so the same lower-case needle finds nothing without + # `contains`. + backend.connection.texts[BUTTON] = "typed" + assert backend.get_value(name="ok", contains=True) == "typed" + assert backend.get_value(name="ok") is None + + +def test_set_value_writes_through_the_editable_interface(backend): + assert backend.set_value("hello", name="OK") is True + assert backend.connection.written == [(BUTTON, "hello")] + + +def test_set_value_on_a_control_that_is_not_there_reports_failure(backend): + assert backend.set_value("hello", name="Cancel") is False + assert backend.connection.written == [] + + +def test_set_value_the_control_refuses_reports_failure(backend): + backend.connection.write_result = False + assert backend.set_value("hello", name="OK") is False + + +def test_a_write_that_fails_on_the_bus_reports_failure(backend): + backend.connection.errors[BUTTON] = DBusError("no EditableText") + assert backend.set_value("hello", name="OK") is False + + +def test_invoke_performs_the_first_action(backend): + assert backend.invoke(name="OK") is True + assert backend.connection.actions == [(BUTTON, 0)] + + +def test_invoke_on_a_control_that_is_not_there_reports_failure(backend): + assert backend.invoke(name="Cancel") is False + + +def test_an_invoke_that_fails_on_the_bus_reports_failure(backend): + backend.connection.errors[BUTTON] = DBusError("no Action") + assert backend.invoke(name="OK") is False + + +def test_set_focus_grabs_it_through_the_component_interface(backend): + assert backend.set_focus(name="OK") is True + assert backend.connection.focused == [BUTTON] + + +def test_set_focus_on_a_control_that_is_not_there_reports_failure(backend): + assert backend.set_focus(name="Cancel") is False + + +def test_a_focus_grab_that_fails_on_the_bus_reports_failure(backend): + backend.connection.errors[BUTTON] = DBusError("no Component") + assert backend.set_focus(name="OK") is False + + +def test_get_state_reports_the_three_bits_it_reads(backend): + backend.connection.states[BUTTON] = (1 << 8) | (1 << 12) | (1 << 25) + state = backend.get_state(name="OK") + assert state == {"enabled": True, "focused": True, "selected": True} + + +def test_get_state_reports_false_for_bits_that_are_clear(backend): + backend.connection.states[BUTTON] = 0 + assert backend.get_state(name="OK") == { + "enabled": False, "focused": False, "selected": False, + } + + +def test_get_state_carries_a_value_only_when_the_control_has_one(backend): + # An absent key and an empty value are different answers: the first says + # the control has no such concept, the second that it is empty. + assert "value" not in backend.get_state(name="OK") + backend.connection.texts[BUTTON] = "" + assert backend.get_state(name="OK")["value"] == "" + + +def test_get_state_carries_a_number_only_when_the_control_has_one(backend): + assert "number" not in backend.get_state(name="OK") + backend.connection.numbers[BUTTON] = 0.0 + assert backend.get_state(name="OK")["number"] == 0.0 + + +def test_get_state_of_a_control_that_is_not_there_is_none(backend): + assert backend.get_state(name="Cancel") is None + + +def test_a_control_pattern_needs_a_backend_that_is_available(): + instance = atspi.LinuxAccessibilityBackend.__new__( + atspi.LinuxAccessibilityBackend) + instance.available = False + for call in (lambda: instance.get_value(name="OK"), + lambda: instance.set_value("x", name="OK"), + lambda: instance.invoke(name="OK"), + lambda: instance.set_focus(name="OK"), + lambda: instance.get_state(name="OK")): + with pytest.raises(AccessibilityNotAvailableError): + call() + + +def test_the_search_gives_up_at_a_bounded_depth(monkeypatch): + """A tree that never ends must not recurse until Python gives up.""" + deep = ("app", "/deep") + connection = FakeConnection( + tree={("registry", "/root"): [APP], APP: [deep], deep: [deep]}, + names={APP: "zenity"}, + ) + monkeypatch.setattr(atspi, "_AtspiConnection", lambda: connection) + instance = atspi.LinuxAccessibilityBackend.__new__( + atspi.LinuxAccessibilityBackend) + instance.available = True + assert instance.get_value(name="nothing here") is None + + +def test_a_branch_the_bus_refuses_ends_the_search_there(monkeypatch): + class RefusingConnection(FakeConnection): + def children(self, reference): + if reference == APP: + raise DBusError("BadWindow") + return super().children(reference) + + connection = RefusingConnection( + tree={("registry", "/root"): [APP]}, names={APP: "zenity"}) + monkeypatch.setattr(atspi, "_AtspiConnection", lambda: connection) + instance = atspi.LinuxAccessibilityBackend.__new__( + atspi.LinuxAccessibilityBackend) + instance.available = True + assert instance.get_value(name="OK") is None + + +def test_an_application_whose_name_cannot_be_read_is_still_walked(monkeypatch): + class NamelessConnection(FakeConnection): + def property(self, reference, name, interface=None): + if reference == APP: + raise DBusError("gone") + return super().property(reference, name, interface) + + connection = NamelessConnection( + tree={("registry", "/root"): [APP], APP: [BUTTON]}, + names={BUTTON: "OK"}, roles={BUTTON: "push button"}) + monkeypatch.setattr(atspi, "_AtspiConnection", lambda: connection) + instance = atspi.LinuxAccessibilityBackend.__new__( + atspi.LinuxAccessibilityBackend) + instance.available = True + [element] = instance.list_elements() + assert element.app_name == "" + + +def test_closing_a_connection_twice_is_harmless(): + """`__exit__` runs on the way out of a `with` and again on a retry.""" + connection = atspi._AtspiConnection() + connection.__exit__() + connection.__exit__() + assert connection._bus is None + + +def test_the_walk_stops_asking_further_applications_once_it_is_full( + monkeypatch): + second = ("other", "/app") + connection = FakeConnection( + tree={("registry", "/root"): [APP, second], + APP: [WINDOW], second: [BUTTON]}, + names={APP: "zenity", WINDOW: "dialog", second: "gedit", + BUTTON: "OK"}, + roles={WINDOW: "dialog", BUTTON: "push button"}, + ) + monkeypatch.setattr(atspi, "_AtspiConnection", lambda: connection) + instance = atspi.LinuxAccessibilityBackend.__new__( + atspi.LinuxAccessibilityBackend) + instance.available = True + assert len(instance.list_elements(max_results=1)) == 1 + + +def test_a_branch_the_bus_refuses_ends_that_branch_of_the_walk(monkeypatch): + class RefusingConnection(FakeConnection): + def children(self, reference): + if reference == WINDOW: + raise DBusError("the dialog closed") + return super().children(reference) + + connection = RefusingConnection( + tree={("registry", "/root"): [APP], APP: [WINDOW], WINDOW: [BUTTON]}, + names={APP: "zenity", WINDOW: "dialog", BUTTON: "OK"}, + roles={WINDOW: "dialog", BUTTON: "push button"}, + ) + monkeypatch.setattr(atspi, "_AtspiConnection", lambda: connection) + instance = atspi.LinuxAccessibilityBackend.__new__( + atspi.LinuxAccessibilityBackend) + instance.available = True + # The window itself was listed before its children were asked for. + assert [e.name for e in instance.list_elements()] == ["dialog"] + + +def test_the_walk_stops_mid_application_once_it_is_full(monkeypatch): + siblings = [("app", f"/b{index}") for index in range(4)] + connection = FakeConnection( + tree={("registry", "/root"): [APP], APP: siblings}, + names={APP: "zenity", **{ref: f"b{index}" + for index, ref in enumerate(siblings)}}, + roles={ref: "push button" for ref in siblings}, + ) + monkeypatch.setattr(atspi, "_AtspiConnection", lambda: connection) + instance = atspi.LinuxAccessibilityBackend.__new__( + atspi.LinuxAccessibilityBackend) + instance.available = True + assert len(instance.list_elements(max_results=2)) == 2 diff --git a/test/unit_test/headless/test_accessibility_linux_bus.py b/test/unit_test/headless/test_accessibility_linux_bus.py new file mode 100644 index 00000000..560ea750 --- /dev/null +++ b/test/unit_test/headless/test_accessibility_linux_bus.py @@ -0,0 +1,390 @@ +"""The AT-SPI wire itself: what the backend actually sends down the bus. + +`test_accessibility_linux.py` replaces `_AtspiConnection` with a fake tree, +which is the right shape for testing the *walk* -- and it leaves the whole +D-Bus call layer beneath it unexecuted. That is where the protocol lives, and +the protocol is where the surprises are: + +* **The accessibility bus is not the session bus.** Its address comes from + `org.a11y.Bus.GetAddress` on the session bus, and everything after that + happens on a second connection to the address that call returns. Talking + AT-SPI to the session bus reaches nobody. +* **An accessible is a *pair*.** The bus name of the owning application plus + an object path inside it -- so references are `(sender, path)` tuples all + the way down, and a call addressed with only a path goes to the wrong + process. +* **The state bitfield arrives as two 32-bit words.** AT-SPI does not send + one 64-bit value, so reading only the first word silently drops every + state above bit 31 -- including SELECTED, which this backend reports. +* **Not implementing an interface is not an error.** A plain text node has no + Component and therefore no rectangle; a walk that treated that as a failure + would fail more often than it succeeded. + +`SessionBus` is replaced rather than `_AtspiConnection`: the whole point is +to run the code that builds those calls. What it is replaced by is a +recorder, so each test can state the exact `(destination, path, interface, +member, signature, body)` that went out. + +The bus is also where this backend gets its blast radius: `_is_available()` +opens a connection at import-decision time on every Linux desktop, so its +failure paths matter as much as its success one. +""" +from __future__ import annotations + +import pytest + +from je_auto_control.utils.accessibility.backends import linux_backend as atspi +from je_auto_control.utils.accessibility.backends.linux_backend import ( + _AtspiConnection, LinuxAccessibilityBackend, +) +from je_auto_control.utils.dbus_client import DBusError, Variant + +ROOT = ("org.a11y.atspi.Registry", "/org/a11y/atspi/accessible/root") +APP = ("app.bus.name", "/org/a11y/atspi/accessible/1") +BUTTON = ("app.bus.name", "/org/a11y/atspi/accessible/2") + + +class _Bus: + """A session bus that answers scripted replies and records the calls.""" + + instances = [] + + def __init__(self, address=None) -> None: + self.address = address + self.calls = [] + self.connected = False + self.closed = False + self.replies = {} + self.errors = {} + self.connect_error = None + _Bus.instances.append(self) + + # -- the SessionBus surface -- + def connect(self): + if self.connect_error is not None: + raise self.connect_error + self.connected = True + + def close(self): + self.closed = True + + def __enter__(self): + self.connect() + return self + + def __exit__(self, *_exception): + self.close() + + def call(self, destination, path, interface, member, signature, body, + timeout=25.0): + self.calls.append((destination, path, interface, member, signature, + list(body), timeout)) + key = (path, interface, member) + if key in self.errors: + raise self.errors[key] + if (interface, member) in self.errors: + raise self.errors[(interface, member)] + if key in self.replies: + return self.replies[key] + return self.replies.get((interface, member), []) + + +@pytest.fixture +def bus(monkeypatch): + """One bus for both connections, so a test can script either.""" + _Bus.instances = [] + instance = _Bus() + instance.replies[("org.a11y.Bus", "GetAddress")] = ["unix:path=/a11y"] + monkeypatch.setattr(atspi, "SessionBus", lambda address=None: instance) + yield instance + _Bus.instances = [] + + +@pytest.fixture +def connection(bus): + with _AtspiConnection() as opened: + yield opened + + +def _call(bus, index=-1): + """One recorded call, without the timeout.""" + return bus.calls[index][:6] + + +# --- finding the accessibility bus -------------------------------------------- + +def test_the_address_is_asked_of_the_session_bus(bus): + with _AtspiConnection(): + pass + destination, path, interface, member, signature, body = _call(bus, 0) + assert (destination, path) == ("org.a11y.Bus", "/org/a11y/bus") + assert (interface, member) == ("org.a11y.Bus", "GetAddress") + assert (signature, body) == ("", []) + + +def test_the_connection_opens_on_the_address_it_was_given(monkeypatch): + # A second connection, to a second bus. Reusing the session bus would + # send AT-SPI calls somewhere that does not speak it. + opened = [] + + def _factory(address=None): + instance = _Bus(address) + instance.replies[("org.a11y.Bus", "GetAddress")] = ["unix:path=/a11y"] + opened.append(instance) + return instance + + monkeypatch.setattr(atspi, "SessionBus", _factory) + with _AtspiConnection(): + pass + assert [b.address for b in opened] == [None, "unix:path=/a11y"] + + +def test_the_connection_is_closed_on_the_way_out(bus): + with _AtspiConnection(): + pass + assert bus.closed is True + + +@pytest.mark.parametrize("reply", [[], [42], None]) +def test_an_address_that_is_not_a_string_is_an_error(bus, reply): + bus.replies[("org.a11y.Bus", "GetAddress")] = reply + with pytest.raises(DBusError, match="GetAddress"): + with _AtspiConnection(): + pass + + +def test_using_the_connection_unopened_says_which_mistake_it_was(bus): + # The class is a context manager on purpose; forgetting the `with` is a + # programming error worth naming rather than an AttributeError. + with pytest.raises(DBusError, match="context manager"): + _AtspiConnection().children(ROOT) + + +def test_the_root_is_the_registry(connection): + assert connection.root == ROOT + + +# --- reads -------------------------------------------------------------------- + +def test_children_are_read_as_sender_and_path_pairs(connection, bus): + bus.replies[("org.a11y.atspi.Accessible", "GetChildren")] = [ + [["app.bus.name", "/org/a11y/atspi/accessible/1"], + ["app.bus.name", "/org/a11y/atspi/accessible/2"]], + ] + assert connection.children(ROOT) == [APP, BUTTON] + + +def test_a_child_entry_that_is_not_a_pair_is_dropped(connection, bus): + # A malformed reply is a bug in the application, not a reason to fail the + # whole walk of everything else on the desktop. + bus.replies[("org.a11y.atspi.Accessible", "GetChildren")] = [ + ["not a pair", ["app.bus.name", "/1"], ["only-one"]], + ] + assert connection.children(ROOT) == [("app.bus.name", "/1")] + + +def test_an_empty_reply_reads_as_no_children(connection, bus): + assert connection.children(ROOT) == [] + + +def test_a_call_is_addressed_to_the_owning_application(connection, bus): + connection.children(BUTTON) + destination, path, interface, member, _signature, _body = _call(bus) + assert destination == "app.bus.name", "the sender half of the reference" + assert path == "/org/a11y/atspi/accessible/2" + assert (interface, member) == ("org.a11y.atspi.Accessible", "GetChildren") + + +def test_a_property_is_unwrapped_from_its_variant(connection, bus): + bus.replies[("org.freedesktop.DBus.Properties", "Get")] = [ + Variant("s", "OK"), + ] + assert connection.property(BUTTON, "Name") == "OK" + _d, _p, interface, member, signature, body = _call(bus) + assert (interface, member) == ("org.freedesktop.DBus.Properties", "Get") + assert (signature, body) == ("ss", ["org.a11y.atspi.Accessible", "Name"]) + + +def test_a_property_that_is_not_a_variant_is_passed_through(connection, bus): + bus.replies[("org.freedesktop.DBus.Properties", "Get")] = ["plain"] + assert connection.property(BUTTON, "Name") == "plain" + + +def test_a_property_with_no_reply_reads_as_none(connection, bus): + assert connection.property(BUTTON, "Name") is None + + +def test_a_property_can_be_asked_of_another_interface(connection, bus): + bus.replies[("org.freedesktop.DBus.Properties", "Get")] = [Variant("d", 5)] + connection.property(BUTTON, "CurrentValue", "org.a11y.atspi.Value") + _d, _p, _i, _m, _signature, body = _call(bus) + assert body == ["org.a11y.atspi.Value", "CurrentValue"] + + +def test_the_role_name_is_read_as_text(connection, bus): + bus.replies[("org.a11y.atspi.Accessible", "GetRoleName")] = ["push button"] + assert connection.role_name(BUTTON) == "push button" + + +def test_a_missing_role_name_reads_as_empty(connection, bus): + assert connection.role_name(BUTTON) == "" + + +def test_the_state_is_assembled_from_both_words(connection, bus): + # AT-SPI sends the bitfield as two 32-bit words. Reading only the first + # would drop every state above bit 31 -- SELECTED among them. + bus.replies[("org.a11y.atspi.Accessible", "GetState")] = [[0, 1 << 25]] + assert connection.state(BUTTON) == 1 << (32 + 25) + + +def test_the_low_word_lands_where_the_backend_looks_for_enabled(connection, + bus): + bus.replies[("org.a11y.atspi.Accessible", "GetState")] = [[1 << 8, 0]] + assert connection.state(BUTTON) & (1 << 8) + + +def test_extra_words_beyond_the_first_two_are_ignored(connection, bus): + bus.replies[("org.a11y.atspi.Accessible", "GetState")] = [[0, 0, 0xFF]] + assert connection.state(BUTTON) == 0 + + +@pytest.mark.parametrize("reply", [[], [[]], [None]]) +def test_a_state_reply_with_nothing_in_it_reads_as_zero(connection, bus, + reply): + bus.replies[("org.a11y.atspi.Accessible", "GetState")] = reply + assert connection.state(BUTTON) == 0 + + +def test_extents_are_asked_in_screen_coordinates(connection, bus): + # Screen is the only coordinate space whose numbers mean anything to a + # caller that is about to click them. + bus.replies[("org.a11y.atspi.Component", "GetExtents")] = [[1, 2, 3, 4]] + assert connection.extents(BUTTON) == (1, 2, 3, 4) + _d, _p, _i, _m, signature, body = _call(bus) + assert (signature, body) == ("u", [0]) + + +def test_an_accessible_with_no_component_interface_has_no_rectangle( + connection, bus): + # A plain text node implements no Component; that is not an error. + bus.errors[("org.a11y.atspi.Component", "GetExtents")] = DBusError("no") + assert connection.extents(BUTTON) == (0, 0, 0, 0) + + +@pytest.mark.parametrize("reply", [[], [[1, 2]]]) +def test_a_short_extents_reply_reads_as_the_origin(connection, bus, reply): + bus.replies[("org.a11y.atspi.Component", "GetExtents")] = reply + assert connection.extents(BUTTON) == (0, 0, 0, 0) + + +def test_text_is_read_over_the_whole_range(connection, bus): + bus.replies[("org.a11y.atspi.Text", "GetText")] = ["hello"] + assert connection.text(BUTTON) == "hello" + _d, _p, _i, _m, signature, body = _call(bus) + assert (signature, body) == ("ii", [0, -1]), "0 to -1 is 'all of it'" + + +def test_an_accessible_with_no_text_interface_has_no_text(connection, bus): + bus.errors[("org.a11y.atspi.Text", "GetText")] = DBusError("no Text") + assert connection.text(BUTTON) is None + + +def test_an_empty_text_reply_reads_as_none(connection, bus): + assert connection.text(BUTTON) is None + + +def test_a_numeric_value_is_read_from_the_value_interface(connection, bus): + bus.replies[("org.freedesktop.DBus.Properties", "Get")] = [ + Variant("d", 0.75), + ] + assert connection.number(BUTTON) == 0.75 + + +@pytest.mark.parametrize("reply,error", [ + ([Variant("s", "not a number")], None), + ([None], None), + (None, DBusError("no Value interface")), +]) +def test_a_control_with_no_number_reads_as_none(connection, bus, reply, error): + if error is not None: + bus.errors[("org.freedesktop.DBus.Properties", "Get")] = error + else: + bus.replies[("org.freedesktop.DBus.Properties", "Get")] = reply + assert connection.number(BUTTON) is None + + +# --- writes ------------------------------------------------------------------- + +def test_an_action_is_performed_by_index(connection, bus): + bus.replies[("org.a11y.atspi.Action", "DoAction")] = [True] + assert connection.do_action(BUTTON, 0) is True + _d, _p, interface, member, signature, body = _call(bus) + assert (interface, member) == ("org.a11y.atspi.Action", "DoAction") + assert (signature, body) == ("i", [0]) + + +def test_an_action_the_control_refuses_reports_failure(connection, bus): + bus.replies[("org.a11y.atspi.Action", "DoAction")] = [False] + assert connection.do_action(BUTTON) is False + + +def test_an_action_with_no_reply_reports_failure(connection, bus): + assert connection.do_action(BUTTON) is False + + +def test_text_is_written_through_the_editable_interface(connection, bus): + bus.replies[("org.a11y.atspi.EditableText", "SetTextContents")] = [True] + assert connection.set_text(BUTTON, "typed") is True + _d, _p, interface, member, signature, body = _call(bus) + assert (interface, member) == ("org.a11y.atspi.EditableText", + "SetTextContents") + assert (signature, body) == ("s", ["typed"]) + + +def test_a_write_the_control_refuses_reports_failure(connection, bus): + bus.replies[("org.a11y.atspi.EditableText", "SetTextContents")] = [False] + assert connection.set_text(BUTTON, "typed") is False + + +def test_focus_is_grabbed_through_the_component_interface(connection, bus): + bus.replies[("org.a11y.atspi.Component", "GrabFocus")] = [True] + assert connection.grab_focus(BUTTON) is True + _d, _p, interface, member, _s, _b = _call(bus) + assert (interface, member) == ("org.a11y.atspi.Component", "GrabFocus") + + +def test_a_focus_grab_the_control_refuses_reports_failure(connection, bus): + assert connection.grab_focus(BUTTON) is False + + +# --- availability ------------------------------------------------------------- + +def test_availability_is_decided_by_reaching_the_root(monkeypatch, bus): + monkeypatch.setattr(atspi.os, "name", "posix") + bus.replies[("org.a11y.atspi.Accessible", "GetChildren")] = [[]] + assert atspi._is_available() is True + + +def test_a_desktop_with_no_accessibility_bus_is_unavailable(monkeypatch, bus): + monkeypatch.setattr(atspi.os, "name", "posix") + bus.connect_error = DBusError("connection refused") + assert atspi._is_available() is False + + +def test_a_bus_that_cannot_be_reached_at_all_is_unavailable(monkeypatch, bus): + monkeypatch.setattr(atspi.os, "name", "posix") + bus.connect_error = OSError("no such file or directory") + assert atspi._is_available() is False + + +def test_off_posix_no_bus_is_even_attempted(monkeypatch, bus): + monkeypatch.setattr(atspi.os, "name", "nt") + assert atspi._is_available() is False + assert bus.calls == [], "not one D-Bus round trip on Windows" + + +def test_the_backend_takes_its_availability_from_the_probe(monkeypatch): + monkeypatch.setattr(atspi, "_is_available", lambda: True) + assert LinuxAccessibilityBackend().available is True + monkeypatch.setattr(atspi, "_is_available", lambda: False) + assert LinuxAccessibilityBackend().available is False diff --git a/test/unit_test/headless/test_accessibility_macos.py b/test/unit_test/headless/test_accessibility_macos.py new file mode 100644 index 00000000..871d10c2 --- /dev/null +++ b/test/unit_test/headless/test_accessibility_macos.py @@ -0,0 +1,314 @@ +"""Walking the macOS accessibility tree, from any square rather than Darwin. + +`backends/macos_backend.py` read 0% everywhere. Its pyobjc imports are inside +the methods, so the module loads on any platform and stubs in `sys.modules` +(`_pyobjc_stub.py`) drive the whole walk from all nine CI squares. + +What the walk has to get right: + +* **The default scope is the *active* application.** Enumerating every + running application's whole tree is thousands of nodes for a caller who + asked about the window in front of them, so an inactive application is + skipped -- unless it was named, which is the one case where the caller + clearly meant it. +* **`max_results` bounds the recursion, not just the answer.** The tree is + deep and the AX API is a round trip per node; a walk that collected + everything and sliced afterwards would pay for the whole desktop. +* **One application's AX failure must not end the enumeration.** Accessibility + is granted per process and revoked at any time, and an application that + refuses is a normal event, not a reason to return nothing. +* **A node with neither role nor title is not an element.** AX reports + structural nodes with nothing on them; carrying them would fill the answer + with rows a caller cannot match on. +""" +from __future__ import annotations + +import types + +import pytest + +from headless import _pyobjc_stub as objc_stub +from headless._pyobjc_stub import AX_FAILURE, AXElement +from je_auto_control.utils.accessibility.backends import macos_backend as mac +from je_auto_control.utils.accessibility.backends.macos_backend import ( + MacOSAccessibilityBackend, _extract_bounds, +) +from je_auto_control.utils.accessibility.element import ( + AccessibilityNotAvailableError, +) + + +class _App: + """One entry of `NSWorkspace.runningApplications()`.""" + + def __init__(self, name: str, pid: int, active: bool = True) -> None: + self._name = name + self._pid = pid + self._active = active + + def isActive(self): # noqa: N802 # reason: the AppKit name + return self._active + + def localizedName(self): # noqa: N802 # reason: the AppKit name + return self._name + + def processIdentifier(self): # noqa: N802 # reason: the AppKit name + return self._pid + + +def _element(role=None, title=None, position=None, size=None, children=None): + attributes = {} + if role is not None: + attributes["AXRole"] = role + if title is not None: + attributes["AXTitle"] = title + if position is not None: + attributes["AXPosition"] = position + if size is not None: + attributes["AXSize"] = size + if children is not None: + attributes["AXChildren"] = children + return AXElement(**attributes) + + +class _World(objc_stub.World): + """The AT-SPI-free half of the pyobjc stub: apps, and a tree per pid.""" + + def __init__(self, apps=None, trees=None) -> None: + super().__init__() + self.apps = list(apps or []) + self.trees = dict(trees or {}) # pid -> root AXElement + self.walk_errors = {} # pid -> exception + + def ax_application(self, pid): + if pid in self.walk_errors: + raise self.walk_errors[pid] + return self.trees.get(int(pid), AXElement()) + + def ax_copy_attribute(self, element, attribute, placeholder): + # An AX read can raise rather than answer with an error code: the + # pyobjc bridge turns some failures into exceptions. + error = getattr(element, "raises", None) + if error is not None: + raise error + return super().ax_copy_attribute(element, attribute, placeholder) + + +@pytest.fixture +def install(monkeypatch): + def _install(world: _World) -> MacOSAccessibilityBackend: + objc_stub.install(monkeypatch, world) + appkit = types.ModuleType("AppKit") + appkit.NSWorkspace = types.SimpleNamespace( + sharedWorkspace=lambda: types.SimpleNamespace( + runningApplications=lambda: list(world.apps))) + monkeypatch.setitem(__import__("sys").modules, "AppKit", appkit) + backend = MacOSAccessibilityBackend() + assert backend.available + return backend + return _install + + +# --- availability ------------------------------------------------------------- + +def test_a_mac_without_pyobjc_refuses_and_names_it(monkeypatch): + monkeypatch.setattr(mac, "_is_available", lambda: False) + backend = MacOSAccessibilityBackend() + with pytest.raises(AccessibilityNotAvailableError, match="pyobjc"): + backend.list_elements() + + +def test_the_probe_reports_what_it_could_import(monkeypatch): + import sys + monkeypatch.setitem(sys.modules, "ApplicationServices", + types.ModuleType("ApplicationServices")) + monkeypatch.setitem(sys.modules, "AppKit", types.ModuleType("AppKit")) + assert mac._is_available() is True + monkeypatch.setitem(sys.modules, "ApplicationServices", None) + assert mac._is_available() is False + + +def test_the_backend_names_the_api_it_walks(install): + assert install(_World()).name == "macos-ax" + + +# --- scoping ------------------------------------------------------------------ + +def test_only_the_active_application_is_walked_by_default(install): + world = _World( + apps=[_App("Safari", 1, active=True), _App("Mail", 2, active=False)], + trees={1: _element(role="AXWindow", title="Safari window"), + 2: _element(role="AXWindow", title="Mail window")}, + ) + names = [e.name for e in install(world).list_elements()] + assert names == ["Safari window"] + + +def test_naming_an_application_reaches_it_even_when_it_is_not_active(install): + world = _World( + apps=[_App("Safari", 1, active=True), _App("Mail", 2, active=False)], + trees={1: _element(role="AXWindow", title="Safari window"), + 2: _element(role="AXWindow", title="Mail window")}, + ) + elements = install(world).list_elements(app_name="Mail") + assert [e.name for e in elements] == ["Mail window"] + assert elements[0].app_name == "Mail" + + +def test_naming_an_application_that_is_not_running_finds_nothing(install): + world = _World(apps=[_App("Safari", 1)], + trees={1: _element(role="AXWindow", title="w")}) + assert install(world).list_elements(app_name="Mail") == [] + + +def test_an_application_with_no_localized_name_is_still_walked(install): + world = _World(apps=[_App(None, 1)], + trees={1: _element(role="AXWindow", title="untitled")}) + [element] = install(world).list_elements() + assert element.app_name == "" + + +def test_the_owning_process_is_carried_on_every_element(install): + world = _World(apps=[_App("Safari", 501)], + trees={501: _element(role="AXWindow", title="w")}) + [element] = install(world).list_elements() + assert element.process_id == 501 + + +# --- the walk ----------------------------------------------------------------- + +def test_the_walk_is_depth_first_through_the_children(install): + leaf = _element(role="AXButton", title="OK") + group = _element(role="AXGroup", title="Buttons", children=[leaf]) + root = _element(role="AXWindow", title="Dialog", children=[group]) + world = _World(apps=[_App("Safari", 1)], trees={1: root}) + names = [e.name for e in install(world).list_elements()] + assert names == ["Dialog", "Buttons", "OK"] + + +def test_a_node_with_neither_role_nor_title_is_not_an_element(install): + structural = _element(children=[_element(role="AXButton", title="OK")]) + world = _World(apps=[_App("Safari", 1)], trees={1: structural}) + names = [e.name for e in install(world).list_elements()] + assert names == ["OK"], "the structural node is walked through, not listed" + + +def test_a_node_with_only_a_role_is_still_an_element(install): + world = _World(apps=[_App("Safari", 1)], + trees={1: _element(role="AXWindow")}) + [element] = install(world).list_elements() + assert (element.role, element.name) == ("AXWindow", "") + + +def test_the_walk_stops_at_max_results(install): + children = [_element(role="AXButton", title=f"b{index}") + for index in range(10)] + root = _element(role="AXWindow", title="Dialog", children=children) + world = _World(apps=[_App("Safari", 1)], trees={1: root}) + assert len(install(world).list_elements(max_results=3)) == 3 + + +def test_max_results_bounds_the_recursion_not_just_the_answer(install): + # Every node is a round trip; collecting the desktop and slicing after + # would pay for all of it. + deep = _element(role="AXButton", title="deep") + nested = _element(role="AXGroup", title="g", children=[deep]) + root = _element(role="AXWindow", title="Dialog", children=[nested]) + world = _World(apps=[_App("Safari", 1)], trees={1: root}) + install(world).list_elements(max_results=1) + assert "AXChildren" not in nested.attributes or not nested.actions + + +def test_the_walk_stops_asking_further_applications_once_it_is_full(install): + world = _World( + apps=[_App("Safari", 1), _App("Mail", 2)], + trees={1: _element(role="AXWindow", title="one"), + 2: _element(role="AXWindow", title="two")}, + ) + assert len(install(world).list_elements(max_results=1)) == 1 + + +def test_a_node_that_refuses_to_list_children_ends_that_branch(install): + root = _element(role="AXWindow", title="Dialog") + root.read_error = AX_FAILURE + world = _World(apps=[_App("Safari", 1)], trees={1: root}) + # The node itself cannot be described either, so nothing comes back -- + # and, crucially, no exception does. + assert install(world).list_elements() == [] + + +def test_a_node_whose_read_raises_is_skipped_rather_than_fatal(install): + # pyobjc turns some AX failures into exceptions rather than error codes, + # and a control that vanished mid-walk is an ordinary event. + root = _element(role="AXWindow", title="Dialog") + root.raises = RuntimeError("AXError -25202") + world = _World(apps=[_App("Safari", 1)], trees={1: root}) + assert install(world).list_elements() == [] + + +def test_the_recursion_guard_holds_even_when_entered_full(install): + # The loop checks before it recurses, so this guard is only reachable by + # entering the walk with a full list -- which is what makes the helper + # safe to call from anywhere, including a future second caller. + world = _World(apps=[_App("Safari", 1)], + trees={1: _element(role="AXWindow", title="w")}) + backend = install(world) + import ApplicationServices as ax_module + results = ["already full"] + backend._walk(ax_module, _element(role="AXWindow", title="w"), + "Safari", 1, results, max_results=1) + assert results == ["already full"] + + +def test_one_application_failing_does_not_end_the_enumeration(install): + world = _World( + apps=[_App("Broken", 1), _App("Safari", 2)], + trees={2: _element(role="AXWindow", title="works")}, + ) + world.walk_errors[1] = RuntimeError("AXError -25211") + names = [e.name for e in install(world).list_elements(app_name=None)] + assert names == ["works"] + + +def test_a_window_title_scope_is_accepted_and_ignored(install): + # AX walks per application already; returning nothing because the scope + # cannot be honoured would be worse than returning the unscoped answer. + world = _World(apps=[_App("Safari", 1)], + trees={1: _element(role="AXWindow", title="Dialog")}) + elements = install(world).list_elements(window_title="something else") + assert [e.name for e in elements] == ["Dialog"] + + +# --- geometry ----------------------------------------------------------------- + +def test_an_elements_bounds_come_from_its_position_and_size(install): + world = _World(apps=[_App("Safari", 1)], trees={ + 1: _element(role="AXWindow", title="w", position=(10, 20), + size=(300, 400)), + }) + [element] = install(world).list_elements() + assert element.bounds == (10, 20, 300, 400) + + +@pytest.mark.parametrize("position,size", [ + (None, (1, 2)), ((1, 2), None), (None, None), +]) +def test_an_element_with_no_geometry_reads_as_the_origin(position, size): + # A control AX will not place is still worth reporting: the caller can + # match on its name even if it cannot click it. + assert _extract_bounds(position, size) == (0, 0, 0, 0) + + +@pytest.mark.parametrize("position,size", [ + ("not a pair", (1, 2)), + ((1, 2), "not a pair"), + ((1,), (1, 2)), + (("x", "y"), (1, 2)), +]) +def test_geometry_that_cannot_be_unpacked_reads_as_the_origin(position, size): + assert _extract_bounds(position, size) == (0, 0, 0, 0) + + +def test_geometry_is_narrowed_to_whole_pixels(): + # AX reports floats; bounds are pixels a caller is about to click. + assert _extract_bounds((1.7, 2.2), (3.9, 4.1)) == (1, 2, 3, 4) diff --git a/test/unit_test/headless/test_accessibility_windows_content.py b/test/unit_test/headless/test_accessibility_windows_content.py new file mode 100644 index 00000000..ee444d43 --- /dev/null +++ b/test/unit_test/headless/test_accessibility_windows_content.py @@ -0,0 +1,731 @@ +"""Reading structured content out of a Windows control. + +Split from `test_accessibility_windows_patterns.py`, which covers acting on a +control; this half is the reads that come back as more than one value -- a +table, a selection, a set of views, the MSAA bridge's fields, a virtualized +row, and the text patterns. + +What they have in common is a second layer of indirection. A grid hands back +a cell which carries its *own* pattern; a selection hands back an element +array which has to be walked by index; a text pattern hands back ranges. Each +of those is another cross-process object that can fail on its own, and the +answer for every one of them is the same: report what could be read, and say +"no" in the type the caller was promised rather than raising out of the +middle of a listing. + +The virtualized-item path is the one worth reading twice. A row 500 places +down a list does not exist as an element until it is realized, so finding it +and realizing it are two calls and skipping the second hands the caller +something that is not there yet. +""" +from __future__ import annotations + +import sys +import types + +import pytest + +from headless._uia_doubles import ( + Automation, ElementArray, Pattern, RawElement, Rect, UiaModule, Unknown, + install_comtypes, +) +from je_auto_control.utils.accessibility.backends import ( + windows_backend as backend_module, +) +from je_auto_control.utils.accessibility.backends.windows_backend import ( + WindowsAccessibilityBackend, _header_names, _read_cell, _read_legacy, + _read_text_attributes, _view_name, +) + +VALUE = backend_module._UIA_VALUE_PATTERN_ID +INVOKE = backend_module._UIA_INVOKE_PATTERN_ID +TOGGLE = backend_module._UIA_TOGGLE_PATTERN_ID +GRID = backend_module._UIA_GRID_PATTERN_ID +GRID_ITEM = backend_module._UIA_GRIDITEM_PATTERN_ID +EXPAND = backend_module._UIA_EXPANDCOLLAPSE_PATTERN_ID +SELECTION_ITEM = backend_module._UIA_SELECTIONITEM_PATTERN_ID +RANGE = backend_module._UIA_RANGEVALUE_PATTERN_ID +SCROLL_ITEM = backend_module._UIA_SCROLLITEM_PATTERN_ID +TEXT = backend_module._UIA_TEXT_PATTERN_ID +ITEM_CONTAINER = backend_module._UIA_ITEMCONTAINER_PATTERN_ID +VIRTUALIZED = backend_module._UIA_VIRTUALIZEDITEM_PATTERN_ID +TABLE = backend_module._UIA_TABLE_PATTERN_ID +TRANSFORM = backend_module._UIA_TRANSFORM_PATTERN_ID +WINDOW = backend_module._UIA_WINDOW_PATTERN_ID +LEGACY = backend_module._UIA_LEGACYIACCESSIBLE_PATTERN_ID +SELECTION = backend_module._UIA_SELECTION_PATTERN_ID +MULTIPLE_VIEW = backend_module._UIA_MULTIPLEVIEW_PATTERN_ID + +_IS_PASSWORD = 30019 + + +@pytest.fixture(autouse=True) +def named_processes(monkeypatch): + monkeypatch.setattr(backend_module, "_process_name", + lambda pid: f"app{pid}.exe" if pid else "") + + +@pytest.fixture +def backend(monkeypatch): + monkeypatch.setattr(backend_module, "_is_available", lambda: True) + instance = WindowsAccessibilityBackend() + instance._automation = Automation() + instance._uia_module = UiaModule() + return instance + + +@pytest.fixture +def found(backend, monkeypatch): + """Answer every search with one raw element, and record what was asked.""" + state = {"raw": None, "filters": []} + + def _find_raw(name, role, app_name, automation_id, window_title=None, + contains=False): + state["filters"].append({"name": name, "role": role, + "app_name": app_name, + "automation_id": automation_id, + "window_title": window_title, + "contains": contains}) + return state["raw"] + + monkeypatch.setattr(backend, "_find_raw", _find_raw) + + def _set(raw): + state["raw"] = raw + return raw + + state["set"] = _set + return state + + +def _control(pattern_id=None, interface_name="", pattern=None, **kwargs): + """A raw element carrying at most one pattern.""" + patterns = {} + if pattern_id is not None: + patterns[pattern_id] = Unknown(interface_name, pattern or Pattern()) + kwargs.setdefault("rect", Rect(0, 0, 1, 1)) + return RawElement(patterns=patterns, **kwargs) + + +def _with(found, pattern_id, interface_name, pattern=None, **kwargs): + """Install a control carrying one pattern, and hand the pattern back.""" + pattern = pattern or Pattern() + found["set"](_control(pattern_id, interface_name, pattern, **kwargs)) + return pattern + + +# --- tables and grids --------------------------------------------------------- + +def _grid(rows, cols, cells=None, item_error=None): + lookup = dict(cells or {}) + + def _get_item(row, column): + if item_error is not None: + raise item_error + return lookup.get((row, column)) + + pattern = Pattern(CurrentRowCount=rows, CurrentColumnCount=cols) + pattern.GetItem = _get_item + return pattern + + +def test_a_table_is_read_row_by_row(backend, found): + cells = {(0, 0): types.SimpleNamespace(CurrentName="a"), + (0, 1): types.SimpleNamespace(CurrentName="b"), + (1, 0): types.SimpleNamespace(CurrentName="c"), + (1, 1): types.SimpleNamespace(CurrentName="d")} + _with(found, GRID, "IUIAutomationGridPattern", _grid(2, 2, cells)) + assert backend.read_table(name="Grid") == [["a", "b"], ["c", "d"]] + + +def test_a_cell_that_is_not_there_reads_as_empty(backend, found): + _with(found, GRID, "IUIAutomationGridPattern", _grid(1, 2, {})) + assert backend.read_table(name="Grid") == [["", ""]] + + +def test_a_cell_the_provider_refuses_reads_as_empty(backend, found): + _with(found, GRID, "IUIAutomationGridPattern", + _grid(1, 1, item_error=OSError("gone"))) + assert backend.read_table(name="Grid") == [[""]] + + +def test_a_table_whose_size_cannot_be_read_is_empty(backend, found): + _with(found, GRID, "IUIAutomationGridPattern", Pattern()) + assert backend.read_table(name="Grid") == [] + + +def test_a_control_that_is_not_a_grid_reads_as_no_table(backend, found): + found["set"](_control()) + assert backend.read_table(name="Grid") == [] + + +def test_a_control_that_is_not_there_reads_as_no_table(backend, found): + assert backend.read_table(name="Grid") == [] + + +def test_table_headers_are_read_from_both_axes(backend, found): + columns = ElementArray([types.SimpleNamespace(CurrentName="Name"), + types.SimpleNamespace(CurrentName="Size")]) + rows = ElementArray([types.SimpleNamespace(CurrentName="1")]) + pattern = Pattern() + pattern.GetCurrentColumnHeaders = lambda: columns + pattern.GetCurrentRowHeaders = lambda: rows + _with(found, TABLE, "IUIAutomationTablePattern", pattern) + assert backend.get_table_headers(name="Grid") == { + "columns": ["Name", "Size"], "rows": ["1"], + } + + +def test_headers_the_provider_refuses_read_as_none(backend, found): + pattern = Pattern() + pattern.errors["GetCurrentColumnHeaders"] = OSError("gone") + _with(found, TABLE, "IUIAutomationTablePattern", pattern) + assert backend.get_table_headers(name="Grid") is None + + +def test_a_control_with_no_table_pattern_has_no_headers(backend, found): + found["set"](_control()) + assert backend.get_table_headers(name="Grid") is None + + +def test_an_array_whose_length_cannot_be_read_is_empty(): + assert _header_names(ElementArray(length_error=OSError("gone"))) == [] + + +def test_an_array_entry_that_cannot_be_read_reads_as_empty(): + array = ElementArray([types.SimpleNamespace(CurrentName="a")], + element_error=OSError("gone")) + assert _header_names(array) == [""] + + +def test_a_grid_cell_carries_its_own_coordinates(backend, found): + cell = types.SimpleNamespace(CurrentName="value") + item = Unknown("IUIAutomationGridItemPattern", + Pattern(CurrentRow=3, CurrentColumn=4, CurrentRowSpan=2, + CurrentColumnSpan=1)) + cell_element = RawElement(patterns={GRID_ITEM: item}) + cell_element.CurrentName = "value" + del cell + _with(found, GRID, "IUIAutomationGridPattern", + _grid(1, 1, {(0, 0): cell_element})) + assert backend.get_grid_cell(0, 0, name="Grid") == { + "value": "value", "row": 3, "column": 4, + "row_span": 2, "column_span": 1, + } + + +def test_a_grid_cell_without_an_item_pattern_reports_where_it_was_asked_for(): + cell = types.SimpleNamespace(CurrentName="value") + assert _read_cell(None, cell, 1, 2) == { + "value": "value", "row": 1, "column": 2, + "row_span": 1, "column_span": 1, + } + + +def test_a_grid_cell_whose_span_cannot_be_read_keeps_the_default(): + cell = types.SimpleNamespace(CurrentName="value") + assert _read_cell(Pattern(CurrentRow=1), cell, 0, 0)["row_span"] == 1 + + +def test_a_grid_cell_that_is_not_there_is_none(backend, found): + _with(found, GRID, "IUIAutomationGridPattern", _grid(1, 1, {})) + assert backend.get_grid_cell(0, 0, name="Grid") is None + + +def test_a_grid_cell_the_provider_refuses_is_none(backend, found): + _with(found, GRID, "IUIAutomationGridPattern", + _grid(1, 1, item_error=OSError("gone"))) + assert backend.get_grid_cell(0, 0, name="Grid") is None + + +def test_a_control_that_is_not_a_grid_has_no_cell(backend, found): + found["set"](_control()) + assert backend.get_grid_cell(0, 0, name="Grid") is None + + +# --- selection and views ------------------------------------------------------ + +def test_a_selection_reports_its_items_and_its_rules(backend, found): + pattern = Pattern(CurrentCanSelectMultiple=True, + CurrentIsSelectionRequired=False) + pattern.GetCurrentSelection = lambda: ElementArray( + [types.SimpleNamespace(CurrentName="one")]) + _with(found, SELECTION, "IUIAutomationSelectionPattern", pattern) + assert backend.get_selection(name="List") == { + "items": ["one"], "can_select_multiple": True, "is_required": False, + } + + +def test_a_selection_the_provider_refuses_is_none(backend, found): + pattern = Pattern() + pattern.errors["GetCurrentSelection"] = OSError("gone") + _with(found, SELECTION, "IUIAutomationSelectionPattern", pattern) + assert backend.get_selection(name="List") is None + + +def test_a_control_with_no_selection_pattern_has_no_selection(backend, found): + found["set"](_control()) + assert backend.get_selection(name="List") is None + + +def _views(names, current=0): + pattern = Pattern(CurrentCurrentView=current) + pattern.GetCurrentSupportedViews = lambda: list(range(len(names))) + pattern.GetViewName = lambda view_id: names[int(view_id)] + return pattern + + +def test_the_views_of_a_control_are_listed_by_name(backend, found): + _with(found, MULTIPLE_VIEW, "IUIAutomationMultipleViewPattern", + _views(["Icons", "Details"], current=1)) + assert backend.list_views(name="Files") == { + "current": "Details", "views": ["Icons", "Details"], + } + + +def test_views_the_provider_refuses_read_as_none(backend, found): + pattern = Pattern(CurrentCurrentView=0) + pattern.errors["GetCurrentSupportedViews"] = OSError("gone") + _with(found, MULTIPLE_VIEW, "IUIAutomationMultipleViewPattern", pattern) + assert backend.list_views(name="Files") is None + + +def test_a_control_with_no_views_has_none(backend, found): + found["set"](_control()) + assert backend.list_views(name="Files") is None + + +def test_a_view_name_that_cannot_be_read_is_empty(): + assert _view_name(Pattern(), "not a number") == "" + + +def test_setting_a_view_matches_it_by_name(backend, found): + pattern = _views(["Icons", "Details"]) + _with(found, MULTIPLE_VIEW, "IUIAutomationMultipleViewPattern", pattern) + assert backend.set_view("Details", name="Files") is True + assert ("SetCurrentView", (1,)) in pattern.calls + + +def test_setting_a_view_that_does_not_exist_fails(backend, found): + _with(found, MULTIPLE_VIEW, "IUIAutomationMultipleViewPattern", + _views(["Icons"])) + assert backend.set_view("Details", name="Files") is False + + +def test_setting_a_view_the_provider_refuses_fails(backend, found): + pattern = Pattern() + pattern.errors["GetCurrentSupportedViews"] = OSError("gone") + _with(found, MULTIPLE_VIEW, "IUIAutomationMultipleViewPattern", pattern) + assert backend.set_view("Details", name="Files") is False + + +def test_setting_a_view_on_a_control_with_none_fails(backend, found): + found["set"](_control()) + assert backend.set_view("Details", name="Files") is False + + +# --- the MSAA bridge ---------------------------------------------------------- + +def test_the_legacy_fields_are_read_into_plain_values(backend, found): + _with(found, LEGACY, "IUIAutomationLegacyIAccessiblePattern", + Pattern(CurrentName="OK", CurrentValue="", CurrentDescription="d", + CurrentDefaultAction="Press", CurrentRole=43, + CurrentState=1048576)) + assert backend.legacy_info(name="OK") == { + "name": "OK", "value": "", "description": "d", + "default_action": "Press", "role": 43, "state": 1048576, + } + + +def test_a_legacy_field_the_provider_will_not_answer_reads_as_none(): + assert _read_legacy(Pattern())["name"] is None + + +def test_a_control_with_no_legacy_bridge_has_no_legacy_info(backend, found): + found["set"](_control()) + assert backend.legacy_info(name="OK") is None + + +def test_the_legacy_default_action_is_performed(backend, found): + pattern = _with(found, LEGACY, "IUIAutomationLegacyIAccessiblePattern") + assert backend.legacy_default_action(name="OK") is True + assert pattern.calls == [("DoDefaultAction", ())] + + +# --- virtualized items -------------------------------------------------------- + +def _container(item, by_property=None, find_error=None): + pattern = Pattern() + + def _find(scope, property_id, value): + if find_error is not None: + raise find_error + if by_property is not None and property_id != by_property: + return None + return item + + pattern.FindItemByProperty = _find + return pattern + + +def test_a_virtual_item_is_found_realized_and_converted(backend, found): + realize = Pattern() + item = RawElement(name="Row 500", rect=Rect(0, 0, 10, 10), + patterns={VIRTUALIZED: Unknown( + "IUIAutomationVirtualizedItemPattern", realize)}) + _with(found, ITEM_CONTAINER, "IUIAutomationItemContainerPattern", + _container(item)) + element = backend.find_virtual_item("Row 500", container_name="List") + assert element.name == "Row 500" + assert realize.calls == [("Realize", ())], ( + "a virtualized row is not a real element until it is realized" + ) + + +def test_a_virtual_item_can_be_looked_up_by_automation_id(backend, found): + item = RawElement(name="Row", rect=Rect(0, 0, 1, 1)) + _with(found, ITEM_CONTAINER, "IUIAutomationItemContainerPattern", + _container(item, by_property=backend_module._UIA_AUTOMATIONID_PROPERTY)) + assert backend.find_virtual_item("row-500", by="automation_id", + container_name="List") is not None + assert backend.find_virtual_item("row-500", container_name="List") is None + + +def test_a_virtual_item_that_is_not_in_the_container_is_none(backend, found): + _with(found, ITEM_CONTAINER, "IUIAutomationItemContainerPattern", + _container(None)) + assert backend.find_virtual_item("Row 500", container_name="List") is None + + +def test_a_container_that_refuses_the_lookup_answers_none(backend, found): + _with(found, ITEM_CONTAINER, "IUIAutomationItemContainerPattern", + _container(None, find_error=OSError("gone"))) + assert backend.find_virtual_item("Row 500", container_name="List") is None + + +def test_a_container_with_no_item_container_pattern_answers_none(backend, + found): + found["set"](_control()) + assert backend.find_virtual_item("Row 500", container_name="List") is None + + +def test_a_container_that_is_not_there_answers_none(backend, found): + assert backend.find_virtual_item("Row 500", container_name="List") is None + + +def test_an_item_that_cannot_be_realized_is_still_returned(backend, found): + realize = Pattern() + realize.errors["Realize"] = OSError("already real") + item = RawElement(name="Row", rect=Rect(0, 0, 1, 1), + patterns={VIRTUALIZED: Unknown( + "IUIAutomationVirtualizedItemPattern", realize)}) + _with(found, ITEM_CONTAINER, "IUIAutomationItemContainerPattern", + _container(item)) + assert backend.find_virtual_item("Row", container_name="List") is not None + + +def test_an_item_with_no_virtualized_pattern_is_returned_as_is(backend, + found): + item = RawElement(name="Row", rect=Rect(0, 0, 1, 1)) + _with(found, ITEM_CONTAINER, "IUIAutomationItemContainerPattern", + _container(item)) + assert backend.find_virtual_item("Row", container_name="List").name == "Row" + + +# --- text --------------------------------------------------------------------- + +class _TextRange: + def __init__(self, text="", attributes=None) -> None: + self._text = text + self._attributes = dict(attributes or {}) + self.selected = 0 + self.attribute_error = None + + def GetText(self, length): # noqa: N802 # reason: the UIA name + return self._text + + def GetAttributeValue(self, attribute_id): # noqa: N802 # UIA name + if self.attribute_error is not None: + raise self.attribute_error + return self._attributes[attribute_id] + + def Select(self): # noqa: N802 # reason: the UIA name + self.selected += 1 + + +def _text_pattern(document="", selection=None, visible=None, + find_result="missing"): + pattern = Pattern(DocumentRange=_TextRange(document)) + if find_result != "missing": + pattern.DocumentRange.FindText = lambda *args: find_result + else: + pattern.DocumentRange.FindText = lambda *args: None + pattern.GetSelection = lambda: selection + pattern.GetVisibleRanges = lambda: visible + return pattern + + +def test_the_document_text_is_read_whole(backend, found): + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(document="the whole document")) + assert backend.document_text(name="Editor") == "the whole document" + + +def test_a_control_with_no_text_pattern_has_no_document(backend, found): + found["set"](_control()) + assert backend.document_text(name="Editor") is None + + +def test_a_control_that_is_not_there_has_no_document(backend, found): + assert backend.document_text(name="Editor") is None + + +def test_a_document_the_provider_refuses_reads_as_none(backend, found): + pattern = Pattern() + _with(found, TEXT, "IUIAutomationTextPattern", pattern) + assert backend.document_text(name="Editor") is None + + +def test_the_selected_text_comes_from_the_first_selected_range(backend, + found): + selection = ElementArray([_TextRange("chosen")]) + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(selection=selection)) + assert backend.selected_text(name="Editor") == "chosen" + + +def test_an_empty_selection_reads_as_the_empty_string(backend, found): + # "" and None are different answers: nothing is selected, against there + # being no text control at all. + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(selection=ElementArray([]))) + assert backend.selected_text(name="Editor") == "" + + +def test_a_selection_the_provider_refuses_reads_as_none(backend, found): + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(selection=ElementArray(length_error=OSError("gone")))) + assert backend.selected_text(name="Editor") is None + + +@pytest.mark.parametrize("method", ["selected_text", "visible_text", + "text_attributes"]) +def test_a_control_with_no_text_pattern_has_no_text_of_any_kind(backend, + found, + method): + found["set"](_control()) + assert getattr(backend, method)(name="Editor") is None + + +def test_the_visible_text_is_the_visible_ranges_joined(backend, found): + visible = ElementArray([_TextRange("first "), _TextRange("second")]) + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(visible=visible)) + assert backend.visible_text(name="Editor") == "first second" + + +def test_visible_text_the_provider_refuses_reads_as_none(backend, found): + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(visible=ElementArray(length_error=OSError("gone")))) + assert backend.visible_text(name="Editor") is None + + +def test_finding_text_reports_whether_a_range_came_back(backend, found): + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(find_result=_TextRange("found"))) + assert backend.find_text("needle", name="Editor") is True + + +def test_finding_text_that_is_not_there_reports_false(backend, found): + _with(found, TEXT, "IUIAutomationTextPattern", _text_pattern()) + assert backend.find_text("needle", name="Editor") is False + + +def test_finding_text_in_a_control_that_is_not_there_reports_false(backend, + found): + assert backend.find_text("needle", name="Editor") is False + + +def test_selecting_text_selects_the_range_it_found(backend, found): + target = _TextRange("found") + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(find_result=target)) + assert backend.select_text("needle", name="Editor") is True + assert target.selected == 1 + + +def test_selecting_text_that_is_not_there_reports_false(backend, found): + _with(found, TEXT, "IUIAutomationTextPattern", _text_pattern()) + assert backend.select_text("needle", name="Editor") is False + + +def test_a_selection_the_control_refuses_reports_false(backend, found): + target = _TextRange("found") + + def _boom(): + raise OSError("read-only") + + target.Select = _boom + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(find_result=target)) + assert backend.select_text("needle", name="Editor") is False + + +def test_a_find_the_provider_refuses_reports_false(backend, found): + pattern = Pattern(DocumentRange=_TextRange()) + + def _boom(*_args): + raise OSError("gone") + + pattern.DocumentRange.FindText = _boom + _with(found, TEXT, "IUIAutomationTextPattern", pattern) + assert backend.find_text("needle", name="Editor") is False + + +_ATTRS = {40005: "Consolas", 40006: 12.0, 40007: 700, 40008: 255, 40014: True} + + +def test_text_attributes_come_from_the_selection_when_there_is_one(backend, + found): + selected = _TextRange("chosen", attributes=_ATTRS) + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(selection=ElementArray([selected]))) + assert backend.text_attributes(name="Editor")["font_name"] == "Consolas" + + +def test_text_attributes_fall_back_to_the_whole_document(backend, found): + pattern = _text_pattern(document="all of it", + selection=ElementArray([])) + pattern.DocumentRange._attributes = dict(_ATTRS) + _with(found, TEXT, "IUIAutomationTextPattern", pattern) + assert backend.text_attributes(name="Editor")["font_size"] == 12.0 + + +def test_a_font_weight_of_seven_hundred_is_bold(): + assert _read_text_attributes(_TextRange(attributes=_ATTRS))["bold"] is True + + +def test_a_lighter_font_weight_is_not_bold(): + attributes = {**_ATTRS, 40007: 400} + assert _read_text_attributes( + _TextRange(attributes=attributes))["bold"] is False + + +def test_an_unreadable_weight_leaves_boldness_unknown(): + # None is not False: "the control did not say" is a different answer from + # "it is not bold", and a caller may want to ask again. + text_range = _TextRange(attributes={}) + text_range.attribute_error = OSError("gone") + assert _read_text_attributes(text_range)["bold"] is None + + +def test_a_selection_that_cannot_be_read_leaves_attributes_none(backend, + found): + _with(found, TEXT, "IUIAutomationTextPattern", + _text_pattern(selection=ElementArray(length_error=OSError("gone")))) + assert backend.text_attributes(name="Editor") is None + + +def test_a_control_with_no_text_pattern_has_no_attributes(backend, found): + found["set"](_control()) + assert backend.text_attributes(name="Editor") is None + + +# --- focus -------------------------------------------------------------------- + +def test_focusing_a_control_calls_set_focus_on_it(backend, found): + raw = found["set"](_control()) + assert backend.set_focus(name="Field") is True + assert raw.focused == 1 + + +def test_focusing_a_control_that_is_not_there_fails(backend, found): + assert backend.set_focus(name="Field") is False + + +def test_a_focus_the_control_refuses_fails(backend, found): + raw = found["set"](_control()) + raw.focus_error = OSError("cannot focus") + assert backend.set_focus(name="Field") is False + + +def test_waiting_for_focus_registers_and_removes_its_handler(backend, + monkeypatch): + install_comtypes(monkeypatch) + automation = backend._automation + result = backend.wait_for_focus_change(timeout=0.01) + assert result is None, "nothing focused inside the timeout" + assert automation.focus_handlers == [], "the handler was removed again" + + +def test_waiting_for_focus_reports_the_element_that_took_it(backend, + monkeypatch): + install_comtypes(monkeypatch) + captured = {} + + def _handler_factory(sink): + captured["sink"] = sink + sink.put({"name": "Field"}) + return object() + + monkeypatch.setattr(backend, "_make_focus_handler", _handler_factory) + assert backend.wait_for_focus_change(timeout=1.0) == {"name": "Field"} + + +def test_waiting_for_focus_without_comtypes_answers_none(backend, + monkeypatch): + monkeypatch.setattr(backend, "_make_focus_handler", lambda sink: None) + assert backend.wait_for_focus_change(timeout=0.01) is None + + +def test_a_provider_that_refuses_the_subscription_answers_none(backend, + monkeypatch): + install_comtypes(monkeypatch) + backend._automation.add_error = OSError("cannot subscribe") + assert backend.wait_for_focus_change(timeout=0.01) is None + + +def test_a_handler_that_cannot_be_removed_does_not_escape(backend, + monkeypatch): + install_comtypes(monkeypatch) + backend._automation.remove_error = OSError("already gone") + assert backend.wait_for_focus_change(timeout=0.01) is None + + +def test_the_focus_handler_reports_the_element_that_was_focused(backend, + monkeypatch): + import queue + install_comtypes(monkeypatch) + sink: "queue.Queue" = queue.Queue() + handler = backend._make_focus_handler(sink) + assert handler is not None + raw = RawElement(name="Field", rect=Rect(0, 0, 1, 1)) + handler.IUIAutomationFocusChangedEventHandler_HandleFocusChangedEvent(raw) + assert sink.get_nowait()["name"] == "Field" + + +def test_a_focus_event_whose_element_is_gone_still_reports_something( + backend, monkeypatch): + import queue + install_comtypes(monkeypatch) + sink: "queue.Queue" = queue.Queue() + handler = backend._make_focus_handler(sink) + broken = RawElement(rect=Rect(0, 0, 1, 1)) + del broken.CurrentName + handler.IUIAutomationFocusChangedEventHandler_HandleFocusChangedEvent( + broken) + assert sink.get_nowait() == {"focused": True} + + +def test_no_comtypes_means_no_focus_handler(backend, monkeypatch): + monkeypatch.setitem(sys.modules, "comtypes", None) + assert backend._make_focus_handler(None) is None + + +def test_a_uia_module_without_the_event_interface_has_no_handler(backend, + monkeypatch): + install_comtypes(monkeypatch) + backend._uia_module = types.SimpleNamespace() + assert backend._make_focus_handler(None) is None + + diff --git a/test/unit_test/headless/test_accessibility_windows_patterns.py b/test/unit_test/headless/test_accessibility_windows_patterns.py new file mode 100644 index 00000000..48c53957 --- /dev/null +++ b/test/unit_test/headless/test_accessibility_windows_patterns.py @@ -0,0 +1,450 @@ +"""What the Windows backend does with a control once it has found one. + +The search that finds it is covered in `test_accessibility_windows_uia.py`; +this file starts from "here is the element" and is about the thirty-odd +control patterns hanging off it -- the half of the backend that reads a +slider's number, presses a button, expands a tree node or pulls the text out +of a document. + +They all go through one indirection, and it is where a mistake hides: + + unknown = raw.GetCurrentPattern(pattern_id) # nothing if unsupported + pattern = unknown.QueryInterface(getattr(uia_module, interface_name)) + +Two ids and one interface name per operation, none of them checked by +anything at runtime -- ask for the ValuePattern id and query the +RangeValuePattern interface and the failure is a `None` that reads exactly +like "no such control". The doubles here refuse a mismatched pair, so each +test below is also an assertion that the operation reaches for the pattern it +means. + +The other thing worth stating is the answer shape. A control that does not +support a pattern is not an error and not an empty value -- it is "no", and +each method has to spell "no" in whatever type it promised: `None` for a +read, `False` for an action, `[]` for a table. A caller cannot recover from +`False` when it meant "there is no such control", so the distinction is the +whole contract. +""" +from __future__ import annotations + +import sys + +import pytest + +from headless._uia_doubles import ( + Automation, Pattern, RawElement, Rect, UiaModule, Unknown, +) +from je_auto_control.utils.accessibility.backends import ( + windows_backend as backend_module, +) +from je_auto_control.utils.accessibility.backends.windows_backend import ( + WindowsAccessibilityBackend, +) + +VALUE = backend_module._UIA_VALUE_PATTERN_ID +INVOKE = backend_module._UIA_INVOKE_PATTERN_ID +TOGGLE = backend_module._UIA_TOGGLE_PATTERN_ID +GRID = backend_module._UIA_GRID_PATTERN_ID +GRID_ITEM = backend_module._UIA_GRIDITEM_PATTERN_ID +EXPAND = backend_module._UIA_EXPANDCOLLAPSE_PATTERN_ID +SELECTION_ITEM = backend_module._UIA_SELECTIONITEM_PATTERN_ID +RANGE = backend_module._UIA_RANGEVALUE_PATTERN_ID +SCROLL_ITEM = backend_module._UIA_SCROLLITEM_PATTERN_ID +TEXT = backend_module._UIA_TEXT_PATTERN_ID +ITEM_CONTAINER = backend_module._UIA_ITEMCONTAINER_PATTERN_ID +VIRTUALIZED = backend_module._UIA_VIRTUALIZEDITEM_PATTERN_ID +TABLE = backend_module._UIA_TABLE_PATTERN_ID +TRANSFORM = backend_module._UIA_TRANSFORM_PATTERN_ID +WINDOW = backend_module._UIA_WINDOW_PATTERN_ID +LEGACY = backend_module._UIA_LEGACYIACCESSIBLE_PATTERN_ID +SELECTION = backend_module._UIA_SELECTION_PATTERN_ID +MULTIPLE_VIEW = backend_module._UIA_MULTIPLEVIEW_PATTERN_ID + +_IS_PASSWORD = 30019 + + +@pytest.fixture(autouse=True) +def named_processes(monkeypatch): + monkeypatch.setattr(backend_module, "_process_name", + lambda pid: f"app{pid}.exe" if pid else "") + + +@pytest.fixture +def backend(monkeypatch): + monkeypatch.setattr(backend_module, "_is_available", lambda: True) + instance = WindowsAccessibilityBackend() + instance._automation = Automation() + instance._uia_module = UiaModule() + return instance + + +@pytest.fixture +def found(backend, monkeypatch): + """Answer every search with one raw element, and record what was asked.""" + state = {"raw": None, "filters": []} + + def _find_raw(name, role, app_name, automation_id, window_title=None, + contains=False): + state["filters"].append({"name": name, "role": role, + "app_name": app_name, + "automation_id": automation_id, + "window_title": window_title, + "contains": contains}) + return state["raw"] + + monkeypatch.setattr(backend, "_find_raw", _find_raw) + + def _set(raw): + state["raw"] = raw + return raw + + state["set"] = _set + return state + + +def _control(pattern_id=None, interface_name="", pattern=None, **kwargs): + """A raw element carrying at most one pattern.""" + patterns = {} + if pattern_id is not None: + patterns[pattern_id] = Unknown(interface_name, pattern or Pattern()) + kwargs.setdefault("rect", Rect(0, 0, 1, 1)) + return RawElement(patterns=patterns, **kwargs) + + +def _with(found, pattern_id, interface_name, pattern=None, **kwargs): + """Install a control carrying one pattern, and hand the pattern back.""" + pattern = pattern or Pattern() + found["set"](_control(pattern_id, interface_name, pattern, **kwargs)) + return pattern + + +# --- the pattern indirection -------------------------------------------------- + +def test_a_control_that_does_not_support_a_pattern_yields_none(backend): + assert backend._pattern(_control(), VALUE, "IUIAutomationValuePattern") is ( + None + ) + + +def test_asking_for_the_wrong_interface_yields_none_rather_than_a_pattern( + backend): + # The id and the interface name are two independent constants; a + # mismatched pair is a silent `None` that reads like "no such control". + raw = _control(VALUE, "IUIAutomationValuePattern") + assert backend._pattern(raw, VALUE, "IUIAutomationRangeValuePattern") is ( + None + ) + + +def test_a_provider_that_fails_the_pattern_query_yields_none(backend): + raw = _control(VALUE, "IUIAutomationValuePattern") + raw.pattern_error = OSError("provider stopped responding") + assert backend._pattern(raw, VALUE, "IUIAutomationValuePattern") is None + + +def test_the_uia_catch_tuple_covers_the_providers_own_error_type(): + # comtypes reports a provider failure as COMError, which derives straight + # from Exception. Until 2026-08-24 the 37 guards in this module named + # `(OSError, AttributeError, ...)` and therefore contained none of them, + # while the two walk guards in the same file already used UIA_ERRORS. + assert TypeError in backend_module._UIA_ERRORS + if sys.platform == "win32": + from _ctypes import COMError + assert COMError in backend_module._UIA_ERRORS + assert not issubclass( + COMError, (OSError, AttributeError, ValueError, TypeError)) + + +@pytest.mark.skipif(sys.platform != "win32", + reason="COMError only exists where comtypes can") +def test_a_window_that_closes_between_the_search_and_the_read_is_contained( + backend, found): + # The race the guard is for: the element was found, and the application + # owning it went away before its value could be read. The answer is "no + # value", not an exception past the executor's containment boundary. + from _ctypes import COMError + raw = _control(VALUE, "IUIAutomationValuePattern", + properties={_IS_PASSWORD: False}) + raw.pattern_error = COMError(-2147220991, "the window closed", None) + found["set"](raw) + assert backend.get_value(name="Field") is None + + +# --- value -------------------------------------------------------------------- + +def test_a_value_is_read_from_the_value_pattern(backend, found): + _with(found, VALUE, "IUIAutomationValuePattern", + Pattern(CurrentValue="typed"), properties={_IS_PASSWORD: False}) + assert backend.get_value(name="Field") == "typed" + + +def test_a_password_fields_value_is_never_handed_back(backend, found): + # UIA is supposed to mask it, but that is a convention a custom-drawn + # control can ignore -- and callers log and forward what they read. + _with(found, VALUE, "IUIAutomationValuePattern", + Pattern(CurrentValue="hunter2"), properties={_IS_PASSWORD: True}) + assert backend.get_value(name="Password") is None + + +def test_a_value_read_forwards_the_scope_it_was_given(backend, found): + _with(found, VALUE, "IUIAutomationValuePattern", + Pattern(CurrentValue="typed"), properties={_IS_PASSWORD: False}) + backend.get_value(name="Field", window_title="Editor", contains=True) + assert found["filters"][-1]["window_title"] == "Editor" + assert found["filters"][-1]["contains"] is True + + +def test_a_control_with_no_value_pattern_has_no_value(backend, found): + found["set"](_control(properties={_IS_PASSWORD: False})) + assert backend.get_value(name="Field") is None + + +def test_a_value_the_provider_will_not_answer_reads_as_none(backend, found): + _with(found, VALUE, "IUIAutomationValuePattern", Pattern(), + properties={_IS_PASSWORD: False}) + assert backend.get_value(name="Field") is None + + +def test_an_empty_value_reads_as_the_empty_string(backend, found): + _with(found, VALUE, "IUIAutomationValuePattern", + Pattern(CurrentValue=None), properties={_IS_PASSWORD: False}) + assert backend.get_value(name="Field") == "" + + +def test_a_control_that_is_not_there_has_no_value(backend, found): + assert backend.get_value(name="Nothing") is None + + +def test_setting_a_value_writes_it_as_text(backend, found): + pattern = _with(found, VALUE, "IUIAutomationValuePattern") + assert backend.set_value(42, name="Field") is True + assert pattern.calls == [("SetValue", ("42",))] + + +def test_setting_a_value_on_a_control_that_is_not_there_fails(backend, found): + assert backend.set_value("x", name="Nothing") is False + + +def test_setting_a_value_the_control_refuses_fails(backend, found): + pattern = _with(found, VALUE, "IUIAutomationValuePattern") + pattern.errors["SetValue"] = OSError("read-only") + assert backend.set_value("x", name="Field") is False + + +# --- invoke and toggle -------------------------------------------------------- + +def test_invoking_presses_the_control(backend, found): + pattern = _with(found, INVOKE, "IUIAutomationInvokePattern") + assert backend.invoke(name="OK") is True + assert pattern.calls == [("Invoke", ())] + + +def test_invoking_a_control_with_no_invoke_pattern_fails(backend, found): + found["set"](_control()) + assert backend.invoke(name="OK") is False + + +def test_invoking_a_control_that_is_not_there_fails(backend, found): + assert backend.invoke(name="OK") is False + + +def test_an_invoke_the_control_refuses_fails(backend, found): + pattern = _with(found, INVOKE, "IUIAutomationInvokePattern") + pattern.errors["Invoke"] = OSError("disabled") + assert backend.invoke(name="OK") is False + + +def test_toggling_flips_the_control(backend, found): + pattern = _with(found, TOGGLE, "IUIAutomationTogglePattern") + assert backend.toggle(name="Enabled") is True + assert pattern.calls == [("Toggle", ())] + + +def test_toggling_a_control_that_cannot_be_toggled_fails(backend, found): + found["set"](_control()) + assert backend.toggle(name="Enabled") is False + + +def test_a_toggle_the_control_refuses_fails(backend, found): + pattern = _with(found, TOGGLE, "IUIAutomationTogglePattern") + pattern.errors["Toggle"] = OSError("disabled") + assert backend.toggle(name="Enabled") is False + + +# --- expand / collapse / select / scroll -------------------------------------- + +@pytest.mark.parametrize("method,expected", [ + ("expand", "Expand"), ("collapse", "Collapse"), +]) +def test_expanding_and_collapsing_reach_the_same_pattern(backend, found, + method, expected): + pattern = _with(found, EXPAND, "IUIAutomationExpandCollapsePattern") + assert getattr(backend, method)(name="Node") is True + assert pattern.calls == [(expected, ())] + + +def test_an_action_on_a_control_without_that_pattern_fails(backend, found): + # `expand`, `select_item`, `scroll_into_view`, `move_element`, + # `resize_element`, `set_range_value`, `set_window_state` and + # `legacy_default_action` all funnel through one helper; this is that + # helper's "the control cannot do it" answer. + found["set"](_control()) + assert backend.expand(name="Node") is False + + +def test_an_action_the_control_refuses_fails(backend, found): + pattern = _with(found, EXPAND, "IUIAutomationExpandCollapsePattern") + pattern.errors["Expand"] = OSError("already expanded") + assert backend.expand(name="Node") is False + + +def test_an_action_on_a_control_that_is_not_there_fails(backend, found): + assert backend.expand(name="Node") is False + + +@pytest.mark.parametrize("code,expected", [ + (0, "collapsed"), (1, "expanded"), (2, "partial"), (3, "leaf"), +]) +def test_an_expand_state_is_reported_by_name(backend, found, code, expected): + _with(found, EXPAND, "IUIAutomationExpandCollapsePattern", + Pattern(CurrentExpandCollapseState=code)) + assert backend.expand_state(name="Node") == expected + + +def test_an_unknown_expand_state_reads_as_none(backend, found): + _with(found, EXPAND, "IUIAutomationExpandCollapsePattern", + Pattern(CurrentExpandCollapseState=9)) + assert backend.expand_state(name="Node") is None + + +def test_an_unreadable_expand_state_reads_as_none(backend, found): + _with(found, EXPAND, "IUIAutomationExpandCollapsePattern", + Pattern(CurrentExpandCollapseState="sideways")) + assert backend.expand_state(name="Node") is None + + +def test_a_control_that_does_not_expand_has_no_expand_state(backend, found): + found["set"](_control()) + assert backend.expand_state(name="Node") is None + + +def test_a_control_that_is_not_there_has_no_expand_state(backend, found): + assert backend.expand_state(name="Node") is None + + +def test_selecting_an_item_reaches_the_selection_item_pattern(backend, found): + pattern = _with(found, SELECTION_ITEM, + "IUIAutomationSelectionItemPattern") + assert backend.select_item(name="Row") is True + assert pattern.calls == [("Select", ())] + + +def test_scrolling_into_view_reaches_the_scroll_item_pattern(backend, found): + pattern = _with(found, SCROLL_ITEM, "IUIAutomationScrollItemPattern") + assert backend.scroll_into_view(name="Row") is True + assert pattern.calls == [("ScrollIntoView", ())] + + +# --- range -------------------------------------------------------------------- + +def test_a_range_is_read_as_three_floats(backend, found): + _with(found, RANGE, "IUIAutomationRangeValuePattern", + Pattern(CurrentValue=3, CurrentMinimum=0, CurrentMaximum=10)) + assert backend.get_range(name="Volume") == { + "value": 3.0, "minimum": 0.0, "maximum": 10.0, + } + + +def test_a_range_that_cannot_be_read_is_none(backend, found): + _with(found, RANGE, "IUIAutomationRangeValuePattern", + Pattern(CurrentValue="loud", CurrentMinimum=0, CurrentMaximum=10)) + assert backend.get_range(name="Volume") is None + + +def test_a_control_with_no_range_has_none(backend, found): + found["set"](_control()) + assert backend.get_range(name="Volume") is None + + +def test_a_control_that_is_not_there_has_no_range(backend, found): + assert backend.get_range(name="Volume") is None + + +def test_setting_a_range_value_writes_it_as_a_float(backend, found): + pattern = _with(found, RANGE, "IUIAutomationRangeValuePattern") + assert backend.set_range_value(7, name="Volume") is True + assert pattern.calls == [("SetValue", (7.0,))] + + +# --- transform and window state ----------------------------------------------- + +def test_moving_a_control_writes_floats(backend, found): + pattern = _with(found, TRANSFORM, "IUIAutomationTransformPattern") + assert backend.move_element(10, 20, name="Panel") is True + assert pattern.calls == [("Move", (10.0, 20.0))] + + +def test_resizing_a_control_writes_floats(backend, found): + pattern = _with(found, TRANSFORM, "IUIAutomationTransformPattern") + assert backend.resize_element(300, 400, name="Panel") is True + assert pattern.calls == [("Resize", (300.0, 400.0))] + + +@pytest.mark.parametrize("state,code", [ + ("normal", 0), ("maximized", 1), ("minimized", 2), ("MAXIMIZED", 1), +]) +def test_a_window_state_is_written_as_its_visual_state_code(backend, found, + state, code): + pattern = _with(found, WINDOW, "IUIAutomationWindowPattern") + assert backend.set_window_state(state, name="Editor") is True + assert pattern.calls == [("SetWindowVisualState", (code,))] + + +def test_an_unknown_window_state_is_refused_before_any_search(backend, found): + assert backend.set_window_state("shaded", name="Editor") is False + assert found["filters"] == [], "it never went looking" + + +@pytest.mark.parametrize("code,expected", [ + (0, "running"), (1, "closing"), (2, "ready"), (3, "blocked_by_modal"), + (4, "not_responding"), +]) +def test_a_window_interaction_state_is_reported_by_name(backend, found, code, + expected): + _with(found, WINDOW, "IUIAutomationWindowPattern", + Pattern(CurrentWindowInteractionState=code)) + assert backend.window_interaction_state(name="Editor") == expected + + +def test_an_unknown_interaction_state_reads_as_none(backend, found): + _with(found, WINDOW, "IUIAutomationWindowPattern", + Pattern(CurrentWindowInteractionState=99)) + assert backend.window_interaction_state(name="Editor") is None + + +def test_an_unreadable_interaction_state_reads_as_none(backend, found): + _with(found, WINDOW, "IUIAutomationWindowPattern", Pattern()) + assert backend.window_interaction_state(name="Editor") is None + + +def test_a_control_that_is_not_a_window_has_no_interaction_state(backend, + found): + found["set"](_control()) + assert backend.window_interaction_state(name="Editor") is None + + +def test_a_control_that_is_not_there_has_no_interaction_state(backend, found): + assert backend.window_interaction_state(name="Editor") is None + + +# --- state -------------------------------------------------------------------- + +def test_the_state_of_a_control_is_read_from_it(backend, found): + found["set"](_control(properties={_IS_PASSWORD: False, + 30029: True, 30045: "typed", + 30046: False})) + assert backend.get_state(name="Field")["value"] == "typed" + + +def test_the_state_of_a_control_that_is_not_there_is_none(backend, found): + assert backend.get_state(name="Field") is None diff --git a/test/unit_test/headless/test_accessibility_windows_query.py b/test/unit_test/headless/test_accessibility_windows_query.py new file mode 100644 index 00000000..b860a374 --- /dev/null +++ b/test/unit_test/headless/test_accessibility_windows_query.py @@ -0,0 +1,447 @@ +"""Where a UIA search starts, and what a control is actually holding. + +`windows_query.py` and `windows_state.py` are the two halves of the Windows +accessibility backend that were split out of it, and both sat near 20% on +every square -- including the Windows ones, because a test that reaches them +needs a UIAutomation provider and a desktop with windows on it. + +Neither needs either. Both take the automation object as an argument, and +every COM import in them is lazy, so a recorder standing in for UIA drives +them from all nine squares. + +The two files encode measurements, and the tests below are what stops those +being quietly undone: + +* **A desktop-rooted `FindAll` is one call that cannot be stopped.** Measured + at ~61 s for 2,085 elements, against 0.14 s starting from one window. So + `search_roots` yields *windows*, front-most first, and `walk_elements` + walks node by node -- asking for 200 elements has to cost 200 elements' + worth of work no matter how large the window is. +* **Properties are cross-process reads.** They come back through a cache + request built once per walk; reading them individually is thousands of + round trips. +* **comtypes returns a wrapper around a NULL pointer, not `None`.** So + `child is not None` is *true* at the end of a sibling list, and the walk + collects a phantom element that raises the moment anything reads it. + Truthiness is the check that works, and a stub that returned `None` would + never catch the difference -- so the one here returns a false-y wrapper, + exactly as comtypes does. +* **An unsupported pattern answers with a default**, not an error: an empty + string for a control that has no value at all. Reading it without asking + whether the pattern exists turns "no such concept" into "it is empty", + which is the more misleading of the two because a caller acts on it. +* **A password field's contents never leave.** UIA is supposed to mask them, + but that is a convention a custom-drawn control can ignore. +""" +from __future__ import annotations + +import sys +import types + +import pytest + +from je_auto_control.utils.accessibility.backends import windows_query as query +from je_auto_control.utils.accessibility.backends.windows_query import ( + CACHED_PROPERTIES, _is_null, search_root, search_roots, walk_elements, +) +from je_auto_control.utils.accessibility.backends.windows_state import ( + TOGGLE_STATES, is_password, read_state, +) +from je_auto_control.utils.accessibility.element import ( + AccessibilityNotAvailableError, +) + + +class _NullPointer: + """What comtypes hands back for "no such element": false-y, not None.""" + + def __bool__(self) -> bool: + return False + + +class _Element: + """A UIA element: a name, some children, and scripted property reads.""" + + def __init__(self, name: str = "", children=None, properties=None, + error=None) -> None: + self.name = name + self.children = list(children or []) + self.properties = dict(properties or {}) + self.error = error + self.reads = [] + + def GetCurrentPropertyValue(self, property_id): # noqa: N802 # UIA name + self.reads.append(property_id) + if self.error is not None: + raise self.error + return self.properties.get(property_id) + + def __repr__(self) -> str: # pragma: no cover - debugging aid + return f"<{self.name}>" + + +class _CacheRequest: + def __init__(self) -> None: + self.properties = [] + + def AddProperty(self, property_id): # noqa: N802 # reason: UIA name + self.properties.append(property_id) + + +class _Walker: + """The control-view walker, over `_Element.children`.""" + + def __init__(self, world) -> None: + self._world = world + + def GetFirstChildElementBuildCache(self, node, request): # noqa: N802 + self._world.cache_requests.append(request) + if node in self._world.first_child_errors: + raise self._world.first_child_errors[node] + return node.children[0] if node.children else _NullPointer() + + def GetNextSiblingElementBuildCache(self, node, request): # noqa: N802 + if node in self._world.sibling_errors: + raise self._world.sibling_errors[node] + for parent in self._world.parents_of(node): + index = parent.children.index(node) + if index + 1 < len(parent.children): + return parent.children[index + 1] + return _NullPointer() + + +class _Automation: + def __init__(self, root=None, handles=None) -> None: + self.root = root or _Element("desktop") + self.handles = dict(handles or {}) # hwnd -> element (or exception) + self.cache_requests = [] + self.first_child_errors = {} + self.sibling_errors = {} + self.ControlViewWalker = _Walker(self) # noqa: N815 # reason: UIA + + def GetRootElement(self): # noqa: N802 # reason: the UIA name + return self.root + + def CreateCacheRequest(self): # noqa: N802 # reason: the UIA name + return _CacheRequest() + + def ElementFromHandle(self, hwnd): # noqa: N802 # reason: the UIA name + found = self.handles.get(hwnd) + if isinstance(found, Exception): + raise found + return found + + def parents_of(self, node): + stack = [self.root, *[h for h in self.handles.values() + if isinstance(h, _Element)]] + seen = set() + while stack: + current = stack.pop() + if id(current) in seen: + continue + seen.add(id(current)) + if node in current.children: + yield current + stack.extend(current.children) + + +@pytest.fixture +def windows(monkeypatch): + """Stand in for `get_all_window_hwnd`, on every platform.""" + listing = [] + + def _get_all_window_hwnd(): + return list(listing) + + module = types.ModuleType( + "je_auto_control.windows.window.windows_window_manage") + module.get_all_window_hwnd = _get_all_window_hwnd + package = types.ModuleType("je_auto_control.windows.window") + package.windows_window_manage = module + monkeypatch.setitem(sys.modules, "je_auto_control.windows.window", package) + monkeypatch.setitem( + sys.modules, "je_auto_control.windows.window.windows_window_manage", + module) + return listing + + +# --- choosing a root ---------------------------------------------------------- + +def test_an_unscoped_search_starts_at_the_desktop(): + automation = _Automation() + assert search_root(automation, None) is automation.root + + +def test_a_window_title_is_matched_as_a_case_insensitive_substring(windows): + target = _Element("notepad") + automation = _Automation(handles={101: target}) + windows.extend([(100, "Calculator"), (101, "Untitled - Notepad")]) + assert search_root(automation, " notePAD ") is target + + +def test_the_first_window_whose_title_matches_wins(windows): + first, second = _Element("first"), _Element("second") + automation = _Automation(handles={1: first, 2: second}) + windows.extend([(1, "Report - Editor"), (2, "Notes - Editor")]) + assert search_root(automation, "Editor") is first + + +def test_a_title_that_matches_nothing_says_so(windows): + windows.append((1, "Calculator")) + with pytest.raises(AccessibilityNotAvailableError, match="Notepad"): + search_root(_Automation(), "Notepad") + + +def test_a_window_with_no_title_is_not_a_match(windows): + windows.append((1, None)) + with pytest.raises(AccessibilityNotAvailableError): + search_root(_Automation(), "anything") + + +def test_a_window_uia_will_not_open_is_skipped(windows): + later = _Element("later") + automation = _Automation(handles={1: None, 2: later}) + windows.extend([(1, "Editor one"), (2, "Editor two")]) + assert search_root(automation, "Editor") is later + + +# --- iterating the roots ------------------------------------------------------ + +def test_an_unscoped_walk_yields_windows_rather_than_the_desktop(windows): + # Same coverage as a desktop-rooted walk, but the caller can stop. That + # is the 0.22 s / 61 s difference the module's docstring measured. + first, second = _Element("one"), _Element("two") + automation = _Automation(handles={1: first, 2: second}) + windows.extend([(1, "one"), (2, "two")]) + assert list(search_roots(automation, None)) == [first, second] + + +def test_the_roots_come_back_in_z_order(windows): + # EnumWindows returns front-to-back, so the window the user is actually + # looking at is searched first. + front, back = _Element("front"), _Element("back") + automation = _Automation(handles={9: front, 1: back}) + windows.extend([(9, "front"), (1, "back")]) + assert [e.name for e in search_roots(automation, None)] == ["front", + "back"] + + +def test_a_window_that_closes_between_enumerate_and_use_is_skipped(windows): + survivor = _Element("survivor") + automation = _Automation(handles={1: OSError("window closed"), + 2: survivor}) + windows.extend([(1, "gone"), (2, "survivor")]) + assert list(search_roots(automation, None)) == [survivor] + + +def test_a_null_pointer_root_is_skipped(windows): + # comtypes answers with a wrapper around NULL, which is not None. + survivor = _Element("survivor") + automation = _Automation(handles={1: _NullPointer(), 2: survivor}) + windows.extend([(1, "phantom"), (2, "survivor")]) + assert list(search_roots(automation, None)) == [survivor] + + +def test_a_scoped_walk_yields_exactly_one_root(windows): + target = _Element("notepad") + automation = _Automation(handles={1: target}) + windows.append((1, "Untitled - Notepad")) + assert list(search_roots(automation, "Notepad")) == [target] + + +@pytest.mark.parametrize("value,expected", [ + (None, True), (_NullPointer(), True), (_Element("real"), False), +]) +def test_null_detection_uses_truthiness_not_identity(value, expected): + assert _is_null(value) is expected + + +# --- walking ------------------------------------------------------------------ + +def _tree(): + leaf_a = _Element("a") + leaf_b = _Element("b") + group = _Element("group", children=[leaf_a, leaf_b]) + tail = _Element("tail") + root = _Element("root", children=[group, tail]) + return root, [group, leaf_a, leaf_b, tail] + + +def test_the_walk_is_depth_first_in_reading_order(): + root, expected = _tree() + automation = _Automation(root=root) + walked = list(walk_elements(automation, root, 10)) + assert [e.name for e in walked] == [e.name for e in expected] + + +def test_the_walk_stops_at_the_limit(): + root, _expected = _tree() + automation = _Automation(root=root) + assert len(list(walk_elements(automation, root, 2))) == 2 + + +@pytest.mark.parametrize("limit", [0, -1]) +def test_a_walk_with_no_budget_asks_for_nothing(limit): + root, _expected = _tree() + automation = _Automation(root=root) + assert list(walk_elements(automation, root, limit)) == [] + assert automation.cache_requests == [], "not one cross-process call" + + +def test_every_property_the_conversion_reads_is_cached_up_front(): + # Each `Current*` read is another cross-process call; the bulk request is + # what makes converting 500 elements cost 0.02 s. + root, _expected = _tree() + automation = _Automation(root=root) + list(walk_elements(automation, root, 10)) + assert automation.cache_requests + assert automation.cache_requests[0].properties == list(CACHED_PROPERTIES) + + +def test_a_node_that_refuses_its_first_child_ends_that_branch(): + root, _expected = _tree() + automation = _Automation(root=root) + automation.first_child_errors[root.children[0]] = OSError("gone") + walked = [e.name for e in walk_elements(automation, root, 10)] + assert walked == ["group", "tail"], "the sibling after it still came back" + + +def test_a_node_that_refuses_a_sibling_keeps_what_it_already_had(): + root, _expected = _tree() + automation = _Automation(root=root) + automation.sibling_errors[root.children[0]] = OSError("gone") + walked = [e.name for e in walk_elements(automation, root, 10)] + assert walked == ["group", "a", "b"], "tail was never reached" + + +def test_a_node_with_more_children_than_the_budget_is_capped(): + children = [_Element(f"c{index}") for index in range(50)] + root = _Element("root", children=children) + automation = _Automation(root=root) + assert len(list(walk_elements(automation, root, 3))) == 3 + + +# --- reading a control's state ------------------------------------------------ + +_IS_PASSWORD = 30019 +_IS_VALUE_AVAILABLE, _VALUE = 30029, 30045 +_IS_TOGGLE_AVAILABLE, _TOGGLE = 30041, 30086 +_IS_SELECTION_AVAILABLE, _SELECTED = 30036, 30079 +_IS_RANGE_AVAILABLE, _RANGE = 30034, 30047 +_READONLY = 30046 +_LEGACY_VALUE = 30093 + + +def _control(properties): + """A UIA element whose property reads answer from a plain dict.""" + return _Element(properties=properties) + + +def test_a_password_field_reports_only_that_it_is_one(): + state = read_state(_control({_IS_PASSWORD: True, + _IS_VALUE_AVAILABLE: True, + _VALUE: "hunter2"})) + assert state == {"password": True} + assert "hunter2" not in str(state) + + +def test_an_element_whose_password_flag_cannot_be_read_counts_as_one(): + # Failing closed: the cost of being wrong the other way is a credential. + element = _control({_IS_VALUE_AVAILABLE: True, _VALUE: "secret"}) + element.error = OSError("provider gone") + assert is_password(element) is True + assert read_state(element) == {"password": True} + + +def test_an_ordinary_field_is_not_a_password(): + assert is_password(_control({_IS_PASSWORD: False})) is False + + +def test_a_value_is_read_only_when_the_pattern_is_supported(): + # An unsupported pattern answers with the default -- an empty string -- + # which reads as "the value is empty" rather than "there is no value". + absent = _control({_IS_PASSWORD: False, _VALUE: ""}) + assert "value" not in read_state(absent) + + +def test_a_supported_value_comes_back_as_text_with_its_read_only_flag(): + state = read_state(_control({_IS_PASSWORD: False, + _IS_VALUE_AVAILABLE: True, + _VALUE: 42, _READONLY: True})) + assert state["value"] == "42" + assert state["read_only"] is True + + +def test_an_empty_supported_value_is_still_a_value(): + state = read_state(_control({_IS_PASSWORD: False, + _IS_VALUE_AVAILABLE: True, _VALUE: None})) + assert state.get("value") is None, "None never reached the state at all" + + +def test_a_control_with_no_value_pattern_falls_back_to_the_legacy_one(): + # Win32 controls with no UIA provider still answer the legacy accessible + # interface, and that is the only value they will ever report. + state = read_state(_control({_IS_PASSWORD: False, + _LEGACY_VALUE: "legacy text"})) + assert state["value"] == "legacy text" + assert "read_only" not in state + + +def test_an_empty_legacy_value_is_not_reported(): + state = read_state(_control({_IS_PASSWORD: False, _LEGACY_VALUE: ""})) + assert "value" not in state + + +@pytest.mark.parametrize("code,expected", sorted(TOGGLE_STATES.items())) +def test_a_toggle_state_is_reported_by_name(code, expected): + state = read_state(_control({_IS_PASSWORD: False, + _IS_TOGGLE_AVAILABLE: True, + _TOGGLE: code})) + assert state["toggle"] == expected + + +def test_an_unknown_toggle_code_is_reported_as_itself(): + state = read_state(_control({_IS_PASSWORD: False, + _IS_TOGGLE_AVAILABLE: True, _TOGGLE: 9})) + assert state["toggle"] == "9" + + +def test_a_selection_state_is_reported_as_a_bool(): + state = read_state(_control({_IS_PASSWORD: False, + _IS_SELECTION_AVAILABLE: True, + _SELECTED: 1})) + assert state["selected"] is True + + +def test_a_range_value_is_reported_as_a_float(): + state = read_state(_control({_IS_PASSWORD: False, + _IS_RANGE_AVAILABLE: True, _RANGE: 3})) + assert state["number"] == 3.0 + assert isinstance(state["number"], float) + + +def test_a_control_that_supports_nothing_reports_nothing(): + assert read_state(_control({_IS_PASSWORD: False})) == {} + + +def test_a_control_that_supports_several_patterns_reports_all_of_them(): + state = read_state(_control({ + _IS_PASSWORD: False, + _IS_VALUE_AVAILABLE: True, _VALUE: "7", _READONLY: False, + _IS_TOGGLE_AVAILABLE: True, _TOGGLE: 1, + _IS_SELECTION_AVAILABLE: True, _SELECTED: True, + _IS_RANGE_AVAILABLE: True, _RANGE: 7.5, + })) + assert state == {"value": "7", "read_only": False, "toggle": "on", + "selected": True, "number": 7.5} + + +def test_the_uia_error_tuple_contains_the_com_error_on_windows(): + # comtypes reports provider failures as COMError, which inherits from + # Exception and from none of the usual suspects -- so an + # `except (OSError, AttributeError)` around a UIA call does not contain + # it, and a window closing mid-walk surfaces exactly that way. + assert OSError in query.UIA_ERRORS + if sys.platform == "win32": + from _ctypes import COMError + assert COMError in query.UIA_ERRORS diff --git a/test/unit_test/headless/test_accessibility_windows_uia.py b/test/unit_test/headless/test_accessibility_windows_uia.py new file mode 100644 index 00000000..0b887923 --- /dev/null +++ b/test/unit_test/headless/test_accessibility_windows_uia.py @@ -0,0 +1,527 @@ +"""Setting up UIAutomation, listing a desktop, and converting an element. + +`backends/windows_backend.py` is the biggest single hole in this project's +coverage. It stayed one on the Windows squares as much as the others: a test +that reaches it needs a UIAutomation provider, a desktop with windows on it, +and applications willing to answer. But every `comtypes` import in the module +is inside a function and the automation object lives on the instance, so +doubles reach all of it -- from every square. + +This file covers getting there and getting back: the automation object, the +listing, the search, and the conversion. The control patterns are in +`test_accessibility_windows_patterns.py`. + +Three decisions in here were made against measurements, and are the kind that +get undone by a well-meaning simplification: + +* **`CUIAutomation8` is asked for first.** It is the only class that hands out + `IUIAutomation2`, which is the only way to bound how long UIA waits on an + application's provider. A full-screen game that never answered made one + `ElementFromHandle` block for 60 seconds; with the timeout set, 1.0 s. Only + the *connect* step is bounded -- how long a legitimate query may take is a + different question. +* **The listing pulls one window at a time.** A desktop-rooted walk measured + ~61 s and cannot be interrupted once started. The budget is checked + *before* the next window is fetched, not after, because obtaining a + window's root element is itself a cross-process call that blocks against a + hung application -- fetching one more root only to find the results already + complete cost exactly that. +* **An `app_name` filter walks more than it keeps.** Most elements in a + window belong to that window's application, so a small overscan is plenty + -- but it is still bounded, or an unmatched filter walks an entire subtree + to return nothing. +""" +from __future__ import annotations + +import sys +import types + +import pytest + +from headless._uia_doubles import ( + Automation, RawElement, Rect, UiaModule, install_comtypes, +) +from je_auto_control.utils.accessibility.backends import ( + windows_backend as backend_module, +) +from je_auto_control.utils.accessibility.backends.windows_backend import ( + WindowsAccessibilityBackend, _convert_uia, _create_automation, + _process_name, _read_properties, _safe_name, +) +from je_auto_control.utils.accessibility.element import ( + AccessibilityNotAvailableError, +) + + +@pytest.fixture(autouse=True) +def named_processes(monkeypatch): + """Name a pid without asking Windows about a pid the test invented.""" + monkeypatch.setattr(backend_module, "_process_name", + lambda pid: f"app{pid}.exe" if pid else "") + + +@pytest.fixture +def backend(monkeypatch): + """An available backend whose automation object is already built.""" + monkeypatch.setattr(backend_module, "_is_available", lambda: True) + instance = WindowsAccessibilityBackend() + instance._automation = Automation() + instance._uia_module = UiaModule() + return instance + + +def _element(name="", control_type=50000, rect=None, process_id=0, + automation_id="", enabled=True, cached=False): + return RawElement(name=name, control_type=control_type, + rect=rect or Rect(0, 0, 10, 20), process_id=process_id, + automation_id=automation_id, enabled=enabled, + cached=cached) + + +# --- availability and the automation object ----------------------------------- + +def test_a_windows_box_without_comtypes_refuses_and_says_how_to_fix_it( + monkeypatch): + monkeypatch.setattr(backend_module, "_is_available", lambda: False) + instance = WindowsAccessibilityBackend() + with pytest.raises(AccessibilityNotAvailableError, match="pip install"): + instance.list_elements() + + +def test_the_probe_reports_whether_comtypes_imports(monkeypatch): + install_comtypes(monkeypatch) + assert backend_module._is_available() is True + monkeypatch.setitem(sys.modules, "comtypes.client", None) + assert backend_module._is_available() is False + + +def test_the_backend_names_the_api_it_uses(backend): + assert backend.name == "windows-uia" + + +def test_the_automation_object_is_built_once_and_kept(monkeypatch): + monkeypatch.setattr(backend_module, "_is_available", lambda: True) + install_comtypes(monkeypatch) + instance = WindowsAccessibilityBackend() + first = instance._ensure_automation() + assert instance._ensure_automation() is first + + +def test_a_missing_uiautomation_dll_is_reported_as_unavailable(monkeypatch): + # The DLL is part of Windows, but a stripped image or a broken + # registration is a real state and must not surface as an OSError. + monkeypatch.setattr(backend_module, "_is_available", lambda: True) + install_comtypes(monkeypatch, module_error=OSError("no such module")) + instance = WindowsAccessibilityBackend() + with pytest.raises(AccessibilityNotAvailableError, + match="UIAutomationCore.dll"): + instance._ensure_automation() + + +def test_the_bounded_provider_wait_is_asked_for_first(monkeypatch): + # CUIAutomation8 is the only class that hands out IUIAutomation2, which + # is the only way to bound how long UIA waits on a provider. + created = install_comtypes(monkeypatch, uia_module=UiaModule()) + automation = _create_automation(UiaModule()) + [(clsid, interface)] = created + assert clsid == backend_module._CLSID_CUIAUTOMATION8 + assert interface == "IUIAutomation2" + assert automation.ConnectionTimeout == 1000 + + +def test_an_older_windows_falls_back_to_the_unbounded_class(monkeypatch): + created = install_comtypes(monkeypatch) + _create_automation(UiaModule(has_iuiautomation2=False)) + [(clsid, interface)] = created + assert clsid == backend_module._CLSID_CUIAUTOMATION + assert interface == "IUIAutomation" + + +def test_a_refused_iuiautomation2_falls_back_rather_than_failing(monkeypatch): + calls = [] + + def _create(clsid, interface=None): + calls.append(clsid) + if clsid == backend_module._CLSID_CUIAUTOMATION8: + raise OSError("class not registered") + return Automation() + + install_comtypes(monkeypatch, create=_create) + assert _create_automation(UiaModule()) is not None + assert calls == [backend_module._CLSID_CUIAUTOMATION8, + backend_module._CLSID_CUIAUTOMATION] + + +# --- listing ------------------------------------------------------------------ + +@pytest.fixture +def listing(monkeypatch): + """Script `search_roots` and `walk_elements` for the listing tests.""" + state = {"roots": [], "under": {}, "walk_error": None, "budgets": [], + "roots_pulled": 0} + + def _search_roots(automation, window_title): + state["window_title"] = window_title + for root in state["roots"]: + state["roots_pulled"] += 1 + yield root + + def _walk_elements(automation, root, limit): + state["budgets"].append(limit) + if state["walk_error"] is not None: + raise state["walk_error"] + for element in state["under"].get(id(root), []): + yield element + + monkeypatch.setattr(backend_module, "search_roots", _search_roots) + monkeypatch.setattr(backend_module, "walk_elements", _walk_elements) + return state + + +def test_a_listing_returns_the_window_and_then_its_contents(backend, listing): + window = _element(name="Editor", process_id=7) + child = _element(name="OK", process_id=7, cached=True) + listing["roots"] = [window] + listing["under"][id(window)] = [child] + names = [e.name for e in backend.list_elements()] + assert names == ["Editor", "OK"] + + +def test_a_scoped_listing_does_not_add_the_window_itself(backend, listing): + # Searching a window's descendants does not include the window element, + # so the unscoped walk adds it back -- and the scoped one must not, or a + # caller asking about one window gets it twice. + window = _element(name="Editor", process_id=7) + child = _element(name="OK", process_id=7, cached=True) + listing["roots"] = [window] + listing["under"][id(window)] = [child] + names = [e.name for e in backend.list_elements(window_title="Editor")] + assert names == ["OK"] + assert listing["window_title"] == "Editor" + + +def test_a_listing_stops_pulling_windows_once_it_has_enough(backend, listing): + # Obtaining a window's root element is itself a cross-process call that + # blocks against a hung application, so the budget is checked before the + # next one is fetched rather than after. + first, second = _element(name="one"), _element(name="two") + listing["roots"] = [first, second] + assert len(backend.list_elements(max_results=1)) == 1 + assert listing["roots_pulled"] == 1 + + +def test_a_listing_asked_for_nothing_walks_nothing(backend, listing): + listing["roots"] = [_element(name="one")] + assert backend.list_elements(max_results=0) == [] + assert listing["roots_pulled"] == 0 + + +def test_a_negative_maximum_is_treated_as_none(backend, listing): + listing["roots"] = [_element(name="one")] + assert backend.list_elements(max_results=-5) == [] + + +def test_an_app_name_filter_keeps_only_that_application(backend, listing): + window = _element(name="Editor", process_id=7) + listing["roots"] = [window] + listing["under"][id(window)] = [ + _element(name="mine", process_id=7, cached=True), + _element(name="theirs", process_id=8, cached=True), + ] + names = [e.name for e in backend.list_elements(app_name="app7.exe")] + assert names == ["Editor", "mine"] + + +def test_an_app_name_filter_walks_further_than_it_keeps(backend, listing): + # Most elements in a window belong to that window's application, so the + # overscan is small -- but an unmatched filter must still be bounded. + window = _element(name="Editor", process_id=7) + listing["roots"] = [window] + backend.list_elements(app_name="app7.exe", max_results=10) + assert listing["budgets"][0] > 10 + assert listing["budgets"][0] < 10 * 100, "bounded, not unbounded" + + +def test_a_filtered_window_that_does_not_match_is_not_listed(backend, + listing): + listing["roots"] = [_element(name="Editor", process_id=8)] + assert backend.list_elements(app_name="app7.exe") == [] + + +def test_an_unresponsive_window_does_not_lose_the_whole_listing(backend, + listing): + window = _element(name="Editor", process_id=7) + listing["roots"] = [window] + listing["walk_error"] = OSError("provider stopped responding") + assert [e.name for e in backend.list_elements()] == ["Editor"] + + +def test_an_element_that_cannot_be_converted_is_skipped(backend, listing): + window = _element(name="Editor", process_id=7) + broken = RawElement(cached=True) + del broken.CachedName + listing["roots"] = [window] + listing["under"][id(window)] = [broken, + _element(name="OK", cached=True)] + assert [e.name for e in backend.list_elements()] == ["Editor", "OK"] + + +def test_a_desktop_with_no_windows_lists_nothing(backend, listing): + assert backend.list_elements() == [] + + +def test_a_listing_stops_mid_window_once_it_has_enough(backend, listing): + # The walk is a generator for exactly this: a window with thousands of + # nodes costs what was asked for, not what it contains. + window = _element(name="Editor") + listing["roots"] = [window] + listing["under"][id(window)] = [ + _element(name=f"c{index}", cached=True) for index in range(5) + ] + names = [e.name for e in backend.list_elements(max_results=2)] + assert names == ["Editor", "c0"] + + +# --- searching for one control ------------------------------------------------ + +@pytest.fixture +def search(monkeypatch, listing): + """The same scripting, for the `_find_raw` path.""" + return listing + + +def test_a_search_returns_the_first_match(backend, search): + window = _element(name="Editor") + wanted = _element(name="OK", cached=True) + search["roots"] = [window] + search["under"][id(window)] = [_element(name="Cancel", cached=True), + wanted] + assert backend._find_raw("OK", None, None, None) is wanted + + +def test_an_unscoped_search_can_match_the_window_itself(backend, search): + window = _element(name="Editor") + search["roots"] = [window] + assert backend._find_raw("Editor", None, None, None) is window + + +def test_a_scoped_search_does_not_match_the_window_itself(backend, search): + window = _element(name="Editor") + search["roots"] = [window] + assert backend._find_raw("Editor", None, None, None, + window_title="Editor") is None + + +def test_a_search_can_match_by_automation_id(backend, search): + window = _element(name="Editor") + wanted = _element(name="", automation_id="okButton", cached=True) + search["roots"] = [window] + search["under"][id(window)] = [wanted] + assert backend._find_raw(None, None, None, "okButton") is wanted + + +def test_a_search_by_automation_id_ignores_a_different_one(backend, search): + window = _element(name="Editor") + search["roots"] = [window] + search["under"][id(window)] = [ + _element(name="OK", automation_id="other", cached=True), + ] + assert backend._find_raw("OK", None, None, "okButton") is None + + +def test_a_search_can_match_a_substring(backend, search): + window = _element(name="Window") + wanted = _element(name="OK and close", cached=True) + search["roots"] = [window] + search["under"][id(window)] = [wanted] + assert backend._find_raw("ok and", None, None, None, + contains=True) is wanted + assert backend._find_raw("ok and", None, None, None) is None + + +def test_a_scoped_search_is_allowed_to_go_deeper(backend, search): + # Naming a window says "it is in here, find it"; not naming one says + # "look around", and stays cheap. A browser window holds thousands of + # nodes and a real target can sit well past any small cap. + window = _element(name="Editor") + search["roots"] = [window] + backend._find_raw("nothing", None, None, None) + unscoped = search["budgets"][0] + search["budgets"].clear() + backend._find_raw("nothing", None, None, None, window_title="Editor") + assert search["budgets"][0] > unscoped + + +def test_a_search_that_fails_on_the_provider_answers_none(backend, search): + search["roots"] = [_element(name="Editor")] + search["walk_error"] = OSError("provider stopped responding") + assert backend._find_raw("OK", None, None, None) is None + + +def test_a_search_of_an_empty_desktop_answers_none(backend, search): + assert backend._find_raw("OK", None, None, None) is None + + +def test_a_search_gives_up_rather_than_walking_the_whole_desktop(backend, + search): + # Unbounded, a target that is not there walks every window on the + # desktop and costs about a minute to answer "no". + first, second = _element(name="one"), _element(name="two") + search["roots"] = [first, second] + search["under"][id(first)] = [ + _element(name=f"c{index}", cached=True) + for index in range(backend_module._FIND_SCAN_LIMIT + 1) + ] + assert backend._find_raw("nothing", None, None, None) is None + assert len(search["budgets"]) == 1, "the second window was never walked" + + +def test_an_element_that_cannot_be_converted_never_matches(backend): + broken = RawElement(cached=True) + del broken.CachedName + assert backend._raw_matches(broken, {"name": "OK"}, cached=True) is False + + +# --- converting an element ---------------------------------------------------- + +def test_an_element_carries_its_name_role_and_geometry(): + raw = _element(name="OK", control_type=50000, + rect=Rect(10, 20, 110, 70), process_id=7, + automation_id="okButton") + element = _convert_uia(raw) + assert element.name == "OK" + assert element.role == "ControlType_50000" + assert element.bounds == (10, 20, 100, 50) + assert element.native_id == "okButton" + assert element.process_id == 7 + + +def test_a_role_is_reported_as_the_raw_control_type(): + # Translation to a friendly name is a separate step on purpose; the + # element carries what UIA said. + assert _convert_uia(_element(control_type=50004)).role == "ControlType_50004" + + +def test_the_owning_application_is_named_from_its_pid(): + assert _convert_uia(_element(process_id=7)).app_name == "app7.exe" + + +def test_a_rectangle_that_is_inside_out_reads_as_no_size(): + # An offscreen or collapsed control reports right < left; a negative + # width would put the element's centre outside the screen. + element = _convert_uia(_element(rect=Rect(100, 100, 10, 10))) + assert element.bounds == (100, 100, 0, 0) + + +def test_a_disabled_control_is_carried_as_disabled(): + assert _convert_uia(_element(enabled=False)).enabled is False + + +def test_a_cached_element_is_read_from_its_cached_properties(): + # The whole cost of a desktop listing is `Current*` reads, one + # cross-process call each; cached elements carry them already. + raw = _element(name="OK", cached=True) + assert _convert_uia(raw, cached=True).name == "OK" + assert _convert_uia(raw, cached=False) is None, "no Current* on it" + + +def test_an_element_that_went_away_mid_conversion_is_none(): + raw = _element(name="OK") + del raw.CurrentName + assert _convert_uia(raw) is None + + +def test_an_element_with_no_name_converts_with_an_empty_one(): + raw = RawElement(name=None, control_type=None, rect=Rect(), + process_id=None, automation_id=None, enabled=False) + element = _convert_uia(raw) + assert (element.name, element.role) == ("", "ControlType_0") + + +def test_a_name_read_that_fails_reads_as_empty(): + raw = _element(name="OK") + del raw.CurrentName + assert _safe_name(raw) == "" + + +def test_a_name_is_read_from_the_current_property(): + assert _safe_name(_element(name="OK")) == "OK" + + +# --- the rich property read --------------------------------------------------- + +def test_the_rich_properties_are_read_one_by_one(): + raw = types.SimpleNamespace( + CurrentIsEnabled=True, CurrentIsOffscreen=False, + CurrentHelpText="press me", CurrentItemStatus="busy", + CurrentAcceleratorKey="Ctrl+S", CurrentAccessKey="s", + CurrentOrientation=1, + ) + assert _read_properties(raw) == { + "enabled": True, "offscreen": False, "help_text": "press me", + "item_status": "busy", "accelerator_key": "Ctrl+S", + "access_key": "s", "orientation": 1, + } + + +def test_a_property_the_provider_will_not_answer_reads_as_none(): + # Each key is present either way: a missing key and a null value are + # different answers to the caller. + assert _read_properties(types.SimpleNamespace()) == { + "enabled": None, "offscreen": None, "help_text": None, + "item_status": None, "accelerator_key": None, "access_key": None, + "orientation": None, + } + + +def test_a_property_of_the_wrong_type_reads_as_none(): + raw = types.SimpleNamespace(CurrentOrientation="sideways") + assert _read_properties(raw)["orientation"] is None + + +def test_get_properties_needs_a_control_to_read_them_from(backend, listing): + assert backend.get_properties(name="OK") is None + + +def test_get_properties_reads_the_control_it_found(backend, listing): + window = _element(name="Editor") + window.CurrentHelpText = "the editor" + listing["roots"] = [window] + assert backend.get_properties(name="Editor")["help_text"] == "the editor" + + +# --- naming a process --------------------------------------------------------- + +@pytest.mark.parametrize("pid", [0, -1]) +def test_a_process_id_that_is_not_one_has_no_name(pid): + assert _process_name(pid) == "" + + +@pytest.mark.skipif(sys.platform == "win32", + reason="on Windows the real lookup runs") +def test_off_windows_no_process_lookup_is_attempted(): + assert _process_name(4321) == "" + + +@pytest.mark.skipif(sys.platform != "win32", reason="Win32 API") +def test_this_interpreters_own_process_names_itself(): + import os + assert _process_name(os.getpid()).lower().startswith("python") + + +@pytest.mark.skipif(sys.platform != "win32", reason="Win32 API") +def test_a_process_that_is_not_there_has_no_name(): + # OpenProcess fails and the name is empty rather than an exception out of + # the middle of a desktop listing. + assert _process_name(0x7FFFFFFF) == "" + + +def test_the_process_name_lookup_is_cached(): + # A desktop listing asks about the same handful of pids thousands of + # times, and each miss is three Win32 round trips. + assert hasattr(_process_name, "cache_info") + before = _process_name.cache_info() + _process_name(0) + _process_name(0) + after = _process_name.cache_info() + assert after.hits > before.hits From 3fbc0a22ddd2dbcbed03fddfaa4db8c3c5c4db22 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 08:03:14 +0800 Subject: [PATCH 13/20] Test the hotkey backends, and stop X11 leaking the key you rebound utils/hotkey/backends was the last backend seam with a real hole -- ocr, vision, llm and agent were already within single digits, and accessibility and window management landed in the last two commits. Three platforms, three unrelated mechanisms: RegisterHotKey plus a message pump, XGrabKey on the root window, a CGEventTap on a run loop. 12.64%, 24.46% and 40.94% all go to 100%, and the selector with them. Whole project 81.89% -> 82.40% on one machine. None of them needed a desktop. The Windows backend takes user32 as an argument to the three methods that do the work, so a recorder drives them anywhere; only the prologue that builds it needs ctypes.wintypes, and those two tests say so. The other two import their libraries inside their loops. The X11 one was leaking. `_sync_one` dropped a changed binding's previous registration from its own table and never called ungrab_key, so the old combo stayed grabbed on the server: swallowed from every application, firing nothing, and impossible for _ungrab_all to release at shutdown because it no longer knew about it. Rebind ctrl+alt+k and ctrl+alt+k is dead system-wide until the process exits. The module already knew the shape of that mistake -- the rollback in _grab_masked carries a comment about leaving grabs held with _registered never updated -- and the Windows backend unregisters at exactly this point. Only the rebinding path was missed. These tests grew the Xlib stub from 12 constants to 18 plus a keysym table. All of them are still compared against the installed library on the Linux squares, and against python-Xlib 0.33 here: 31 for 31. --- CHANGELOG.md | 10 + Progress.md | 27 +- WHATS_NEW.md | 33 ++ architecture_explore.md | 10 +- .../utils/hotkey/backends/linux_backend.py | 8 + test/unit_test/headless/_xlib_stub.py | 92 +++- .../headless/test_hotkey_backend_linux.py | 367 +++++++++++++ .../headless/test_hotkey_backend_win_mac.py | 508 ++++++++++++++++++ .../headless/test_xlib_stub_values.py | 17 + 9 files changed, 1063 insertions(+), 9 deletions(-) create mode 100644 test/unit_test/headless/test_hotkey_backend_linux.py create mode 100644 test/unit_test/headless/test_hotkey_backend_win_mac.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 27275e85..e22a5e06 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -387,6 +387,16 @@ only when documented here with a migration path. ### Fixed +- **Changing a hotkey's combo on X11 left the old key grabbed for the life of + the daemon.** `LinuxHotkeyBackend._sync_one` dropped the previous + registration from its own table without calling `ungrab_key`, so the *old* + combo stayed grabbed on the X server: it was swallowed from every + application, fired nothing, and `_ungrab_all` could not release it at + shutdown because it no longer knew about it. Rebinding `ctrl+alt+k` to + something else made `ctrl+alt+k` dead system-wide until the process exited. + The Windows backend has always unregistered at the same point; the X11 one + now does too. Unaffected on Windows and macOS. + - **A window closing mid-call let a COM error escape every Windows accessibility read.** `comtypes` reports a provider failure as `COMError`, which derives straight from `Exception` — the reason diff --git a/Progress.md b/Progress.md index 957c0d58..a7d88ba1 100644 --- a/Progress.md +++ b/Progress.md @@ -383,7 +383,7 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl | `utils/usb` | 573 / 2,137 | 73.2% | | `utils/mcp_server` | 573 / 4,618 | 87.6% | | ~~`wrapper/window_backends`~~ | ~~361 / 477~~ | **2026-08-24 補完（100%）** | -| `utils/hotkey` | 221 / 426 | 48.1% | +| ~~`utils/hotkey`~~ | ~~221 / 426~~ | **2026-08-24 backends 全數 100%** | | `utils/rest_api` | 146 / 808 | 81.9% | **要動地板，只能補在每一格都跑得到的程式碼上。** 地板取的是最低那一格， @@ -539,6 +539,31 @@ control pattern——寫的是 `(OSError, AttributeError, …)`，一個都攔 race 很小但真實：`_find_raw` 自己會攔，所以曝露的是「找到之後、讀之前」 那一段。已全部改用同一個 tuple；這只會**放寬**攔截範圍。 +#### 2026-08-24：`utils/hotkey/backends` 補完，**所有 backend seam 都有覆蓋了** + +`CLAUDE.md` 列的 backend seam 有八個（accessibility／ocr／vision／llm／agent／ +hotkey／usb／usbip）。ocr、vision、llm、agent 本來就只差個位數， +accessibility 與 window 這兩天補掉，剩下的就是 hotkey：三個平台、三套完全 +不同的機制（`RegisterHotKey` + 訊息幫浦／`XGrabKey`／`CGEventTap` + run loop）， +12.64%／24.46%／40.94% **全部到 100%**。 + +一樣都不需要桌面。Windows 那支把 `user32` 當**參數**傳給真正做事的三個方法， +所以錄音機式的替身在哪裡都能驅動；只有組出 `user32` 的開頭需要 +`ctypes.wintypes`，那兩支測試標成只在 Windows 跑。 + +**又抓到一個真的錯，這次是資源洩漏。** X11 的 `_sync_one` 在 combo 改掉時 +只把舊登記從自己的表裡拿掉，**沒有 `ungrab_key`**——舊的組合鍵於是一直被 +grab 在 X server 上，被所有應用程式吞掉、什麼也不觸發，而且 `_ungrab_all` +關機時也放不掉（它已經不知道那筆了）。使用者把 `ctrl+alt+k` 改成別的， +`ctrl+alt+k` 就在整個桌面上死到行程結束為止。 + +同一個檔案其實知道這個形狀——`_grab_masked` 的回滾註解就寫著「留著不放、 +`_registered` 又沒更新，會洩漏 grab 並在每次輪詢時噴 BadAccess」——而 Windows +那支一直都在同一個位置 unregister。漏的只有改綁定這條路。 + +X11 那批測試也把 Xlib stub 從 12 個常數長到 18 個外加一張 keysym 表， +全部照舊在 Linux 兩格對真貨比對（本機也用 python-Xlib 0.33 實測過：31 中 31）。 + ### mypy：整包把關，**豁免清單已經清空** `TODO` → **完成（2026-08-22）** diff --git a/WHATS_NEW.md b/WHATS_NEW.md index 851b9ea2..39521aa7 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -207,6 +207,39 @@ je_auto_control.windows.window import windows_window_manage` resolves by double under its own dotted name does nothing on a machine where the real module is importable — the package it is read off has to be the double. +### Every Backend Seam Is Covered Now, And The Last One Was Hiding A Leak + +`utils/hotkey/backends/` was the last of the project's backend seams with a +real hole in it — the accessibility, window, OCR, vision, LLM and agent seams +are all covered. Three platforms, three completely different mechanisms: +`RegisterHotKey` and a message pump, `XGrabKey` on the root window, a +`CGEventTap` on a run loop. All three go to 100%, from 12.64%, 24.46% and +40.94%. + +None of them needed a desktop. The Windows backend takes `user32` as an +*argument* to the three methods that do the work, so a recorder drives them +from anywhere; only the prologue that builds it needs `ctypes.wintypes`, and +those two tests say so. The other two import their platform libraries inside +their loops. + +**Changing a hotkey's combo on X11 left the old key grabbed for the life of +the daemon.** `_sync_one` dropped the previous registration from its own +table and never called `ungrab_key`, so the old combo stayed grabbed on the X +server: swallowed from every application, firing nothing, and impossible for +`_ungrab_all` to release at shutdown because it no longer knew about it. +Rebind `ctrl+alt+k` and `ctrl+alt+k` is dead system-wide until the process +exits. + +The same module already knew the shape of that mistake — the rollback in +`_grab_masked` carries a comment saying that leaving grabs held with +`_registered` never updated "leaks the grab and spams BadAccess on every +following poll" — and the Windows backend unregisters at exactly this point. +Only the rebinding path was missed. + +The X11 tests are also what grew the Xlib stub from 12 constants to 18 plus a +keysym table, all of them still compared against the installed library on the +Linux squares (and verified here against python-Xlib 0.33: 31 for 31). + ### The Accessibility Backends, Including The One Nobody Could Reach `utils/accessibility` was the second-largest gap in the project and the one diff --git a/architecture_explore.md b/architecture_explore.md index 5f1a4a63..142b2078 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -20,7 +20,7 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | | Python 模組總數（含周邊子專案） | 1,032 | -| 程式碼總行數 | 141,339 | +| 程式碼總行數 | 141,347 | | `je_auto_control/utils/` 子套件數 | 310 | | `AC_*` 動作指令數（`known_commands()` 實測） | 773 | | 套件門面 `__all__` 公開名稱數 | 1,238 | @@ -320,11 +320,11 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.3 排程、觸發與背景監看 -> 11 個套件、約 3,554 行。 +> 11 個套件、約 3,562 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | -| `utils/hotkey/` | 727 | 全域熱鍵守護行程，把 OS 層熱鍵綁到 action 檔（Win/macOS/X11 三後端） | +| `utils/hotkey/` | 735 | 全域熱鍵守護行程，把 OS 層熱鍵綁到 action 檔（Win/macOS/X11 三後端） | | `utils/idle_keepawake/` | 212 | 偵測使用者閒置時間並在無人值守執行期間阻止系統睡眠 | | `utils/lock_session/` | 163 | 鎖定工作站、等待解鎖並分類鎖定狀態轉換 | | `utils/observer/` | 220 | 反應式畫面觀察者，在出現／消失／變化時觸發 | @@ -1044,7 +1044,7 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `utils/assertion/` | 3 | 863 | | `osx/` | 17 | 915 | | `autocontrol-lsp/` | 8 | 744 | -| `utils/hotkey/` | 7 | 727 | +| `utils/hotkey/` | 7 | 735 | | 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 673 | 47,683 | -| **總計** | **1,026** | **141,274** | +| **總計** | **1,026** | **141,282** | diff --git a/je_auto_control/utils/hotkey/backends/linux_backend.py b/je_auto_control/utils/hotkey/backends/linux_backend.py index 2066d17a..db061ba2 100644 --- a/je_auto_control/utils/hotkey/backends/linux_backend.py +++ b/je_auto_control/utils/hotkey/backends/linux_backend.py @@ -106,6 +106,14 @@ def _sync_one(self, root, binding: HotkeyBinding) -> None: if prior is not None and prior[0] == binding.combo: return if prior is not None: + # Release the old key before forgetting it. Dropping it from + # `_registered` alone leaves the grab held on the server for the + # life of the process — the *previous* combo keeps being consumed + # from every application and fires nothing, and `_ungrab_all` + # cannot release what it no longer knows about. The Windows + # backend has always unregistered here. + _combo, prior_mask, prior_keycode = prior + self._ungrab_masked(root, prior_keycode, prior_mask) self._registered.pop(binding.binding_id, None) try: mask, keycode = _combo_to_x11(binding.combo) diff --git a/test/unit_test/headless/_xlib_stub.py b/test/unit_test/headless/_xlib_stub.py index 2ca08d2c..a6fce2d1 100644 --- a/test/unit_test/headless/_xlib_stub.py +++ b/test/unit_test/headless/_xlib_stub.py @@ -30,17 +30,38 @@ # --- X.h ----------------------------------------------------------------- -#: `X.h` event masks and sentinels the window backend actually names. +#: `X.h` event masks, modifier masks and sentinels the backends name. X_CONSTANTS = { "NONE": 0, "CurrentTime": 0, "AnyPropertyType": 0, + "KeyPress": 2, "KeyPressMask": 1 << 0, "KeyReleaseMask": 1 << 1, "ButtonPressMask": 1 << 2, "ButtonReleaseMask": 1 << 3, "SubstructureNotifyMask": 1 << 19, "SubstructureRedirectMask": 1 << 20, + # Modifier masks, for the hotkey backend's grabs. The two lock masks are + # what it re-grabs each combo under, so a hotkey still fires with NumLock + # or CapsLock on. + "ShiftMask": 1 << 0, + "LockMask": 1 << 1, + "ControlMask": 1 << 2, + "Mod1Mask": 1 << 3, + "Mod2Mask": 1 << 4, + "Mod4Mask": 1 << 6, + "GrabModeSync": 0, + "GrabModeAsync": 1, +} + +#: Keysyms `XK.string_to_keysym` answers for, by their `keysymdef.h` values. +#: Latin-1 letters are their ASCII code point; the named keys are 0xFF00-range +#: function keysyms. +KEYSYMS = { + "a": 0x0061, "b": 0x0062, "k": 0x006B, "q": 0x0071, "z": 0x007A, + "1": 0x0031, "Return": 0xFF0D, "Tab": 0xFF09, "Escape": 0xFF1B, + "space": 0x0020, "F5": 0xFFC2, "Page_Up": 0xFF55, "Delete": 0xFFFF, } #: `Xatom.h` predefined atoms, by their fixed protocol numbers. @@ -82,6 +103,11 @@ def __init__(self, display, window_id: int, *, properties=None, self.parent_id = parent_id self.sent = [] self.mapped = None + self.attributes = {} + self.grabs = [] + self.ungrabs = [] + self.grab_errors = {} # (keycode, mask) -> exception + self.ungrab_errors = {} self.property_error = None self.tree_error = None self.geometry_error = None @@ -123,6 +149,30 @@ def map(self): def unmap(self): self.mapped = False + def change_attributes(self, **kwargs): + self.attributes = dict(kwargs) + + def grab_key(self, keycode, mask, owner_events, pointer_mode, key_mode): + error = self.grab_errors.get((keycode, mask)) + if error is not None: + raise error + self.grabs.append((keycode, mask, owner_events, pointer_mode, + key_mode)) + + def ungrab_key(self, keycode, mask): + if (keycode, mask) in self.ungrab_errors: + raise self.ungrab_errors[(keycode, mask)] + self.ungrabs.append((keycode, mask)) + + @property + def grabbed(self): + """The `(keycode, mask)` pairs currently held, in grab order.""" + held = [(keycode, mask) for keycode, mask, *_rest in self.grabs] + for pair in self.ungrabs: + if pair in held: + held.remove(pair) + return held + class Display: """One X connection: an atom table, a root, and a window table.""" @@ -134,6 +184,10 @@ class Display: def __init__(self) -> None: self.atoms = {} self.flushes = 0 + self.syncs = 0 + self.closed = False + self.events = [] + self.keycodes = {} # keysym -> keycode self._windows = {} self.root = Window(self, 1) self._windows[1] = self.root @@ -154,6 +208,21 @@ def create_resource_object(self, kind: str, window_id: int) -> Window: def flush(self) -> None: self.flushes += 1 + def sync(self) -> None: + self.syncs += 1 + + def close(self) -> None: + self.closed = True + + def keysym_to_keycode(self, keysym: int) -> int: + return self.keycodes.get(keysym, 0) + + def pending_events(self) -> int: + return len(self.events) + + def next_event(self): + return self.events.pop(0) + # -- test helpers -- def atom_name(self, atom: int): for name, value in self.atoms.items(): @@ -177,11 +246,16 @@ def set_root_property(self, name: str, value) -> None: self.root.properties[name] = value -def install(monkeypatch, display=None) -> Display: +def install(monkeypatch, display=None, display_error=None) -> Display: """Put the stub in `sys.modules` and hand back the display it opens. Shadows the real `Xlib` for the duration wherever one is installed, so the same test reads the same on every square. + + `display_error` makes `Display()` raise while leaving the rest of the + package importable, which is the shape of a Wayland or headless session: + python-Xlib is a hard dependency on Linux, so it is always *there* — it + is the connection that is not. """ display = display if display is not None else Display() @@ -193,6 +267,11 @@ def install(monkeypatch, display=None) -> Display: for name, value in XATOM_CONSTANTS.items(): setattr(xatom_module, name, value) + # `XK.string_to_keysym` answers 0 for a name X does not know, which is + # the "unknown key" branch the hotkey backend has to report. + xk_module = types.ModuleType("Xlib.XK") + xk_module.string_to_keysym = lambda name: KEYSYMS.get(name, 0) + event_module = types.SimpleNamespace( ClientMessage=lambda **kwargs: Event("ClientMessage", **kwargs), KeyPress=lambda **kwargs: Event("KeyPress", **kwargs), @@ -203,17 +282,24 @@ def install(monkeypatch, display=None) -> Display: protocol_module = types.ModuleType("Xlib.protocol") protocol_module.event = event_module + def _open_display(*args, **kwargs): + if display_error is not None: + raise display_error + return display + display_module = types.ModuleType("Xlib.display") - display_module.Display = lambda *args, **kwargs: display + display_module.Display = _open_display package = types.ModuleType("Xlib") package.X = x_module package.Xatom = xatom_module + package.XK = xk_module package.protocol = protocol_module package.display = display_module for name, module in (("Xlib", package), ("Xlib.X", x_module), ("Xlib.Xatom", xatom_module), + ("Xlib.XK", xk_module), ("Xlib.protocol", protocol_module), ("Xlib.display", display_module)): monkeypatch.setitem(sys.modules, name, module) diff --git a/test/unit_test/headless/test_hotkey_backend_linux.py b/test/unit_test/headless/test_hotkey_backend_linux.py new file mode 100644 index 00000000..1e05f243 --- /dev/null +++ b/test/unit_test/headless/test_hotkey_backend_linux.py @@ -0,0 +1,367 @@ +"""Grabbing a hotkey on X11, from any square rather than only Linux. + +`utils/hotkey/backends/` is the last of the project's backend seams with a +real hole in it -- accessibility and window management are covered now, and +the OCR / vision / LLM / agent seams were already close to full. This is the +X11 half: 138 statements at 24%. + +The same fact makes it reachable: every `Xlib` import is inside a function, +so a stub in `sys.modules` (`_xlib_stub.py`) drives the whole thing from all +nine CI squares. + +What the grab has to get right, and what breaks silently if it does not: + +* **A hotkey is grabbed four times, not once.** X reports NumLock and + CapsLock as modifier bits in the event state, and a grab registered + without them simply does not match while either is on. So every combo is + registered under all four lock combinations, and the match ignores those + same bits again on the way back. +* **A failed grab has to roll back the ones that already took.** X refuses a + grab another client already holds, and it refuses it per lock-variant -- + so a combo can be half-grabbed. Leaving those held, with `_registered` + never updated, leaks the grab and spams `BadAccess` on every following + poll. +* **The match is exact on the modifiers, not a superset.** `ctrl+k` must not + fire on `ctrl+shift+k`: the state has to equal the mask once the lock bits + are masked out, or every binding shadows the ones above it. + +The stub's constants and keysyms carry their real values, and +`test_xlib_stub_values.py` compares every one of them against the installed +library on the Linux squares. +""" +from __future__ import annotations + +import threading + +import pytest + +from headless import _xlib_stub +from je_auto_control.utils.hotkey.backends.linux_backend import ( + LinuxHotkeyBackend, _combo_to_x11, _lock_all_mask, _lock_mask_variants, +) +from je_auto_control.utils.hotkey.hotkey_daemon import ( + BackendContext, HotkeyBinding, +) + +_X = _xlib_stub.X_CONSTANTS +CTRL = _X["ControlMask"] +SHIFT = _X["ShiftMask"] +ALT = _X["Mod1Mask"] +SUPER = _X["Mod4Mask"] +NUM_LOCK = _X["Mod2Mask"] +CAPS_LOCK = _X["LockMask"] + +#: What `_combo_to_x11` resolves "k" to, given the keycodes installed below. +K_KEYCODE = 45 + + +@pytest.fixture +def display(monkeypatch): + """An X display that resolves the keysyms these combos use.""" + fake = _xlib_stub.install(monkeypatch) + fake.keycodes = { + _xlib_stub.KEYSYMS["k"]: K_KEYCODE, + _xlib_stub.KEYSYMS["q"]: 24, + _xlib_stub.KEYSYMS["Return"]: 36, + _xlib_stub.KEYSYMS["F5"]: 71, + _xlib_stub.KEYSYMS["Page_Up"]: 112, + } + return fake + + +@pytest.fixture +def backend(): + return LinuxHotkeyBackend() + + +def _binding(binding_id="b1", combo="ctrl+alt+k"): + return HotkeyBinding(binding_id=binding_id, combo=combo, + script_path="script.json") + + +def _context(bindings, fired=None, stop_after=1): + """A context whose stop event trips after `stop_after` polls.""" + stop = threading.Event() + state = {"polls": 0} + + def _get_bindings(): + state["polls"] += 1 + if state["polls"] >= stop_after: + stop.set() + return list(bindings) + + return BackendContext(stop_event=stop, get_bindings=_get_bindings, + fire=(fired if fired is not None else []).append) + + +# --- turning a combo into a grab ---------------------------------------------- + +@pytest.mark.parametrize("combo,mask", [ + ("k", 0), + ("ctrl+k", CTRL), + ("shift+k", SHIFT), + ("alt+k", ALT), + ("win+k", SUPER), + ("ctrl+alt+k", CTRL | ALT), + ("ctrl+shift+alt+win+k", CTRL | SHIFT | ALT | SUPER), +]) +def test_a_combo_becomes_an_x11_modifier_mask(display, combo, mask): + assert _combo_to_x11(combo) == (mask, K_KEYCODE) + + +@pytest.mark.parametrize("combo,keycode", [ + ("ctrl+enter", 36), ("ctrl+return", 36), ("ctrl+f5", 71), + ("ctrl+pageup", 112), +]) +def test_a_named_key_resolves_through_its_keysym(display, combo, keycode): + # "pageup" is not what X calls it -- `Page_Up` is -- so the table in the + # module is the only thing between the user's spelling and a grab. + assert _combo_to_x11(combo)[1] == keycode + + +def test_a_single_character_key_needs_no_table_entry(display): + assert _combo_to_x11("ctrl+Q")[1] == 24, "upper case is lowered first" + + +def test_a_multi_character_key_x_does_not_know_is_refused(display): + with pytest.raises(ValueError, match="unsupported hotkey key"): + _combo_to_x11("ctrl+mediaplay") + + +def test_a_key_with_no_keysym_is_refused(display): + # `string_to_keysym` answers 0 for a name X has never heard of; grabbing + # keycode 0 would register a hotkey on nothing. + with pytest.raises(ValueError, match="unknown X keysym"): + _combo_to_x11("ctrl+¥") + + +def test_resolving_a_combo_closes_the_display_it_opened(display): + _combo_to_x11("ctrl+k") + assert display.closed is True + + +# --- the lock-mask variants --------------------------------------------------- + +def test_a_hotkey_is_grabbed_under_every_lock_combination(display): + # X reports NumLock and CapsLock as modifier bits, and a grab registered + # without them does not match while either is on. + assert _lock_mask_variants() == [0, NUM_LOCK, CAPS_LOCK, + NUM_LOCK | CAPS_LOCK] + + +def test_the_lock_bits_are_masked_out_of_the_event_state(display): + assert _lock_all_mask() == NUM_LOCK | CAPS_LOCK + + +def test_without_xlib_the_variants_degrade_to_the_plain_grab(monkeypatch): + import sys + monkeypatch.setitem(sys.modules, "Xlib", None) + assert _lock_mask_variants() == [0] + assert _lock_all_mask() == 0 + + +# --- registering and unregistering -------------------------------------------- + +def test_a_new_binding_is_grabbed_four_times(backend, display): + backend._sync(display, display.root, [_binding()]) + assert display.root.grabbed == [ + (K_KEYCODE, CTRL | ALT | extra) for extra in _lock_mask_variants() + ] + + +def test_a_grab_owns_the_key_rather_than_observing_it(backend, display): + # `owner_events=True` with async modes is what consumes the key, matching + # Windows `RegisterHotKey` semantics. + backend._sync(display, display.root, [_binding()]) + _keycode, _mask, owner_events, pointer, keyboard = display.root.grabs[0] + assert owner_events is True + assert pointer == keyboard == _X["GrabModeAsync"] + + +def test_a_binding_that_has_not_changed_is_not_re_grabbed(backend, display): + bindings = [_binding()] + backend._sync(display, display.root, bindings) + before = len(display.root.grabs) + backend._sync(display, display.root, bindings) + assert len(display.root.grabs) == before + + +def test_a_binding_whose_combo_changed_releases_the_old_key(backend, + display): + # Until 2026-08-24 the old grab was only forgotten, never released: + # the previous combo stayed consumed from every application, fired + # nothing, and `_ungrab_all` could not free what it no longer knew + # about. The Windows backend has always unregistered here. + backend._sync(display, display.root, [_binding(combo="ctrl+k")]) + backend._sync(display, display.root, [_binding(combo="ctrl+q")]) + assert (24, CTRL) in display.root.grabbed + assert (K_KEYCODE, CTRL) not in display.root.grabbed + assert backend._registered["b1"][0] == "ctrl+q" + + +def test_a_binding_that_disappeared_is_ungrabbed(backend, display): + backend._sync(display, display.root, [_binding()]) + backend._sync(display, display.root, []) + assert display.root.grabbed == [] + assert backend._registered == {} + + +def test_an_unparseable_combo_is_logged_and_skipped(backend, display): + backend._sync(display, display.root, + [_binding(combo="ctrl+mediaplay"), _binding("b2", "ctrl+k")]) + assert list(backend._registered) == ["b2"], "the good one still took" + + +def test_a_refused_grab_rolls_back_the_variants_that_took(backend, display): + # X refuses per lock-variant, so a combo can be half-grabbed. Leaving + # those held with `_registered` never updated leaks the grab and spams + # BadAccess on every following poll. + display.root.grab_errors[(K_KEYCODE, CTRL | ALT | CAPS_LOCK)] = ( + RuntimeError("BadAccess")) + backend._sync(display, display.root, [_binding()]) + assert display.root.grabbed == [] + assert backend._registered == {} + + +def test_a_rollback_that_itself_fails_does_not_escape(backend, display): + display.root.grab_errors[(K_KEYCODE, CTRL | ALT | NUM_LOCK)] = ( + RuntimeError("BadAccess")) + display.root.ungrab_errors[(K_KEYCODE, CTRL | ALT)] = ( + RuntimeError("BadWindow")) + backend._sync(display, display.root, [_binding()]) + assert backend._registered == {} + + +def test_an_ungrab_that_fails_is_not_fatal(backend, display): + backend._sync(display, display.root, [_binding()]) + display.root.ungrab_errors[(K_KEYCODE, CTRL | ALT)] = ( + RuntimeError("BadWindow")) + backend._sync(display, display.root, []) + assert backend._registered == {} + + +def test_a_sync_flushes_the_grabs_to_the_server(backend, display): + # X buffers requests; grabs that are never synced are grabs the server + # has not made yet. + before = display.syncs + backend._sync(display, display.root, [_binding()]) + assert display.syncs > before + + +# --- matching an event back to a binding -------------------------------------- + +def _key_press(keycode=K_KEYCODE, state=CTRL | ALT): + import types as _types + return _types.SimpleNamespace(type=_X["KeyPress"], detail=keycode, + state=state) + + +def test_a_matching_key_press_fires_its_binding(backend, display): + backend._sync(display, display.root, [_binding()]) + display.events = [_key_press()] + fired = [] + backend._drain(display, fired.append) + assert fired == ["b1"] + + +@pytest.mark.parametrize("extra", [NUM_LOCK, CAPS_LOCK, NUM_LOCK | CAPS_LOCK]) +def test_a_hotkey_fires_with_the_locks_on(backend, display, extra): + backend._sync(display, display.root, [_binding()]) + display.events = [_key_press(state=CTRL | ALT | extra)] + fired = [] + backend._drain(display, fired.append) + assert fired == ["b1"] + + +def test_a_press_with_an_extra_modifier_does_not_fire(backend, display): + # `ctrl+k` must not fire on `ctrl+shift+k`, or every binding shadows the + # ones with more modifiers. + backend._sync(display, display.root, [_binding(combo="ctrl+k")]) + display.events = [_key_press(state=CTRL | SHIFT)] + fired = [] + backend._drain(display, fired.append) + assert fired == [] + + +def test_a_press_with_a_missing_modifier_does_not_fire(backend, display): + backend._sync(display, display.root, [_binding()]) + display.events = [_key_press(state=CTRL)] + fired = [] + backend._drain(display, fired.append) + assert fired == [] + + +def test_a_press_of_another_key_does_not_fire(backend, display): + backend._sync(display, display.root, [_binding()]) + display.events = [_key_press(keycode=99)] + fired = [] + backend._drain(display, fired.append) + assert fired == [] + + +def test_an_event_that_is_not_a_key_press_is_ignored(backend, display): + import types as _types + backend._sync(display, display.root, [_binding()]) + display.events = [_types.SimpleNamespace(type=99, detail=K_KEYCODE, + state=CTRL | ALT)] + fired = [] + backend._drain(display, fired.append) + assert fired == [] + + +def test_every_queued_event_is_drained_in_one_poll(backend, display): + backend._sync(display, display.root, [_binding()]) + display.events = [_key_press(), _key_press(), _key_press()] + fired = [] + backend._drain(display, fired.append) + assert fired == ["b1", "b1", "b1"] + assert display.pending_events() == 0 + + +# --- the run loop ------------------------------------------------------------- + +def test_the_loop_asks_the_root_for_key_presses(backend, display): + backend.run_forever(_context([_binding()])) + assert display.root.attributes == {"event_mask": _X["KeyPressMask"]} + + +def test_the_loop_registers_polls_and_tears_down(backend, display): + fired = [] + backend.run_forever(_context([_binding()], fired)) + assert display.root.grabs, "it grabbed while running" + assert display.root.grabbed == [], "and released everything on the way out" + assert backend._registered == {} + assert display.closed is True + + +def test_the_loop_fires_a_hotkey_pressed_while_it_runs(backend, display): + display.events = [_key_press()] + fired = [] + backend.run_forever(_context([_binding()], fired)) + assert fired == ["b1"] + + +def test_a_session_with_no_display_gives_up_quietly(monkeypatch, backend): + # Wayland, or a daemon started outside a session. There is nothing to + # grab, and taking the daemon's thread down with an exception would take + # every other binding with it. + _xlib_stub.install(monkeypatch, + display_error=RuntimeError("no DISPLAY")) + backend.run_forever(_context([_binding()])) + assert backend._registered == {} + + +def test_a_teardown_ungrab_that_fails_does_not_escape(backend, display): + # X11 ungrab races are non-fatal: the window can already be gone. The + # daemon's thread is on its way out and must not raise there. + backend._sync(display, display.root, [_binding()]) + display.root.ungrab_errors = { + (K_KEYCODE, CTRL | ALT | extra): RuntimeError("BadWindow") + for extra in _lock_mask_variants() + } + backend._ungrab_all(display, display.root) + assert backend._registered == {} + + +def test_the_backend_names_the_protocol_it_uses(backend): + assert backend.name == "linux-x11" diff --git a/test/unit_test/headless/test_hotkey_backend_win_mac.py b/test/unit_test/headless/test_hotkey_backend_win_mac.py new file mode 100644 index 00000000..ee458383 --- /dev/null +++ b/test/unit_test/headless/test_hotkey_backend_win_mac.py @@ -0,0 +1,508 @@ +"""Registering a hotkey on Windows and on macOS, and picking a backend. + +The X11 half is in `test_hotkey_backend_linux.py`. These two do the same job +through completely different machinery -- `RegisterHotKey` plus a message +pump on Windows, a `CGEventTap` plus a run loop on macOS -- and both sat at +around 12% and 41%. + +Neither needed a desktop. The Windows backend takes `user32` as an *argument* +to the three methods that do the work, so a recorder drives them anywhere; +only the prologue that builds it needs `ctypes.wintypes`, and that part is +marked Windows-only. The macOS one imports Quartz and CoreFoundation inside +its loop, so stubs in `sys.modules` reach it. + +What the two share, and what these tests are about: + +* **Sync is a diff, not a re-registration.** A binding whose combo has not + changed must not be unregistered and registered again on every poll -- + that is a window during which the hotkey does not work, four times a + second. +* **A combo the parser rejects must not take the others down.** Bindings come + from a user-edited file; one bad line is normal. +* **The teardown has to release what it took.** An OS-level hotkey survives + the process that registered it on neither platform, but a leaked + registration inside a long-lived daemon means a key that is swallowed and + fires nothing. +* **macOS consumes the event by returning None from the tap** and defers the + actual firing to the polling thread, because the tap callback runs on the + run loop and calling a user script there would block every key on the + system until it finished. +""" +from __future__ import annotations + +import sys +import threading +import types + +import pytest + +from je_auto_control.utils.hotkey import backends as backends_mod +from je_auto_control.utils.hotkey.backends import macos_backend as mac +from je_auto_control.utils.hotkey.backends.macos_backend import ( + MacOSHotkeyBackend, _combo_to_macos, _primary_key_to_keycode, +) +from je_auto_control.utils.hotkey.backends.windows_backend import ( + WindowsHotkeyBackend, +) +from je_auto_control.utils.hotkey.hotkey_daemon import ( + BackendContext, HotkeyBinding, parse_combo, +) + + +def _binding(binding_id="b1", combo="ctrl+alt+k"): + return HotkeyBinding(binding_id=binding_id, combo=combo, + script_path="script.json") + + +def _context(bindings, fired=None, stop_after=1): + stop = threading.Event() + state = {"polls": 0} + + def _get_bindings(): + state["polls"] += 1 + if state["polls"] >= stop_after: + stop.set() + return list(bindings) + + return BackendContext(stop_event=stop, get_bindings=_get_bindings, + fire=(fired if fired is not None else []).append) + + +# --- the selector ------------------------------------------------------------- + +@pytest.fixture +def on_platform(monkeypatch): + def _set(name: str): + monkeypatch.setattr(backends_mod.sys, "platform", name) + return _set + + +@pytest.mark.parametrize("platform,expected", [ + ("win32", "windows"), ("darwin", "macos"), ("linux", "linux-x11"), + ("linux2", "linux-x11"), +]) +def test_each_platform_gets_its_own_backend(on_platform, platform, expected): + on_platform(platform) + assert backends_mod.get_backend().name == expected + + +def test_a_platform_with_no_hotkey_backend_says_so(on_platform): + # Unlike the accessibility and window seams there is no null backend + # here: a daemon that silently registered nothing would look like a + # daemon whose hotkeys never fire. + on_platform("sunos5") + with pytest.raises(NotImplementedError, match="sunos5"): + backends_mod.get_backend() + + +# --- Windows: RegisterHotKey -------------------------------------------------- + +class _User32: + """The three user32 entry points the backend calls, as a recorder.""" + + def __init__(self, register_result=True) -> None: + self.registered = [] + self.unregistered = [] + self.register_result = register_result + self.RegisterHotKey = _Stub(self._register) # noqa: N815 + self.UnregisterHotKey = _Stub(self._unregister) # noqa: N815 + self.PeekMessageW = _Stub(lambda *args: 0) # noqa: N815 + + def _register(self, hwnd, reg_id, modifiers, vk): + self.registered.append((reg_id, modifiers, vk)) + return 1 if self.register_result else 0 + + def _unregister(self, hwnd, reg_id): + self.unregistered.append(reg_id) + return 1 + + @property + def held(self): + """Registration ids still outstanding, in the order they were taken.""" + out = [reg_id for reg_id, _mods, _vk in self.registered] + for reg_id in self.unregistered: + if reg_id in out: + out.remove(reg_id) + return out + + +class _Stub: + """A user32 function pointer: callable, and it accepts argtypes/restype.""" + + def __init__(self, call) -> None: + self._call = call + self.argtypes = None + self.restype = None + + def __call__(self, *args): + return self._call(*args) + + +@pytest.fixture +def user32(): + return _User32() + + +@pytest.fixture +def win_backend(): + return WindowsHotkeyBackend() + + +def test_a_new_binding_is_registered_with_its_parsed_combo(win_backend, + user32): + win_backend._sync(user32, [_binding()]) + [(_reg_id, modifiers, vk)] = user32.registered + assert (modifiers, vk) == parse_combo("ctrl+alt+k") + + +def test_each_registration_gets_its_own_id(win_backend, user32): + win_backend._sync(user32, [_binding("b1", "ctrl+k"), + _binding("b2", "ctrl+q")]) + ids = [reg_id for reg_id, _m, _v in user32.registered] + assert len(set(ids)) == 2 + + +def test_a_binding_that_has_not_changed_is_not_re_registered(win_backend, + user32): + bindings = [_binding()] + win_backend._sync(user32, bindings) + win_backend._sync(user32, bindings) + assert len(user32.registered) == 1 + assert user32.unregistered == [], "the hotkey never stopped working" + + +def test_a_binding_whose_combo_changed_releases_the_old_key(win_backend, + user32): + win_backend._sync(user32, [_binding(combo="ctrl+k")]) + first_id = user32.registered[0][0] + win_backend._sync(user32, [_binding(combo="ctrl+q")]) + assert first_id in user32.unregistered + assert len(user32.held) == 1 + + +def test_a_binding_that_disappeared_is_unregistered(win_backend, user32): + win_backend._sync(user32, [_binding()]) + win_backend._sync(user32, []) + assert user32.held == [] + assert win_backend._registered == {} + + +def test_an_unparseable_combo_is_logged_and_skipped(win_backend, user32): + win_backend._sync(user32, [_binding(combo="ctrl+"), _binding("b2", "ctrl+k")]) + assert list(win_backend._registered) == ["b2"] + + +def test_a_registration_windows_refuses_is_not_remembered(win_backend): + # Another application already owns the combo. Remembering it would mean + # never retrying, and reporting it as bound would be a lie. + refusing = _User32(register_result=False) + win_backend._sync(refusing, [_binding()]) + assert win_backend._registered == {} + + +def test_a_hotkey_message_fires_the_binding_it_belongs_to(win_backend, + user32): + win_backend._sync(user32, [_binding("b1", "ctrl+k"), + _binding("b2", "ctrl+q")]) + second_id = user32.registered[1][0] + fired = [] + win_backend._dispatch(second_id, fired.append) + assert fired == ["b2"] + + +def test_a_hotkey_message_for_an_unknown_id_fires_nothing(win_backend, + user32): + win_backend._sync(user32, [_binding()]) + fired = [] + win_backend._dispatch(9999, fired.append) + assert fired == [] + + +@pytest.mark.skipif(sys.platform != "win32", + reason="ctypes.wintypes is Windows-only") +def test_the_message_pump_registers_and_releases(monkeypatch, win_backend): + import ctypes + recorder = _User32() + monkeypatch.setattr(ctypes, "WinDLL", + lambda name, **kwargs: recorder) + win_backend.run_forever(_context([_binding()])) + assert recorder.registered, "it registered while running" + assert recorder.held == [], "and released on the way out" + assert win_backend._registered == {} + + +@pytest.mark.skipif(sys.platform != "win32", + reason="ctypes.wintypes is Windows-only") +def test_a_hotkey_message_reaches_the_binding_through_the_pump(monkeypatch, + win_backend): + import ctypes + recorder = _User32() + messages = {"left": 1} + + def _peek(msg_pointer, hwnd, low, high, remove): + if not messages["left"]: + return 0 + messages["left"] -= 1 + msg = msg_pointer._obj + msg.message = 0x0312 # WM_HOTKEY + msg.wParam = recorder.registered[0][0] + return 1 + + recorder.PeekMessageW = _Stub(_peek) + monkeypatch.setattr(ctypes, "WinDLL", lambda name, **kwargs: recorder) + fired = [] + win_backend.run_forever(_context([_binding()], fired)) + assert fired == ["b1"] + + +@pytest.mark.skipif(sys.platform != "win32", + reason="ctypes.wintypes is Windows-only") +def test_a_message_that_is_not_a_hotkey_is_pumped_past(monkeypatch, + win_backend): + # The queue carries whatever Windows posts to the thread; only WM_HOTKEY + # means one of ours fired. + import ctypes + recorder = _User32() + messages = {"left": 2} + + def _peek(msg_pointer, hwnd, low, high, remove): + if not messages["left"]: + return 0 + messages["left"] -= 1 + msg = msg_pointer._obj + msg.message = 0x0113 if messages["left"] else 0x0312 # WM_TIMER + msg.wParam = recorder.registered[0][0] + return 1 + + recorder.PeekMessageW = _Stub(_peek) + monkeypatch.setattr(ctypes, "WinDLL", lambda name, **kwargs: recorder) + fired = [] + win_backend.run_forever(_context([_binding()], fired)) + assert fired == ["b1"], "the timer was skipped, the hotkey was not" + + +def test_the_windows_backend_names_the_api_it_uses(win_backend): + assert win_backend.name == "windows" + + +# --- macOS: CGEventTap -------------------------------------------------------- + +_FLAG_SHIFT = 1 << 17 +_FLAG_CONTROL = 1 << 18 +_FLAG_ALT = 1 << 19 +_FLAG_CMD = 1 << 20 + + +@pytest.mark.parametrize("combo,mask", [ + ("k", 0), + ("ctrl+k", _FLAG_CONTROL), + ("shift+k", _FLAG_SHIFT), + ("alt+k", _FLAG_ALT), + ("win+k", _FLAG_CMD), + ("ctrl+alt+k", _FLAG_CONTROL | _FLAG_ALT), +]) +def test_a_combo_becomes_a_quartz_flags_mask(combo, mask): + assert _combo_to_macos(combo)[0] == mask + + +@pytest.mark.parametrize("key,keycode", [ + ("a", 0), ("z", 6), ("m", 46), # letters + ("1", 18), ("6", 22), ("0", 29), # digits, in Carbon's odd order + ("return", 36), ("enter", 36), ("space", 49), ("esc", 53), + ("escape", 53), ("f1", 122), ("f12", 111), ("pageup", 116), + ("A", 0), ("F1", 122), # case does not matter +]) +def test_a_key_name_becomes_a_carbon_virtual_keycode(key, keycode): + assert _primary_key_to_keycode(key) == keycode + + +def test_a_key_carbon_has_no_code_for_is_refused(): + with pytest.raises(ValueError, match="unsupported hotkey key"): + _primary_key_to_keycode("mediaplay") + + +@pytest.fixture +def mac_backend(): + return MacOSHotkeyBackend() + + +def test_a_new_binding_is_remembered_with_its_mask_and_keycode(mac_backend): + mac_backend._sync([_binding()]) + assert mac_backend._registered == { + "b1": ("ctrl+alt+k", _FLAG_CONTROL | _FLAG_ALT, 40), + } + + +def test_a_binding_that_has_not_changed_is_left_alone(mac_backend): + bindings = [_binding()] + mac_backend._sync(bindings) + first = mac_backend._registered["b1"] + mac_backend._sync(bindings) + assert mac_backend._registered["b1"] is first + + +def test_a_binding_whose_combo_changed_is_replaced(mac_backend): + mac_backend._sync([_binding(combo="ctrl+k")]) + mac_backend._sync([_binding(combo="ctrl+q")]) + assert mac_backend._registered["b1"][0] == "ctrl+q" + + +def test_a_binding_that_disappeared_is_forgotten(mac_backend): + mac_backend._sync([_binding()]) + mac_backend._sync([]) + assert mac_backend._registered == {} + + +def test_an_unparseable_combo_is_logged_and_skipped_on_macos(mac_backend): + mac_backend._sync([_binding(combo="ctrl+mediaplay"), + _binding("b2", "ctrl+k")]) + assert list(mac_backend._registered) == ["b2"] + + +def test_a_matching_key_event_names_its_binding(mac_backend): + mac_backend._sync([_binding()]) + assert mac_backend._match(40, _FLAG_CONTROL | _FLAG_ALT) == "b1" + + +def test_an_event_with_extra_modifiers_does_not_match(mac_backend): + mac_backend._sync([_binding(combo="ctrl+k")]) + assert mac_backend._match(40, _FLAG_CONTROL | _FLAG_SHIFT) is None + + +def test_an_event_for_another_key_does_not_match(mac_backend): + mac_backend._sync([_binding()]) + assert mac_backend._match(99, _FLAG_CONTROL | _FLAG_ALT) is None + + +def test_pending_fires_are_drained_in_order(mac_backend): + mac_backend._pending_fires = ["b1", "b2"] + fired = [] + mac_backend._drain_fires(fired.append) + assert fired == ["b1", "b2"] + assert mac_backend._pending_fires == [] + + +# --- the macOS run loop ------------------------------------------------------- + +class _Quartz(types.ModuleType): + """The Quartz surface the tap setup touches.""" + + kCGKeyboardEventKeycode = "keycode" # noqa: N815 # reason: Quartz name + kCGEventKeyDown = 10 # noqa: N815 + kCGHIDEventTap = 0 # noqa: N815 + kCGHeadInsertEventTap = 0 # noqa: N815 + kCGEventTapOptionDefault = 0 # noqa: N815 + + def __init__(self, tap="tap") -> None: + super().__init__("Quartz") + self.tap = tap + self.enabled = [] + self.sources_added = [] + self.sources_removed = [] + self.callback = None + self.tap_mask = None + + def CGEventGetIntegerValueField(self, event, field): # noqa: N802 + return event["keycode"] + + def CGEventGetFlags(self, event): # noqa: N802 # reason: Quartz name + return event["flags"] + + def CGEventTapCreate(self, tap_point, place, options, mask, # noqa: N802 + callback, refcon): + self.callback = callback + self.tap_mask = mask + return self.tap + + def CFMachPortCreateRunLoopSource(self, allocator, tap, order): # noqa: N802 + return "source" + + def CFRunLoopGetCurrent(self): # noqa: N802 # reason: Quartz name + return "run-loop" + + def CFRunLoopAddSource(self, loop, source, mode): # noqa: N802 + self.sources_added.append((loop, source, mode)) + + def CFRunLoopRemoveSource(self, loop, source, mode): # noqa: N802 + self.sources_removed.append((loop, source, mode)) + + def CGEventTapEnable(self, tap, enable): # noqa: N802 + self.enabled.append(bool(enable)) + + +@pytest.fixture +def quartz(monkeypatch): + def _install(tap="tap"): + module = _Quartz(tap) + monkeypatch.setitem(sys.modules, "Quartz", module) + core = types.ModuleType("CoreFoundation") + core.kCFRunLoopDefaultMode = "default-mode" + core.CFRunLoopRunInMode = lambda mode, seconds, once: None + monkeypatch.setitem(sys.modules, "CoreFoundation", core) + return module + return _install + + +def test_the_tap_listens_for_key_down_only(mac_backend, quartz): + module = quartz() + mac_backend.run_forever(_context([_binding()])) + assert module.tap_mask == 1 << module.kCGEventKeyDown + + +def test_the_tap_is_enabled_while_running_and_disabled_after(mac_backend, + quartz): + module = quartz() + mac_backend.run_forever(_context([_binding()])) + assert module.enabled == [True, False] + assert module.sources_added and module.sources_removed + + +def test_a_mac_without_accessibility_permission_gives_up_with_a_reason( + mac_backend, quartz): + # `CGEventTapCreate` returns None rather than failing when the grant is + # missing, so this is the only signal there is. + module = quartz(tap=None) + mac_backend.run_forever(_context([_binding()])) + assert module.enabled == [], "nothing to enable, and nothing to remove" + + +def test_a_mac_without_pyobjc_gives_up_quietly(mac_backend, monkeypatch): + monkeypatch.setitem(sys.modules, "Quartz", None) + mac_backend.run_forever(_context([_binding()])) + assert mac_backend._registered == {} + + +def test_a_matching_key_is_consumed_and_queued(mac_backend, quartz): + # Returning None consumes the event; the fire is deferred to the polling + # thread because the callback runs on the run loop, where calling a user + # script would block every key on the system. + module = quartz() + fired = [] + mac_backend.run_forever(_context([_binding()], fired, stop_after=2)) + event = {"keycode": 40, "flags": _FLAG_CONTROL | _FLAG_ALT} + assert module.callback(None, None, event, None) is None + mac_backend._drain_fires(fired.append) + assert fired == ["b1"] + + +def test_a_key_that_matches_nothing_is_passed_through(mac_backend, quartz): + module = quartz() + mac_backend.run_forever(_context([_binding()])) + event = {"keycode": 99, "flags": 0} + assert module.callback(None, None, event, None) is event + + +def test_an_unrelated_modifier_bit_is_masked_off_before_matching(mac_backend, + quartz): + # macOS sets bits for caps lock, the numeric keypad and more; only the + # four modifier flags take part in the comparison. + module = quartz() + mac_backend.run_forever(_context([_binding()])) + event = {"keycode": 40, + "flags": _FLAG_CONTROL | _FLAG_ALT | (1 << 16) | (1 << 21)} + assert module.callback(None, None, event, None) is None + + +def test_the_macos_backend_names_the_api_it_uses(mac_backend): + assert mac_backend.name == "macos" + assert mac.MacOSHotkeyBackend is MacOSHotkeyBackend diff --git a/test/unit_test/headless/test_xlib_stub_values.py b/test/unit_test/headless/test_xlib_stub_values.py index 65c8a47b..35e51ab5 100644 --- a/test/unit_test/headless/test_xlib_stub_values.py +++ b/test/unit_test/headless/test_xlib_stub_values.py @@ -44,6 +44,23 @@ def test_every_predefined_atom_matches_the_installed_library(name): ) +@pytest.mark.parametrize("name", sorted(_xlib_stub.KEYSYMS)) +def test_every_keysym_matches_what_the_installed_library_resolves(name): + # `XK.string_to_keysym` is what turns "Page_Up" into the number a grab is + # registered against; a wrong one grabs the wrong key. + from Xlib import XK + assert _xlib_stub.KEYSYMS[name] == XK.string_to_keysym(name), ( + f"the stub's keysym for {name!r} is not what python-Xlib resolves" + ) + + +def test_an_unknown_key_name_resolves_to_zero(): + # Zero is how X says "no such keysym", and the hotkey backend reports it + # as an unsupported key rather than grabbing keycode 0. + from Xlib import XK + assert XK.string_to_keysym("not-a-key") == 0 + + def test_the_event_classes_the_stub_fakes_all_exist(): # A stub that answers for an event class the library does not have would # let a backend build a message nothing can send. From 2c947ba63dbb8000bf7a67b125cbb01ec3ad3120 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 08:04:13 +0800 Subject: [PATCH 14/20] Record where the coverage climb stands: 82.40 locally, floor waits on CI --- Progress.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/Progress.md b/Progress.md index a7d88ba1..c01141a0 100644 --- a/Progress.md +++ b/Progress.md @@ -244,9 +244,16 @@ capability enum 值與 variadic `ei_seat_bind_capabilities`、event-type enum **2026-08-23 維護者拍板：下一個覆蓋率目標是 80。** 這一條留到那時候。 型別那一節留著是因為它記的那幾個坑之後還會踩到。 -### 覆蓋率：地板 75，下一站 80 +### 覆蓋率：地板 75，目標 80——本機已過，等九宮格 -`TODO` — 目標 **80**（2026-08-23 拍板）。地板本身照舊只跟著實測的最低那一格走。 +`TODO` — 目標 **80**（2026-08-23 拍板）。**2026-08-24 本機（Windows／3.14）已到 +82.40%**（77.04% 起算，四批：WebRTC 那一族、window backends、accessibility、 +hotkey backends），但**地板照舊只跟著九宮格最低那一格走**，所以 +`fail_under` 還停在 75，要等 CI 的 `coverage report` 印出來才動。 +本機與最低那一格上一次相差約 1.3 點（77.04 對 75.79）。 + +**這一條要收掉，只差兩件事**：讀一次矩陣把地板提上去，以及下面第 2 項那個 +`DECIDE`（掃不到的那批 adapter）。其餘計畫內的項目都做完了。 `fail_under` 一度從 35 提到 50，理由寫在 `pyproject.toml`。**那兩個數字都低了大約 24 點**，而原因不在測試，在量測的起點： From fc50ffb8793ec35d84d5b6bd842cac8c668e604e Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 11:25:12 +0800 Subject: [PATCH 15/20] Read the AX value constants from the framework that declares them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Quartz.kAXValueCGPointType` does not exist. pyobjc builds ApplicationServices on top of HIServices, which declares the AXValue type constants, and Quartz has no such parent — so the attribute lookup raised AttributeError on the one platform this backend runs on. It took out `move()` and `_point()`, and through `_point` the frame match that picks which accessibility window a Quartz window is, which is the first thing every other action does. The stub answered for those names on Quartz, which is why all nine squares were green. `test_pyobjc_stub_names` exists to stop precisely that, and it did: the two macOS squares failed the moment the stub was measured against the real frameworks. The names move to `AX_NAMES`, so they are now checked against ApplicationServices, where the wheel's `HIServices/_metadata.py` defines them. --- architecture_explore.md | 8 ++++---- .../wrapper/window_backends/macos_backend.py | 11 +++++++---- test/unit_test/headless/_pyobjc_stub.py | 9 ++++++--- 3 files changed, 17 insertions(+), 11 deletions(-) diff --git a/architecture_explore.md b/architecture_explore.md index 142b2078..25a3f624 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -20,7 +20,7 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | | Python 模組總數（含周邊子專案） | 1,032 | -| 程式碼總行數 | 141,347 | +| 程式碼總行數 | 141,350 | | `je_auto_control/utils/` 子套件數 | 310 | | `AC_*` 動作指令數（`known_commands()` 實測） | 773 | | 套件門面 `__all__` 公開名稱數 | 1,238 | @@ -179,7 +179,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `wrapper/auto_control_image.py` | 83 | 影像 API：`locate_all_image`、`locate_image_center`、`locate_and_click`。 | | `wrapper/auto_control_record.py` | 114 | 錄製 API：`record`／`stop_record`／`record_to_json`（支援 stop event 與逾時）。 | | `wrapper/auto_control_window.py` | 278 | 視窗管理門面：列舉、尋找、聚焦、等待、關閉、顯示狀態、幾何、所屬行程 PID、依行程列舉／最小化視窗、不搶焦點的投遞式輸入（目前僅 Windows 實作）。 | -| `wrapper/window_backends/` | 985 | 視窗管理的平台縫（`base` / `windows_backend` / `x11_backend` / `macos_backend` / `null_backend`）。放在 `wrapper/` 而不是 `utils/`，因為它必須 import `windows/`、`linux_with_x11/`、`osx/`，而 `utils/` 在分層上在那三者之上。 | +| `wrapper/window_backends/` | 988 | 視窗管理的平台縫（`base` / `windows_backend` / `x11_backend` / `macos_backend` / `null_backend`）。放在 `wrapper/` 而不是 `utils/`，因為它必須 import `windows/`、`linux_with_x11/`、`osx/`，而 `utils/` 在分層上在那三者之上。 | ### 5.3 平台後端 @@ -1032,7 +1032,7 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `utils/usb/` | 17 | 4,281 | | `je_auto_control/`（頂層 3 檔） | 3 | 2,367 | | `utils/accessibility/` | 13 | 2,840 | -| `wrapper/` | 19 | 3,514 | +| `wrapper/` | 19 | 3,517 | | `windows/` | 23 | 1,906 | | `utils/rest_api/` | 8 | 1,751 | | `utils/agent/` | 8 | 1,250 | @@ -1046,5 +1046,5 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 735 | | 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 673 | 47,683 | -| **總計** | **1,026** | **141,282** | +| **總計** | **1,026** | **141,285** | diff --git a/je_auto_control/wrapper/window_backends/macos_backend.py b/je_auto_control/wrapper/window_backends/macos_backend.py index edaf30e0..1aaa5d99 100644 --- a/je_auto_control/wrapper/window_backends/macos_backend.py +++ b/je_auto_control/wrapper/window_backends/macos_backend.py @@ -237,10 +237,14 @@ def move(self, window_id: int, x: int, y: int, window = self._ax_window(window_id) if window is None: return False + # The AXValue type constants come from ApplicationServices, not + # Quartz: pyobjc builds ApplicationServices on top of HIServices, + # which declares them, and Quartz has no such parent. Spelling them + # `Quartz.` raised AttributeError on the one platform this runs on. position = ax.AXValueCreate( - Quartz.kAXValueCGPointType, Quartz.CGPoint(float(x), float(y))) + ax.kAXValueCGPointType, Quartz.CGPoint(float(x), float(y))) size = ax.AXValueCreate( - Quartz.kAXValueCGSizeType, + ax.kAXValueCGSizeType, Quartz.CGSize(float(width), float(height))) moved = ax.AXUIElementSetAttributeValue(window, "AXPosition", position) resized = ax.AXUIElementSetAttributeValue(window, "AXSize", size) @@ -279,11 +283,10 @@ def _best_match(candidates: list, wanted_origin: Tuple[int, int], def _point(value: Any) -> Tuple[int, int]: """Read an ``AXValue`` point as ``(x, y)``, or ``(-1, -1)``.""" import ApplicationServices as ax - import Quartz if value is None: return (-1, -1) - ok, point = ax.AXValueGetValue(value, Quartz.kAXValueCGPointType, None) + ok, point = ax.AXValueGetValue(value, ax.kAXValueCGPointType, None) if not ok or point is None: return (-1, -1) return (int(point.x), int(point.y)) diff --git a/test/unit_test/headless/_pyobjc_stub.py b/test/unit_test/headless/_pyobjc_stub.py index 8685a786..b370ab52 100644 --- a/test/unit_test/headless/_pyobjc_stub.py +++ b/test/unit_test/headless/_pyobjc_stub.py @@ -36,7 +36,7 @@ #: The rest of the surface the backend names, by module. QUARTZ_NAMES = WINDOW_KEYS + ( "kCGWindowListOptionOnScreenOnly", "kCGWindowListExcludeDesktopElements", - "kCGNullWindowID", "kAXValueCGPointType", "kAXValueCGSizeType", + "kCGNullWindowID", "CGWindowListCopyWindowInfo", "CGPoint", "CGSize", ) APPKIT_NAMES = ( @@ -47,6 +47,9 @@ "AXUIElementCreateApplication", "AXUIElementCopyAttributeValue", "AXUIElementSetAttributeValue", "AXUIElementPerformAction", "AXValueCreate", "AXValueGetValue", + # HIServices declares these and ApplicationServices inherits them; + # Quartz does not, which is what the macOS squares caught. + "kAXValueCGPointType", "kAXValueCGSizeType", ) #: What an AX call returns when it worked. Zero, and the backend reads it as @@ -164,8 +167,6 @@ def install(monkeypatch, world: World) -> World: quartz.kCGWindowListOptionOnScreenOnly = 1 quartz.kCGWindowListExcludeDesktopElements = 16 quartz.kCGNullWindowID = 0 - quartz.kAXValueCGPointType = "point" - quartz.kAXValueCGSizeType = "size" quartz.CGWindowListCopyWindowInfo = world.copy_window_info quartz.CGPoint = AXPoint quartz.CGSize = AXSize @@ -182,6 +183,8 @@ def install(monkeypatch, world: World) -> World: services.AXUIElementPerformAction = world.ax_perform_action services.AXValueCreate = world.ax_value_create services.AXValueGetValue = world.ax_value_get + services.kAXValueCGPointType = "point" + services.kAXValueCGSizeType = "size" for name, module in (("Quartz", quartz), ("AppKit", appkit), ("ApplicationServices", services)): From 7b98e8a37a09435dd80f844e84ea2448d11637ca Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 11:25:18 +0800 Subject: [PATCH 16/20] Raise the fixture's error through a parameter, not a dict subscript MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `raise state["walk_error"]` reads as `raise None` to an analyser that infers the slot from the dict literal that declared it, and the `is not None` guard above does not narrow that inference — Codacy's pylint reported it and held the PR red while every Actions job was green. Passing the error through a parameter says what it is at the point it is raised. --- .../headless/test_accessibility_windows_uia.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/test/unit_test/headless/test_accessibility_windows_uia.py b/test/unit_test/headless/test_accessibility_windows_uia.py index 0b887923..6ebaa8dd 100644 --- a/test/unit_test/headless/test_accessibility_windows_uia.py +++ b/test/unit_test/headless/test_accessibility_windows_uia.py @@ -78,6 +78,17 @@ def _element(name="", control_type=50000, rect=None, process_id=0, cached=cached) +def _raise(error): + """Raise the error a fixture parked in its state dict. + + Raising the subscript directly reads as ``raise None`` to an analyser + that infers the slot from the dict literal declaring it, and the + ``is not None`` guard does not narrow that inference. Passing the error + through a parameter says what it is where it is raised. + """ + raise error + + # --- availability and the automation object ----------------------------------- def test_a_windows_box_without_comtypes_refuses_and_says_how_to_fix_it( @@ -169,7 +180,7 @@ def _search_roots(automation, window_title): def _walk_elements(automation, root, limit): state["budgets"].append(limit) if state["walk_error"] is not None: - raise state["walk_error"] + _raise(state["walk_error"]) for element in state["under"].get(id(root), []): yield element From fdf40d51445c7a70821c3c308ef46a513b95e190 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 11:25:32 +0800 Subject: [PATCH 17/20] Build the real object when an adapter's callee is a class MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The sweep stopped at `Optional[X]`, containers, scalars and dataclasses, so 134 adapters whose callee is a class sat out — the biggest remaining group. The contract is readable there too: a constructor's parameters carry annotations, so an instance can be built from them exactly as a dataclass is built from its fields. Measured, 112 of the 134 are reachable that way. Two rules keep it honest rather than merely running: A callee that *returns* a class gets a stub returning a real instance, so `get_egress_policy().is_allowed(url)` and `parse_baggage(header).to_dict()` run for real against the declared shape. A callee that *is* a class is left alone entirely. Replacing it would throw away the client's arguments — the very thing under test — and take its alternative constructors with it, which is how a stub standing in for a class first showed up as `'function' object has no attribute 'from_dict'`. Three consequences fall out of running the genuine objects: Sample values now follow a declared object's own `properties` instead of flattening it to `{}`. A tool that names `kind` as required is describing a payload no client would send empty, and `{}` tests the sample, not the wiring. A class defined beside a third-party import stays out. `S3ArtifactStore` builds from its annotations perfectly well and then reaches for `boto3` on the first method call — the existing "the adapter picks its own backend" rule, one level down, and eight entries that would all have read "boto3" kept off the documented-exception list. Both sweeps run in a directory of their own. A checkpoint store handed the sample path creates a SQLite database where it stands; unisolated it left `sample.txt` and `value-for-db` in the repository and made one case depend on whether another had run first. 586 MCP adapters (was 554) and 537 executor adapters (was 471) now run. Locally 82.40% to 82.75%. Four adapters need more than any annotation promises and are named with a reason each, as before. --- test/unit_test/headless/_contract_sweep.py | 133 +++++++++++++++++- .../headless/test_adapter_registry_sweep.py | 27 +++- .../headless/test_rest_route_sweep.py | 14 ++ 3 files changed, 168 insertions(+), 6 deletions(-) diff --git a/test/unit_test/headless/_contract_sweep.py b/test/unit_test/headless/_contract_sweep.py index 69484e96..a7408055 100644 --- a/test/unit_test/headless/_contract_sweep.py +++ b/test/unit_test/headless/_contract_sweep.py @@ -23,6 +23,7 @@ Nothing here is a test; the file is named so pytest does not collect it. """ import ast +import collections.abc import dataclasses import importlib import inspect @@ -53,6 +54,9 @@ # before delegating enforces it. Without this the sweep hands `ac_rrule_next` # the generic sample and watches `datetime.fromisoformat` reject it -- which # says nothing about the adapter and everything about the sample. +#: How far into a nested object the sample builder will follow `properties`. +_MAX_SCHEMA_DEPTH = 4 + _STRING_FORMATS: Dict[str, str] = { "date-time": "2026-01-02T03:04:05", "date": "2026-01-02", @@ -60,8 +64,15 @@ } -def _sample_value(spec: Dict[str, Any], name: str) -> Any: - """Return a value satisfying one JSON-Schema property node.""" +def _sample_value(spec: Dict[str, Any], name: str, depth: int = 0) -> Any: + """Return a value satisfying one JSON-Schema property node. + + A declared ``object`` is built from its own ``properties`` rather than + left empty: a tool whose ``anchor`` argument names ``kind`` as required + is describing a payload no client would send as ``{}``, and handing the + adapter ``{}`` tests the sample, not the wiring. ``depth`` stops a schema + that describes itself. + """ enum = spec.get("enum") if isinstance(enum, list) and enum: return enum[0] @@ -70,7 +81,13 @@ def _sample_value(spec: Dict[str, Any], name: str) -> Any: kind = next((entry for entry in kind if entry != "null"), "string") if kind == "array": item = spec.get("items") - return [_sample_value(item, name)] if item else [] + return [_sample_value(item, name, depth + 1)] if item else [] + if kind == "object": + properties = spec.get("properties") or {} + if not properties or depth >= _MAX_SCHEMA_DEPTH: + return {} + return {key: _sample_value(sub, key, depth + 1) + for key, sub in properties.items()} if kind == "string": return _STRING_FORMATS.get(spec.get("format"), f"value-for-{name}") return _SCALARS.get(kind, f"value-for-{name}") @@ -84,6 +101,18 @@ def _sample_value(spec: Dict[str, Any], name: str) -> Any: list: list, dict: dict, set: set, tuple: tuple, } +# An annotation is free to promise the abstract protocol rather than the +# concrete type, and `Mapping[str, X]` originates at `collections.abc.Mapping`, +# which cannot be instantiated. The emptiest value satisfying each is the +# builtin that registers as it. +_ABSTRACT_CONTAINERS: Dict[Any, Callable[[], Any]] = { + collections.abc.Mapping: dict, collections.abc.MutableMapping: dict, + collections.abc.Sequence: list, collections.abc.MutableSequence: list, + collections.abc.Iterable: list, collections.abc.Iterator: iter([]).__iter__, + collections.abc.Collection: list, + collections.abc.Set: set, collections.abc.MutableSet: set, +} + def _value_for_generic(origin: Any, arguments: Tuple[Any, ...], @@ -98,6 +127,8 @@ def _value_for_generic(origin: Any, arguments: Tuple[Any, ...], return tuple(_value_for(entry, seen) for entry in arguments) if origin in (list, set, frozenset, dict): return origin() + if origin in _ABSTRACT_CONTAINERS: + return _ABSTRACT_CONTAINERS[origin]() raise ValueError(f"unmodelled container {origin!r}") @@ -124,6 +155,81 @@ def _dataclass_value(cls: type, seen: FrozenSet[type]) -> Any: return cls(**arguments) +def _constructed_value(cls: type, seen: FrozenSet[type]) -> Any: + """Build a real instance of a first-party class from its own ``__init__``. + + This is `_dataclass_value` one level along, and for the same reason: a + constructor's parameters carry annotations, so "what does calling this + promise the caller?" stays a question the program can answer. The instance + is the genuine class, not a double -- its methods run, its ``to_dict()`` + runs, and an adapter that calls one with the wrong arguments still raises. + That is what keeps "the adapter ran" and "the adapter was checked" from + coming apart, which a double answering every method would not. + + Only the constructor's *required* parameters are filled: a default is the + class's own statement of what the caller may leave out. + + Raising :class:`ValueError` leaves the adapter out of the sweep, which is + the right answer for all three ways this can fail -- a class outside the + package (the adapter is choosing a backend), a parameter the contract + never described, and zero values that break an invariant the annotation + cannot express (``rate must be positive`` for a token bucket, say). + """ + if cls in seen: + raise ValueError(f"self-referential constructor on {cls!r}") + if not getattr(cls, "__module__", "").startswith(_PACKAGE + "."): + raise ValueError(f"not a first-party class: {cls!r}") + if _module_picks_a_backend(cls): + raise ValueError(f"{cls!r} is defined beside a third-party import") + try: + signature = inspect.signature(cls.__init__) + hints = typing.get_type_hints(cls.__init__) + except (TypeError, ValueError, NameError) as error: + raise ValueError(f"unreadable constructor on {cls!r}: {error}") from error + arguments = {} + for name, parameter in signature.parameters.items(): + if name == "self" or parameter.default is not parameter.empty: + continue + if parameter.kind not in (parameter.POSITIONAL_OR_KEYWORD, + parameter.KEYWORD_ONLY): + continue + if name not in hints: + raise ValueError(f"{cls!r} takes an unannotated {name}") + arguments[name] = _value_for(hints[name], seen | {cls}) + try: + return cls(**arguments) + except Exception as error: # noqa: BLE001 # reason: any refusal means the contract's zero values do not build one, and the adapter leaves the sweep + raise ValueError( + f"{cls!r} refuses its contract's zero values: {error}") from error + + +def _module_picks_a_backend(cls: type) -> bool: + """Return True when the module defining ``cls`` imports a third-party one. + + The sweeps already refuse an adapter that imports a third-party package, + because what it returns then depends on what is installed on the machine + rather than on any contract. A real instance moves that question one level + down: ``S3ArtifactStore`` constructs from its annotations perfectly well + and then reaches for ``boto3`` the moment a method is called. Same rule, + same seam, one level along -- and it keeps eight adapters off the + documented-exception list, where they would all have said "boto3". + """ + try: + source = inspect.getsource(sys.modules[cls.__module__]) + except (OSError, TypeError, KeyError): + return True + for node in ast.walk(ast.parse(source)): + if isinstance(node, ast.Import): + if any(not _is_stdlib(alias.name) + and not alias.name.startswith(_PACKAGE) for alias in node.names): + return True + elif isinstance(node, ast.ImportFrom) and node.module and not node.level: + root = node.module.split(".")[0] + if not _is_stdlib(root) and root != _PACKAGE: + return True + return False + + def _value_for(annotation: Any, seen: FrozenSet[type] = frozenset()) -> Any: """Build the emptiest value a return annotation allows. @@ -144,6 +250,8 @@ def _value_for(annotation: Any, seen: FrozenSet[type] = frozenset()) -> Any: return _ZEROS[annotation]() if isinstance(annotation, type) and dataclasses.is_dataclass(annotation): return _dataclass_value(annotation, seen) + if isinstance(annotation, type): + return _constructed_value(annotation, seen) raise ValueError(f"unmodelled return annotation {annotation!r}") @@ -334,8 +442,25 @@ def _contract_stubs(adapter: Any) -> Optional[List[Tuple[Any, str, Any]]]: if callee is None: return None owner, attribute = callee + target = getattr(owner, attribute) + if isinstance(target, type): + # A class callee is left alone: the adapter constructs the real + # object out of the client's own arguments, which is both the + # strongest form of "run against the declared type" and the only + # one that keeps the arguments under test. Replacing the class + # with something returning a prepared instance would throw those + # arguments away -- and take the class's own constructors with + # them, since a stub standing in for a class has no `from_dict`. + # `_constructed_value` below still has to succeed, so a class the + # contract cannot build stays out of the sweep rather than being + # run blind. + try: + _value_for(target) + except (ValueError, TypeError, NameError): + return None + continue try: - hints = typing.get_type_hints(getattr(owner, attribute)) + hints = typing.get_type_hints(target) stubs.append((owner, attribute, _value_for(hints.get("return", inspect.Signature.empty)))) diff --git a/test/unit_test/headless/test_adapter_registry_sweep.py b/test/unit_test/headless/test_adapter_registry_sweep.py index 091a150c..44fa7819 100644 --- a/test/unit_test/headless/test_adapter_registry_sweep.py +++ b/test/unit_test/headless/test_adapter_registry_sweep.py @@ -60,14 +60,37 @@ _NEEDS_MORE_THAN_THE_CONTRACT = { # Uses the return value as a context manager; the annotation is untyped. "ac_list_monitors", - # Its single import is a class, so there is no return annotation to build. - "ac_anchor_click", # Indexes a key out of a Dict[str, Any] the annotation cannot promise. "ac_tween_drag", "ac_voice_dispatch", + # Its anchor's `kind` selects a locator backend, and the first value the + # enum offers is `image` -- so the call leaves the adapter and goes into + # OpenCV template matching against a real screen. + "ac_anchor_click", + # Needs a viewer that registered earlier. The registry it is handed is + # real and empty, and refusing an unknown viewer is what it is for. + "ac_presence_update_cursor", + "ac_presence_set_role", + # Needs an interaction recorded into the cassette first; a fresh one + # misses by design. + "ac_http_replay", } +@pytest.fixture(autouse=True) +def _in_a_directory_of_its_own(tmp_path, monkeypatch): + """Run every sweep case in an empty directory. + + An adapter whose callee is a class builds the real object out of the + client's own arguments, and some of those objects are stores: a + checkpoint store handed the sample file path creates a SQLite database + where it stands. In the repository that leaves files behind and makes one + case depend on whether another ran first; here each case gets a directory + nobody else can see. + """ + monkeypatch.chdir(tmp_path) + + # === Reading a value out of a declared type ================================= # Sample values by JSON-Schema type. Strings carry their property name so a diff --git a/test/unit_test/headless/test_rest_route_sweep.py b/test/unit_test/headless/test_rest_route_sweep.py index 3af0361c..cbd52a45 100644 --- a/test/unit_test/headless/test_rest_route_sweep.py +++ b/test/unit_test/headless/test_rest_route_sweep.py @@ -52,6 +52,20 @@ _NOT_JSON_ROUTES = {("GET", "/dashboard"), ("GET", "/docs"), ("GET", "/metrics")} +@pytest.fixture(autouse=True) +def _in_a_directory_of_its_own(tmp_path, monkeypatch): + """Run every sweep case in an empty directory. + + An adapter whose callee is a class builds the real object out of the + client's own arguments, and some of those objects are stores: a + checkpoint store handed the sample file path creates a SQLite database + where it stands. In the repository that leaves files behind and makes one + case depend on whether another ran first; here each case gets a directory + nobody else can see. + """ + monkeypatch.chdir(tmp_path) + + def _routes() -> List[Tuple[str, str, Any]]: """Every `(method, path, handler)` the dispatcher can reach.""" return ([("GET", path, handler) From 5efb03e7c8ecd4ab649f639579fd7d0bc8c840ff Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 11:42:24 +0800 Subject: [PATCH 18/20] Stop what the sweep starts, and refuse to leave a thread behind MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Running the genuine object means an adapter whose job is to start something really starts it. `AC_usb_watch_start` left a hotplug poller running for the rest of the session, and on Windows that poller shells out to PowerShell every interval — so `test_wayland_libei`, which patches `subprocess.run` across the process and reads the first argv it recorded, read the poller's instead of its own. One square of the matrix, and nowhere else. Every `_start` in both registries has a `_stop` sibling — all eight of them — so the sweep runs it, which also covers the stop adapter from the only state where stopping means anything. The autouse guard is the part that matters beyond these two: the next adapter to grow a thread fails here, by name, instead of becoming somebody else's flake three files away. Also cache the module scan behind the class-callee rule; the sweeps drop from 17.9s to 12.4s. --- test/unit_test/headless/_contract_sweep.py | 10 ++-- .../headless/test_adapter_registry_sweep.py | 47 +++++++++++++++++++ 2 files changed, 53 insertions(+), 4 deletions(-) diff --git a/test/unit_test/headless/_contract_sweep.py b/test/unit_test/headless/_contract_sweep.py index a7408055..9b403660 100644 --- a/test/unit_test/headless/_contract_sweep.py +++ b/test/unit_test/headless/_contract_sweep.py @@ -25,6 +25,7 @@ import ast import collections.abc import dataclasses +import functools import importlib import inspect import json @@ -179,7 +180,7 @@ def _constructed_value(cls: type, seen: FrozenSet[type]) -> Any: raise ValueError(f"self-referential constructor on {cls!r}") if not getattr(cls, "__module__", "").startswith(_PACKAGE + "."): raise ValueError(f"not a first-party class: {cls!r}") - if _module_picks_a_backend(cls): + if _module_picks_a_backend(cls.__module__): raise ValueError(f"{cls!r} is defined beside a third-party import") try: signature = inspect.signature(cls.__init__) @@ -203,8 +204,9 @@ def _constructed_value(cls: type, seen: FrozenSet[type]) -> Any: f"{cls!r} refuses its contract's zero values: {error}") from error -def _module_picks_a_backend(cls: type) -> bool: - """Return True when the module defining ``cls`` imports a third-party one. +@functools.lru_cache(maxsize=None) +def _module_picks_a_backend(module_path: str) -> bool: + """Return True when the module at ``module_path`` imports a third-party one. The sweeps already refuse an adapter that imports a third-party package, because what it returns then depends on what is installed on the machine @@ -215,7 +217,7 @@ def _module_picks_a_backend(cls: type) -> bool: documented-exception list, where they would all have said "boto3". """ try: - source = inspect.getsource(sys.modules[cls.__module__]) + source = inspect.getsource(sys.modules[module_path]) except (OSError, TypeError, KeyError): return True for node in ast.walk(ast.parse(source)): diff --git a/test/unit_test/headless/test_adapter_registry_sweep.py b/test/unit_test/headless/test_adapter_registry_sweep.py index 44fa7819..52bc39cf 100644 --- a/test/unit_test/headless/test_adapter_registry_sweep.py +++ b/test/unit_test/headless/test_adapter_registry_sweep.py @@ -36,6 +36,7 @@ """ import inspect import json +import threading import typing from typing import Any, Callable, Dict, List @@ -91,6 +92,41 @@ def _in_a_directory_of_its_own(tmp_path, monkeypatch): monkeypatch.chdir(tmp_path) +@pytest.fixture(autouse=True) +def _leaves_no_background_work_behind(): + """No sweep case may outlive itself in a thread. + + Running the genuine object rather than a stand-in means an adapter whose + job is to *start* something really starts it: `AC_usb_watch_start` leaves + a hotplug poller running, and on Windows that poller shells out to + PowerShell every interval. Any later test that patches `subprocess.run` + process-wide and reads the first call it recorded then sees the poller's + argv instead of its own -- which is how `test_wayland_libei` failed on one + square of the matrix and nowhere else. + + The sweep stops what it starts, below; this is the guard that says so, and + it names the next adapter to grow a thread instead of letting it become + somebody else's flake. + """ + before = {thread.ident for thread in threading.enumerate()} + yield + left = [thread.name for thread in threading.enumerate() + if thread.ident not in before and thread.is_alive()] + assert not left, f"the case left these threads running: {left}" + + +def _stop_whatever_it_started(name: str, dispatch) -> None: + """Run the `_stop` sibling of a `_start` adapter, where there is one. + + Every `_start` in either registry has one, which is what makes this a + convention rather than a special case -- and running it is coverage of + the stop adapter too, from the only state where stopping means anything. + """ + if not name.endswith("_start"): + return + dispatch(name[:-len("_start")] + "_stop") + + # === Reading a value out of a declared type ================================= # Sample values by JSON-Schema type. Strings carry their property name so a @@ -161,6 +197,14 @@ def _unique_by_handler(tools: List[MCPTool]) -> List[MCPTool]: return unique +def _invoke_by_name(name: str) -> None: + """Call one tool by name with what its own schema declares, if it exists.""" + for tool in REGISTRY: + if tool.name == name: + tool.invoke(_tool_arguments(tool.input_schema, required_only=False)) + return + + REGISTRY = build_default_tool_registry(read_only=False, aliases=False) _TOOLS = _unique_by_handler(REGISTRY) DELEGATING = [(tool, *delegation(tool.handler)) for tool in _TOOLS @@ -213,6 +257,7 @@ def test_tool_result_survives_json(tool, monkeypatch): install_stubs(monkeypatch, contract_stubs(tool.handler)) result = tool.invoke(_tool_arguments(tool.input_schema, required_only=False)) + _stop_whatever_it_started(tool.name, lambda stop: _invoke_by_name(stop)) assert is_serialisable(result, (MCPContent,)), ( f"{tool.name} returned {type(result).__name__}, which json.dumps " "cannot encode") @@ -296,6 +341,8 @@ def test_command_dispatches_and_records_json(command, monkeypatch): install_stubs(monkeypatch, contract_stubs(adapter)) payload = _command_arguments(command, adapter) record = executor.execute_action([[command, payload]]) + _stop_whatever_it_started(command, lambda stop: executor.execute_action( + [[stop, _command_arguments(stop, executor.event_dict[stop])]])) assert record, f"{command} produced no execution record" try: json.dumps(record) From 682cec63ffb56718fdd1227eae23b09d64f61ab2 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 11:42:31 +0800 Subject: [PATCH 19/20] Patch the bridge on the module that actually reads it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `send_file` hands the work to `FileTransferSender`, which reads `get_bridge` out of `webrtc_files`. Both fixtures patched it on the viewer and the host instead, so that one path kept the real bridge: the chunks were queued onto a background event loop while the assertion read `sent[0]` straight afterwards. It passed wherever the loop won the race, which was every square until a loaded runner lost it — macos-14/3.10 and ubuntu-22.04/3.10 in the same round, one on the host side and one on the viewer side. `test_webrtc_file_transfer` already patched the right module, which is why it never flaked. --- test/unit_test/headless/test_webrtc_host_channels.py | 8 ++++++++ test/unit_test/headless/test_webrtc_viewer_control.py | 8 ++++++++ 2 files changed, 16 insertions(+) diff --git a/test/unit_test/headless/test_webrtc_host_channels.py b/test/unit_test/headless/test_webrtc_host_channels.py index 81a8de50..e76d7eb0 100644 --- a/test/unit_test/headless/test_webrtc_host_channels.py +++ b/test/unit_test/headless/test_webrtc_host_channels.py @@ -44,6 +44,14 @@ def bridge(monkeypatch): fake = Bridge() monkeypatch.setattr(host_module, "get_bridge", lambda: fake) + # `send_file` hands the work to `FileTransferSender`, which reads + # `get_bridge` out of `webrtc_files` -- patching it here alone left that + # path on the real bridge, so the chunks landed on a background event + # loop and the assertion read `sent[0]` before anything was in it. It + # passed wherever the loop won the race, which was every square until a + # loaded runner lost it. + from je_auto_control.utils.remote_desktop import webrtc_files + monkeypatch.setattr(webrtc_files, "get_bridge", lambda: fake) return fake diff --git a/test/unit_test/headless/test_webrtc_viewer_control.py b/test/unit_test/headless/test_webrtc_viewer_control.py index d3640780..c66d41d5 100644 --- a/test/unit_test/headless/test_webrtc_viewer_control.py +++ b/test/unit_test/headless/test_webrtc_viewer_control.py @@ -43,6 +43,14 @@ def bridge(monkeypatch): fake = Bridge() monkeypatch.setattr(viewer_module, "get_bridge", lambda: fake) + # `send_file` hands the work to `FileTransferSender`, which reads + # `get_bridge` out of `webrtc_files` -- patching it here alone left that + # path on the real bridge, so the chunks landed on a background event + # loop and the assertion read `sent[0]` before anything was in it. It + # passed wherever the loop won the race, which was every square until a + # loaded runner lost it. + from je_auto_control.utils.remote_desktop import webrtc_files + monkeypatch.setattr(webrtc_files, "get_bridge", lambda: fake) return fake From 0e8e25b76acb64d991540ee83e8ba138354b523a Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Mon, 24 Aug 2026 11:54:07 +0800 Subject: [PATCH 20/20] Raise the floor to 81, which meets the target set on 2026-08-23 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The nine-way matrix runs 81.40% (ubuntu-22.04 / 3.14) to 82.73% (windows-2022 / 3.12), read from `coverage report` — the number `fail_under` is compared against — and not from the XML artifact, which omits branches and reads about two points high. Floored to the integer below the lowest square, as every step of this ratchet has been. Five batches took it from 75: the WebRTC family, the window backends, the accessibility backends, the hotkey backends, and the class-callee half of the adapter sweep. Progress.md closes the coverage item and keeps the section for the traps that will be stepped in again — where the measurement starts, that the floor can only be read from the report, and that only code reachable on all nine squares moves it. --- Progress.md | 48 +++++++++++++--------- WHATS_NEW.md | 107 +++++++++++++++++++++++++++++++++++++++++++++++++ pyproject.toml | 14 ++++++- 3 files changed, 148 insertions(+), 21 deletions(-) diff --git a/Progress.md b/Progress.md index c01141a0..82b2a8c1 100644 --- a/Progress.md +++ b/Progress.md @@ -244,16 +244,19 @@ capability enum 值與 variadic `ei_seat_bind_capabilities`、event-type enum **2026-08-23 維護者拍板：下一個覆蓋率目標是 80。** 這一條留到那時候。 型別那一節留著是因為它記的那幾個坑之後還會踩到。 -### 覆蓋率：地板 75，目標 80——本機已過，等九宮格 +### 覆蓋率：目標 80 達成，地板提到 81——**這一條結束了** -`TODO` — 目標 **80**（2026-08-23 拍板）。**2026-08-24 本機（Windows／3.14）已到 -82.40%**（77.04% 起算，四批：WebRTC 那一族、window backends、accessibility、 -hotkey backends），但**地板照舊只跟著九宮格最低那一格走**，所以 -`fail_under` 還停在 75，要等 CI 的 `coverage report` 印出來才動。 -本機與最低那一格上一次相差約 1.3 點（77.04 對 75.79）。 +`TODO` → **完成（2026-08-24）**。2026-08-23 拍板的目標是 **80**；九宮格實測 +最低那一格 **81.40%**（ubuntu-22.04／3.14），最高 82.73%（windows-2022／3.12）， +`fail_under` 隨之從 75 提到 **81**。 -**這一條要收掉，只差兩件事**：讀一次矩陣把地板提上去，以及下面第 2 項那個 -`DECIDE`（掃不到的那批 adapter）。其餘計畫內的項目都做完了。 +從 75 到 81 是五批：WebRTC 那一族、window backends、accessibility backends、 +hotkey backends，以及最後這一批——掃不到的那批 adapter（下面第 2 項那條 +`DECIDE`，2026-08-24 拍板走「建真的實例」）。 + +**這一整節之後只剩參考價值**：它記的幾個坑（量測起點、地板只能從 +`coverage report` 讀、只補跑得到九格的程式碼）之後每次動覆蓋率都還會踩到， +所以留著。下一個目標由維護者再定。 `fail_under` 一度從 35 提到 50，理由寫在 `pyproject.toml`。**那兩個數字都低了大約 24 點**，而原因不在測試，在量測的起點： @@ -368,15 +371,19 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl 三支共用的機器搬進 `test/unit_test/headless/_contract_sweep.py`，各自只留自己的 參數來源。 -#### 現在的九宮格（2026-08-24 實測，本 PR 最後一次 run） +#### 九宮格怎麼走到 81 的（每一列都是 `coverage report` 實測） -| | 最低 | 最高 | -| --- | --- | --- | -| 上一版（PR #486 首輪） | 69.67%（ubuntu-22.04／3.14） | 70.97%（windows-2022／3.12） | -| 這一版 | **75.79%**（ubuntu-22.04／3.14） | 76.99%（windows-2022，3.11／3.12） | +| | 最低 | 最高 | 地板 | +| --- | --- | --- | ---: | +| PR #486 首輪 | 69.67%（ubuntu-22.04／3.14） | 70.97%（windows-2022／3.12） | 69 | +| 補完 WebRTC 之後 | 75.79%（ubuntu-22.04／3.14） | 76.99%（windows-2022，3.11／3.12） | 75 | +| 補完後四批之後 | **81.40%**（ubuntu-22.04／3.14） | 82.73%（windows-2022／3.12） | **81** | -地板隨之從 69 提到 **75**。這九個數字讀的是 `coverage report`（也就是 -`fail_under` 真正比對的那個），不是 artifact 的 `line-rate`——理由見上一節。 +這九個數字讀的是 `coverage report`（也就是 `fail_under` 真正比對的那個）， +不是 artifact 的 `line-rate`——理由見上一節。最低那一格始終是 Linux、最高那一格 +始終是 Windows，因為門面 import 進來的是它自己那個平台的後端；換句話說 +**剩下的那 18 點裡有一部分是任何單一平台都拿不到的**，要再往上只能補在九格 +都跑得到的程式碼上。 #### 剩下的 4 點在哪裡（2026-08-24，ubuntu-22.04／3.14 的 artifact） @@ -400,11 +407,11 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl | 檔案 | 每一格都沒蓋到 | 備註 | | --- | ---: | --- | -| `utils/executor/action_executor.py` | 589 | 掃不到的那批 adapter：被呼叫者是 class（沒有回傳標注可讀）、或 adapter 伸手進兩個模組 | +| ~~`utils/executor/action_executor.py`~~ | ~~589~~ | **2026-08-24 補到 79.76%**（本機實測 77.31% → 79.76%，未覆蓋 589 → 498） | | ~~`utils/accessibility/backends/windows_backend.py`~~ | ~~446~~ | **2026-08-24 補完（99.42%）**，順便修掉 37 個攔不到 `COMError` 的 except | | ~~`utils/remote_desktop/webrtc_viewer.py`~~ | ~~354~~ | **2026-08-24 補完（100%）** | | ~~`utils/remote_desktop/webrtc_host.py`~~ | ~~351~~ | **2026-08-24 補完（100%）** | -| `utils/mcp_server/tools/_handlers.py` | 348 | 同 `action_executor.py` | +| ~~`utils/mcp_server/tools/_handlers.py`~~ | ~~348~~ | **2026-08-24 補到 86.73%**（本機實測 83.23% → 86.73%，未覆蓋 348 → 262） | | `utils/remote_desktop/signaling_server.py` | 155 | **CI 動不了**：要 `[signaling]` extra（fastapi／uvicorn），沒裝 | | ~~`utils/remote_desktop/multi_viewer.py`~~ | ~~146~~ | **2026-08-24 補完（100%）** | | ~~`wrapper/window_backends/x11_backend.py`~~ | ~~133~~ | **2026-08-24 補完（100%）**。「只有 Linux 那兩格跑得到」是錯的，見下 | @@ -414,9 +421,10 @@ REST 那一支的參數來自 `rest_openapi.build_openapi_spec()`——與 handl 1. ~~`webrtc_host`／`webrtc_viewer`／`multi_viewer`／`webrtc_transport`／ `webrtc_audio`~~ **2026-08-24 做完**，見下一節。 -2. 掃不到的那批 adapter：目前卡在「被呼叫者是 class」。要嘛從 class 自己的方法標注 - 長出一個替身物件，要嘛承認那批不掃——**得先決定，因為前者會讓「跑起來了」和 - 「驗到了東西」分家**。 +2. ~~掃不到的那批 adapter：卡在「被呼叫者是 class」~~ **2026-08-24 做完**。 + 原本寫的兩個選項是個假二選一：第三條路是照 class 自己的 `__init__` 標注 + **建真的實例**，不用方法替身，所以「跑起來了」和「驗到了東西」不會分家。 + 維護者拍板走這條，見下下節。 3. ~~`wrapper/window_backends` 與 `utils/accessibility`~~ **2026-08-24 兩個都整包補完**，見下下節。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index 39521aa7..e2a0de2c 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -2,6 +2,113 @@ ## What's new (2026-08-24) +### The Floor Is 81, And The Target Set On 2026-08-23 Is Met + +The nine-way matrix now runs **81.40%** (ubuntu-22.04 / 3.14) to 82.73% +(windows-2022 / 3.12), so `fail_under` goes from 75 to 81 — floored to the +integer below the lowest square, the convention every step of this ratchet has +used. The target the maintainer set on 2026-08-23 was 80. + +Five batches got it there from 75: the WebRTC family, the window backends, the +accessibility backends, the hotkey backends, and the class-callee half of the +adapter sweep. Linux is still the low corner and Windows the high one, for the +reason that has held all along — the facade imports the backend of whatever +platform it is running on, so some of the remaining 18 points cannot be reached +from any single square, and raising this number again means covering code that +runs on all nine. + +### Two Races That Only A Loaded Runner Could Lose + +Both showed up as squares failing in a matrix where the same tests had passed +the round before, which is the shape of a race rather than a regression. + +**A poller that outlived its test.** `AC_usb_watch_start` really starts a +hotplug poller now that the sweep runs genuine objects, and on Windows that +poller shells out to PowerShell every interval. `test_wayland_libei` patches +`subprocess.run` across the whole process and reads the first argv it recorded +— so it read the poller's. Every `_start` in both registries has a `_stop` +sibling, all eight of them, so the sweep now runs it, and an autouse guard +fails any case that leaves a thread behind. The next adapter to grow one is +named here rather than becoming somebody else's flake three files away. + +**A bridge patched on the wrong module.** `send_file` hands the work to +`FileTransferSender`, which reads `get_bridge` out of `webrtc_files`; the host +and viewer fixtures patched it on the host and the viewer. That left the real +bridge on that one path, queueing the chunks onto a background event loop while +the assertion read `sent[0]` immediately afterwards. It passed wherever the +loop won the race — every square, until two lost it in the same round. +`test_webrtc_file_transfer` had always patched the right module, which is why +it never flaked. + +### The AX Constants Came From The Wrong Framework, And Nine Green Squares Said Otherwise + +`Quartz.kAXValueCGPointType` does not exist. pyobjc builds `ApplicationServices` +on top of `HIServices`, which is where the AXValue type constants are declared, +and `Quartz` has no such parent — so on macOS the lookup raised `AttributeError`. +That took out `move()`, `_point()`, and through `_point` the frame comparison +that decides which accessibility element a given Quartz window is, which every +other window action depends on. + +Every square was green while this was true, because the pyobjc stub answered +for the names on `Quartz` — the failure mode the stub was written to create and +`test_pyobjc_stub_names` was written to catch. It caught it on the first run: +the two macOS squares, where the real frameworks are installed, failed on +exactly those two names. The stub now declares them where they live, so they +are checked against `ApplicationServices`. + +The ground truth came out of the wheel rather than a guess: +`HIServices/_metadata.py` defines both constants, and +`ApplicationServices/__init__.py` names `(Quartz, HIServices, CoreText)` as its +parents while `Quartz` names no such thing. + +### When The Callee Is A Class, Build The Real Thing + +The sweep could read a contract made of `Optional[X]`, containers, scalars and +dataclasses. 134 adapters had a callee that is a class, and sat out — the +biggest group left. A constructor's parameters carry annotations too, so an +instance can be built from them the same way a dataclass is built from its +fields; measured, 112 of the 134 are reachable that way. + +What decides whether that is a test or just an execution is which of two shapes +the callee has, and they are handled oppositely: + +**A callee that returns a class** gets a stub returning a real instance. The +adapter then runs its real continuation: `get_egress_policy().is_allowed(url)` +and `parse_baggage(header).to_dict()` execute against the declared shape rather +than against something that answers every method. + +**A callee that is a class** is left alone. Stubbing it was tried first and is +wrong twice over: the prepared instance discards the client's arguments, which +are the thing under test — `CoordinateSpace` divided by a zero `model_w` that no +client sent — and a stub standing in for a class has no alternative +constructors, which surfaced as `'function' object has no attribute 'from_dict'`. +Letting the adapter build the genuine object out of the genuine arguments has +neither problem. + +Running real objects rather than stand-ins had three consequences worth naming: + +**A declared object is no longer flattened to `{}`.** A tool that names `kind` +as required is describing a payload no client would send empty; handing the +adapter `{}` tested the sample rather than the wiring. Samples now follow the +schema's own `properties`, bounded by depth. + +**A class defined beside a third-party import stays out.** `S3ArtifactStore` +builds from its annotations perfectly well and reaches for `boto3` on the first +method call. That is the existing "the adapter is choosing its own backend" +rule one level down, and applying it there kept eight entries that would all +have read "boto3" off the documented-exception list. + +**Both sweeps now run in a directory of their own.** A checkpoint store handed +the sample path creates a SQLite database where it stands: unisolated, the +sweep left `sample.txt` and `value-for-db` in the repository, and one case +passed or failed depending on whether another had run first. + +586 MCP adapters (from 554) and 537 executor adapters (from 471) now get +called. Four need more than any annotation can promise and are named with a +reason each — a presence registry that is real and empty and correctly refuses +an unknown viewer, a cassette with nothing recorded in it, and an anchor whose +`kind` enum sends the call into OpenCV template matching against a real screen. + ### A Whole Subsystem Was Being Measured At Zero, And Its Tests Were Skipping `quality.yml` now installs the `[webrtc]` extra. The coverage number was the diff --git a/pyproject.toml b/pyproject.toml index 16648211..1e81f484 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -189,7 +189,19 @@ precision = 2 # leaves them out, so the XML reads about 2 points higher (77.78% against # 75.79% on ubuntu-22.04 / 3.14, same run). A floor set from the artifact # would be a floor the suite cannot clear. -fail_under = 75 +# +# Measured 2026-08-24 on the run that turned every square green, after the +# WebRTC modules, the window and accessibility backends, the hotkey backends +# and the class-callee half of the adapter sweep: the nine-way matrix runs +# 81.40% (ubuntu-22.04 / 3.14) to 82.73% (windows-2022 / 3.12). That clears +# the 80 the maintainer set on 2026-08-23. Floored to the integer below the +# lowest square, the same convention every step of this ratchet has used: 81. +# +# The spread is still about 1.3 points and Windows is still the high corner, +# for the reason given above — the facade imports the backend of the platform +# it is running on, so part of the remainder is unreachable from any single +# square. Raising this number again means covering code that runs on all nine. +fail_under = 81 [tool.mypy] python_version = "3.10"