From 443d9832193ede2656822389f4b574b9ae1e7e3d Mon Sep 17 00:00:00 2001 From: MendixMau Date: Fri, 21 Aug 2026 17:30:09 +0800 Subject: [PATCH 01/11] Fix coherence-cadence.sh's proven-module glob and add full-harness-audit skill coherence-cadence.sh derived "proven modules" from architecture/modules/*/ subdirectories per module-brief.md's canonical layout, but a project using flat -brief.md files (a spec drift) made the glob match zero directories, silently reporting 0 proven modules forever regardless of how many verify-module.sh passes had actually run. Now reads .claude/loop/verify//summary.tsv directly, which is the project's own record of what ran and doesn't depend on a second directory matching a convention it may not follow. Also adds skills/full-harness-audit.md: a consolidated map of the whole testing harness (layer stack, the three easily-confused review passes, the five journey rungs, the cadence/obligations layer, and what fires automatically through the build-plan/module-brief/ledger timeline) plus a reusable trigger prompt for a genuine full click-through audit. Routed via ROUTING.md. Co-Authored-By: Claude Sonnet 5 --- ROUTING.md | 1 + project-bin/coherence-cadence.sh | 18 ++- skills/full-harness-audit.md | 231 +++++++++++++++++++++++++++++++ 3 files changed, 245 insertions(+), 5 deletions(-) create mode 100644 skills/full-harness-audit.md diff --git a/ROUTING.md b/ROUTING.md index 0cf7299..f3114ee 100644 --- a/ROUTING.md +++ b/ROUTING.md @@ -173,6 +173,7 @@ picks the row up. That is the whole procedure — there is no second list to rem | Tracking scope delta between the BRD and the built state | `skills/learned-skill-scope-delta.md` | review | 6 | ondemand | | Writing or reading docs/report.json — the append-only contract every instrument writes to and every renderer reads; open BEFORE building a new instrument or a second renderer | `skills/report-schema.md` | test,review,gate | 5,6 | ondemand | | Installing, extending, debugging or porting the verification harness — which part owns what, which parts run standalone, and what a missing part must report | `skills/harness-architecture.md` | test,review | 5,6 | ondemand | +| A user asks for "a full end-to-end test," "does everything work," or a click-through proof — before running anything, or if a prior pass turns out to have covered one module and gotten called e2e | `skills/full-harness-audit.md` | test,review | 5,6 | ondemand | | Checking whether the whole journey hangs together rather than each piece — finds correctly-built components nothing reaches, which per-element conformance and UI tests both miss | `skills/process-coherence-pass.md` | review | 5,6 | ondemand | | After every module's CONFIRM stage — counts proven modules since the last cluster/full coherence pass and exits DUE once the threshold is reached, so the cadence isn't left to memory | `project-bin/coherence-cadence.sh` | mdl,gate,review | 5,6 | ondemand | | Turning an already-rigorous run into a narrated proof a stakeholder can trust without running anything | `skills/e2e-evidence-report.md` | test,review | 6 | ondemand | diff --git a/project-bin/coherence-cadence.sh b/project-bin/coherence-cadence.sh index a9ca84f..78b3f30 100755 --- a/project-bin/coherence-cadence.sh +++ b/project-bin/coherence-cadence.sh @@ -45,17 +45,25 @@ done cd "$ROOT" || exit 2 -MODULES_DIR="$ROOT/architecture/modules" +VERIFY_DIR="$ROOT/.claude/loop/verify" MARKER="$ROOT/.claude/loop/coherence/last-cluster-pass.tsv" mkdir -p "$(dirname "$MARKER")" -# Every module that has ever completed a verify-module.sh pass (proven, per the header note above). +# Every module that has ever completed a verify-module.sh pass (proven, per the header note +# above). Source of truth is .claude/loop/verify//summary.tsv directly — do NOT derive +# module names from architecture/modules/*/ subdirectories, even though module-brief.md:70 +# specifies exactly that layout (one directory per module). A project that drifted from spec — +# flat .md / -brief.md files at architecture/modules/ root, no per-module +# subdirectory — made this loop match zero directories, silently reporting 0 proven modules +# forever regardless of how many verify passes had actually run. verify/ is this project's own +# record of what ran; trust that over a second directory that may not match the convention it's +# supposed to. Found on a real project, 2026-08-21. PROVEN="" -if [ -d "$MODULES_DIR" ]; then - for d in "$MODULES_DIR"/*/; do +if [ -d "$VERIFY_DIR" ]; then + for d in "$VERIFY_DIR"/*/; do [ -d "$d" ] || continue m="$(basename "$d")" - [ -f "$ROOT/.claude/loop/verify/$m/summary.tsv" ] && PROVEN="${PROVEN}${m} + [ -f "$d/summary.tsv" ] && PROVEN="${PROVEN}${m} " done fi diff --git a/skills/full-harness-audit.md b/skills/full-harness-audit.md new file mode 100644 index 0000000..32b7809 --- /dev/null +++ b/skills/full-harness-audit.md @@ -0,0 +1,231 @@ +# Skill: Full harness audit — running the whole testing stack for real, not by title + +**When to use:** the user asks for a "full end-to-end test," a "click-through proof," or "does +everything actually work" — and especially when they've already caught a report calling something +"e2e" that only covered one module or one action. Also use this as the entry point whenever you +are not sure which of the harness's many skills applies; it routes you to the right one instead of +guessing. + +**What this is:** a map of the whole testing harness — every layer, every instrument, what each +one catches that its neighbors don't — plus a single reusable trigger prompt that actually engages +all of it in one connected pass. It does not replace any skill it names; it is the thing that +stops those skills from being individually correct and collectively never invoked. + +**Why this exists.** On a live project (2026-08-21), an agent ran a real, narrated, +headed-browser click-through of one module, called it "the e2e test," and produced a report that +looked complete. The user asked a direct question — "this is not really e2e, is it?" — and it +wasn't: one module walked fully, one action spot-checked, four other built modules untouched, three +of four personas never logged in as. Separately, the same session found that +`process-coherence-pass.md` and `wiring-sweep.md` were both named in `gate-check.sh`'s required- +skill lists and both had a formal row in `obligations.tsv`, and neither had ever produced an +artifact in that project's history — not once, on any module. Being *listed* is not being *run*. +This skill exists so the next session doesn't have to re-derive either lesson from scratch. + +--- + +## 1. The layer stack — what exists, and who owns it + +Five layers. Layer 2 (model-side) never calls layer 3 (runtime), and layer 3 never calls layer 2 — +layer 1's orchestrators are the only place they meet. Full architectural detail, seams, and the +honest-degradation contract: `harness-architecture.md`. This section is the map; that file is the +territory. + +``` +LAYER 4 content journeys/.journey.json · coverage-ledger.md · BRDs +LAYER 3 runtime engine journey-runner.js · monkey.js · report-normalize/render.js +LAYER 2 model-side test-stack-up.sh · conformance-check.sh · graph-sweep.sh · + coverage-check.sh · fixture-manifest.sh +LAYER 1 orchestrators review-module.sh (model-only) · verify-module.sh (full pass) +LAYER 0 spec (skills) journey-proof.md · module-review.md · wiring-sweep.md · + process-coherence-pass.md +``` + +**The exit contract**, everywhere: `PASS` (rc 0, measured correct) / `FINDING` (rc 1, measured +wrong) / `FAULT` (rc 2, did not run at all). A fault is absent, not amber — never let a missing +check read as a clean one. Full discipline: `journey-proof.md` §"Verdict discipline", +`testing-shape.md` §4. + +## 2. Three passes that sound alike and get blurred — read this before running any of them + +These are the ones a session reaches for interchangeably and shouldn't: + +| Skill | Question it answers | What only it can catch | +|---|---|---| +| `module-review.md` (the LOOK) | Does this module render right, interaction by interaction? | A Save button that silently 4xx'd — the mechanical audit can't see this | +| `wiring-sweep.md` | Does every visible affordance on this page actually do something? | Dead wiring from the page/element side — a button with no action behind it | +| `process-coherence-pass.md` | Do individually-correct pieces actually chain into the process a requirement describes? | A component that's correctly built, correctly granted, referenced nowhere — none of the other passes look for this because each one is scoped to a single module or a single page | + +None substitutes for the others. A module can pass `module-review.md` and `wiring-sweep.md` on +every page and still fail `process-coherence-pass.md` if nothing outside the module ever calls into +it. Run all three; don't let a pass on one stand in for the others in a report. + +## 3. The five journey rungs — what "the golden path is proven" actually requires + +One journey = one persona walking a path with carried state, asserted in this order (each rung is +only meaningful if the one above held): + +| # | Rung | Catches | +|---|---|---| +| 1 | Landing guard | without it, every later assertion runs against the *previous* page | +| 2 | What the screen says | the machine can do the work right and the screen can still lie about it | +| 3 | Ordered spans | existence-only checks can't tell a skipped step from a reordered one | +| 4 | Data effects (3 claims: delta / assocMustBeSet / mustPointAt) | saved vs. saved-with-link vs. saved-with-the-*right*-link | +| 5 | Outcome | per-step deltas can each be right while the journey's net result is wrong | + +Full spec, mutants, and the false-green register this exists to close: `journey-proof.md`, +`testing-shape.md` §4. + +## 4. The cadence layer — the thing most likely to be silently unmet + +`obligations.tsv` (`bin/lib/obligations.tsv`) is the mechanical trip-wire: it declares which passes +are owed, by whom, on what cadence, and what artifact discharges each one. Read it directly before +trusting any summary that claims coverage — it is the ground truth for what's owed, not a +description of what usually happens. + +| Obligation | Owner | Scope | Discharged by | +|---|---|---|---| +| `look` | review | module | `design/ui-reviews/ui-review-*.html` | +| `sweep` | test | module | `.claude/loop/sweep//sweep.md` | +| `journeys` | test | module | `.claude/loop/verify//summary.tsv` | +| `coherence` | architect | cluster (every 2–3 proven modules) | `.claude/loop/coherence/last-cluster-pass.tsv` | + +**Check every row against the filesystem, not against whether the relevant skill is *named* in +`gate-check.sh`'s required-skills list.** A skill being in that list means the gate will ask for its +artifact — it does not mean anyone has produced one. `bin/coherence-cadence.sh` mechanizes the +`coherence` row specifically — it counts modules proven since the last recorded pass and exits 1 +(DUE) at threshold. Run it before any full-harness audit; do not assume "not due" without running +it, and do not trust a "not due" verdict on a project whose `architecture/modules/` doesn't follow +`module-brief.md`'s one-directory-per-module layout (see the fixed bug below — a project that +drifted from that convention made the script always report zero, silently, with no error). + +**Known-fixed bug, worth re-checking on any project inheriting an older script copy:** +`coherence-cadence.sh` used to derive "which modules count" by listing `architecture/modules/*/` +subdirectories. `module-brief.md:70` specifies one directory per module — but a project that +instead keeps flat `.md` / `-brief.md` files at that path (a spec drift, not a +supported alternative) made the glob match zero directories, so the script reported **0 proven +modules, always**, no matter how many `.claude/loop/verify//summary.tsv` files existed. +Fixed in `project-bin/coherence-cadence.sh` (and propagated by `sync-project.sh`) to read +`.claude/loop/verify/` directly — that directory is the actual record of what ran, and doesn't +depend on a second directory matching a convention it may not follow. Found and fixed on +A real project, 2026-08-21. + +## 4.5. What fires automatically — the build-plan / module-brief / ledger timeline + +Everything above is reached for on demand. This is the part that isn't optional — the fixed +sequence every module and every project walks through regardless of whether anyone remembers to +invoke a skill by name. Read this before assuming a stage's testing obligations were met just +because the pipeline moved past it. + +**Module brief — the test plan gets decided, not yet run.** Written once at brief sign-off, +before the module's first script (`module-brief.md`'s "Test plan" section): +- which rungs apply (UI always, Data always, Unit yes/no, Trace yes/no) and the **Base set** — the + scripts that must all be done before testing opens at all; +- the **Journeys** table, compiled straight into `journeys/.journey.json` — a projection + of the brief's own Roles/journeys and Golden-path tables, never a separately hand-authored list; +- the **Interactive elements** table — the wiring sweep's denominator, decided here so the sweep + later has something concrete to count against. + +**Stage 4 gate — coverage ledger.** Once, before the build plan is signed off; re-run after any +BRD edit and again before each slice build (`coverage-ledger.md`). Every BRD leaf claimed by a +build-plan row or explicitly catalogued with a reason; UNCLAIMED/PHANTOM/DOUBLE-CLAIMED must all +be empty. **No ledger yet means `conformance-check.sh` and `coverage-check.sh` report FAULT on +every module that follows, forever, until one exists** — this is the correct response to a +genuinely missing input, not an instrument bug, and it is the single most common way a project +ends up with a wall of FAULT rows that look like tooling failure. + +**Per module — the fixed five-step close-out.** Every module, Stage 5, before the next one opens: + +``` +1 BUILD MDL drafted and validated (mxcli check --references, 0 errors) +2 GATE snapshot → exec → mxbuild → auto-restore on failure; 0 errors, lint clean +3 PROVE verify-module.sh, one shot: UI + Data journey rungs, wiring-sweep, monkey pass — + fires once the brief's Base set is fully checked off, never before +4 LOOK the human-eyes pass, every screen in the module — nothing mechanical substitutes + for this; a green PROVE buys no exemption from it +5 CONFIRM one report, denominator stated explicitly, then the next module opens +``` + +**Every 2–3 proven modules — the cluster cadence.** Counted automatically after each module's +CONFIRM step by `coherence-cadence.sh`, which exits DUE at threshold; `process-coherence-pass.md` +then runs and records itself when done. This is the only check that crosses module boundaries. +**Being named in a gate's required-skills list is not the same as this having ever fired** — check +the recorded-pass marker file (`.claude/loop/coherence/last-cluster-pass.tsv`) directly. + +**Stage 6 — whole-project gate.** Once, before cutover: every module's module-review report +exists with zero open P1, a full-app `process-coherence-pass.md` if the cluster cadence hasn't +already covered the tail, optionally `e2e-evidence-report.md` to package the result for a +stakeholder audience. + +## 5. Running a full harness audit — the reusable trigger + +Use this prompt verbatim (substitute the project name) when someone asks for "a full end-to-end +test," "does everything work," or a click-through proof that should actually mean what it says. + +``` +Run a full harness audit of — every layer, every module, every persona, +one connected pass. This is not a single-module walkthrough; if scope has to be cut, name +exactly what was cut in the final report, never round up to "full coverage." + +0. Read harness-architecture.md and journey-proof.md if you haven't this session. Confirm + PROJECT.md's acknowledged toolkit commit matches HEAD (conversion-runbook.md ritual). + Check for other live sessions touching the .mpr (ListAgents) and coordinate before any + mxcli exec / docker rebuild. + +1. Run bin/coherence-cadence.sh. Read its verdict — do not assume "not due" without running + it fresh. If it reports DUE, run process-coherence-pass.md's 4-pass review BEFORE the + click-through, since an audit built on modules that don't actually chain together isn't + proving what it claims to prove. Record with --record when the pass completes. + +2. For every module that has been opened for work (not just the one most recently touched): + a. bin/verify-module.sh — read every rung's individual verdict, not just the + top-line summary. A "journeys: FINDING" or any "*: FAULT" row is not covered by an + overall green. + b. wiring-sweep.md on that module's main pages — click every visible affordance, not + just the happy-path buttons. Confirm nothing is dead-wired. + c. module-review.md's LOOK stage — human-eyes pass, does an interaction silently fail + in a way the mechanical audit can't see. + +3. For every persona with materially different access (check the navigation profile, not + an assumed role list) — log in as each one, confirm menu/page differences, and re-walk + at minimum the one journey most likely to differ by role. Don't run every module as one + persona and call that "full coverage across personas." + +4. Back every claimed pass with DB evidence (OQL) or a runtime log line, per testing-shape.md + §4 — call out explicitly wherever a step only has UI-observed evidence. If a dialog or + error appears, read what it actually says before calling it a pass — a raw system "Error" + dialog is a different finding than a designed empty-state message, even when neither one + crashes. + +5. If something is broken, name it and keep going — do not silently route around it. Log it + as its own finding with reproduction steps. + +6. Consolidate into ONE e2e-evidence-report.md-format artifact: self-contained HTML, + organized by persona then by journey, headline states the denominator plainly + ("N of M modules × P of Q personas actually exercised live, this pass"). Update + docs/BUILD-LOG.md or the project's demo script wherever this pass contradicts a stale + claim there. + +7. Cross-check every obligations.tsv row against the filesystem (§4 above), not against + whether gate-check.sh's required-skills list names the relevant skill. Report which + obligations are actually discharged, which are FAULT, and for `coherence` specifically + whether bin/coherence-cadence.sh itself can even count correctly on this project's + architecture/modules/ layout before trusting its verdict. + +Do not call the result "e2e" in the final summary unless every module and every +materially-distinct persona was actually walked this pass. State exactly what was covered. +``` + +## 6. Reporting the result + +Package with `e2e-evidence-report.md`'s format and rigor discipline — read that file for the exact +per-step field list (action / screenshot / DB evidence / log evidence / verdict) and the +denominator rule. This skill tells you *what* to run and in what order; `e2e-evidence-report.md` +tells you how to package what you found so a stakeholder can trust it on sight. + +--- + +**Related, not superseded:** `harness-architecture.md` (the machine), `journey-proof.md` (the +spec), `testing-shape.md` (the vocabulary and false-green register), `module-review.md` / +`wiring-sweep.md` / `process-coherence-pass.md` (the three passes §2 disambiguates), +`e2e-evidence-report.md` (the report format), `bin/lib/obligations.tsv` (the ground truth for +what's owed). From 9fc2870d38fe8fb440db13297177238a9351171c Mon Sep 17 00:00:00 2001 From: MendixMau Date: Fri, 21 Aug 2026 21:09:31 +0700 Subject: [PATCH 02/11] full-harness-audit: add full-ui-loop, design-audit.js, ux-audit skill entries Rounds out the three-passes table with the whole-app LOOK variant (full-ui-loop) and names the two UI-focused checks that live inside/ alongside LOOK: design-audit.js (mechanical, runs automatically every module) and learned-skill-ux-audit.md (heavier, standalone, on-demand design-system gap analysis). All three were real, already-built parts of the harness that the first pass at this skill left off the map. Co-Authored-By: Claude Sonnet 5 --- skills/full-harness-audit.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/skills/full-harness-audit.md b/skills/full-harness-audit.md index 32b7809..f651d0f 100644 --- a/skills/full-harness-audit.md +++ b/skills/full-harness-audit.md @@ -59,6 +59,25 @@ None substitutes for the others. A module can pass `module-review.md` and `wirin every page and still fail `process-coherence-pass.md` if nothing outside the module ever calls into it. Run all three; don't let a pass on one stand in for the others in a report. +`module-review.md`'s LOOK stage also has a whole-app variant: the same human-eyes pass run once +across every page in every custom-built module, instead of one module at a time — reach for it +before a demo or after a batch of build work spanning several modules. It's not a different check, +just a different scope; a project may wire it as its own command (e.g. `/full-ui-loop`) rather than +adding a fourth row to the table above. + +**Two UI-focused checks live inside/alongside LOOK and are easy to lose track of:** + +- **`design-audit.js`** — not a skill, a Playwright script LOOK's Stage 4b runs automatically. Purely + mechanical: invented CSS classes never promoted to the deployed theme, raw `class=` where a + sanctioned design-system property exists, axe accessibility (serious/critical, one `h1`, + landmarks), horizontal overflow at three widths. Has a `--static-only` mode for when the app + isn't running. This is the answer when someone wants "just a UI check, not the whole human pass." +- **`learned-skill-ux-audit.md`** — a heavier, standalone, on-demand pass: Playwright screenshots + the live app and the design-system reference side by side, then an agent scores gaps across + color, typography, component-pattern reuse, and design-system features never built into the live + app. Triggered explicitly ("run a UX audit", "compare to design system", `/ux-audit`) — it is + **not** part of the automatic per-module close-out, unlike `design-audit.js`. + ## 3. The five journey rungs — what "the golden path is proven" actually requires One journey = one persona walking a path with carried state, asserted in this order (each rung is From 8b5a8bb7e992ec82e0d0d76d784a201bd9d48264 Mon Sep 17 00:00:00 2001 From: MendixMau Date: Fri, 21 Aug 2026 21:27:03 +0700 Subject: [PATCH 03/11] Correct the DECISION prohibition and add the workflow-object corruption class MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit learned-workflow-patterns.md Warning 1 read as an absolute "never emit a DECISION activity" while the same file's §13 already scoped that corruption to a v0.16.0 binary defect — an internal contradiction that sends people to Studio Pro to hand-build gateways mxcli v0.18.0 writes correctly. Warning 1 is now a binary-version gate, matching §13's shape. New sections, from 2026-08-20/21 build experience on v0.18.0: - §14 referencing a not-yet-created microflow from a workflow body corrupts the stored workflow object; the corruption survives creating the microflow afterwards and resurfaces later as CE0495 on untouched sibling activities. Same class as BUG-92. Includes the checker gap, the snapshot-bisect procedure, and an untested recovery path. - §15 DECISION vs CALL MICROFLOW — decided by what the expression must reach, not by how simple the logic looks (bracket-predicate traversal is CE0117). - §16 OUTCOMES is optional; the both-branches-empty dead-branch smell and the two legitimate fixes. - §17 fewer moving parts as a corruption-risk criterion, not style. - §18 confirm PARALLEL SPLIT isn't already there before flagging a fan-out. - §19 no BOUNDARY EVENT TIMER without documented SLA evidence. - §20 `= empty` on an association: valid in an IF, invalid in a RETRIEVE WHERE (XPath has no such comparison). Also adds preflight STOP #20 for the create-before-reference ordering rule, and updates the three routing descriptors. Promoted from personal-toolkit skills/workflow-patterns.md; client project names anonymized per the convention already used in this file. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 2 +- ROUTING.md | 2 +- agents/mdl-agent.md | 2 +- skills/learned-mdl-preflight.md | 1 + skills/learned-workflow-patterns.md | 274 ++++++++++++++++++++++++++-- 5 files changed, 264 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 7692359..d9e29c9 100644 --- a/README.md +++ b/README.md @@ -554,7 +554,7 @@ Every mxcli project has a `.ai-context/skills/` directory (bundled by `mxcli ini | Task | Skill to load | |---|---| -| Writing or debugging a Mendix native Workflow (CREATE WORKFLOW/USER TASK/OUTCOMES) — syntax, the 11 workflow microflow statements, and the binary-version $Type corruption class | `skills/learned-workflow-patterns.md` | +| Writing or debugging a Mendix native Workflow (CREATE WORKFLOW/USER TASK/OUTCOMES) — syntax, the 11 workflow microflow statements, DECISION vs CALL MICROFLOW, and the two corruption classes (binary-version $Type, and create-before-reference) | `skills/learned-workflow-patterns.md` | **Build · Integration** diff --git a/ROUTING.md b/ROUTING.md index f3114ee..6f90ff1 100644 --- a/ROUTING.md +++ b/ROUTING.md @@ -146,7 +146,7 @@ picks the row up. That is the whole procedure — there is no second list to rem | Always relevant for | Load this | Agent(s) | Stage(s) | Tier | |---|---|---|---|---| -| Writing or debugging a Mendix native Workflow (CREATE WORKFLOW/USER TASK/OUTCOMES) — syntax, the 11 workflow microflow statements, and the binary-version $Type corruption class | `skills/learned-workflow-patterns.md` | mdl | 5 | ondemand | +| Writing or debugging a Mendix native Workflow (CREATE WORKFLOW/USER TASK/OUTCOMES) — syntax, the 11 workflow microflow statements, DECISION vs CALL MICROFLOW, and the two corruption classes (binary-version $Type, and create-before-reference) | `skills/learned-workflow-patterns.md` | mdl | 5 | ondemand | #### Build · Integration diff --git a/agents/mdl-agent.md b/agents/mdl-agent.md index fd40820..668b71f 100644 --- a/agents/mdl-agent.md +++ b/agents/mdl-agent.md @@ -64,7 +64,7 @@ a rule below names an asset (e.g. "the wireframe", "the brief"), it means the pa | `project-bin/lint-gate.sh` | Running lint as a gate rather than a report — per-rule ratchet against a committed baseline, plus the crash and collapse guards that stop a blind rule passing | | `skills/improvement-register.md` | Any review pass that runs more than once — module-review, coherence, monkey, wiring-sweep: findings accumulate across runs, a per-run report cannot show a trend | | `skills/wiring-sweep.md` | Every module before it is called done — does every clickable thing actually do something; run AFTER the happy-path journey is green, never before | -| `skills/learned-workflow-patterns.md` | Writing or debugging a Mendix native Workflow (CREATE WORKFLOW/USER TASK/OUTCOMES) — syntax, the 11 workflow microflow statements, and the binary-version $Type corruption class | +| `skills/learned-workflow-patterns.md` | Writing or debugging a Mendix native Workflow (CREATE WORKFLOW/USER TASK/OUTCOMES) — syntax, the 11 workflow microflow statements, DECISION vs CALL MICROFLOW, and the two corruption classes (binary-version $Type, and create-before-reference) | | `skills/rest-integration-first-time-right.md` | Building a REST integration (consumed or published) for the first time on a project — the checks that avoid a rebuild after the first live call | | `skills/bug-submission-checklist.md` | Preparing an mxcli/Studio Pro bug for submission — scope pinning, read-back-vs-write-path verification, gate-sensitivity negative controls, severity scoping, before it's called filable | | `skills/empty-widget-triage.md` | A page/grid/combobox renders empty (blank cells, zero rows, zero options) during UI review or an e2e run — before assuming a single cause | diff --git a/skills/learned-mdl-preflight.md b/skills/learned-mdl-preflight.md index 2a71ef4..f6c803d 100644 --- a/skills/learned-mdl-preflight.md +++ b/skills/learned-mdl-preflight.md @@ -54,6 +54,7 @@ Once you've picked a mode per operation, run the STOP table below against every | 18 | Write `CALL MICROFLOW` inside a **workflow body** without a `WITH` parameter mapping when the target microflow has parameters | See [[learned-workflow-patterns]] §7/§13 for the full workflow-authoring pattern and the related binary-version `$Type` corruption class. **STOP → always add `WITH (Module.MF.ParamName = '$workflowContext')` for every parameter the target microflow declares.** `mxcli check` and `mxcli check --references` both pass — the corruption is invisible until SP is opened or the app is run. Symptom in SP: activity shows as a red pin that cannot be double-clicked; symptom at runtime: `"No new model classes have arrived within ten seconds, aborting model initialization (Class 'Workflows$CallMicroflowTask' could not be found)"`. | Confirmed 2026-07-24 (a PLM parts-flow project, mxcli v0.16.0): script 09 wrote 5 `CALL MICROFLOW` steps — `WF_ACT_AssistPrefill_Prefill`, `WF_ACT_RiskAgent_RiskFlag`, `WF_ACT_PLMStub_AutoFeasibleCheck`, `WF_ACT_PLMStub_SendToDownstream`, `WF_ACT_PLMStub_Archive` — without `WITH` clauses. All 5 target microflows take `($PLMStub: PLM.PLMStub)`. mxcli silently created `CallMicroflowTask` nodes with no parameter binding, exposing the red-pin corruption only on SP open. **Fix pattern:** `CALL MICROFLOW Module."MFName" WITH (Module."MFName"."ParamName" = '$workflowContext');` — fully qualify both the microflow and its parameter name. Omit `WITH` only when the target microflow has zero parameters. Fix script: `mdlsource/3c-workflow/09b-plm-workflow-fix-callmf.mdl` (CREATE OR REPLACE the whole workflow). | | 19 | Write **any workflow MDL script** (CREATE WORKFLOW, ALTER WORKFLOW, or any script that adds user-task outcomes or call-microflow steps to an existing workflow) | See [[learned-workflow-patterns]] for the full syntax reference, warnings, and build order before scripting. **Read the BRD lifecycle section and wireframe for that feature BEFORE writing a single line.** Verify: (a) every user-task's allowed outcomes and their exact names, (b) which microflows belong inside which outcome branch vs unconditionally after the task, (c) whether a CALL MICROFLOW step is actually needed in MDL vs drag-and-drop in SP (prefer SP drag if it avoids the red-pin bug). Only start scripting once you can tick all three. | Confirmed real incident 2026-07-24 (a PLM parts-flow project): `09i-plm-workflow-usertasks-only.mdl` was written without checking the BRD, giving `Release_Decision` a single outcome `'ReleaseDecided'` instead of `'Release'` + `'ReleaseHold'`, and placing `SendToDownstream`/`Archive` unconditionally after the task instead of inside the `Release` branch only. The BRD (F004) and wireframe clearly document both outcomes and the conditional branching; checking first would have caught both mistakes before a single line was written. | | 16 | Write an access-rule XPath constraint with a **path expression inside the `[%CurrentUser/...%]` substitution token** — `[Assoc = '[%CurrentUser/Module.OtherAssoc%]']`, especially crossing module boundaries | **Never put a path inside the CurrentUser substitution. Keep it bare (`'[%CurrentUser%]'`) and do the ENTIRE multi-hop/cross-module traversal on the left-hand side instead**, qualifying the entity name at every module crossing (same rule as the "cross-module association expression paths" MDL gotcha, extended here to access-rule XPath constraints, not just microflow expressions). Was: `[FeasibilityDecision_Supplier = '[%CurrentUser/PLM.Account_Supplier%]']`. **Fix (confirmed working):** `[PLM.FeasibilityDecision_Supplier/PLM.Supplier/Administration.Account_Supplier = '[%CurrentUser%]']` — walk Supplier→Account fully on the left, compare directly to the (already-an-Account) CurrentUser token. | This is genuinely invalid Mendix XPath, not an mxcli-specific bug — `CE0161` "Error(s) in XPath constraint" is Mendix correctly rejecting it every time it's checked, which is why it kept resurfacing across three separate incidents on a PLM parts-flow project (2026-07-22/23, `bug-logs/mxcli-bugs.md`) that were originally misdiagnosed as mxcli corrupting the domain model on unrelated grant rewrites. In fact the "unrelated entity broke too" symptom was just two independently-broken rules (both using the same invalid CurrentUser-path form) surfacing together once a full `mx check` ran — not one rule corrupting the other. Once rewritten to the correct bare-CurrentUser/full-LHS-chain form, the earlier "Studio Pro GUI only, module-wide" restriction no longer applies — **confirmed mxcli-safe 2026-07-23** via `test-05-currentuser-xpath-chain.mdl` (rebuilt all 4 role grants on the affected entity through mxcli exec, reproducing the corrected XPath verbatim, `mx check: 0 errors`). Use mxcli directly for this pattern; no GUI fallback needed. | +| 20 | Write a workflow that `CALL MICROFLOW`s a microflow whose `create/modify microflow` statement is **not in the same script and same exec** | **STOP → move every callee's create/modify statement above the `create or modify/replace workflow` statement in the same file.** Referencing a microflow that exists nowhere in the model corrupts the stored workflow object itself, and the corruption survives creating the microflow afterwards. `mxcli check --references` and `mxcli exec` both pass clean — the checker does not validate `CALL MICROFLOW` targets inside a workflow body against the live model. Native `mx check` reports only the symptom (CE1613), not the damage. See [[learned-workflow-patterns]] §14 for the incident, the bisect procedure, and recovery. | Confirmed 2026-08-21 (a QA-sampling approval project, mxcli v0.18.0): a fix script called a `DEC_*` microflow that only ever existed in a superseded, never-executed script. CE1613 appeared a day later; creating the microflow cleared CE1613 but the workflow then failed `CE0495 "Duplicate name"` across 8 sibling activities the fix had never touched. Same defect class as BUG-92 (orphaned records surviving an edit, invisible to `DESCRIBE`). | **Default to mxcli for:** entities/attributes/enums, associations (after SHOW ASSOCIATIONS check), microflows (without inline assoc-sets), demo users, module roles/grants, navigation. diff --git a/skills/learned-workflow-patterns.md b/skills/learned-workflow-patterns.md index 41af2aa..20ff654 100644 --- a/skills/learned-workflow-patterns.md +++ b/skills/learned-workflow-patterns.md @@ -1,9 +1,10 @@ # Mendix Native Workflows in MDL **Applies to:** any mxcli project scripting a Mendix native Workflow (definition, task -pages, targeting, starting instances) from MDL. Everything here is scriptable; the two -exceptions are a data-driven `DECISION` gateway, which must still be added by hand (§8), -and a visual check in Studio Pro, which is not optional (§8, Warning 3). +pages, targeting, starting instances) from MDL. Everything here is scriptable on a current +binary, including the `DECISION` gateway (§15 — but read Warning 1 first, it is *not* +scriptable on `v0.16.0`). The one thing that is never optional is a visual check in Studio +Pro (§8, Warning 3). **Verified on:** Mendix 11.13.0, mxcli v0.17.0/v0.18.0. Every MDL form that appears in `build/workflow-example.mdl` was confirmed with `mxcli check`. Forms discussed only in @@ -390,28 +391,39 @@ Then **wire the back-reference** on the very next line, as above — see §3 for ## 8. Warnings -### Warning 1 — do not emit a `DECISION` activity inside a workflow +### Warning 1 — `DECISION` corrupts the `.mpr` on old binaries; check your version -**This is a known open mxcli defect, and it corrupts your `.mpr` silently.** +**This was an absolute prohibition until 2026-08-20. It no longer is — but only on a +current binary.** Corrected 2026-08-21 against mxcli **v0.18.0**, where `DECISION` writes +correctly and is often the preferred construct (§15). + +**The defect, on affected builds:** - **Symptom:** the `.mpr` becomes unloadable. Studio Pro and the native `mx` loader fail with `Mendix.Modeler.Storage.StorageLoadException`. - **Cause:** mxcli writes the decision's outcome label as a raw string into a field the - native loader requires to be a real `EnumerationValueIdentifier`. This is unconditional — - it does not depend on the expression's type. + native loader requires to be a real `EnumerationValueIdentifier`. Unconditional — it does + not depend on the expression's type. - **Repro:** something as trivial as `DECISION '1 = 1'` inside a `CREATE WORKFLOW` body. - **Why it is invisible:** `mxcli check`, `mxcli exec` and `DESCRIBE WORKFLOW` all report success. Nothing in the mxcli toolchain sees it. You find out when someone opens the project. -- **There is no MDL-only workaround.** -**What to do instead:** express the branch as **user task outcomes** (§6) — an outcome with -a nested activity block *is* an exclusive branch, and covers most real gateways. Where you -genuinely need a data-driven gateway with no user decision behind it, flatten the flow to -its happy path in MDL, leave a plain MDL comment at the insertion point describing the -gateway precisely, and add that one activity by hand in Studio Pro. +**What to do:** this is the same *binary-version* story as §13 — run `mxcli --version` +first and treat it as a gate, not a folk rule. + +- **On a confirmed-clean binary (v0.18.0 or later):** write the `DECISION` from MDL. Use + §15 to decide whether a given gate should be a `DECISION` at all, then **verify what was + actually stored** (§13's `strings`-on-the-unit check, plus a real `mx check` and an SP + open) before moving on. A passing `mxcli check` still proves nothing here. +- **On `v0.16.0` or an unverified binary:** the old rule stands in full — there is no + MDL-only workaround. Express the branch as **user task outcomes** (§6), since an outcome + with a nested activity block *is* an exclusive branch and covers most real gateways. + Where you genuinely need a data-driven gateway with no user decision behind it, flatten + the flow to its happy path in MDL, leave a plain MDL comment at the insertion point + describing the gateway precisely, and add that one activity by hand in Studio Pro. -`PARALLEL SPLIT` is unaffected. +`PARALLEL SPLIT` is unaffected on every binary (§18). ### Warning 2 — `DESCRIBE MICROFLOW` lies about `CALL WORKFLOW` @@ -585,6 +597,240 @@ of write, regardless of which binary produced it. --- +## 14. Referencing a not-yet-created microflow from a workflow body corrupts the stored workflow + +**This is a distinct defect class from Warning 1 and §13.** Those are binary-version bugs +in how an activity is *written*. This one is a corruption of the stored workflow object +that survives fixing the thing that caused it. + +**Incident** (a QA-sampling approval project, mxcli v0.18.0, 2026-08-20/21): a fix script +rebuilt a workflow with a `call microflow Module.DEC_CoarseClassification` activity, but +that microflow did not exist in the project yet — its `create or modify microflow` +statement lived only in an earlier, superseded script that was never executed. `mxcli +check --references` passed clean and `mxcli exec` completed with no error. Only a later, +separate native `mx check` caught CE1613 (`… no longer exists`). Creating the missing +microflow afterward cleared CE1613 — but **not** the underlying damage. Roughly a day +later the workflow failed with `CE0495 "Duplicate name ''"` across 8 sibling +`CALL MICROFLOW` activities that the fix had never touched. + +**Root cause, confirmed by elimination, not assumed.** It is not a workflow-grammar limit +and not a shared naming namespace between branch subtrees — both theories were disproven +by cloning the exact 17-call nested structure fresh into a throwaway `_TEST` workflow (0 +errors), and separately by retargeting all 17 calls to a project-unique dummy microflow +(still 0 errors). The corruption lives inside the specific stored workflow object's own +Unit-blob storage tree: an orphaned/duplicated activity record left behind by the original +bad write, surviving even after the dangling reference was repaired. **Same defect class +as `bug-logs/mxcli-bugs.md` BUG-92** (orphaned widgets surviving page edits, invisible to +`DESCRIBE`) — here in a workflow instead of a page. The exact write-sequence trigger could +not be reproduced fresh in an isolated `_TEST` object, so treat this as a strong candidate +mechanism, not a nailed-down repro. + +**The reference-checker gap that lets it happen:** `mxcli check --references` and `mxcli +exec` do **not** validate `CALL MICROFLOW` targets inside a workflow body against the live +model. A workflow written against a genuinely nonexistent microflow — not "created later +in the same script," which mxcli's checker does correctly special-case, but never created +anywhere at all — passes both clean. Only native `mx check` catches it (CE1613), once, +well after the damage is done. + +> **Hard rule.** Within any single script that writes a workflow calling one or more +> microflows, the `create/modify microflow` statements for every callee must come +> **before** the `create or modify/replace workflow` statement that references them — same +> file, same exec. Never split "create the workflow" and "create a microflow it calls" +> across two exec passes, even if you intend to run the second immediately after. +> `mxcli check --references` will not catch the ordering mistake; only `mx check`, and even +> then only the symptom, not the corruption it leaves behind. + +**If you suspect this has already happened** (a `CE0495`/duplicate-name error appears on a +workflow that was *not* the target of your most recent edit): don't assume the most recent +script caused it. Bisect via mxcli's auto-snapshots (`.mpr-snapshots//`) — swap +each candidate snapshot's `.mpr` **in place** into the live project directory (preserving +relative paths such as `theme/`, or you get spurious `CE6083` errors from an incomplete +sandbox) and run a real `./mxcli docker check` against each — never `--references`, which +cannot see it — working backward until you find the first clean one. The corrupted state +can predate the change you were about to fix. + +**Recovery — untested against a real corrupted object as of 2026-08-21.** Try `create or +replace workflow` with the byte-identical current definition (same control flow, no +behavior change) first: this is the mechanism BUG-92's page-rebuild precedent uses to force +full regeneration of the stored activity tree. It could not be verified in isolation +because a throwaway copy could not be driven into the corrupted state to test the fix. If +that doesn't clear it, the fallback is `DROP WORKFLOW` + recreate from the same literal +definition — higher risk to inbound references (role grants, `CALL WORKFLOW` start sites, +WF task pages). Either way: **verify with native `mx check` afterward, not +`--references`**, and confirm first that no deployed runtime paired with this `.mpr` has +in-flight instances — a full regeneration can orphan in-flight tasks, and that cannot be +determined from the design-time `.mpr` alone. + +--- + +## 15. `DECISION` vs. `CALL MICROFLOW` — when a boolean gate belongs in the workflow + +**Read Warning 1 first.** On a clean binary, `DECISION` is writable from MDL and is often +the *better* choice. Native workflow has a `DECISION` activity — an inline exclusive +gateway — that needs no microflow at all: + +``` +DECISION [''] [COMMENT ''] + OUTCOMES '' -> { } ...; +``` + +It evaluates an expression directly against the workflow's context data. What decides +whether a boolean gate should be a `DECISION` or stay a `CALL MICROFLOW … OUTCOMES true -> +{} false -> {}` is **not** "is the logic simple" — it's **what the expression needs to +reach**: + +- **Direct attribute access on the context parameter** (`$WorkflowContext/SomeField = + 'SomeValue'` — no association hop, no filter predicate) → `DECISION`. No side effects, no + commit, nothing a microflow buys you, and one fewer stored microflow artifact referenced + by the workflow (§17 for why that matters beyond style). +- **Anything needing a `RETRIEVE` with a `WHERE` predicate, a bracketed + association-traversal filter, a commit, or multi-step logic** → stays a `CALL MICROFLOW`. + Confirmed empirically (2026-08-21): workflow expressions reject the bracket-predicate + traversal form (`$Context/Module.Assoc_Entity[SomeAttr = 'X']`) with `CE0117` — the same + constraint already noted for `CALL MICROFLOW … WITH` binding expressions in §7, and it + applies just as hard inside a `DECISION` expression. A decision that must find "the one + child row matching a key" (filtering a to-many association down to a single row by an + enum/string key) **cannot** be expressed as a `DECISION` — write it as a microflow with a + real `RETRIEVE … WHERE … LIMIT 1`. + +In short: `DECISION` replaces a microflow whose entire body is "read one field, no +retrieve, return a bool." It does not replace a microflow that has to go find the field +first. + +--- + +## 16. `OUTCOMES` is optional — the dead-branch smell + +`CALL MICROFLOW`'s outcome clause is optional in the grammar: + +``` +CALL MICROFLOW Module.MF [COMMENT ''] + [OUTCOMES '' { } ...]; +``` + +Whether a workflow needs outcome branches at all is driven entirely by the called +microflow's **return type**, not by habit: + +- Returns **Void** → no `OUTCOMES` clause needed. The workflow runs the activity and moves + on. +- Returns **Boolean** → the workflow is forced into a two-outcome `true -> { } false -> { }` + shape, whether or not either branch does anything. +- Returns an **enumeration** → one outcome per enum value the workflow actually branches on. + +**The smell:** a `CALL MICROFLOW` whose microflow returns Boolean and whose `true -> { }` +and `false -> { }` outcomes are *both* empty. The workflow computed a boolean and threw it +away — nothing is decided by the branch. This is easy to miss because it looks structurally +identical to a real decision point; only reading the outcome *bodies*, not the outcome +names, reveals it's dead. + +Two legitimate fixes — pick based on whether the boolean carries real business consequence: + +1. **The result genuinely doesn't affect control flow** (a denormalization/stamping helper + whose only failure mode is "no matching row found," already logged internally) → change + the microflow's return type from Boolean to Void, delete its `$Success` variable, and + drop the `OUTCOMES` clause at every call site. Move any error handling that depended on + the boolean (a `LOG ERROR` on the not-found path) **inside** the microflow body first — + the caller can no longer react to it. +2. **The result does carry consequence** ("did the version-fork actually get created," "did + the terminal status-close actually commit") → don't silently drop it. Either make the + `false` branch do something (log at ERROR, escalate, retry, route to a recovery task) or, + at minimum, confirm via `SHOW CALLERS OF` and a read of the callee that failure is + already unreachable before flattening to Void. A boolean quietly discarded on a + *terminal* activity is a different risk profile than one discarded on a per-station + stamp — the same "empty branches" shape can be a harmless no-op or a swallowed failure, + depending entirely on what the callee does on the false path. + +**Signature-change risk:** Boolean → Void is a public signature change. Always run `SHOW +CALLERS OF Module.TheMicroflow` first. If the workflow is the only caller — common for +`ACT_*_ByKey`/shim microflows built specifically for one workflow's `WITH`-clause +limitations — the change is low-risk, but every `CALL MICROFLOW` call site must be updated +**in the same script and the same exec** as the signature change. Same discipline as §14's +create-before-reference rule, and for the same reason: never leave a workflow definition and +a microflow signature out of sync between two execs. + +--- + +## 17. Fewer moving parts is a corruption-risk criterion, not a style preference + +§14's corruption class has a direct design-time countermeasure: **every separately-created +microflow a workflow references is one more chance to get create-before-reference ordering +wrong — across one more script, one more exec, one more future edit.** Collapsing a +microflow-wrapped boolean gate into a native `DECISION` (§15) doesn't just read cleaner; it +permanently removes an artifact-and-reference pair from the workflow's dependency surface. + +Treat "does this activity need to be a separate stored microflow at all, or can it be a +native construct" as a concrete corruption-risk question when auditing an existing workflow +or planning a new one — not only a readability one. This is not a licence to collapse +everything: a microflow doing a real `RETRIEVE`/commit/multi-step mutation still has to be a +microflow (§15). But a pure attribute-equality gate wrapped in a microflow *only* because +that's how the workflow was first drafted is exactly the avoidable moving part this targets. + +--- + +## 18. `PARALLEL SPLIT` — confirm it isn't already there before flagging a fan-out + +Don't assume a set of sibling human tasks described informally as "done in parallel" (in a +module brief, BRD, or prose) is modeled sequentially just because they're adjacent in a +script or listed one after another in a doc. Read the live `DESCRIBE WORKFLOW` output before +flagging a missing `PARALLEL SPLIT`. Confirmed 2026-08-21 on a three-station sub-review: the +workflow already had a genuine `parallel split` with three `path N { user task … }` +branches, each independently completable. `DESCRIBE WORKFLOW` renders nested `path` blocks +clearly, so this is a cheap check — do it before recommending a structural change that +already exists. + +--- + +## 19. `BOUNDARY EVENT TIMER` — don't add one without a documented business trigger + +`BOUNDARY EVENT TIMER` (Mendix 10.6.0+) is available and syntactically simple: + +``` +user task ReviewTask 'Review' + outcomes 'Done' { } + boundary event timer 'P3D' { + call microflow Module.WF_Escalate; + }; +``` + +or after the fact: `ALTER WORKFLOW INSERT BOUNDARY EVENT ON TIMER '' +{ }`. Availability is not a reason to add one to every long-running human task. +Before recommending it, search the actual requirements evidence — BRDs, blueprint, +source-migration triage notes — for SLA/timeout/escalation/deadline language tied to *that +specific task*. Absence of that evidence is a legitimate "no action" finding. Don't invent +an SLA the business never asked for because the construct exists and the task is +human-facing. (Confirmed practice, 2026-08-21: a full-text search across an approval +module's BRDs and migration-triage notes for SLA/timeout/escalation/deadline/overdue +language turned up nothing tied to any of its 15 user tasks — correctly documented as "no +boundary event needed," not silently skipped.) + +This partly supersedes the "Notes on scope" caveat below: boundary-event body syntax has now +been used in a real build. + +--- + +## 20. `= empty` on an association: valid in an `IF`, invalid in a `RETRIEVE WHERE` + +Found 2026-08-21 writing a utility to find context rows with no wired `System.Workflow`. + +``` +RETRIEVE $Runs FROM Module.Entity WHERE RunStatus = X and Assoc_Ref = empty; +``` + +passed both `mxcli check` and `mxcli check --references` clean, but native `mx check` failed +with `CE0161 "Error(s) in XPath constraint"`. A `RETRIEVE … WHERE` clause compiles to an +XPath constraint, and **XPath has no `= empty` comparison for an association reference** — +that syntax exists only in the microflow expression language (IF conditions, decisions). +Fix: filter the retrieve on plain attributes only, then check `$Var/Module.Association = +empty` inside an `IF` in the loop body. + +Separately, that `IF` check itself first failed with `CE0117 "Error(s) in expression"` when +written bare as `$Run/AssocName = empty`. Association references in microflow expressions +need the **module-qualified** association name: `$Run/Module.Entity_Other = empty`. Same +silent-pass-then-native-fail pattern — `mxcli check`/`--references` caught neither defect, +only a real `mx check` run did. + +--- + ## Notes on scope Workflow **timer** and **wait-for-notification** activities are referenced by From 372e2ac1790305b3a163a609c01fb9898913c9f3 Mon Sep 17 00:00:00 2001 From: MendixMau Date: Fri, 21 Aug 2026 21:34:18 +0700 Subject: [PATCH 04/11] Correct the false Studio Pro dependency in the cloud/mobile prompts commands/mobile-dev-loop-prompt.md asserted that the build gate requires Studio Pro and that a headless container therefore cannot run it. That is wrong, and it was stopping cloud sessions at a boundary that does not exist: the gate is mxbuild, a plain binary that project-bin/exec.sh invokes directly, and exec.sh's only Studio Pro coupling is a lock check (refuse to write while SP holds the .mpr) which is a no-op where no SP exists. Both prompts also predate mxcli run --local / --hub / --test-endpoint, which the toolkit did not mention anywhere. A container can now stand the app up itself and publish it at a public URL, so the hand-rolled ngrok/cloudflared tunnel is no longer the only route in. - mobile-dev-loop-prompt.md: dated correction note; DO-NOT list reframed as out-of-scope-for-this-prompt rather than impossible-here. - mobile-auto-test-prompt.md: stand-it-up-here is now the preferred option. - testing-shape.md: mxcli run --local flag table, runtime.log note. - e2e-harness-base.md: BASE_URL no longer assumes localhost:8080. Co-Authored-By: Claude Opus 5 (1M context) --- commands/mobile-auto-test-prompt.md | 25 +++++++++++---- commands/mobile-dev-loop-prompt.md | 50 ++++++++++++++++------------- skills/e2e-harness-base.md | 7 ++-- skills/testing-shape.md | 20 ++++++++++++ 4 files changed, 71 insertions(+), 31 deletions(-) diff --git a/commands/mobile-auto-test-prompt.md b/commands/mobile-auto-test-prompt.md index bafe63a..9834910 100644 --- a/commands/mobile-auto-test-prompt.md +++ b/commands/mobile-auto-test-prompt.md @@ -14,6 +14,14 @@ it has no filesystem in common with your laptop and cannot reach `localhost` on exists so that gap gets stated up front, once, instead of being rediscovered as a confusing `fault` deep into a run. +**Updated 2026-08-21 — the container can now stand the app up itself.** This file used to present +a laptop-hosted app plus a hand-rolled ngrok/cloudflared tunnel as the only options. It is no +longer the shortest path: given the project's repo, this session can run the app locally with +`mxcli docker run` or `mxcli run --local`, and publish it with `mxcli run --local --hub` (a public +URL via `mxcli tunnel-hub`, with `ApplicationRootUrl` set correctly for that origin). Option (c) +below is now the default; (a) and (b) remain valid when you specifically want to test an app +instance that already exists elsewhere. + --- ``` @@ -28,12 +36,17 @@ Setup (do this first, in order): 2. From the toolkit clone, run `bin/doctor.sh` and fix anything it reports missing before continuing — it names exactly what's absent (Python, mxcli binary, etc.) and how to get it. 3. This app must be RUNNING and reachable from this container, plus its database reachable - for OQL/DB assertions. This container cannot reach a laptop's localhost. Either: - (a) it's deployed at a reachable URL — set APP_URL to that, and PG_HOST/PG_PORT (or the - M2EE admin port) to the reachable DB, or - (b) tunnel the local instance out (ngrok/cloudflared) and set APP_URL/PG_HOST to the - tunnel address: . - If neither is possible, stop and say so rather than guessing — a runtime instrument that + for OQL/DB assertions. This container cannot reach a laptop's localhost. In order of + preference: + (c) **stand it up here** — `mxcli docker run -p .mpr --wait`, or `mxcli run --local` + for a Docker-free warm loop (needs JDK 21 and a reachable PostgreSQL whose database + already exists). Add `--hub` if I need to click through it from a browser myself. + Verify with `mxcli docker status`; never infer "up" from the absence of an error. + (a) it's already deployed at a reachable URL — set APP_URL to that, and PG_HOST/PG_PORT (or + the M2EE admin port) to the reachable DB, or + (b) tunnel an existing local instance out and set APP_URL/PG_HOST to the tunnel address: + . + If none is possible, stop and say so rather than guessing — a runtime instrument that can't reach the app should report `fault`/`INVALID`, never be skipped silently or faked green. Skill to follow: skills/existing-app-assurance.md — Track B (Regression / e2e test net). diff --git a/commands/mobile-dev-loop-prompt.md b/commands/mobile-dev-loop-prompt.md index 887c9f1..77199dd 100644 --- a/commands/mobile-dev-loop-prompt.md +++ b/commands/mobile-dev-loop-prompt.md @@ -1,25 +1,30 @@ --- -description: Self-contained prompt to paste into a fresh Claude Code (mobile/cloud) session to run the drafting/static-check half of iterative-build-loop.md — it stops at the Studio Pro-dependent gate rather than faking it +description: Self-contained prompt to paste into a fresh Claude Code (mobile/cloud) session to run the drafting/static-check half of iterative-build-loop.md against an existing project repo --- Copy everything below the line into a new Claude Code session that has no access to your local machine. Fill in the `<...>` blanks first. -**Why this is scoped the way it is:** unlike a pure e2e test run, the build loop's gate needs -**Studio Pro** — a Windows/macOS GUI app — open and reachable at several mandatory points (close -SP → exec → mxbuild gate → reopen SP → Run Locally). The toolkit's own SP automation -(`save-sp.sh`, `restart-sp.sh`) is macOS-only (`osascript`/`lsof`/`open -a`). A headless cloud -container cannot drive a GUI app it has no display for, and there is no tunnel-equivalent fix for -that the way there is for a running web app. So this prompt deliberately does the half of the -loop that has no GUI dependency, and stops cleanly at the handoff instead of pretending to -complete the rest. +**Why this is scoped the way it is.** This prompt is the *drafting* half of the build loop, for +when you want MDL written and statically checked against a project repo without standing up a +runtime. That is a scope choice, not a platform limit. + +> **Correction, 2026-08-21 — this file previously claimed the build gate requires Studio Pro. It +> does not, and that claim blocked cloud sessions at a boundary that isn't real.** The gate is +> `mxbuild`, a plain binary that `project-bin/exec.sh` invokes directly; `exec.sh`'s only Studio +> Pro coupling is a *lock check* (refuse to write while SP holds the `.mpr`), which is a no-op +> where no SP exists. `mxcli new` creates a project headlessly, `mxcli docker run` or +> `mxcli run --local` runs it, `mxcli run --hub` exposes it at a public URL, and +> `mxcli playwright` / `mxcli oql` / `mxcli test --local` exercise it. A container can run the +> whole loop. For the full headless build-and-prove run, use the full-e2e cloud prompt in +> `personal-toolkit/prompts/` instead of this one. --- ``` This is a fresh Claude Code session with no prior context. Task: draft and static-check the next -phase of a Mendix build plan, using mxcli-project-toolkit's iterative-build-loop.md — WITHOUT -Studio Pro, which this session cannot reach. +phase of a Mendix build plan, using mxcli-project-toolkit's iterative-build-loop.md. Scope is +deliberately static: MDL drafted and applied, model-side checks run, no runtime stood up. Setup (do this first, in order): 1. Add and clone these repos: @@ -27,9 +32,9 @@ Setup (do this first, in order): - https://github.com/mendixlabs/mxcli.git (the mxcli CLI — build/install it) - (contains the .mpr this session will read/write against) -2. From the toolkit clone, run `bin/doctor.sh `. It will (correctly) warn that - Studio Pro automation is unavailable here — that's expected, not a problem to fix. Everything - else it reports missing, fix before continuing. +2. From the toolkit clone, run `bin/doctor.sh `. A warning that Studio Pro + automation is unavailable is expected here and is not a problem to fix — nothing in this + prompt's scope uses it. Everything else it reports missing, fix before continuing. 3. Confirm no live Studio Pro elsewhere holds a lock on this project's .mpr (check for a `*.mpr.lock` file, or ask me). mxcli must never touch a .mpr while Studio Pro has it open, including reads — if uncertain, stop and ask rather than risk corrupting the model. @@ -51,20 +56,19 @@ Scope — run through "The Build Loop" steps 1-9 ONLY, then stop: - project-bin/graph-sweep.sh --module - project-bin/coverage-check.sh, against the module's coverage ledger -DO NOT attempt, and do not report as done — hand these back to me explicitly instead: -- Gate: BUILD (bin/exec.sh's mxbuild run + snapshot/auto-restore) — exec.sh's SP handling - requires a live, reachable Studio Pro. -- Reopening Studio Pro, "Update security", or any Cmd+S save. +OUT OF SCOPE for this prompt — do not attempt, and hand these back to me explicitly instead. +These are excluded because this run is deliberately static, NOT because they are impossible here: +- Gate: BUILD (project-bin/exec.sh's mxbuild run + snapshot/auto-restore). Runs fine headless; + it is out of scope only because nothing here stands up a runtime to prove the result. - Gate: UI (module-review.md's PROVE/LOOK stages, project-bin/verify-module.sh, the happy-path - walk) — these need a running app + reachable DB, same as the e2e test prompt - (commands/mobile-auto-test-prompt.md) requires, and this session has neither by default. + walk) — these need a running app + reachable DB, which this prompt does not set up. - Renaming any script to its `done-` prefix — that rename only happens after the FULL gate - (mxbuild + SP reopen + happy-path) passes, which this session cannot verify. + (mxbuild + happy-path) passes, which this session does not verify. Deliverable: the drafted/applied MDL for this phase, syntax-clean per step 6, plus the results of the four static instruments in step 7, plus a short handoff note listing exactly what's left -(mxbuild gate, SP reopen, happy-path walk, module-review.md, done- rename) for me to run locally -or in a session that can reach Studio Pro / the running app. +(mxbuild gate, happy-path walk, module-review.md, done- rename) for me to run in a session that +stands up the app. If you hit anything requiring a judgement call outside this scope (an ambiguous CE error, a requirements gap, whether to touch a shared module), stop and ask rather than guessing — same diff --git a/skills/e2e-harness-base.md b/skills/e2e-harness-base.md index 074093e..f0d4711 100644 --- a/skills/e2e-harness-base.md +++ b/skills/e2e-harness-base.md @@ -14,7 +14,8 @@ Build after completing a module build phase: - Domain model + all microflows done - Pages implemented and reachable via navigation - Seed data loaded (ACT_SeedData_Run executed) -- App running locally (`mxcli docker run -p App.mpr --wait`) +- App running locally (`mxcli docker run -p App.mpr --wait`, or `mxcli run --local` for a + Docker-free warm loop — see `testing-shape.md` for the flag table) --- @@ -23,7 +24,9 @@ Build after completing a module build phase: - Node.js available - Playwright installed: `npm init -y && npm i -D playwright` - `npx playwright install chromium` -- App running at `http://localhost:8080` +- App running at `http://localhost:8080` — or, when this session cannot reach that host (a + cloud container, a devcontainer, a phone), at the public URL from `mxcli run --local --hub`. + Point the harness `BASE_URL` at whichever one actually answers; never assume `localhost:8080`. - A working data-assertion instrument. **Prefer the M2EE admin API** (`mxcli oql --direct`, `adminPort = runtime port + 10`, token from the project's own m2ee config) — see `learned-db-assertions.md`. The `psql.exe` config further down is the Windows-only diff --git a/skills/testing-shape.md b/skills/testing-shape.md index 61625aa..9bfd88f 100644 --- a/skills/testing-shape.md +++ b/skills/testing-shape.md @@ -280,6 +280,26 @@ and only report a blocker if you cannot. | `mxcli docker reload` | rebuild + hot reload after an exec | | `mxcli docker down` | tear down | +**Docker-free alternative — `mxcli run --local`.** Keeps an `mxbuild --serve` process and a +standalone Mendix runtime hot: cold first build ~10-15s, then an incremental rebuild ~1s that is +hot-applied without a restart for page/microflow/text changes (entity, view and association changes +still restart the runtime — the metamodel is reconciled only at startup). Needs Mendix 11.x, JDK 21, +and a reachable PostgreSQL whose database **already exists**; no Docker daemon. Runtime stack traces +and microflow `LOG` output land in `/.mxcli/runtime.log` — the browser only shows a +generic dialog, so that file is where a server-side error is actually readable. + +| Flag | Use | +|---|---| +| `--watch` | rebuild and hot-apply on every model change | +| `--hub` | expose the running app at a public URL through `mxcli tunnel-hub` — a chisel client reverse-tunnels out over 443 and the runtime boots with `ApplicationRootUrl` set to the hub URL, so the app works under that origin. Implies `--local`. | +| `--test-endpoint` | host mxcli's token-guarded test endpoint, so `mxcli test -p --attach` runs against this already-warm app — a couple of seconds instead of ~30 | + +**`--hub` is what makes a container-hosted run reachable.** A cloud/devcontainer session has no +shared filesystem with a laptop and cannot serve `localhost` to one. Before `--hub` the only answer +was a hand-rolled ngrok/cloudflared tunnel; it is now a flag. `mxcli test --local` likewise boots on +mxcli's own runtime (ports 8081/8091, its own `_test` database), so a warm `run --local` +loop can keep serving while tests run. + > ### 🔴 Docker is NOT a safe default if the app calls host services by `localhost` > > **Confirmed the hard way, 2026-08-06, on the first real run of this skill.** Inside a container, From f9fc7fac30b0563c77d7b6ad3e71f0702176407f Mon Sep 17 00:00:00 2001 From: MendixMau Date: Sat, 22 Aug 2026 20:33:30 +0700 Subject: [PATCH 05/11] rest-integration skill v2: retire RULE 0 + occurrence patch, add the import range trap Re-measured on mxcli v0.18.0 / Mendix 11.13.0 with a runtime mxcli test: - entity names need not match JSON element names (proven with zero matches) - JSON structure root 0..1 instantiates fine since v0.17 (BUG-LOCAL-14 fixed) - NEW: import from mapping with no range keyword writes ForceSingleOccurrence=1 and returns EMPTY silently; write 'all'. Repro scripts in fixtures/. Co-Authored-By: Claude Fable 5 --- fixtures/import-mapping-range-repro/flip.mdl | 31 ++ .../import.test.mdl | 6 + fixtures/import-mapping-range-repro/log.mdl | 24 + .../import-mapping-range-repro/maptest.mdl | 52 ++ skills/rest-integration-first-time-right.md | 466 +++++++++--------- 5 files changed, 351 insertions(+), 228 deletions(-) create mode 100644 fixtures/import-mapping-range-repro/flip.mdl create mode 100644 fixtures/import-mapping-range-repro/import.test.mdl create mode 100644 fixtures/import-mapping-range-repro/log.mdl create mode 100644 fixtures/import-mapping-range-repro/maptest.mdl diff --git a/fixtures/import-mapping-range-repro/flip.mdl b/fixtures/import-mapping-range-repro/flip.mdl new file mode 100644 index 0000000..7f3abb1 --- /dev/null +++ b/fixtures/import-mapping-range-repro/flip.mdl @@ -0,0 +1,31 @@ +drop import mapping ApiTest.IMM_SearchRoutes; +drop association "ApiTest"."PageInfo_SearchResponse"; +drop association "ApiTest"."Route_SearchResponse"; +create association "ApiTest"."SearchResponse_PageInfo" from "ApiTest"."SearchResponse" to "ApiTest"."PageInfo" type Reference owner Both; +create association "ApiTest"."SearchResponse_Route" from "ApiTest"."SearchResponse" to "ApiTest"."Route" type ReferenceSet owner Default; +create import mapping "ApiTest"."IMM_SearchRoutes" + with json structure "ApiTest"."JSON_SearchRoutes" +{ + create ApiTest.SearchResponse { + create ApiTest.SearchResponse_PageInfo/ApiTest.PageInfo = pagination { + Page = page, PageSize = pageSize, TotalItems = totalItems, TotalPages = totalPages + }, + create ApiTest.SearchResponse_Route/ApiTest.Route = items { + RouteId = routing_id, RouteCode = route_code, RouteName = route_name, + IsCurrent = is_current, EffectiveFrom = effective_from, RowVersion = row_version + } + } +}; +create or modify microflow "ApiTest"."TEST_ImportCount" () returns Integer as $Out +begin + declare $Json String = '{"pagination":{"page":1,"pageSize":20,"totalItems":2,"totalPages":1},"items":[{"routing_id":"r-1","route_code":"RT-001","route_name":"SMT Main","is_current":true,"effective_from":"2026-01-01T00:00:00","row_version":1},{"routing_id":"r-2","route_code":"RT-002","route_name":"THT Line","is_current":false,"effective_from":"2026-02-01T00:00:00","row_version":3}]}'; + $Response = import from mapping ApiTest."IMM_SearchRoutes"($Json); + retrieve $Routes from $Response/ApiTest."SearchResponse_Route"; + retrieve $Page from $Response/ApiTest."SearchResponse_PageInfo"; + $Count = count($Routes); + declare $Out Integer = $Count * 100; + if $Page != empty then + set $Out = $Out + $Page/TotalItems; + end if; + return $Out; +end; diff --git a/fixtures/import-mapping-range-repro/import.test.mdl b/fixtures/import-mapping-range-repro/import.test.mdl new file mode 100644 index 0000000..0fdf44d --- /dev/null +++ b/fixtures/import-mapping-range-repro/import.test.mdl @@ -0,0 +1,6 @@ +/** + * @test import mapping with non-matching entity names instantiates root, pagination and 2 items + * @expect $result = 202 + */ +$result = call microflow ApiTest.TEST_ImportCount(); +/ diff --git a/fixtures/import-mapping-range-repro/log.mdl b/fixtures/import-mapping-range-repro/log.mdl new file mode 100644 index 0000000..f388af3 --- /dev/null +++ b/fixtures/import-mapping-range-repro/log.mdl @@ -0,0 +1,24 @@ +create or modify microflow "ApiTest"."TEST_ImportCount" () returns Integer as $Out +begin + declare $Json String = '{"pagination":{"page":1,"pageSize":20,"totalItems":2,"totalPages":1},"items":[{"routing_id":"r-1","route_code":"RT-001","route_name":"SMT Main","is_current":true,"effective_from":"2026-01-01T00:00:00","row_version":1},{"routing_id":"r-2","route_code":"RT-002","route_name":"THT Line","is_current":false,"effective_from":"2026-02-01T00:00:00","row_version":3}]}'; + $Response = import from mapping ApiTest."IMM_SearchRoutes"($Json) all on error { + log error node 'MapTest' 'RESULT mapping THREW type={1} msg={2}' with ({1} = $latestError/ErrorType, {2} = $latestError/Message); + return -2; + }; + if $Response = empty then + log error node 'MapTest' 'RESULT root=EMPTY'; + return -1; + end if; + retrieve $Routes from $Response/ApiTest."SearchResponse_Route"; + retrieve $Page from $Response/ApiTest."SearchResponse_PageInfo"; + $Count = count($Routes); + declare $Out Integer = $Count * 100; + if $Page != empty then + set $Out = $Out + $Page/TotalItems; + log info node 'MapTest' 'RESULT page present totalItems={1}' with ({1} = toString($Page/TotalItems)); + else + log warning node 'MapTest' 'RESULT page EMPTY'; + end if; + log info node 'MapTest' 'RESULT items={1} out={2}' with ({1} = toString($Count), {2} = toString($Out)); + return $Out; +end; diff --git a/fixtures/import-mapping-range-repro/maptest.mdl b/fixtures/import-mapping-range-repro/maptest.mdl new file mode 100644 index 0000000..61241d4 --- /dev/null +++ b/fixtures/import-mapping-range-repro/maptest.mdl @@ -0,0 +1,52 @@ +create module "ApiTest"; + +create non-persistent entity "ApiTest"."SearchResponse" ( + "TotalItems": integer, + "Page": integer +); +create non-persistent entity "ApiTest"."PageInfo" ( + "Page": integer, + "PageSize": integer, + "TotalItems": integer, + "TotalPages": integer +); +create non-persistent entity "ApiTest"."Route" ( + "RouteId": string(100), + "RouteCode": string(50), + "RouteName": string(200), + "IsCurrent": boolean, + "EffectiveFrom": datetime, + "RowVersion": integer +); +create association "ApiTest"."PageInfo_SearchResponse" from "ApiTest"."PageInfo" to "ApiTest"."SearchResponse"; +create association "ApiTest"."Route_SearchResponse" from "ApiTest"."Route" to "ApiTest"."SearchResponse"; + +create json structure "ApiTest"."JSON_SearchRoutes" + snippet $${ + "pagination": { "page": 1, "pageSize": 20, "totalItems": 2, "totalPages": 1 }, + "items": [ + { "routing_id": "r-1", "route_code": "RT-001", "route_name": "SMT Main", "is_current": true, "effective_from": "2026-01-01T00:00:00", "row_version": 1 }, + { "routing_id": "r-2", "route_code": "RT-002", "route_name": "THT Line", "is_current": false, "effective_from": "2026-02-01T00:00:00", "row_version": 3 } + ] +}$$; + +create import mapping "ApiTest"."IMM_SearchRoutes" + with json structure "ApiTest"."JSON_SearchRoutes" +{ + create "ApiTest"."SearchResponse" { + create "ApiTest"."PageInfo_SearchResponse"/"ApiTest"."PageInfo" = pagination { + "Page" = page, + "PageSize" = pageSize, + "TotalItems" = totalItems, + "TotalPages" = totalPages + }, + create "ApiTest"."Route_SearchResponse"/"ApiTest"."Route" = items { + "RouteId" = routing_id, + "RouteCode" = route_code, + "RouteName" = route_name, + "IsCurrent" = is_current, + "EffectiveFrom" = effective_from, + "RowVersion" = row_version + } + } +}; diff --git a/skills/rest-integration-first-time-right.md b/skills/rest-integration-first-time-right.md index 641371d..94e2565 100644 --- a/skills/rest-integration-first-time-right.md +++ b/skills/rest-integration-first-time-right.md @@ -1,291 +1,301 @@ -# REST integration in Mendix via mxcli — getting it right the first time +# REST-fed filterable DataGrid in Mendix via mxcli — first time right -**Applies to:** any mxcli project modelling a REST-backed entity. +**Applies to:** any mxcli project that shows data from a REST API in a page: search/filter bar → +microflow calls the API → import mapping → non-persistent rows → DataGrid. The "Route List / +Route Detail" shape. -**Read before modelling any REST-backed entity.** Not after the grid comes up empty. - -Derived from one 6-hour debugging session on a production REST integration (2026-07-29) in which -**four independent defects all presented as the same symptom: an empty grid.** Nothing in the -tooling distinguished them. This file exists so the next integration costs 40 minutes. +**Status:** v2, rewritten 2026-08-22. Every claim below was re-measured on **mxcli v0.18.0 + +Mendix 11.13.0** with a runtime test (`mxcli test`, Docker), not inferred from `DESCRIBE`, +`check --references` or mxbuild. v1 of this skill (2026-07-29) carried two rules that are now +**proven false** — see §0. If you hold a copy of v1, discard it. --- -## RULE 0 — NAMING IS KING - -**Every entity name must exactly match the JSON structure's element name, at every object -level, including the root.** - -``` -schema element entity must be called -───────────────── ───────────────────── -Root Root -Pagination Pagination -ItemsItem ItemsItem -NodesItem NodesItem -EdgesItem EdgesItem -``` +## 0. What changed since v1 — read this even if you know the old skill -Nothing else in this file matters if this is wrong. A mismatch does not warn, does not error, -and does not fail the build. The mapping simply instantiates nothing and every downstream panel -reads empty. +| v1 said | Measured 2026-08-22 on v0.18.0 | Consequence | +|---|---|---| +| **RULE 0 — entity name must equal the JSON element name at every level (`Root`, `Pagination`, `ItemsItem`)** | **False.** `SearchResponse`/`PageInfo`/`Route` mapped to `(Object)`/`pagination`/`items` — not one name matches — and the runtime import returned root + pagination + 2 items. | **Name entities after the domain.** No shared `Root` NPE, no union `ItemsItem`, no `custom name map` gymnastics. The mapping binds by explicit element path; name equality was never a Mendix rule. | +| **mxcli writes JSON structures DEAD (root occurrence `0..0`); run `fix-json-occurrences.sh --fix` before every mapping** | **Fixed since v0.17.0.** v0.17/v0.18 write root `0..1`, array items `0..*`. A fresh, unpatched structure imported correctly. Patching the root to `1..1` changed nothing. | **Retire the patch step.** The fixer script still prints `DEAD` for anything ≠ `1..1` — that verdict is now a false alarm. | +| *(not known)* | **NEW trap — the `import from mapping` activity's range.** Omit the range keyword and mxcli (v0.17 **and** v0.18) writes `ForceSingleOccurrence=1`. On an object-rooted mapping that activity **returns EMPTY, throws nothing, logs nothing** — at `check`, `--references`, mxbuild and runtime. `DESCRIBE` shows it as a trailing `first`. Write **`all`**. | This is the defect the folk-fix "re-select the mapping in Studio Pro" was silently repairing: SP resets the flag to 0. It is also the likeliest cause behind BUG-84 ("array child never populated") and a project's 0-row read-model pages. | -**Proven directly (2026-07-29).** `IMM_PagedThings` returned `(empty)` in the debugger while -its root targeted `ThingsResponse`. Renaming that entity to `Root` — changing nothing -else — made it work. Same for the child: `ItemsItem`, not `ThingVersion`. +The whole v1 diagnosis chain (naming → occurrences → re-select in SP) was three symptoms of +**one flag on the calling activity**, never on the structure or the mapping. -**Do not argue with this from a counterexample.** `IMM_SearchThings` works with semantic names -(`SearchThingsResponse` / `Things`) because it was authored in Studio Pro. Every mapping -mxcli generated with mismatched names came back empty. I used that one counterexample to -dismiss the rule three separate times while the user kept fixing it by hand. If a mapping -returns nothing, check the names first, not last. +--- -### The consequence you must design around +## 1. The pattern, end to end -Every paged endpoint yields the same element names — `Root`, `Pagination`, `ItemsItem` — and -entity names are unique per module. So two paged endpoints in one module cannot each own a -`Root`. +Five documents, one page. Build in this order; each step has a read-back. -**Share the entities.** One `Root`, one `Pagination`, one `ItemsItem` carrying the union of -fields, reused by every mapping in the module: +### 1.1 Filter object — one non-persistent entity per search page ``` -Root ──Root_ItemsItem──▶ ItemsItem (versions AND effectivities) - ──Root_NodesItem──▶ NodesItem - ──Root_EdgesItem──▶ EdgesItem +create non-persistent entity "Mod"."RouteFilter" ( + "RoutingCode": string(50), + "RoutingType": enumeration("Mod"."ENUM_RoutingType"), + "LifecycleState": enumeration("Mod"."ENUM_LifecycleState"), + "PageNo": integer default 1, + "PageSize": integer default 20, + "TotalItems": integer default 0, + "TotalPages": integer default 1 +); ``` -These are non-persistent read-model DTOs, not domain entities. A shared, slightly baggy shape is -the correct trade — fields the calling endpoint doesn't supply are simply empty. - -The alternatives are worse: per-structure Custom Name edits (manual, per structure, in Studio -Pro) or one module per endpoint (sprawl). +- Do **not** name it `Filter` — reserved word, the page's datasource call silently breaks. +- Paging counters live here so the page footer can show them without a second call. -### And the association must match too - -Naming alone isn't enough — the shape has to be right as well: +### 1.2 Response entities — domain names, parent owns the children ``` -Root_ItemsItem Root → ItemsItem ReferenceSet ✅ parent owns, loader walks forward -ChildNode_Root ChildNode → Root Reference ❌ backwards, and singular +create non-persistent entity "Mod"."SearchResponse" (); -- wrapper, may have 0 attributes +create non-persistent entity "Mod"."PageInfo" ( "Page": integer, "PageSize": integer, + "TotalItems": integer, "TotalPages": integer ); +create non-persistent entity "Mod"."Route" ( "RouteId": string(100), "RouteCode": string(50), … ); + +create association "Mod"."SearchResponse_PageInfo" + from "Mod"."SearchResponse" to "Mod"."PageInfo" type Reference owner Both; +create association "Mod"."SearchResponse_Route" + from "Mod"."SearchResponse" to "Mod"."Route" type ReferenceSet owner Default; ``` -A reverse walk on a non-persistent entity resolves as a database query against objects that were -never in a database. Silently zero. - ---- - -## Why this is hard — the thing to internalise - -Every layer in this stack swallows its own failure: - -| layer | how it fails | what you see | -|---|---|---| -| JSON structure with root occurrence 0 | produces no root object | empty list | -| Import mapping with unbound elements | matches nothing | empty list | -| Reverse association retrieve (non-persistent) | resolves as a DB query | empty list | -| `rest call` throwing | caught by `on error` | empty list | -| Stale deployment | runs yesterday's model | empty list | - -`mxcli check`, `--references` and `mxbuild` pass on **all** of them. So does the page render. -You get one symptom for five causes and no signal to separate them. That is the whole problem — -it is not that REST is hard. +- **Direction matters, names don't.** The wrapper → child direction lets the microflow walk + *forward* (`$Response/Mod.SearchResponse_Route`). A child → parent association is walked in + reverse, which on non-persistent objects resolves as a *database query* and returns nothing, + silently. (Re-confirmed 2026-08-22: with child-owned associations the test could not even + compile `$Page/TotalItems` — the reverse walk types as a list.) +- A flat `GET /things/{id}` response can map straight onto the real domain entity; the wrapper + is only for paged/nested responses. +- Reuse the wrapper/PageInfo pair across every paged endpoint in the module — that is the + "reuse standard mapping entities" you want, and it needs no generic names to work. -There is also a hidden layer people don't model: it is **not** API ↔ entity. It is -**API → JSON structure → import mapping → entity**. The structure is invisible in most views, -has no error state, and mxcli generates it broken. +### 1.3 JSON structure — from a captured payload, never from the contract ---- - -## The protocol - -### 1. Capture the real payload before modelling anything - -```bash -curl -s "$BASE/endpoint" -H "$AUTH" | python3 -m json.tool > analysis/json-samples/JSON_Thing.json +``` +create json structure "Mod"."JSON_SearchRoutes" + snippet $${ "pagination": { "page": 1, "pageSize": 20, "totalItems": 2, "totalPages": 1 }, + "items": [ { "routing_id": "r-1", "route_code": "RT-001", "is_current": true, + "effective_from": "2026-01-01T00:00:00", "row_version": 1 } ] }$$; ``` -One file per structure, named after it. **Never model from the contract or an OpenAPI spec.** -Real example: the contract implied a field held a code; the payload showed it holds an id -(`"node-003-1"` vs `"N01"`). Only the bytes tell you. - -Check every endpoint — sibling endpoints often share an identical `{pagination, items}` wrapper and -differ only inside `items`. That similarity is how the wrong payload ends up in the right -structure, and it is invisible until you read the item fields. - -### 2. Name entity attributes from the payload — this is the biggest lever - -Mendix derives each schema element's **Custom Name** from the JSON key by capitalising the first -character and keeping underscores: - -| JSON key | Custom Name | your attribute should be | -|---|---|---| -| `routing_id` | `Routing_id` | `Routing_id` | -| `product_family_code` | `Product_family_code` | `Product_family_code` | -| `is_current` | `Is_current` | `Is_current` | +- `curl` the real endpoint and paste the bytes. Contracts lie about types and field names. +- Give every field a **typed, non-null sample** — `null` in the snippet infers Unknown. +- Occurrences are written correctly on v0.17+. No patch step. -**Why it matters more than it looks.** Studio Pro's *Map automatically* pairs elements to -attributes by **exact name**. With PascalCase entities it matches nothing, silently skips every -field, and — if no entity is assigned — invents a parallel entity that *does* match, leaving you -with two competing domain models. +### 1.4 Import mapping — explicit paths, raw keys on the right -With matching names you never hand-bind anything. That is the real prize: **hand-made bindings -reference schema elements by internal ID, so regenerating a structure destroys every one of them** -and leaves red dots plus CE0272. Name-matched mappings survive regeneration and re-bind in one -click. +``` +create import mapping "Mod"."IMM_SearchRoutes" + with json structure "Mod"."JSON_SearchRoutes" +{ + create Mod.SearchResponse { + create Mod.SearchResponse_PageInfo/Mod.PageInfo = pagination { + Page = page, PageSize = pageSize, TotalItems = totalItems, TotalPages = totalPages + }, + create Mod.SearchResponse_Route/Mod.Route = items { + RouteId = routing_id, RouteCode = route_code, IsCurrent = is_current, + EffectiveFrom = effective_from, RowVersion = row_version + } + } +}; +``` -Do **not** rename UI-only entities (filter holders, view models). They are not mapping targets; -keep them in your normal convention. +- **Left** of `=` is the attribute; **right** is the raw JSON key as the API sends it. Since + #882 (v0.18.0) either the raw key or Mendix's exposed name (`Routing_id`) resolves — but + `DESCRIBE` prints the exposed form, so its output is still not a safe paste-back. +- **Unquoted identifiers inside the mapping body.** This inverts the always-quote rule; quotes + here are stored literally and produce 58× CE1613. +- **Never `create or modify` a mapping that contains an array** — drop, then create. -It is not a mechanical PascalCase→snake_case transform. Derive from the payload: -`ThingVersionId` → `Thing_version_id`, not `Version_id`. +### 1.5 The search microflow — URL as one string, `all`, explicit error path, log ladder -### 2b. Map automatically matches OBJECT names too — align Custom Names, not entities +``` +create microflow "Mod"."ACT_Route_Search" ("Filter": "Mod"."RouteFilter") +returns List of "Mod"."Route" as $Routes +begin + $NoRoutes = create list of Mod."Route"; -- error-path return, declared FIRST + declare $CodeParam String = if $Filter/RoutingCode = empty then '' else $Filter/RoutingCode; + declare $TypeParam String = if $Filter/RoutingType = empty then '' else toString($Filter/RoutingType); + declare $Url String = @Mod.ApiBaseUrl + '/routes' + + '?routingCode=' + $CodeParam + '&routingType=' + $TypeParam + + '&page=' + toString($Filter/PageNo) + '&pageSize=' + toString($Filter/PageSize); + log info node 'RouteSearch' '[1] url={1}' with ({1} = $Url); + + $Json = rest call get '{1}' with ({1} = $Url) + header 'Accept' = 'application/json' + timeout 30 returns String on error continue; + log info node 'RouteSearch' '[2] json len={1}' with ({1} = toString(length($Json))); + + $Response = import from mapping Mod."IMM_SearchRoutes"($Json) all on error { + log error node 'RouteSearch' '[3E] mapping THREW type={1} msg={2}' + with ({1} = $latestError/ErrorType, {2} = $latestError/Message); + return $NoRoutes; + }; + if $Response = empty then + log warning node 'RouteSearch' '[3] mapping returned EMPTY — range flag or dead structure'; + return $NoRoutes; + end if; + + retrieve $Routes from $Response/Mod."SearchResponse_Route"; + retrieve $Page from $Response/Mod."SearchResponse_PageInfo"; + $Count = count($Routes); + log info node 'RouteSearch' '[4] items={1}' with ({1} = toString($Count)); + if $Page != empty then + change $Filter (TotalItems = $Page/TotalItems, TotalPages = $Page/TotalPages); + end if; + return $Routes; +end; +``` -**Attribute names alone are not enough.** *Map automatically* matches at two levels: +Why each line is there: -| schema element | must equal | +| Line | Reason | |---|---| -| the **ROOT** object (always named `Root`) | the **entity** name — so the entity must literally be called `Root` | -| every nested object (`Pagination`, `ItemsItem`, `NodesItem`, `EdgesItem`) | the **entity** name | -| the **leaf** elements (`routing_version_id`…) | the **attribute** names | - -**The root counts.** This is the part that is easy to miss and cost hours: a mapping whose root -object targets a semantically-named entity (`ThingsResponse`) instantiates NOTHING under -auto-map, and the failure is invisible — `check --references` is green, the build stops reporting -it, and the only symptom is the response variable reading `(empty)` in the debugger. - -**Consequence — one auto-mappable structure of a given shape per module.** Every paged endpoint -of that shape produces `Root` / `Pagination` / `ItemsItem`. Entity names are unique within a -module, so the second structure of that shape cannot have its entities named the same way. Either -give each structure distinct Custom Names by hand, or accept that only one mapping per module is -auto-mappable and hand-bind the rest. - -If the object name does not match any entity, Studio Pro **creates a new one** — plus a new -association — rather than reusing yours. Observed 2026-07-29: one module accumulated -`Item`, `ItemItems`, `Items`, `ItemsItem` and `Pagination`, all duplicating hand-built -`Thing` / `ThingVersion` / `SearchThingsPagination`, from repeated auto-map runs. Each new -mapping silently retargeted to the duplicate, so the microflow's retrieve over the ORIGINAL -association returned nothing while the mapping "worked". - -**Fix direction: change the schema's Custom Name, not the entity.** Mendix derives `ItemsItem` -from the array's item object, but the **Custom Name column is editable**. Set it to your entity's -name (`ThingVersion`) and auto-map binds to the real entity. - -Custom Name is a display/binding label with no other meaning. An entity name carries domain -meaning and is referenced across pages, microflows and access rules. Bend the one that costs -nothing. - -Do this for the object element BEFORE running Map automatically the first time — retargeting an -already-generated mapping means deleting the duplicate entity and its association too. - -### 3. JSON structures: mxcli writes them DEAD (BUG-LOCAL-14) - -Every structure mxcli creates gets **root occurrence `0..0`**. A root at 0 never instantiates, so -the mapping yields nothing — with no error at any gate. Measured, not recalled: +| `$NoRoutes` first | a custom `on error {}` opens a branch that never reaches the retrieve; it must return something or CE0108 | +| unset enum → `''` | any non-blank value is an exact-match filter server-side; `toString(empty)` poisons the URL | +| one `{1}` for the whole URL | multi-placeholder templates do not substitute — they go out literally | +| `@Mod.ApiBaseUrl` constant | a hard-coded `localhost` is the container under Docker; constant syntax proven | +| `on error continue` on `rest call` | the only handler it accepts; a block is CE6035. An empty body is how failure presents | +| **`all`** on `import from mapping` | **see §0** — omitted = `ForceSingleOccurrence=1` = silent EMPTY | +| `[3]` empty check | distinguishes "threw" from "mapped nothing" — the two must never look alike | +| `$Count = count($Routes)` | bare `$x = count(...)` is the aggregate form; `length()` is for strings | +| counters from `$Page` | `count($Routes)` only sees one page, never the true total | + +### 1.6 The page — filter bar above the grid, both inside the filter dataview ``` -create json structure Mod."JSON_Test" snippet $${...}$$; - → ROOT occ = 0..0 +create page "Mod"."Route_List" ( + Title: 'Routes', + Layout: Atlas_Core.Atlas_TopBar, + Params: { $FilterObj: Mod.RouteFilter } +) { + dataview dvFilter (DataSource: $FilterObj, Class: 'card') { + layoutgrid filterBar { + row row1 { + column col1 (DesktopWidth: AutoFill) { textbox txtCode (Label: 'Routing code', Attribute: RoutingCode) } + column col2 (DesktopWidth: AutoFill) { combobox cbType (Label: 'Type', Attribute: RoutingType) } + } + row row2 { + column col1 (DesktopWidth: AutoFill) { + container actions (Class: 'page-actions') { + actionbutton btnReset (Caption: 'Reset', + Action: microflow Mod.ACT_Filter_Reset(FilterObj: $FilterObj)) + actionbutton btnSearch (Caption: 'Search', ButtonStyle: Primary, + Action: microflow Mod.ACT_Filter_Apply(FilterObj: $FilterObj)) + } + } + } + } + datagrid dgRoutes (DataSource: MICROFLOW Mod.ACT_Route_Search(Filter: $currentObject)) { + column RouteCode (Attribute: RouteCode, Caption: 'Code', Sortable: false) + column Lifecycle (Caption: 'Lifecycle', ShowContentAs: customContent) { + dynamictext txtBadge (Attribute: LifecycleState, Class: 'badge badge-info') + } + } + } +}; ``` -Two ways out: - -**(a) Regenerate in Studio Pro** — paste the snippet, Refresh, confirm the top `(Object)` row reads -Occurrence `1`. Reliable, manual, does not scale. - -**(b) Patch the bytes** — `bin/fix-json-occurrences.sh --fix`. The value is two integers; rewriting -them does not change file length. mxcli stores them as **int32 (BSON 0x10)**, Studio Pro as -**int64 (0x12)** — handle both. Verified: model loads, `DESCRIBE` round-trips, independent reader -confirms `1..1`. *Not yet verified: Studio Pro acceptance and runtime mapping output — confirm both -before trusting it on a real build.* - -Either way, **verify occurrence the moment the structure exists**, before building anything on it. - -Multi-array structures (e.g. `nodes` + `edges`) must be checked on **every** array. A half-fixed -structure gives you a populated first list and a permanently empty second one, which looks exactly -like a mapping bug and is not. - -### 4. Instrument the microflow while writing it, not when it breaks - -Five log lines. They are what make the failure modes distinguishable: +- The grid must sit **inside** the dataview that supplies the filter object — a sibling gets + CE1571 (scope, not a missing argument). +- Put the filter widgets in a `layoutgrid` **outside** the grid, not in the grid's `controlbar`: + control-bar widgets resolve against the *row* entity and CE1613 on every combobox. +- A DataGrid-2 `textfilter`/`dropdownfilter` filters the rows the datasource already returned. + For server-side filtering the inputs bind to the filter object, and the API does the work. +- Search/Reset are a **no-op write with `refresh`** — there is no "re-run datasource" action: ``` -[1] URL -[2] json len -[3] mapped -[3E] THREW -[4] retrieved +create microflow "Mod"."ACT_Filter_Apply" ("FilterObj": "Mod"."RouteFilter") +begin + change $FilterObj (PageNo = 1) refresh; +end; ``` -Read them as a decision tree: +- An opener microflow creates the filter object and shows the page: + `$F = create Mod.RouteFilter; show page Mod.Route_List(FilterObj: $F);` +- Binding an Action onto an **existing** button needs `alter page … replace btn with {…}` + in full; `set` cannot bind actions. A datasource can be set after the fact with + `alter page … { set DataSource = microflow … on dgRoutes }`, but list-view datasource + **arguments** are dropped by ALTER — write list-view datasources inline at CREATE time. +- Detail page: same shape without the filter object — `dataview (DataSource: $Route)` and + child `listview`s whose datasource microflows take `$currentObject/RouteId`. -- `[2]` zero → the call failed. Check `[3E]`. -- `[2]` large, `[3]` zero → **dead structure or unbound mapping.** -- `[3]` fine, `[4]` zero → **wrong association direction.** -- All fine, page empty → **stale deployment**, or the page reads a different variable. +--- -The single log line `[2] json length=4774` is what finally cracked the reference session. Add it -first, not last. +## 2. Prove it before the page exists — the 60-second runtime test -### 5. Never let `on error` return empty silently +Deploy-and-click is the slow oracle. `mxcli test` is the fast one, and it is the only one +that catches the §0 trap. Put this beside the mapping on day one: ``` -$Response = import from mapping Mod.IMM_Thing($Json) on error { - log error node 'X' '[3E] mapping THREW — type=' + $latestError/ErrorType - + ' message=' + $latestError/Message; - return $NoResults; -}; +-- tests/import.test.mdl +/** + * @test import mapping instantiates root, pagination and the items array + * @expect $result = 202 + */ +$result = call microflow Mod.TEST_ImportCount(); +/ ``` -An empty list meaning "it threw" and an empty list meaning "no matches" must never look the same. - -Also: mxcli **silently attaches `on error rollback`** to `import from mapping` and `call microflow` -when you don't specify one (BUG-LOCAL-11). Always write the handler explicitly, and read it back. - -### 6. Response wrapper owns the children; always retrieve forward - -Create a wrapper entity per response and give it a **reference set** to the child entity, owned by -the parent: - ``` -create association Mod."ThingResponse_Thing" - from Mod."ThingResponse" to Mod."Thing" [reference_set] owner both; +create microflow "Mod"."TEST_ImportCount" () returns Integer as $Out +begin + declare $Json String = ''; + $Response = import from mapping Mod."IMM_SearchRoutes"($Json) all; + if $Response = empty then return -1; end if; + retrieve $Routes from $Response/Mod."SearchResponse_Route"; + retrieve $Page from $Response/Mod."SearchResponse_PageInfo"; + $Count = count($Routes); + declare $Out Integer = $Count * 100; + if $Page != empty then set $Out = $Out + $Page/TotalItems; end if; + return $Out; +end; ``` -Then `retrieve $Items from $Response/Mod.ThingResponse_Thing` walks **forward**. - -A reverse walk (child → parent) on **non-persistent** entities is resolved by a database query and -silently returns nothing, because non-persistent objects are not in the database. This defect cost -a full day twice in the same module. - -### 7. Deploy before concluding anything - -A saved model is not a running model. After any structure or mapping edit, **redeploy** before -judging the result. And note: `deployment/model`'s *directory* mtime does not move when files -inside are overwritten — it is not a deploy timestamp. Check a file inside it. - -### 8. One session owns writes - -Two agents (or two terminals) writing one `.mpr` means measurements go stale between reading and -reasoning. Symptoms appear and vanish for no visible cause. +`./mxcli test tests/ -p app.mpr` (Docker; `--local` cannot run on macOS — the cached mxbuild +is the Linux build). The encoding `items×100 + totalItems` makes every failure mode a different +number: `-1` root never instantiated (range flag), `200` pagination branch unbound, `2` items +branch unbound, `202` correct. **`mxcli test` writes a `MxTest` module into the `.mpr` and +restores it afterwards — it is a model write; ask first on a real project.** --- -## Verification checklist — before calling an integration done +## 3. Verification checklist — before calling the integration done ```bash -./bin/fix-json-occurrences.sh # every structure root 1..1? -./mxcli -p app.mpr -c "DESCRIBE IMPORT MAPPING Mod.IMM_X" # bindings present? -curl -s "$URL" | python3 -m json.tool | head # payload still the shape you modelled? +./mxcli -p app.mpr -c "DESCRIBE MICROFLOW Mod.ACT_Route_Search" | grep "import from" +# must end in "all" — a trailing "first" is the silent-EMPTY flag +./mxcli -p app.mpr -c "DESCRIBE IMPORT MAPPING Mod.IMM_SearchRoutes" # both branches present? +curl -s "$URL" | python3 -m json.tool | head # payload still the shape you modelled? +./mxcli test tests/ -p app.mpr # 202 ``` -Then in the running app, read the `[1]`–`[4]` log lines. **A clean build is not a working page** — -mxbuild is blind to every failure in the table at the top of this file. +Then in the running app read `[1]`–`[4]`. Decision tree: `[2]` zero → call failed; +`[2]` large, `[3]` EMPTY → **range flag** (was: dead structure); `[3]` fine, `[4]` zero → +association direction; all fine, page empty → stale deployment or a different variable. + +**A clean build is not a working page.** mxbuild is blind to every row in that tree. --- +## 4. Things that still hold from v1 + +- Capture the real payload before modelling anything; only the bytes tell you a "code" field + holds an id. +- Deploy before concluding anything — `deployment/model`'s directory mtime is not a deploy + timestamp. +- One session owns writes to a given `.mpr`. +- mxcli silently attaches `on error rollback` to `import from mapping` and `call microflow` + when you write no handler (BUG-LOCAL-11); a rollback swallows the exception. Write the + handler, read it back. +- Fixture facts cost hours: an endpoint that genuinely returns 0 children looks exactly like + a broken mapping. Know which fixture id has data before you debug. + ## Related - `learned-mdl-preflight.md` — write-mode choice and the STOP table -- `tool-output-is-not-ground-truth.md` — why read-back is mandatory -- project `bug-logs/mxcli-bugs.md` — BUG-LOCAL-09 … -17 +- `learned-page-patterns.md` — DataGrid-2 filter placement for database-backed grids +- `tool-output-is-not-ground-truth.md` — why the runtime test, not `DESCRIBE`, is the oracle +- `bugs/bug-84-import-mapping-array-child-never-populated.md` — re-run its discriminating test + with `all` before filing anything +- project `bug-logs/mxcli-bugs.md` — BUG-LOCAL-14 (resolved), BUG-LOCAL-33 (the range flag) From 4f75473ce7757f5b7e035c43ad5e4a82611a40e9 Mon Sep 17 00:00:00 2001 From: MendixMau Date: Tue, 25 Aug 2026 16:42:07 +0800 Subject: [PATCH 06/11] Fix Windows: unrun mxbuild gate, and gate-check's O(n^2) fork storm MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found during a Windows training round where gate-check took 5-15 minutes per run and participants switched it off. CORRECTNESS — the mxbuild gate never ran on Windows at all. _common.sh looked for Studio Pro only at /Applications/*.app and exec.sh got JAVA_HOME only from /usr/libexec/java_home, both macOS-only. Under Git Bash both resolved empty, so exec.sh's guard `[ -x "$MXBUILD" ] && [ -x "$JAVA_EXE" ]` was false and the whole gate block was skipped. It reported `skipped` rather than a false `pass` — it was built "three states, not two" for this reason — but a skip without a stated cause is a skip nobody acts on, so every MDL exec on a Windows machine went unverified. That is the BSON-corruption class that iterative-build-loop.md:243 names mxbuild as the only reliable detector for. - mxtk_platform() + Windows branches in find_sp_app/find_mxbuild (C:\Program Files\Mendix\\modeler\mxbuild.exe, version-sorted) - find_java/find_java_exe, preferring Studio Pro's own bundled JRE - exec.sh now says WHY the gate will be skipped, naming the missing binary PERFORMANCE — gate-check forked ~1,130-5,790 processes per run. _ob_waiver() ran per (obligation x unit) and re-parsed the ENTIRE register on each call, forking `tr` AND `sed` per line: O(register x obligations x units). Free at ~4ms/fork on macOS; on Git Bash, where MSYS emulates fork() and a spawn measured 152ms on a training laptop, it was minutes. - register parsed ONCE per path into a memoized cache, one awk pass - _ob_lc(): fork-free lowercase via a global (bash 3.2 has no ${var,,}) - also fixes a portability bug the rewrite subsumes: `sed 's/^[ \t]*//'` read \t as a real tab under GNU sed but as literal backslash-or-t under BSD sed, so macOS silently stripped a leading 't' from keys VB-USI-main 5,790 -> 226 forks (Windows ~880s -> ~34s) fieldrun-keyist 1,127 -> 186 forks (Windows ~171s -> ~28s) scaling is now ~flat per module instead of ~+800 forks each ESCAPE HATCHES — MXTK_SKIP_GATES=1 (exit 0, loud on stderr, never reads as a pass) and MXTK_NO_FETCH=1. The protocol-freshness `git fetch` also ran on every invocation with no timeout over an HTTPS remote; it is now capped by git's own low-speed abort with GIT_TERMINAL_PROMPT=0. No `timeout` wrapper: macOS ships neither timeout nor gtimeout. Verified: gate-check output byte-identical across 4 projects x 11 stages (controlled for advise()'s one-shot state). test-bug02-register.sh 8 ok, test-bug03-gates.sh 9 ok. test-source-sufficiency-gate.sh has 2 failures that reproduce identically with these changes stashed — pre-existing, not from here. Co-Authored-By: Claude Opus 5 (1M context) --- bin/gate-check.sh | 41 ++++++++++++- bin/lib/obligation-check.sh | 85 +++++++++++++++++++++----- project-bin/_common.sh | 118 ++++++++++++++++++++++++++++++++++-- project-bin/exec.sh | 19 +++++- 4 files changed, 238 insertions(+), 25 deletions(-) diff --git a/bin/gate-check.sh b/bin/gate-check.sh index e67488e..cfe2168 100755 --- a/bin/gate-check.sh +++ b/bin/gate-check.sh @@ -76,6 +76,32 @@ if [ $# -lt 1 ]; then exit 1 fi +# --- Escape hatches, for when the gate itself is the thing blocking the room ------------- +# +# MXTK_SKIP_GATES=1 — return "no verdict" immediately and do no work at all. Added during a +# training round (2026-08-25) where gate-check took 5-15 minutes per invocation on participant +# machines that could not be reproduced here: on this Mac a full run is 1.9s empty / 7.8s on a +# real project, and the artifact tree walk — the previously known slow path, see the prune +# comment further down — costs 0.30s worst case over a 107k-file project. +# +# Exits 0 DELIBERATELY, including for a stage-specific run that would normally exit non-zero +# on failure. A skipped gate must not read as a failed one, and it must not wedge a caller +# that branches on the exit code. It is loud on stderr for the same reason the WAIVED verdict +# exists: a check nobody performed has to say so, every time, or it becomes green-by-absence. +# +# MXTK_NO_FETCH=1 — keep every gate, drop only the network. The protocol-freshness check runs +# `git fetch` against the toolkit remote on EVERY invocation (see TOOLKIT_REF below) over an +# HTTPS remote with no timeout, so a proxy that swallows packets or a credential prompt nobody +# is there to answer stalls the whole run. The fallback that this forces is already designed +# and already non-blocking: local HEAD, labelled UNVERIFIED. Prefer this over SKIP_GATES — +# it keeps the verdicts and only gives up the "is your toolkit current?" answer. +if [ "${MXTK_SKIP_GATES:-0}" = "1" ]; then + echo "gate-check: SKIPPED — MXTK_SKIP_GATES=1 is set in this environment." >&2 + echo " No stage was evaluated and no verdict was produced. This is NOT a pass." >&2 + echo " Re-enable with: unset MXTK_SKIP_GATES" >&2 + exit 0 +fi + PROJECT_DIR="$1" REQUESTED_STAGE="${2:-}" @@ -1409,8 +1435,19 @@ TOOLKIT_HEAD="$(git -C "$TOOLKIT_DIR" rev-parse --short HEAD 2>/dev/null || echo # fall back to local HEAD so offline work isn't stranded, but label the verdict UNVERIFIED so a # pass that proved nothing cannot read as a pass that did. SYNC_REF_LABEL="origin" -if git -C "$TOOLKIT_DIR" rev-parse --abbrev-ref '@{u}' >/dev/null 2>&1 \ - && git -C "$TOOLKIT_DIR" fetch --quiet 2>/dev/null; then +# The fetch is the only network call in this script and it had no timeout and no opt-out, so a +# slow or captive network turned a local file check into an unbounded wait. Two guards, both of +# which fall through to the existing UNVERIFIED path rather than failing: +# - MXTK_NO_FETCH=1 skips it outright. +# - Otherwise it is capped by git's own low-speed abort (~8s). A `timeout`/`gtimeout` wrapper +# was rejected: macOS ships NEITHER, so it would have silently broken every trainer's Mac. +# GIT_TERMINAL_PROMPT=0 stops it blocking forever on credentials nobody is there to type. +if [ "${MXTK_NO_FETCH:-0}" = "1" ]; then + TOOLKIT_REF="$TOOLKIT_HEAD" + SYNC_REF_LABEL="local HEAD — UNVERIFIED, fetch skipped (MXTK_NO_FETCH=1)" +elif git -C "$TOOLKIT_DIR" rev-parse --abbrev-ref '@{u}' >/dev/null 2>&1 \ + && GIT_TERMINAL_PROMPT=0 GIT_SSH_COMMAND="${GIT_SSH_COMMAND:-ssh -o BatchMode=yes -o ConnectTimeout=5}" \ + git -C "$TOOLKIT_DIR" -c http.lowSpeedLimit=1000 -c http.lowSpeedTime=8 fetch --quiet 2>/dev/null; then TOOLKIT_REF="$(git -C "$TOOLKIT_DIR" rev-parse --short '@{u}' 2>/dev/null || echo "unknown")" else TOOLKIT_REF="$TOOLKIT_HEAD" diff --git a/bin/lib/obligation-check.sh b/bin/lib/obligation-check.sh index 35a032f..4f1bb1e 100755 --- a/bin/lib/obligation-check.sh +++ b/bin/lib/obligation-check.sh @@ -73,36 +73,91 @@ _ob_register_lines() { # Spelling, matching the per-stage waivers gate-check.sh already reads: # Waived obligation look/Orders: integration module, no pages # Waived obligation sweep: this project tests in the client's own suite (all units) +# --- Register parsing: ONCE per file, not once per lookup --------------------------------- +# +# WHY (2026-08-25). _ob_waiver() is called for every (obligation x unit) pair, and each call +# re-read the whole register, forking `tr` AND `sed` for EVERY LINE. Measured on a one-module +# project: 603 `sed` + 345 `tr` out of 1,127 total forks in a single gate-check run — and it +# scaled at roughly +800 forks per module, so it got worse exactly as a project grew. +# +# That is free on Linux/macOS (~4ms a fork) and ruinous under Git Bash on Windows, where MSYS +# emulates fork() with CreateProcess and a spawn measured 152ms on a training laptop. 1,127 +# forks x 152ms = 171s, which is why gate-check "hung" for 5-15 minutes there and got switched +# off. See the same root cause in project-bin/_common.sh's platform note. +# +# The parse is now one awk pass, memoized per register path. Trim + lowercase happen inside +# that pass rather than per line per lookup. +# +# NOTE ON \t: the old `sed 's/^[ \t]*//'` was not portable. BSD sed (macOS) does not read \t as +# tab inside a bracket expression — it read it as the literal characters backslash and 't', so +# a key beginning with 't' had it stripped. GNU sed (Git Bash, Linux) read a real tab. The +# awk below trims real whitespace on every platform, which is what the code always meant. +_OB_REG_NORM="" # cached "keyvalue" lines, key lowercased, both trimmed +_OB_REG_NORM_FILE="" # which register path _OB_REG_NORM was built from + +_ob_register_norm() { # sets _OB_REG_NORM + local reg="$1" + [ "$_OB_REG_NORM_FILE" = "$reg" ] && return 0 + _OB_REG_NORM_FILE="$reg" + _OB_REG_NORM="$(_ob_register_lines "$reg" | awk ' + { i = index($0, ":"); if (i == 0) next + k = substr($0, 1, i-1); v = substr($0, i+1) + gsub(/^[ \t]+/, "", k); gsub(/[ \t]+$/, "", k) + gsub(/^[ \t]+/, "", v); gsub(/[ \t]+$/, "", v) + if (v == "") next + print tolower(k) "\t" v }')" + return 0 +} + +# _ob_lc — sets _OB_LC to the lowercased string. Sets a global instead of echoing +# because "$(_ob_lc x)" would fork a subshell, which is the cost this whole change removes. +# Hand-rolled because bash 3.2 (macOS's /bin/bash) has no ${var,,}. +_OB_LC="" +_ob_lc() { + local s="$1" out="" c i=0 n=${#1} + while [ $i -lt $n ]; do + c=${s:$i:1} + case $c in + A) c=a ;; B) c=b ;; C) c=c ;; D) c=d ;; E) c=e ;; F) c=f ;; G) c=g ;; H) c=h ;; + I) c=i ;; J) c=j ;; K) c=k ;; L) c=l ;; M) c=m ;; N) c=n ;; O) c=o ;; P) c=p ;; + Q) c=q ;; R) c=r ;; S) c=s ;; T) c=t ;; U) c=u ;; V) c=v ;; W) c=w ;; X) c=x ;; + Y) c=y ;; Z) c=z ;; + esac + out="$out$c"; i=$((i+1)) + done + _OB_LC="$out" +} + _ob_waiver() { local reg="$1" ob="$2" unit="$3" line key val want_u want_a # Compared case-insensitively: reg_field() lowercases the key it reads, and a waiver that # silently missed because someone typed "orders" for module "Orders" would be a waiver the # author believes is in force and the gate does not — the worst of both. - want_u="$(printf 'waived obligation %s/%s' "$ob" "$unit" | tr '[:upper:]' '[:lower:]')" - want_a="$(printf 'waived obligation %s' "$ob" | tr '[:upper:]' '[:lower:]')" - while IFS= read -r line; do - key="${line%%:*}"; val="${line#*:}" - [ "$key" = "$line" ] && continue - key="$(printf '%s' "$key" | tr '[:upper:]' '[:lower:]' | sed 's/^[ \t]*//;s/[ \t]*$//')" - val="$(printf '%s' "$val" | sed 's/^[ \t]*//;s/[ \t]*$//')" - [ -n "$val" ] || continue + _ob_lc "waived obligation $ob/$unit"; want_u="$_OB_LC" + _ob_lc "waived obligation $ob"; want_a="$_OB_LC" + _ob_register_norm "$reg" + local IFS=$'\n' + for line in $_OB_REG_NORM; do + key="${line%%$'\t'*}"; val="${line#*$'\t'}" if [ "$key" = "$want_u" ] || [ "$key" = "$want_a" ]; then printf '%s\n' "$val"; return 0 fi - done < <(_ob_register_lines "$reg") + done return 1 } _ob_adopted_stage() { local reg="$1" line key val - while IFS= read -r line; do - key="${line%%:*}"; val="${line#*:}" - [ "$key" = "$line" ] && continue - key="$(printf '%s' "$key" | tr '[:upper:]' '[:lower:]' | sed 's/^[ \t]*//;s/[ \t]*$//')" + _ob_register_norm "$reg" + local IFS=$'\n' + for line in $_OB_REG_NORM; do + key="${line%%$'\t'*}"; val="${line#*$'\t'}" [ "$key" = "adopted at stage" ] || continue - printf '%s\n' "$(printf '%s' "$val" | sed 's/^[ \t]*//;s/[ \t]*$//' | awk '{print $1}')" + # First whitespace-delimited word, as `awk '{print $1}'` gave. Value is already trimmed, + # so the leading-blank case awk tolerated cannot arise here. + printf '%s\n' "${val%%[ ]*}" return 0 - done < <(_ob_register_lines "$reg") + done return 1 } diff --git a/project-bin/_common.sh b/project-bin/_common.sh index 8eded93..2e73c2d 100755 --- a/project-bin/_common.sh +++ b/project-bin/_common.sh @@ -82,7 +82,39 @@ find_mpr() { } # --------------------------------------------------------------------------- -# find_sp_app — newest installed Studio Pro .app bundle. +# mxtk_platform — "macos" | "windows" | "linux". +# +# WHY THIS EXISTS (2026-08-25). Everything below used to assume macOS: Studio Pro +# was looked for at /Applications/*.app and Java at /usr/libexec/java_home. Under +# Git Bash on Windows both lookups return nothing, so exec.sh's gate guard +# `[ -x "$MXBUILD" ] && [ -x "$JAVA_EXE" ]` was false on every run and the whole +# mxbuild block was skipped. The gate reported `skipped` — honestly, it was built +# "three states, not two" for exactly this reason — but nobody reads a skip as a +# problem, so every MDL exec on a Windows machine went unverified. That is the +# BSON-corruption class iterative-build-loop.md:243 says mxbuild is the ONLY +# reliable detector for. Found during a Windows training round. +# +# $OSTYPE is set by the shell; `uname -s` is the fallback for shells that do not +# export it. Git Bash reports MINGW64_NT-*, MSYS2 reports MSYS_NT-*. +# --------------------------------------------------------------------------- +mxtk_platform() { + case "${OSTYPE:-}" in + darwin*) echo macos ; return ;; + msys*|cygwin*|win*) echo windows ; return ;; + esac + case "$(uname -s 2>/dev/null)" in + Darwin) echo macos ;; + MINGW*|MSYS*|CYGWIN*|Windows*) echo windows ;; + *) echo linux ;; + esac +} + +# --------------------------------------------------------------------------- +# find_sp_app — newest installed Studio Pro root. +# +# Returns an .app bundle on macOS and a version directory on Windows; callers +# must go through find_mxbuild() rather than appending a path themselves, because +# the layout below the root differs per platform. # # Version-sorted, NOT lexically sorted: with 11.9.0 and 11.13.0 both installed, # a plain `sort` picks 11.9.0 as "highest" because '9' > '1' at the third @@ -92,9 +124,45 @@ find_mpr() { # --------------------------------------------------------------------------- find_sp_app() { if [ -n "${MENDIX_APP:-}" ]; then echo "$MENDIX_APP"; return 0; fi - local list - list=$(ls -d /Applications/Mendix\ Studio\ Pro*.app 2>/dev/null) || true - [ -z "$list" ] && { echo "ERROR: no 'Mendix Studio Pro *.app' in /Applications" >&2; return 1; } + local list="" root + case "$(mxtk_platform)" in + macos) + list=$(ls -d /Applications/Mendix\ Studio\ Pro*.app 2>/dev/null) || true + [ -z "$list" ] && { echo "ERROR: no 'Mendix Studio Pro *.app' in /Applications" >&2; return 1; } + ;; + windows) + # Studio Pro installs as C:\Program Files\Mendix\\ . Both Program + # Files roots are checked, plus whatever Windows says they are — a machine + # with a relocated install (D:\ is common on managed laptops, and the + # training round's Git lived on D:) is not reachable by hardcoded /c. + # NB: `PROGRAMFILES(X86)` cannot be expanded as ${...} — parentheses are not + # legal in a bash identifier and it fails at RUNTIME with "bad substitution" + # while passing `bash -n` cleanly. printenv is the only way to read it. + for root in "${ProgramW6432:-}" "${PROGRAMFILES:-}" \ + "$(printenv 'PROGRAMFILES(X86)' 2>/dev/null)" \ + "/c/Program Files" "/c/Program Files (x86)" \ + "/d/Program Files" "/d/Mendix" "/c/Mendix"; do + [ -n "$root" ] || continue + # Env vars arrive in Windows form (C:\Program Files); make them POSIX. + case "$root" in + [A-Za-z]:*) root="/$(printf '%s' "${root%%:*}" | tr '[:upper:]' '[:lower:]')${root#*:}" + root=$(printf '%s' "$root" | tr '\\' '/') ;; + esac + [ -d "$root/Mendix" ] && root="$root/Mendix" + [ -d "$root" ] || continue + list="$list$(ls -d "$root"/*/ 2>/dev/null)" + done + list=$(printf '%s\n' "$list" | sed 's:/*$::' | grep -v '^$') || true + [ -z "$list" ] && { + echo "ERROR: no Mendix Studio Pro install found under Program Files\\Mendix." >&2 + echo " Set MENDIX_APP= or MXBUILD_PATH=." >&2 + return 1; } + ;; + *) + echo "ERROR: Studio Pro does not run on this platform; set MXBUILD_PATH to skip discovery." >&2 + return 1 + ;; + esac if printf '1.10\n1.9\n' | sort -V >/dev/null 2>&1; then printf '%s\n' "$list" | sort -V | tail -1 @@ -105,12 +173,50 @@ find_sp_app() { fi } -# find_mxbuild — the mxbuild binary inside the chosen SP app. $MXBUILD_PATH overrides. +# find_mxbuild — the mxbuild binary inside the chosen SP install. $MXBUILD_PATH overrides. find_mxbuild() { if [ -n "${MXBUILD_PATH:-}" ]; then echo "$MXBUILD_PATH"; return 0; fi local app app=$(find_sp_app) || return 1 - echo "$app/Contents/modeler/mxbuild" + case "$(mxtk_platform)" in + windows) echo "$app/modeler/mxbuild.exe" ;; + *) echo "$app/Contents/modeler/mxbuild" ;; + esac +} + +# find_java — JAVA_HOME for the mxbuild invocation. $JAVA_HOME wins if already set. +# +# /usr/libexec/java_home is a macOS binary and does not exist anywhere else, so on +# Windows this used to leave JAVA_HOME empty and JAVA_EXE as the literal "/bin/java". +# Studio Pro ships its own JRE, which is the right one to use — it matches the +# mxbuild it is paired with — so that is tried before any system Java. +find_java() { + if [ -n "${JAVA_HOME:-}" ] && [ -d "$JAVA_HOME" ]; then echo "$JAVA_HOME"; return 0; fi + local app jh + if [ "$(mxtk_platform)" = macos ] && [ -x /usr/libexec/java_home ]; then + jh=$(/usr/libexec/java_home 2>/dev/null) && [ -n "$jh" ] && { echo "$jh"; return 0; } + fi + # Studio Pro's bundled JRE. + if app=$(find_sp_app 2>/dev/null); then + for jh in "$app/jre" "$app/Contents/jre" "$app/runtime/jre"; do + [ -d "$jh" ] && { echo "$jh"; return 0; } + done + fi + # System Java, resolved from the java on PATH (two levels up from bin/java). + local j + j=$(command -v java 2>/dev/null) && [ -n "$j" ] && { + jh=$(dirname "$(dirname "$j")"); [ -d "$jh" ] && { echo "$jh"; return 0; }; } + return 1 +} + +# find_java_exe — the java binary itself, matching find_java's home. +find_java_exe() { + local jh + jh=$(find_java) || return 1 + case "$(mxtk_platform)" in + windows) echo "$jh/bin/java.exe" ;; + *) echo "$jh/bin/java" ;; + esac } # project_name — the .mpr basename without extension, for user-facing messages. diff --git a/project-bin/exec.sh b/project-bin/exec.sh index 79e7bb2..480d166 100755 --- a/project-bin/exec.sh +++ b/project-bin/exec.sh @@ -166,8 +166,23 @@ echo "→ Snapshotting model..." ./bin/snapshot-mpr.sh MXBUILD="$(find_mxbuild)" || true -JAVA_HOME=$(/usr/libexec/java_home 2>/dev/null || true) -JAVA_EXE="${JAVA_HOME}/bin/java" +# Was: JAVA_HOME=$(/usr/libexec/java_home ...) — a macOS-only binary, so on Windows +# JAVA_HOME came back empty, JAVA_EXE was the literal "/bin/java", the gate guard +# below failed its -x test and the entire mxbuild block was skipped on every run. +# find_java/find_java_exe (project-bin/_common.sh) resolve per platform and prefer +# Studio Pro's own bundled JRE, which is the one paired with this mxbuild. +JAVA_HOME="$(find_java 2>/dev/null || true)" +JAVA_EXE="$(find_java_exe 2>/dev/null || true)" + +# Say WHY the gate will be skipped, at the point we can still name the cause. The +# gate's own "skipped" verdict is honest but arrives without a reason, and a skip +# nobody can explain is a skip nobody acts on — which is how Windows machines ran +# unverified execs for an entire training round. +if [ ! -x "$MXBUILD" ] || [ ! -x "$JAVA_EXE" ]; then + echo "⚠ mxbuild gate will be SKIPPED — this exec will NOT be model-verified." >&2 + [ -x "$MXBUILD" ] || echo " mxbuild not found/executable: ${MXBUILD:-} (set MXBUILD_PATH=)" >&2 + [ -x "$JAVA_EXE" ] || echo " java not found/executable: ${JAVA_EXE:-} (set JAVA_HOME=)" >&2 +fi # ── JSON reader ────────────────────────────────────────────────────────────── # The mxbuild gate reads its verdict out of a JSON errors file, so it needs a From dcd45f801b9467275d258aa8627dabbd257736f6 Mon Sep 17 00:00:00 2001 From: MendixMau Date: Tue, 25 Aug 2026 22:48:08 +0800 Subject: [PATCH 07/11] Bind the module brief to the write, and give verification a row Three changes from a customer training round's post-mortem (2026-08-25), each traced to an artifact rather than a hunch. exec.sh gains guard 5: no module brief, no write. The brief was enforced only by check_build_ready() -- a command nobody is obliged to run -- so a project that never declared itself build-ready was never asked for one. That project executed 25 scripts against a module whose architecture/modules// held only definition.md. Satisfied by either the separate file or a "## Module brief -- " heading in the plan, because single-module projects should merge the two rather than keep documents that overlap ~70%. Fires on writes INTO a module, including "create or modify module role" (roles are the brief's access table); not on bare CREATE MODULE, which is scaffolding an empty shell -- and which keeps tests/wave2/test-bug07-08.sh's fixture working. brd-to-build-plan.md Step 5 gains a row schema with four kinds -- BUILD, PROVE, RUN, HARNESS -- one number sequence, and a Skills column where "none" is valid but blank is not. That plan had 35 build rows and zero verification rows, and not because anyone dropped them: every column of the schema in use described a build, so there was nowhere to write "prove the mapping returns 202". The brief becomes row 0 of its module's phase, which is what gives the guard above a state to read. module-brief.md's "Build skills to read first" asked only about Workflow and Agent decisions, so integration fell through: skill-routing.tsv had rest-integration-first-time-right.md in build/integration and nothing ever named it. All 15 "import from mapping" calls shipped without the range keyword (silent EMPTY, clean at check/--references/mxbuild/runtime) and all 4 wrapper associations pointed child->parent. It now covers every build group and is a roll-up of the rows' Skills column, not a second authoring pass. iterative-build-loop.md:136 said the brief is enforced manually. It isn't now. Tested: 12/12 case matrix (comment-stripped phantom modules, both brief forms, wrong-module heading, FORCE_EXEC, platform skip, multi-module, bare CREATE MODULE) plus a 28-script replay of that round's real day 1 -- first refusal at script 1 of 28. No wave2 fixture was run; test-bug07-08.sh was verified by inspection, which is how the CREATE MODULE regression was caught before it shipped. Co-Authored-By: Claude Opus 5 (1M context) --- project-bin/exec.sh | 89 +++++++++++++++++++++++++++++++++- skills/brd-to-build-plan.md | 49 +++++++++++++++++++ skills/iterative-build-loop.md | 2 +- skills/module-brief.md | 33 ++++++++++--- 4 files changed, 165 insertions(+), 8 deletions(-) diff --git a/project-bin/exec.sh b/project-bin/exec.sh index 480d166..9017f3d 100755 --- a/project-bin/exec.sh +++ b/project-bin/exec.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # exec.sh — the guard chain around a model write. # -# concurrent-writer guard → mxcli check → snapshot → baseline → exec +# concurrent-writer guard → module-brief guard → mxcli check → snapshot → baseline → exec # → mxbuild gate → auto-restore on regression → SP reopen # # Usage: ./bin/exec.sh @@ -79,6 +79,93 @@ if [ -n "$MPR_DIRTY" ]; then echo " (FORCE_EXEC set — proceeding despite uncommitted changes)" fi +# 5. No module brief for a module this script writes to. +# +# WHY. The brief is the mdl-agent's single per-module input: the access-table slice, +# screens-per-role, field-level validation rules, edge cases, and the wireframe->page map. +# Nothing else in the project carries them. Without it the agent synthesises them from +# training data, and every access-rights / wrong-binding / invented-validation incident +# traces back to that. It was previously enforced only by check_build_ready() in +# gate-check.sh — a separate command nobody is obliged to run — so a project that never +# declared itself build-ready was never asked for one. Real incident (a customer training round, +# 2026-08-25): 12 domain scripts and 13 feature scripts executed against a module whose +# architecture/modules// held only definition.md. Binding it to the write is the point; +# there is nothing to rewrite when it fires, unlike a missing skill, so firing late still +# costs only the brief. +# +# Satisfied by EITHER form, because single-module projects merge the brief into the plan +# rather than maintaining two documents that overlap ~70%: +# a) architecture/modules//module-brief.md +# b) a "## Module brief — " heading in architecture/build-plan.md +# +# Platform and marketplace modules are skipped — this project does not author their briefs. +brief_missing="" +if [ -f "$SCRIPT" ]; then + # Comments FIRST, before any pattern match. A `-- ... CREATE MODULE errors if ...` line in a + # real workshop script otherwise yielded a module named "errors" and would have demanded a + # brief for it forever. A guard that cries wolf gets switched off — same reasoning as + # bin/check-portability.sh's "NOT on the denylist" note. POSIX awk, no python/perl: Git Bash + # on Windows has to run this too. + # Assigned single-quoted, invoked double-quoted: bash does not re-expand the *result* of a + # variable expansion, so awk's $0 survives. The alternative (inlining the program in quotes) + # was tried and mangled it — $0 expanded to the shell script's own name. + awk_strip_comments=' + { + s = $0; out = ""; i = 1 + while (i <= length(s)) { + c = substr(s, i, 2) + if (!inblk && c == "/*") { inblk = 1; i += 2; continue } + if (inblk && c == "*/") { inblk = 0; i += 2; continue } + if (!inblk && c == "--") { break } + if (!inblk) out = out substr(s, i, 1) + i++ + } + print out + }' + script_body=$(awk "$awk_strip_comments" "$SCRIPT" 2>/dev/null) + + # Modules written to by this script: the qualified name on a create/alter/drop target. + script_modules=$(printf '%s\n' "$script_body" \ + | grep -oiE '(create|alter|drop)[[:space:]]+(or[[:space:]]+modify[[:space:]]+)?(persistent[[:space:]]+|non-persistent[[:space:]]+)?[a-z_ ]*[[:space:]]"?([A-Za-z_][A-Za-z0-9_]*)"?\.' \ + | grep -oE '"?[A-Za-z_][A-Za-z0-9_]*"?\.$' | tr -d '".' | sort -u) + # DELIBERATELY NOT `CREATE MODULE ` on its own. Creating an empty module is scaffolding; the + # brief describes what goes *inside* one, so demanding it before the module exists is backwards. + # It also keeps tests/wave2/test-bug07-08.sh working — its fixture project execs + # `CREATE MODULE "Nope";` with no architecture/ at all, and would otherwise fail every case. + # + # Everything else about a module DOES need the brief first, including `create or modify module + # role .` — module roles are the subject of the brief's access table, so authoring them + # unspecified is the exact gap this guard exists to close. Replaying that round's real day + # 1 (28 scripts, empty architecture/) the first refusal lands on script 1, at the module-role + # line in 01-app-scaffold.mdl. That is the intended blast radius: the brief is row 0, so nothing + # numbered after it may run first. + + for m in $(echo "$script_modules" | grep -v '^$' | sort -u); do + case "$m" in + System|Administration|Atlas_Core|Atlas_Web_Content|Atlas_UI_Resources|MxModelReflection |CommunityCommons|Encryption|NanoflowCommons|WebActions|DeepLink|MxTest) continue ;; + esac + [ -f "$PROJECT_ROOT/architecture/modules/$m/module-brief.md" ] && continue + grep -qiE "^#+[[:space:]]*Module brief[[:space:]]*(—|-|:)[[:space:]]*$m[[:space:]]*$" \ + "$PROJECT_ROOT/architecture/build-plan.md" 2>/dev/null && continue + brief_missing="$brief_missing $m" + done +fi +if [ -n "$brief_missing" ]; then + echo "✗ No module brief for:$brief_missing — refusing to write a module nothing has specified." + echo "" + for m in $brief_missing; do + echo " $m — expected architecture/modules/$m/module-brief.md" + echo " or a '## Module brief — $m' section in architecture/build-plan.md" + done + echo "" + echo " The brief is row 0 of this module's phase. It carries the access table, screens-per-role," + echo " validation rules, edge cases, wireframe->page map and test plan — see module-brief.md." + echo " → Draft it (ba-agent translation mode, pulling architect-agent), then sign it off in chat." + echo " Override (proceeds with nothing having specified this module): FORCE_EXEC=1 ./bin/exec.sh $SCRIPT" + [ "$FORCE" = "1" ] || exit 1 + echo " (FORCE_EXEC set — proceeding with no module brief for:$brief_missing)" +fi + echo $$ > "$LOCK" trap 'rm -f "$LOCK"' EXIT # ───────────────────────────────────────────────────────────────────────────── diff --git a/skills/brd-to-build-plan.md b/skills/brd-to-build-plan.md index 5eb219d..03d6ac1 100644 --- a/skills/brd-to-build-plan.md +++ b/skills/brd-to-build-plan.md @@ -245,6 +245,55 @@ See `design-artifacts.md` and `learned-stylegallery.md` for full process. ## Step 5: Produce the Numbered Script Sequence +### Row schema — four kinds, one number sequence + +Every row is one of four kinds, sharing one numbering and one dependency graph. **A row may say +`not built`; it may never be absent.** No caption stands in for several items. + +| Col | Meaning | +|---|---| +| # | step number, dependency-ordered, never reused | +| Kind | `BUILD` / `PROVE` / `RUN` / `HARNESS` | +| Step | the `.mdl` script, the command, or the harness to invoke | +| Produces / Proves | the model artifact, or what a pass demonstrates | +| Depends on | prior row number(s) | +| **Skills** | required reading before authoring this row, from `bin/lib/skill-routing.tsv`. **`none` is a valid value but must be written** — blank is not. The module brief's "Build skills to read first" is the union of this column across the module's rows. | +| State | `built` / `partial` / `not built` / `blocked` | + +- **`BUILD`** — writes model. One entity, one microflow group (≤6), or one page section. +- **`PROVE`** — headless and mechanical, mid-phase. `mxcli test`, a `curl`, a `DESCRIBE` read-back. +- **`RUN`** — deploy, reopen Studio Pro, walk the happy path as a demo user. +- **`HARNESS`** — invoke a named harness or skill (`journey-runner.js`, `design-audit.js`, …). + +`PROVE`/`RUN`/`HARNESS` rows carry **no test content** — only what to run, what counts as a pass, +and what they depend on. Content lives in `testing-shape.md`, the brief's Test plan, and the +harness. This is the *moment*; the shape is Step 1's column and the detail is the brief. + +> **RULE: every phase ends in at least one verification row. A phase of only `BUILD` rows is +> malformed — reject it and add the row.** + +**Why this is a schema and not advice (a customer training round, 2026-08-25).** That plan had 35 build +rows and one coverage ledger at the end — zero verification rows. Nobody omitted them: the row +schema in use was `# / script / produces / depends on / write mode / state`, every column of which +describes a build. There was nowhere to write "prove the mapping returns 202" or "run the journey", +so no one did, and a day's work went unverified on machines where the mxbuild gate was also +silently skipping. Give verification a row and it can be numbered, depended on, counted, and +rendered in `build-plan.html` like anything else. + +### Row 0 of every phase is that module's brief + +The brief (`module-brief.md`) is the first row of its module's phase — not stockpiled upfront, not +produced after the previous module's gate. That placement makes it early enough to be an input and +late enough not to be speculative, and it gives it a state that `build-plan-status.sh` can show and +`project-bin/exec.sh`'s module-brief guard can enforce. + +**Single-module projects: merge it.** The brief's sections become sections of the build plan under +a `## Module brief — ` heading (the form the exec guard also accepts). A project-wide +ordering over one module is that module's ordering; two documents at ~70% overlap is how one of +them ends up unwritten — which is exactly what happened to the workshop project above. + +--- + Combine Steps 1–4 into a concrete, ordered list. If Phase 2 UI Scaffold is confirmed, it appears as a block between Phase 1 and the first feature module: diff --git a/skills/iterative-build-loop.md b/skills/iterative-build-loop.md index 3a37820..16d1ad4 100644 --- a/skills/iterative-build-loop.md +++ b/skills/iterative-build-loop.md @@ -133,7 +133,7 @@ still had the old (wrong) description and two already-resolved open questions st Run this before scripting each module: -- [ ] **Module brief exists and passes its ready-check.** `architecture/modules//module-brief.md` must exist (authored by `ba-agent` translation mode, per `module-brief.md`) with every ready-check box ticked: every screen has a wireframe, the access table covers every element, no open business question blocks this phase, write mode chosen for every STOP-row element. **No brief, or an unchecked ready-check item touching this phase → STOP.** Produce/complete the brief first — do not let the `mdl-agent` synthesize the module from raw BRDs. This is the just-in-time gate: mechanical `gate-check.sh` cannot enforce it (briefs don't all exist at Stage 4), so it is enforced here, manually, per module. +- [ ] **Module brief exists and passes its ready-check.** `architecture/modules//module-brief.md` must exist (authored by `ba-agent` translation mode, per `module-brief.md`) with every ready-check box ticked: every screen has a wireframe, the access table covers every element, no open business question blocks this phase, write mode chosen for every STOP-row element. **No brief, or an unchecked ready-check item touching this phase → STOP.** Produce/complete the brief first — do not let the `mdl-agent` synthesize the module from raw BRDs. This is the just-in-time gate. `gate-check.sh` cannot require all briefs at Stage 4 (they don't all exist yet), but it is no longer manual-only: **`project-bin/exec.sh` refuses a write to a module with no brief** — guard 5, satisfied by either `architecture/modules//module-brief.md` or a `## Module brief — ` heading in the build plan (the single-module merged form), overridable with `FORCE_EXEC=1`. The brief is row 0 of the module's phase (`brd-to-build-plan.md` Step 5), so in the normal phased flow the guard cannot fire; it is the net for building off-plan. - [ ] Read source screenshots for this module top-to-bottom - [ ] Read the feature doc (F-doc or BRD) for this module - [ ] Extract the build checklist from the feature doc: diff --git a/skills/module-brief.md b/skills/module-brief.md index 00907cf..8bc9d7d 100644 --- a/skills/module-brief.md +++ b/skills/module-brief.md @@ -73,7 +73,13 @@ list into the brief, stop — link it instead and synthesize the *decision* abou a model state that later changes is a brief that lies. The **first** module's brief is produced at Stage 4 alongside the build plan; the rest are produced as each module's build begins. - Do **not** stockpile all briefs upfront. `gate-check.sh` cannot mechanically require all briefs - at Stage 4 for this reason — the brief gate is a **manual pre-module check** in the build loop. + at Stage 4 for this reason. It is not a manual-only check any more, though: the brief is **row 0 + of its module's phase** (`brd-to-build-plan.md` Step 5), and `project-bin/exec.sh` guard 5 + refuses any write to a module that has no brief. Row 0 placement is what makes the guard + unreachable in the normal flow — early enough to be an input, late enough not to be speculative. +- **Single-module projects: merge the brief into the build plan** under a `## Module brief — ` + heading (the exec guard accepts that form). Two documents at ~70% overlap is how one goes + unwritten. --- @@ -192,10 +198,23 @@ rather than wondering whether they missed a file. ### Domain summary (exact names live in the domain MDL — link above) - Entities: · key associations: -### Build skills to read first (from PROJECT.md's Workflow scope / Agent-wiring scope decisions) - +### Build skills to read first (every build group, not just Workflow/Agent) + - - Date: Wed, 26 Aug 2026 06:31:08 +0800 Subject: [PATCH 08/11] sync: name the two exec.sh fixes a stale project is missing known_fix_note had one entry. A project scaffolded before 2026-08-25 has an exec.sh that differs from the toolkit's, so sync correctly classifies it as locally modified and reports drift rather than overwriting -- it cannot tell "the user hardened this" from "the toolkit moved on", and blind-overwriting would destroy real work in six projects on this machine. The consequence is that the drift line was the ONLY thing standing between a Windows project and an exec.sh whose mxbuild gate never runs, and it said only how many lines differed. Now it names both misses: the Windows mxbuild gate (macOS-only path resolution meant every exec on Git Bash went unverified) and the module-brief guard. A reader can decide whether --upgrade-bin is worth it without diffing 29k of shell. Co-Authored-By: Claude Opus 5 (1M context) --- bin/sync-project.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/bin/sync-project.sh b/bin/sync-project.sh index 3036d3d..41ae095 100755 --- a/bin/sync-project.sh +++ b/bin/sync-project.sh @@ -643,6 +643,8 @@ known_fix_note() { case "$1" in verify-module.sh) echo "bin/verify-module.sh is missing the design-audit wiring fix (toolkit, 2026-08-21) — wires tests/e2e/design-audit.js (rungs 6-7, UI/a11y) into the composed pass as an informational, non-gating rung. Recommended upgrade." ;; + exec.sh) + echo "bin/exec.sh predates two fixes worth naming. (1) The WINDOWS MXBUILD GATE (toolkit, 2026-08-25): find_sp_app/find_mxbuild and JAVA_HOME resolution were macOS-only, so under Git Bash the gate block was skipped entirely and every exec on a Windows machine went UNVERIFIED — it reported 'skipped', not a false pass, but a skip nobody acts on is the same outcome. (2) The MODULE-BRIEF GUARD (toolkit, 2026-08-25): refuses a write to a module with no module-brief.md (or no '## Module brief — ' section in the build plan), overridable with FORCE_EXEC=1. Strongly recommended upgrade on Windows — without (1) nothing checks your builds." ;; esac } From 70e298a1209c547c460790dca504e4eb0a803626 Mon Sep 17 00:00:00 2001 From: MendixMau Date: Wed, 26 Aug 2026 06:57:28 +0800 Subject: [PATCH 09/11] Make the module brief a build-plan row; exec.sh advises, never refuses The brief was enforced by a hard block in exec.sh guard 5, which blocked a-la-carte use: a project with no architecture/ at all was refused a write by a pipeline artifact it never opted into. Guard 5 now warns and continues, has no FORCE_EXEC path, and is silent entirely in projects with no architecture/build-plan.md. The ordering guarantee moves to the document that actually steers a build: every phase opens with a numbered BRIEF row (a fifth row kind), phrased check-then- create -- does the brief exist and cover this phase's rows, create if absent, extend if thin. One brief per module, grown across its phases rather than rewritten, so a short phase adds a short increment. A rule in a skill file is followed by whoever loaded the skill; a numbered row with a State cell is followed by whoever works the plan. Co-Authored-By: Claude Opus 5 (1M context) --- project-bin/exec.sh | 49 ++++++++++++++++------------- skills/brd-to-build-plan.md | 57 ++++++++++++++++++++++++++++------ skills/iterative-build-loop.md | 2 +- skills/module-brief.md | 28 +++++++++++------ 4 files changed, 95 insertions(+), 41 deletions(-) diff --git a/project-bin/exec.sh b/project-bin/exec.sh index 9017f3d..f42d9e4 100755 --- a/project-bin/exec.sh +++ b/project-bin/exec.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # exec.sh — the guard chain around a model write. # -# concurrent-writer guard → module-brief guard → mxcli check → snapshot → baseline → exec +# concurrent-writer guard → module-brief advisory → mxcli check → snapshot → baseline → exec # → mxbuild gate → auto-restore on regression → SP reopen # # Usage: ./bin/exec.sh @@ -79,28 +79,36 @@ if [ -n "$MPR_DIRTY" ]; then echo " (FORCE_EXEC set — proceeding despite uncommitted changes)" fi -# 5. No module brief for a module this script writes to. +# 5. Module brief advisory (WARNS, never blocks). # -# WHY. The brief is the mdl-agent's single per-module input: the access-table slice, +# WHY THE BRIEF. It is the mdl-agent's single per-module input: the access-table slice, # screens-per-role, field-level validation rules, edge cases, and the wireframe->page map. # Nothing else in the project carries them. Without it the agent synthesises them from # training data, and every access-rights / wrong-binding / invented-validation incident -# traces back to that. It was previously enforced only by check_build_ready() in -# gate-check.sh — a separate command nobody is obliged to run — so a project that never -# declared itself build-ready was never asked for one. Real incident (a customer training round, -# 2026-08-25): 12 domain scripts and 13 feature scripts executed against a module whose -# architecture/modules// held only definition.md. Binding it to the write is the point; -# there is nothing to rewrite when it fires, unlike a missing skill, so firing late still -# costs only the brief. +# traces back to that. Real incident (a customer training round, 2026-08-25): 12 domain +# scripts and 13 feature scripts executed against a module whose architecture/modules// +# held only definition.md. +# +# WHY THIS ONLY WARNS. The brief is a build-plan artifact — "write the module brief" is a +# numbered row at the head of each phase (brd-to-build-plan.md Step 5). That is where the +# ordering guarantee belongs, because that is the document every build follows. exec.sh is +# not the pipeline's gatekeeper and must not become one: the toolkit is deliberately usable +# a la carte, and refusing a write is the wrong answer to "this project chose not to run the +# pipeline". A hard block here would also have needed FORCE_EXEC on every legitimate +# non-pipeline write, and a guard routinely overridden teaches people to override guards. +# +# So: no architecture/build-plan.md means the project never opted in, and this block says +# nothing at all. With a build plan, a missing brief means a row was skipped — worth saying +# once, out loud, and then getting out of the way. # # Satisfied by EITHER form, because single-module projects merge the brief into the plan # rather than maintaining two documents that overlap ~70%: # a) architecture/modules//module-brief.md -# b) a "## Module brief — " heading in architecture/build-plan.md +# b) a "## Module brief - " heading in architecture/build-plan.md # -# Platform and marketplace modules are skipped — this project does not author their briefs. +# Platform and marketplace modules are skipped - this project does not author their briefs. brief_missing="" -if [ -f "$SCRIPT" ]; then +if [ -f "$SCRIPT" ] && [ -f "$PROJECT_ROOT/architecture/build-plan.md" ]; then # Comments FIRST, before any pattern match. A `-- ... CREATE MODULE errors if ...` line in a # real workshop script otherwise yielded a module named "errors" and would have demanded a # brief for it forever. A guard that cries wolf gets switched off — same reasoning as @@ -151,19 +159,18 @@ if [ -f "$SCRIPT" ]; then done fi if [ -n "$brief_missing" ]; then - echo "✗ No module brief for:$brief_missing — refusing to write a module nothing has specified." + echo "⚠ No module brief for:$brief_missing — proceeding, but nothing has specified this module." echo "" for m in $brief_missing; do echo " $m — expected architecture/modules/$m/module-brief.md" - echo " or a '## Module brief — $m' section in architecture/build-plan.md" + echo " or a '## Module brief - $m' section in architecture/build-plan.md" done echo "" - echo " The brief is row 0 of this module's phase. It carries the access table, screens-per-role," - echo " validation rules, edge cases, wireframe->page map and test plan — see module-brief.md." - echo " → Draft it (ba-agent translation mode, pulling architect-agent), then sign it off in chat." - echo " Override (proceeds with nothing having specified this module): FORCE_EXEC=1 ./bin/exec.sh $SCRIPT" - [ "$FORCE" = "1" ] || exit 1 - echo " (FORCE_EXEC set — proceeding with no module brief for:$brief_missing)" + echo " This project has a build plan, so a row was skipped: the brief is the first row of" + echo " this module's phase. It carries the access table, screens-per-role, validation rules," + echo " edge cases, wireframe->page map and test plan — see module-brief.md." + echo " → Write it before the next script, or the MDL after this one is guesswork too." + echo "" fi echo $$ > "$LOCK" diff --git a/skills/brd-to-build-plan.md b/skills/brd-to-build-plan.md index 03d6ac1..ffbf28c 100644 --- a/skills/brd-to-build-plan.md +++ b/skills/brd-to-build-plan.md @@ -245,21 +245,23 @@ See `design-artifacts.md` and `learned-stylegallery.md` for full process. ## Step 5: Produce the Numbered Script Sequence -### Row schema — four kinds, one number sequence +### Row schema — five kinds, one number sequence -Every row is one of four kinds, sharing one numbering and one dependency graph. **A row may say +Every row is one of five kinds, sharing one numbering and one dependency graph. **A row may say `not built`; it may never be absent.** No caption stands in for several items. | Col | Meaning | |---|---| | # | step number, dependency-ordered, never reused | -| Kind | `BUILD` / `PROVE` / `RUN` / `HARNESS` | +| Kind | `BRIEF` / `BUILD` / `PROVE` / `RUN` / `HARNESS` | | Step | the `.mdl` script, the command, or the harness to invoke | | Produces / Proves | the model artifact, or what a pass demonstrates | | Depends on | prior row number(s) | | **Skills** | required reading before authoring this row, from `bin/lib/skill-routing.tsv`. **`none` is a valid value but must be written** — blank is not. The module brief's "Build skills to read first" is the union of this column across the module's rows. | | State | `built` / `partial` / `not built` / `blocked` | +- **`BRIEF`** — writes no model. *Check the brief for this phase's module exists and covers this + phase's rows; create it if absent, extend it if thin.* Exactly one per phase, always first. - **`BUILD`** — writes model. One entity, one microflow group (≤6), or one page section. - **`PROVE`** — headless and mechanical, mid-phase. `mxcli test`, a `curl`, a `DESCRIBE` read-back. - **`RUN`** — deploy, reopen Studio Pro, walk the happy path as a demo user. @@ -280,15 +282,52 @@ so no one did, and a day's work went unverified on machines where the mxbuild ga silently skipping. Give verification a row and it can be numbered, depended on, counted, and rendered in `build-plan.html` like anything else. -### Row 0 of every phase is that module's brief +### The first row of every phase is a `BRIEF` row -The brief (`module-brief.md`) is the first row of its module's phase — not stockpiled upfront, not -produced after the previous module's gate. That placement makes it early enough to be an input and -late enough not to be speculative, and it gives it a state that `build-plan-status.sh` can show and -`project-bin/exec.sh`'s module-brief guard can enforce. +Every phase opens with a numbered `BRIEF` row, and it is written as a **check-then-create**, not as +an unconditional authoring task: + +> *Check `architecture/modules//module-brief.md` exists and covers this phase's rows. +> Create it if absent; extend it if it does not.* + +Phase 1 row 1, phase 2 row 1, phase 3 row 1 — every phase, including phases of a module already +briefed. Nothing else in the phase may depend on anything earlier than it. + +**Make it a row, not a rule.** A rule stated in a skill file is followed by whoever loaded the skill; +a numbered row is followed by whoever works the plan, which is everyone, on every build, in every +entry mode — *the plan is what the agent is being steered by.* The row also carries a `State` cell, +so `build-plan-status.sh` shows it unwritten and the phase cannot report complete around it. + +**And it is a step, not a gate.** The distinction matters: a gate is something you fail, and the +project stops; a step is something you do, and the project continues. `project-bin/exec.sh` warns +when a script writes to a module with no brief but never refuses (see its guard 5 comment), because +the toolkit is usable à la carte and a project that never opted into the pipeline should not have +its writes blocked by a pipeline artifact. Ordering belongs to the document that orders things. + +### How big is a brief for a short phase? + +**A brief is per module and grows across that module's phases — it is not rewritten per phase.** The +first `BRIEF` row that hits a module creates it; every later one appends the sections its own rows +need. So the size follows the phase, and a two-row phase legitimately adds two or three lines. + +What a `BRIEF` row must add is fixed by *what this phase's rows touch*, not by a word count: + +| This phase's rows include… | The brief must gain… | +|---|---| +| any element with access rules | those elements' rows in the access table | +| a page or snippet | its wireframe → page mapping, and the screens-per-role entry | +| an entity or attribute | its field-level validation rules | +| an integration call | its error paths — 401 / 404 / 500 / empty | +| anything at all | the union of that phase's `Skills` column, into "Build skills to read first" | + +If a phase's rows touch none of those, the row is satisfied by confirming the existing brief already +covers them, and its `State` becomes `built` with a one-line note. **That is a real outcome, not a +skipped step** — the value is in having looked. What is *not* acceptable is a brief that is one +sentence because writing it felt like overhead: if the phase builds pages with no wireframe mapping +and no access rows, the phase is underspecified and the brief is telling you so. **Single-module projects: merge it.** The brief's sections become sections of the build plan under -a `## Module brief — ` heading (the form the exec guard also accepts). A project-wide +a `## Module brief — ` heading (the form `exec.sh`'s advisory also recognises). A project-wide ordering over one module is that module's ordering; two documents at ~70% overlap is how one of them ends up unwritten — which is exactly what happened to the workshop project above. diff --git a/skills/iterative-build-loop.md b/skills/iterative-build-loop.md index 16d1ad4..c911b9c 100644 --- a/skills/iterative-build-loop.md +++ b/skills/iterative-build-loop.md @@ -133,7 +133,7 @@ still had the old (wrong) description and two already-resolved open questions st Run this before scripting each module: -- [ ] **Module brief exists and passes its ready-check.** `architecture/modules//module-brief.md` must exist (authored by `ba-agent` translation mode, per `module-brief.md`) with every ready-check box ticked: every screen has a wireframe, the access table covers every element, no open business question blocks this phase, write mode chosen for every STOP-row element. **No brief, or an unchecked ready-check item touching this phase → STOP.** Produce/complete the brief first — do not let the `mdl-agent` synthesize the module from raw BRDs. This is the just-in-time gate. `gate-check.sh` cannot require all briefs at Stage 4 (they don't all exist yet), but it is no longer manual-only: **`project-bin/exec.sh` refuses a write to a module with no brief** — guard 5, satisfied by either `architecture/modules//module-brief.md` or a `## Module brief — ` heading in the build plan (the single-module merged form), overridable with `FORCE_EXEC=1`. The brief is row 0 of the module's phase (`brd-to-build-plan.md` Step 5), so in the normal phased flow the guard cannot fire; it is the net for building off-plan. +- [ ] **Module brief exists and passes its ready-check.** `architecture/modules//module-brief.md` must exist (authored by `ba-agent` translation mode, per `module-brief.md`) with every ready-check box ticked: every screen has a wireframe, the access table covers every element, no open business question blocks this phase, write mode chosen for every STOP-row element. **No brief, or an unchecked ready-check item touching this phase → STOP.** Produce/complete the brief first — do not let the `mdl-agent` synthesize the module from raw BRDs. This is the just-in-time gate. `gate-check.sh` cannot require all briefs at Stage 4 (they don't all exist yet), so the ordering guarantee lives where the build actually gets its orders: every phase opens with a **`BRIEF` row** (`brd-to-build-plan.md` Step 5) — a check-then-create carrying its own `State` cell: does the brief exist and cover this phase's rows, and if not, write or extend it. `project-bin/exec.sh` **warns** if a script writes to a module with no brief — satisfied by either `architecture/modules//module-brief.md` or a `## Module brief — ` heading in the build plan (the single-module merged form) — but it does not refuse, and says nothing at all in a project with no build plan. À-la-carte use of this toolkit is a supported choice; the warning is a signal that a planned row was skipped, not a gate. - [ ] Read source screenshots for this module top-to-bottom - [ ] Read the feature doc (F-doc or BRD) for this module - [ ] Extract the build checklist from the feature doc: diff --git a/skills/module-brief.md b/skills/module-brief.md index 8bc9d7d..fe8d7c4 100644 --- a/skills/module-brief.md +++ b/skills/module-brief.md @@ -68,17 +68,25 @@ list into the brief, stop — link it instead and synthesize the *decision* abou ## Location & Timing - **Location:** `architecture/modules//module-brief.md` — one directory per module, holding its brief (and any per-module assets alongside it). -- **Timing (just-in-time):** the brief for module N is produced only after module N−1 has passed - its full build gate. Same rule as MDL phasing (`brd-to-build-plan.md`) — a brief written against - a model state that later changes is a brief that lies. The **first** module's brief is produced - at Stage 4 alongside the build plan; the rest are produced as each module's build begins. -- Do **not** stockpile all briefs upfront. `gate-check.sh` cannot mechanically require all briefs - at Stage 4 for this reason. It is not a manual-only check any more, though: the brief is **row 0 - of its module's phase** (`brd-to-build-plan.md` Step 5), and `project-bin/exec.sh` guard 5 - refuses any write to a module that has no brief. Row 0 placement is what makes the guard - unreachable in the normal flow — early enough to be an input, late enough not to be speculative. +- **Timing — it is a row in the build plan.** Every phase opens with a `BRIEF` row + (`brd-to-build-plan.md` Step 5), written as a **check-then-create**: *does the brief for this + phase's module exist and cover this phase's rows? Create it if absent, extend it if thin.* Not + stockpiled upfront, not produced after the previous module's gate — a brief written against a + model state that later changes is a brief that lies, and a brief written after the build is a + transcript rather than an input. +- **It grows; it is not rewritten per phase.** One brief per module, extended by each phase's + `BRIEF` row with the sections that phase's rows require — access rows for elements with access + rules, wireframe→page mapping for pages, validation rules for attributes, error paths for + integration calls, and always the union of that phase's `Skills` column. A short phase therefore + adds a short increment; see `brd-to-build-plan.md` → "How big is a brief for a short phase?". +- **A step, not a gate.** Do not stockpile all briefs upfront — `gate-check.sh` cannot mechanically + require them all at Stage 4 for that reason. The ordering guarantee lives in the build plan, where + it is a numbered row with a `State` cell rather than a rule someone has to have read. + `project-bin/exec.sh` **warns** — never refuses — when a script writes to a module with no brief, + and only in projects that have a build plan at all; à-la-carte use is a supported choice and must + not be blocked by a pipeline artifact. - **Single-module projects: merge the brief into the build plan** under a `## Module brief — ` - heading (the exec guard accepts that form). Two documents at ~70% overlap is how one goes + heading (`exec.sh`'s advisory recognises that form). Two documents at ~70% overlap is how one goes unwritten. --- From 18cf86707e5ab5cca5b20bf27d838460ddccb716 Mon Sep 17 00:00:00 2001 From: MendixMau Date: Wed, 26 Aug 2026 06:57:45 +0800 Subject: [PATCH 10/11] Protocol notice in plain language; ack without needing a terminal The freshness notice was written for someone who already knew what a "protocol commit" was: two abbreviated SHAs, the word "ack", and skill file paths. Run in front of TAMs in an enablement session, it reads as something being broken, and the only reachable answer was a shell command. Output is now plain-language by default with commit ids, paths and diffstat behind --verbose. The TTY refusal is replaced, not removed: an ack still asserts a human was told, but "a human" is no longer equated with "a human at a terminal". --ack-protocol --verbose without a TTY is a read-only preview that records nothing and instructs the agent to summarise and ask; --ack-protocol --approved-in-chat records the answer. The log line carries which route was used. There is still no env-var auto-yes. test-bug06-freshness.sh helpers pass --verbose so every existing technical assertion holds verbatim, plus a new control for the plain default. Co-Authored-By: Claude Opus 5 (1M context) --- bin/gate-check.sh | 140 +++++++++++++++++++++------- bin/init-project.sh | 22 +++-- tests/wave2/test-bug06-freshness.sh | 29 +++++- 3 files changed, 150 insertions(+), 41 deletions(-) diff --git a/bin/gate-check.sh b/bin/gate-check.sh index cfe2168..ba0c333 100755 --- a/bin/gate-check.sh +++ b/bin/gate-check.sh @@ -6,7 +6,7 @@ # actual project state. A stage-specific run answers one question and writes nothing: that is the # invocation used in agent loops and hooks, and a read-only query must leave no trace. # -# Usage: bin/gate-check.sh [--html|--no-html] [--ack-protocol|--force-stale] +# Usage: bin/gate-check.sh [--html|--no-html] [--ack-protocol [--approved-in-chat]|--force-stale] [--verbose] # [--adopt --reason "..."] # [--waive --reason "..."] # [stage] @@ -46,6 +46,8 @@ set -uo pipefail # The obvious `set -- $POSITIONAL` form word-splits the path and was rejected for that reason. HTML_MODE="auto" ACK_PROTOCOL=0 +APPROVED_IN_CHAT=0 +PROTOCOL_VERBOSE=0 FORCE_STALE="${MXTK_ACK_STALE:-0}" STRICT_PROTOCOL="${MXTK_STRICT_PROTOCOL:-0}" ADOPT_STAGE="" @@ -58,6 +60,8 @@ while [ $# -gt 0 ]; do --html) HTML_MODE="always" ;; --no-html) HTML_MODE="never" ;; --ack-protocol) ACK_PROTOCOL=1 ;; + --approved-in-chat) APPROVED_IN_CHAT=1 ;; + --verbose) PROTOCOL_VERBOSE=1 ;; --adopt) shift; ADOPT_STAGE="${1:-}" ;; --waive) shift; WAIVE_STAGE="${1:-}" ;; --reason) shift; WAIVER_REASON="${1:-}" ;; @@ -1556,35 +1560,69 @@ fi SYNC_BLOCKING=0 [ "$STRICT_PROTOCOL" = "1" ] && [ "$SYNC_STATUS" = "NOTICE" ] && SYNC_BLOCKING=1 -printf "Sync (Protocol freshness): %s — %s\n" "$SYNC_STATUS" "$SYNC_NOTE" +# The one-line verdict is jargon by construction — it names commits and paths so a maintainer +# can act on it without a second command. Non-verbose readers get the same verdict in words. +if [ "$PROTOCOL_VERBOSE" = "1" ]; then + printf "Sync (Protocol freshness): %s — %s\n" "$SYNC_STATUS" "$SYNC_NOTE" +else + case "$SYNC_STATUS" in + PASS) printf "Toolkit updates: up to date.\n" ;; + WARN) printf "Toolkit updates: the shared toolkit moved, but nothing this stage uses changed. No action needed.\n" ;; + NOTICE) printf "Toolkit updates: available, not yet reviewed. Nothing is blocked.\n" ;; + *) printf "Toolkit updates: %s\n" "$SYNC_STATUS" ;; + esac +fi if [ "$SYNC_STATUS" = "NOTICE" ]; then # Lead with the CHOICE, not with the diagnosis. The old message named files, showed no diff, # offered no command, and did not say where the value it wanted was written — so the only # available response was to go and find the four-step ritual in another file. echo "" - echo " ⚠ $SYNC_HEADLINE" - [ -n "$SYNC_DETAIL" ] && printf ' %s\n' "$SYNC_DETAIL" - # An UPGRADE IS AN OFFER, NOT A DEMAND (2026-08-20). When the toolkit grows a new artifact or - # a new stage requirement, the projects that see it first are the ones already mid-build — and - # for them "the protocol changed" often means nothing needs to happen: they did that analysis - # their own way, or they are past the stage entirely. Saying only "the toolkit moved" leaves - # them to guess whether they are now behind, and the safe-looking guess (regenerate it) is the - # expensive one. So name the stages it touches and put the "not needed here" answer on the - # same screen as the "yes please" one, with equal billing. + # LAYERED, and plain-language by default (2026-08-26). The old notice was written for someone + # who already knew what a "protocol commit" was: it led with two abbreviated SHAs, said "ack", + # and named skill file paths. A TAM running this in an enablement session reads that and can + # only conclude something is broken. Nothing about the SHAs helps them decide; the decision is + # "has the shared toolkit changed in a way that affects what I am about to build?" — so ask + # exactly that, in those words, and put the identifiers behind --verbose for the people who + # actually diff things. + if [ "$PROTOCOL_VERBOSE" = "1" ]; then + echo " ⚠ $SYNC_HEADLINE" + [ -n "$SYNC_DETAIL" ] && printf ' %s\n' "$SYNC_DETAIL" + else + echo " ⚠ The shared toolkit has been updated since this project last checked." + if [ -n "${SYNC_RELEVANT:-}" ]; then + SYNC_N="$(printf '%s' "$SYNC_RELEVANT" | wc -w | tr -d ' ')" + echo " $SYNC_N guidance file(s) that this stage relies on changed. Nothing is broken and" + echo " nothing is blocked — this is asking whether you want to take the update." + fi + fi if [ -n "${SYNC_RELEVANT:-}" ]; then # shellcheck disable=SC2086 — SYNC_RELEVANT is a deliberate list. SYNC_STAGES="$(stages_for_protocol_files $SYNC_RELEVANT)" [ -n "$SYNC_STAGES" ] && echo " Stages this touches: $SYNC_STAGES" + if [ "$PROTOCOL_VERBOSE" != "1" ]; then + echo " For the file list and commit ids, re-run with --verbose." + fi + fi + echo "" + if [ "$PROTOCOL_VERBOSE" = "1" ]; then + echo " A) $SYNC_OPT_A" + else + echo " If you are an agent: run \"$0 $PROJECT_DIR --ack-protocol --verbose\" to see what" + echo " changed, summarise it for the user in plain language, ask whether to take the" + echo " update, and only if they say yes re-run with --ack-protocol --approved-in-chat." + echo " Do not decide this on the user's behalf and do not ask them to type anything." + echo "" + echo " A) Take the update — the agent records it for you once you say yes." fi - echo " A) $SYNC_OPT_A" if [ "$SYNC_BLOCKING" = "1" ]; then echo " B) Proceed anyway: $0 --force-stale $PROJECT_DIR ${REQUESTED_STAGE:-}" echo " (--strict-protocol is on, so this run WILL stop; the bypass is logged.)" else - echo " B) Ignore for now — nothing is blocked. This notice repeats until acked." - echo " C) Not needed here — if the change asks for artifacts this project already covered" - echo " its own way, or is past: $0 --waive --reason \"...\" $PROJECT_DIR" - echo " Acking (A) never obliges you to produce anything; it records that you read it." + echo " B) Decide later — nothing is blocked. This notice repeats until you answer." + echo " C) Not needed here — if the update asks for work this project already did its own" + echo " way, or is past: $0 --waive --reason \"...\" $PROJECT_DIR" + echo " Taking the update (A) never obliges you to produce anything; it records that" + echo " you were told." fi echo "" fi @@ -1718,24 +1756,56 @@ if [ "$ACK_PROTOCOL" = "1" ]; then echo "Cannot acknowledge: the toolkit clone at $TOOLKIT_DIR has no resolvable commit." >&2 exit 1 fi - # Non-interactive: REFUSE, and name the path forward at the moment of refusal. + # Non-interactive: the ack still needs a HUMAN, but it no longer needs a TERMINAL. # - # An ack asserts that a HUMAN READ THE DIFF. Prompting for that with no human present is - # theatre, and an unattended auto-yes (the proposal's MXTK_ACK_YES=1) is worse than theatre: - # an agent would write PROTOCOL-ACK — a record asserting a read that never happened — into the - # one log this design relies on to tell reads from rubber-stamps, poisoning that signal - # permanently and silently. So an agent session takes the explicit, logged bypass instead: - # PROTOCOL-BYPASS is honest about what actually occurred and is countable. There is no - # MXTK_ACK_YES, and the refusal below always names --force-stale and says it is logged. - if [ ! -t 0 ]; then - echo "Refusing to acknowledge protocol non-interactively (no TTY on stdin)." >&2 - echo "An ack asserts a human read the diff; nothing here can make that true." >&2 - echo " A) A human runs: $0 $PROJECT_DIR --ack-protocol" >&2 - echo " B) Proceed anyway: $0 --force-stale $PROJECT_DIR ${REQUESTED_STAGE:-}" >&2 + # The original rule refused any ack without a TTY, because an ack asserts that a human read + # the diff and an unattended auto-yes would write a record of a read that never happened — + # poisoning the one signal that tells reads from rubber-stamps. That reasoning is intact and + # this code still enforces it. What it got wrong was equating "a human" with "a human at a + # terminal". The people this now runs in front of — TAMs in an enablement session, consultants + # in a workshop — are humans who will never open a shell, and refusing them left the notice + # repeating forever with no reachable answer, which is its own kind of rubber-stamp. + # + # So there are two honest ways to ack, and the log records WHICH: + # at a terminal — the human ran this, saw the diffstat, optionally paged the full diff + # approved in chat — the agent ran --ack-protocol --verbose, summarised what changed in + # plain language, ASKED, and the human said yes + # Both are real reads. Neither is an agent deciding alone: --approved-in-chat is a claim the + # agent makes on the record, in a countable log line, and an agent that sets it without having + # asked has falsified an audit trail rather than skipped a step. That is the same trust model + # every other write in this toolkit already runs on. + # + # There is still no env-var auto-yes, and --approved-in-chat is deliberately not implied by + # anything: it must be typed, once, per ack, after the question was actually put. + if [ ! -t 0 ] && [ "$APPROVED_IN_CHAT" != "1" ]; then + if [ "$PROTOCOL_VERBOSE" = "1" ] && [ -n "${RECORDED:-}" ] \ + && git -C "$TOOLKIT_DIR" cat-file -e "${RECORDED}^{commit}" 2>/dev/null; then + # The agent asked to SEE it. Show it, record nothing, and say what to do next. + echo "What changed in the shared toolkit since this project last checked:" + echo "" + # shellcheck disable=SC2086 — PROTOCOL_PATHS is a deliberate multi-pathspec list. + git -C "$TOOLKIT_DIR" --no-pager diff --stat "$RECORDED" "$TOOLKIT_REF" -- $PROTOCOL_PATHS + echo "" + # shellcheck disable=SC2086 + git -C "$TOOLKIT_DIR" --no-pager log --oneline "$RECORDED..$TOOLKIT_REF" -- $PROTOCOL_PATHS + echo "" + echo "NOTHING HAS BEEN RECORDED. This was a read-only preview." + echo "Agent: summarise the above for the user in plain language — what changed and what it" + echo "means for what they are building — then ask whether to take the update. Only if they" + echo "say yes, re-run: $0 $PROJECT_DIR --ack-protocol --approved-in-chat" + exit 0 + fi + echo "Refusing to record this without a human having been asked." >&2 + echo "An ack asserts a person was told what changed; nothing here can make that true." >&2 + echo " A) Agent: preview it, ask the user, then record their answer:" >&2 + echo " $0 $PROJECT_DIR --ack-protocol --verbose (shows it, records nothing)" >&2 + echo " $0 $PROJECT_DIR --ack-protocol --approved-in-chat (after they say yes)" >&2 + echo " B) Human at a terminal: $0 $PROJECT_DIR --ack-protocol" >&2 + echo " C) Proceed anyway: $0 --force-stale $PROJECT_DIR ${REQUESTED_STAGE:-}" >&2 echo " — that works, and records a PROTOCOL-BYPASS line in docs/BUILD-LOG.md." >&2 if [ "$STRICT_PROTOCOL" != "1" ]; then echo " Note: without --strict-protocol nothing is blocked anyway — this run can" >&2 - echo " simply proceed, and the notice repeats until a human acks it." >&2 + echo " simply proceed, and the notice repeats until someone answers it." >&2 fi exit 1 fi @@ -1749,8 +1819,10 @@ if [ "$ACK_PROTOCOL" = "1" ]; then # shellcheck disable=SC2086 — PROTOCOL_PATHS is a deliberate multi-pathspec list. git -C "$TOOLKIT_DIR" --no-pager diff --stat "$RECORDED" "$TOOLKIT_REF" -- $PROTOCOL_PATHS echo "" + if [ "$APPROVED_IN_CHAT" = "1" ]; then reply=y; else printf "Print the full diff before acknowledging? [d=diff / y=ack / n=abort] " read -r reply + fi case "$reply" in d|D) # shellcheck disable=SC2086 git -C "$TOOLKIT_DIR" diff "$RECORDED" "$TOOLKIT_REF" -- $PROTOCOL_PATHS @@ -1766,16 +1838,20 @@ if [ "$ACK_PROTOCOL" = "1" ]; then # No usable prior ack (a fresh scaffold, or a sha this clone has never seen). There is no # diff to show, so there is nothing to pretend was read — stamp it and say so in the log. echo "$REGISTER records no toolkit commit this clone can diff from." + if [ "$APPROVED_IN_CHAT" = "1" ]; then reply=y; else printf "Record the current protocol commit %s as this project's baseline? [y/N] " "$TOOLKIT_REF" read -r reply + fi case "$reply" in y|Y) ;; *) echo "Aborted — Toolkit commit line unchanged."; exit 1 ;; esac ACK_STAT="no prior ack to diff from — recorded as a baseline" ACK_FILES="(baseline)" ACK_FROM="${RECORDED:-none}" fi if register_stamp_commit "$TOOLKIT_REF"; then - build_log_append "PROTOCOL-ACK $ACK_FROM -> $TOOLKIT_REF ($ACK_STAT) files: $ACK_FILES" - echo "Acknowledged $TOOLKIT_REF — $REGISTER updated, recorded in $BUILD_LOG." + ACK_HOW="read at a terminal" + [ "$APPROVED_IN_CHAT" = "1" ] && ACK_HOW="approved in chat after the agent summarised it" + build_log_append "PROTOCOL-ACK $ACK_FROM -> $TOOLKIT_REF [$ACK_HOW] ($ACK_STAT) files: $ACK_FILES" + echo "Update taken — $REGISTER now records $TOOLKIT_REF, logged in $BUILD_LOG ($ACK_HOW)." exit 0 fi echo "Could not find a 'Toolkit commit:' line in $REGISTER — add one reading:" >&2 diff --git a/bin/init-project.sh b/bin/init-project.sh index 3277ec0..e42946d 100755 --- a/bin/init-project.sh +++ b/bin/init-project.sh @@ -285,15 +285,23 @@ session that will touch the pipeline: 1. \`git -C $TOOLKIT_ROOT pull --ff-only\` 2. \`$TOOLKIT_ROOT/bin/gate-check.sh \` — it reports protocol freshness and tells you whether anything you depend on moved. -3. If it says so: re-read the named files, then - \`$TOOLKIT_ROOT/bin/gate-check.sh --ack-protocol\`. That one command shows the - diffstat, offers the full diff, rewrites the \`Toolkit commit:\` line in \`PROJECT.md\`, and - records which files and how many lines you accepted in \`docs/BUILD-LOG.md\`. +3. If it says an update is available, **you (the agent) handle it — never ask the user to type + a command.** Run \`$TOOLKIT_ROOT/bin/gate-check.sh --ack-protocol --verbose\`, + which shows what changed and **records nothing**. Re-read the named files. Then tell the user + in plain language what changed and what it means for what they are building, and ask whether + to take the update. Only if they say yes: + \`$TOOLKIT_ROOT/bin/gate-check.sh --ack-protocol --approved-in-chat\` — that + rewrites the \`Toolkit commit:\` line in \`PROJECT.md\` and records which files, how many + lines, and that it was approved in chat, in \`docs/BUILD-LOG.md\`. 4. State in chat which commit you're working from. -Steps 1-3 used to be a four-step manual ritual; \`--ack-protocol\` replaces the middle of it. -It is interactive on purpose — an ack asserts a human read the diff, so it refuses when there -is no TTY and points an unattended caller at \`--force-stale\`, which works and is logged. +An ack asserts a human was told what changed, so it cannot happen unattended: without a TTY and +without \`--approved-in-chat\` it refuses and names the routes out. \`--approved-in-chat\` is a +claim you are making on the record — set it only after you actually asked and they actually +answered. There is no env-var auto-yes. + +By default the output is written for whoever is in the room, not for a maintainer: plain +language, no commit ids, no file paths. Add \`--verbose\` for the ids, paths and diffstat. **Protocol staleness NEVER blocks a gate.** It prints a notice with lettered options and the stage verdict is reported on its own merits either way — a project is not broken because diff --git a/tests/wave2/test-bug06-freshness.sh b/tests/wave2/test-bug06-freshness.sh index f1852e3..f546601 100755 --- a/tests/wave2/test-bug06-freshness.sh +++ b/tests/wave2/test-bug06-freshness.sh @@ -78,13 +78,18 @@ push_change() { # push_change git -C "$TK" push -q origin HEAD:refs/heads/main } -run() { "$GC" --no-html "$PROJ" ${1:+$1} 2>&1; } +# --verbose on the helpers (2026-08-26). The default rendering became plain-language by decision: +# gate-check now leads with "Toolkit updates: ..." and words, and puts commit ids, file paths and +# diffstat behind --verbose, because the people running this in enablement sessions are not the +# people who diff skill files. Every technical assertion below is about that verbose surface and +# still holds verbatim against it. The plain default gets its own control at the end of the file. +run() { "$GC" --no-html --verbose "$PROJ" ${1:+$1} 2>&1; } # `run X | grep -q ...` is a TRAP under `set -o pipefail`: grep -q exits on the first match and # SIGPIPEs gate-check, whose 141 then becomes the pipeline's status — so a matching pattern # reports NO MATCH. It cost three false "CONTROL BROKEN" results while writing this file. Every # content assertion below therefore goes through this helper, which uses a herestring, not a pipe. saw() { grep -q "$2" <<<"$1"; } -rc() { "$GC" --no-html "$PROJ" ${1:+$1} >/dev/null 2>&1; echo "$?"; } +rc() { "$GC" --no-html --verbose "$PROJ" ${1:+$1} >/dev/null 2>&1; echo "$?"; } sync_status() { run "${1:-}" | grep '^Sync' | sed 's/^Sync (Protocol freshness): \([A-Z]*\).*/\1/'; } # A notice is only useful if a human sees the choice. Every control checks all three parts. notice_ok() { # notice_ok