Skip to content

fix: meaningful diagnostics, S-first layering, worker retry, representability surfacing - #74

Merged
ryandmonk merged 3 commits into
mainfrom
fix/diagnostics-and-providers
Aug 10, 2026
Merged

fix: meaningful diagnostics, S-first layering, worker retry, representability surfacing#74
ryandmonk merged 3 commits into
mainfrom
fix/diagnostics-and-providers

Conversation

@ryandmonk

Copy link
Copy Markdown
Contributor

Phase 1 presentation + robustness from the ratified P1+P2 milestone. Feature-detected against dspack-emit 0.7 / dspack-gen 0.4 — no dependency bumps in this PR (bump follows once emit 0.7.0 is on npm).

  • Per-instance findings: shared catalogGateFindings() (composer-core) consumes emit 0.7's errorDetails when present — one finding per component instance with an honest Component#id target (the hardcoded a2ui@0.9.1 target is gone), message capped at 3 errors (+N more), full raw strings preserved on finding.evidence. Fallback for emit 0.6 caps the joined message and keeps everything on evidence. Agent twin (project.ts, record-fixture.ts) uses the same helper.
  • Build failures: reasons deduped across A2UI versions ("both A2UI versions" target), S-layer findings lead with A-gate cascades collapsed to one summary line, failed-lint-exhausted turns whose last attempt carries representability now surface the emitter refusal instead of "unknown".
  • Checks view: errors → warnings → infos; the fidelity wall (193 rows on the shipped shadcn example) collapses to one disclosure line; findings with evidence get a raw-evidence expander.
  • Worker: one bounded retry on unusable provider output (the measured-transient 502 class — 4/4 recovered on retry in the field); 429/busy/timeout/kill-switch never retried; fixed a leaked 45s race timer found in passing.
  • Fail-first: 7 composer-core + 4 worker failures on pre-change code pasted in the test commit body. composer-core 68, composer 29, agent 44, typecheck green; no e2e edits (testids verified against the suites).

🤖 Generated with Claude Code

ryandmonk and others added 3 commits August 10, 2026 17:51
…ayered build failures, representability, worker retry

New assertions for the ratified diagnostics + robustness slice, written and
run BEFORE any implementation. Failing output on the pre-change tree:

pnpm --filter @dspack-studio/composer-core test:

 ❯ src/diagnostics.test.ts (7 tests | 7 failed) 8ms
   × catalogGateFindings … a failing gate with errorDetails yields ONE finding per instance …
     → catalogGateFindings is not a function
   × catalogGateFindings … without errorDetails the single finding keeps the caller's target, caps the joined message …
     → catalogGateFindings is not a function
   × catalogGateFindings … three or fewer errors are never marked truncated …
     → catalogGateFindings is not a function
   × buildFailure … identical catalog-gate errors under both A2UI versions render ONCE …
     → expected [ { gate: 'A3', …(3) }, …(3) ] to have a length of 2 but got 4
   × buildFailure … identical-message repetition within one version is capped to one row …
     → expected [ { gate: 'A3', …(3) }, …(3) ] to have a length of 2 but got 4
   × buildFailure … when S-gate failures explain the turn, A-gate reasons collapse to one summary …
     → expected [ { gate: 'A3', …(3) }, …(7) ] to have a length of 2 but got 8
   × buildFailure … a failed-lint-exhausted whose last attempt carries a representability refusal …
     → expected undefined to deeply equal { pass: false, …(1) }
 Test Files  1 failed | 2 passed (3)
      Tests  7 failed | 61 passed (68)

pnpm --filter composer test:

 ❯ worker/propose.test.mjs (5 tests | 4 failed) 9ms
   × retries ONCE when the provider returns prose instead of JSON, then succeeds
     → expected 502 to be 200 // Object.is equality
   × a truncated fenced payload also gets exactly one retry …
     → expected 502 to be 200 // Object.is equality
   × two unusable outputs return the existing 502 provider-unavailable shape — and there is NO third call
     → expected "spy" to be called 2 times, but got 1 times
   ✓ provider rate-limiting (429 shape) is NEVER retried — one call, classified busy
   × the model-call+parse helper is exported and retries exactly once (unit, injected fake env.AI)
     → callModel is not a function
 Test Files  1 failed | 4 passed (5)
      Tests  4 failed | 25 passed (29)

The one passing new test (429 never retried) deliberately pins EXISTING
behavior that must survive the retry change — the zone limit is not hammered.

Notes on scaffolding: catalogGateFindings is reached via a namespace import
so its absence fails per-test (TypeError) instead of failing the module at
load time and masking the behavioral buildFailure failures; the folded
attempt's representability access is cast (as any) until the field exists.
Both are flipped to direct typed access in the implementation commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… failures, representability, worker bounded retry

Makes the fail-first suite from the previous commit green. Four defects from
the field report, one principle: genuine errors are layered, never dropped.

composer-core findings — catalogGateFindings (shared by browser + agent):
* dspack-emit >= 0.7 errorDetails (feature-detected via Array.isArray) yield
  ONE finding per invalid component instance, target `Component#id` — the
  hardcoded "a2ui@0.9.1" target (and its dead deep-link) is gone from the
  detail path. Message = first 3 error strings + " (+N more)"; the COMPLETE
  raw strings ride the new optional ComposerFinding.evidence field.
* Without errorDetails (0.6): the single finding keeps the caller's version
  target, caps the joined message the same way (the measured 3,278-char
  table cell), and keeps ALL raw strings on evidence.

composer-core build — buildFailure presentation:
* emitGateReasons dedupes A-gate reasons by (gate, code, message): a reason
  under both A2UI versions renders once targeted "both A2UI versions";
  identical-message repetition within one version collapses too (the ~100-row
  panel becomes the distinct facts; the full audit report stays on the turn).
* S-first layering: when the last attempt's S-gate reasons exist, A-gate
  reasons collapse to one summary ("N catalog-gate findings — see Checks for
  detail") — S findings already explain the failure.
* dspack-gen >= 0.4 attempts[].representability (feature-detected) is kept
  by the fold on TurnAttempt and surfaced as a structured reason for
  failed-lint-exhausted turns (stoppedAt "attempt N · representability")
  instead of "unknown … without structured evidence".

apps/composer validation.ts + apps/agent project.ts / record-fixture.ts:
the twin gate-mapping loops now call catalogGateFindings (cross-referenced
comments); record-fixture prints one line per instance the same way.

worker propose.mjs — bounded retry on unusable OUTPUT only:
* callModel = runModel (one provider call; rejection classified, never
  retried — 429/busy/timeout/kill-switch must not hammer the zone limit)
  + parseProposal (fence-strip + JSON.parse; throw = unusable output).
  Exactly one retry on unusable output; second failure rides the existing
  502 provider-unavailable path. Exported for unit tests; the race timer is
  now cleared (no dangling 45s handle).

Also flips the fail-first scaffolding to direct typed imports (namespace
access + `as any` were only there so the pre-change tree could run the file).

pnpm --filter @dspack-studio/composer-core test: 68 passed (68)
pnpm --filter composer test: 29 passed (29)
pnpm --filter agent test: 44 passed (44)
pnpm test (all packages) + pnpm -r typecheck: green

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…es collapsed; evidence expanders; representability labels

Checks (validate-view):
* Layering: error findings first, then warnings, then infos (stable sort, so
  gate order holds within a severity) — the view no longer opens on a
  fidelity-warning wall (measured ~67 rows on a real project; 193 on the
  shipped shadcn example) before anything project-specific.
* Fidelity findings live behind a native <details> disclosure, COLLAPSED by
  default: "N fidelity notes — how this contract projects onto A2UI". They
  are the projection record, not unresolved work; nothing is hidden — one
  click away, rows rendered by the same FindingRow.
* Findings carrying `evidence` (capped catalog-gate messages) get a
  "full gate output (N errors)" expander: monospace, scrollable, complete.
* Every e2e-pinned testid survives verbatim: finding-${gate}-${code},
  acknowledged-${target}, emit-status, validate-status, run-validate,
  run-emit (grepped e2e/ — only composer-agent.spec.ts touches this view).

Build (build-view):
* An attempt whose folded record carries representability { pass: false }
  (dspack-gen >= 0.4, feature-detected upstream in composer-core) shows
  "representability FAIL" beside its gates (refusal as title text) and its
  repair discloses as "repair sent — representability". Repair message
  rendering itself is unchanged — the text already flows.

Verified live against the dev server (both example projects): errors with
their ACKNOWLEDGED chips sort to the top, 28/193 fidelity notes collapse to
one summary line, the disclosure opens on click with finding-fidelity-*
testids intact.

pnpm -r typecheck green; pnpm test + composer (29) + agent (44) all passing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 10, 2026 22:15
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 10, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
dspack-studio-composer a7871e3 Aug 10 2026, 10:17 PM

@ryandmonk
ryandmonk merged commit c2d88e2 into main Aug 10, 2026
2 of 3 checks passed
@ryandmonk
ryandmonk deleted the fix/diagnostics-and-providers branch August 10, 2026 22:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves diagnostics fidelity and presentation across composer-core, the Composer app, and the hosted Worker by introducing structured, per-instance catalog-gate findings (when available), deduplicating/collapsing build-failure reasons, surfacing representability refusals, and adding a bounded retry for model-output parse failures.

Changes:

  • Add catalogGateFindings() to generate per-component-instance A-gate findings with capped messages and full raw evidence, and adopt it in browser + agent emit reporting.
  • Improve build-failure summarization (A-gate dedupe across versions, S-first layering) and surface dspack-gen representability refusals in turn attempts/failures.
  • Update Checks UI to sort by severity, collapse fidelity notes, and add an expander for full raw evidence; add Worker callModel() with a single retry on unusable provider output plus tests.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
packages/composer-core/src/index.ts Re-export catalogGateFindings and related types for shared use.
packages/composer-core/src/findings.ts Add evidence to findings and implement catalogGateFindings() (per-instance + capped message).
packages/composer-core/src/diagnostics.test.ts Add fail-first tests for catalog-gate findings and build-failure layering/dedup/representability surfacing.
packages/composer-core/src/build.ts Deduplicate A-gate reasons, collapse A-gate walls when S-gates explain failure, and surface representability refusal.
apps/composer/worker/propose.test.mjs Add tests for bounded retry behavior on unusable provider output (and no retry on provider failures).
apps/composer/worker/propose.mjs Implement runModel (timer cleanup) + callModel with one retry on parse-only failures; export helper for tests.
apps/composer/app/views/validate-view.tsx Sort findings by severity, collapse fidelity notes, and add evidence expander rendering.
apps/composer/app/views/build-view.tsx Display representability failures and annotate repair details accordingly.
apps/composer/app/validation.ts Use catalogGateFindings() for browser emit gate reporting.
apps/agent/src/record-fixture.ts Use catalogGateFindings() to print per-instance gate diagnostics in fixture recording.
apps/agent/src/project.ts Use catalogGateFindings() for agent emit gate reporting.
Suppressed comments (2)

packages/composer-core/src/findings.ts:155

  • gate.errors is treated as string[], but the input is a loose CatalogGateLike coming from external reports. If errors is unexpectedly not an array, this path will attach a non-array value as evidence and break cappedMessage/UI consumers. Guard with Array.isArray.
    });
  }
  const evidence = gate.errors ?? [];
  return [withEvidence(finding(gateId, code, "error", fallbackTarget, cappedMessage(evidence, code || "gate failed")), evidence)];
}

apps/composer/app/views/validate-view.tsx:52

  • data-testid for the evidence block only uses gate + code, which can collide across multiple findings (especially with per-instance A-gate findings). Include target (or another stable discriminator) so each evidence expander is uniquely addressable.
            </summary>
            <pre
              data-testid={`finding-evidence-${f.gate}-${f.code}`}
              style={{ fontFamily: "var(--mono)", fontSize: 11, whiteSpace: "pre-wrap", maxHeight: 200, overflow: "auto", color: "var(--fg-dim)", margin: "4px 0 0" }}
            >

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +236 to +247
for (const error of (gate.errors ?? ["gate failed"]) as string[]) {
const key = `${gate.gate}${gate.name ?? ""}${error}`;
const entry = merged.get(key);
if (entry) {
if (version) entry.versions.add(version);
continue;
}
merged.set(key, {
reason: { gate: String(gate.gate), code: String(gate.name ?? ""), message: error },
versions: new Set(version ? [version] : []),
});
}
Comment on lines +145 to +150
const target =
typeof detail?.component === "string" && typeof detail?.id === "string"
? `${detail.component}#${detail.id}`
: fallbackTarget;
const evidence = (detail?.errors ?? []).map(detailErrorString);
return withEvidence(finding(gateId, code, "error", target, cappedMessage(evidence, code || "gate failed")), evidence);
Comment on lines +23 to +25
return (
<tr style={{ borderTop: "1px solid var(--line-soft)" }} data-testid={`finding-${f.gate}-${f.code}`}>
<td style={{ padding: "6px 8px", fontFamily: "var(--mono)", fontSize: 12 }}>
Comment on lines +74 to +76
{rows.map((f, i) => (
<FindingRow key={i} f={f} />
))}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants