test(executor-e2e): cover conclusion work-item filing for signal safe outputs

[Copilot]

Summary

The signal safe outputs (noop, missing-tool, missing-data) have no ADO write path — the executor only records them in safe-outputs-executed.ndjson. Their user-visible effect is produced one job later by the Conclusion job, which files work items from that manifest. The harness stopped at the executor record, so nothing exercised signal → manifest → work item end to end.

Changes:

• Post-execute phase (scenario.ts, runner.ts, execute-cli.ts) — new optional postExecute(ctx, state, run) hook that runs after a successful ado-aw execute and before assert, handed the safe-output dir and parsed records. Failures report phase post-execute; SkipError skips; cleanup still runs. runExecute now returns safeOutputDir, and spawnCollect is exported (with a label) so both child processes share the same hang protection.

• conclusion-cli.ts — spawns the compiled conclusion.js with the same flat AW_* env contract the compiler emits (AW_SAFE_OUTPUT_DIR, AW_PIPELINE_NAME, AW_<TOOL>_TITLE_PREFIX, …). The bundle is a build artifact, so scenarios skip when EXECUTOR_E2E_CONCLUSION_BUNDLE is unset or missing rather than failing.

• scenarios/conclusion.ts — four scenarios asserting the filed work item via ADO REST:

  Scenario	Proves
  conclusion-noop	work item created with configured title/type/tags, body carries the rendered report + stats
  conclusion-missing-tool	same incl. reported tool name; a second reporter run appends exactly one comment instead of filing a duplicate
  conclusion-missing-data	same incl. data type and reason
  conclusion-report-as-work-item-false	per-tool opt-out files nothing

  Titles are unique per build ([ado-aw-e2e conclusion] ado-aw-det-<buildId>-<scenario>) so concurrent runs never dedup into each other's item; each is deleted in cleanup. Work-item lookup polls because WIQL lags creation — which also guarantees the reporter's own dedup query can see the item before the append run.

• Pipeline / docs — azure-pipelines.yml builds conclusion.js, passes its path, and adds scripts/ado-script/src/conclusion/** to the PR path filter; coverage documented in tests/executor-e2e/README.md and docs/conclusion.md.

Test plan

• npm run typecheck and npm test (1190 tests) in scripts/ado-script.
• New unit tests: three runner tests pinning post-execute ordering, failure and skip handling; conclusion-cli.test.ts pinning the env contract, the skip paths, and non-zero bundle exit.
• Manually ran the freshly built conclusion.js over a hand-written manifest to confirm the signal-detection and opt-out log lines the scenarios assert on.
• The live ADO assertions run only in the executor-e2e pipeline (AgentPlayground); they skip locally without a built bundle.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟢 Approval recommended

The changes are well-scoped to test harness + pipeline wiring, with added unit tests and no apparent contract mismatches in the Conclusion env/config handling.

Pull request overview

Extends the deterministic executor-e2e harness to exercise the real Conclusion reporter (conclusion.js) end-to-end for signal safe outputs, ensuring the “signal → executed manifest → filed work item” path is covered in CI and documented.

Changes:

• Added an optional postExecute phase to scenarios and runner plumbing so post-Stage-3 consumers (Conclusion) can run before assertions.
• Introduced a conclusion-cli wrapper plus four new conclusion-* E2E scenarios that run the compiled bundle, then assert/cleanup the resulting work items via ADO REST.
• Updated the executor-e2e pipeline and docs to build/pass the conclusion.js artifact and describe the new coverage.

File summaries

File	Description
tests/executor-e2e/README.md	Documents new Conclusion scenarios and local env requirements.
tests/executor-e2e/azure-pipelines.yml	Builds conclusion.js, passes its path to the harness, and updates PR path filters.
scripts/ado-script/src/executor-e2e/scenarios/index.ts	Registers the new Conclusion scenarios in deterministic order.
scripts/ado-script/src/executor-e2e/scenarios/conclusion.ts	Adds four end-to-end Conclusion scenarios asserting work-item filing/dedup/opt-out.
scripts/ado-script/src/executor-e2e/scenario.ts	Adds PostExecuteRun and optional postExecute hook to the scenario contract.
scripts/ado-script/src/executor-e2e/runner.ts	Executes postExecute after a successful ado-aw execute, with skip/failure handling.
scripts/ado-script/src/executor-e2e/execute-cli.ts	Returns safeOutputDir and exports spawnCollect for shared hang protection.
scripts/ado-script/src/executor-e2e/conclusion-cli.ts	New wrapper that runs compiled conclusion.js with the compiler’s flat AW_* env contract and skips when absent.
scripts/ado-script/src/executor-e2e/tests/runner.test.ts	Adds unit tests pinning postExecute ordering, error handling, and skip behavior.
scripts/ado-script/src/executor-e2e/tests/conclusion-cli.test.ts	Adds unit tests for bundle resolution, env contract passing, and non-zero exit behavior.
docs/conclusion.md	Documents unit vs end-to-end test coverage for Conclusion reporting.

Review details

• Files reviewed: 11/11 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.