Skip to content

[ops] broker — safely retry failed current-main workflow jobs #148

Description

@ayhammouda

Vision — automated project maintainer

Context

Main 44c80f91aea9e133c42f81df2ed83c3c53448faf remains incompletely validated after #147. CI run 37368971571 and Scorecard run 37368971799 each have jobs with no runner and no executed steps; their annotations say “The job was not acquired by Runner of type hosted even after multiple attempts”. Four executed Python 3.12/3.13 matrix jobs, audit, CodeQL, installed package smoke and product regression passed. The Python 3.14 jobs and Scorecard were cancelled, not passed. GitHub Status reports Actions operational on 2026-10-08. The existing pdctl API allowlist cannot rerun workflows. No source-code failure is established.

References: AGENTS.md, AGENT-EXECUTION-PIPELINE.md ownership amendments, ops/vision/README.md; https://docs.github.com/en/rest/actions/workflow-runs#re-run-failed-jobs-from-a-workflow-run ; https://www.githubstatus.com/api/v2/summary.json .

Goal

Prepare a narrowly validated, broker-only failed-job rerun operation so transient hosted-runner failures need not strand verified development.

Acceptance criteria

  • A dedicated pdctl rerun RUN_ID HEAD_SHA (or equivalently narrow command) checks the fixed repository, exact 40-hex head, completed failed/cancelled run, and current-main identity before requesting only GitHub's rerun-failed-jobs endpoint.
  • Refuse mismatched/stale SHAs, other repositories, successful or active runs, arbitrary endpoints/options and malformed identifiers; verify event/branch metadata and disallow unreviewed pull-request code.
  • Preserve existing serialization/credential boundary. No arbitrary general API allowlist expansion, credentials returned, privileged installation, protection changes, skipping/weakening gates or direct host GitHub use.
  • Add meaningful positive/negative mocked broker/client tests and exact locked canonical gates; record commands/exits and an isolated committed bundle.
  • Document that owner review, independent prepublication/exact-head verification and root-owned installation remain required; repository merge alone does not update installed policy. No installation is authorized to workers.

Scope boundaries

In scope: smallest ops/vision client/control/tests/documentation change for retrying failed jobs of current-main runs. Vision explicitly authorizes this policy/operational proposal. Existing two-symbol feature delivery and #138 repair remain preserved and paused; do not change their code.
Out of scope: workflow edits, force-push/no-op commits to trigger CI, dispatching arbitrary refs, rerunning unreviewed PRs, releases, product code, dependency/corpus/scorer changes, credential or root-policy access.

Forbidden-territory reminder

Do not install or alter the live root-owned broker; this is only a reviewed repository proposal. Do not weaken tests or protection. Use existing injected/mocked test patterns; run contributed code only in pd-implementer.

Validation commands

uv sync --locked --dev; uv run --locked ruff check src/ tests/ benchmarks/ ops/ .github/scripts/; uv run --locked pyright src/ benchmarks/; uv run --locked pytest --tb=short -q; doctor if a valid worker index is available, otherwise disclose unavailable rather than claim pass. Include focused broker/client tests and git diff --check.

PR template

Refs this issue; exact base/head/tree; checked acceptance with evidence; canonical command exits and limitations; independent review pending; supervisor review: broker authorization surface.

Recovery

Stop after two failed repairs without a new hypothesis. A rejected/unavailable configured openai/gpt-6.1-sol worker route is a blocker, not permission to switch models. Return a committed bundle and JSON decision/handoff to /var/lib/python-docs/exchange/implementation. If unable to finish, checkpoint diagnosis without a passing claim.

Effort estimate

One 900-second isolated worker attempt. Root installation cannot be completed by this role.

Product decision and outcome review

User problem: a transient infrastructure failure prevents trustworthy post-merge validation and delays delivery. Baseline: two current-main failed runs, three unstarted cancelled jobs, no permitted retry. Target: safely retry only those failed current-main jobs without mutating source or reducing gates. Non-goals listed above. Review 2026-10-10; retain only if the narrow retry can be independently audited and improves recovery, revisit/remove if the broker exposes equivalent authorized functionality. No paid calls or product-quality claim.

Activity

  1. ayhammouda commented on Oct 8, 2026

    @ayhammouda
    OwnerAuthor

    Vision — automated project maintainer

    The narrow recovery proposal is committed at 6745681dd418dc6a09f6184e9533ad35e4fc9269, based on current main 44c80f91aea9e133c42f81df2ed83c3c53448faf (tree 0ac3d6d15cf35e411f437329d14f897fbf03b681). I imported the verified bundle and statically reviewed all four changed files. No product/workflow/dependency/corpus/scorer or existing-test changes.

    It admits only an exact current-main failed/cancelled completed push run in this repository, checks both repositories and run metadata, rechecks main immediately before a bodyless failed-jobs-only retry, and preserves the broker lock and general API allowlist. Temporary authentication only: App mode explicitly refuses under its unchanged actions:read permission ceiling. A retry acceptance is never reported as passing CI.

    Fresh developer evidence: 86 focused broker/client tests, 666 full tests, locked lint/type checks and diff checks pass. An initial two-line lint failure was repaired and retained in evidence. Doctor passes on a symbol-only index, not a full documentation index; no product-regression claim follows. The owner ran no contributed code.

    Not published, independently approved, installed or used for a live retry. Prepared decision and bundle are checkpointed for mandatory independent prepublication review at the start of the next full review window, followed by exact-head verification and hosted checks. Merge alone cannot update the root-owned broker; authorized operator installation remains separate. Main's two Python 3.14 jobs and Scorecard are still unvalidated, not declared fixed. Existing #138 and two-symbol deliveries remain preserved. Outcome review: 2026-10-10.

  2. ayhammouda commented on Oct 8, 2026

    @ayhammouda
    OwnerAuthor

    Vision — automated project maintainer

    Mandatory independent prepublication review was attempted on the retained #148 proposal. The broker rejected it with ValueError: Independent review rejected: incomplete check evidence (exit 1). The unpublished broker-created head is 55724bb2538f92d624adc4c162cee6d3fe7afea3, based on 44c80f91aea9e133c42f81df2ed83c3c53448faf, and its tree is exactly the reviewed developer tree 0ac3d6d15cf35e411f437329d14f897fbf03b681. Verifier session: 407c5b13-a617-45b5-aef2-e584f83b3e0d; recorded failure count: 1.

    No runnable branch/PR was published, no policy was installed, and no CI retry or merge occurred. The exposed broker result does not identify the missing/mismatched/nonzero command evidence, so I am not interpreting this as a reproduced product defect or retrying unchanged content. Exact developer bundle and decision remain preserved for evidence-based recovery.

    Live current-main checks remain unchanged: both Python 3.14 jobs and Scorecard were cancelled before runner assignment; executed matrix jobs, product checks, audit and CodeQL pass. Those missing jobs are still not counted as passes. #138 preparation may continue in isolation, but its publication/merge remains gated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions