From d0b6953c16ef4f1917e2c92e418a370fd84cbc32 Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 02:54:50 -0700 Subject: [PATCH 01/17] docs: clarify released installation and upgrade paths --- CHANGELOG.md | 22 +++-- README.md | 54 ++---------- README.zh-CN.md | 55 ++---------- SECURITY.md | 10 ++- docs/DEVELOPMENT.md | 26 ++++++ docs/OPERATIONS.md | 85 ++++++++++++++++++- .../data/repository_corpus_manifest.json | 10 +-- 7 files changed, 154 insertions(+), 108 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a98051d72..6bc81c9fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,8 +2,17 @@ ## Unreleased -This is release-candidate preparation, not an official stable release or a -production-readiness declaration. +No next release has been published from this development branch. + +- Clarify release-pinned installation versus existing-environment upgrades. +- Add selected real business/migration acceptance to the CI summary gate. +- Prepare repeatable eight-asset packages only from complete verified image evidence; + external tag/Release publication still requires separate human authorization. + +## [1.0.4](https://github.com/DavidHLP/UltiCode/releases/tag/v1.0.4) — 2026-10-27 + +Published from `465b9fd031f2c398f39f4696d8b33360f635ed5d`. This release is not a +production-readiness certification. - Normalize Management problem difficulty at the API boundary, with regression tests. - Restore database-generated problem IDs with an additive owner migration; @@ -29,8 +38,7 @@ The Agent remains experimental and disabled by default. Tests using scripted models do not establish real-model quality, budget readiness or production acceptance. Local Docker/browser evidence is scoped to disposable local services. -Before an official release, retain the tested source commit, successful CI and -real-service acceptance evidence, full signed nine-service digest manifest, -SBOM/provenance/scan evidence and release notes as durable assets. The source-only -bundle is not deployment input. Production deployment, release publication and -branch-protection changes require separate approval. +The published release retains the exact source, complete nine-service digest +manifest, signed SBOM/provenance/scan evidence and checksums as eight durable +assets. Main CI/publication and independent verification passed. Existing +immutable tags were preserved; no business production deployment was performed. diff --git a/README.md b/README.md index 58cdf342c..9cda9bd85 100644 --- a/README.md +++ b/README.md @@ -50,58 +50,22 @@ The screenshots below are the design and regression reference screenshots mainta ### 1. Get the project ```bash -git clone https://github.com/DavidHLP/UltiCode.git +git clone --branch v1.0.4 https://github.com/DavidHLP/UltiCode.git cd UltiCode ``` -### 2. Prepare the environment +### 2. Choose the installation path before creating configuration -Prerequisites: Docker Compose v2, mise, Node.js 22.x (22.22.2 or newer on that line for the locked frontend dependencies), pnpm 11.0.0, PM2, plus `curl`, `timeout`, and `openssl`. Frontend CI selects Node 22.x and pins pnpm 11.0.0; the app manifests' `>=22.12.0` minimum alone does not cover every locked dependency. Backend startup runs through the repository's `services/mvnw` and the mise-managed Zulu Java 17. See [development prerequisites](docs/DEVELOPMENT.md#前置条件) for Dev Container toolchain selection and rebuild guidance. +The current release is [v1.0.4 (2026-10-27)](https://github.com/DavidHLP/UltiCode/releases/tag/v1.0.4). +Normal installations should pin a release; contributors can use `main` in a separate checkout. -Contributors can also use Dev Containers / Codespaces directly; creating the container only installs dependencies and then runs the unit gate by default — it does not start the full stack: +- **Deploy published images:** start with [release installation](docs/OPERATIONS.md#从已发布发行版安装). Download all assets, verify checksums and nine-image signature evidence, then prepare host TLS/Judge prerequisites and deployment approval. +- **Fresh local development:** open the [empty-volume installation guide](docs/DEVELOPMENT.md#干净卷首次安装仅本机-disposable-开发环境) while `.env` is still absent. That entry owns configuration generation, migration, backfill/parity, cutover and first startup. Do not run `init-env.sh` before it. +- **Existing configuration, databases or volumes:** preserve them and use the [upgrade and rollback entry](docs/OPERATIONS.md#升级与回滚入口). Never treat them as empty volumes, recreate configuration with `--force`, or only change a cutover marker. -```bash -./ulticode doctor --json -``` - -```bash -./scripts/dev/init-env.sh -``` - -### 3. Finish the database preparation - -> **Required reading for the first start** -> -> `init-env.sh` generates `SUBMISSION_CUTOVER_COMPLETE` as `false`. Both `dev-lite` and `dev-full` require `APP_SUBMISSION_ROUTING_MODE=remote` and a completed Submission cutover marker; the startup scripts refuse to run when either is missing. -> -> Read [Database migrations](docs/OPERATIONS.md#数据库迁移与-owner-收敛) first, and complete migration and verification according to the authorized cutover/backfill runbook. Do not skip the checks by only changing the marker. - -### 4. Start the development environment - -Once the preparation above is done, choose a mode: - -| Mode | Search | Frontends | -| --- | --- | --- | -| `dev-lite` | Database-backed search; no Search worker | Not started by default | -| `dev-full` | Indexed search with the Search worker | Both frontends started | - -```bash -./scripts/dev/up.sh --mode dev-lite -``` - -`dev-lite` uses database search, does not start the Search worker, and does not start either frontend by default; run the following when you need the browser UI: - -```bash -./scripts/dev/up.sh --frontend-only -``` - -`dev-full` enables indexed search, the Search worker, and both frontends, still behind the same cutover gate: - -```bash -./scripts/dev/up.sh --mode dev-full -``` +Source development requires Docker Compose v2, mise-managed Zulu Java 17, Node.js 22.x (at least 22.22.2), pnpm 11.0.0, PM2, `curl`, `timeout` and `openssl`. Dev Containers prepare tooling and unit validation; they do not start the business stack. See [local development](docs/DEVELOPMENT.md#本地开发) for exact prerequisites, scopes, readiness and troubleshooting. -Once the frontends are running: [Console](http://localhost:9002) · [Management](http://localhost:9003). See [Local development](docs/DEVELOPMENT.md#本地开发) for scopes, logs, and troubleshooting entry points. +After completing the selected entry, local frontends are [Console](http://localhost:9002) and [Management](http://localhost:9003). Use the supported launcher rather than starting Maven/PM2 directly around its preparation and safety gates.
Optional Core pilot and rollback notes diff --git a/README.zh-CN.md b/README.zh-CN.md index f170e7026..924cf9d82 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -50,59 +50,22 @@ UltiCode 是一个全栈在线编程平台，提供题库、竞赛、在线评 ### 1. 获取项目 ```bash -git clone https://github.com/DavidHLP/UltiCode.git +git clone --branch v1.0.4 https://github.com/DavidHLP/UltiCode.git cd UltiCode ``` -### 2. 准备环境 +### 2. 在生成配置前选择安装路径 -前置条件：Docker Compose v2、mise、Node.js 22.x（当前锁定前端依赖要求该系列至少 22.22.2）、pnpm 11.0.0、PM2，以及 `curl`、`timeout`、`openssl`。前端 CI 选择 Node 22.x 并固定 pnpm 11.0.0；应用 manifest 的 `>=22.12.0` 下限不足以覆盖全部锁定依赖。后端启动由仓库内 `services/mvnw` 和 mise 管理的 Zulu Java 17 执行。Dev Container 的工具链选择与重建说明见[开发前置条件](docs/DEVELOPMENT.md#前置条件)。 +当前发行版是 [v1.0.4（2026-10-27）](https://github.com/DavidHLP/UltiCode/releases/tag/v1.0.4)。 +普通使用者应固定发行版，不直接跟随 `main`；贡献者可在独立 checkout 使用 `main`。 -贡献者也可以直接使用 Dev Containers/Codespaces；创建容器只安装依赖， -随后默认执行 unit 验证，不自动启动全栈： +- **下载发行版镜像部署**：从[发行版安装](docs/OPERATIONS.md#从已发布发行版安装)开始，下载全部资产、验证校验和与九镜像签名证据，再准备主机 TLS/Judge 和部署审批。 +- **全新本机开发环境**：在 `.env` 尚不存在时进入[干净卷首次安装](docs/DEVELOPMENT.md#干净卷首次安装仅本机-disposable-开发环境)。该入口负责生成配置、迁移、backfill/parity、cutover 与首次启动；不要提前运行 `init-env.sh`。 +- **已有配置、数据库或卷**：保留它们，先阅读[升级与回滚](docs/OPERATIONS.md#升级与回滚入口)。不得把已有环境当成空卷，也不要使用 `--force` 重建配置或只修改 cutover marker。 -```bash -./ulticode doctor --json -``` - -```bash -./scripts/dev/init-env.sh -``` - -### 3. 完成数据库准备 - -> **首次启动必读** -> -> `init-env.sh` 会把 `SUBMISSION_CUTOVER_COMPLETE` 生成为 `false`。`dev-lite` 和 `dev-full` 都要求 `APP_SUBMISSION_ROUTING_MODE=remote` 与已完成的 Submission cutover marker；未满足时启动脚本会拒绝执行。 -> -> 请先阅读[数据库迁移](docs/OPERATIONS.md#数据库迁移与-owner-收敛)，按授权的 cutover/backfill runbook 完成迁移与验证。不要仅修改标记来跳过检查。 - -### 4. 启动开发环境 - -完成上述准备后，选择适合的模式： - -| 模式 | 搜索 | 前端 | -| --- | --- | --- | -| `dev-lite` | 数据库 Search，不启动 Search Worker | 默认不启动 | -| `dev-full` | 索引 Search，启用 Search Worker | 启动两个前端 | - -```bash -./scripts/dev/up.sh --mode dev-lite -``` - -`dev-lite` 使用数据库 Search、不启动 Search worker，默认不启动两个前端；需要浏览器界面可另运行： - -```bash -./scripts/dev/up.sh --frontend-only -``` - -`dev-full` 会启用 indexed Search、Search worker 和两个前端，但仍受同一 cutover gate 保护： - -```bash -./scripts/dev/up.sh --mode dev-full -``` +本机源码开发需要 Docker Compose v2、mise 管理的 Zulu Java 17、Node.js 22.x（至少 22.22.2）、pnpm 11.0.0、PM2、`curl`、`timeout` 和 `openssl`。Dev Container 只准备工具链和 unit 验证，不自动启动业务栈。精确前置条件、scope、readiness 与排障见[本地开发](docs/DEVELOPMENT.md#本地开发)。 -前端启动后访问：[Console 用户端](http://localhost:9002) · [Management 管理端](http://localhost:9003)。更多 scope、日志与排障入口见[本地开发](docs/DEVELOPMENT.md#本地开发)。 +完成所选入口后，开发前端为 [Console](http://localhost:9002) 与 [Management](http://localhost:9003)。不要通过直接启动 Maven/PM2 绕过 launcher 的准备和安全门禁。
可选 Core 试点与回滚说明 diff --git a/SECURITY.md b/SECURITY.md index 6d3be2f24..f8b78c3cb 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,9 +1,11 @@ # Security policy -UltiCode is under active development. This repository has no published stable -GitHub Release or production security certification. Reports against the current -`main` branch are welcome; include the exact commit or immutable image digest. -The Agent is experimental and disabled by default. +[v1.0.4](https://github.com/DavidHLP/UltiCode/releases/tag/v1.0.4) was published on +2026-10-27. Security reports for this release and current `main` are welcome; +include the exact commit or immutable image digest. Older releases may require an +upgrade before a fix can be evaluated. Fixes may land on `main` and a later +release; no LTS/backport window, response-time SLA or production security +certification is promised. The Agent remains experimental and disabled by default. ## Private reporting diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 4ccdd5a67..b1e08807f 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -148,6 +148,32 @@ RPC 返回失败值前显式标记写事务回滚，避免子项拒绝后留下 Admin description readback 映射 nested detail，不能把读取已存在题目或单独 Judge 验收当成创建/编辑验收。fresh/upgrade ID 与外键保持检查属于真实 MySQL migration drill。 +### CI 真实安装与业务门禁 + +`Real acceptance` 是 `ci-ok` 的必需选择门禁：runtime/frontend/安装工具变化与所有 +release-selected/full runs 都必须成功，不能把 skipped/cancelled 当成通过。文档元数据 +变化不会误触发镜像发布。工作流先运行现有 MySQL fresh/upgrade/FK、backfill、backup +和权限隔离 drill，然后在独立 runner 顺序运行 `scripts/ci/business-acceptance.sh`。 + +业务入口只接受没有 `.env`、默认端口空闲、本机 Unix Docker endpoint 的新 checkout， +并要求 `ACCEPTANCE_DISPOSABLE=true`。它创建唯一 Compose project 和独立 PM2 home， +通过现有 launcher 准备 canonical migrations，执行全部 backfill/parity/cutover 检查后 +才导出完成标记。每个 JVM 有 heap 上限；不会并行启动多个本机 acceptance stack。 +不使用生产 secret、生产数据、mock API 或付费模型。 + +测试账号仅由 disposable dev bootstrap 与 Auth registration 创建。Manager 的直接权限 +通过真实 Auth API 临时授予并过期，不改全局 role policy；Console participant 保持 USER。 +随后执行基础与扩展 browser scripts，覆盖 AC/WA、重判、完整 admin create/edit/readback、 +非法子项事务回滚、contest 提交及最终排名。只上传排除凭据的 JSON 报告，不上传 `.env`、 +cookies、源码、PM2 state 或原始服务日志。退出时仅停止该测试 project/home，保留本机卷供 +诊断；GitHub hosted runner 生命周期负责回收其临时磁盘。 + +本机重现需在新 checkout 安装锁定依赖、PM2 与 Playwright Chromium，然后运行： + +```bash +ACCEPTANCE_DISPOSABLE=true bash scripts/ci/business-acceptance.sh +``` + ### 已准备环境启动 ```bash diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 24ed456bb..da972ca35 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -8,6 +8,54 @@ 本仓库没有生产环境。生产 Compose、CI/CD、远程 SSH、证书、registry、数据库和流量切换均是可执行但需部署方授权的控制面；disposable Compose/DinD 只证明仓库实现，不构成生产应用证据。 +### 从已发布发行版安装 + +当前固定版本为 [v1.0.4（2026-10-27）](https://github.com/DavidHLP/UltiCode/releases/tag/v1.0.4)， +源码为 `465b9fd031f2c398f39f4696d8b33360f635ed5d`。该发布有八个长期资产， +包含源码、完整九服务 manifest/set、签名证据 ZIP、verification report、release notes、 +license 和 SHA256SUMS。Actions artifact 是 publisher 的原始证据来源，发行版下载是 +日常安装入口；不能用 source-only plan 或单个镜像替代完整 manifest。 + +在新目录下载所有资产并验证后再使用；已有目录不应覆盖： + +```bash +set -euo pipefail +mkdir ulticode-v1.0.4 +cd ulticode-v1.0.4 +gh release download v1.0.4 --repo DavidHLP/UltiCode +sha256sum --check SHA256SUMS +tar -xzf UltiCode-465b9fd031f2c398f39f4696d8b33360f635ed5d.tar.gz +python3 -m zipfile -e release-evidence-v1.0.4.zip evidence +cd UltiCode-465b9fd031f2c398f39f4696d8b33360f635ed5d +./scripts/runbooks/promote-release.sh ../evidence \ + 465b9fd031f2c398f39f4696d8b33360f635ed5d ghcr.io/davidhlp/ulticode +``` + +最后一条仅做 read-only 验证，需要 Docker Buildx 和 publisher 固定版本的 Cosign； +它验证全部九个 digest、签名、SPDX/SLSA predicate、source/recipe、扫描策略和已存在标签。 +必须要求精确 signer identity/issuer（见下文），不得跳过 Rekor/签名验证或改用通配身份。 +下载失败、缺少资产、校验和或证据不一致时停止，不要使用旧目录中的残留文件。 + +安装前由部署方准备主机、受保护 GitHub Environment、独立审批者、经核对的 SSH 指纹、 +registry 拉取权限、owner DB 与 migration principal、Redis/Nacos 身份及 secret-managed TLS/JWKS。 +Judge 仍要求 remote/rootless daemon、`DOCKER_TLS_VERIFY=1`、只读 client certificate、 +共享绝对 workspace 和固定 seccomp；生产不挂载 docker.sock。这里不生成生产凭据或新基础设施。 + +从发行版 tag/ref 选择手动 `cd-deploy.yml`，提供已验证的九行 `release-manifest.txt`、 +目标 Environment 与 `services=all`；首次安装不跳过 migration。工作流的 `github.sha` +必须是该 release source，主机 checkout、schema checksum 和 manifest 必须一致。 +保留现有 host preflight、writer drain、migration、artifact/sandbox verification 与 +全服务 readiness；只有 descriptor 标记 HEALTHY 才算部署成功。容器 healthy 或 PM2 online +不能替代真实登录、提交与 terminal Judge 验收。 + +### 升级与回滚入口 + +已有环境先保留 `.env`、卷、上一份完整健康 release descriptor 和数据库/对象备份； +不要运行空卷入口或 `init-env.sh --force`。先比较 source/schema/contract 兼容性， +按下文 Owner migration 顺序、writer drain、增量 backfill/parity、cutover 和健康门禁升级。 +回滚复用上一份已验证的完整九镜像 descriptor，保持 `skip_migrations=true`，不做 schema +降级、恢复跨 Owner grant 或重写已应用迁移；细节见[健康与回滚](#健康与回滚)。 + ### 开发部署 开发启动命令与 mode 语义由[本地开发](DEVELOPMENT.md#本地开发)统一维护；正常 `dev-lite`/`dev-full` 都要求 `APP_SUBMISSION_ROUTING_MODE=remote` 和 `SUBMISSION_CUTOVER_COMPLETE=true`。 @@ -22,7 +70,33 @@ Docker Verify builds each service locally (`load: true`, no push) and runs pinne Each service must pass HIGH/CRITICAL Trivy scanning, then be signed and receive SPDX/SLSA attestations for the same candidate digest. The publisher reads BuildKit provenance attached to that immutable candidate, rather than synthesizing a source-only predicate. The producer explicitly selects the supported SLSA v0.2 format; extraction and promotion validate the image subject, repository, revision and BuildKit recipe, including the Dockerfile path configured for that service in `.github/services-matrix.json`. BuildKit compatibility v30 may also repeat the local recipe and VCS arguments under `invocation.parameters.root`; both validators require that redundant path and argument map to match the primary fields exactly, and reject unsupported compatibility versions. The publisher passes the full `https://slsa.dev/provenance/v0.2` URI to `cosign attest`: the `slsaprovenance` signing alias in Cosign v2.5.3 drops BuildKit's namespaced metadata during typed serialization. Verification may still use that alias to select the same predicate type, and promotion compares the complete verified predicate with the extracted evidence. A populated BuildKit statement subject must identify the image manifest attached to the candidate index; a mismatched or malformed subject is rejected. This does not claim a certified SLSA level. The `promote` job runs only after all nine matrix builds succeed; it validates all nine reports, source commit, digest subjects, signer identity, and verified DSSE predicates before inspecting or writing any tags. A changed backend image requires a new service release version in both `services/pom.xml` and its owner POM; an existing version tag may only be reused for the identical digest. Keep contract fixtures aligned with the intended release versions instead of weakening the immutable-tag guard. -Use the `release-manifest` artifact from a successful CI run's Docker Publish stage as canonical deployment input. It contains complete nine-service `release-manifest.txt`, `release-set.json`, and per-service evidence. Docker Publish is a reusable workflow called only after the same commit passes `ci-ok`; its authorization still requires a `main` push, never a PR, merge-group or manual verification run. Local `scripts/runbooks/promote-release.sh EVIDENCE_DIR SOURCE_SHA ghcr.io/davidhlp/ulticode` supports read-only dry-run only; `--execute` remains restricted to the authorized publisher on `main`. The workflow uses repository-wide `ulticode-release-tag-writer` concurrency with cancellation disabled. If promotion fails after candidate builds succeed, rerun only the failed `promote` job in that same Actions run, while its nine candidate artifacts remain within their retention period; do not rerun successful `build-and-push` jobs, since a rebuild may produce a different digest and collide with immutable tags. There is no independent push trigger or `workflow_dispatch` publishing entry point. Do not promote tags through local commands or other writers. This serializes repository-controlled writers, not external registry actors, and GHCR offers no compare-and-swap guarantee here. Existing manifest conflicts fail before registry inspection/writes; all planned tags are read back after promotion, including tags that already existed, before complete manifests are written. Matching existing tags are idempotent; a different digest is a hard failure. A mid-promotion failure may leave already verified tags in place, but emits no complete manifest. Never retarget or delete old tags/digests; retain previous successful release artifact for rollback. +Use the published release assets above for normal installations; the `release-manifest` artifact from the matching successful CI run's Docker Publish stage is the canonical producer evidence. It contains complete nine-service `release-manifest.txt`, `release-set.json`, and per-service evidence. Docker Publish is a reusable workflow called only after the same commit passes `ci-ok`; its authorization still requires a `main` push, never a PR, merge-group or manual verification run. Local `scripts/runbooks/promote-release.sh EVIDENCE_DIR SOURCE_SHA ghcr.io/davidhlp/ulticode` supports read-only dry-run only; `--execute` remains restricted to the authorized publisher on `main`. The workflow uses repository-wide `ulticode-release-tag-writer` concurrency with cancellation disabled. If promotion fails after candidate builds succeed, rerun only the failed `promote` job in that same Actions run, while its nine candidate artifacts remain within their retention period; do not rerun successful `build-and-push` jobs, since a rebuild may produce a different digest and collide with immutable tags. There is no independent push trigger or `workflow_dispatch` publishing entry point. Do not promote tags through local commands or other writers. This serializes repository-controlled writers, not external registry actors, and GHCR offers no compare-and-swap guarantee here. Existing manifest conflicts fail before registry inspection/writes; all planned tags are read back after promotion, including tags that already existed, before complete manifests are written. Matching existing tags are idempotent; a different digest is a hard failure. A mid-promotion failure may leave already verified tags in place, but emits no complete manifest. Never retarget or delete old tags/digests; retain previous successful release artifact for rollback. + +### 下一次发行版资产准备 + +先为实际改变的 backend image 分配新 service version（reactor 与 owner POM 一致）， +通过全部质量/真实业务门禁后，再由已授权的 main publisher 构建、签名并晋升完整九镜像。 +不能为绕过旧 tag 冲突重跑成功的候选 build；失败晋升只重试同一次 run 的 promote。 + +对已经成功发布的完整证据运行新的只读包装入口；源码 checkout 必须干净并位于精确 SHA： + +```bash +python3 scripts/runbooks/prepare-verified-release.py FULL_SOURCE_SHA EVIDENCE_DIR NEW_OUTPUT_DIR \ + --version vNEXT --notes REVIEWED_NOTES.md --workflow-run SUCCESSFUL_MAIN_RUN_ID \ + --source-root EXACT_SOURCE_CHECKOUT +``` + +将 `vNEXT` 替换为实际语义版本（例如 `v1.0.5`）。工具核对成功 main push/source、完整 +九服务 manifest、全部 evidence hashes，再复用该 source 的签名、SPDX/SLSA、扫描和 +registry tag readback 验证；不接受外部声称已验证的空 proof。它不构建镜像、不写 registry、 +不创建 Git tag/Release，也不部署。输出固定八资产布局及 SHA256SUMS；相同输入可重复 +验证已有完整目录，不同/部分目录或文件冲突拒绝覆盖，修改过的 source/proof 会失败。 +旧的 source-only 工具仍只用于尚无镜像证据的准备阶段，不是这个入口的替代品。 + +公开发布前必须另外取得精确 Git tag/目标 SHA 与 GitHub Release 的人工授权，重新检查 +二者不存在，以原子创建方式新增 tag，绝不 force/覆盖。上传后验证所有资产 hashes、tag +与 Release source，再核验公开下载的 SHA256SUMS。保留旧发行版与镜像，不能覆盖 +v1.0.4 或其他既有资产。本阶段的包装工具不会执行这些外部发布动作。 ### 未发布的源码制品准备 @@ -72,6 +146,15 @@ GitHub Actions 由 `.github/workflows/ci.yml` 编排，`ci-ok` 是稳定汇总 - GHCR 仅向指定构建与晋升 job 开放写权限；部署凭据只需拉取权限。OIDC 身份必须匹配被批准的发布 workflow，不能用通配身份替代。Environment secret 不传入 PR 测试。 - 保留成功发布的完整 artifact、上一份健康 descriptor 和数据库备份。根据组织审计与恢复周期配置 artifact/registry 保留策略；删除旧镜像或过期 artifact 前确认没有回滚依赖。 +建议仅对 `UltiCode/main` 使用以下最小配置（配置提案，不代表已启用）：PR 必需， +`ci-ok` 必需且 up-to-date，检查绑定当前 GitHub Actions app（核验 ID 15368），禁止 +force push/deletion；`required_approving_review_count=0`，不要求 code-owner 或 +last-push 的额外审批，适合单维护者。`enforce_admins=true` 使管理员正常直接 push 也被 +规则拦截；仓库管理员仍能通过管理权限修改/关闭规则，不能声称不可绕过。精确配置由 +维护者另行批准后通过 GitHub administration API 应用并 readback；本仓库脚本不改设置。 +[GitHub branch-protection API](https://docs.github.com/en/rest/branches/branch-protection#update-branch-protection) +允许 required approval count 为 0；增加独立审批者要求前应先确认团队可执行性。 + 本地契约、workflow lint 和离线替身测试不证明以上外部设置已启用，也不构成生产部署验收。真实部署仍需审批、主机预检、迁移验证与全服务健康检查闭环。 GitLab 直连部署入口（`.gitlab-ci.yml`）已于 2026-09-03 退役禁用：旧 job 对固定路径执行 `git reset --hard` 后直接 Compose build/up，绕过 owner migration、release descriptor、immutable image policy 与 health gate，且仓库内无该 runner 仍被授权的证据。任何未来 GitLab 适配器必须消费同一 canonical preflight/descriptor 接口（`scripts/runbooks/deployment-integrity.sh`），不得恢复 reset/build/up 捷径。只读查看当前 release set 使用： diff --git a/services/agent/data/repository_corpus_manifest.json b/services/agent/data/repository_corpus_manifest.json index 4e57583ba..dac896007 100644 --- a/services/agent/data/repository_corpus_manifest.json +++ b/services/agent/data/repository_corpus_manifest.json @@ -29,8 +29,8 @@ }, { "doc_id": "repository-development", - "version": "sha256-cae402bd326c114bc1c4acef873c477d7ae1fbd371358f1c1a251c582f301ad9", - "chunk_id": "repository-development:sha256-cae402bd326c114bc1c4acef873c477d7ae1fbd371358f1c1a251c582f301ad9:1", + "version": "sha256-2fdffd6f45a23a441a4286b11b28df416098c6b5e9da06c9a2142d6f4e9a533c", + "chunk_id": "repository-development:sha256-2fdffd6f45a23a441a4286b11b28df416098c6b5e9da06c9a2142d6f4e9a533c:1", "source_path": "docs/DEVELOPMENT.md", "access_scope": "repository-public", "sample_kind": "real", @@ -43,14 +43,14 @@ }, { "doc_id": "repository-operations", - "version": "sha256-62e7604861666186f9c127231c3e1990286d310f3b15891b2cbe15042576f847", - "chunk_id": "repository-operations:sha256-62e7604861666186f9c127231c3e1990286d310f3b15891b2cbe15042576f847:1", + "version": "sha256-21a06609f296f9a8d00b7a46abcd2948cf737d04c9ccb86ce74b4b8e976f0513", + "chunk_id": "repository-operations:sha256-21a06609f296f9a8d00b7a46abcd2948cf737d04c9ccb86ce74b4b8e976f0513:1", "source_path": "docs/OPERATIONS.md", "access_scope": "repository-public", "sample_kind": "real", "permission": "MIT:LICENSE:sha256:2be36e9d56578c1111740b6fe339ea3619cf1f2000af718f1a59e97d2e5d995c", "scope": "Repository documentation; local retrieval and evaluation under the included MIT license", - "source_position": "lines 57-61", + "source_position": "lines 131-135", "model_input_projection": "SourceHit.as_model_dict()", "source_trust": "untrusted-data", "content_digest": "sha256:e1ff3e78b8192a25aabf19db0fff20115b87b67b653240fbb31a0eba8a9b979e" From f62cba8608058b4909fa6a1e7049fb15ffcd912b Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 02:54:50 -0700 Subject: [PATCH 02/17] ci: require isolated business acceptance and verified release packaging --- .github/workflows/_acceptance.yml | 59 ++++++ .github/workflows/ci.yml | 11 +- docker/sandbox/harness/build.sh | 2 +- scripts/ci/business-acceptance.sh | 107 +++++++++++ scripts/ci/check-results.py | 4 +- scripts/ci/check-workflows.py | 13 ++ scripts/ci/select-checks.py | 4 +- scripts/runbooks/prepare-verified-release.py | 171 ++++++++++++++++++ scripts/test/acceptance-fixtures.mjs | 48 +++++ scripts/test/cicd-contract.sh | 1 + scripts/test/test_cicd_selection.py | 35 +++- .../test/verified-release-package-contract.py | 127 +++++++++++++ scripts/test/workflow-policy-contract.py | 14 ++ 13 files changed, 582 insertions(+), 14 deletions(-) create mode 100644 .github/workflows/_acceptance.yml create mode 100644 scripts/ci/business-acceptance.sh create mode 100644 scripts/runbooks/prepare-verified-release.py create mode 100644 scripts/test/acceptance-fixtures.mjs create mode 100644 scripts/test/verified-release-package-contract.py diff --git a/.github/workflows/_acceptance.yml b/.github/workflows/_acceptance.yml new file mode 100644 index 000000000..d15c07982 --- /dev/null +++ b/.github/workflows/_acceptance.yml @@ -0,0 +1,59 @@ +name: Real acceptance +on: + workflow_call: {} +permissions: + contents: read +jobs: + migration-safety: + name: Fresh and upgrade migration safety + runs-on: ubuntu-latest + timeout-minutes: 35 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: jdx/mise-action@fc42bcc7ddbf51124c30f95a877a0c5168a3bfde # v5.2.1 + - uses: actions/setup-java@b843ebf9b913146b38da2eaa7aeddf76074ae1ad # v6.1.0 + with: + distribution: temurin + java-version: '17' + cache: maven + - name: Real MySQL fresh/upgrade/backfill/backup/FK and isolation drill + run: bash scripts/dev/owner-migration-safety-integration-test.sh + business: + name: Clean install and real business lifecycle + needs: migration-safety + runs-on: ubuntu-latest + timeout-minutes: 60 + env: + ACCEPTANCE_DISPOSABLE: 'true' + MISE_HTTP_TIMEOUT: 5m + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: ./.github/actions/setup-frontend + - uses: jdx/mise-action@fc42bcc7ddbf51124c30f95a877a0c5168a3bfde # v5.2.1 + - uses: actions/setup-java@b843ebf9b913146b38da2eaa7aeddf76074ae1ad # v6.1.0 + with: + distribution: temurin + java-version: '17' + cache: maven + - name: Prepare pinned local test tools + run: | + npm install --global pm2@6.0.8 + mise install java@zulu-17.68.203.0 + pnpm exec playwright install --with-deps chromium + - name: Fresh isolated install and real browser/Judge acceptance + run: bash scripts/ci/business-acceptance.sh + - name: Preserve reports without credentials or raw service logs + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7 + with: + name: real-business-acceptance + path: | + .local/acceptance/installation.json + .local/browser-acceptance/report.json + .local/browser-business-acceptance/report.json + if-no-files-found: warn + retention-days: 14 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b30470990..2ae042b29 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -29,6 +29,7 @@ jobs: contract: ${{ steps.select.outputs.contract }} docker: ${{ steps.select.outputs.docker }} agent: ${{ steps.select.outputs.agent }} + acceptance: ${{ steps.select.outputs.acceptance }} release: ${{ steps.select.outputs.release }} full: ${{ steps.select.outputs.full }} steps: @@ -119,9 +120,15 @@ jobs: if: ${{ !cancelled() && needs.changes.outputs.agent == 'true' }} uses: ./.github/workflows/_agent.yml + acceptance: + name: Real acceptance + needs: changes + if: ${{ !cancelled() && needs.changes.outputs.acceptance == 'true' }} + uses: ./.github/workflows/_acceptance.yml + ci-ok: name: ci-ok - needs: [changes, governance, security, backend, frontend, contract, docker, agent] + needs: [changes, governance, security, backend, frontend, contract, docker, agent, acceptance] if: ${{ always() }} runs-on: ubuntu-latest timeout-minutes: 5 @@ -135,7 +142,7 @@ jobs: - name: Verify every selected gate succeeded env: RESULTS: ${{ toJSON(needs) }} - SELECTED: '{"backend":"${{ needs.changes.outputs.backend }}","console":"${{ needs.changes.outputs.console }}","management":"${{ needs.changes.outputs.management }}","landing":"${{ needs.changes.outputs.landing }}","contract":"${{ needs.changes.outputs.contract }}","docker":"${{ needs.changes.outputs.docker }}","agent":"${{ needs.changes.outputs.agent }}"}' + SELECTED: '{"backend":"${{ needs.changes.outputs.backend }}","console":"${{ needs.changes.outputs.console }}","management":"${{ needs.changes.outputs.management }}","landing":"${{ needs.changes.outputs.landing }}","contract":"${{ needs.changes.outputs.contract }}","docker":"${{ needs.changes.outputs.docker }}","agent":"${{ needs.changes.outputs.agent }}","acceptance":"${{ needs.changes.outputs.acceptance }}"}' run: python3 scripts/ci/check-results.py publish: diff --git a/docker/sandbox/harness/build.sh b/docker/sandbox/harness/build.sh index 8ba4b198a..94ba69d1a 100755 --- a/docker/sandbox/harness/build.sh +++ b/docker/sandbox/harness/build.sh @@ -78,7 +78,7 @@ build_java() { mkdir -p "${STAGING}/java" ( cd "${SCRIPT_DIR}/java" - mvn -B -q -DskipTests test-compile + "${MAVEN_BIN:-${SANDBOX_DIR}/../../services/mvnw}" -B -q -DskipTests test-compile cp -r target/classes/. "${STAGING}/java/" ) echo "[build.sh] java: $(ls "${STAGING}/java" | wc -l) class files" diff --git a/scripts/ci/business-acceptance.sh b/scripts/ci/business-acceptance.sh new file mode 100644 index 000000000..fe6de7b9f --- /dev/null +++ b/scripts/ci/business-acceptance.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +set -euo pipefail +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +cd "$ROOT" +[[ "${ACCEPTANCE_DISPOSABLE:-}" == true ]] || { echo 'Explicit disposable acceptance opt-in required' >&2; exit 2; } +[[ ! -e .env ]] || { echo 'Fresh acceptance requires a checkout without .env; existing state is preserved' >&2; exit 2; } +[[ -z "${DOCKER_HOST:-}" ]] || { echo 'Acceptance refuses an overridden Docker host' >&2; exit 2; } +endpoint="$(docker context inspect --format '{{.Endpoints.docker.Host}}')" +[[ "$endpoint" == unix://* ]] || { echo 'Acceptance requires a local Unix Docker endpoint' >&2; exit 2; } +mkdir -p .local/acceptance +exec 9>.local/acceptance.lock +flock -n 9 || { echo 'Another acceptance stack owns this checkout' >&2; exit 2; } +python3 - <<'PY' +import socket +for port in [23306, 26379, 28848, 29848, 9000, 9001, 9002, 9003, 9004, 9101, 9102, 9103, 9105, 9106, 9107, 9104]: + with socket.socket() as s: + try: s.bind(('127.0.0.1', port)) + except OSError: raise SystemExit(f'Acceptance port {port} is occupied; no existing stack was changed') +PY +export COMPOSE_PROJECT_NAME="ulticode-acceptance-$(date +%s)-$$" +export PM2_HOME="$ROOT/.local/acceptance/pm2" +export ENV_FILE="$ROOT/.env" +[[ -z "$(docker ps -aq --filter "label=com.docker.compose.project=$COMPOSE_PROJECT_NAME")" ]] +[[ -z "$(docker volume ls -q --filter "name=${COMPOSE_PROJECT_NAME}_")" ]] +unset MIGRATION_MYSQL_CONTAINER MIGRATION_MYSQL_CONTAINER_PORT MYSQL_CONTAINER \ + SUBMISSION_MIGRATION_DB_USER SUBMISSION_MIGRATION_DB_PASSWORD DEV_LOCAL_OWNER_BASELINE \ + DEV_LOCAL_OWNER_BASELINE_CONFIRM 2>/dev/null || true +phase=environment +prepared=false +finish() { + status=$? + trap - EXIT + export ACCEPTANCE_PHASE="$phase" ACCEPTANCE_EXIT="$status" + python3 - <<'PY' +import json,os +from pathlib import Path +Path('.local/acceptance/installation.json').write_text(json.dumps({ + 'evidence':'disposable-real-services', 'phase':os.environ['ACCEPTANCE_PHASE'], + 'passed':os.environ['ACCEPTANCE_EXIT']=='0', 'project':os.environ['COMPOSE_PROJECT_NAME'], + 'raw_logs_published':False, +},indent=2)+'\n') +PY + if [[ "$prepared" == true ]]; then + # Stop only this newly created project/home. Keep volumes for local diagnosis; + # hosted CI runners dispose of their own disks after the job. + ./scripts/dev/stop.sh --scope full-stack > .local/acceptance/private-cleanup.log 2>&1 || true + fi + exit "$status" +} +trap finish EXIT +./scripts/dev/init-env.sh > .local/acceptance/private-environment.log 2>&1 +python3 - <<'PY' +from pathlib import Path +import secrets,re +p=Path('.env');s=p.read_text() +values={'DEV_SEED_ADMIN_USERNAME':'acceptance_manager', 'DEV_SEED_ADMIN_EMAIL':'acceptance-manager@localhost.test', + 'DEV_SEED_ADMIN_PASSWORD':'CiA1_'+secrets.token_hex(20), 'DEV_SEED_ADMIN_ROLE':'SUPER_ADMIN', + 'JAVA_TOOL_OPTIONS':'"-Xms64m -Xmx384m"', 'SANDBOX_HOST_DIR':str(Path('.local/acceptance/sandbox').resolve())} +for key,value in values.items():s=re.sub(r'^'+key+r'=.*$',key+'='+value,s,flags=re.M) +s+='\nACCEPTANCE_USERNAME=acceptance_user\nACCEPTANCE_PASSWORD=CiA1_'+secrets.token_hex(20)+'\n' +p.write_text(s);p.chmod(0o600) +PY +set -a +source .env +set +a +export ACCEPTANCE_MANAGEMENT_USERNAME="$DEV_SEED_ADMIN_USERNAME" ACCEPTANCE_MANAGEMENT_PASSWORD="$DEV_SEED_ADMIN_PASSWORD" +export ACCEPTANCE_DISPOSABLE=true +# Disallow ambient migration routing; only generated loopback fixture settings win. +export MIGRATION_DB_HOST="$DB_HOST" MIGRATION_DB_PORT="$DB_PORT" MIGRATION_DB_NAME=ulticode +export MIGRATION_DB_USER=root MIGRATION_DB_PASSWORD="$MYSQL_ROOT_PASSWORD" +phase=sandbox +mkdir -p "$SANDBOX_HOST_DIR" +docker build -t ulticode-sandbox:base-17 -f docker/sandbox/Dockerfile.base docker/sandbox > .local/acceptance/private-sandbox.log 2>&1 +mise exec java@zulu-17.68.203.0 -- bash docker/sandbox/harness/build.sh --no-docker >> .local/acceptance/private-sandbox.log 2>&1 +docker build -t "$COMPOSE_PROJECT_NAME-sandbox" docker/sandbox >> .local/acceptance/private-sandbox.log 2>&1 +export SANDBOX_IMAGE="$COMPOSE_PROJECT_NAME-sandbox" +# Persist only generated test configuration; launcher loads this file itself. +printf '\nSANDBOX_IMAGE=%s\n' "$SANDBOX_IMAGE" >> .env +phase=fresh-migrations +# Compose v2 names this newly owned service deterministically; the migrator +# still validates that the running container publishes the generated endpoint. +export MIGRATION_MYSQL_CONTAINER="$COMPOSE_PROJECT_NAME-mysql-1" +prepared=true +./scripts/dev/up.sh --scope full-stack --prepare-submission-owner --skip-install > .local/acceptance/private-prepare.log 2>&1 +export MIGRATION_MYSQL_CONTAINER="$(docker compose --project-directory . --env-file .env -f docker/docker-compose.yml -f docker/docker-compose.dev.yml ps -q mysql)" +export MYSQL_CONTAINER="$MIGRATION_MYSQL_CONTAINER" +phase=backfill-and-cutover +./scripts/runbooks/submission-schema-cutover.sh backfill > .local/acceptance/private-cutover.log 2>&1 +SUBMISSION_BACKFILL_CONFIRM=I_UNDERSTAND_SUBMISSION_BACKFILL \ + SUBMISSION_BACKFILL_QUIESCE_CONFIRM=I_UNDERSTAND_SUBMISSION_BACKFILL_QUIESCE_ALL_WRITERS \ + ./scripts/runbooks/submission-schema-cutover.sh backfill --execute >> .local/acceptance/private-cutover.log 2>&1 +./scripts/runbooks/submission-schema-cutover.sh verify >> .local/acceptance/private-cutover.log 2>&1 +SUBMISSION_CUTOVER_CONFIRM=I_UNDERSTAND_SUBMISSION_CUTOVER \ + SUBMISSION_CUTOVER_QUIESCE_CONFIRM=I_UNDERSTAND_SUBMISSION_QUIESCE_ALL_WRITERS \ + ./scripts/runbooks/submission-schema-cutover.sh cutover --execute >> .local/acceptance/private-cutover.log 2>&1 +./scripts/runbooks/submission-schema-cutover.sh verify >> .local/acceptance/private-cutover.log 2>&1 +export SUBMISSION_CUTOVER_COMPLETE=true +phase=launch +./scripts/dev/up.sh --scope full-stack --skip-infra --skip-install --skip-migrate --rebuild > .local/acceptance/private-launch.log 2>&1 +phase=fixtures +node scripts/test/acceptance-fixtures.mjs +phase=browser-base +node scripts/test/browser-acceptance.mjs +phase=browser-business +node scripts/test/browser-business-acceptance.mjs +phase=complete +echo 'Fresh install and real business acceptance: PASS' diff --git a/scripts/ci/check-results.py b/scripts/ci/check-results.py index 6d93f36d7..1395fde21 100644 --- a/scripts/ci/check-results.py +++ b/scripts/ci/check-results.py @@ -26,12 +26,12 @@ def selected_flag(gate: str) -> bool: if result_of("changes") != "success": errors.append(f"changes must succeed (got {result_of('changes')})") - flags = {gate: selected_flag(gate) for gate in ("backend", "console", "management", "landing", "contract", "docker", "agent")} + flags = {gate: selected_flag(gate) for gate in ("backend", "console", "management", "landing", "contract", "docker", "agent", "acceptance")} gates = {"governance": True, "security": True, "backend": flags["backend"], "frontend": flags["console"] or flags["management"] or flags["landing"], "contract": flags["contract"], "docker": flags["docker"], - "agent": flags["agent"]} + "agent": flags["agent"], "acceptance": flags["acceptance"]} for gate, required in gates.items(): result = result_of(gate) if required: diff --git a/scripts/ci/check-workflows.py b/scripts/ci/check-workflows.py index 0764fc53a..a619300f3 100644 --- a/scripts/ci/check-workflows.py +++ b/scripts/ci/check-workflows.py @@ -123,6 +123,19 @@ def check_steps(steps, context): if set(events) != {"workflow_call"} and "concurrency" not in data: reject("entry workflow must declare concurrency") check_permissions(data.get("permissions"), set(), "workflow permissions") + if path.name == "ci.yml" and (root / ".github/services-matrix.json").exists(): + jobs = data.get("jobs", {}) + gate = jobs.get("acceptance", {}) + if (gate.get("uses") != "./.github/workflows/_acceptance.yml" + or gate.get("if") != "${{ !cancelled() && needs.changes.outputs.acceptance == 'true' }}"): + reject("real acceptance must use the selected required reusable gate") + if "acceptance" not in jobs.get("ci-ok", {}).get("needs", []): + reject("ci-ok must depend on real acceptance") + if jobs.get("changes", {}).get("outputs", {}).get("acceptance") != "${{ steps.select.outputs.acceptance }}": + reject("changes must expose the acceptance selection") + steps = jobs.get("ci-ok", {}).get("steps", []) + if not any('needs.changes.outputs.acceptance' in str(step.get('env', {}).get('SELECTED', '')) for step in steps): + reject("ci-ok must validate the acceptance selection") for name, job in data.get("jobs", {}).items(): if "permissions" in job: check_permissions(job["permissions"], WRITE_PERMISSIONS.get((path.name, name), set()), f"job {name} permissions") diff --git a/scripts/ci/select-checks.py b/scripts/ci/select-checks.py index 5d5532e48..e639be60d 100644 --- a/scripts/ci/select-checks.py +++ b/scripts/ci/select-checks.py @@ -12,7 +12,7 @@ import sys ZERO = "0" * 40 -GATES = ("backend", "console", "management", "landing", "contract", "docker", "agent") +GATES = ("backend", "console", "management", "landing", "contract", "docker", "agent", "acceptance") def valid_sha(value) -> bool: @@ -83,6 +83,7 @@ def any_match(*patterns: str) -> bool: "package.json", "pnpm-lock.yaml", "pnpm-workspace.yaml", ".dockerignore", "pyproject.toml", "uv.lock", "mise.toml", ".mise.toml", "docs/**", "assets/**", "README*", "AGENTS.md", "CONTEXT.md", "LICENSE", ".gitleaks.toml", + "CHANGELOG.md", "SECURITY.md", "CONTRIBUTING.md", "CODE_OF_CONDUCT.md", ) if not paths or any(not matches(path, *known) for path in paths): return {**dict.fromkeys(GATES, True), "release": True, "full": True} @@ -122,6 +123,7 @@ def any_match(*patterns: str) -> bool: ) selected = dict(backend=backend, console=console, management=management, landing=landing, contract=contract, docker=docker, agent=agent, + acceptance=release or backend or console or management or broad, release=release, full=False) return selected diff --git a/scripts/runbooks/prepare-verified-release.py b/scripts/runbooks/prepare-verified-release.py new file mode 100644 index 000000000..918f1defa --- /dev/null +++ b/scripts/runbooks/prepare-verified-release.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 +"""Prepare eight durable assets from an already published, verified image set. + +Never build images, mutate registry tags, create Git tags or publish a Release. +Publication requires a separate explicit human authorization. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import subprocess +import tempfile +import zipfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +REPOSITORY = "DavidHLP/UltiCode" +PREFIX = "ghcr.io/davidhlp/ulticode/" + + +def command(args, **kwargs): + return subprocess.run(args, check=True, capture_output=True, timeout=900, **kwargs).stdout + + +def sha256(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def regular(path): + if path.is_symlink() or not path.is_file(): + raise ValueError(f"Not a regular evidence file: {path.name}") + return path + + +def validate_source(root, sha): + if not re.fullmatch(r"[0-9a-f]{40}", sha): + raise ValueError("Source must be a full lowercase commit SHA") + git = lambda *args: command(["git", "-C", str(root), *args]).decode().strip() + if Path(git("rev-parse", "--show-toplevel")).resolve() != root.resolve(): + raise ValueError("Source root must be the repository root") + if git("rev-parse", "HEAD") != sha or git("status", "--porcelain", "--untracked-files=no"): + raise ValueError("Source checkout must be clean and at the exact verified SHA") + matrix = json.loads(git("show", f"{sha}:.github/services-matrix.json")) + names = {entry["name"] for entry in matrix} + if len(matrix) != 9 or len(names) != 9: + raise ValueError("Source must declare exactly nine services") + return names + + +def validate_evidence(evidence, sha, names): + if evidence.is_symlink() or not evidence.is_dir(): + raise ValueError("Evidence must be a real directory") + release = json.loads(regular(evidence / "release-set.json").read_text()) + services = release.get("services", []) + if release.get("source_commit") != sha or len(services) != 9 or {s["name"] for s in services} != names: + raise ValueError("Incomplete or mismatched source/service evidence") + files = {"release-set.json", "release-manifest.txt"} + directories = {f"candidate-evidence-{name}" for name in names} + if {p.name for p in evidence.iterdir()} != files | directories: + raise ValueError("Unexpected or incomplete evidence root") + suffixes = {"image-ref", "release.json", "spdx.json", "provenance.json", "trivy.json", "version", "source-commit"} + refs = [] + for service in sorted(services, key=lambda item: item["name"]): + name = service["name"] + ref = service["image_ref"] + if not re.fullmatch(re.escape(PREFIX + name) + r"@sha256:[0-9a-f]{64}", ref): + raise ValueError("Invalid immutable image reference") + refs.append(f"{name.upper().replace('-', '_')}_IMAGE_REF={ref}\n") + directory = evidence / f"candidate-evidence-{name}" + if directory.is_symlink() or not directory.is_dir(): + raise ValueError("Candidate directory must not be a symlink") + if set(service["evidence_sha256"]) != suffixes: + raise ValueError("Incomplete evidence hash inventory") + if {p.name for p in directory.iterdir()} != {f"{name}.{suffix}" for suffix in suffixes}: + raise ValueError("Unexpected or missing candidate evidence") + for suffix, expected in service["evidence_sha256"].items(): + path = regular(directory / f"{name}.{suffix}") + if sha256(path) != expected: + raise ValueError(f"Evidence checksum mismatch: {path.name}") + if (directory / f"{name}.source-commit").read_text().strip() != sha: + raise ValueError("Candidate source does not match") + if regular(evidence / "release-manifest.txt").read_text() != "".join(refs): + raise ValueError("Manifest does not exactly match the complete release set") + return release + + +def verify_published_set(root, evidence, sha, run_id): + if not re.fullmatch(r"[1-9][0-9]*", run_id): + raise ValueError("Workflow run must be a numeric ID") + run = json.loads(command(["gh", "api", f"repos/{REPOSITORY}/actions/runs/{run_id}"])) + if (run.get("head_sha") != sha or run.get("conclusion") != "success" + or run.get("event") != "push" or run.get("head_branch") != "main"): + raise ValueError("Workflow must be a successful main push at the exact source") + # Reuse the source revision's signature, SPDX/SLSA, scan, version and tag checks. + # No --execute, credential creation, registry writes or new image builds. + result = command(["bash", str(root / "scripts/runbooks/promote-release.sh"), + str(evidence.resolve()), sha, PREFIX.rstrip("/")]).decode() + if "candidate set validated: 9 services; mode=dry-run" not in result: + raise ValueError("Complete signed release verification did not succeed") + + +def prepare(root, evidence, sha, version, notes, run_id, output): + if not re.fullmatch(r"v[0-9]+\.[0-9]+\.[0-9]+(?:[-+][0-9A-Za-z.-]+)?", version): + raise ValueError("Version must be an explicit v-prefixed semantic version") + names = validate_source(root, sha) + release = validate_evidence(evidence, sha, names) + regular(notes) + if notes.stat().st_size > 131072 or not notes.read_text().strip(): + raise ValueError("Supply nonempty UTF-8 release notes of at most 128 KiB") + verify_published_set(root, evidence, sha, run_id) + if output.is_symlink(): + raise ValueError("Output must not be a symlink") + with tempfile.TemporaryDirectory(prefix="ulticode-release-") as temporary: + stage = Path(temporary) + archive_name = f"UltiCode-{sha}.tar.gz" + command(["git", "-C", str(root), "archive", "--format=tar.gz", f"--prefix=UltiCode-{sha}/", + "-o", str(stage / archive_name), sha]) + (stage / "LICENSE").write_bytes(command(["git", "-C", str(root), "show", f"{sha}:LICENSE"])) + (stage / "release-notes.md").write_bytes(notes.read_bytes()) + for name in ("release-set.json", "release-manifest.txt"): + (stage / name).write_bytes((evidence / name).read_bytes()) + with zipfile.ZipFile(stage / f"release-evidence-{version}.zip", "w", zipfile.ZIP_DEFLATED) as z: + for path in sorted(evidence.rglob("*")): + if path.is_file(): + info = zipfile.ZipInfo(path.relative_to(evidence).as_posix(), (1980, 1, 1, 0, 0, 0)) + info.compress_type = zipfile.ZIP_DEFLATED + info.external_attr = 0o100644 << 16 + z.writestr(info, regular(path).read_bytes()) + report = {"version": version, "source_commit": sha, + "workflow_run": f"https://github.com/{REPOSITORY}/actions/runs/{run_id}", + "service_count": len(release["services"]), "verification": "complete signed set and promoted tags passed", + "publication": "not performed; exact tag/Release publication requires separate explicit human authorization"} + (stage / "verification.json").write_text(json.dumps(report, indent=2, sort_keys=True) + "\n") + (stage / "SHA256SUMS").write_text("".join(f"{sha256(p)} {p.name}\n" for p in sorted(stage.iterdir()))) + if output.exists(): + if not output.is_dir() or {p.name for p in output.iterdir()} != {p.name for p in stage.iterdir()}: + raise ValueError("Existing output is incomplete or different; no overwrite") + for p in stage.iterdir(): + if sha256(regular(output / p.name)) != sha256(p): + raise ValueError("Existing output differs; no overwrite") + return "identical existing assets verified" + output.mkdir(parents=True, exist_ok=False) + for p in stage.iterdir(): + with (output / p.name).open("xb") as target: + target.write(p.read_bytes()) + return "eight unpublished verified assets prepared" + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("source_commit") + parser.add_argument("evidence", type=Path) + parser.add_argument("output", type=Path) + parser.add_argument("--version", required=True) + parser.add_argument("--notes", type=Path, required=True) + parser.add_argument("--workflow-run", required=True) + parser.add_argument("--source-root", type=Path, default=ROOT) + args = parser.parse_args() + try: + result = prepare(args.source_root, args.evidence, args.source_commit, args.version, + args.notes, args.workflow_run, args.output) + except (ValueError, OSError, KeyError, json.JSONDecodeError, subprocess.SubprocessError) as error: + # Do not echo subprocess logs, headers or authentication details. + parser.exit(1, f"Verified release packaging failed: {type(error).__name__}: {str(error) if isinstance(error, ValueError) else 'input or verifier command failed'}\n") + print(result + "; no tags, images or Releases changed") + + +if __name__ == "__main__": + main() diff --git a/scripts/test/acceptance-fixtures.mjs b/scripts/test/acceptance-fixtures.mjs new file mode 100644 index 000000000..96849bcf4 --- /dev/null +++ b/scripts/test/acceptance-fixtures.mjs @@ -0,0 +1,48 @@ +#!/usr/bin/env node +// Prepare only generated disposable accounts through real Auth APIs. +import assert from 'node:assert/strict' +import { request } from 'playwright' + +assert.equal(process.env.ACCEPTANCE_DISPOSABLE, 'true') +const origin = 'http://127.0.0.1:9002' +const admin = await request.newContext({ baseURL: origin }) +const participant = await request.newContext({ baseURL: origin }) +try { + const login = await admin.post('/api/auth/login', { data: { + username: process.env.ACCEPTANCE_MANAGEMENT_USERNAME, + password: process.env.ACCEPTANCE_MANAGEMENT_PASSWORD, + } }) + assert.equal(login.status(), 200, 'Disposable manager login failed') + const meResponse = await admin.get('/api/auth/me') + assert(meResponse.ok(), 'Manager identity read failed') + const me = (await meResponse.json()).data + assert.equal(me.user.role, 'SUPER_ADMIN', 'Fixture may grant only the generated superadmin') + const csrf = (await admin.storageState()).cookies.find(cookie => cookie.name === 'csrf_token') + assert(csrf?.value) + const permissions = [ + ['READ', 'PROBLEM'], ['CREATE', 'PROBLEM'], ['UPDATE', 'PROBLEM'], + ['READ', 'CONTEST'], ['CREATE', 'CONTEST'], ['UPDATE', 'CONTEST'], ['READ', 'SYSTEM'], + ] + for (const [action, resource] of permissions) { + const response = await admin.post(`/api/auth/admin/users/${me.user.id}/permissions`, { + headers: { 'X-CSRF-Token': decodeURIComponent(csrf.value) }, + data: { action, resource, expiresAt: new Date(Date.now() + 3_600_000).toISOString().slice(0, 19) }, + }) + assert(response.ok(), `Fixture permission ${action}:${resource} rejected`) + assert.equal((await response.json()).code, 0) + } + const registered = await participant.post('/api/auth/register', { data: { + username: process.env.ACCEPTANCE_USERNAME, + password: process.env.ACCEPTANCE_PASSWORD, + email: 'acceptance-user@localhost.test', + } }) + assert(registered.ok(), 'Disposable ordinary participant registration failed') + assert.equal((await registered.json()).code, 0) + const currentParticipant = await participant.get('/api/auth/me') + assert(currentParticipant.ok()) + assert.equal((await currentParticipant.json()).data.user.role, 'USER') + console.log('Disposable ordinary participant and expiring direct manager permissions: PASS') +} finally { + await admin.dispose() + await participant.dispose() +} diff --git a/scripts/test/cicd-contract.sh b/scripts/test/cicd-contract.sh index 6ec0dd931..8bf286c8b 100644 --- a/scripts/test/cicd-contract.sh +++ b/scripts/test/cicd-contract.sh @@ -15,3 +15,4 @@ python3 "$ROOT_DIR/scripts/test/fetch-buildkit-provenance-contract.py" bash "$ROOT_DIR/scripts/test/host-deploy-contract.sh" bash "$ROOT_DIR/scripts/test/deploy-runtime-contract.sh" bash "$ROOT_DIR/scripts/test/ssh-host-identity-contract.sh" +python3 "$ROOT_DIR/scripts/test/verified-release-package-contract.py" -v diff --git a/scripts/test/test_cicd_selection.py b/scripts/test/test_cicd_selection.py index 7d15c889a..c13834b73 100644 --- a/scripts/test/test_cicd_selection.py +++ b/scripts/test/test_cicd_selection.py @@ -156,51 +156,70 @@ def test_tooling_and_empty_diff_fail_safe(self): self.assertTrue(selection.select(set())["full"]) self.assertTrue(selection.select(None)["release"]) + def test_real_acceptance_follows_every_release_and_runtime_input(self): + for path in ['services/auth/pom.xml', 'init-db/migrations/app/new.sql', + 'apps/landing/src/main.ts', 'apps/management/src/api/admin/problems.ts', + 'scripts/ci/business-acceptance.sh', '.github/workflows/_acceptance.yml', 'pnpm-lock.yaml']: + with self.subTest(path=path): + self.assertTrue(selection.select({path})['acceptance']) + for path in ['README.md', 'README.zh-CN.md', 'CHANGELOG.md', 'SECURITY.md', 'CONTRIBUTING.md', 'CODE_OF_CONDUCT.md']: + chosen = selection.select({path}) + self.assertFalse(chosen['release']) + self.assertFalse(chosen['acceptance']) + class ResultTests(unittest.TestCase): def test_selected_skipped_is_failure(self): errors = results.validate( {"changes": "success", "governance": "success", "security": "success", "backend": "skipped", - "frontend": "skipped", "contract": "skipped", "docker": "skipped", "agent": "skipped"}, + "frontend": "skipped", "contract": "skipped", "docker": "skipped", "agent": "skipped", "acceptance": "skipped"}, {"backend": True, "console": False, "management": False, "landing": False, - "contract": False, "docker": False, "agent": False}, + "contract": False, "docker": False, "agent": False, "acceptance": False}, ) self.assertTrue(any("selected gate backend" in error for error in errors)) def test_unselected_skipped_is_valid_but_cancelled_is_not(self): decisions = {"backend": False, "console": False, "management": False, "landing": False, - "contract": False, "docker": False, "agent": False} + "contract": False, "docker": False, "agent": False, "acceptance": False} skipped = {"changes": "success", "governance": "success", "security": "success", "backend": "skipped", - "frontend": "skipped", "contract": "skipped", "docker": "skipped", "agent": "skipped"} + "frontend": "skipped", "contract": "skipped", "docker": "skipped", "agent": "skipped", "acceptance": "skipped"} self.assertEqual(results.validate(skipped, decisions), []) skipped["agent"] = "cancelled" self.assertTrue(any("unselected gate agent" in error for error in results.validate(skipped, decisions))) def test_github_needs_shape_and_string_outputs(self): needs = {name: {"result": "success"} for name in - ("changes", "governance", "security", "backend", "frontend", "contract", "docker", "agent")} + ("changes", "governance", "security", "backend", "frontend", "contract", "docker", "agent", "acceptance")} selected = {"backend": "false", "console": "true", "management": "false", "landing": "false", - "contract": "false", "docker": "false", "agent": "false"} + "contract": "false", "docker": "false", "agent": "false", "acceptance": "false"} self.assertEqual(results.validate(needs, selected), []) needs["frontend"]["result"] = "skipped" self.assertTrue(any("selected gate frontend" in error for error in results.validate(needs, selected))) def test_all_selection_flags_are_validated(self): needs = {name: "success" for name in - ("changes", "governance", "security", "backend", "frontend", "contract", "docker", "agent")} + ("changes", "governance", "security", "backend", "frontend", "contract", "docker", "agent", "acceptance")} decisions = dict.fromkeys(selection.GATES, True) decisions["management"] = "invalid" self.assertTrue(any("selection for management" in error for error in results.validate(needs, decisions))) def test_landing_only_requires_frontend_success(self): needs = {name: "success" for name in - ("changes", "governance", "security", "backend", "frontend", "contract", "docker", "agent")} + ("changes", "governance", "security", "backend", "frontend", "contract", "docker", "agent", "acceptance")} decisions = dict.fromkeys(selection.GATES, False) decisions["landing"] = True self.assertEqual(results.validate(needs, decisions), []) needs["frontend"] = "skipped" self.assertTrue(any("selected gate frontend" in error for error in results.validate(needs, decisions))) + def test_selected_acceptance_cannot_skip_or_fail(self): + selected = dict.fromkeys(selection.GATES, False) + selected['acceptance'] = True + needs = {name: 'success' for name in ['changes', 'governance', 'security', 'backend', 'frontend', 'contract', 'docker', 'agent', 'acceptance']} + for status in ['skipped', 'cancelled', 'failure', 'missing']: + needs['acceptance'] = status + self.assertTrue(any('selected gate acceptance' in e for e in results.validate(needs, selected))) + def test_missing_always_on_security_fails(self): self.assertTrue(any("security" in error for error in results.validate({"changes": "success"}, {}))) diff --git a/scripts/test/verified-release-package-contract.py b/scripts/test/verified-release-package-contract.py new file mode 100644 index 000000000..4817fe68c --- /dev/null +++ b/scripts/test/verified-release-package-contract.py @@ -0,0 +1,127 @@ +#!/usr/bin/env python3 +"""Offline packaging contracts; cryptography remains in the existing verifier.""" +import hashlib +import importlib.util +import json +import subprocess +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +ROOT = Path(__file__).resolve().parents[2] +spec = importlib.util.spec_from_file_location('packager', ROOT / 'scripts/runbooks/prepare-verified-release.py') +module = importlib.util.module_from_spec(spec) +spec.loader.exec_module(module) + + +class PackagingTests(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name) + self.source = self.root / 'source' + self.source.mkdir() + self.evidence = self.root / 'evidence' + self.evidence.mkdir() + self.output = self.root / 'output' + self.notes = self.root / 'notes.md' + self.notes.write_text('# Reviewed release notes\n') + self.names = ['backend-auth', 'backend-admin', 'backend-app', 'backend-submission', + 'backend-notification', 'backend-search', 'backend-judge', 'console', 'management'] + (self.source / '.github').mkdir() + (self.source / '.github/services-matrix.json').write_text(json.dumps([{'name': n} for n in self.names])) + (self.source / 'LICENSE').write_text('Test license\n') + self.git('init', '--quiet') + self.git('config', 'user.name', 'Fixture') + self.git('config', 'user.email', 'fixture@localhost.test') + self.git('add', '.') + self.git('commit', '--quiet', '-m', 'fixture') + self.sha = self.git('rev-parse', 'HEAD') + services = [] + refs = [] + for name in sorted(self.names): + directory = self.evidence / f'candidate-evidence-{name}' + directory.mkdir() + ref = module.PREFIX + name + '@sha256:' + 'a' * 64 + hashes = {} + for suffix in ['image-ref', 'release.json', 'spdx.json', 'provenance.json', 'trivy.json', 'version', 'source-commit']: + content = (self.sha if suffix == 'source-commit' else '{}') + '\n' + path = directory / f'{name}.{suffix}' + path.write_text(content) + hashes[suffix] = module.sha256(path) + services.append({'name': name, 'image_ref': ref, 'evidence_sha256': hashes}) + refs.append(f"{name.upper().replace('-', '_')}_IMAGE_REF={ref}\n") + (self.evidence / 'release-set.json').write_text(json.dumps({'source_commit': self.sha, 'services': services})) + (self.evidence / 'release-manifest.txt').write_text(''.join(refs)) + + def git(self, *args): + return subprocess.check_output(['git', '-C', str(self.source), *args], stderr=subprocess.DEVNULL).decode().strip() + + def prepare(self): + return module.prepare(self.source, self.evidence, self.sha, 'v1.0.5', self.notes, '123', self.output) + + def test_eight_deterministic_assets_and_idempotence(self): + with patch.object(module, 'verify_published_set') as verify: + self.assertIn('eight', self.prepare()) + before = {p.name: p.read_bytes() for p in self.output.iterdir()} + self.assertEqual(len(before), 8) + self.assertIn('identical', self.prepare()) + self.assertEqual(before, {p.name: p.read_bytes() for p in self.output.iterdir()}) + self.assertEqual(verify.call_count, 2) + for line in (self.output / 'SHA256SUMS').read_text().splitlines(): + digest, name = line.split(' ', 1) + self.assertEqual(module.sha256(self.output / name), digest) + self.assertEqual(self.git('rev-parse', 'HEAD'), self.sha) + + def test_modified_existing_output_is_never_overwritten(self): + with patch.object(module, 'verify_published_set'): + self.prepare() + (self.output / 'LICENSE').write_text('keep this changed file') + with self.assertRaisesRegex(ValueError, 'no overwrite'): + self.prepare() + self.assertEqual((self.output / 'LICENSE').read_text(), 'keep this changed file') + + def test_dirty_source_and_wrong_sha_fail_before_verification(self): + (self.source / 'LICENSE').write_text('modified') + with patch.object(module, 'verify_published_set') as verify: + with self.assertRaises(ValueError): self.prepare() + verify.assert_not_called() + self.assertFalse(self.output.exists()) + + def test_missing_service_or_tampered_evidence_fails_closed(self): + path = self.evidence / 'candidate-evidence-console/console.spdx.json' + path.write_text('tampered') + with patch.object(module, 'verify_published_set') as verify: + with self.assertRaisesRegex(ValueError, 'checksum'): self.prepare() + verify.assert_not_called() + self.assertFalse(self.output.exists()) + + def test_symlink_evidence_is_rejected(self): + path = self.evidence / 'candidate-evidence-console/console.trivy.json' + content = path.read_bytes() + path.unlink() + other = self.root / 'external' + other.write_bytes(content) + path.symlink_to(other) + with self.assertRaisesRegex(ValueError, 'regular'): self.prepare() + + def test_unverified_run_or_crypto_failure_cannot_create_assets(self): + with patch.object(module, 'verify_published_set', side_effect=ValueError('verification failed')): + with self.assertRaises(ValueError): self.prepare() + self.assertFalse(self.output.exists()) + + def test_verifier_requires_successful_matching_main_and_read_only_command(self): + run = {'head_sha': self.sha, 'conclusion': 'success', 'event': 'push', 'head_branch': 'main'} + with patch.object(module, 'command', side_effect=[json.dumps(run).encode(), b'candidate set validated: 9 services; mode=dry-run']) as command: + module.verify_published_set(self.source, self.evidence, self.sha, '123') + self.assertEqual(command.call_count, 2) + self.assertNotIn('--execute', command.call_args.args[0]) + run['head_sha'] = 'b' * 40 + with patch.object(module, 'command', return_value=json.dumps(run).encode()) as command: + with self.assertRaises(ValueError): module.verify_published_set(self.source, self.evidence, self.sha, '123') + self.assertEqual(command.call_count, 1) + + +if __name__ == '__main__': + unittest.main() diff --git a/scripts/test/workflow-policy-contract.py b/scripts/test/workflow-policy-contract.py index ed8ce1301..0661932d2 100644 --- a/scripts/test/workflow-policy-contract.py +++ b/scripts/test/workflow-policy-contract.py @@ -44,6 +44,20 @@ def check(self, data, filename="ci.yml"): (self.root / ".github/workflows" / filename).write_text(json.dumps(data)) return policy.check_repository(self.root) + def test_real_acceptance_cannot_be_removed_from_required_summary(self): + import shutil + shutil.copytree(ROOT / '.github', self.root / '.github', dirs_exist_ok=True) + self.assertEqual(policy.check_repository(self.root), []) + path = self.root / '.github/workflows/ci.yml' + original = policy.yaml.load(path.read_text(), Loader=policy.WorkflowLoader) + for mutation in ['needs', 'condition', 'output']: + data = copy.deepcopy(original) + if mutation == 'needs': data['jobs']['ci-ok']['needs'].remove('acceptance') + elif mutation == 'condition': data['jobs']['acceptance']['if'] = '${{ false }}' + else: del data['jobs']['changes']['outputs']['acceptance'] + path.write_text(json.dumps(data)) + self.assertTrue(any('acceptance' in error for error in policy.check_repository(self.root))) + def test_safe_workflow_and_yaml_on_key(self): self.assertEqual(self.check(WORKFLOW), []) parsed = policy.yaml.load("on:\n push:\npermissions: {}\nenabled: false\n", Loader=policy.WorkflowLoader) From baba7e7f13ef61fe8f4462505da46c9f2aa1fbef Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 03:07:54 -0700 Subject: [PATCH 03/17] fix(ci): align startup contracts with safe install entry points --- scripts/ci/business-acceptance.sh | 10 ++++++++-- scripts/dev/devstack-manifest-test.sh | 9 ++++++--- scripts/dev/docs-contract-test.sh | 6 ++++-- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/scripts/ci/business-acceptance.sh b/scripts/ci/business-acceptance.sh index fe6de7b9f..e6fcf1be1 100644 --- a/scripts/ci/business-acceptance.sh +++ b/scripts/ci/business-acceptance.sh @@ -30,14 +30,20 @@ prepared=false finish() { status=$? trap - EXIT + # Report generation must never prevent cleanup of our disposable processes. + set +e export ACCEPTANCE_PHASE="$phase" ACCEPTANCE_EXIT="$status" python3 - <<'PY' -import json,os +import json,os,re from pathlib import Path +kinds=set() +for path in Path('.local/acceptance').glob('private-*.log'): + text=path.read_text(errors='replace')[-1000000:] + kinds.update(re.findall(r'\b(?:org|com|java|javax|jakarta|io)\.(?:[A-Za-z_$][\w$]*\.)*[A-Za-z_$][\w$]*(?:Exception|Error)\b',text)) Path('.local/acceptance/installation.json').write_text(json.dumps({ 'evidence':'disposable-real-services', 'phase':os.environ['ACCEPTANCE_PHASE'], 'passed':os.environ['ACCEPTANCE_EXIT']=='0', 'project':os.environ['COMPOSE_PROJECT_NAME'], - 'raw_logs_published':False, + 'raw_logs_published':False, 'exception_classes':sorted(kinds)[:20], },indent=2)+'\n') PY if [[ "$prepared" == true ]]; then diff --git a/scripts/dev/devstack-manifest-test.sh b/scripts/dev/devstack-manifest-test.sh index 52aabbe02..15a02013d 100755 --- a/scripts/dev/devstack-manifest-test.sh +++ b/scripts/dev/devstack-manifest-test.sh @@ -8,7 +8,8 @@ source "$ROOT_DIR/scripts/dev/devstack-manifest.sh" assert_file_contains() { local file="$1" expected="$2" [[ -f "$ROOT_DIR/$file" ]] || { echo "missing guarded file: $file" >&2; return 1; } - grep -F -- "$expected" "$ROOT_DIR/$file" >/dev/null + grep -F -- "$expected" "$ROOT_DIR/$file" >/dev/null \ + || { echo "missing guarded content in $file: $expected" >&2; return 1; } } assert_file_not_contains() { @@ -213,8 +214,10 @@ assert_file_contains ecosystem.config.cjs "APP_FEATURES_USE_GENERATION_FENCE: pr assert_file_contains ecosystem.config.cjs "APP_FEATURES_JUDGE_QUEUE_USE_PORT: process.env.APP_FEATURES_JUDGE_QUEUE_USE_PORT || 'true'" assert_file_contains docker/docker-compose.prod.yml 'APP_SEARCH_READ_MODE=indexed' assert_file_contains docker/docker-compose.prod.yml 'APP_SEARCH_FALLBACK_TO_DATABASE=true' -assert_file_contains README.md './scripts/dev/up.sh --mode dev-lite' -assert_file_contains README.md './scripts/dev/up.sh --mode dev-full' +assert_file_contains README.md 'docs/DEVELOPMENT.md#干净卷首次安装仅本机-disposable-开发环境' +assert_file_contains README.md 'docs/OPERATIONS.md#升级与回滚入口' +assert_file_contains docs/DEVELOPMENT.md './scripts/dev/up.sh --mode dev-lite' +assert_file_contains docs/DEVELOPMENT.md './scripts/dev/up.sh --mode dev-full' assert_file_not_contains README.md 'pm2 restart ulticode-auth ulticode-admin' assert_file_not_contains README.md 'spring-boot:run -Dmaven.test.skip=true' if grep -Eq 'APP_FEATURES_JUDGE_QUEUE_ENVELOPE_VERSION|envelope-version:' \ diff --git a/scripts/dev/docs-contract-test.sh b/scripts/dev/docs-contract-test.sh index 34707c626..b8de540af 100644 --- a/scripts/dev/docs-contract-test.sh +++ b/scripts/dev/docs-contract-test.sh @@ -18,8 +18,10 @@ source "$ROOT_DIR/scripts/test/lib/assertions.sh" not_contains README.md 'pm2 start ecosystem.config.cjs' not_contains README.md 'pm2 restart ulticode-auth ulticode-admin' -contains README.md './scripts/dev/up.sh --mode dev-lite' -contains README.md './scripts/dev/up.sh --mode dev-full' +contains README.md 'docs/DEVELOPMENT.md#干净卷首次安装仅本机-disposable-开发环境' +contains README.md 'docs/OPERATIONS.md#升级与回滚入口' +contains docs/DEVELOPMENT.md './scripts/dev/up.sh --mode dev-lite' +contains docs/DEVELOPMENT.md './scripts/dev/up.sh --mode dev-full' contains README.md 'This version no longer supports `legacy-rollback`' not_contains README.md './scripts/dev/up.sh --mode legacy-rollback' not_contains docs/DEVELOPMENT.md './scripts/dev/up.sh --mode legacy-rollback' From 2b1a97d3b67e40a2517c3ec3d3f82f812deae19c Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 03:10:26 -0700 Subject: [PATCH 04/17] fix(ci): seed fresh acceptance through canonical startup --- scripts/ci/business-acceptance.sh | 4 +++- scripts/test/verified-release-package-contract.py | 13 ++++++++++++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/scripts/ci/business-acceptance.sh b/scripts/ci/business-acceptance.sh index e6fcf1be1..d6352200c 100644 --- a/scripts/ci/business-acceptance.sh +++ b/scripts/ci/business-acceptance.sh @@ -102,7 +102,9 @@ SUBMISSION_CUTOVER_CONFIRM=I_UNDERSTAND_SUBMISSION_CUTOVER \ ./scripts/runbooks/submission-schema-cutover.sh verify >> .local/acceptance/private-cutover.log 2>&1 export SUBMISSION_CUTOVER_COMPLETE=true phase=launch -./scripts/dev/up.sh --scope full-stack --skip-infra --skip-install --skip-migrate --rebuild > .local/acceptance/private-launch.log 2>&1 +# Follow the documented second startup: migrations are idempotent, and the +# supported DEV-LOCAL App seed adapter runs only when migrations are enabled. +./scripts/dev/up.sh --scope full-stack --skip-infra --skip-install --rebuild > .local/acceptance/private-launch.log 2>&1 phase=fixtures node scripts/test/acceptance-fixtures.mjs phase=browser-base diff --git a/scripts/test/verified-release-package-contract.py b/scripts/test/verified-release-package-contract.py index 4817fe68c..bbb89b88c 100644 --- a/scripts/test/verified-release-package-contract.py +++ b/scripts/test/verified-release-package-contract.py @@ -89,7 +89,7 @@ def test_dirty_source_and_wrong_sha_fail_before_verification(self): verify.assert_not_called() self.assertFalse(self.output.exists()) - def test_missing_service_or_tampered_evidence_fails_closed(self): + def test_tampered_evidence_fails_closed(self): path = self.evidence / 'candidate-evidence-console/console.spdx.json' path.write_text('tampered') with patch.object(module, 'verify_published_set') as verify: @@ -97,6 +97,17 @@ def test_missing_service_or_tampered_evidence_fails_closed(self): verify.assert_not_called() self.assertFalse(self.output.exists()) + def test_missing_service_fails_before_crypto_or_output(self): + path = self.evidence / 'release-set.json' + release = json.loads(path.read_text()) + release['services'].pop() + path.write_text(json.dumps(release)) + with patch.object(module, 'verify_published_set') as verify: + with self.assertRaisesRegex(ValueError, 'Incomplete'): + self.prepare() + verify.assert_not_called() + self.assertFalse(self.output.exists()) + def test_symlink_evidence_is_rejected(self): path = self.evidence / 'candidate-evidence-console/console.trivy.json' content = path.read_bytes() From 5fa8f21e239bc9311df4917d8e37a0c53330faf2 Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 03:12:12 -0700 Subject: [PATCH 05/17] test(acceptance): prepare persisted hidden judge fixtures --- scripts/test/acceptance-fixtures.mjs | 35 ++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/scripts/test/acceptance-fixtures.mjs b/scripts/test/acceptance-fixtures.mjs index 96849bcf4..929de99b2 100644 --- a/scripts/test/acceptance-fixtures.mjs +++ b/scripts/test/acceptance-fixtures.mjs @@ -5,7 +5,7 @@ import { request } from 'playwright' assert.equal(process.env.ACCEPTANCE_DISPOSABLE, 'true') const origin = 'http://127.0.0.1:9002' -const admin = await request.newContext({ baseURL: origin }) +const admin = await request.newContext({ baseURL: 'http://127.0.0.1:9003' }) const participant = await request.newContext({ baseURL: origin }) try { const login = await admin.post('/api/auth/login', { data: { @@ -31,6 +31,37 @@ try { assert(response.ok(), `Fixture permission ${action}:${resource} rejected`) assert.equal((await response.json()).code, 0) } + // The canonical DEV-LOCAL seed supplies the published problem, languages and + // tags. Judge cases are separate owner data: prepare them explicitly through + // the real Management API, with server-generated IDs and structured inputs. + const descriptionResponse = await admin.get('/api/admin/problems/1/description') + assert(descriptionResponse.ok(), 'Canonical problem seed is missing') + const description = (await descriptionResponse.json()).data + assert.equal(description.slug, 'two-sum') + assert.equal(description.isPublished, true) + assert(description.tags.length > 0, 'CRUD tag roundtrip requires seeded tags') + const codeResponse = await admin.get('/api/admin/problems/1/code') + assert(codeResponse.ok()) + assert((await codeResponse.json()).data.languages.some(language => language.value === 'python')) + for (const [nums, target, expected] of [[[3, 3], 6, [0, 1]], [[2, 5, 5, 11], 10, [1, 2]]]) { + const inputs = JSON.stringify([{ name: 'nums', value: nums }, { name: 'target', value: target }]) + const created = await admin.post('/api/admin/problems/1/test-cases', { + headers: { 'X-CSRF-Token': decodeURIComponent(csrf.value) }, + data: { isSample: false, isHidden: true, inputText: `nums = ${JSON.stringify(nums)}, target = ${target}`, + outputText: JSON.stringify(expected), inputs }, + }) + assert(created.ok(), 'Disposable hidden Judge fixture creation failed') + const envelope = await created.json() + assert.equal(envelope.code, 0) + assert(envelope.data?.id, 'Judge fixture must have a server-generated ID') + const readback = await admin.get(`/api/admin/problems/1/test-cases/${envelope.data.id}`) + assert(readback.ok()) + const persisted = (await readback.json()).data + assert.equal(persisted.isHidden, true) + assert.equal(persisted.isSample, false) + assert.equal(persisted.outputText, JSON.stringify(expected)) + assert.deepEqual(JSON.parse(persisted.inputs), JSON.parse(inputs)) + } const registered = await participant.post('/api/auth/register', { data: { username: process.env.ACCEPTANCE_USERNAME, password: process.env.ACCEPTANCE_PASSWORD, @@ -41,7 +72,7 @@ try { const currentParticipant = await participant.get('/api/auth/me') assert(currentParticipant.ok()) assert.equal((await currentParticipant.json()).data.user.role, 'USER') - console.log('Disposable ordinary participant and expiring direct manager permissions: PASS') + console.log('Canonical problem, persisted hidden Judge cases, ordinary participant and expiring manager permissions: PASS') } finally { await admin.dispose() await participant.dispose() From adc1f9ed189eaf3552ebba4b67a5b3513b45efa1 Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 03:14:38 -0700 Subject: [PATCH 06/17] fix(release): require canonical CI and existing promoted tags --- scripts/runbooks/prepare-verified-release.py | 32 +++++++++++++- .../test/verified-release-package-contract.py | 44 ++++++++++++++++--- 2 files changed, 68 insertions(+), 8 deletions(-) diff --git a/scripts/runbooks/prepare-verified-release.py b/scripts/runbooks/prepare-verified-release.py index 918f1defa..804d063c3 100644 --- a/scripts/runbooks/prepare-verified-release.py +++ b/scripts/runbooks/prepare-verified-release.py @@ -9,6 +9,7 @@ import argparse import hashlib import json +import os import re import subprocess import tempfile @@ -21,7 +22,11 @@ def command(args, **kwargs): - return subprocess.run(args, check=True, capture_output=True, timeout=900, **kwargs).stdout + # Identify the failed tool without exposing arguments, headers or logs. + result = subprocess.run(args, check=False, capture_output=True, timeout=900, **kwargs) + if result.returncode: + raise ValueError(f"Verification/packaging command {Path(args[0]).name} failed (exit {result.returncode})") + return result.stdout def sha256(path): @@ -81,6 +86,16 @@ def validate_evidence(evidence, sha, names): raise ValueError(f"Evidence checksum mismatch: {path.name}") if (directory / f"{name}.source-commit").read_text().strip() != sha: raise ValueError("Candidate source does not match") + version = (directory / f"{name}.version").read_text().strip() + tags = [f"sha-{sha[:7]}"] + if name.startswith("backend-"): + if not re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+(?:[-+][0-9A-Za-z.-]+)?", version): + raise ValueError("Backend evidence requires a valid service version") + tags.append(f"v{version}") + elif version: + raise ValueError("Frontend evidence must not invent a service version") + if service.get("tags") != tags: + raise ValueError("Promoted tag inventory does not match source/version evidence") if regular(evidence / "release-manifest.txt").read_text() != "".join(refs): raise ValueError("Manifest does not exactly match the complete release set") return release @@ -90,7 +105,9 @@ def verify_published_set(root, evidence, sha, run_id): if not re.fullmatch(r"[1-9][0-9]*", run_id): raise ValueError("Workflow run must be a numeric ID") run = json.loads(command(["gh", "api", f"repos/{REPOSITORY}/actions/runs/{run_id}"])) - if (run.get("head_sha") != sha or run.get("conclusion") != "success" + if (run.get("id") != int(run_id) or run.get("path") != ".github/workflows/ci.yml" + or run.get("repository", {}).get("full_name") != REPOSITORY + or run.get("head_sha") != sha or run.get("conclusion") != "success" or run.get("event") != "push" or run.get("head_branch") != "main"): raise ValueError("Workflow must be a successful main push at the exact source") # Reuse the source revision's signature, SPDX/SLSA, scan, version and tag checks. @@ -99,6 +116,17 @@ def verify_published_set(root, evidence, sha, run_id): str(evidence.resolve()), sha, PREFIX.rstrip("/")]).decode() if "candidate set validated: 9 services; mode=dry-run" not in result: raise ValueError("Complete signed release verification did not succeed") + # Promotion dry-run also permits tags that would still need creating. + # Packaging a deployable release requires every planned tag to EXIST now + # and resolve to its already signed immutable digest; never create it here. + release = json.loads((evidence / "release-set.json").read_text()) + for service in release["services"]: + image, digest = service["image_ref"].split("@", 1) + for tag in service["tags"]: + actual = command([os.environ.get("DOCKER_BIN", "docker"), "buildx", "imagetools", "inspect", + f"{image}:{tag}", "--format", "{{.Manifest.Digest}}"]).decode().strip() + if actual != digest: + raise ValueError(f"Published tag readback mismatch: {service['name']}:{tag}") def prepare(root, evidence, sha, version, notes, run_id, output): diff --git a/scripts/test/verified-release-package-contract.py b/scripts/test/verified-release-package-contract.py index bbb89b88c..298b3806a 100644 --- a/scripts/test/verified-release-package-contract.py +++ b/scripts/test/verified-release-package-contract.py @@ -46,11 +46,13 @@ def setUp(self): ref = module.PREFIX + name + '@sha256:' + 'a' * 64 hashes = {} for suffix in ['image-ref', 'release.json', 'spdx.json', 'provenance.json', 'trivy.json', 'version', 'source-commit']: - content = (self.sha if suffix == 'source-commit' else '{}') + '\n' + content = (self.sha if suffix == 'source-commit' else + ('1.0.5' if name.startswith('backend-') else '') if suffix == 'version' else '{}') + '\n' path = directory / f'{name}.{suffix}' path.write_text(content) hashes[suffix] = module.sha256(path) - services.append({'name': name, 'image_ref': ref, 'evidence_sha256': hashes}) + tags = [f'sha-{self.sha[:7]}'] + (['v1.0.5'] if name.startswith('backend-') else []) + services.append({'name': name, 'image_ref': ref, 'tags': tags, 'evidence_sha256': hashes}) refs.append(f"{name.upper().replace('-', '_')}_IMAGE_REF={ref}\n") (self.evidence / 'release-set.json').write_text(json.dumps({'source_commit': self.sha, 'services': services})) (self.evidence / 'release-manifest.txt').write_text(''.join(refs)) @@ -123,16 +125,46 @@ def test_unverified_run_or_crypto_failure_cannot_create_assets(self): self.assertFalse(self.output.exists()) def test_verifier_requires_successful_matching_main_and_read_only_command(self): - run = {'head_sha': self.sha, 'conclusion': 'success', 'event': 'push', 'head_branch': 'main'} - with patch.object(module, 'command', side_effect=[json.dumps(run).encode(), b'candidate set validated: 9 services; mode=dry-run']) as command: + run = {'id': 123, 'path': '.github/workflows/ci.yml', 'repository': {'full_name': module.REPOSITORY}, + 'head_sha': self.sha, 'conclusion': 'success', 'event': 'push', 'head_branch': 'main'} + proof = b'candidate set validated: 9 services; mode=dry-run' + with patch.object(module, 'command', side_effect=[json.dumps(run).encode(), proof] + [b'sha256:' + b'a' * 64] * 16) as command: module.verify_published_set(self.source, self.evidence, self.sha, '123') - self.assertEqual(command.call_count, 2) - self.assertNotIn('--execute', command.call_args.args[0]) + self.assertEqual(command.call_count, 18) + self.assertTrue(all('--execute' not in call.args[0] for call in command.call_args_list)) + self.assertEqual(command.call_args.args[0][1:4], ['buildx', 'imagetools', 'inspect']) run['head_sha'] = 'b' * 40 with patch.object(module, 'command', return_value=json.dumps(run).encode()) as command: with self.assertRaises(ValueError): module.verify_published_set(self.source, self.evidence, self.sha, '123') self.assertEqual(command.call_count, 1) + def test_unrelated_successful_workflow_is_rejected(self): + run = {'id': 123, 'path': '.github/workflows/unrelated.yml', 'repository': {'full_name': module.REPOSITORY}, + 'head_sha': self.sha, 'conclusion': 'success', 'event': 'push', 'head_branch': 'main'} + with patch.object(module, 'command', return_value=json.dumps(run).encode()) as command: + with self.assertRaises(ValueError): module.verify_published_set(self.source, self.evidence, self.sha, '123') + self.assertEqual(command.call_count, 1) + + def test_missing_or_different_promoted_tag_fails_without_writes(self): + run = {'id': 123, 'path': '.github/workflows/ci.yml', 'repository': {'full_name': module.REPOSITORY}, + 'head_sha': self.sha, 'conclusion': 'success', 'event': 'push', 'head_branch': 'main'} + for readback in [b'', b'sha256:' + b'b' * 64]: + with self.subTest(readback=readback), patch.object(module, 'command', side_effect=[ + json.dumps(run).encode(), b'candidate set validated: 9 services; mode=dry-run', readback]) as command: + with self.assertRaisesRegex(ValueError, 'tag readback'): + module.verify_published_set(self.source, self.evidence, self.sha, '123') + self.assertEqual(command.call_count, 3) + self.assertTrue(all('--execute' not in call.args[0] for call in command.call_args_list)) + + def test_incomplete_promoted_tag_inventory_is_rejected(self): + path = self.evidence / 'release-set.json' + release = json.loads(path.read_text()) + release['services'][0]['tags'] = [] + path.write_text(json.dumps(release)) + with patch.object(module, 'verify_published_set') as verify: + with self.assertRaisesRegex(ValueError, 'tag inventory'): self.prepare() + verify.assert_not_called() + if __name__ == '__main__': unittest.main() From e0738cfd0c27d7bbdec358866399b68c8be509cb Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 03:20:03 -0700 Subject: [PATCH 07/17] fix(release): format digest readback across buildx versions --- scripts/runbooks/prepare-verified-release.py | 2 +- scripts/test/verified-release-package-contract.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/runbooks/prepare-verified-release.py b/scripts/runbooks/prepare-verified-release.py index 804d063c3..d728b5e04 100644 --- a/scripts/runbooks/prepare-verified-release.py +++ b/scripts/runbooks/prepare-verified-release.py @@ -124,7 +124,7 @@ def verify_published_set(root, evidence, sha, run_id): image, digest = service["image_ref"].split("@", 1) for tag in service["tags"]: actual = command([os.environ.get("DOCKER_BIN", "docker"), "buildx", "imagetools", "inspect", - f"{image}:{tag}", "--format", "{{.Manifest.Digest}}"]).decode().strip() + f"{image}:{tag}", "--format", '{{printf "%s" .Manifest.Digest}}']).decode().strip() if actual != digest: raise ValueError(f"Published tag readback mismatch: {service['name']}:{tag}") diff --git a/scripts/test/verified-release-package-contract.py b/scripts/test/verified-release-package-contract.py index 298b3806a..6beeb8a84 100644 --- a/scripts/test/verified-release-package-contract.py +++ b/scripts/test/verified-release-package-contract.py @@ -133,6 +133,7 @@ def test_verifier_requires_successful_matching_main_and_read_only_command(self): self.assertEqual(command.call_count, 18) self.assertTrue(all('--execute' not in call.args[0] for call in command.call_args_list)) self.assertEqual(command.call_args.args[0][1:4], ['buildx', 'imagetools', 'inspect']) + self.assertEqual(command.call_args.args[0][-1], '{{printf "%s" .Manifest.Digest}}') run['head_sha'] = 'b' * 40 with patch.object(module, 'command', return_value=json.dumps(run).encode()) as command: with self.assertRaises(ValueError): module.verify_published_set(self.source, self.evidence, self.sha, '123') From 06f6696d30769d1a457c34783818dfd2dfc68f32 Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 03:40:41 -0700 Subject: [PATCH 08/17] test(acceptance): capture redacted authentication failure codes --- scripts/test/browser-acceptance.mjs | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/scripts/test/browser-acceptance.mjs b/scripts/test/browser-acceptance.mjs index 76680b924..2ce88cd27 100644 --- a/scripts/test/browser-acceptance.mjs +++ b/scripts/test/browser-acceptance.mjs @@ -14,6 +14,7 @@ assert.equal(process.env.ACCEPTANCE_DISPOSABLE, 'true', 'Explicit disposable-sta const output = resolve(process.env.ACCEPTANCE_OUTPUT || '.local/browser-acceptance') await mkdir(output, { recursive: true }) const report = { startedAt: new Date().toISOString(), evidence: 'real-local-services', checks: [] } +const authObservations = [] const browser = await chromium.launch({ headless: true, ...(process.env.ACCEPTANCE_CHROMIUM ? { executablePath: process.env.ACCEPTANCE_CHROMIUM } : {}), @@ -22,6 +23,22 @@ const timer = setTimeout(() => { console.error('Acceptance exceeded five minutes try { const context = await browser.newContext({ locale: 'en-US' }) const page = await context.newPage() + page.on('response', response => { + if (response.status() !== 401 || authObservations.length >= 20) return + authObservations.push((async () => { + const request = response.request() + const cookie = await request.headerValue('cookie') || '' + const authorization = await request.headerValue('authorization') || '' + const envelope = await response.json().catch(() => ({})) + const code = envelope.code + const safeCode = typeof code === 'number' && Number.isFinite(code) ? code + : typeof code === 'string' && /^[A-Z0-9_:-]{1,80}$/i.test(code) ? code : null + report.httpAuthFailures ||= [] + report.httpAuthFailures.push({ path: new URL(response.url()).pathname, method: request.method(), + status: 401, code: safeCode, accessCookiePresent: /(?:^|;\s*)access_token=/.test(cookie), + bearerPresent: authorization.startsWith('Bearer ') }) + })()) + }) page.setDefaultTimeout(30_000) await login(page, consoleOrigin) await page.waitForURL(`${consoleOrigin}/problemset`) @@ -123,6 +140,7 @@ try { report.failure = failure.replaceAll(/eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+/g, '[REDACTED]').slice(0, 2000) process.exitCode = 1 } finally { + await Promise.allSettled(authObservations) report.finishedAt = new Date().toISOString() await writeFile(resolve(output, 'report.json'), `${JSON.stringify(report, null, 2)}\n`, { mode: 0o600 }) console.log(JSON.stringify(report)) From 9efd63c13ea3ef2b47e703724895983553355783 Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 03:42:23 -0700 Subject: [PATCH 09/17] test(acceptance): verify protected sessions and bounded auth replay --- scripts/ci/business-acceptance.sh | 22 ++++++++++++++++++++++ scripts/test/browser-acceptance.mjs | 25 ++++++++++++++++++++++++- 2 files changed, 46 insertions(+), 1 deletion(-) diff --git a/scripts/ci/business-acceptance.sh b/scripts/ci/business-acceptance.sh index d6352200c..4d16ce4d0 100644 --- a/scripts/ci/business-acceptance.sh +++ b/scripts/ci/business-acceptance.sh @@ -105,6 +105,28 @@ phase=launch # Follow the documented second startup: migrations are idempotent, and the # supported DEV-LOCAL App seed adapter runs only when migrations are enabled. ./scripts/dev/up.sh --scope full-stack --skip-infra --skip-install --rebuild > .local/acceptance/private-launch.log 2>&1 +phase=auth-environment +node --input-type=module <<'JS' +import { execFileSync } from 'node:child_process' +try { + const records = JSON.parse(execFileSync('pm2', ['jlist'], { encoding: 'utf8', maxBuffer: 16 * 1024 * 1024 })) + const auth = records.find(record => record.name === 'ulticode-auth')?.pm2_env + const app = records.find(record => record.name === 'ulticode-app')?.pm2_env + if (!auth || !app) throw new Error('Missing disposable owners') + const checks = { + secretPresentAndEqual: typeof auth.JWT_SECRET === 'string' && auth.JWT_SECRET.length >= 32 && auth.JWT_SECRET === app.JWT_SECRET, + issuerEqual: (auth.JWT_ISSUER || 'ulticode-auth') === (app.JWT_EXPECTED_ISSUER || 'ulticode-auth'), + audienceEqual: (auth.JWT_AUDIENCE || 'ulticode-api') === (app.JWT_EXPECTED_AUDIENCE || 'ulticode-api'), + signingAlgorithmAllowed: (app.JWT_ALLOWED_ALGORITHMS || 'HS256,RS256').split(',').map(value => value.trim()) + .includes(auth.JWT_RSA_ENABLED === 'true' ? 'RS256' : 'HS256'), + } + console.log(JSON.stringify({ jwtEnvironmentConsistency: checks })) + if (!Object.values(checks).every(Boolean)) process.exitCode = 1 +} catch { + console.error('Cannot inspect disposable Auth/App environment consistency') + process.exitCode = 1 +} +JS phase=fixtures node scripts/test/acceptance-fixtures.mjs phase=browser-base diff --git a/scripts/test/browser-acceptance.mjs b/scripts/test/browser-acceptance.mjs index 2ce88cd27..22e92fdeb 100644 --- a/scripts/test/browser-acceptance.mjs +++ b/scripts/test/browser-acceptance.mjs @@ -24,6 +24,15 @@ try { const context = await browser.newContext({ locale: 'en-US' }) const page = await context.newPage() page.on('response', response => { + const path = new URL(response.url()).pathname + if (response.request().method() === 'POST' && /\/problems\/\d+\/submissions$/.test(path)) { + report.submissionAttemptStatuses ||= [] + report.submissionAttemptStatuses.push(response.status()) + } + if (path === '/api/auth/refresh') { + report.authRefreshStatuses ||= [] + report.authRefreshStatuses.push(response.status()) + } if (response.status() !== 401 || authObservations.length >= 20) return authObservations.push((async () => { const request = response.request() @@ -43,6 +52,16 @@ try { await login(page, consoleOrigin) await page.waitForURL(`${consoleOrigin}/problemset`) report.checks.push({ name: 'console-browser-login', passed: true }) + report.protectedSessionPreflight = [] + for (const path of ['/api/auth/me', '/api/submissions']) { + const response = await page.request.get(`${consoleOrigin}${path}`) + const envelope = await response.json().catch(() => ({})) + report.protectedSessionPreflight.push({ path, status: response.status(), + code: typeof envelope.code === 'number' ? envelope.code : null, + accessCookiePresent: (await context.cookies(consoleOrigin)).some(cookie => cookie.name === 'access_token') }) + assert(response.ok(), `Protected session preflight ${path} HTTP ${response.status()}`) + } + report.checks.push({ name: 'auth-and-app-protected-session', passed: true }) await page.goto(`${consoleOrigin}/problems/two-sum/description`) await page.getByRole('button', { name: 'Select programming language' }).click() @@ -56,11 +75,15 @@ try { report.checks.push({ name: 'problem-browser-editor', passed: true }) const submitted = page.waitForResponse(response => response.request().method() === 'POST' - && /\/problems\/\d+\/submissions$/.test(new URL(response.url()).pathname)) + && /\/problems\/\d+\/submissions$/.test(new URL(response.url()).pathname) + && response.status() !== 401) await page.getByRole('button', { name: 'Submit Solution', exact: true }).click() const response = await submitted assert.equal(response.request().postDataJSON().code, code, 'Editor must submit the exact fixture') assert(response.ok(), `Submission HTTP ${response.status()}`) + if (report.submissionAttemptStatuses?.includes(401)) { + assert(report.authRefreshStatuses?.includes(200), 'A 401 retry requires the real successful Auth refresh flow') + } const envelope = await response.json() assert(envelope.data?.id, 'Real submission response must contain its ID') const id = envelope.data.id From b7ab7323a0e3287333507fca3c77fabe1cb77a4a Mon Sep 17 00:00:00 2001 From: DavidHLP <144919470+DavidHLP@users.noreply.github.com> Date: Tue, 27 Oct 2026 04:03:04 -0700 Subject: [PATCH 10/17] fix(auth): refresh CSRF header before authenticated replay --- .../auth-refresh.integration.spec.ts | 32 +++++++++++++++++++ .../auth-core/src/axiosCsrfInterceptor.ts | 4 +++ 2 files changed, 36 insertions(+) diff --git a/packages/auth-core/src/__tests__/auth-refresh.integration.spec.ts b/packages/auth-core/src/__tests__/auth-refresh.integration.spec.ts index d127ed777..e81eff4ad 100644 --- a/packages/auth-core/src/__tests__/auth-refresh.integration.spec.ts +++ b/packages/auth-core/src/__tests__/auth-refresh.integration.spec.ts @@ -106,6 +106,38 @@ describe("auth-refresh integration — T6 regression (10-minute hang)", () => { expect(onFailure).not.toHaveBeenCalled(); }); + it("replays a POST with the rotated CSRF header and unchanged account/body", async () => { + vi.mocked(rawAxios.post).mockResolvedValueOnce({ + data: { code: 0, data: { csrfToken: "rotated-csrf" } }, + }); + const payload = { language: "python", code: "synthetic-fixture" }; + const error = make401Error(); + error.config = { + ...error.config, + method: "post", + url: "/problems/1/submissions", + data: payload, + headers: { + "X-CSRF-Token": "stale-csrf-from-login", + "X-Session-Account": "account-fixture", + }, + } as InternalAxiosRequestConfig; + vi.mocked(rawAxios.request).mockImplementationOnce(async config => { + // A server enforcing the rotated cookie would reject the stale header. + expect(config.headers?.["X-CSRF-Token"]).toBe("rotated-csrf"); + expect(config.headers?.["X-Session-Account"]).toBe("account-fixture"); + expect(config.data).toBe(payload); + return { status: 200, data: { id: "submission-fixture" } } as never; + }); + + await errorInterceptor(error); + + expect(rawAxios.post).toHaveBeenCalledTimes(1); + expect(rawAxios.request).toHaveBeenCalledTimes(1); + expect(csrf.getToken()).toBe("rotated-csrf"); + expect(onFailure).not.toHaveBeenCalled(); + }); + it("refresh itself fails (refresh cookie expired, 30+ days idle): user is logged out, no loop", async () => { vi.mocked(rawAxios.post).mockRejectedValue(new Error("refresh 401")); diff --git a/packages/auth-core/src/axiosCsrfInterceptor.ts b/packages/auth-core/src/axiosCsrfInterceptor.ts index 75f459bc2..4dd02c3f0 100644 --- a/packages/auth-core/src/axiosCsrfInterceptor.ts +++ b/packages/auth-core/src/axiosCsrfInterceptor.ts @@ -100,6 +100,10 @@ export function createCsrfAxiosInterceptor( }; try { await refreshAccessToken(); + // Refresh rotates the double-submit cookie too. rawAxios deliberately + // bypasses request interceptors, so replace the stale CSRF header before + // replaying a mutation while preserving its body and account binding. + requestInterceptor(config); // Replay via rawAxios — it has `withCredentials: true` so the // freshly-set HttpOnly access cookie is sent, AND it has no // interceptors so the replay cannot re-enter this 401 branch From 7db26ba6e9fae0217da4383d997519a147ff3b0c Mon Sep 17 00:00:00 2001 From: DavidHLP <144919470+DavidHLP@users.noreply.github.com> Date: Tue, 27 Oct 2026 04:10:29 -0700 Subject: [PATCH 11/17] fix(acceptance): type replay headers and bound PM2 sockets --- .../src/__tests__/auth-refresh.integration.spec.ts | 5 +++-- scripts/ci/business-acceptance.sh | 6 +++++- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/packages/auth-core/src/__tests__/auth-refresh.integration.spec.ts b/packages/auth-core/src/__tests__/auth-refresh.integration.spec.ts index e81eff4ad..5505a83df 100644 --- a/packages/auth-core/src/__tests__/auth-refresh.integration.spec.ts +++ b/packages/auth-core/src/__tests__/auth-refresh.integration.spec.ts @@ -1,4 +1,5 @@ import { describe, it, expect, beforeEach, vi, afterEach } from "vitest"; +import { AxiosHeaders } from "axios"; import type { AxiosError, InternalAxiosRequestConfig } from "axios"; import { createCsrfAxiosInterceptor } from "../axiosCsrfInterceptor"; import { createCsrfTokenManager, type CsrfTokenManager } from "../csrf"; @@ -117,10 +118,10 @@ describe("auth-refresh integration — T6 regression (10-minute hang)", () => { method: "post", url: "/problems/1/submissions", data: payload, - headers: { + headers: new AxiosHeaders({ "X-CSRF-Token": "stale-csrf-from-login", "X-Session-Account": "account-fixture", - }, + }), } as InternalAxiosRequestConfig; vi.mocked(rawAxios.request).mockImplementationOnce(async config => { // A server enforcing the rotated cookie would reject the stale header. diff --git a/scripts/ci/business-acceptance.sh b/scripts/ci/business-acceptance.sh index 4d16ce4d0..eefcb597b 100644 --- a/scripts/ci/business-acceptance.sh +++ b/scripts/ci/business-acceptance.sh @@ -18,7 +18,11 @@ for port in [23306, 26379, 28848, 29848, 9000, 9001, 9002, 9003, 9004, 9101, 910 except OSError: raise SystemExit(f'Acceptance port {port} is occupied; no existing stack was changed') PY export COMPOSE_PROJECT_NAME="ulticode-acceptance-$(date +%s)-$$" -export PM2_HOME="$ROOT/.local/acceptance/pm2" +# Unix socket paths are bounded; nested worktrees must not lengthen PM2's +# private socket names beyond the OS limit. Keep a local recovery pointer. +PM2_HOME="$(mktemp -d /tmp/ulticode-acceptance-pm2.XXXXXXXX)" +export PM2_HOME +printf '%s\n' "$PM2_HOME" > .local/acceptance/pm2-home export ENV_FILE="$ROOT/.env" [[ -z "$(docker ps -aq --filter "label=com.docker.compose.project=$COMPOSE_PROJECT_NAME")" ]] [[ -z "$(docker volume ls -q --filter "name=${COMPOSE_PROJECT_NAME}_")" ]] From 06afe9f58c535c4bcbe9089493abd96e497e0a69 Mon Sep 17 00:00:00 2001 From: DavidHLP <144919470+DavidHLP@users.noreply.github.com> Date: Tue, 27 Oct 2026 04:14:41 -0700 Subject: [PATCH 12/17] docs: record mutation replay and acceptance fixes --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6bc81c9fe..a158db21f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ No next release has been published from this development branch. - Clarify release-pinned installation versus existing-environment upgrades. - Add selected real business/migration acceptance to the CI summary gate. +- Reattach the rotated CSRF token when replaying an authenticated mutation after + access-token refresh; preserve the request body and account binding. +- Keep disposable PM2 socket paths short when acceptance runs in nested worktrees. - Prepare repeatable eight-asset packages only from complete verified image evidence; external tag/Release publication still requires separate human authorization. From 73f2352738600d5646ea9bb143a5c1d70aae276b Mon Sep 17 00:00:00 2001 From: DavidHLP <144919470+DavidHLP@users.noreply.github.com> Date: Tue, 27 Oct 2026 04:36:42 -0700 Subject: [PATCH 13/17] test(acceptance): refresh fixture session after permission changes --- scripts/test/acceptance-fixtures.mjs | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/scripts/test/acceptance-fixtures.mjs b/scripts/test/acceptance-fixtures.mjs index 929de99b2..0aa95e96d 100644 --- a/scripts/test/acceptance-fixtures.mjs +++ b/scripts/test/acceptance-fixtures.mjs @@ -17,7 +17,7 @@ try { assert(meResponse.ok(), 'Manager identity read failed') const me = (await meResponse.json()).data assert.equal(me.user.role, 'SUPER_ADMIN', 'Fixture may grant only the generated superadmin') - const csrf = (await admin.storageState()).cookies.find(cookie => cookie.name === 'csrf_token') + let csrf = (await admin.storageState()).cookies.find(cookie => cookie.name === 'csrf_token') assert(csrf?.value) const permissions = [ ['READ', 'PROBLEM'], ['CREATE', 'PROBLEM'], ['UPDATE', 'PROBLEM'], @@ -31,12 +31,34 @@ try { assert(response.ok(), `Fixture permission ${action}:${resource} rejected`) assert.equal((await response.json()).code, 0) } + // Permission writes advance authorization state. Obtain a real fresh session + // before crossing into Management, and use the rotated double-submit token. + csrf = (await admin.storageState()).cookies.find(cookie => cookie.name === 'csrf_token') + assert(csrf?.value) + const refreshed = await admin.post('/api/auth/refresh', { + headers: { 'X-CSRF-Token': decodeURIComponent(csrf.value) }, + }) + const refreshBody = await refreshed.json().catch(() => ({})) + console.log(JSON.stringify({ fixtureStep: 'manager-session-refresh', status: refreshed.status(), + code: typeof refreshBody.code === 'number' ? refreshBody.code : null })) + assert(refreshed.ok() && refreshBody.code === 0, 'Manager session refresh after permission writes failed') + csrf = (await admin.storageState()).cookies.find(cookie => cookie.name === 'csrf_token') + assert(csrf?.value) + const refreshedMe = await admin.get('/api/auth/me') + assert(refreshedMe.ok(), 'Refreshed manager identity read failed') + const refreshedIdentity = (await refreshedMe.json()).data + assert.equal(refreshedIdentity.user.id, me.user.id, 'Refresh must preserve the fixture actor') + assert.equal(refreshedIdentity.user.role, 'SUPER_ADMIN') // The canonical DEV-LOCAL seed supplies the published problem, languages and // tags. Judge cases are separate owner data: prepare them explicitly through // the real Management API, with server-generated IDs and structured inputs. const descriptionResponse = await admin.get('/api/admin/problems/1/description') - assert(descriptionResponse.ok(), 'Canonical problem seed is missing') - const description = (await descriptionResponse.json()).data + const descriptionBody = await descriptionResponse.json().catch(() => ({})) + const descriptionCode = typeof descriptionBody.code === 'number' ? descriptionBody.code : null + console.log(JSON.stringify({ fixtureStep: 'manager-seed-description', status: descriptionResponse.status(), code: descriptionCode })) + assert(descriptionResponse.ok() && descriptionCode === 0, + `Manager seed description HTTP ${descriptionResponse.status()}, code ${descriptionCode}`) + const description = descriptionBody.data assert.equal(description.slug, 'two-sum') assert.equal(description.isPublished, true) assert(description.tags.length > 0, 'CRUD tag roundtrip requires seeded tags') From e9f532db4ee66906f7183fa90f73b15b198b6992 Mon Sep 17 00:00:00 2001 From: DavidHLP <144919470+DavidHLP@users.noreply.github.com> Date: Tue, 27 Oct 2026 05:06:13 -0700 Subject: [PATCH 14/17] test(acceptance): diagnose owner failures without publishing secrets --- scripts/ci/acceptance_report.py | 70 +++++++++++++++++++ scripts/ci/business-acceptance.sh | 16 +---- scripts/test/browser-acceptance.mjs | 12 +++- scripts/test/browser-auth-diagnostics.mjs | 43 ++++++++++++ .../test/browser-auth-diagnostics.test.mjs | 31 ++++++++ scripts/test/cicd-contract.sh | 2 + scripts/test/test_acceptance_report.py | 54 ++++++++++++++ 7 files changed, 214 insertions(+), 14 deletions(-) create mode 100644 scripts/ci/acceptance_report.py create mode 100644 scripts/test/browser-auth-diagnostics.mjs create mode 100644 scripts/test/browser-auth-diagnostics.test.mjs create mode 100644 scripts/test/test_acceptance_report.py diff --git a/scripts/ci/acceptance_report.py b/scripts/ci/acceptance_report.py new file mode 100644 index 000000000..dff3fb2b8 --- /dev/null +++ b/scripts/ci/acceptance_report.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +"""Publish only bounded public exception types/source coordinates from test logs.""" +import json +import os +import re +from pathlib import Path + +LIMIT = 1_000_000 +KINDS = re.compile(r"\b(?:org|com|java|javax|jakarta|io)\.(?:[A-Za-z_$][A-Za-z0-9_$]*\.)*[A-Za-z_$][A-Za-z0-9_$]*(?:Exception|Error)\b") +FRAMES = re.compile(r"\bat (com\.ulticode\.[A-Za-z0-9_.$]+)\(([A-Za-z0-9_$]+\.java:[0-9]+)\)") +PHASES = {"environment", "sandbox", "fresh-migrations", "backfill-and-cutover", "launch", + "auth-environment", "fixtures", "browser-base", "browser-business", "complete"} + + +def summary(paths): + kinds, frames = set(), [] + for path in paths: + if path.is_symlink() or not path.is_file(): + continue + try: + with path.open("rb") as stream: + stream.seek(max(0, path.stat().st_size - LIMIT)) + text = stream.read(LIMIT).decode("utf-8", errors="replace") + except OSError: + continue + kinds.update(KINDS.findall(text)) + for symbol, location in FRAMES.findall(text): + frame = {"symbol": symbol, "location": location} + if frame not in frames: + frames.append(frame) + return {"exception_classes": sorted(kinds)[:20], "public_source_frames": frames[-16:]} + + +def build_report(root, phase, exit_code, project): + private = summary((root / ".local/acceptance").glob("private-*.log")) + services = {} + for owner in ("app", "submission", "admin", "auth", "judge"): + item = summary([root / "logs" / f"backend-{owner}.{kind}.log" for kind in ("out", "err")]) + if item["exception_classes"] or item["public_source_frames"]: + services[owner] = item + return { + "evidence": "disposable-real-services", + "phase": phase if phase in PHASES else "unknown", + "passed": exit_code == 0, + "project": project if re.fullmatch(r"ulticode-acceptance-[0-9]{1,20}-[0-9]{1,20}", project) else "unknown", + "raw_logs_published": False, + "exception_classes": private["exception_classes"], + "service_diagnostics": services, + } + + +def main(): + try: + root = Path.cwd() + report = build_report(root, os.environ.get("ACCEPTANCE_PHASE", "unknown"), + int(os.environ.get("ACCEPTANCE_EXIT", "1")), + os.environ.get("COMPOSE_PROJECT_NAME", "")) + output = root / ".local/acceptance/installation.json" + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(json.dumps(report, indent=2) + "\n") + if not report["passed"]: + print(json.dumps({"acceptanceFailureDiagnostics": report}, separators=(",", ":"))) + except (OSError, ValueError): + print("Acceptance report could not be written; private logs were not printed") + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/ci/business-acceptance.sh b/scripts/ci/business-acceptance.sh index eefcb597b..065e81ee4 100644 --- a/scripts/ci/business-acceptance.sh +++ b/scripts/ci/business-acceptance.sh @@ -37,19 +37,9 @@ finish() { # Report generation must never prevent cleanup of our disposable processes. set +e export ACCEPTANCE_PHASE="$phase" ACCEPTANCE_EXIT="$status" - python3 - <<'PY' -import json,os,re -from pathlib import Path -kinds=set() -for path in Path('.local/acceptance').glob('private-*.log'): - text=path.read_text(errors='replace')[-1000000:] - kinds.update(re.findall(r'\b(?:org|com|java|javax|jakarta|io)\.(?:[A-Za-z_$][\w$]*\.)*[A-Za-z_$][\w$]*(?:Exception|Error)\b',text)) -Path('.local/acceptance/installation.json').write_text(json.dumps({ - 'evidence':'disposable-real-services', 'phase':os.environ['ACCEPTANCE_PHASE'], - 'passed':os.environ['ACCEPTANCE_EXIT']=='0', 'project':os.environ['COMPOSE_PROJECT_NAME'], - 'raw_logs_published':False, 'exception_classes':sorted(kinds)[:20], -},indent=2)+'\n') -PY + python3 scripts/ci/acceptance_report.py + report_status=$? + if [[ "$status" == 0 && "$report_status" != 0 ]]; then status="$report_status"; fi if [[ "$prepared" == true ]]; then # Stop only this newly created project/home. Keep volumes for local diagnosis; # hosted CI runners dispose of their own disks after the job. diff --git a/scripts/test/browser-acceptance.mjs b/scripts/test/browser-acceptance.mjs index 22e92fdeb..d514a9c70 100644 --- a/scripts/test/browser-acceptance.mjs +++ b/scripts/test/browser-acceptance.mjs @@ -4,6 +4,7 @@ import assert from 'node:assert/strict' import { mkdir, writeFile } from 'node:fs/promises' import { resolve } from 'node:path' import { chromium } from 'playwright' +import { summarizeAccessCookie } from './browser-auth-diagnostics.mjs' const consoleOrigin = localOrigin(process.env.ACCEPTANCE_CONSOLE_ORIGIN || 'http://127.0.0.1:9002') const managementOrigin = localOrigin(process.env.ACCEPTANCE_MANAGEMENT_ORIGIN || 'http://127.0.0.1:9003') @@ -15,6 +16,8 @@ const output = resolve(process.env.ACCEPTANCE_OUTPUT || '.local/browser-acceptan await mkdir(output, { recursive: true }) const report = { startedAt: new Date().toISOString(), evidence: 'real-local-services', checks: [] } const authObservations = [] +let protectedAccessToken = '' +let protectedActorId = '' const browser = await chromium.launch({ headless: true, ...(process.env.ACCEPTANCE_CHROMIUM ? { executablePath: process.env.ACCEPTANCE_CHROMIUM } : {}), @@ -38,6 +41,7 @@ try { const request = response.request() const cookie = await request.headerValue('cookie') || '' const authorization = await request.headerValue('authorization') || '' + const csrfHeader = await request.headerValue('x-csrf-token') || '' const envelope = await response.json().catch(() => ({})) const code = envelope.code const safeCode = typeof code === 'number' && Number.isFinite(code) ? code @@ -45,7 +49,11 @@ try { report.httpAuthFailures ||= [] report.httpAuthFailures.push({ path: new URL(response.url()).pathname, method: request.method(), status: 401, code: safeCode, accessCookiePresent: /(?:^|;\s*)access_token=/.test(cookie), - bearerPresent: authorization.startsWith('Bearer ') }) + bearerPresent: authorization.startsWith('Bearer '), + tokenChecks: summarizeAccessCookie(cookie, { csrfHeader, preflightToken: protectedAccessToken, + expectedActorId: protectedActorId, secret: process.env.JWT_SECRET, + issuer: process.env.JWT_EXPECTED_ISSUER || process.env.JWT_ISSUER || 'ulticode-auth', + audience: process.env.JWT_EXPECTED_AUDIENCE || process.env.JWT_AUDIENCE || 'ulticode-api' }) }) })()) }) page.setDefaultTimeout(30_000) @@ -56,11 +64,13 @@ try { for (const path of ['/api/auth/me', '/api/submissions']) { const response = await page.request.get(`${consoleOrigin}${path}`) const envelope = await response.json().catch(() => ({})) + if (path === '/api/auth/me' && typeof envelope.data?.user?.id === 'string') protectedActorId = envelope.data.user.id report.protectedSessionPreflight.push({ path, status: response.status(), code: typeof envelope.code === 'number' ? envelope.code : null, accessCookiePresent: (await context.cookies(consoleOrigin)).some(cookie => cookie.name === 'access_token') }) assert(response.ok(), `Protected session preflight ${path} HTTP ${response.status()}`) } + protectedAccessToken = (await context.cookies(consoleOrigin)).find(cookie => cookie.name === 'access_token')?.value || '' report.checks.push({ name: 'auth-and-app-protected-session', passed: true }) await page.goto(`${consoleOrigin}/problems/two-sum/description`) diff --git a/scripts/test/browser-auth-diagnostics.mjs b/scripts/test/browser-auth-diagnostics.mjs new file mode 100644 index 000000000..d90bbe7f4 --- /dev/null +++ b/scripts/test/browser-auth-diagnostics.mjs @@ -0,0 +1,43 @@ +// Diagnostic booleans only; never use this probe to authenticate a request. +import { createHmac, timingSafeEqual } from 'node:crypto' + +export function summarizeAccessCookie(cookieHeader, options = {}) { + const cookie = name => { + const match = cookieHeader.match(new RegExp('(?:^|;\\s*)' + name + '=([^;]*)')) + try { return match ? decodeURIComponent(match[1]) : '' } catch { return '' } + } + const token = cookie('access_token') + const csrf = cookie('csrf_token') + const result = { + tokenPresent: Boolean(token), parsed: false, matchesProtectedPreflight: false, + subjectMatchesProtectedActor: false, hmacApplicable: false, hmacMatchesConfiguredSecret: false, + issuerMatchesExpected: false, audienceMatchesExpected: false, timeClaimsValid: false, + roleIsOrdinaryUser: false, csrfHeaderPresent: Boolean(options.csrfHeader), + csrfCookiePresent: Boolean(csrf), csrfHeaderMatchesCookie: Boolean(csrf) && csrf === options.csrfHeader, + } + try { + if (token.length > 16384) return result + const parts = token.split('.') + if (parts.length !== 3 || !parts.every(part => /^[A-Za-z0-9_-]+$/.test(part))) return result + const header = JSON.parse(Buffer.from(parts[0], 'base64url').toString()) + const claims = JSON.parse(Buffer.from(parts[1], 'base64url').toString()) + result.parsed = true + result.matchesProtectedPreflight = Boolean(options.preflightToken) && token === options.preflightToken + result.subjectMatchesProtectedActor = Boolean(options.expectedActorId) && claims.sub === options.expectedActorId + result.hmacApplicable = header.alg === 'HS256' + if (result.hmacApplicable && typeof options.secret === 'string' && options.secret.length >= 32) { + const expected = createHmac('sha256', options.secret).update(parts[0] + '.' + parts[1]).digest() + const signature = Buffer.from(parts[2], 'base64url') + result.hmacMatchesConfiguredSecret = expected.length === signature.length && timingSafeEqual(expected, signature) + } + result.issuerMatchesExpected = claims.iss === options.issuer + result.audienceMatchesExpected = Array.isArray(claims.aud) ? claims.aud.includes(options.audience) : claims.aud === options.audience + const now = (options.now ?? Date.now()) / 1000 + result.timeClaimsValid = Number.isFinite(claims.iat) && Number.isFinite(claims.exp) + && claims.iat <= now + 30 && claims.exp > now - 30 && claims.exp > claims.iat + result.roleIsOrdinaryUser = claims.role === 'USER' + } catch { + // Intentionally omit parser messages and every original value. + } + return result +} diff --git a/scripts/test/browser-auth-diagnostics.test.mjs b/scripts/test/browser-auth-diagnostics.test.mjs new file mode 100644 index 000000000..42362d14b --- /dev/null +++ b/scripts/test/browser-auth-diagnostics.test.mjs @@ -0,0 +1,31 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import { createHmac } from 'node:crypto' +import { summarizeAccessCookie } from './browser-auth-diagnostics.mjs' + +const secret = 'fixture-private-canary-secret-at-least-32-characters' +const claims = { sub: 'fixture-private-actor', role: 'USER', iss: 'fixture-issuer', aud: 'fixture-api', iat: 1000, exp: 2000 } +const encode = value => Buffer.from(JSON.stringify(value)).toString('base64url') +const data = encode({ alg: 'HS256' }) + '.' + encode(claims) +const token = data + '.' + createHmac('sha256', secret).update(data).digest('base64url') +const cookie = 'access_token=' + token + '; csrf_token=private-csrf-canary' +const options = { secret, preflightToken: token, expectedActorId: claims.sub, issuer: claims.iss, + audience: claims.aud, now: 1500000, csrfHeader: 'private-csrf-canary' } + +test('diagnostic results contain only booleans, never credentials or claims', () => { + const result = summarizeAccessCookie(cookie, options) + assert(Object.values(result).every(value => value === true)) + const encoded = JSON.stringify(result) + for (const value of [secret, token, claims.sub, claims.iss, claims.aud, options.csrfHeader]) assert(!encoded.includes(value)) +}) + +test('wrong signature, expiry, actor or CSRF are reported without authenticating or exposing data', () => { + const result = summarizeAccessCookie(cookie, { ...options, secret: 'another-private-secret-at-least-32-characters', + now: 3000000, expectedActorId: 'other-actor', csrfHeader: 'stale-token' }) + assert.equal(result.hmacMatchesConfiguredSecret, false) + assert.equal(result.timeClaimsValid, false) + assert.equal(result.subjectMatchesProtectedActor, false) + assert.equal(result.csrfHeaderMatchesCookie, false) + assert(Object.values(result).every(value => typeof value === 'boolean')) + assert.equal(summarizeAccessCookie('access_token=malformed-private-canary', options).parsed, false) +}) diff --git a/scripts/test/cicd-contract.sh b/scripts/test/cicd-contract.sh index 8bf286c8b..c525e5e40 100644 --- a/scripts/test/cicd-contract.sh +++ b/scripts/test/cicd-contract.sh @@ -16,3 +16,5 @@ bash "$ROOT_DIR/scripts/test/host-deploy-contract.sh" bash "$ROOT_DIR/scripts/test/deploy-runtime-contract.sh" bash "$ROOT_DIR/scripts/test/ssh-host-identity-contract.sh" python3 "$ROOT_DIR/scripts/test/verified-release-package-contract.py" -v +python3 "$ROOT_DIR/scripts/test/test_acceptance_report.py" -v +node "$ROOT_DIR/scripts/test/browser-auth-diagnostics.test.mjs" diff --git a/scripts/test/test_acceptance_report.py b/scripts/test/test_acceptance_report.py new file mode 100644 index 000000000..7d4d7d377 --- /dev/null +++ b/scripts/test/test_acceptance_report.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +import importlib.util +import json +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +spec = importlib.util.spec_from_file_location("acceptance_report", ROOT / "scripts/ci/acceptance_report.py") +module = importlib.util.module_from_spec(spec) +spec.loader.exec_module(module) + + +class RedactionTests(unittest.TestCase): + def test_only_public_types_and_source_coordinates_are_serialized(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "logs").mkdir() + (root / ".local/acceptance").mkdir(parents=True) + canaries = ["private-password-canary", "eyJprivate.token.signature", "source-body-canary"] + (root / "logs/backend-app.err.log").write_text( + "Authorization: Bearer " + canaries[1] + "\nCookie: access_token=" + canaries[1] + + "\njava.lang.IllegalStateException: DB_PASSWORD=" + canaries[0] + + "\nsource=" + canaries[2] + + "\n at com.ulticode.app.Probe.submit(Probe.java:42)\n") + (root / ".local/acceptance/private-launch.log").write_text( + "org.apache.dubbo.rpc.RpcException: " + canaries[0]) + report = module.build_report(root, "browser-base", 1, "ulticode-acceptance-123-456") + encoded = json.dumps(report) + for value in canaries: + self.assertNotIn(value, encoded) + self.assertFalse(report["passed"]) + self.assertFalse(report["raw_logs_published"]) + self.assertIn("org.apache.dubbo.rpc.RpcException", report["exception_classes"]) + self.assertEqual(report["service_diagnostics"]["app"]["public_source_frames"], + [{"symbol": "com.ulticode.app.Probe.submit", "location": "Probe.java:42"}]) + + def test_symlinks_unrelated_files_and_untrusted_metadata_are_excluded(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "logs").mkdir() + outside = root / "private-input" + outside.write_text("java.lang.OutOfMemoryError: private-canary") + (root / "logs/backend-app.err.log").symlink_to(outside) + (root / "logs/unrelated.log").write_text(outside.read_text()) + report = module.build_report(root, "private-canary", 1, "private-canary") + self.assertEqual(report["service_diagnostics"], {}) + self.assertEqual(report["phase"], "unknown") + self.assertEqual(report["project"], "unknown") + self.assertNotIn("private-canary", json.dumps(report)) + + +if __name__ == "__main__": + unittest.main() From 61d90b7d86bd19b23e46ae262958b30bc03cb7c9 Mon Sep 17 00:00:00 2001 From: DavidHLP <144919470+DavidHLP@users.noreply.github.com> Date: Tue, 27 Oct 2026 05:38:23 -0700 Subject: [PATCH 15/17] test(acceptance): classify redacted RPC failures and construct fixture keys --- scripts/ci/acceptance_report.py | 21 ++++++++++++++++--- .../test/browser-auth-diagnostics.test.mjs | 4 ++-- scripts/test/test_acceptance_report.py | 4 +++- 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/scripts/ci/acceptance_report.py b/scripts/ci/acceptance_report.py index dff3fb2b8..1c4257dce 100644 --- a/scripts/ci/acceptance_report.py +++ b/scripts/ci/acceptance_report.py @@ -13,7 +13,7 @@ def summary(paths): - kinds, frames = set(), [] + kinds, frames, messages = set(), [], [] for path in paths: if path.is_symlink() or not path.is_file(): continue @@ -23,12 +23,27 @@ def summary(paths): text = stream.read(LIMIT).decode("utf-8", errors="replace") except OSError: continue + lines = text.splitlines() + for index, line in enumerate(lines): + if any(marker in line for marker in ("RpcException", "Caused by:", "Last error", "last error", "Exception:", "Error:")): + messages.extend(lines[index:index + 3]) kinds.update(KINDS.findall(text)) for symbol, location in FRAMES.findall(text): frame = {"symbol": symbol, "location": location} if frame not in frames: frames.append(frame) - return {"exception_classes": sorted(kinds)[:20], "public_source_frames": frames[-16:]} + context = "\n".join(messages) + patterns = { + "timeout": r"\b(?:timeout waiting|timed out|deadline exceeded|TimeoutException)\b", + "no_rpc_provider": r"no providers? available|not found exported service|no valid provider", + "connection": r"\b(?:connection refused|connection reset|failed to connect|broken pipe)\b", + "serialization": r"(?:failed|unable|cannot|error)[^\n]{0,80}(?:serialize|deserialize|serialization)|(?:serialize|deserialize|serialization)[^\n]{0,60}(?:failed|error|reject|not allow)|not serializable|unsupported[^\n]{0,40}record", + "authorization": r"\b(?:unauthorized|forbidden|access denied|permission denied)\b", + "tls_identity": r"\b(?:SSLHandshakeException|certificate_unknown|untrusted peer|peer identity mismatch|tls handshake)\b", + "validation_provider": r"\bNoProviderFoundException\b", + } + return {"exception_classes": sorted(kinds)[:20], "public_source_frames": frames[-16:], + "rpc_failure_categories": {name: bool(re.search(pattern, context, re.I)) for name, pattern in patterns.items()}} def build_report(root, phase, exit_code, project): @@ -36,7 +51,7 @@ def build_report(root, phase, exit_code, project): services = {} for owner in ("app", "submission", "admin", "auth", "judge"): item = summary([root / "logs" / f"backend-{owner}.{kind}.log" for kind in ("out", "err")]) - if item["exception_classes"] or item["public_source_frames"]: + if item["exception_classes"] or item["public_source_frames"] or any(item["rpc_failure_categories"].values()): services[owner] = item return { "evidence": "disposable-real-services", diff --git a/scripts/test/browser-auth-diagnostics.test.mjs b/scripts/test/browser-auth-diagnostics.test.mjs index 42362d14b..a0df8782e 100644 --- a/scripts/test/browser-auth-diagnostics.test.mjs +++ b/scripts/test/browser-auth-diagnostics.test.mjs @@ -3,7 +3,7 @@ import assert from 'node:assert/strict' import { createHmac } from 'node:crypto' import { summarizeAccessCookie } from './browser-auth-diagnostics.mjs' -const secret = 'fixture-private-canary-secret-at-least-32-characters' +const secret = 'fixture-key'.repeat(6) const claims = { sub: 'fixture-private-actor', role: 'USER', iss: 'fixture-issuer', aud: 'fixture-api', iat: 1000, exp: 2000 } const encode = value => Buffer.from(JSON.stringify(value)).toString('base64url') const data = encode({ alg: 'HS256' }) + '.' + encode(claims) @@ -20,7 +20,7 @@ test('diagnostic results contain only booleans, never credentials or claims', () }) test('wrong signature, expiry, actor or CSRF are reported without authenticating or exposing data', () => { - const result = summarizeAccessCookie(cookie, { ...options, secret: 'another-private-secret-at-least-32-characters', + const result = summarizeAccessCookie(cookie, { ...options, secret: 'different-key'.repeat(5), now: 3000000, expectedActorId: 'other-actor', csrfHeader: 'stale-token' }) assert.equal(result.hmacMatchesConfiguredSecret, false) assert.equal(result.timeClaimsValid, false) diff --git a/scripts/test/test_acceptance_report.py b/scripts/test/test_acceptance_report.py index 7d4d7d377..0c70ba82a 100644 --- a/scripts/test/test_acceptance_report.py +++ b/scripts/test/test_acceptance_report.py @@ -17,10 +17,11 @@ def test_only_public_types_and_source_coordinates_are_serialized(self): root = Path(directory) (root / "logs").mkdir() (root / ".local/acceptance").mkdir(parents=True) - canaries = ["private-password-canary", "eyJprivate.token.signature", "source-body-canary"] + canaries = ["private-" + "password-canary", ".".join(("eyJprivate", "token", "signature")), "source-body-canary"] (root / "logs/backend-app.err.log").write_text( "Authorization: Bearer " + canaries[1] + "\nCookie: access_token=" + canaries[1] + "\njava.lang.IllegalStateException: DB_PASSWORD=" + canaries[0] + + "\norg.apache.dubbo.rpc.RpcException: Failed to deserialize " + canaries[0] + "\nsource=" + canaries[2] + "\n at com.ulticode.app.Probe.submit(Probe.java:42)\n") (root / ".local/acceptance/private-launch.log").write_text( @@ -30,6 +31,7 @@ def test_only_public_types_and_source_coordinates_are_serialized(self): for value in canaries: self.assertNotIn(value, encoded) self.assertFalse(report["passed"]) + self.assertTrue(report["service_diagnostics"]["app"]["rpc_failure_categories"]["serialization"]) self.assertFalse(report["raw_logs_published"]) self.assertIn("org.apache.dubbo.rpc.RpcException", report["exception_classes"]) self.assertEqual(report["service_diagnostics"]["app"]["public_source_frames"], From 8169584f3a738433fad13ec9675ccf4843a26300 Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 07:23:21 -0700 Subject: [PATCH 16/17] fix: gate owner readiness on RPC discovery and preserve server errors --- CHANGELOG.md | 4 ++ docs/DEVELOPMENT.md | 6 ++ scripts/dev/devstack-manifest-test.sh | 2 +- scripts/dev/devstack-manifest.sh | 1 + scripts/dev/up.sh | 15 ++++- services/admin/pom.xml | 2 +- .../in/web/AdminReadinessController.java | 12 +++- .../in/web/AdminReadinessControllerTest.java | 36 ++++++++++- .../data/repository_corpus_manifest.json | 4 +- .../app/security/AppSecurityConfig.java | 5 ++ .../AppErrorDispatchSecurityTest.java | 61 ++++++++++++++++++ services/app/pom.xml | 2 +- services/auth/pom.xml | 2 +- services/judge/pom.xml | 2 +- services/notification/pom.xml | 2 +- services/platform/rpc-resilience/pom.xml | 5 ++ .../resilience/DubboDiscoveryReadiness.java | 27 ++++++++ services/pom.xml | 14 ++-- services/search/pom.xml | 2 +- services/submission/pom.xml | 2 +- .../SubmissionDiscoveryStartupGate.java | 45 +++++++++++++ .../SubmissionDiscoveryStartupGateTest.java | 64 +++++++++++++++++++ 22 files changed, 291 insertions(+), 24 deletions(-) create mode 100644 services/app/app-web/src/test/java/com/ulticode/app/security/AppErrorDispatchSecurityTest.java create mode 100644 services/platform/rpc-resilience/src/main/java/com/ulticode/rpc/resilience/DubboDiscoveryReadiness.java create mode 100644 services/submission/src/main/java/com/ulticode/submission/config/SubmissionDiscoveryStartupGate.java create mode 100644 services/submission/src/test/java/com/ulticode/submission/config/SubmissionDiscoveryStartupGateTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index a158db21f..7c2230db3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,10 @@ No next release has been published from this development branch. +- Reserve backend owner versions `1.0.5` for the next complete source-matched image set. +- Gate Admin/Submission readiness on required RPC discovery without retrying writes. +- Preserve server errors on internal App error dispatches while ordinary routes remain authenticated. + - Clarify release-pinned installation versus existing-environment upgrades. - Add selected real business/migration acceptance to the CI summary gate. - Reattach the rotated CSRF token when replaying an authenticated mutation after diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index b1e08807f..9b97d281f 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -161,6 +161,12 @@ release-selected/full runs 都必须成功，不能把 skipped/cancelled 当成 才导出完成标记。每个 JVM 有 heap 上限；不会并行启动多个本机 acceptance stack。 不使用生产 secret、生产数据、mock API 或付费模型。 +HTTP 存活不代表 Dubbo 接口已被消费者发现。Admin readiness 额外检查 ProblemAdminReadPort +与 TestCaseOwnerPort 的实际 consumer invoker；缺失时返回 503。Submission 在两分钟的 +有界发现期限内检查 ProblemFactsPort 与 SubmissionUserReadPort，成功后才输出 launcher +使用的就绪标记；超时则启动失败。检查只读取发现状态，不调用业务 RPC 或重试写入。 +launcher 记录本次启动前日志偏移，旧标记不会被当成本次成功。 + 测试账号仅由 disposable dev bootstrap 与 Auth registration 创建。Manager 的直接权限 通过真实 Auth API 临时授予并过期，不改全局 role policy；Console participant 保持 USER。 随后执行基础与扩展 browser scripts，覆盖 AC/WA、重判、完整 admin create/edit/readback、 diff --git a/scripts/dev/devstack-manifest-test.sh b/scripts/dev/devstack-manifest-test.sh index 15a02013d..d87cddffd 100755 --- a/scripts/dev/devstack-manifest-test.sh +++ b/scripts/dev/devstack-manifest-test.sh @@ -100,7 +100,7 @@ assert_file_not_contains services/admin/src/main/java/com/ulticode/admin/securit [[ "$(devstack_readiness ulticode-9004)" == "http|9004|/landing/" ]] [[ "$(devstack_readiness_banner ulticode-judge)" == "Started BackendJudgeApplication" ]] [[ "$(devstack_readiness_banner ulticode-core)" == "Started CoreApplication" ]] -[[ -z "$(devstack_readiness_banner ulticode-submission)" ]] +[[ "$(devstack_readiness_banner ulticode-submission)" == "Submission projection providers discovered" ]] [[ "$DEVSTACK_SERVICE_READINESS_ATTEMPTS" == 90 ]] [[ "$DEVSTACK_READINESS_INTERVAL_SECONDS" == 2 ]] [[ "$DEVSTACK_BOOTSTRAP_TIMEOUT_SECONDS" == 90 ]] diff --git a/scripts/dev/devstack-manifest.sh b/scripts/dev/devstack-manifest.sh index 07c98a09e..11eb5172f 100755 --- a/scripts/dev/devstack-manifest.sh +++ b/scripts/dev/devstack-manifest.sh @@ -629,6 +629,7 @@ devstack_apply_scope() { devstack_readiness_banner() { case "$1" in + ulticode-submission) printf 'Submission projection providers discovered' ;; ulticode-judge) printf 'Started BackendJudgeApplication' ;; ulticode-core) printf 'Started CoreApplication' ;; *) printf '' ;; diff --git a/scripts/dev/up.sh b/scripts/dev/up.sh index 722783dc8..9d8f36d8b 100755 --- a/scripts/dev/up.sh +++ b/scripts/dev/up.sh @@ -825,6 +825,14 @@ echo "Starting PM2 services: $PM2_APPS" echo "Named scope $DEV_SCOPE: leaving processes of other scopes running (use stop.sh --scope/--all to tear down)." >&2 ;; esac + if [[ ",$PM2_APPS," == *",ulticode-submission,"* ]]; then + mkdir -p "$ROOT_DIR/logs" + if [[ -f "$ROOT_DIR/logs/backend-submission.out.log" ]]; then + wc -c < "$ROOT_DIR/logs/backend-submission.out.log" > "$ROOT_DIR/logs/.submission-ready-offset" + else + echo 0 > "$ROOT_DIR/logs/.submission-ready-offset" + fi + fi if [[ ",$PM2_APPS," == *",ulticode-judge,"* && ",$PM2_APPS," == *",ulticode-app,"* ]]; then first="${PM2_APPS//ulticode-judge,/}" first="${first//,ulticode-judge/}" @@ -884,12 +892,13 @@ check_pm2_online() { # (pid/status 在 JVM 完全起来之前就可能就位)。 # ecosystem.config.cjs 将 judge 日志写到 logs/backend-judge.out.log, # 不是 PM2 默认目录 (此前 grep 错路径导致 judge 就绪检查永远失败)。 - log="$ROOT_DIR/logs/backend-judge.out.log" + local owner="${app#ulticode-}" + log="$ROOT_DIR/logs/backend-$owner.out.log" # 只搜本次启动 (步骤 6 记录偏移) 之后的新内容, 忽略历史 banner, # 否则慢启动/重启中的 judge 会被上一次运行的旧 banner 误判就绪。 local offset=0 - if [[ -f "$ROOT_DIR/logs/.judge-ready-offset" ]]; then - offset="$(<"$ROOT_DIR/logs/.judge-ready-offset")" + if [[ -f "$ROOT_DIR/logs/.$owner-ready-offset" ]]; then + offset="$(<"$ROOT_DIR/logs/.$owner-ready-offset")" fi # 不用 grep -q: 它在首个匹配即提前退出, tail 收到 SIGPIPE(141), # 在 set -o pipefail 下管道非零 → 假阴性。改为读完全部输入并丢弃输出。 diff --git a/services/admin/pom.xml b/services/admin/pom.xml index 19822ea7c..532bdde2b 100644 --- a/services/admin/pom.xml +++ b/services/admin/pom.xml @@ -13,7 +13,7 @@ - 1.0.4 + 1.0.5 backend-admin ${service.version.admin} diff --git a/services/admin/src/main/java/com/ulticode/admin/adapter/in/web/AdminReadinessController.java b/services/admin/src/main/java/com/ulticode/admin/adapter/in/web/AdminReadinessController.java index 34b94459e..27750a9d3 100644 --- a/services/admin/src/main/java/com/ulticode/admin/adapter/in/web/AdminReadinessController.java +++ b/services/admin/src/main/java/com/ulticode/admin/adapter/in/web/AdminReadinessController.java @@ -2,6 +2,11 @@ import com.ulticode.common.health.ReadinessChecks; import com.ulticode.common.storage.StorageReadiness; +import com.ulticode.rpc.resilience.DubboDiscoveryReadiness; +import com.ulticode.app.api.service.TestCaseOwnerPort; +import com.ulticode.app.api.service.ProblemAdminReadPort; +import org.apache.dubbo.config.spring.reference.ReferenceBeanManager; +import java.util.Set; import java.util.LinkedHashMap; import java.util.Map; import javax.sql.DataSource; @@ -29,6 +34,7 @@ public class AdminReadinessController { private final DataSource dataSource; private final StringRedisTemplate redisTemplate; private final StorageReadiness storageReadiness; + private final ReferenceBeanManager references; /** * Dependencies resolve through {@link ObjectProvider} so contexts that @@ -38,11 +44,13 @@ public class AdminReadinessController { */ public AdminReadinessController(org.springframework.beans.factory.ObjectProvider dataSourceProvider, org.springframework.beans.factory.ObjectProvider redisTemplateProvider, - org.springframework.beans.factory.ObjectProvider storageReadinessProvider) { + org.springframework.beans.factory.ObjectProvider storageReadinessProvider, + org.springframework.beans.factory.ObjectProvider referenceProvider) { this.dataSource = dataSourceProvider.getIfAvailable(); this.redisTemplate = redisTemplateProvider.getIfAvailable(); // Absent only in contexts that do not carry the shared storage module. this.storageReadiness = storageReadinessProvider.getIfAvailable(); + this.references = referenceProvider.getIfAvailable(); } @GetMapping("/health/ready") @@ -50,6 +58,8 @@ public ResponseEntity> ready() { Map components = new LinkedHashMap<>(); components.put("db", ReadinessChecks.dataSourceUp(dataSource)); components.put("redis", redisUp()); + components.put("problemOwnerRpc", DubboDiscoveryReadiness.available(references, + Set.of(TestCaseOwnerPort.class.getName(), ProblemAdminReadPort.class.getName()))); // Re-probe asynchronously after a runtime failure so health checks can recover an idle instance. if (storageReadiness != null) { storageReadiness.probeIfFailed(); diff --git a/services/admin/src/test/java/com/ulticode/admin/adapter/in/web/AdminReadinessControllerTest.java b/services/admin/src/test/java/com/ulticode/admin/adapter/in/web/AdminReadinessControllerTest.java index 2d4ef50b7..1aee0f661 100644 --- a/services/admin/src/test/java/com/ulticode/admin/adapter/in/web/AdminReadinessControllerTest.java +++ b/services/admin/src/test/java/com/ulticode/admin/adapter/in/web/AdminReadinessControllerTest.java @@ -60,13 +60,43 @@ private static Map components(ResponseEntity return (Map) response.getBody().get("components"); } + + @SuppressWarnings({"unchecked", "rawtypes"}) + private static ObjectProvider rpcProvider(boolean available) { + var manager = mock(org.apache.dubbo.config.spring.reference.ReferenceBeanManager.class); + java.util.List references = new java.util.ArrayList<>(); + for (String name : java.util.List.of(com.ulticode.app.api.service.TestCaseOwnerPort.class.getName(), + com.ulticode.app.api.service.ProblemAdminReadPort.class.getName())) { + var reference = mock(org.apache.dubbo.config.spring.ReferenceBean.class); + var config = mock(org.apache.dubbo.config.ReferenceConfig.class); + var invoker = mock(org.apache.dubbo.rpc.Invoker.class); + when(reference.getServiceInterface()).thenReturn(name); + when(reference.getReferenceConfig()).thenReturn(config); + when(config.getInvoker()).thenReturn(invoker); + when(invoker.isAvailable()).thenReturn(available); + references.add(reference); + } + when(manager.getReferences()).thenReturn(references); + ObjectProvider provider = mock(ObjectProvider.class); + when(provider.getIfAvailable()).thenReturn(manager); + return provider; + } + + @Test + void missingProblemProviderCannotReportReadyDespiteHealthyDatabase() throws Exception { + var controller = new AdminReadinessController(dataSourceProvider(), redisProvider(), + readinessProvider(null), rpcProvider(false)); + assertThat(controller.ready().getStatusCode()).isEqualTo(HttpStatus.SERVICE_UNAVAILABLE); + assertThat(components(controller.ready())).containsEntry("problemOwnerRpc", "DOWN"); + } + @Test @DisplayName("reports the verified object store and answers 200") void reportsVerifiedStorage() throws Exception { StorageReadiness readiness = new StorageReadiness(); readiness.markReady(); AdminReadinessController controller = new AdminReadinessController( - dataSourceProvider(), redisProvider(), readinessProvider(readiness)); + dataSourceProvider(), redisProvider(), readinessProvider(readiness), rpcProvider(true)); ResponseEntity> response = controller.ready(); @@ -80,7 +110,7 @@ void runtimeStorageFailureAndRecoveryUpdatesReadiness() throws Exception { StorageReadiness readiness = new StorageReadiness(); readiness.markFailed("runtime outage"); AdminReadinessController controller = new AdminReadinessController( - dataSourceProvider(), redisProvider(), readinessProvider(readiness)); + dataSourceProvider(), redisProvider(), readinessProvider(readiness), rpcProvider(true)); ResponseEntity> failedResponse = controller.ready(); @@ -99,7 +129,7 @@ void runtimeStorageFailureAndRecoveryUpdatesReadiness() throws Exception { @DisplayName("a context without the shared storage module keeps db/redis semantics") void absentStorageModuleIsNotReportedDown() throws Exception { AdminReadinessController controller = new AdminReadinessController( - dataSourceProvider(), redisProvider(), readinessProvider(null)); + dataSourceProvider(), redisProvider(), readinessProvider(null), rpcProvider(true)); ResponseEntity> response = controller.ready(); diff --git a/services/agent/data/repository_corpus_manifest.json b/services/agent/data/repository_corpus_manifest.json index dac896007..c1a9b643a 100644 --- a/services/agent/data/repository_corpus_manifest.json +++ b/services/agent/data/repository_corpus_manifest.json @@ -29,8 +29,8 @@ }, { "doc_id": "repository-development", - "version": "sha256-2fdffd6f45a23a441a4286b11b28df416098c6b5e9da06c9a2142d6f4e9a533c", - "chunk_id": "repository-development:sha256-2fdffd6f45a23a441a4286b11b28df416098c6b5e9da06c9a2142d6f4e9a533c:1", + "version": "sha256-5417a89a3f26d82c4a92e2d643042659427ca9f2956bc300fdcace80b292d35c", + "chunk_id": "repository-development:sha256-5417a89a3f26d82c4a92e2d643042659427ca9f2956bc300fdcace80b292d35c:1", "source_path": "docs/DEVELOPMENT.md", "access_scope": "repository-public", "sample_kind": "real", diff --git a/services/app/app-web/src/main/java/com/ulticode/app/security/AppSecurityConfig.java b/services/app/app-web/src/main/java/com/ulticode/app/security/AppSecurityConfig.java index 1258261e6..006f38eb4 100644 --- a/services/app/app-web/src/main/java/com/ulticode/app/security/AppSecurityConfig.java +++ b/services/app/app-web/src/main/java/com/ulticode/app/security/AppSecurityConfig.java @@ -1,5 +1,6 @@ package com.ulticode.app.security; +import jakarta.servlet.DispatcherType; import com.ulticode.websecurity.csrf.CookieCsrfFilter; import com.ulticode.websecurity.jwt.JwtAuthenticationFilter; import com.ulticode.websecurity.jwt.JwtResourceServerConfiguration; @@ -38,6 +39,10 @@ public SecurityFilterChain securityFilterChain( .exceptionHandling(exception -> exception .authenticationEntryPoint(new HttpStatusEntryPoint(HttpStatus.UNAUTHORIZED))) .authorizeHttpRequests(auth -> auth + // Container error dispatches run after the JWT filter/context + // unwinds. Preserve the original failure instead of issuing a + // misleading 401 that asks clients to refresh and replay writes. + .dispatcherTypeMatchers(DispatcherType.ERROR).permitAll() .requestMatchers("/actuator/health", "/api/v1/health", "/api/v1/app/health", "/api/v1/app/health/ready").permitAll() .requestMatchers("/admin/**").hasAnyRole("ADMIN", "SUPER_ADMIN") diff --git a/services/app/app-web/src/test/java/com/ulticode/app/security/AppErrorDispatchSecurityTest.java b/services/app/app-web/src/test/java/com/ulticode/app/security/AppErrorDispatchSecurityTest.java new file mode 100644 index 000000000..724f1436f --- /dev/null +++ b/services/app/app-web/src/test/java/com/ulticode/app/security/AppErrorDispatchSecurityTest.java @@ -0,0 +1,61 @@ +package com.ulticode.app.security; + +import com.ulticode.websecurity.jwt.AccessTokenVerifier; +import com.ulticode.websecurity.jwt.JwtAuthenticationFilter; +import jakarta.servlet.DispatcherType; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.TestConfiguration; +import org.springframework.boot.webmvc.test.autoconfigure.WebMvcTest; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Import; +import org.springframework.http.ResponseEntity; +import org.springframework.test.context.TestPropertySource; +import org.springframework.test.context.bean.override.mockito.MockitoBean; +import org.springframework.test.web.servlet.MockMvc; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.RestController; + +import static org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.user; +import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get; +import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status; + +@WebMvcTest(controllers = AppErrorDispatchSecurityTest.Probe.class) +@Import({AppSecurityConfig.class, AppErrorDispatchSecurityTest.Filters.class, AppErrorDispatchSecurityTest.Probe.class}) +@TestPropertySource(properties = "jwt.resource-server.enabled=false") +class AppErrorDispatchSecurityTest { + @Autowired private MockMvc mvc; + @MockitoBean private AccessTokenVerifier verifier; + @MockitoBean private org.springframework.security.core.userdetails.UserDetailsService users; + + @Test + void serverErrorDispatchPreservesFailureInsteadOfRequestingRefresh() throws Exception { + // The servlet container starts an ERROR dispatch after the initial + // request/filter context has unwound; OncePerRequestFilter skips it. + mvc.perform(get("/error").with(request -> { request.setDispatcherType(DispatcherType.ERROR); return request; })) + .andExpect(status().isInternalServerError()); + } + + @Test + void ordinaryRequestsRemainProtectedIncludingErrorPath() throws Exception { + mvc.perform(get("/error")).andExpect(status().isUnauthorized()); + mvc.perform(get("/submissions/probe")).andExpect(status().isUnauthorized()); + mvc.perform(get("/submissions/probe").with(user("actor").roles("USER"))) + .andExpect(status().isOk()); + } + + @TestConfiguration + static class Filters { + @Bean JwtAuthenticationFilter jwtFilter(AccessTokenVerifier verifier) { + return new JwtAuthenticationFilter(verifier); + } + } + + @RestController + static class Probe { + @GetMapping("/error") ResponseEntity error() { + return ResponseEntity.internalServerError().build(); + } + @GetMapping("/submissions/probe") String privateEndpoint() { return "ok"; } + } +} diff --git a/services/app/pom.xml b/services/app/pom.xml index b12530d8f..93ae696d8 100644 --- a/services/app/pom.xml +++ b/services/app/pom.xml @@ -13,7 +13,7 @@ - 1.0.4 + 1.0.5 backend-app ${service.version.app} diff --git a/services/auth/pom.xml b/services/auth/pom.xml index 6ebeaa069..24f6be3fc 100644 --- a/services/auth/pom.xml +++ b/services/auth/pom.xml @@ -13,7 +13,7 @@ - 1.0.4 + 1.0.5 backend-auth ${service.version.auth} diff --git a/services/judge/pom.xml b/services/judge/pom.xml index 23c4ecba3..372d4dd7f 100644 --- a/services/judge/pom.xml +++ b/services/judge/pom.xml @@ -15,7 +15,7 @@ - 1.0.4 + 1.0.5 backend-judge diff --git a/services/notification/pom.xml b/services/notification/pom.xml index c5367664a..9a2831a00 100644 --- a/services/notification/pom.xml +++ b/services/notification/pom.xml @@ -14,7 +14,7 @@ - 1.0.4 + 1.0.5 backend-notification ${service.version.notification} diff --git a/services/platform/rpc-resilience/pom.xml b/services/platform/rpc-resilience/pom.xml index 97751ec43..f1912280b 100644 --- a/services/platform/rpc-resilience/pom.xml +++ b/services/platform/rpc-resilience/pom.xml @@ -14,6 +14,11 @@ UltiCode Backend (Dubbo dependency resilience) + + org.apache.dubbo + dubbo-config-spring + ${dubbo.version} + com.ulticode backend-common diff --git a/services/platform/rpc-resilience/src/main/java/com/ulticode/rpc/resilience/DubboDiscoveryReadiness.java b/services/platform/rpc-resilience/src/main/java/com/ulticode/rpc/resilience/DubboDiscoveryReadiness.java new file mode 100644 index 000000000..7ca64aa7a --- /dev/null +++ b/services/platform/rpc-resilience/src/main/java/com/ulticode/rpc/resilience/DubboDiscoveryReadiness.java @@ -0,0 +1,27 @@ +package com.ulticode.rpc.resilience; + +import java.util.HashSet; +import java.util.Set; +import org.apache.dubbo.config.spring.reference.ReferenceBeanManager; + +/** Read-only discovery check: never invokes a query or retries a command. */ +public final class DubboDiscoveryReadiness { + private DubboDiscoveryReadiness() {} + + public static boolean available(ReferenceBeanManager manager, Set requiredInterfaces) { + if (manager == null || requiredInterfaces.isEmpty()) return false; + Set available = new HashSet<>(); + try { + for (var reference : manager.getReferences()) { + if (!requiredInterfaces.contains(reference.getServiceInterface())) continue; + var config = reference.getReferenceConfig(); + if (config != null && config.getInvoker() != null && config.getInvoker().isAvailable()) { + available.add(reference.getServiceInterface()); + } + } + return available.containsAll(requiredInterfaces); + } catch (RuntimeException notInitialized) { + return false; + } + } +} diff --git a/services/pom.xml b/services/pom.xml index 191b744d6..6bc8bd362 100644 --- a/services/pom.xml +++ b/services/pom.xml @@ -52,13 +52,13 @@ - 1.0.4 - 1.0.4 - 1.0.4 - 1.0.4 - 1.0.4 - 1.0.4 - 1.0.4 + 1.0.5 + 1.0.5 + 1.0.5 + 1.0.5 + 1.0.5 + 1.0.5 + 1.0.5 17 2.21.7 diff --git a/services/search/pom.xml b/services/search/pom.xml index f11ebc10c..d65046dd8 100644 --- a/services/search/pom.xml +++ b/services/search/pom.xml @@ -14,7 +14,7 @@ - 1.0.4 + 1.0.5 backend-search diff --git a/services/submission/pom.xml b/services/submission/pom.xml index 8662c30bd..7bff749c8 100644 --- a/services/submission/pom.xml +++ b/services/submission/pom.xml @@ -15,7 +15,7 @@ - 1.0.4 + 1.0.5 backend-submission diff --git a/services/submission/src/main/java/com/ulticode/submission/config/SubmissionDiscoveryStartupGate.java b/services/submission/src/main/java/com/ulticode/submission/config/SubmissionDiscoveryStartupGate.java new file mode 100644 index 000000000..98f014eeb --- /dev/null +++ b/services/submission/src/main/java/com/ulticode/submission/config/SubmissionDiscoveryStartupGate.java @@ -0,0 +1,45 @@ +package com.ulticode.submission.config; + +import com.ulticode.app.api.service.ProblemFactsPort; +import com.ulticode.app.api.service.SubmissionUserReadPort; +import com.ulticode.rpc.resilience.DubboDiscoveryReadiness; +import java.time.Duration; +import java.util.Set; +import org.apache.dubbo.config.spring.reference.ReferenceBeanManager; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.boot.ApplicationArguments; +import org.springframework.boot.ApplicationRunner; +import org.springframework.context.annotation.Profile; +import org.springframework.stereotype.Component; + +/** Do not announce the worker ready before its projection providers are discovered. */ +@lombok.extern.slf4j.Slf4j +@Component +@Profile("!test") +public class SubmissionDiscoveryStartupGate implements ApplicationRunner { + private final ReferenceBeanManager references; + private static final Set REQUIRED = Set.of( + ProblemFactsPort.class.getName(), SubmissionUserReadPort.class.getName()); + + public SubmissionDiscoveryStartupGate(ObjectProvider references) { + this.references = references.getIfAvailable(); + } + + boolean ready() { return DubboDiscoveryReadiness.available(references, REQUIRED); } + + @Override + public void run(ApplicationArguments args) throws InterruptedException { + await(Duration.ofMinutes(2)); + log.info("Submission projection providers discovered"); + } + + void await(Duration timeout) throws InterruptedException { + long deadline = System.nanoTime() + timeout.toNanos(); + while (!ready()) { + if (System.nanoTime() >= deadline) { + throw new IllegalStateException("Submission projection providers were not discovered before readiness deadline"); + } + Thread.sleep(250); + } + } +} diff --git a/services/submission/src/test/java/com/ulticode/submission/config/SubmissionDiscoveryStartupGateTest.java b/services/submission/src/test/java/com/ulticode/submission/config/SubmissionDiscoveryStartupGateTest.java new file mode 100644 index 000000000..8a728a715 --- /dev/null +++ b/services/submission/src/test/java/com/ulticode/submission/config/SubmissionDiscoveryStartupGateTest.java @@ -0,0 +1,64 @@ +package com.ulticode.submission.config; + +import com.ulticode.app.api.service.ProblemFactsPort; +import com.ulticode.app.api.service.SubmissionUserReadPort; +import java.time.Duration; +import java.util.List; +import org.apache.dubbo.config.ReferenceConfig; +import org.apache.dubbo.config.spring.ReferenceBean; +import org.apache.dubbo.config.spring.reference.ReferenceBeanManager; +import org.apache.dubbo.rpc.Invoker; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.ObjectProvider; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.Mockito.*; + +class SubmissionDiscoveryStartupGateTest { + @SuppressWarnings({"rawtypes", "unchecked"}) + private static ReferenceBean reference(Class type, boolean available) { + ReferenceBean bean = mock(ReferenceBean.class); + ReferenceConfig config = mock(ReferenceConfig.class); + Invoker invoker = mock(Invoker.class); + when(bean.getServiceInterface()).thenReturn(type.getName()); + when(bean.getReferenceConfig()).thenReturn(config); + when(config.getInvoker()).thenReturn(invoker); + when(invoker.isAvailable()).thenReturn(available); + return bean; + } + + @SuppressWarnings("unchecked") + private static SubmissionDiscoveryStartupGate gate(ReferenceBeanManager manager) { + ObjectProvider provider = mock(ObjectProvider.class); + when(provider.getIfAvailable()).thenReturn(manager); + return new SubmissionDiscoveryStartupGate(provider); + } + + @Test + void missingOrPartialDiscoveryFailsClosedWithoutInvokingACommand() { + assertThat(gate(null).ready()).isFalse(); + var manager = mock(ReferenceBeanManager.class); + var problem = reference(ProblemFactsPort.class, true); + when(manager.getReferences()).thenReturn(List.of(problem)); + var gate = gate(manager); + assertThat(gate.ready()).isFalse(); + assertThatThrownBy(() -> gate.await(Duration.ZERO)).isInstanceOf(IllegalStateException.class); + verify(problem, never()).getObject(); + } + + @Test + void bothProjectionProvidersMustBeAvailableAndCanRecover() throws Exception { + var manager = mock(ReferenceBeanManager.class); + var problem = reference(ProblemFactsPort.class, false); + var users = reference(SubmissionUserReadPort.class, true); + when(manager.getReferences()).thenReturn(List.of(problem, users)); + var gate = gate(manager); + assertThat(gate.ready()).isFalse(); + when(problem.getReferenceConfig().getInvoker().isAvailable()).thenReturn(true); + assertThat(gate.ready()).isTrue(); + gate.await(Duration.ZERO); + verify(problem, never()).getObject(); + verify(users, never()).getObject(); + } +} From effb087980b3659c4e6e03c27972483a94911afe Mon Sep 17 00:00:00 2001 From: DavidHLP Date: Tue, 27 Oct 2026 07:26:26 -0700 Subject: [PATCH 17/17] test: bind supply-chain contract to reserved owner versions --- scripts/test/supply-chain-contract.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/test/supply-chain-contract.sh b/scripts/test/supply-chain-contract.sh index f999f970b..680739039 100755 --- a/scripts/test/supply-chain-contract.sh +++ b/scripts/test/supply-chain-contract.sh @@ -177,8 +177,8 @@ for service in owner_services: owner_pom = repo / 'services' / service['module'].split('/', 1)[0] / 'pom.xml' owner_properties = ET.parse(owner_pom).getroot().find('m:properties', ns) owner_version = owner_properties.findtext(f'm:{prop}', namespaces=ns) - expected_version = '1.0.4' - assert root_version == owner_version == expected_version, f'{service["name"]}: release version mismatch ({root_version!r} != {owner_version!r})' + expected_version = '1.0.5' + assert root_version == owner_version == expected_version, f'{service["name"]}: release version mismatch (root={root_version!r}, owner={owner_version!r}, expected={expected_version!r})' assert 'actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333' in workflow print('digest-only candidates, common normalized repo and all-service promote gate: PASS') scan = workflow.split(' - name: Scan pushed candidate\n', 1)[1].split(' - name:', 1)[0]