From 0a0f38991cf23f0efdb859812487b4e8efa5c527 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sun, 11 Oct 2026 06:01:02 +0900 Subject: [PATCH 01/12] =?UTF-8?q?design:=20build=20dependencies=20on=20one?= =?UTF-8?q?=20pipeline=20(#797)=20=E2=80=94=20design,=20repro=20and=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...-build-dependencies-one-pipeline-design.md | 729 ++++++++++++++++++ ...1-build-dependencies-one-pipeline-repro.sh | 199 +++++ .agents/docs/README.md | 4 +- 3 files changed, 931 insertions(+), 1 deletion(-) create mode 100644 .agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md create mode 100644 .agents/docs/2026-10-11-build-dependencies-one-pipeline-repro.sh diff --git a/.agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md b/.agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md new file mode 100644 index 00000000..37386684 --- /dev/null +++ b/.agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md @@ -0,0 +1,729 @@ +--- +subject: build-program +status: active +--- + +# 构建依赖的一条管线：配置与产物两轴、包的实例、库接口、兼容系统与工作空间收尾（#797） + +- 日期：2026-10-11。状态：修订 3，待 review。 +- 基线：`main` `2bf99ee1`（2026.10.10.2 加 bootstrap 提交）。 +- 证据：本地 Linux x86_64，gcc 16.1.0，用 `main` 构建的 mcpp（版本串 2026.10.10.1，本文涉及的代码路径与 + `main` 相同）。复现脚本 [2026-10-11-build-dependencies-one-pipeline-repro.sh](2026-10-11-build-dependencies-one-pipeline-repro.sh) + 只打印观察结果，不做断言；下文 R1–R8、W1–W2 引用其中的用例，§11 的性能基线另行记录了命令。 +- 范围：mcpp#797 正文与评论；同一领域读代码与实测发现的缺陷；兼容系统；构建数据库对构建程序的描述； + 工作空间清单的语义。**一个 PR 完成**，不分阶段，不留临时方案。 + +**修订 3 的变化（review 结论）：** +- 修订 2 的五个决策点全部确认：插件由导入判定；有构建程序的包，其构建依赖的库默认编译（`host-module = false` 退出）； + 构建数据库的新集合命名；`[workspace.dependencies]` 的继承规则；`mcpp pack` 拒绝插件根。 +- 构建程序、宿主实例与引擎模块**只**进入 `mcpp emit build-database` 的 S1 文档，这是 mcpp 与 mcppls 生态内部的联动； + `compile_commands.json`（`mcpp build` 写出的与 `--spec compile-commands` 输出的）完全不变，不为其他 LSP 做适配（§7.2）。 +- 新增"同一个库既在 `[dependencies]` 又在 `[build-dependencies]`"的验收矩阵（§3.5），作为本 PR 必须测通的条件。 + 实测基线：同一消费方两张表（R2）失败；工作空间中两个成员分别使用（R2b）、经另一个库传递链接加直接构建依赖（R2c） + 今天可以工作，列为不得退化的保护用例。 +- 自我 review 的修正见 §18。 + +**修订 2 的变化（review 结论）：** +- 移除新键 `lib`。SPEC-004 §5.2 规定已发布描述符中的键不得改名，而 `host-module` 已在已发布描述符中； + 它保留原名，含义精确为"构建程序使用这个依赖的库"，并且在 `[build-dependencies]` 中成为默认（§3.1、§3.2）。 +- 评估插件生态（插件依赖插件与库、插件包自己的构建与测试）。插件单元仍由导入引擎接口判定； + "插件单元经 `[dependencies]` 进入目标即拒绝"一条收窄为"进入**目标配置的实例**即拒绝"：配置沿依赖传播， + 插件依赖插件写在 `[dependencies]` 中，在宿主实例里成立；插件库作为根时以构建程序配置规划（§2.4、§4）。 + 新增 R7（插件包今天无法单独构建）。 +- 兼容系统做成登记表：每条兼容读法有 id、弃用版本、移除下限与规范写法；警告、文档、`mcpp why compat`、 + 测试与 CI 都由这张表派生（§6）。移除下限 `2027.1` 已确认。 +- 构建数据库与 `compile_commands.json` 描述构建程序、宿主实例与引擎模块，为 mcppls 准备基础设施（§7，R8）。修订 3 收窄为只进 S1 文档。 +- 新增工作空间清单语义的完整对照（虚拟根与带根包的根），以及两个新发现：虚拟根旁的 `build.mcpp` + 被静默忽略（W1）、`[workspace.dependencies]` 中的 `features` 被静默丢弃（W2）（§8）。 +- 新增性能基线与预算（§11）、可观测性（§12）。 +- 宿主图从"一个合成包含多个 `bin`"改为"一个合成工作空间，每个构建程序一个合成成员"，每个程序的依赖与 + 链接各自正确，共享包仍只编译一次（§4.3，自我 review 的修正）。 +- 程序缓存身份对 path 包改用 stat 摘要加内容回退，避免今天检查 host 模块条目时观察到的 52 ms（§4.4、§11）。 + +## 0. 结论 + +1. **两条轴。** section 决定**配置**：`[dependencies]`、`[dev-dependencies]` 为目标配置，`[build-dependencies]` + 为构建程序配置（宿主）。边上的请求决定**产物**：库、`tools`、`artifacts`。 + `host-module` 不再决定放置，只表示"构建程序使用这个库"，并且在 `[build-dependencies]` 中默认成立。 +2. **包的实例。** 节点是 `(包, 配置)`。两张表中同名的包得到两个实例；版本统一，特性按配置激活。 +3. **构建程序图。** 每个宿主配置一张图，每个构建程序是其中一个合成成员，依赖是本包 `[build-dependencies]` + 中的库。这张图走现有的子构建与工作空间分组规划、由 ninja 构建：依赖按完整的包编译，`build.mcpp` + 本身也在其中编译，因此 BMI 与使用它的代码由同一个 flag 生成器产生。 + `host_module_compile`、`host_module_store`、单独编译接口单元的路径整体删除。 +4. **插件。** 导入 `mcpp.core` 的单元是插件单元，只能出现在构建程序配置的实例中。 + 插件可以依赖插件与普通库（写在 `[dependencies]`，随配置传播）；插件库作为根时以构建程序配置规划， + 所以 `mcpp build`、`mcpp test` 在插件包里直接可用。 +5. **库接口一个定义。** 接口根（`[lib] path`、Form B 新增的 `lib`、约定路径）的 `export import` 闭包， + 加上已启用特性加入的接口单元，就是公开模块；目标、构建程序、`mcpp pack`、校验读同一个函数。 +6. **兼容系统。** 一张登记表，按日期版本给出弃用与移除下限（`2027.1`），警告、文档、报告、测试、CI 都由它派生。 +7. **可观测性与编辑器支持。** `mcpp emit build-database`（S1）描述构建程序、宿主实例与引擎模块，供 mcppls 使用，`compile_commands.json` 不变；`mcpp why build-programs`、 + `mcpp why compat` 与日志标签让每一步可以解释。 +8. **工作空间。** 修复 R4、R5、W1、W2；虚拟根与带根包的根逐表写明语义。 + +## 1. 问题与证据 + +### 1.1 复现 + +| 编号 | 用例 | 观察（mcpp 2026.10.10.1） | 原因 | +|---|---|---|---| +| R1 | 头文件库的模块包装（toml++ 形状），`[build-dependencies]` 中 `host-module = true` | `fatal error: hl/hl.hpp: No such file or directory` | 宿主编译只有编译器、标准、`base` 与已有 BMI（`host_module_compile.cppm:35`、`:545`），不带提供方的编译上下文 | +| R2 | 同一包在两张表中（include 改为相对路径以绕过 R1） | 目标侧 `hl: error: failed to read compiled module` | 两次声明合并为一条边（`graph.cpp:279`），`hostModule` 被并入（`:300`），`reachable(targetEdgesOnly)` 跳过它（`features.cpp:1304`），`sources` 被清空 | +| R3 | `mid` 的 `[build-dependencies]` 为 `hl` 请求特性 `x`，根的 `[dependencies]` 也依赖 `hl` | 目标编译带上 `-DHL_FEATURE_X` | `aggregatedRequest`（`features.cpp:73`）对所有入边取并集，不分配置 | +| R4 | 带 `[package]` 的根，`[workspace.xlings.overrides]`，根包以 path 依赖成员 | `error: 'sub' states [xlings.overrides] … it is a dependency of this build` | `merge_xlings`（`project.cppm:253`）在每个位置复制 overrides，依赖位置随后被 `sources.cpp:334` 拒绝 | +| R5 | 同上，改为根自己的 `[xlings.overrides]`（位置读法） | 普通 `mcpp build` 根包也出现 `manifest/workspace-position` 警告，随后同样失败 | 依赖位置照样走位置读法并发出警告，而根包自己的条目本来就在这次构建的环境中 | +| R6 | path host 模块所 include 的头文件从 42 改为 43 | 构建程序不重编不重跑，产物仍是 42 | 程序缓存身份只哈希接口文件（`build_program.cppm:1269`） | +| R7 | 一个插件包（`import mcpp;`）作为根构建，即 `examples/08-build-rules/rules-embed` 的形状 | `module 'mcpp' imported but not provided`，构建失败 | 根总是以目标配置规划，引擎模块只在构建程序的编译中存在 | +| R8 | `mcpp emit build-database` 一个带 `build.mcpp`、导入 host 模块的工程 | 翻译单元只有 `main.cpp` 与 `std.cc`；`build.mcpp`、host 模块与引擎模块都不出现 | 构建程序的编译在计划之外。本 PR 只改 S1 文档（§7.2） | +| R2b | 工作空间中 m1 的 `[dependencies]` 与 m2 的 `[build-dependencies]` 使用同一个库 | 两者都工作（`ok: m1 42`、`ok: m2 42`） | 两条边属于不同的消费方，没有合并；保护用例 | +| R2c | 根经库 `lib` 传递地链接 `hl`，同时把 `hl` 写在自己的 `[build-dependencies]` 中 | 工作（`ok: 42 42`） | 同上；保护用例 | +| W1 | 虚拟根（没有 `[package]`）旁放一个会失败的 `build.mcpp` | 没有任何输出，构建成功 | 包表在虚拟根上会被报告（"acts on no package"），`build.mcpp` 不在这份清单里 | +| W2 | `[workspace.dependencies] hl = { path = …, features = ["x"] }`，成员 `hl.workspace = true` | 特性 `x` 没有生效，也没有警告 | `merge_workspace_deps`（`project.cppm:53`）只复制 version、path、git | + +### 1.2 读代码所得 + +- 同一条轴两种读法：`features.cpp:1304` 读 `hostModule || buildOnly`，`scan.cpp:94` 只读 `buildOnly`。 +- docs/05 说"表决定是否进入目标，请求只决定产物"，docs/30 说 `host-module = true` 的依赖写在 `[dependencies]` 里也不进目标； + 代码实现的是后者。`examples/08-build-rules/app` 正是 `[dependencies] … host-module = true` 的写法。 +- 导出面两套定义：普通路径以 `sources` 中所有 `export module` 单元为可导入，接口根只用于警告 + （`validate.cppm:120`、`scan.cpp:951`）；宿主路径要求接口根并只取列出的接口单元（`features.cpp:920`）。 + SPEC-008 的公开模块计算在 `scan.cpp:951-970` 内联，`pack/library_pipeline.cppm:244` 另有一份。 +- Form B 没有 `lib` 键（`xpkg.cppm:258`）。index 中 7 个用 `generated_files` 合成接口单元的包都无法给出接口根。 +- 规则依赖规则的写法（docs/30）让规则的**接口**导入它 `[build-dependencies]` 中的包， + `examples/08-build-rules/rules-tidy` 即是：这是旧模型下唯一可行的写法。 + +## 2. 模型 + +### 2.1 两条轴 + +| 轴 | 由什么决定 | 取值 | 含义 | +|---|---|---|---| +| **配置** | section | 目标：`[dependencies]`、`[dev-dependencies]`；构建程序：`[build-dependencies]` | 依赖的代码为哪个配置编译、链接进谁 | +| **产物** | 边上的请求 | 库（`host-module` 在构建程序配置中控制它）、`tools = [...]`、`artifacts = [...]` | 要依赖包的哪些产物 | + +- 配置沿 `[dependencies]` 边向下传播：一个宿主实例的 `[dependencies]` 也是宿主实例（Cargo、Bazel、Conan 相同）。 +- 一个包的 `[build-dependencies]` 只服务**它自己的**构建程序。 +- `tools` 与配置无关，总是为构建机器构建（SPEC-004 §10.1）；`artifacts` 以消费方的目标构建（§10.3），只出现在目标配置中。 + +### 2.2 包的实例 + +- 节点是 `(包, 配置)`。被目标边到达的包有目标实例，被构建程序配置的库边到达的包有宿主实例，两者可以并存。 +- **一个包一个版本。** 主图的解析已经覆盖两种配置（worklist 会走 `buildDependencies`），锁文件不变；宿主图以主图的解析为钉（§4.3）。 +- **特性按配置激活**（R3，Cargo resolver v2）。构建环境（`feature-xlings`）取两种配置激活结果的并集： + 规则在消费方构建中运行动作所需的载荷，属于消费方的构建环境（SPEC-004 §4.3"两条轴的并集"）。 + +### 2.3 构建程序 + +构建程序是构建程序配置中的一个程序：主文件是 `build.mcpp`（或由规则特性合成的程序），依赖是本包构建程序配置中的库边， +再加上经 `reexport` 传递来的库（provision 图）。它的编译与链接走计划与 ninja；它的**运行**（环境契约、指令协议、 +运行缓存、诊断回放）不变。 + +### 2.4 插件 + +**判定。** 导入 `mcpp.core`（或永久等价写法 `mcpp`，SPEC-007 R9.1）的单元是**插件单元**。引擎接口只在构建程序配置中存在。 + +**规则。** + +| 编号 | 规则 | +|---|---| +| P1 | 插件单元只能出现在构建程序配置的实例中。一个**目标配置的实例**的编译集合中出现插件单元时，规划期拒绝 | +| P2 | 配置沿 `[dependencies]` 传播。插件 A 用到插件 B 或普通库 C，在 A 的 `[dependencies]` 中声明 B、C；A 被消费方的 `[build-dependencies]` 到达时，A、B、C 都是宿主实例 | +| P3 | 一个包的**库**含有插件单元（在当前启用的特性下）时，它作为构建的根（`mcpp build`、`mcpp test`、`-p` 选中的成员、tools 子构建的根）以构建程序配置规划：宿主工具链，`--target` 指向他处时拒绝并写出原因 | +| P4 | 一个包的**程序目标**（`bin`、`app`）的单元导入引擎接口，而该包不是 P3 的插件库时，规划期拒绝：引擎接口属于构建程序，而 `build.mcpp` 才是构建程序 | +| P5 | `mcpp pack` 拒绝以构建程序配置规划的根：插件以源码分发，宿主 BMI 不可移植 | + +**P1 的消息**写出：单元、它所属的包、到达该包的边所在的 section，以及正确的写法。消费方写错 section 时， +指向 `[build-dependencies]`；依赖链中间某个**目标**库错误地依赖了插件时，写出这条链。 + +**为什么由导入判定，而不是声明（例如 Cargo 的 `[lib] proc-macro = true`）。** + +| | 由导入判定（采用） | 包级声明 | +|---|---|---| +| 事实所在 | 单元的导入，一处 | 声明与导入两处，可能不一致 | +| 混合包（`mcpp:plugins`：基础库 `mcpp.plugins` 只导入 `std`，被 `mcpp-embed` 工具使用；插件单元在特性之后） | 按启用的特性自然成立 | 包级声明无法表达；需要逐特性声明 | +| 已发布的插件包 | 不需要改动 | 都要加声明 | +| 与 2026.10.10.1"模块角色取自声明"的原则 | 一致：角色取自单元自己的声明 | 另起一个来源 | +| 根的配置需要在工具链解析之前确定 | 需要一次预扫描（§4.8，§11 有预算） | 加载期即可知 | + +预扫描的代价是确定的、可测量的；声明的代价是一个事实有两个来源。因此采用由导入判定。 + +**生态场景评估。** + +| 场景 | 写法 | 结果 | +|---|---|---| +| S1 第三方插件基于 L2 `mcpp:plugins` | 插件的 `[dependencies.mcpp] plugins = { features = ["plugins-core"] }` | 宿主图中两者都是宿主实例 ✔（旧写法 `[build-dependencies] … host-module = true` 按兼容读法成立，带警告） | +| S2 插件用普通库（toml++、fmt） | 插件的 `[dependencies]` | ✔ 今天不可能（R1） | +| S3 插件用另一个插件，后者有 `tools` | 照常 | ✔ `tools` 不受影响 | +| S4 插件包自己构建、测试 | `mcpp build` / `mcpp test` | ✔ P3；今天失败（R7）。插件的单元测试可以写成普通的 `tests/*.cpp`，导入 `mcpp.core` 与测试工具，不必再借消费方的构建程序 | +| S5 库把插件转交给它的消费方 | 库的 `[build-dependencies] p = { reexport = true }` | ✔ provision 传播不变 | +| S6 两个插件依赖同一个库、特性不同 | — | ✔ 同一宿主图内特性合并（与 Cargo 同一配置内的合并一致） | +| S7 两个消费方的标准不同 | — | ✔ 不同宿主配置组，各一张图 | +| S8 混合包 `mcpp:plugins` | 不变 | ✔ 目标侧只用基础库；构建程序配置启用插件特性 | +| S9 插件包自己有 `build.mcpp`，用另一个插件 | 插件的 `[build-dependencies]` | ✔ 宿主图中递归一层 | +| S10 菱形：消费方的两个构建依赖都依赖 `mcpp:plugins` | — | ✔ 宿主图中一个实例 | +| S11 两个插件导出同名模块 | — | 宿主图规划期报模块冲突（现有规则），写出两个包 | +| S12 目标库误用 `[dependencies]` 依赖插件 | — | P1 拒绝，消息写出依赖链 | + +### 2.5 库接口（SPEC-008 的延伸） + +一个函数 `library_interface(包, 配置, 已启用特性, 模块图)` 给出： + +| 量 | 定义 | +|---|---| +| 接口根 | `[lib] path`；Form B 的 `lib`；都没有时为约定路径 `src/<包名末段>.<模块接口扩展名>`（文件存在时）。至多一个 | +| 公开模块 | 接口根声明的模块及其 `export import` 传递闭包，**加上**已启用特性的 `sources` 中声明主模块接口的单元及其转出闭包 | +| 可导入模块 | 该配置编译集合中的全部导出主模块（可导入而不公开的模块产生 W3，SPEC-008 第一阶段） | +| 发布闭包 | 公开模块接口传递导入的全部单元（SPEC-008 I5） | + +- 特性一项修订 SPEC-008 I2：特性是否启用由消费方决定，`export import` 无法表达"只在特性启用时转出"， + 因此特性加入的接口单元就是该特性的接口。`mcpp:plugins` 的 `rules-*`、`plugins-core` 正是这种形状。 +- 读者全部改用这个函数：目标侧 W3（`scan.cpp`）、构建程序的导入检查、`exports` 断言（`validate.cppm`）、 + `mcpp pack` 的发布闭包（`library_pipeline.cppm`）、`rule_module` 校验、诊断、构建数据库（§8）。 + +### 2.6 行业对照 + +| 工具 | 构建期依赖 | 同包两用 | 管线 | 特性/配置隔离 | 插件 | +|---|---|---|---|---|---| +| Cargo | `[build-dependencies]`、proc-macro | 两次编译 | 同一条，换 target | resolver v2 分开 | `[lib] proc-macro = true`，就是 host 库 | +| Bazel | `cfg = "exec"` | 两次配置 | 同一条 | 每配置一套 flag | exec 配置下的依赖 | +| Conan 2 | `tool_requires` / traits | 两个 context、两个节点 | 同一条，两套 profile | 按 context | 不单独建模 | +| vcpkg / Meson / Zig | `host: true` / `native: true` / `b.graph.host` | 各一份 | 同一条 | 按 triplet / machine / target | 不单独建模 | + +mcpp 多一条约束：BMI 不能跨编译器、标准或语言选项使用。宿主实例必须与 `build.mcpp` 由同一个工具链、同一个标准、 +同一个 flag 生成器编译，这就是 §2.3 让 `build.mcpp` 进入宿主图的原因；也是不采用 #797 A 方案（交给 tools 子构建、 +各自为根）的原因：多个宿主库共享一个依赖时，各自为根会编译并链接多份。 + +## 3. 清单语义与使用场景的变化 + +### 3.1 `host-module`：不加新键 + +- **不加 `lib` 键。** SPEC-004 §5.2：已进入已发布描述符的键禁止改名。`host-module` 已在 index 与 + `mcpp:plugins` 的消费方清单中，改名只会给每个用户制造一次迁移和一段两种拼法并存的时期， + 换来的只是名字更贴切。 +- **含义精确化：** `host-module = true` 表示"构建程序使用这个依赖的库"：它的公开模块可导入，公开头文件可见， + 对象参与链接。名字只说了模块，是历史原因；SPEC-004 §5.2 要求把这类不一致写成规则并注明来源，本文照做。 +- **默认值：** 在 `[build-dependencies]` 中，边没有列出 `tools`、`artifacts` 时，`host-module` 默认为 `true`； + 列出时默认为 `false`，两者都要时写 `host-module = true`（与今天 `mcpp:plugins` 的 + `features = […], host-module = true, tools = ["mcpp-embed"]` 写法相同）。请求声明了 `rule_module` 的特性时， + 隐含 `true`（与今天相同）。`host-module = false` 可以显式退出（只要依赖的目录或载荷时）。 +- 在 `[build-dependencies]` 中写 `host-module = true` 而没有 `tools` 是在陈述默认值，**不给诊断** + （与 `default-features = true` 相同）。 + +### 3.2 使用场景：之前与之后 + +| 场景 | 之前 | 之后 | +|---|---|---| +| 构建程序用一个普通库（toml++、nlohmann.json） | 要写 `host-module = true`；头文件库的模块包装编不过（R1）；Form B 包缺接口根直接拒绝 | `[build-dependencies] tomlplusplus = "3.4.0"`，`import tomlplusplus;` 即可 | +| 构建程序用一个 C 库（cJSON）或带非接口源文件的库 | 不可能：只编译接口单元，没有链接 | 完整编译并链接 | +| 构建依赖只用来取工具（protoc） | `{ tools = ["protoc"] }` | 不变，不编译库 | +| 既要工具又要库（`mcpp:plugins` + `mcpp-embed`） | `host-module = true, tools = [...]` | 不变 | +| 规则特性（`features = ["rules-spirv"]`） | 隐含 `host-module` | 不变 | +| 同一个包既链接进目标，又在构建程序中使用 | 两张表各写一次，构建失败（R2） | 两张表各写一次，两个实例，各自正确 | +| `[dependencies] x = { host-module = true }`（`examples/08-build-rules/app`） | 不进目标，只给构建程序 | 行为不变，警告建议移到 `[build-dependencies]`；`2027.1` 起移除这种读法 | +| 插件依赖插件 | 写在插件的 `[build-dependencies]`（`host-module = true`），由接口导入 | 写在插件的 `[dependencies]`；旧写法行为不变并给出警告 | +| 插件包自己 `mcpp build` / `mcpp test` | 失败（R7） | 以构建程序配置构建与测试 | +| 只为取目录而写的 `[build-dependencies] x = "1.0"`（x 是库），且本包有构建程序 | 不编译 | 编译 x 的宿主实例（与 Cargo 相同）；只要目录时写 `host-module = false`。没有构建程序的包不受影响：宿主图只为构建程序存在 | +| 构建依赖请求的特性 | 泄漏进目标实例（R3） | 只作用于宿主实例；依赖这种泄漏的工程需要在目标边上写出特性 | +| Form B 包声明接口根 | 无法声明 | `lib = "mcpp_generated/fmt_module.cppm"` | + +### 3.3 Form B 的 `lib` 与 `modules` + +- **`lib = "<路径>"`**（新，属于新增而非改名）≡ `[lib] path`。路径规则同 `sources`：相对版本目录，接受 `*/` 包装层。 +- **`modules = { ... }`** ≡ `[modules] exports`：对公开模块集合的**断言**。存在时与 §2.5 派生的公开模块（基础部分）逐项比较； + 不一致时本包构建警告、`--strict` 下错误、作为依赖时不报。它不指定接口根：一个事实一个声明位置。 +- 更旧的引擎把 `lib` 当作未知键警告并跳过（`xpkg.cppm:238`），目标使用不受影响；index 以 `min_mcpp` 门控即可。 + +### 3.4 同一个包写在两张表中 + +两条边、两个实例，**不合并**。同一 section 内的多次声明（无条件与 `[feature-deps.]`、`[target..dependencies]`） +照旧合并请求（`graph.cpp:617`）。docs/05 中 "the ordinary declaration wins … requested of the one edge" 一段改写。 + +### 3.5 同一个库同时用于目标与构建程序：验收矩阵 + +这是本 PR 必须测通的条件（e2e 903）。库 `hl` 是头文件库加模块包装的形状：接口单元的全局模块片段以 +`#include ` 引入自己的头文件，`[build] include_dirs = ["include"]`，带一个特性 `x`。 + +| 编号 | 写法 | 期望 | 今天 | +|---|---|---|---| +| M1 | 同一消费方：`[dependencies] hl` 与 `[build-dependencies] hl`（不写 `host-module`） | 目标导入 `hl`、`build.mcpp` 导入 `hl` 都成功；两个实例，目标实例的 BMI 在目标计划中，宿主实例的在宿主图中 | 失败（R1，绕过 R1 后仍失败：R2） | +| M2 | 同 M1，构建依赖显式写 `host-module = true` | 与 M1 相同（陈述默认值，无诊断） | 失败（R1、R2） | +| M3 | 同 M1，两侧特性不同：目标边 `features = []`，构建依赖边 `features = ["x"]` | 宿主实例有 `x`，目标实例没有（`-DHL_X` 只出现在宿主图的编译中） | 目标实例收到 `x`（R3） | +| M4 | 同 M1，`hl` 来自本地 index 的 Form B 并声明 `lib` | 与 M1 相同；版本只解析一次，`mcpp.lock` 中一条 | 失败（无接口根） | +| M5 | 工作空间：m1 的 `[dependencies] hl`，m2 的 `[build-dependencies] hl` | 两者都工作 | 工作（R2b，绕过 R1 时） | +| M6 | 根经库 `lib` 传递地链接 `hl`，并在 `[build-dependencies]` 中直接写 `hl` | 两者都工作 | 工作（R2c，绕过 R1 时） | +| M7 | 一个依赖包 `d` 的 `build.mcpp` 导入 `hl`（`d` 的构建依赖），而根在 `[dependencies]` 中链接 `hl` | 两者都工作，`hl` 的目标实例只编译一次、宿主实例只编译一次 | 依赖的构建程序走同一条 host 模块路径，受 R1 限制 | +| M8 | M1 在交叉构建中（`--target aarch64-linux-gnu`） | 目标实例为 aarch64 编译，宿主实例为宿主编译，构建程序在宿主上运行 | 失败（R1、R2） | +| M9 | M1 的构建数据库 | S1 文档中 `hl`（目标）与 `hl:host`（宿主）两个集合，参数各自与实际编译一致；`compile_commands.json` 与修订前逐字相同（只有目标单元） | 无宿主集合 | +| M10 | M1 之后编辑 `hl/include/hl/hl.hpp` | 目标实例与宿主实例都重编，构建程序重跑，产物带新值 | 构建程序不重跑（R6） | + +两个实例各自提供模块 `hl`，但处在两次规划（目标计划与宿主图）中，不构成同名模块冲突；依赖缓存的键因配置 +（工具链、profile、特性）不同而不同，两份条目并存。 + +## 4. 引擎架构 + +### 4.1 边模型 + +`DependencyEdge`（`state.cppm:120`）： + +- 删除 `hostModule` 字段；`buildOnly` 改为 `configuration`（`Target | BuildProgram`），由 section 决定并沿 `[dependencies]` 继承 + （今天 `buildOnly` 已是继承的：`graph.cpp:1926`）。 +- 新增 `wantsLibrary`：§3.1 的默认规则在清单规范化时求值，兼容读法也在那里完成，图代码看不到 `host-module` 的历史含义。 +- `recordDependencyEdge` 的合并键加上 `configuration`；`graph.cpp:279-306` 的跨 section 合并删除（R2）。 +- 实例：`reachable` 只看 `configuration`，结果写入 `PackageRoot::targetInstance`；`compilesHere(i)`（`graph.cpp:2031`） + 加上这个条件；`plan.cppm:2288`、`:2631` 与 `scan.cpp:94` 改读 `compilesHere`。**不再修改清单**（删除清空 `sources` 的做法）。 + +### 4.2 特性按配置激活 + +`activeFeaturesByPackage` 分为目标与构建程序两张表；`aggregatedRequest` 只汇总同一配置的入边。编译与 `[features]` +的折叠只用目标表；provision 与 `feature-xlings` 用两表的并集；构建程序表交给宿主图作为输入，宿主图自己的激活结果 +必须与之相同，不同即为内部错误（带两边的集合写入日志）。 + +### 4.3 宿主图 + +**分组键。** 构建程序配置的工具链身份（指纹）与 C++ 标准（所属包的 `cppStandard`）。标准不同即为不同组（e2e 857 判据 B）。 + +**合成工作空间。** 每组一个内存中的虚拟工作空间（不落盘），每个构建程序一个合成成员： + +```toml +# 合成成员 . 的清单（身份在保留命名空间 `_build` 下：`_build..`） +[package] +name = "." +standard = "<组的标准>" +[build] +sources = [] # 明确为空：成员的根目录是消费方包的根目录，不能取到它的 src/** +[dependencies] # 该包构建程序配置的库边，加上 reexport 传来的；路径锚定到该包 +"". = { version = "=<主图解析的版本>" | path = "…", features = [...], default-features = … } +[targets.build-program] +kind = "bin" +main = "build.mcpp" # 或合成的规则程序（写到宿主图工作目录中） +``` + +用工作空间而不是一个多 `bin` 的包：每个程序的依赖、导入可见性与链接各自正确；共享的包在同一配置组的一次规划中 +只出现一次（SPEC-005 R3.3 已有的工作空间分组语义），因此仍只编译一次（e2e 857 判据 A）。 + +**引擎包。** `mcpp` 与 `mcpp.core` 的接口文本写入按版本区分的只读目录 `$MCPP_HOME/engine//modules/` +（`mcpp.cppm`、`mcpp_core.cppm`），作为引擎包。宿主图中有单元导入它时，它是该单元所在包的隐式依赖。依赖缓存把它当作 +不可变来源，全局条目仍是 `build-cache/v1/pkg/_engine/mcpp-build-module@<版本>//`，`mcpp cache list/verify/gc` +看到的不变。源文件目录与 BMI 条目分开，构建数据库指向前者（§8）。目标配置中没有引擎包（P1）。 + +**执行。** 现有子构建入口 `prepare_build(BuildOverrides)`（与 `step6_build_tool`，`features.cpp:1620`，同一个入口）： + +| 覆盖项 | 值 | +|---|---| +| `preloaded_workspace` *(新)* | 合成工作空间（根与成员的清单都在内存中） | +| `target_triple` | `""`（宿主） | +| `inherited_build_program_toolchain` *(新)* | 已解析的工具链对象（共享指针），不从字符串重新解析，保证与今天编译构建程序的工具链相同 | +| `resolution_pins` *(新)* | 主图对闭包中每个包的解析（身份 → 版本或来源）。工具链阶段的程序（SPEC-007 R9.9）在主图解析之前运行，没有钉 | +| `features` | §4.2 的构建程序表 | +| profile | 内部 profile：`opt = 0`、`debug = false`，与今天构建程序的编译相同 | +| 链接 | 今天的"静态助手"策略（musl、MinGW，`build_program.cppm:1258`）由合成成员陈述；Windows 的 UTF-8 清单（#693）由合成成员的资源陈述 | +| `work_dir` | `<工作区>/target/.build-mcpp/host/<组键前 12 位>/`（路径短，mcpp#641） | +| `tool_depth` | 加 1，复用嵌套上限与环检测 | + +**并发。** 配置组可以在多个线程上构建（`ui.cppm:473`）。宿主图的工作目录按组键命名，用 `platform::fs::FileLock` +（阻塞获取，带等待日志）互斥；两个目标配置组的宿主配置相同时共用一张宿主图，后到的一方等待后 ninja 无事可做（e2e 912）。 + +**目标（goal）。** 只构建缓存失效的程序（§4.4）。index 包经依赖缓存全局复用；path 包的产物留在工作区（e2e 857 判据 C、D 的性质不变， +位置变为 `target/.build-mcpp/host/…`）。 + +**宿主实例自己的构建程序**在宿主图的 `prepare_build` 中按同样机制运行（递归一层，§2.4 场景 S9）。 + +**`.mcpp` 源文件**在计划中按 C++ 翻译单元处理，命令带强制语言参数（GCC/Clang 的 `-x c++`，cl 的 `/TP`），与今天相同（`dial.forceCxxLangArgv`）。 + +### 4.4 构建程序：编译交给宿主图，运行不变 + +- `run_build_program_impl` 中的 `compile_binary`（`build_program.cppm:1533` 起：std 模块暂存、`mcpp` 模块、host 模块循环、 + 链接策略、响应文件）由"宿主图产出的程序路径"替代。运行、指令解析、运行缓存、`report_stated_diagnostics`、 + `run_answering_requests` 不变。 +- **缓存身份**（替代 `compilerIdentity`）：构建程序工具链指纹、标准、引擎版本、`build.mcpp` 的内容摘要，以及闭包中每个包的 + `(身份, 版本, 来源摘要, 构建程序配置的特性)`。index 包的来源摘要是版本加修订；path 包用源树的 **stat 摘要** + （路径、大小、修改时间），stat 有变化时再算内容摘要并记录，规则与 ninja 的重启判断相同；源树范围与 tools 库键相同 + （不含带自己 `mcpp.toml` 的子目录，SPEC-004 §10.1）。身份在主图解析后即可算出，**不需要规划宿主图**；全部程序新鲜时 + 宿主图不规划（R6 由此修复：头文件的改动改变 stat 与内容摘要）。 +- 判定为失效时，日志写出**哪一个分量**变了（§12）。 +- **预编译线程**（`target_side.cpp:2080-2130`）删除：同一组失效的程序在一次 ninja 中并行编译。 +- **编译失败**按成员归属到包：`build.mcpp of failed to compile`，附编译器输出、宿主图工作目录与单独复现的命令 + （tools 子构建的 `subContext` 先例）。 +- R9.7（未启用特性提供的模块要写出特性）与 e2e 624（缺失模块要写出键）的诊断移到宿主图扫描阶段，措辞不变。 + +### 4.5 provision + +`Kind::HostModule` 改名为 `Kind::HostLibrary`，由 `configuration == BuildProgram && wantsLibrary` 的边产生； +`reexport` 的不动点（`provisions.cppm:142`）不变。`hostModuleProvidersByConsumer`、`hostModulesByConsumer`、 +`dormantFeaturesByConsumer` 由每个程序的依赖集（合成成员的输入）替代。 + +### 4.6 删除 + +| 删除 | 由谁承担 | +|---|---| +| `src/build/host_module_compile.cppm`（622 行）、`src/build/host_module_store.cppm`（381 行）、`tests/unit/test_host_module_store.cpp` | 宿主图与依赖缓存 | +| `host_module_units`（`features.cpp:920`）及 `step6_host_module_registration` 中的排序、冲突、命名 | 宿主图的模块图与 `library_interface` | +| `buildTimeOnly` 与清空 `sources` | `targetInstance` 与 `compilesHere` | +| 构建程序的预编译线程 | ninja | +| docs/30 "units are otherwise compiled alone, so they import std, mcpp and nothing else" | 不再成立 | + +### 4.7 文件布局 + +`prepare/features.cpp` 现为 2277 行（上限 2500）。宿主图的分组、合成工作空间、身份与子构建放进新文件 +`src/build/prepare/host_graph.cpp`。`library_interface` 是新模块 `mcpp.build.library_interface`：它需要模块图， +而 `mcpp.manifest.types` 不能再增加导入（`toml.cppm:5180` 记录的 GCC 16.1 ICE）。兼容登记表在 `mcpp.pm.compat`（§6）。 + +### 4.8 根的配置（P3） + +工具链解析之前，对根（与 `-p` 选中的成员）当前启用特性下的**库**源文件做一次预扫描，只读导入声明（与主扫描同一个文本扫描器）。 +预扫描结果按文件保存在 `PrepareState` 中，主扫描直接取用，同一文件不读两次。含插件单元则以构建程序配置规划， +状态行写出 `Configuration build-program (host <三元组>, <工具链>)`，日志写出触发的单元。 + +## 5. 诊断 + +按 `src/diag.cppm` 的不变量与已有先例分级。 + +| 情形 | 级别 | 消息要点 | +|---|---|---| +| P1：目标配置的实例中出现插件单元 | 规划期拒绝 | 单元、包、边所在的 section 与依赖链，以及 `[build-dependencies]` 写法 | +| P3：插件库作为根且 `--target` 指向他处 | 规划期拒绝 | 插件在构建机器上运行，`--target` 不适用；写出触发的单元 | +| P4：程序目标导入引擎接口 | 规划期拒绝 | 引擎接口属于构建程序，指向 `build.mcpp` | +| P5：`mcpp pack` 构建程序配置的根 | 拒绝 | 插件以源码分发 | +| 构建程序导入的模块没有依赖提供 | 规划期拒绝（今天已是） | 候选的 `[build-dependencies]` 条目；未启用特性时写出特性（R9.7） | +| 构建程序导入仅作为他人前提到达的模块 | 规划期拒绝（今天已是） | 措辞不变 | +| 构建程序导入可导入但不公开的模块 | 警告 `build/interface`（SPEC-008 W3，今天只对目标） | 与目标侧相同 | +| 显式声明的接口根不存在 | 本包构建：错误；作为依赖：警告 | 只对能修改它的包报错 | +| `exports` / Form B `modules` 与派生公开集合不一致 | 本包构建：警告，`--strict` 下错误；作为依赖：不报 | 列出多出与缺少的名字 | +| `rule_module` 不是该特性的接口单元之一 | 加载期拒绝 | 列出该特性实际提供的模块 | +| 兼容读法 | 警告 `manifest/deprecated`（只对能修改的清单；`--strict` 下错误） | 由登记表生成：读法、规范写法、弃用版本、移除下限（§6） | +| 宿主图中某个程序编译失败 | 构建期错误 | 包、`build.mcpp`、编译器输出、工作目录、复现命令 | +| W1：虚拟根旁的 `build.mcpp` | 警告，`--strict` 下错误（与虚拟根上的包表相同） | 它不属于任何包；写进某个成员 | +| W2：`[workspace.dependencies]` 中的边请求键（§8.2） | 警告，`--strict` 下错误 | 该键属于每个成员自己的边 | + +"A package offering build rules must have a lib root" 这条错误消失（接口根不再是宿主消费的前提）。 + +## 6. 兼容系统 + +### 6.1 原则（写入 docs/51 §Deprecation） + +- 被弃用的写法或读法照常工作，每次使用给出警告，写出规范写法、弃用版本与移除下限。 +- 移除下限是一个 CalVer 月份；从弃用它的发布起至少两个完整日历月；只在下限及以后的发布中移除。 +- docs/51 "`mcpp.toml` keys: Additive" 的例外只有登记表中的条目。 +- 警告只对能修改该清单的包给出（根、工作空间成员、工作空间内的 path 依赖）；依赖包中的用法由 `mcpp why compat` 报告， + index 由 `mcpp xpkg parse --format json` 的 `compat` 字段在 CI 中检查。 + +### 6.2 登记表 + +`mcpp.pm.compat` 中一张表，每条一行： + +| 字段 | 例 | +|---|---| +| `id` | `workspace-position` | +| `since` | `2026.10.10.1` | +| `floor` | `2027.1` | +| `what` | `[toolchain] on a workspace root reaching members by position` | +| `canonical` | `[workspace.toolchain]` | +| `domain` | `manifest/workspace-position` | +| `docs` | `docs/07-workspace.md#…` | + +本 PR 登记的条目： + +| id | 读法 | 规范写法 | +|---|---|---| +| `workspace-position` | 工作空间根自己的 `[toolchain]`、`[indices]`、`[profile.*]`、`[target.*]` 标量、`[xlings]` 按位置到达成员 | `[workspace.X]` | +| `host-module-in-dependencies` | `[dependencies] x = { host-module = true }`，以及请求规则特性的 `[dependencies]` 边（Form B `deps` 同） | `[build-dependencies]` | +| `plugin-imports-from-build-dependencies` | 插件的库单元导入本包 `[build-dependencies]` 中 `host-module = true` 的条目 | 写进本包的 `[dependencies]` | +| `dotted-package-name` | `name = "ns.name"`、`"ns.name" = "…"`（已有的旧兼容，纳入登记表） | `namespace = …` | + +### 6.3 由登记表派生 + +| 派生物 | 内容 | +|---|---| +| 警告文本 | "`` is deprecated since mcpp `` and is removed from mcpp ``. Write ``." | +| `mcpp why compat`（新话题，`--format json` 为 `mcpp.why.compat`） | 本工程及其依赖用到的每条兼容读法、出现位置、`since`/`floor` | +| docs/51 的弃用表 | `.github/tools/check_compat_registry.py` 核对 docs/51 中英文表格与登记表一致 | +| 代码标记 | 每个 `COMPAT()` 注释都必须对应一条登记，每条登记至少有一个标记（同一个 CI 工具检查） | +| 强制删除 | 单元测试：`MCPP_VERSION` 达到某条的 `floor` 时失败，写出该 id 与它的标记位置 | +| 新条目的约束 | 同一单元测试检查 `floor` 至少比 `since` 晚两个完整日历月 | + +内部的 "COMPAT, remove in 1.0.0" 注释（resolver、package_fetcher、legacy_dirs、compat 的回退候选）不是对外接口， +改为 `COMPAT(<主题>)` 内部标签，不进入登记表、不承诺版本（登记表只登记用户写得出来的东西）。CHANGELOG 不改。 + +## 7. 引擎模块与构建数据库（mcppls 的基础设施） + +### 7.1 引擎模块的位置 + +- `$MCPP_HOME/engine//modules/{mcpp.cppm, mcpp_core.cppm}`：只读，首次需要时写入（原子重命名），内容只取决于 mcpp 版本。 +- `mcpp env`（`mcpp.env`）新增 `engineModules` 字段，编辑器与 mcppls 可以直接读到。 +- `mcpp cache gc` 回收不再被任何已安装 mcpp 版本使用的目录（与 `_engine` 缓存条目同一规则）。 + +### 7.2 SPEC-005 的变化（v1.8，只增不改，只涉及 S1 文档） + +`build.mcpp` 是 mcpp 生态专属的文件，引擎模块是 mcpp 自己的模块，消费它们的是 mcppls。因此新内容**只**进入 +`mcpp emit build-database` 的 S1 文档；`mcpp build` 写出的 `compile_commands.json` 与 +`mcpp emit build-database --spec compile-commands` 都不变（R4.1 不变），不为其他 LSP 做适配。 +插件库作为根时（P3）根计划本身就是构建程序配置，它自己的单元照常出现在两种文档中；引擎包的单元只出现在 S1 的 +`mcpp:core` 集合中。 + +| 规则 | 内容 | +|---|---| +| 新集合 | 每个构建程序一个集合 `<包>:build`（单元为 `build.mcpp` 或合成程序）；每个宿主实例一个集合 `<包>:host`；引擎包为 `mcpp:core`；宿主的标准库模块与目标的不同时为 `mcpp:std:host` | +| 名字的兼容 | 已有集合的名字不变：构建程序配置**不**计为 R3.3 的"多个配置"，因此不触发 `<配置>/` 前缀；新集合以保留后缀区分（与 `<包>:test`、`mcpp:std` 的先例相同） | +| `ide.configuration` | 新集合为 `build-program` | +| `ide.kind` | `<包>:build` 为 `executable`，`<包>:host`、`mcpp:core` 为 `library` | +| `visible-sets` | 新集合只互相可见（宿主图是另一次规划，R3.4 的理由同样成立） | +| 单元的 `arguments` | 取自宿主图计划中该单元的编译记录，与实际编译逐字相同（R3.7 的同一规则，记录来源是宿主图而不是根计划），`.mcpp` 单元带强制语言参数 | +| `provides` | `mcpp:core` 的单元提供 `mcpp` 与 `mcpp.core` | +| `watch` | 增加每个构建程序依赖的源码包的源文件 glob（宿主实例）与引擎模块目录 | +| `--spec compile-commands` 与 `compile_commands.json` | 不包含这些单元，与修订前逐字相同 | +| 不编译（R2.2） | 不变：宿主图在 emit 中只被规划；构建程序照常运行（R2.5），需要编译时与今天一样编译。宿主图在 `mcpp build` 时把自己的描述写到工作目录，emit 在宿主图新鲜时直接读取，不重新规划 | + +### 7.3 给 mcppls 仓库的适配 issue（实现完成、发布之后提出） + +要点：新集合与命名规则；`ide.configuration = build-program`；`.mcpp` 单元按 C++ 处理（参数中已有强制语言）； +引擎模块的只读源码位置（`mcpp env` 的 `engineModules`）及其 `provides`；`<包>:build` 的可见集合；`watch` 新增项； +插件库作为根时整个文档只有构建程序配置（P3）。附一个示例工程与 `mcpp emit build-database` 的输出。 + +## 8. 工作空间 + +### 8.1 一张对照表：虚拟根与带根包的根 + +| 写在工作空间根清单中的东西 | 虚拟根（没有 `[package]`） | 带根包的根 | +|---|---|---| +| `[workspace]`、`[workspace.X]` | 给每个成员（带根包时，根包也是成员 `"."`） | 同左 | +| `[package]`、`[build]`、`[targets]`、`[dependencies]`、`[dev-dependencies]`、`[build-dependencies]`、`[features]`、`[resources]`、`[test]`、`[hooks]`、`[runtime]` | 不作用于任何包：警告，`--strict` 拒绝（已实现） | 根包自己的 | +| `build.mcpp` | **今天静默忽略（W1）→ 改为与上一行相同：警告，`--strict` 拒绝** | 根包自己的构建程序，只作用于根包（实测：成员的编译不带它的指令） | +| `[toolchain]`、`[indices]`、`[profile.*]`、`[target.*]` 标量、`[xlings]` | 兼容读法 `workspace-position`：按位置给成员，每次使用警告；同一张表两种写法时拒绝 | 根包自己的；兼容读法只在成员确实收到值时警告 | +| `[workspace.dependencies]` | 成员以 `x.workspace = true` 选用 | 同左；根包也可以选用（实测可用） | +| `[xlings.overrides]` 与 `[workspace.xlings.overrides]` | 只在根位置给（**修复 R4**） | 同左 | +| 宿主图的工作目录 | `<工作空间根>/target/.build-mcpp/host/` | 同左 | + +### 8.2 修复 + +| 编号 | 修复 | +|---|---| +| R4 | `merge_xlings` 拆开：`deps`/`workspace` 条目照旧在每个位置继承（W5）；`overrides` 移到 `merge_layer_root_position`/`inherit_workspace_root_position`。工作空间层与位置读法同样处理 | +| R5 | 依赖位置经位置读法收到 `[xlings]` 条目时，只在**这次构建的根不属于该工作空间**时警告（例如外部工程依赖 git 托管工作空间的成员）；否则构建的根已经把这些条目带进图的环境（`step6_xlings_workspace_from_graph` 收集图中所有包），收到与否不改变构建。判据：`state.runtimeWorkspaceRoot` 是否等于成员所在工作空间的根 | +| W1 | 虚拟根旁的 `build.mcpp` 与包表同等对待 | +| W2 | `[workspace.dependencies]` 的继承复制 `features`（工作空间的在前，成员的追加，与 Cargo 相同）与 `default-features`（成员写了则以成员为准）；其余边请求键（`host-module`、`tools`、`artifacts`、`reexport`、`visibility`）属于每个成员自己的边——同一条目可以被 `[dependencies]` 与 `[build-dependencies]` 继承，含义不同——写在 `[workspace.dependencies]` 中给出警告，`--strict` 拒绝。今天这些键被静默忽略，所以只警告、不拒绝（docs/51：不得静默忽略，也不得让能构建的清单停止构建） | +| 警告示例 | 位置警告的示例按成员实际收到的部分写（overrides 写 `[workspace.xlings.overrides]`，条目写 `[workspace.xlings.workspace]`） | +| 版本文本 | 由登记表生成（§6） | + +## 9. 兼容性 + +| 今天能构建的清单 | 之后 | +|---|---| +| `[build-dependencies] x = { host-module = true }`（规则包、`mcpp:plugins`） | 照常构建，无诊断（陈述默认值）；宿主实例按完整的包编译，`include_dirs` 等生效 | +| `[dependencies] x = { host-module = true }` 或请求规则特性 | 照常构建，不进目标，警告 `host-module-in-dependencies` | +| 插件接口导入 `[build-dependencies]` 中的包（`rules-tidy`） | 照常构建，警告 `plugin-imports-from-build-dependencies` | +| Form B 包没有接口根，从 `build.mcpp` 导入 | 今天失败；之后可导入，本包构建时有 W1（SPEC-008） | +| index 的 toml++（Form A，相对路径 include） | 照常构建 | +| 没有构建程序的工程 | 不规划宿主图，路径与今天相同 | +| 只 `#include`、不导入模块的 `build.mcpp` | 在宿主图中编译（一个没有依赖的成员）；引号 include 按源文件目录解析，结果不变 | + +写进 CHANGELOG 的行为变化：宿主实例编译依赖的**全部**源（今天"只取列出的 sources"是为了避免把普通库的单元单独编译，单独编译已不存在）； +目标实例不再收到只由构建依赖请求的特性（R3）；有构建程序的包，其 `[build-dependencies]` 中的库默认被编译（`host-module = false` 退出）。 + +## 10. 跨平台 + +| 平台 | 要点 | +|---|---| +| Linux glibc | 程序的 RPATH 由计划的链接规则给出（e2e 901 的工具库路径规则），与今天 `default-v2` 等价，实测（O2） | +| Linux musl、MinGW | "静态助手"策略由合成成员陈述 | +| Windows MSVC / clang-cl | UTF-8 清单由合成成员的资源陈述；工作目录短；`/reference` 由计划的 BmiTraits 生成；`.mcpp` 带 `/TP` | +| macOS | 部署目标由计划的 `min_platform_version` 给出，与今天的判断一致 | +| 交叉构建 | 宿主图 `target_triple = ""`，用构建程序的工具链；引擎包的缓存条目按宿主工具链区分 | +| Termux / Android 宿主 | 与 tools 子构建同一条路径（`posix_shell()`），实测（O4） | + +## 11. 性能 + +### 11.1 基线（本地，mcpp 2026.10.10.1，3 个成员各有导入 `mcpp` 与一个 path host 模块的 `build.mcpp`） + +| 情形 | 时间 | 说明 | +|---|---|---| +| 冷构建 `--workspace`（全局缓存中已有引擎模块） | 1.56 s | 构建程序预编译线程 + host 模块存储 | +| no-op `--workspace` | 0.05 s | 整个构建的快速路径，不进入 prepare | +| 编辑 `m1/build.mcpp` 中的值 | 1.24 s | 其中 host 模块条目检查 52 ms、`build.mcpp` 编译约 500 ms、主 ninja 625 ms | +| 很小的工程 no-op / `--configure-only` | 0.05 s / 0.04 s | 进程启动加规划的量级 | + +### 11.2 预算与手段 + +| 路径 | 预算 | 手段 | +|---|---|---| +| 整个构建的 no-op | 不变 | 快速路径不受影响（它在 prepare 之前） | +| prepare 内、全部程序新鲜 | 程序身份计算 ≤ 10 ms（3 个程序） | stat 摘要；不规划宿主图（§4.4） | +| 一个程序失效 | ≤ 今天（1.24 s） | 宿主图规划 ≤ 50 ms（子构建，引擎包与 index 包命中依赖缓存）；ninja 只编译该程序 | +| 冷构建 | ≤ 今天（1.56 s） | 同组程序一次 ninja 并行；共享包编译一次 | +| 插件根的预扫描 | ≤ 5 ms / 100 个文件 | 只读导入，结果交给主扫描复用 | +| 宿主图锁等待 | 记录等待时长 | 日志 `host-graph: waiting for ` | + +这些预算在探测 PR 中按 #748 的工作负载在三个平台上实测（O1），达不到时在实现前回到设计。 + +## 12. 可观测性 + +| 层面 | 内容 | +|---|---| +| 日志标签（`mcpp.log`，`--verbose` 与 `~/.mcpp/log/mcpp.log`） | `edges`：每条边的配置、产物与来源 section；`features`：每个实例两张表的激活结果；`host-graph`：分组、合成成员、钉、工作目录、锁、规划与 ninja 的 `ScopedTimer`；`build-program`：每个程序的身份分量、**失效原因（哪个分量变了）**、运行；`interface`：每个包的接口根、公开模块与来源；`compat`：每次兼容读法的使用 | +| 记录文件 | 宿主图工作目录中的 `host-graph.json`：组键、成员、每个程序的依赖与身份分量、上次构建时间；与 `resolution.json` 并列 | +| 命令 | `mcpp why build-programs`（`mcpp.why.build-programs`）：每个程序所在的组、依赖、实例、是否新鲜及原因；`mcpp why deps` 写出每条边的配置与每个包的实例；`mcpp why compat`（§6.3） | +| 进度 | 宿主图作为一个分组（docs/50 "The group record"）：`Compiling build programs (host <三元组>, <工具链>, <标准>): m1, m2, m3` | +| 失败 | 宿主图的失败带工作目录与复现命令；`MCPP_TOOL_BUILD_VERBOSE` 同样作用于宿主图的内层输出 | + +## 13. 测试 + +### 13.1 新增 e2e（编号接在 901 之后；每个都必须在 2026.10.10.2 上失败） + +| 编号 | 判据 | +|---|---| +| 902 | 头文件库的模块包装（path、本地 index 的 Form A、Form B + `lib`）可以从 `build.mcpp` 导入，GMF 用 `<…>`（R1）；不写 `host-module` 也可以 | +| 903 | §3.5 的验收矩阵 M1–M10（M5、M6 在今天通过，作为不得退化的保护；其余在 2026.10.10.2 上失败） | +| 904 | 只由构建依赖请求的特性不进目标实例，宿主实例拿到它（R3） | +| 905 | 宿主库带非接口源文件、C 源文件和自己的 `[dependencies]`，全部链接进构建程序（#797 Q4、Q5） | +| 906 | 编辑 path 宿主库所 include 的头文件，程序重编重跑；不编辑时 `--verbose` 显示宿主图未被规划（R6 与预算） | +| 907 | Form B `lib`；`modules` 断言不一致时的诊断与 `--strict`；`lib` 指向不存在的文件：本包构建报错，作为依赖时只警告 | +| 908 | `host-module` 默认值：只要 `tools` 时不编译库；加 `host-module = true` 两者都有；规则特性隐含；`host-module = false` 退出 | +| 909 | 兼容登记表：每条读法行为不变并给出带 `since`/`2027.1` 的警告；index 包不警告；`mcpp why compat` 列出它们；`--strict` 下为错误 | +| 910 | 插件：P1（`[dependencies]` 到插件被拒绝并写出链）、P3（插件包 `mcpp build`、`mcpp test` 成功，`--target` 他处被拒绝）、P4、P5；插件依赖插件与普通库（S1、S2） | +| 911 | 工作空间：R4、R5、W1、W2 | +| 912 | 两个配置组宿主配置相同时，宿主库只编译一次（锁与共享） | +| 913 | S1 文档：`<包>:build`、`<包>:host`、`mcpp:core` 集合存在，参数与宿主图的实际编译逐字相同，已有集合名字不变，`watch` 含新项；`compile_commands.json` 与 `--spec compile-commands` 的输出与修订前逐字相同；`mcpp env` 的 `engineModules` 指向存在的只读源码 | + +### 13.2 单元测试 + +`library_interface`（接口根、约定、特性接口、转出闭包、断言比较）；边模型（配置继承、`wantsLibrary` 默认规则）； +程序身份（每个分量的变化都改变身份，stat 摘要回退到内容摘要）；兼容登记表（下限强制、`floor - since` 下限、 +警告文本）；插件预扫描（只读导入、结果复用）。 + +### 13.3 已有测试 + +189、309–312、610、622、624、630、633、798、815、819、831 **不改断言**通过（行为兼容）。857 判据 C/D 与 859 断言的是 +`host-modules/` 存储布局，改为断言同样的性质（编译一次、path 包不进全局缓存、引擎模块进全局缓存、输入一致时复用）， +位置改为宿主图与依赖缓存。894 的警告文本改为登记表生成的文本。 + +### 13.4 本轮已做的测量 + +R1–R8、R2b、R2c、W1–W2 在复现脚本中（2026-10-11 全量运行一次，结果与 §1.1 一致）；§11.1 的基线；`examples/08-build-rules/app`（`[dependencies] … host-module = true` +与规则依赖规则）今天可以构建（1.25 s），它是兼容读法必须保持的用例。 + +## 14. 文档与规范 + +- **SPEC-004**：新增 §12"依赖的库与配置"（两条轴、实例、一个包一个版本、按配置激活特性、`host-module` 的精确含义与默认值、 + §5.2 所要求的历史注记）；§9 第 13 条的移除下限；`[workspace.dependencies]` 的继承键（W2）；变更记录。 +- **SPEC-005**（构建数据库）v1.8：§7.2，只改 §3（S1 文档）与 §6（`watch`），§4 不变。 +- **SPEC-007**（构建插件）：插件单元的定义与 P1–P5；构建程序在宿主图中编译；引擎模块的位置；变更记录。 +- **SPEC-008**（库接口）0.2：I1 扩展到配置；I2 加入特性接口；Form B `lib`；`exports` 为断言；§4 诊断表；第二阶段的条件不变。 +- **docs（中英）**：04 §2.4 `[lib]`；05 `[build-dependencies]`（两条轴、`host-module` 含义与默认、两表同名）； + 30 构建程序与构建依赖的库（删除 "compiled alone"）；31 插件写作（依赖写在 `[dependencies]`、插件包的构建与测试、`rule_module`）； + 07 工作空间（§8 的对照表）；50 新的 `why` 话题与分组；51 弃用策略与登记表；03 示例；23 `mcpp env` 的 `engineModules`。 +- **CHANGELOG**：Added、Changed、Fixed（R1–R8、W1、W2）、Removed、Deprecated（登记表条目）。 + +## 15. 生态同步 + +| 仓库 | 内容 | 时机 | +|---|---|---| +| mcpp（本 PR） | `examples/08-build-rules`、`09-heterogeneous`、`10-graphics`、`12-a-new-device-language` 改为规范写法；`rules-tidy` 的依赖移到 `[dependencies]` | 本 PR | +| mcpp-index | 6 个 Form B 包加 `lib`；`marzer.tomlplusplus` 回到 Form B 加 `lib`，include 改回 ``，按 revision 递增；`min_mcpp` 门控；CI 读取 `mcpp xpkg parse --format json` 的 `compat` 字段 | mcpp 发布之后 | +| mcpp-plugins | 消费方测试中多余的 `host-module = true` 可保留（陈述默认值）；fixture 测试可以改写为插件包自己的 `mcpp test`（S4）；README；`mcpp = ">=<本发布>"` | mcpp 发布之后 | +| mcppls | 适配 issue（§7.3） | mcpp 发布之后 | + +## 16. 决策 + +| 编号 | 决策 | +|---|---| +| D1 | 配置由 section 决定，产物由边上的请求决定 | +| D2 | 不加新键；`host-module` 保留原名，含义为"构建程序使用这个库"，在 `[build-dependencies]` 中没有 `tools`/`artifacts` 时默认成立 | +| D3 | 节点为 `(包, 配置)`；两表同名为两个实例；版本统一；特性按配置激活；环境取并集 | +| D4 | 宿主图：按（工具链身份、标准）分组，合成工作空间、每个构建程序一个合成成员，经子构建规划与构建，以主图解析为钉 | +| D5 | 引擎模块是按版本存放的只读引擎包，只在构建程序配置中存在；全局缓存条目位置不变 | +| D6 | 宿主图 profile 为 `opt = 0`、`debug = false`，与今天相同；不新增用户可配置项 | +| D7 | 宿主图工作目录按组键命名并加锁；index 包经依赖缓存全局复用，path 包留在工作区 | +| D8 | 程序缓存身份覆盖整个闭包；path 包用 stat 摘要加内容回退；全部新鲜时不规划宿主图 | +| D9 | `library_interface` 是库接口的唯一定义 | +| D10 | Form B `lib` ≡ `[lib] path`；`exports`/`modules` 是断言 | +| D11 | 特性加入的接口单元是该特性的公开模块；`rule_module` 必须是其中之一（SPEC-008 I2 修订） | +| D12 | 接口根不是宿主消费的前提；显式声明而不存在：本包报错，依赖警告 | +| D13 | 构建程序只能导入本包构建依赖（及 reexport 传来的）的模块；目标侧导入传递依赖模块的现状不变（改变它会让能构建的清单停止构建） | +| D14 | 删除 `host_module_compile`、`host_module_store`、`host_module_units`、清空 sources、预编译线程 | +| D15 | 插件单元由导入引擎接口判定；P1–P5 | +| D16 | 兼容登记表与 §6.3 的派生物；移除下限 `2027.1` | +| D17 | 只有 S1 文档按 §7.2 描述构建程序、宿主实例与引擎模块，名字只增不改；`compile_commands.json` 不变 | +| D18 | 工作空间：§8.2 的六项修复 | +| D19 | 工具链阶段的构建程序（R9.9）用同一套宿主图，bootstrap 工具链，没有钉 | +| D20 | 性能预算（§11.2）与可观测性（§12）是实现的验收条件 | + +## 17. 待测量（探测 PR） + +| 编号 | 问题 | 判据 | +|---|---|---| +| O1 | #748 工作负载在 Linux、macOS、Windows 上的冷构建、程序失效、全部新鲜、no-op | §11.2 的预算 | +| O2 | Linux glibc 程序的 RPATH 与 `default-v2` 运行一致；musl、MinGW 静态链接 | 三平台 e2e 通过 | +| O3 | Windows MSVC 与 clang-cl：UTF-8 清单、`/reference`、`/TP`、路径长度 | Windows runner 上 902–913 与已有测试通过 | +| O4 | macOS 部署目标；Termux 宿主 | 对应 runner 与 Android 探测通过 | +| O5 | 交叉构建（`aarch64-linux-gnu`、`wasm32-emscripten`）的宿主图与引擎包缓存键 | 程序可运行，缓存条目按宿主工具链区分 | +| O6 | 子构建规划宿主图的实际耗时（`ScopedTimer`） | ≤ 50 ms | + +## 18. 自我 review + +### 18.1 修订 2 → 3 + +| 问题 | 修正 | +|---|---| +| 修订 2 把新集合也放进 `compile_commands.json`，会改变所有读者看到的内容，而这些单元只对 mcppls 有意义 | 只进 S1（§7.2），e2e 913 断言 `compile_commands.json` 逐字不变 | +| S1 中新单元的 `arguments` 原写作"与 `compile_commands.json` 的条目同一条记录"（R3.7），而新单元不再进入后者 | 改为取自宿主图计划的编译记录 | +| 插件根（P3）的引擎包单元在 `compile_commands.json` 中是否出现没有说明 | 引擎包单元只在 S1 的 `mcpp:core` 中出现，插件根自己的单元照常 | +| "同一个库两侧使用"只有一条测试（903 的 R2），没有覆盖特性不同、Form B、依赖的构建程序、交叉构建、头文件改动 | §3.5 的矩阵 M1–M10，M5、M6 是今天能工作的保护用例 | +| 目标与宿主两个实例都提供模块 `hl`，是否触发同名模块冲突没有说明 | 两次规划各自一张模块图，不冲突；依赖缓存的两份条目并存（§3.5 末段） | + +### 18.2 修订 1 → 2 + +| 问题 | 修正 | +|---|---| +| 新键 `lib` 违反 SPEC-004 §5.2（已发布的键不改名），并让每个用户迁移一次 | 撤回，`host-module` 精确化（D2） | +| "插件经 `[dependencies]` 进入目标即拒绝"会挡住插件依赖插件，也没有回答插件包自己怎么构建（R7） | 改为配置传播加 P1–P5 | +| 一个合成包含多个 `bin`：所有程序共享依赖、链接与可见性，一个程序的依赖会出现在另一个程序的链接中 | 合成工作空间，每个程序一个成员 | +| 程序身份对 path 包的源树摘要每次都读内容（今天检查一次 host 模块条目要 52 ms） | stat 摘要加内容回退 | +| 构建程序配置的集合若计为"多个配置"，会给已有集合加前缀，破坏现有的构建数据库读者 | 新集合用保留后缀，不计为配置（§7.2） | +| 合成成员的根目录是消费方包的根目录，默认的 `sources` 推断会取到消费方的 `src/**` | 合成成员明确 `sources = []` | +| `FileLock::try_acquire` 只尝试一次 | 宿主图需要阻塞获取并记录等待（§4.3） | +| 插件根的预扫描与主扫描重复读文件 | 结果交给主扫描复用（§4.8） | +| 兼容读法警告若对 index 包给出，用户无法处理 | 只对能修改的清单警告，依赖中的用法由 `mcpp why compat` 报告 | +| `examples/08-build-rules/app` 用的是要弃用的写法 | 列入兼容测试（909）与示例更新（§15） | + +## 19. PR 组织 + +一个 PR，分支 `release/<下一版本>`，按依赖顺序提交（每个提交都能编译，测试在最后一个提交全部通过）： + +1. 兼容登记表、派生的警告、`mcpp why compat`、CI 工具与单元测试；docs/51 策略；`workspace-position` 迁入登记表。 +2. `library_interface` 模块与单元测试；`scan.cpp`、`validate.cppm`、`library_pipeline.cppm` 改用它；Form B `lib`；`exports` 断言。 +3. 清单规范化：`host-module` 的默认值与兼容读法；`[workspace.dependencies]` 的继承键（W2）。 +4. 边模型：`configuration`、`wantsLibrary`、实例谓词、按配置激活特性；删除清空 sources。 +5. 引擎包与 `mcpp env` 字段；宿主图（`prepare/host_graph.cpp`）；构建程序的编译改走宿主图；程序身份。 +6. 插件规则 P1–P5 与根的预扫描。 +7. 删除 `host_module_compile`、`host_module_store`、预编译线程；诊断迁移。 +8. S1 文档的新集合（SPEC-005 v1.8）；`compile_commands.json` 不变的回归断言。 +9. 工作空间：R4、R5、W1、警告示例。 +10. 可观测性：日志标签、`host-graph.json`、`mcpp why build-programs`、进度分组。 +11. e2e 902–913、已有测试的更新、examples。 +12. 规范（SPEC-004、005、007、008）、docs 中英文、CHANGELOG、版本号。 + +在 review 结论与探测 PR（O1–O6）完成之前，不开始实现。 + +## 20. 实施计划与任务依赖（goal 启动时写入） + +版本：`2026.10.11.1`（发布日期变化时在最后一个提交中改）。分支 `release/2026.10.11.1`，mcpp 一个 PR； +mcpp-index、mcpp-plugins 各一个 PR（发布之后）；mcppls 一个 issue。 + +| 任务 | 内容 | 依赖 | 验证 | +|---|---|---|---| +| T1 | 兼容登记表（`mcpp.pm.compat`）、警告生成、`mcpp why compat`、下限强制的单元测试、`check_compat_registry.py`；`workspace-position` 迁入 | — | 单元测试；894 的新文本 | +| T2 | 工作空间：R4、R5、W1、W2、警告示例 | T1 | e2e 911 | +| T3 | 清单：`host-module` 的三态（未写/真/假）与默认值；`[dependencies]` 中的兼容读法移到 `buildDependencies`；Form B `lib`；Form B `deps` 同规则 | T1 | 单元测试（toml、xpkg） | +| T4 | 边模型：`configuration`、`wantsLibrary`、`targetInstance`、`compilesHere`；特性按配置；删除跨 section 合并与清空 sources | T3 | e2e 903（M1–M3 的目标侧）、904、310 | +| T5 | `library_interface` 模块及其读者；Form B `lib`、`exports` 断言、`rule_module` 校验 | — | 单元测试；e2e 907 | +| T6 | 引擎包：`$MCPP_HOME/engine/<版本>/modules/`；`mcpp env` 的 `engineModules` | — | e2e 913（部分） | +| T7 | 宿主图：合成工作空间（写到 `target/.build-mcpp/host/<键>/`，可在该目录直接复现）、子构建、钉、工具链继承、内部 profile、锁、`.mcpp` 源；构建程序的编译改走宿主图；程序身份；删除 `host_module_compile`、`host_module_store`、预编译线程 | T4、T5、T6 | e2e 902、903、905、906、912；已有 189、309–312、610、622、624、630、633、798、815、819、831、857、859 | +| T8 | 插件 P1–P5 与根的预扫描 | T7 | e2e 910 | +| T9 | S1 文档的新集合 | T7 | e2e 913 | +| T10 | 可观测性：日志标签、`host-graph.json`、`mcpp why build-programs`、`why deps` 的配置、进度分组 | T7 | e2e 906 的 `--verbose` 断言 | +| T11 | e2e 902–913、单元测试、已有测试的更新 | 各自的任务 | 本地全量 e2e；CI 三平台 | +| T12 | 规范（SPEC-004、005、007、008）、docs 中英文、CHANGELOG、版本号 | 全部 | `check_docs_*`、`check_*` | +| T13 | examples（08、09、10、12）改为规范写法 | T3 | `build_examples.sh` | +| T14 | CI 全绿、生态级自我 review、合入、发布、GitCode、xim-pkgindex、clean-room 验证、mcpp-index 与 mcpp-plugins PR、mcppls issue | 全部 | — | + +关键路径是 T3 → T4 → T7 → T8/T9/T10 → T11。T1、T2、T5、T6 与关键路径并行。 +宿主图的合成工作空间**写到磁盘**（修订 3 写作"不落盘"）：它由引擎生成，落盘之后可以在该目录直接运行 +`mcpp build` 复现宿主图的构建，这是可观测性要求的一部分；内容不变时不重写，ninja 保持增量。 diff --git a/.agents/docs/2026-10-11-build-dependencies-one-pipeline-repro.sh b/.agents/docs/2026-10-11-build-dependencies-one-pipeline-repro.sh new file mode 100644 index 00000000..ec0d7afa --- /dev/null +++ b/.agents/docs/2026-10-11-build-dependencies-one-pipeline-repro.sh @@ -0,0 +1,199 @@ +#!/usr/bin/env bash +# Observations for .agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md. +# Prints what each case does; asserts nothing. Usage: MCPP=/path/to/mcpp bash +# +# R1 #797 body: a module wrapper over a header library, imported by build.mcpp +# through `host-module = true` (its GMF #includes the package's own header). +# R2 #797 comment: one package in [dependencies] and [build-dependencies]. +# R2b the same library split across two workspace members: m1 links it, m2's +# build.mcpp imports it (works today; a regression guard). +# R2c a library reached by the target through another library, and named +# directly in [build-dependencies] (works today; a regression guard). +# R3 a feature requested only by a [build-dependencies] edge reaches the +# target instance of the same package. +# R4 [workspace.xlings.overrides] on a rooted workspace whose root package +# depends on a member by path. +# R5 the same with the root's own [xlings] by position: the warning appears +# on a plain `mcpp build` of the root package. +# R6 a header that a path host module includes is edited; the build +# program is neither recompiled nor rerun (its identity hashes the +# interface file only). +# R7 a build plugin built as the root of its own build (the shape of +# examples/08-build-rules/rules-embed): the engine module does not exist. +# R8 `mcpp emit build-database` and compile_commands.json do not describe +# build.mcpp, the host modules or the engine module. +# W1 a build.mcpp beside a workspace root without [package] is ignored +# without a word (the package tables there are reported). +# W2 `features` written in [workspace.dependencies] is dropped without a +# word when a member inherits the entry. +set -u +MCPP="${MCPP:-mcpp}" +T=$(mktemp -d) +trap 'rm -rf "$T"' EXIT +show() { echo "── $1"; shift; ( "$@" ) 2>&1 | grep -E "error|warning|fatal|FEATURE|ok:" | head -8; echo; } + +# ── a header library with a module wrapper (the toml++ shape) ─────────────── +mklib() { # $1 = dir + mkdir -p "$1/include/hl" "$1/src" + cat > "$1/include/hl/hl.hpp" <<'EOF' +#pragma once +namespace hl { inline int answer() { return 42; } } +EOF + cat > "$1/src/hl.cppm" <<'EOF' +module; +#include +export module hl; +export namespace hl { using hl::answer; } +#ifdef HL_FEATURE_X +export namespace hl { inline int feature_x() { return 1; } } +#endif +EOF + cat > "$1/mcpp.toml" <<'EOF' +[package] +name = "hl" +version = "0.1.0" +[lib] +path = "src/hl.cppm" +[build] +sources = ["src/hl.cppm"] +include_dirs = ["include"] +[features] +x = { defines = ["HL_FEATURE_X"] } +[targets.hl] +kind = "lib" +EOF +} +mklib "$T/hl" +# The same library with the include made relative to the unit -- the shape +# mcpp-index uses today to get past R1 -- so that R2 isolates the edge merge. +mklib "$T/hlrel" +sed -i 's|#include |#include "../include/hl/hl.hpp"|' "$T/hlrel/src/hl.cppm" + +mkapp() { # $1 = dir, $2 = manifest tail + mkdir -p "$1/src" + printf '[package]\nname = "app"\nversion = "0.1.0"\n[targets.app]\nkind = "bin"\nmain = "src/main.cpp"\n%s\n' "$2" > "$1/mcpp.toml" +} + +# R1 +mkapp "$T/r1" $'[build-dependencies]\nhl = { path = "../hl", host-module = true }' +echo 'int main() { return 0; }' > "$T/r1/src/main.cpp" +printf 'import std;\nimport hl;\nint main() { std::println("mcpp:cxxflag=-DANSWER={}", hl::answer()); }\n' > "$T/r1/build.mcpp" +show "R1 host-module over a header library" bash -c "cd $T/r1 && $MCPP build" + +# R2 +mkapp "$T/r2" $'[dependencies]\nhl = { path = "../hlrel" }\n[build-dependencies]\nhl = { path = "../hlrel", host-module = true }' +printf 'import std;\nimport hl;\nint main() { std::println("ok: {}", hl::answer()); }\n' > "$T/r2/src/main.cpp" +printf 'import std;\nimport hl;\nint main() { std::println("mcpp:cxxflag=-DANSWER={}", hl::answer()); }\n' > "$T/r2/build.mcpp" +show "R2 one package in both tables (target import)" bash -c "cd $T/r2 && $MCPP build" + +# R2b +mkdir -p "$T/r2b" +printf '[workspace]\nmembers = ["m1", "m2"]\n' > "$T/r2b/mcpp.toml" +mkapp "$T/r2b/m1" $'[dependencies]\nhl = { path = "../../hlrel" }' +sed -i 's/name = "app"/name = "m1"/; s/targets.app/targets.m1/' "$T/r2b/m1/mcpp.toml" +printf 'import std;\nimport hl;\nint main() { std::println("ok: m1 {}", hl::answer()); }\n' > "$T/r2b/m1/src/main.cpp" +mkapp "$T/r2b/m2" $'[build-dependencies]\nhl = { path = "../../hlrel", host-module = true }' +sed -i 's/name = "app"/name = "m2"/; s/targets.app/targets.m2/' "$T/r2b/m2/mcpp.toml" +printf 'import std;\nint main() { std::println("ok: m2 {}", ANSWER); }\n' > "$T/r2b/m2/src/main.cpp" +printf 'import std;\nimport hl;\nint main() { std::println("mcpp:cxxflag=-DANSWER={}", hl::answer()); }\n' > "$T/r2b/m2/build.mcpp" +show "R2b workspace split" bash -c "cd $T/r2b && $MCPP build --workspace && $MCPP run -p m1 && $MCPP run -p m2" + +# R2c +mkdir -p "$T/lib/src" +printf '[package]\nname = "lib"\nversion = "0.1.0"\n[dependencies]\nhl = { path = "../hlrel" }\n[targets.lib]\nkind = "lib"\n' > "$T/lib/mcpp.toml" +printf 'export module lib;\nimport hl;\nexport int via_lib() { return hl::answer(); }\n' > "$T/lib/src/lib.cppm" +mkapp "$T/r2c" $'[dependencies]\nlib = { path = "../lib" }\n[build-dependencies]\nhl = { path = "../hlrel", host-module = true }' +printf 'import std;\nimport lib;\nint main() { std::println("ok: {} {}", via_lib(), ANSWER); }\n' > "$T/r2c/src/main.cpp" +cp "$T/r2b/m2/build.mcpp" "$T/r2c/build.mcpp" +show "R2c transitive target use and a direct build dependency" bash -c "cd $T/r2c && $MCPP build && $MCPP run" + +# R3 -- the library itself is a dependency of `mid`, which asks for feature x +# only in its [build-dependencies]; the app imports hl for its target. +mkdir -p "$T/mid/src" +printf '[package]\nname = "mid"\nversion = "0.1.0"\n[build-dependencies]\nhl = { path = "../hl", features = ["x"] }\n[targets.mid]\nkind = "lib"\n' > "$T/mid/mcpp.toml" +printf 'export module mid;\nexport int mid_v() { return 1; }\n' > "$T/mid/src/mid.cppm" +mkapp "$T/r3" $'[dependencies]\nhl = { path = "../hl" }\nmid = { path = "../mid" }' +cat > "$T/r3/src/main.cpp" <<'EOF' +import std; +import hl; +int main() { +#ifdef HL_FEATURE_X + std::println("FEATURE x reached the target"); +#endif + std::println("ok: {}", hl::answer()); +} +EOF +show "R3 a build-only feature request and the target instance" bash -c "cd $T/r3 && $MCPP build && $MCPP run" +grep -rhoE -- "-DHL_FEATURE_X" "$T/r3/target" 2>/dev/null | head -1 | sed 's/^/FEATURE define in target compile: /' + +# R4, R5 +mkws() { # $1 = dir, $2 = root tail + mkdir -p "$1/src" "$1/sub/src" + printf '[package]\nname = "root"\nversion = "0.1.0"\n[targets.root]\nkind = "bin"\nmain = "src/main.cpp"\n[dependencies]\nsub = { path = "sub" }\n[workspace]\nmembers = ["sub"]\n%s\n' "$2" > "$1/mcpp.toml" + echo 'int main() { return 0; }' > "$1/src/main.cpp" + printf '[package]\nname = "sub"\nversion = "0.1.0"\n[targets.sub]\nkind = "lib"\n' > "$1/sub/mcpp.toml" + printf 'export module sub;\nexport int f() { return 1; }\n' > "$1/sub/src/sub.cppm" +} +mkws "$T/r4" $'[workspace.xlings.overrides]\n"xim:cmake" = "/usr/bin/cmake"' +show "R4 [workspace.xlings.overrides] with the root depending on a member" bash -c "cd $T/r4 && $MCPP build" +mkws "$T/r5" $'[xlings.overrides]\n"xim:cmake" = "/usr/bin/cmake"' +show "R5 the root's own [xlings] by position, plain build of the root" bash -c "cd $T/r5 && $MCPP build" + +# R6 -- uses the relative-include library so that the host compile succeeds. +mkapp "$T/r6" $'[build-dependencies]\nhl = { path = "../hlrel", host-module = true }' +printf 'import std;\nint main() { std::println("ok: ANSWER={}", ANSWER); }\n' > "$T/r6/src/main.cpp" +printf 'import std;\nimport hl;\nint main() { std::println("mcpp:cxxflag=-DANSWER={}", hl::answer()); }\n' > "$T/r6/build.mcpp" +show "R6 before the header edit" bash -c "cd $T/r6 && $MCPP build >/dev/null && $MCPP run" +sed -i 's/return 42/return 43/' "$T/hlrel/include/hl/hl.hpp" +show "R6 after editing the header to 43" bash -c "cd $T/r6 && $MCPP build >/dev/null && $MCPP run" + +# R7 +mkdir -p "$T/r7/src" +printf '[package]\nname = "rule"\nversion = "0.1.0"\n[targets.rule]\nkind = "lib"\n' > "$T/r7/mcpp.toml" +printf 'export module rule;\nimport std;\nimport mcpp;\nexport namespace rule { inline bool compile() { return true; } }\n' > "$T/r7/src/rule.cppm" +show "R7 a build plugin as the root of its own build" bash -c "cd $T/r7 && $MCPP build" + +# R8 -- r6 has a build.mcpp importing a host module +( cd "$T/r6" && $MCPP emit build-database --format json 2>/dev/null ) > "$T/db.json" +python3 - "$T/db.json" <<'PY' +import json, sys +doc = json.load(open(sys.argv[1]))["data"]["database"] +def sources(o): + if isinstance(o, dict): + for k, v in o.items(): + if k == "source" and isinstance(v, str): yield v.replace("\\", "/").split("/")[-1] + else: yield from sources(v) + elif isinstance(o, list): + for v in o: yield from sources(v) +units = sorted(set(sources(doc))) +print("── R8 build database") +print("ok: translation units:", units) +print("ok: build.mcpp / hl.cppm / mcpp.cppm described:", + "build.mcpp" in units, "hl.cppm" in units, "mcpp.cppm" in units) +PY +echo + +# W1 +mkdir -p "$T/w1/m/src" +printf '[workspace]\nmembers = ["m"]\n' > "$T/w1/mcpp.toml" +printf 'int main() { return 1; }\n' > "$T/w1/build.mcpp" # would fail the build if it ran +printf '[package]\nname = "m"\nversion = "0.1.0"\n[targets.m]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$T/w1/m/mcpp.toml" +echo 'int main() { return 0; }' > "$T/w1/m/src/main.cpp" +show "W1 build.mcpp beside a virtual root (no output = ignored silently)" bash -c "cd $T/w1 && $MCPP build" + +# W2 +mkdir -p "$T/w2/m/src" +printf '[workspace]\nmembers = ["m"]\n[workspace.dependencies]\nhl = { path = "../hl", features = ["x"] }\n' > "$T/w2/mcpp.toml" +printf '[package]\nname = "m"\nversion = "0.1.0"\n[dependencies]\nhl.workspace = true\n[targets.m]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$T/w2/m/mcpp.toml" +cat > "$T/w2/m/src/main.cpp" <<'CPP' +import std; +import hl; +int main() { +#ifdef HL_FEATURE_X + std::println("ok: feature x inherited"); +#else + std::println("FEATURE x from [workspace.dependencies] was dropped"); +#endif +} +CPP +show "W2 features in [workspace.dependencies]" bash -c "cd $T/w2 && $MCPP build && $MCPP run -p m" diff --git a/.agents/docs/README.md b/.agents/docs/README.md index 52585134..08730041 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -330 records. +331 records. ## By subject @@ -26,6 +26,7 @@ Records that declare one. Everything else is listed by date below. ### build-program +- [构建依赖的一条管线：配置与产物两轴、包的实例、库接口、兼容系统与工作空间收尾（#797）](2026-10-11-build-dependencies-one-pipeline-design.md) — active - [Four upstream asks from a UI framework: what each one is under mcpp's design, and the combined plan](2026-09-13-four-upstream-asks-from-a-ui-framework.md) — landed ### design @@ -124,6 +125,7 @@ Records that declare one. Everything else is listed by date below. ### 2026-10 +- [构建依赖的一条管线：配置与产物两轴、包的实例、库接口、兼容系统与工作空间收尾（#797）](2026-10-11-build-dependencies-one-pipeline-design.md) — active - [一处来源，一种含义：工作空间的传递语义、生成输出、模块角色与首次运行（#786、#785、#778、#790）](2026-10-10-one-source-one-meaning-design.md) — landed - [LLVM 23.1.3 Part 2：Linux aarch64 默认工具链与 glibc 生态闭环](2026-10-08-llvm-2313-linux-aarch64-ecosystem-part2-design.md) — active - [LLVM 23.1.3 全平台统一默认:跨仓库联动方案(mcpp × xim-pkgindex)](2026-10-07-llvm-2313-unified-default-cross-repo-design.md) — active From 656484224716c5421951c0fe3a7e2cdcd73f8cc6 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sun, 11 Oct 2026 06:11:05 +0900 Subject: [PATCH 02/12] Edges carry the configuration their table states; features per instance; compat registry; workspace overrides at the root position (#797) --- modules/buildmcpp/src/provisions.cppm | 5 +- modules/manifest/src/compat/registry.cppm | 144 ++++++++++++++ .../src/compat/workspace_position.cppm | 15 +- modules/manifest/src/dep_spec.cppm | 28 ++- modules/manifest/src/manifest.cppm | 1 + modules/manifest/src/toml.cppm | 16 ++ src/build/prepare/config.cpp | 11 +- src/build/prepare/features.cpp | 188 ++++++++++-------- src/build/prepare/graph.cpp | 147 +++++++++----- src/build/prepare/graph_load.cpp | 7 +- src/build/prepare/manifest.cpp | 19 +- src/build/prepare/scan.cpp | 2 +- src/build/prepare/state.cppm | 54 +++-- src/modgraph/scanner.cppm | 9 +- src/project.cppm | 114 +++++++++-- 15 files changed, 568 insertions(+), 192 deletions(-) create mode 100644 modules/manifest/src/compat/registry.cppm diff --git a/modules/buildmcpp/src/provisions.cppm b/modules/buildmcpp/src/provisions.cppm index 1596f3b9..c186ac80 100644 --- a/modules/buildmcpp/src/provisions.cppm +++ b/modules/buildmcpp/src/provisions.cppm @@ -124,7 +124,7 @@ struct Provision { // Templated on the edge type for the reason `dep_graph` is: `DependencyEdge` // is a local struct inside prepare_build. An edge must expose // `consumerPackageIndex`, `dependencyPackageIndex`, `reexport`, -// `requestedTools` and `hostModule`. +// `requestedTools`, `buildTable` and `wantsLibrary`. struct Propagation { std::vector> visible; std::vector> exported; @@ -144,7 +144,8 @@ Propagation propagate(const std::vector& edges, std::size_t packageCount) s.insert(Provision{ Kind::DepDir, e.dependencyPackageIndex, {} }); for (auto const& t : e.requestedTools) s.insert(Provision{ Kind::Tool, e.dependencyPackageIndex, t }); - if (e.hostModule) + // The build program uses this dependency's library (mcpp#797). + if (e.buildTable && e.wantsLibrary) s.insert(Provision{ Kind::HostModule, e.dependencyPackageIndex, {} }); return s; }; diff --git a/modules/manifest/src/compat/registry.cppm b/modules/manifest/src/compat/registry.cppm new file mode 100644 index 00000000..26fffb3d --- /dev/null +++ b/modules/manifest/src/compat/registry.cppm @@ -0,0 +1,144 @@ +// mcpp.pm.compat.registry - every deprecated spelling or reading a user can +// write, in one table. +// +// docs/51 states that `mcpp.toml` keys are additive: an existing key keeps its +// meaning. A reading that is to be removed is an exception to that promise, +// and the exceptions are exactly the rows of `kDeprecations`. Each row states +// the release that deprecated the reading (`since`) and the CalVer month from +// which it is removed (`floor`). Everything a user sees about a deprecation is +// derived from the row: +// +// the warning deprecation_clause(): "deprecated since mcpp +// and removed from mcpp " +// `mcpp why compat` record_use() collects each use with its location +// docs/51 .github/tools/check_compat_registry.py checks that the +// table in docs/51 (en and zh) lists every row +// the code every `COMPAT()` marker names a row, and every row +// has a marker (the same tool) +// the removal tests/unit/test_pm_compat.cpp fails once MCPP_VERSION +// reaches a row's floor, naming the id +// +// A COMPAT marker of the form `COMPAT(internal:)` names engine-internal +// compatibility that no manifest can spell (legacy index file names, older +// call signatures). It is not a promise to anyone and has no row. + +export module mcpp.pm.compat.registry; + +import std; + +export namespace mcpp::pm::compat { + +struct Deprecation { + std::string_view id; // stable, kebab-case; the marker's argument + std::string_view since; // the first release that warns + std::string_view floor; // CalVer `YYYY.M`: absent from every release >= floor + std::string_view what; // the reading, as a noun phrase + std::string_view canonical; // what to write instead + std::string_view domain; // the diagnostic domain of the warning + std::string_view docs; // where the reading and its replacement are described +}; + +// The month from which the readings deprecated in 2026.10 are removed. A row +// may name a later floor; docs/51 requires at least two whole calendar months +// between `since` and `floor`. +inline constexpr std::string_view kRemovalFloor = "2027.1"; + +inline constexpr Deprecation kDeprecations[] = { + { + "workspace-position", "2026.10.10.1", kRemovalFloor, + "a workspace root's own [toolchain], [indices], [profile.], " + "[target.] scalars or [xlings] reaching members by position", + "[workspace.toolchain], [workspace.indices], [workspace.profile.], " + "[workspace.target.], [workspace.xlings]", + "manifest/workspace-position", "docs/07-workspace.md", + }, + { + "host-module-in-dependencies", "2026.10.11.1", kRemovalFloor, + "a [dependencies] entry that asks for `host-module = true` or for a " + "feature declaring `rule_module`, read as a build dependency", + "the same entry in [build-dependencies]", + "manifest/deprecated", "docs/05-dependencies.md", + }, + { + "plugin-imports-from-build-dependencies", "2026.10.11.1", kRemovalFloor, + "a build plugin's library importing a package it lists in its own " + "[build-dependencies] with `host-module = true`", + "the package in the plugin's [dependencies]", + "manifest/deprecated", "docs/31-authoring-a-rule-package.md", + }, +}; + +// The row named `id`, or null. +inline const Deprecation* find_deprecation(std::string_view id) { + for (auto const& d : kDeprecations) + if (d.id == id) return &d; + return nullptr; +} + +// "deprecated since mcpp 2026.10.10.1 and removed from mcpp 2027.1" +inline std::string deprecation_clause(const Deprecation& d) { + return std::format("deprecated since mcpp {} and removed from mcpp {}", d.since, d.floor); +} + +// The `(year, month)` of a version or of a floor: the first two dot-separated +// numbers. Both are CalVer, so their order is the order of these pairs. +inline std::optional> calver_month(std::string_view v) { + int parts[2] = {0, 0}; + for (int i = 0; i < 2; ++i) { + const auto dot = v.find('.'); + const auto head = v.substr(0, dot); + if (head.empty()) return std::nullopt; + auto [p, ec] = std::from_chars(head.data(), head.data() + head.size(), parts[i]); + if (ec != std::errc{} || p != head.data() + head.size()) return std::nullopt; + if (i == 0 && dot == std::string_view::npos) return std::nullopt; + v = dot == std::string_view::npos ? std::string_view{} : v.substr(dot + 1); + } + return std::pair{parts[0], parts[1]}; +} + +// Whether a release of `version` is at or past the floor of `d`. +inline bool past_floor(const Deprecation& d, std::string_view version) { + auto v = calver_month(version), f = calver_month(d.floor); + return v && f && *v >= *f; +} + +// ── Uses of a deprecated reading in this process (`mcpp why compat`) ────── + +struct DeprecationUse { + std::string id; + std::string where; // the manifest that wrote it + std::string detail; // the table, key or entry, as the warning names it + bool warned = false; // false: in a manifest the user cannot edit +}; + +// Recorded by every reader that applies a deprecated reading, warned or not. +void record_use(std::string_view id, std::string where, std::string detail, bool warned); +// Every use recorded so far, de-duplicated, in the order first seen. +std::vector recorded_uses(); + +} // namespace mcpp::pm::compat + +namespace mcpp::pm::compat { + +namespace { +std::mutex& uses_mutex() { static std::mutex m; return m; } +std::vector& uses_store() { static std::vector v; return v; } +} // namespace + +void record_use(std::string_view id, std::string where, std::string detail, bool warned) { + std::lock_guard lock(uses_mutex()); + auto& v = uses_store(); + for (auto& u : v) + if (u.id == id && u.where == where && u.detail == detail) { + u.warned = u.warned || warned; + return; + } + v.push_back({std::string(id), std::move(where), std::move(detail), warned}); +} + +std::vector recorded_uses() { + std::lock_guard lock(uses_mutex()); + return uses_store(); +} + +} // namespace mcpp::pm::compat diff --git a/modules/manifest/src/compat/workspace_position.cppm b/modules/manifest/src/compat/workspace_position.cppm index 9cdc99d8..5d64189c 100644 --- a/modules/manifest/src/compat/workspace_position.cppm +++ b/modules/manifest/src/compat/workspace_position.cppm @@ -1,8 +1,8 @@ // mcpp.pm.compat.workspace_position - the workspace root's own tables, given // to its members by POSITION. // -// COMPAT(workspace-position): removed with the rest of mcpp.pm.compat in -// mcpp 1.0.0 (DEPRECATION SCHEDULE in compat.cppm). +// COMPAT(workspace-position): a row of the deprecation registry +// (compat/registry.cppm), which states when it is removed. // // Before 2026.10.10.1 a workspace root's `[toolchain]`, `[indices]`, // `[profile.]`, the scalar rows of `[target.]` and its @@ -37,6 +37,7 @@ export module mcpp.pm.compat.workspace_position; import std; import mcpp.manifest.types; +import mcpp.pm.compat.registry; export namespace mcpp::pm::compat { @@ -118,13 +119,16 @@ inherit_root_position_by_position(mcpp::manifest::Manifest& member, return used; } -// What the warning for one table says. +// What the warning for one table says. `xlings.overrides` is the overrides +// half of `[xlings]`, named apart because its example and its mirror differ. std::string position_warning(std::string_view table, std::string_view member) { std::string_view example = table == "toolchain" ? "[workspace.toolchain]\n default = \"llvm@23.1.3\"" : table == "indices" ? "[workspace.indices]\n acme = { path = \"vendor/index\" }" : table == "profile" ? "[workspace.profile.release]\n opt = 3" : table == "target" ? "[workspace.target.x86_64-linux-musl]\n linkage = \"static\"" + : table == "xlings.overrides" + ? "[workspace.xlings.overrides]\n \"xim:cmake\" = \"/usr/bin/cmake\"" : "[workspace.xlings.workspace]\n cmake = \"4.0.2\""; const std::string mirror = table == "profile" ? "[workspace.profile.]" : table == "target" ? "[workspace.target.]" @@ -134,10 +138,9 @@ std::string position_warning(std::string_view table, std::string_view member) { : std::format("[{}]", table); return std::format( "{} on the workspace root reaches {} by its position; this reading is " - "deprecated and is removed in mcpp 1.0.0. Write it as {} to give it to every " - "member, for example:\n {}", + "{}. Write it as {} to give it to every member, for example:\n {}", original, member.empty() ? std::string("every member") : std::format("member '{}'", member), - mirror, example); + deprecation_clause(*find_deprecation("workspace-position")), mirror, example); } } // namespace mcpp::pm::compat diff --git a/modules/manifest/src/dep_spec.cppm b/modules/manifest/src/dep_spec.cppm index cc72fc57..01b53191 100644 --- a/modules/manifest/src/dep_spec.cppm +++ b/modules/manifest/src/dep_spec.cppm @@ -76,19 +76,24 @@ struct DependencySpec { // takes the dependency's programs and not its code: nothing of the // package is linked into the consumer through this edge. std::vector artifacts; - // #355 step 5: compile this dependency's lib-root module interface FOR THE - // HOST and make it importable from the consumer's build.mcpp — the - // mechanism behind reusable build rules distributed as ordinary packages - // (`import mcpp.rules.protobuf;`), instead of a second, non-C++ rule DSL. + // `host-module`: the build program uses this dependency's LIBRARY -- its + // public modules are importable from build.mcpp, its public headers are + // visible, and its objects are linked (mcpp#797; SPEC-004 §12). The name is + // older than the meaning: it was written when only a lib-root module was + // compiled for the host, and a key in published descriptors is never + // renamed (SPEC-004 §5.2). // - // Compiled ALONGSIDE build.mcpp with the SAME flags, not by a separate - // sub-build. That is not an optimisation: a BMI is only usable by a - // compile that agrees with it on standard, dialect flags and compiler - // identity, and two independently-resolved builds have no reason to. The - // shared-compile construction makes that agreement structural rather than - // something to verify — the same class of failure as `module X CRC - // mismatch`, which this project has paid for before. + // In `[build-dependencies]` it defaults to true unless the edge lists + // `tools` or `artifacts`, so `hostModuleDeclared` tells the default from a + // statement. In `[dependencies]` it is the compatibility reading + // `host-module-in-dependencies`: the entry is read as a build dependency. + // + // The library is compiled in the build program's dependency graph, the + // same plan that compiles build.mcpp, so a BMI and the program importing it + // come from one flag producer by construction (the reason the old + // shared-compile path existed). bool hostModule = false; + bool hostModuleDeclared = false; // #359: re-export everything this edge provides at BUILD time — the tools // above, the host module above, and the dependency's directory — to this // package's own consumers, transitively. @@ -109,6 +114,7 @@ struct DependencySpec { // are private-only.) bool reexport = false; bool defaultFeatures = true; // consumer opt-out: `default-features = false` + bool defaultFeaturesDeclared = false; // A member edge (workspace design 2026-09-29 §15): the edge from a // workspace plan's root to a selected member. It puts the member in the // graph and links nothing into the root, as an `artifacts` edge does. diff --git a/modules/manifest/src/manifest.cppm b/modules/manifest/src/manifest.cppm index a6ba5605..877c2372 100644 --- a/modules/manifest/src/manifest.cppm +++ b/modules/manifest/src/manifest.cppm @@ -23,3 +23,4 @@ export import mcpp.manifest.flag_words; export import mcpp.manifest.cfg_selector; export import mcpp.manifest.key_registry; export import mcpp.pm.compat.workspace_position; +export import mcpp.pm.compat.registry; diff --git a/modules/manifest/src/toml.cppm b/modules/manifest/src/toml.cppm index 292570fe..d611d74f 100644 --- a/modules/manifest/src/toml.cppm +++ b/modules/manifest/src/toml.cppm @@ -2246,6 +2246,7 @@ std::expected parse_document(const t::Document& documen // the default pseudo-feature is not seeded for this dependency. if (auto it = sub.find("default-features"); it != sub.end() && it->second.is_bool()) { spec.defaultFeatures = it->second.as_bool(); + spec.defaultFeaturesDeclared = true; } // #355: `tools = ["protoc"]` — HOST binaries this consumer wants from // the dependency. Same shape as `features`, and deliberately on the @@ -2265,6 +2266,7 @@ std::expected parse_document(const t::Document& documen // module importable from build.mcpp (reusable rules as packages). if (auto it = sub.find("host-module"); it != sub.end() && it->second.is_bool()) { spec.hostModule = it->second.as_bool(); + spec.hostModuleDeclared = true; } // #359: `reexport = true` — hand this edge's build-time provisions // (tools, host module, dependency dir) on to THIS package's consumers. @@ -2288,6 +2290,20 @@ std::expected parse_document(const t::Document& documen spec.gitRev = it->second.as_string(); spec.gitRefKind = "branch"; } + // [workspace.dependencies] states a dependency's SOURCE (and the + // features every member asks of it). The other requests belong to a + // member's own edge, and one entry may be inherited into both + // [dependencies] and [build-dependencies], where they mean different + // things; they were dropped without a word (mcpp#797, W2). + if (section.starts_with("workspace.dependencies")) { + for (std::string_view key : {"host-module", "tools", "artifacts", "reexport", + "visibility", "linkage"}) + if (sub.contains(std::string(key))) + m.schemaWarnings.push_back(std::format( + "[{}.\"{}\"] '{}' is a request of a member's own edge and is not " + "inherited; write it in the member's entry beside `workspace = true`", + section, fqName, key)); + } if (auto it = sub.find("workspace"); it != sub.end() && it->second.is_bool() && it->second.as_bool()) { spec.inheritWorkspace = true; return {}; // version will be filled in by workspace merge diff --git a/src/build/prepare/config.cpp b/src/build/prepare/config.cpp index 84d2590c..f5a19777 100644 --- a/src/build/prepare/config.cpp +++ b/src/build/prepare/config.cpp @@ -634,15 +634,22 @@ unfolded_defines_error(const mcpp::manifest::Manifest& m) { // dependency, a member of a git-hosted workspace, and a member inside an // index package's archive. The same commit then compiles the same way in its // own checkout and in every consumer's graph. +// +// `reportPosition` is false for a member of the workspace the build itself is +// rooted in: the root received the same position-read entries and was told +// about them, so telling the dependency too reports a use that changes +// nothing (mcpp#797, R5). `[xlings.overrides]` never travels here: it belongs +// to the root of a build (R4). std::optional inherit_as_workspace_member(mcpp::manifest::Manifest& member, const mcpp::manifest::Manifest& workspace, const std::filesystem::path& workspaceRoot, - const std::filesystem::path& memberDir) { + const std::filesystem::path& memberDir, + bool reportPosition) { mcpp::project::inherit_workspace_package(member, workspace); mcpp::project::merge_workspace_deps(member, workspace, workspaceRoot); mcpp::project::inherit_workspace_build(member, workspace, workspaceRoot); - mcpp::project::inherit_workspace_xlings(member, workspace); + mcpp::project::inherit_workspace_xlings(member, workspace, reportPosition); mcpp::project::inherit_workspace_layer_rows(member, workspace, workspaceRoot); return mcpp::project::workspace_inheritance_error(member, memberDir); } diff --git a/src/build/prepare/features.cpp b/src/build/prepare/features.cpp index d899371e..023c9dbf 100644 --- a/src/build/prepare/features.cpp +++ b/src/build/prepare/features.cpp @@ -8,6 +8,7 @@ import mcpp.build.prepare_inputs; import std; import mcpp.diag; +import mcpp.log; import mcpp.build.refusal; import mcpp.build.version_floor; import mcpp.home; @@ -69,12 +70,28 @@ namespace mcpp::build { // function of PrepareState&, the same treatment #719 gave every closure // that a later phase needed. +// The feature request for one instance of a package, aggregated over the +// edges that reach that instance (Cargo semantics: requested features union, +// `default-features` stays on unless every edge opts out). The target +// instance is asked by ordinary edges from consumers that have a target +// instance; the build-program instance by `[build-dependencies]` edges and by +// ordinary edges from consumers that have a build-program instance (mcpp#797). +enum class Instance { Target, BuildProgram }; static std::pair, bool> -aggregatedRequest(PrepareState& state, std::size_t depPkgIndex) { +aggregatedRequest(PrepareState& state, std::size_t depPkgIndex, Instance instance) { std::vector feats; bool anyEdge = false, anyDefault = false; for (auto const& edge : state.dependencyEdges) { if (edge.dependencyPackageIndex != depPkgIndex) continue; + const auto c = edge.consumerPackageIndex; + const bool consumerTarget = c == 0 || (c < state.packages.size() + && !state.packages[c].buildTimeOnly); + const bool consumerHost = c < state.packages.size() + && state.packages[c].hostInstance; + const bool serves = instance == Instance::Target + ? !edge.buildTable && consumerTarget + : edge.buildTable || consumerHost; + if (!serves) continue; anyEdge = true; if (edge.defaultFeatures) anyDefault = true; for (auto const& f : edge.requestedFeatures) @@ -84,6 +101,74 @@ aggregatedRequest(PrepareState& state, std::size_t depPkgIndex) { return { std::move(feats), anyEdge ? anyDefault : true }; } +// WHICH INSTANCES EACH PACKAGE HAS (mcpp#797). +// +// A package has a target instance when a chain of ordinary edges reaches it +// from the root, and a build-program instance when a `[build-dependencies]` +// edge reaches it or a package that has one. The test used to be "is every +// edge into it a host-module or build-only edge", and a host-module request +// merged onto an ordinary edge then removed a library the target imports +// (R2). Stated as reachability, a package the project depends on directly +// keeps its target instance whatever else also reaches it, and a build +// dependency's own `[dependencies]` stay out of the target with it. +// +// Nothing is cleared from a manifest: `compilesHere` reads the result, and the +// link flags a package without a target instance contributed during +// resolution are withdrawn from the root's link line. +static void step6_instances(PrepareState& state) { + const auto n = state.packages.size(); + if (n == 0) return; + std::vector target(n, false), host(n, false); + std::vector stack{0}; + target[0] = true; + while (!stack.empty()) { + const auto p = stack.back(); + stack.pop_back(); + for (auto const& e : state.dependencyEdges) { + if (e.consumerPackageIndex != p || e.buildTable) continue; + const auto d = e.dependencyPackageIndex; + if (d >= n || target[d]) continue; + target[d] = true; + stack.push_back(d); + } + } + for (auto const& e : state.dependencyEdges) + if (e.buildTable && e.dependencyPackageIndex < n && !host[e.dependencyPackageIndex]) { + host[e.dependencyPackageIndex] = true; + stack.push_back(e.dependencyPackageIndex); + } + while (!stack.empty()) { + const auto p = stack.back(); + stack.pop_back(); + for (auto const& e : state.dependencyEdges) { + if (e.consumerPackageIndex != p) continue; + const auto d = e.dependencyPackageIndex; + if (d >= n || host[d]) continue; + host[d] = true; + stack.push_back(d); + } + } + for (std::size_t d = 1; d < n; ++d) { + state.packages[d].hostInstance = host[d]; + // A package no edge mentions is a bookkeeping gap, not a build + // dependency: it keeps its target instance. + state.packages[d].buildTimeOnly = !target[d] && host[d]; + auto const& pkg = state.packages[d].manifest.package; + mcpp::log::info("edges", std::format( + "{}{}: {}{}", pkg.namespace_.empty() ? "" : pkg.namespace_ + ".", pkg.name, + target[d] ? "target" : "", host[d] ? (target[d] ? " + build-program" : "build-program") + : "")); + } + for (auto const& [key, rec] : state.resolved) { + const auto p = rec.depIndex + 1; + if (p >= n || !state.packages[p].buildTimeOnly) continue; + auto& ldflags = state.m->buildConfig.ldflags; + for (auto const& flag : rec.linkFlagsAdded) + if (auto pos = std::ranges::find(ldflags, flag); pos != ldflags.end()) + ldflags.erase(pos); + } +} + static void step6_check_version_floors_closure(PrepareState& state) { // ─── Feature activation (Cargo-style, additive) ──────────────────── // activated(pkg) = pkg.[features].default ∪ features requested for it @@ -553,9 +638,17 @@ static std::expected step6_activate_features(PrepareState& st // activation AGREE with resolution (mergeActiveFeatureDeps, which reads // the true per-edge spec): a transitive dep's requested features and its // consumer's `default-features = false` are no longer silently dropped. + state.buildProgramFeaturesByPackage.assign(state.packages.size(), {}); for (std::size_t i = 1; i < state.packages.size(); ++i) { auto& pname = state.packages[i].manifest.package.name; - auto [req, depDefaultFeatures] = aggregatedRequest(state, i); + if (state.packages[i].hostInstance) { + auto [hreq, hdef] = aggregatedRequest(state, i, Instance::BuildProgram); + state.buildProgramFeaturesByPackage[i] = + feature_closure(state.packages[i].manifest, hreq, hdef); + } + auto [req, depDefaultFeatures] = aggregatedRequest( + state, i, state.packages[i].buildTimeOnly ? Instance::BuildProgram + : Instance::Target); if (!req.empty() && !state.packages[i].manifest.featuresMap.empty()) { for (auto& f : req) { if (state.packages[i].manifest.featuresMap.contains(f)) continue; @@ -620,9 +713,13 @@ static std::expected step6_device_extensions_and_rules(Prepar if (edge.consumerPackageIndex != ci) continue; if (edge.dependencyPackageIndex >= state.packages.size()) continue; auto const& dep = state.packages[edge.dependencyPackageIndex]; + // The features of the instance this edge reaches: a rule is + // requested through `[build-dependencies]` (mcpp#797). + const auto& byInstance = edge.buildTable ? state.buildProgramFeaturesByPackage + : state.activeFeaturesByPackage; const auto& depFeatures = - edge.dependencyPackageIndex < state.activeFeaturesByPackage.size() - ? state.activeFeaturesByPackage[edge.dependencyPackageIndex] + edge.dependencyPackageIndex < byInstance.size() + ? byInstance[edge.dependencyPackageIndex] : edge.requestedFeatures; for (auto const& f : depFeatures) { auto it = dep.manifest.featureDeviceExtensions.find(f); @@ -1248,86 +1345,6 @@ step6_host_module_registration(PrepareState& state) { } } - // A build rule is BUILD-TIME ONLY. Registering the module is not - // enough: the package is still an ordinary node of the consumer's - // graph, so its interface was ALSO compiled as a normal library and - // linked into the target. That is wrong on its own terms — a rule - // has no business in the consumer's binary — and it made the - // feature nearly unusable, because in that second compile the - // bundled `mcpp` module does not exist: any rule that actually used - // the API it exists to wrap died with `fatal error: module 'mcpp' - // not found` (2026.8.5.1). - // - // Emptying the source globs is how a package is removed from the - // compile set here — the same mechanism the feature-gated-sources - // drop above uses. Resolution is untouched: the package still lands - // on disk, which is what `resolve_lib_root_path` just read. - // - // Guarded on EVERY edge into the package being a host-module edge. - // A package can legitimately be both a rule and a library, and - // silently dropping its objects then would surface as an undefined - // reference far from here. (The predicate used to be "no consumer - // other than the root", which said the same thing only while the - // root was the only possible requester.) - // - // Stated as FORWARD REACHABILITY rather than as exclusion, and the - // difference is not cosmetic. - // - // The predicate used to be per-edge: "every in-edge into this - // package is a host-module edge". That is right about the rule - // itself and wrong about everything BEHIND it — a rule's own - // `[dependencies]` are reached by ordinary edges, so they kept - // their globs and were compiled and LINKED INTO THE CONSUMER'S - // BINARY, while the rule could not even import them. Both halves - // were wrong, and the sharper harm was that a rule's dependency - // versions took part in the consumer's real resolution, so a rule - // could create a version conflict in a project that never asked - // for one. - // - // Exclusion would also get the dual-role case backwards. A package - // the project depends on directly must stay in the target even - // when some rule's build dependencies also reach it: the - // build-time path never subtracts from what the project asked to - // link. Asking "can the target reach it" answers both cases with - // one rule and no special case. - { - auto reachable = [&](bool targetEdgesOnly) { - std::vector seen(state.packages.size(), false); - if (state.packages.empty()) return seen; - std::vector stack{0}; - seen[0] = true; - while (!stack.empty()) { - auto p = stack.back(); - stack.pop_back(); - for (auto const& e : state.dependencyEdges) { - if (e.consumerPackageIndex != p) continue; - if (targetEdgesOnly && (e.hostModule || e.buildOnly)) - continue; - auto d = e.dependencyPackageIndex; - if (d >= seen.size() || seen[d]) continue; - seen[d] = true; - stack.push_back(d); - } - } - return seen; - }; - const auto viaTarget = reachable(/*targetEdgesOnly=*/true); - const auto viaAny = reachable(/*targetEdgesOnly=*/false); - for (std::size_t d = 1; d < state.packages.size(); ++d) { - // `viaAny` is the guard that keeps this from acting on a - // package no edge ever mentioned. Such a package is a - // bookkeeping gap, not a build dependency, and clearing - // its sources would turn that gap into an undefined - // reference a long way from here. - if (viaTarget[d] || !viaAny[d]) continue; - state.packages[d].buildTimeOnly = true; - auto& dm = state.packages[d].manifest; - dm.buildConfig.sources.clear(); - dm.buildConfig.featureSources.clear(); - dm.modules.sources.clear(); - } - } - if (state.overrides.tool_depth >= mcpp::build::tool_store::kMaxDepth && !toolRequests.empty()) { return std::unexpected(std::format( @@ -1898,7 +1915,7 @@ static std::expected step6_dependency_build_programs(PrepareS // Same edge-graph aggregation as feature activation above, so a // dep build.mcpp sees the SAME active feature set the dep is built // with (incl. transitive requests / default-features opt-out). - auto [req, depDefaultFeatures] = aggregatedRequest(state, i); + auto [req, depDefaultFeatures] = aggregatedRequest(state, i, Instance::Target); auto dirSafe = [](std::string s) { for (auto& c : s) if (c == '/' || c == '\\' || c == ':') c = '_'; return s; @@ -2253,6 +2270,7 @@ static std::expected step6_capability_binding(PrepareState& s std::expected phase6_features_and_host_tools(PrepareState& state) { step6_check_version_floors_closure(state); + step6_instances(state); if (auto r = step6_activate_features(state); !r) return std::unexpected(r.error()); if (auto r = step6_device_extensions_and_rules(state); !r) return std::unexpected(r.error()); diff --git a/src/build/prepare/graph.cpp b/src/build/prepare/graph.cpp index 5859cf49..d3bc9a67 100644 --- a/src/build/prepare/graph.cpp +++ b/src/build/prepare/graph.cpp @@ -8,6 +8,7 @@ import mcpp.build.prepare_inputs; import std; import mcpp.diag; +import mcpp.log; import mcpp.build.refusal; import mcpp.xlings.address_set; import mcpp.build.version_floor; @@ -244,7 +245,7 @@ static void recordDependencyEdge( std::size_t consumerDepIndex, std::size_t dependencyPackageIndex, const mcpp::manifest::DependencySpec& spec, - bool buildOnly, + bool buildTable, const std::string& writtenKey) { const auto consumerPackageIndex = packageIndexForConsumer(consumerDepIndex); @@ -263,27 +264,50 @@ static void recordDependencyEdge( .key = writtenKey, .table = spec.declaredIn, }); + + // A REQUESTED FEATURE THAT IS A BUILD RULE IMPLIES THE LIBRARY. + // + // A feature declaring `rule_module` is usable only by a build program, + // so requesting it asks for the dependency's library there. Requiring + // `host-module` beside it was a second spelling of one fact. + bool ruleFeature = false; + { + auto const& depManifest = state.packages[dependencyPackageIndex].manifest; + for (auto const& f : spec.features) + if (depManifest.featureRuleModule.contains(f)) { ruleFeature = true; break; } + } + // COMPAT(host-module-in-dependencies): a `[dependencies]` entry asking + // for `host-module = true`, or for a rule feature, served the build + // program and not the target before the section decided the + // configuration (mcpp#797). It keeps that meaning, reported where the + // manifest can be edited. + if (!buildTable && ((spec.hostModuleDeclared && spec.hostModule) || ruleFeature)) { + buildTable = true; + state.reportCompatUse("host-module-in-dependencies", consumerPackageIndex, + std::format("[dependencies] {}", writtenKey)); + } + const bool wantsLibrary = buildTable + && (ruleFeature + || (spec.hostModuleDeclared ? spec.hostModule + : spec.tools.empty() && spec.artifacts.empty())); + + // One edge per (consumer, dependency, visibility, configuration). The + // same package written in [dependencies] and [build-dependencies] of + // one consumer is two edges and two instances (mcpp#797, R2): merging + // them made the build-program request read as "not in the target". const auto visibility = parseVisibility(spec.visibility); auto same = [&](const DependencyEdge& edge) { return edge.consumerPackageIndex == consumerPackageIndex && edge.dependencyPackageIndex == dependencyPackageIndex - && edge.visibility == visibility; + && edge.visibility == visibility + && edge.buildTable == buildTable; }; auto it = std::find_if(state.dependencyEdges.begin(), state.dependencyEdges.end(), same); if (it != state.dependencyEdges.end()) { - // One consumer naming one dependency in BOTH tables. The ordinary - // declaration wins, because the build-time path never subtracts - // from what the project asked to link — stating the rule the other - // way round would let a `[build-dependencies]` line quietly drop a - // library the target needs. - if (!buildOnly) it->buildOnly = false; - // AND THE SECOND DECLARATION'S REQUESTS ARE KEPT (#649 E7). Both - // declarations name one edge, so what each asks of the dependency - // is asked of that edge: this used to return here and lose the - // second one's `tools`, `features`, `host-module` and `reexport` - // without a word, under `--strict` too. The rule is the one - // `mergeActiveFeatureDeps` already applies to a feature's - // restatement: additive fields union, `default-features` stays on + // One consumer restating one dependency in one table (an + // unconditional entry and a `[target..*]` or `[feature-deps]` + // row): every declaration's requests are requested of the one edge + // (#649 E7). Additive fields union, `default-features` stays on // unless every declaration opts out. for (auto const& t : spec.tools) if (std::ranges::find(it->requestedTools, t) == it->requestedTools.end()) @@ -297,30 +321,10 @@ static void recordDependencyEdge( == it->requestedFeatures.end()) it->requestedFeatures.push_back(f); it->defaultFeatures = it->defaultFeatures || spec.defaultFeatures; - if (spec.hostModule) it->hostModule = true; - else if (dependencyPackageIndex < state.packages.size()) - for (auto const& f : spec.features) - if (state.packages[dependencyPackageIndex].manifest.featureRuleModule.contains(f)) { - it->hostModule = true; - break; - } + it->wantsLibrary = it->wantsLibrary || wantsLibrary; it->reexport = it->reexport || spec.reexport; return; } - // A REQUESTED FEATURE THAT IS A BUILD RULE IMPLIES `host-module`. - // - // `host-module = true` says "compile this dependency's interface unit - // for the host so my build program can import it", and a feature - // declaring `rule_module` has already said that is the only way to use - // it. Requiring both was a second spelling of one fact, and the failure - // when only the feature was written landed in the consumer's build as - // an unresolved import rather than in the line that was incomplete. - bool hostModule = spec.hostModule; - if (!hostModule && dependencyPackageIndex < state.packages.size()) { - auto const& depManifest = state.packages[dependencyPackageIndex].manifest; - for (auto const& f : spec.features) - if (depManifest.featureRuleModule.contains(f)) { hostModule = true; break; } - } state.dependencyEdges.push_back(DependencyEdge{ .consumerPackageIndex = consumerPackageIndex, .dependencyPackageIndex = dependencyPackageIndex, @@ -329,9 +333,9 @@ static void recordDependencyEdge( .defaultFeatures = spec.defaultFeatures, .requestedTools = spec.tools, .requestedArtifacts = spec.artifacts, - .hostModule = hostModule, + .buildTable = buildTable, + .wantsLibrary = wantsLibrary, .reexport = spec.reexport, - .buildOnly = buildOnly, }); } @@ -615,6 +619,7 @@ static std::expected mergeActiveFeatureDeps( == dst.features.end()) dst.features.push_back(f2); dst.hostModule = dst.hostModule || spec.hostModule; + dst.hostModuleDeclared = dst.hostModuleDeclared || spec.hostModuleDeclared; dst.reexport = dst.reexport || spec.reexport; } } @@ -1043,7 +1048,7 @@ step4b_identity_version_merge(PrepareState& state, WorklistItemCtx& ctx, if (!secPackage) return std::unexpected(secPackage.error()); state.packages.push_back(std::move(*secPackage)); recordDependencyEdge(state, item.consumerDepIndex, depPackageIndex, - spec, item.buildOnly, name); + spec, item.fromBuildTable, name); auto linkFlagsAdded = propagateLinkFlags(state, secStage, *state.dep_manifests.back()); ResolvedKey mangledKey{key.ns, mangledPackage}; @@ -1086,7 +1091,7 @@ step4b_identity_version_merge(PrepareState& state, WorklistItemCtx& ctx, // no re-fetch needed; just record this consumer edge. recordDependencyEdge(state, item.consumerDepIndex, it->second.depIndex + 1, - spec, item.buildOnly, name); + spec, item.fromBuildTable, name); return {}; } @@ -1146,7 +1151,7 @@ step4b_identity_version_merge(PrepareState& state, WorklistItemCtx& ctx, state.packages[it->second.depIndex + 1] = std::move(*mergedPackage); recordDependencyEdge(state, item.consumerDepIndex, it->second.depIndex + 1, - spec, item.buildOnly, name); + spec, item.fromBuildTable, name); it->second.version = *merged; it->second.linkFlagsAdded = std::move(linkFlagsAdded); @@ -1284,7 +1289,7 @@ step4b_handle_already_resolved(PrepareState& state, WorklistItemCtx& ctx, if (it->second.depIndex + 1 < state.packages.size()) { recordDependencyEdge(state, item.consumerDepIndex, it->second.depIndex + 1, - spec, item.buildOnly, name); + spec, item.fromBuildTable, name); } return {}; } @@ -1360,7 +1365,7 @@ step4b_handle_already_resolved(PrepareState& state, WorklistItemCtx& ctx, if (it->second.depIndex + 1 < state.packages.size()) { recordDependencyEdge(state, item.consumerDepIndex, it->second.depIndex + 1, - spec, item.buildOnly, name); + spec, item.fromBuildTable, name); } return {}; return {}; @@ -1610,8 +1615,10 @@ step4b_acquire_dependency_source(PrepareState& state, WorklistItemCtx& ctx) { // git-hosted workspace inherits from ITS repository, anchored at // the clone. auto inheritAsMember = [&](const mcpp::manifest::Manifest& ws, - const std::filesystem::path& wsRoot) { - return inherit_as_workspace_member(*dep_manifest, ws, wsRoot, dep_root); + const std::filesystem::path& wsRoot, + bool reportPosition) { + return inherit_as_workspace_member(*dep_manifest, ws, wsRoot, dep_root, + reportPosition); }; // A rooted workspace's own package, the member "." of a workspace // plan (§15), is the workspace's manifest: it reads its own @@ -1623,13 +1630,15 @@ step4b_acquire_dependency_source(PrepareState& state, WorklistItemCtx& ctx) { mcpp::project::merge_workspace_deps(*dep_manifest, *state.wsManifest, state.runtimeWorkspaceRoot); mcpp::project::inherit_as_root_package(*dep_manifest, - state.runtimeWorkspaceRoot); + state.runtimeWorkspaceRoot, + /*asBuildRoot=*/false); } else if (depIsMember) { - if (auto bad = inheritAsMember(*state.wsManifest, state.runtimeWorkspaceRoot)) + if (auto bad = inheritAsMember(*state.wsManifest, state.runtimeWorkspaceRoot, + /*reportPosition=*/false)) return std::unexpected(*bad); } else if (!gitMember.empty()) { if (auto rm = mcpp::manifest::load(gitMemberCloneRoot / "mcpp.toml")) { - if (auto bad = inheritAsMember(*rm, gitMemberCloneRoot)) + if (auto bad = inheritAsMember(*rm, gitMemberCloneRoot, /*reportPosition=*/true)) return std::unexpected(*bad); } } @@ -1824,7 +1833,7 @@ step4b_finalize_dependency(PrepareState& state, WorklistItemCtx& ctx) { } state.packages.push_back(std::move(*depPackage)); recordDependencyEdge(state, item.consumerDepIndex, depPackageIndex, spec, - item.buildOnly, name); + item.fromBuildTable, name); // Record this dep as resolved so future encounters of the same // (ns, name) hit the fast path (skip / merge / conflict). @@ -1923,7 +1932,8 @@ step4b_finalize_dependency(PrepareState& state, WorklistItemCtx& ctx) { state.worklist.push_back({child_name, childReq, thisDepLabel + " (build-dep)", childReq.version, selfIdx, ctx.dep_root, - item.devOnly, /*buildOnly=*/true}); + item.devOnly, /*buildOnly=*/true, + /*fromBuildTable=*/true}); } return {}; } @@ -2028,10 +2038,41 @@ static void step4b_define_lookup_closures(PrepareState& state) { }; // A selected workspace member is compiled here whatever its targets are // (workspace design 2026-09-29 §15). + // + // NOT a package with no target instance (mcpp#797): one reached only + // through `[build-dependencies]` is compiled in the build program's graph, + // not here. `targetInstance` is decided by `step6_instances`; until then + // every package counts. state.compilesHere = [&](std::size_t i) { + if (i < state.packages.size() && i > 0 && state.packages[i].buildTimeOnly) + return false; return i == 0 || !state.isProgramOnlyPackage(state.packages[i].manifest) || state.isArtifactPackage(i) || state.packages[i].selectedMember; }; + state.editablePackage = [&](std::size_t i) { + if (i == 0 || i >= state.packages.size()) return true; + auto const& pkg = state.packages[i]; + if (pkg.selectedMember || state.isWorkspaceMemberPackage(i)) return true; + if (mcpp::build::path_is_under_any(pkg.root, state.storeRoots)) return false; + const auto home = !state.runtimeWorkspaceRoot.empty() ? state.runtimeWorkspaceRoot + : *state.root; + auto rel = pkg.root.lexically_normal().lexically_relative(home.lexically_normal()); + return !rel.empty() && *rel.begin() != ".."; + }; + state.reportCompatUse = [&](std::string_view id, std::size_t i, std::string detail) { + auto const* d = mcpp::pm::compat::find_deprecation(id); + if (!d) return; + const bool warn = state.editablePackage(i); + const auto where = i < state.packages.size() + ? (state.packages[i].root / "mcpp.toml").string() : std::string{}; + mcpp::pm::compat::record_use(id, where, detail, warn); + mcpp::log::info("compat", std::format("{} {}: {}{}", id, where, detail, + warn ? "" : " (not editable here; not warned)")); + if (!warn) return; + mcpp::diag::warning(d->domain, std::format( + "{}: {} is {}. Write {} instead.", where, detail, mcpp::pm::compat::deprecation_clause(*d), + d->canonical)); + }; state.appendUniquePath = @@ -2263,6 +2304,10 @@ step4b_define_provisioning_closures(PrepareState& state) { } auto& consumer = state.packages[edge.consumerPackageIndex]; auto const& dependency = state.packages[edge.dependencyPackageIndex]; + // A build dependency serves the build program, which is planned + // in its own graph: its include directories and interface + // defines do not reach this consumer's target compile (mcpp#797). + if (edge.buildTable) continue; // A package of programs publishes no usage requirements to its // consumers (#649 E6): nothing of it is compiled or linked here. if (edge.dependencyPackageIndex > 0 @@ -2437,7 +2482,7 @@ std::expected phase4b_graph_worklist(PrepareState& state) { injectCliForwards(n, req); state.worklist.push_back({n, req, mainPkgLabel + " (build-dep)", req.version, kMainConsumer, {}, /*devOnly=*/false, - /*buildOnly=*/true}); + /*buildOnly=*/true, /*fromBuildTable=*/true}); } diff --git a/src/build/prepare/graph_load.cpp b/src/build/prepare/graph_load.cpp index 544faaac..6ab48918 100644 --- a/src/build/prepare/graph_load.cpp +++ b/src/build/prepare/graph_load.cpp @@ -111,8 +111,13 @@ static void step4a_define_split_and_identity_closures(PrepareState& state) { const std::size_t rootClaims = claims.size(); for (std::size_t i = 0; i < state.packages.size(); ++i) { const auto& man = state.packages[i].manifest; - const auto feats = i < state.activeFeaturesByPackage.size() + // Both instances' features: a payload a feature declares serves + // this build whichever instance activated it (mcpp#797). + auto feats = i < state.activeFeaturesByPackage.size() ? state.activeFeaturesByPackage[i] : std::vector{}; + if (i < state.buildProgramFeaturesByPackage.size()) + for (auto const& f : state.buildProgramFeaturesByPackage[i]) + if (std::ranges::find(feats, f) == feats.end()) feats.push_back(f); for (auto const& spec : applicable_xlings_addresses( man, feats, state.toolPurpose, /*isRoot=*/i == 0 || state.packages[i].selectedMember)) { diff --git a/src/build/prepare/manifest.cpp b/src/build/prepare/manifest.cpp index 72de01f3..4b487e03 100644 --- a/src/build/prepare/manifest.cpp +++ b/src/build/prepare/manifest.cpp @@ -157,17 +157,30 @@ report_manifest_statements(const mcpp::manifest::Manifest& m, bool strict) { // What the workspace root's own manifest states about itself, reported once // for the command that plans its members (SPEC-004 §9.10): // W7 a package table on a root without [package] acts on no package -- -// warning, an error under --strict; +// warning, an error under --strict; so does a `build.mcpp` beside such +// a root, which no package owns and which never runs (mcpp#797, W1); // the workspace layer read by an engine older than this one is ignored by // it without a word (2026.10.8.1 builds with a different toolchain), so a // root that writes `[workspace.X]` without stating this engine as its floor // gets a note. static std::expected -report_workspace_root(const mcpp::manifest::Manifest& ws, bool strict) { +report_workspace_root(const mcpp::manifest::Manifest& ws, const std::filesystem::path& wsRoot, + bool strict) { for (auto const& w : ws.virtualRootPackageTables) { if (strict) return std::unexpected(w); mcpp::diag::warning("manifest/schema", w); } + if (ws.package.name.empty()) { + std::error_code ec; + if (std::filesystem::is_regular_file(wsRoot / "build.mcpp", ec)) { + const std::string w = + "build.mcpp is written beside a workspace root without [package], so it " + "belongs to no package and does not run; move it into the member whose " + "build it configures"; + if (strict) return std::unexpected(w); + mcpp::diag::warning("manifest/schema", w); + } + } if (ws.layer) { const auto have = mcpp::xpkg_version::parse(mcpp::MCPP_VERSION); const auto floor = ws.workspace.inherited.mcppFloor; @@ -191,7 +204,7 @@ static std::expected select_workspace_members(PrepareState& state, const std::filesystem::path& wsRoot, mcpp::manifest::Manifest ws, const std::vector& selection) { - if (auto r = report_workspace_root(ws, state.overrides.strict); !r) return r; + if (auto r = report_workspace_root(ws, wsRoot, state.overrides.strict); !r) return r; const auto all = mcpp::project::workspace_members(ws, wsRoot); { auto first = mcpp::project::load_member_manifest(ws, wsRoot, selection.front()); diff --git a/src/build/prepare/scan.cpp b/src/build/prepare/scan.cpp index 14ceb30e..c7e33bb9 100644 --- a/src/build/prepare/scan.cpp +++ b/src/build/prepare/scan.cpp @@ -91,7 +91,7 @@ static std::expected step11_scan_sources(PrepareState& state) }; std::map> codeEdges; for (auto const& e : state.dependencyEdges) { - if (!e.requestedArtifacts.empty() || e.buildOnly) continue; + if (!e.requestedArtifacts.empty() || e.buildTable) continue; codeEdges[e.consumerPackageIndex].push_back(e.dependencyPackageIndex); } for (std::size_t i = 0; i < state.packages.size(); ++i) { diff --git a/src/build/prepare/state.cppm b/src/build/prepare/state.cppm index ceef6022..bbd62bcb 100644 --- a/src/build/prepare/state.cppm +++ b/src/build/prepare/state.cppm @@ -130,18 +130,25 @@ constexpr std::size_t kMainConsumer = static_cast(-1); // THIS plan for its target. A package asked for them is scanned and // configured here even when every target it declares is a program. std::vector requestedArtifacts; - // #355 step 5 / #359: does this edge ask for the dependency's lib-root - // interface as a HOST module, and does it hand its build-time - // provisions on to this consumer's own consumers? - bool hostModule = false; + // WHICH CONFIGURATION THE EDGE SERVES, AND WHAT IT ASKS FOR (mcpp#797). + // + // `buildTable`: the declaration is in `[build-dependencies]` (or is a + // compatibility reading of one, `host-module-in-dependencies`), so the + // edge serves the consumer's BUILD PROGRAM and never its target. It is + // the declaration's own table and is not inherited: a build + // dependency's `[dependencies]` are ordinary edges, and they serve the + // build program only because the package they start from does. Which + // packages have a target instance follows from reachability over the + // edges that are not `buildTable` (`step6_instances`). + // + // `wantsLibrary`: on a `buildTable` edge, the build program uses the + // dependency's library (`host-module`, default true unless the edge + // lists `tools` or `artifacts`, implied by a `rule_module` feature). + // On an ordinary edge the library is what the edge means, and the + // field is not read. + bool buildTable = false; + bool wantsLibrary = false; bool reexport = false; - // Did this edge come from `[build-dependencies]`? Such an edge serves - // the BUILD and never the target, and the property is inherited by - // everything reachable through it. It is a property of the edge and - // not of the package: the same package may be an ordinary dependency - // of someone else in the same build, and then it does reach the - // target. - bool buildOnly = false; }; struct GraphRequest { @@ -220,6 +227,10 @@ constexpr std::size_t kMainConsumer = static_cast(-1); // target that also provides a tool is written once, in // `[dependencies]`, with a `tools` request on it. bool buildOnly = false; + // The item was declared in its consumer's `[build-dependencies]`. NOT + // inherited, unlike `buildOnly`: it is the table of this one + // declaration, which is what an edge records (DependencyEdge). + bool fromBuildTable = false; }; struct DeclaringManifest { @@ -560,7 +571,15 @@ struct PrepareState { // the moment that former's phase returns, which the compiler cannot // detect (nothing outside the escaping closure's own body names them). std::vector packages; + // The features active in the instance this plan compiles: the target + // instance's where the package has one, the build-program instance's + // otherwise (mcpp#797). A request made only through `[build-dependencies]` + // does not reach a target instance (R3). std::vector> activeFeaturesByPackage; + // The features of each package's build-program instance (empty where it + // has none). The build program's graph activates the same set; payloads a + // feature declares serve this build in either instance. + std::vector> buildProgramFeaturesByPackage; // #734 E7: per consumer, the dormant features of its host-module providers. std::map> dormantFeaturesByConsumer; @@ -642,6 +661,16 @@ struct PrepareState { std::function(std::size_t)> bareBindingsFor; std::function compilesHere; + // A use of a deprecated reading (mcpp.pm.compat.registry) by the manifest + // of package `i`: warned where that manifest can be edited -- the root, a + // workspace member, a path package outside the package store -- and + // recorded for `mcpp why compat` either way. + std::function + reportCompatUse; + // Whether the manifest of package `i` is one the person running the build + // can edit (SPEC-008 §4: a diagnostic is given to the package that can act + // on it). + std::function editablePackage; std::function isProgramOnlyPackage; std::function&, const std::vector&)> appendUniqueFlags; @@ -820,7 +849,8 @@ std::optional inherit_as_workspace_member(mcpp::manifest::Manifest& member, const mcpp::manifest::Manifest& workspace, const std::filesystem::path& workspaceRoot, - const std::filesystem::path& memberDir); + const std::filesystem::path& memberDir, + bool reportPosition = true); std::optional> workspace_listing(const std::filesystem::path& memberDir, const std::filesystem::path& bound); diff --git a/src/modgraph/scanner.cppm b/src/modgraph/scanner.cppm index 7e5736e1..0e8a7048 100644 --- a/src/modgraph/scanner.cppm +++ b/src/modgraph/scanner.cppm @@ -141,9 +141,14 @@ struct PackageRoot { UsageRequirements publicUsage; UsageRequirements linkUsage; bool usageResolved = false; - // Reached only through build-time edges: resolved for host tools/rules, - // but neither its sources nor its products belong to the target plan. + // No TARGET INSTANCE (mcpp#797): no chain of ordinary edges reaches it + // from the root. It is resolved here -- one version per package, and its + // tools, directory and payloads serve this build -- and it is compiled in + // the build program's graph, not in this plan. bool buildTimeOnly = false; + // A BUILD-PROGRAM INSTANCE: reached through a `[build-dependencies]` edge + // and then any edges. A package may have both instances (mcpp#797, R2). + bool hostInstance = false; // A selected workspace member that the plan builds (workspace design // 2026-09-29 §15), and the directory below `bin/` its products are placed // in (empty: `bin/` itself). False on the plan's root and on every diff --git a/src/project.cppm b/src/project.cppm index 9945b985..9176c708 100644 --- a/src/project.cppm +++ b/src/project.cppm @@ -74,6 +74,18 @@ export void merge_workspace_deps(mcpp::manifest::Manifest& member, spec.git = wsSpec.git; spec.gitRev = wsSpec.gitRev; spec.gitRefKind = wsSpec.gitRefKind; + // The features every member asks of it come first and a member's own + // follow; `default-features` is the member's when it states one + // (Cargo's rule). They were not copied, so a feature written in + // [workspace.dependencies] was dropped without a word (mcpp#797, W2). + std::vector features = wsSpec.features; + for (auto const& f : spec.features) + if (std::ranges::find(features, f) == features.end()) features.push_back(f); + spec.features = std::move(features); + if (!spec.defaultFeaturesDeclared && wsSpec.defaultFeaturesDeclared) { + spec.defaultFeatures = wsSpec.defaultFeatures; + spec.defaultFeaturesDeclared = true; + } if (!spec.path.empty() && !wsRoot.empty()) { std::filesystem::path p(spec.path); if (p.is_relative()) { @@ -195,14 +207,24 @@ bool lists(std::span names, std::string_view name) { // no other reading of these tables, so every use is reported; a root with one // reads them as its own package's first, and only a member that actually // received a value through the old rule is told. +// +// `report` is false where the value received changes nothing in the build: a +// member reached as a dependency of a build whose root is in the same +// workspace, which received the same entries itself (mcpp#797, R5). The use +// is still recorded for `mcpp why compat`. void warn_position(const mcpp::manifest::Manifest& workspace, const mcpp::manifest::Manifest& member, - std::string_view table, bool received) { + std::string_view table, bool received, bool report = true) { const bool virtualRoot = workspace.package.name.empty(); if (!virtualRoot && !received) return; - mcpp::diag::warning("manifest/workspace-position", - mcpp::pm::compat::position_warning( - table, virtualRoot ? std::string_view{} : member.package.name)); + const auto text = mcpp::pm::compat::position_warning( + table, virtualRoot ? std::string_view{} : member.package.name); + mcpp::pm::compat::record_use("workspace-position", workspace.sourcePath.string(), + std::format("[{}] -> {}", table, + virtualRoot ? std::string("every member") + : member.package.name), + report); + if (report) mcpp::diag::warning("manifest/workspace-position", text); } // `[xlings]` entries and `[target..xlings]` rows from `top` and `rows`, @@ -213,9 +235,18 @@ void warn_position(const mcpp::manifest::Manifest& workspace, // the emitter's per-platform view (`workspaceByPlatform`), so a published // member's descriptor states only what the member itself declared. Returns // whether the member received anything. +// +// TWO PARTS, TWO POSITIONS (mcpp#797, R4). The entries say which payloads a +// package needs and travel in every position (W5). `[xlings.overrides]` says +// where a payload comes from on THIS machine: it belongs to the root of a +// build, and a dependency stating one is refused (`dependency_override_ +// refusal`). Copying the workspace's overrides into a member reached as a +// dependency made a workspace's own statement fail its own build. +enum class XlingsPart { Entries, Overrides }; bool merge_xlings(mcpp::manifest::Manifest& member, const mcpp::manifest::XlingsConfig* top, - std::span rows) { + std::span rows, + XlingsPart part) { // `(namespace, name)`, the identity `mcpp.xlings.address_set` defines, // spelled here from the same parser rather than imported. Importing that // module here makes GCC 16.1 fail with an internal compiler error @@ -234,6 +265,14 @@ bool merge_xlings(mcpp::manifest::Manifest& member, // with the pin and the tier each address carries. auto take = [&](const mcpp::manifest::XlingsConfig& from, mcpp::manifest::XlingsConfig& to) { + if (part == XlingsPart::Overrides) { + // `[xlings.overrides]` states where a payload comes from on THIS + // machine, so a member built as the root of its own build takes + // the workspace's statement unless it makes its own (mcpp#755). + for (auto const& [pkg, o] : from.overrides) + if (to.overrides.try_emplace(pkg, o).second) received = true; + return; + } std::vector taken; for (auto const& a : from.deps) { if (own.contains(package_key(a))) continue; @@ -247,11 +286,6 @@ bool merge_xlings(mcpp::manifest::Manifest& member, } received = received || !taken.empty(); to.deps.insert(to.deps.begin(), taken.begin(), taken.end()); - // `[xlings.overrides]` states where a payload comes from on THIS - // machine, so a member built as the root of its own build takes the - // workspace's statement unless it makes its own (mcpp#755). - for (auto const& [pkg, o] : from.overrides) - if (to.overrides.try_emplace(pkg, o).second) received = true; }; if (top) take(*top, member.xlings); @@ -349,14 +383,22 @@ void merge_layer_root_position(mcpp::manifest::Manifest& member, // An xlings entry describes the environment a build runs in, so it is // inherited without an opt-in; only dependencies, which are graph edges, need // `.workspace = true` (`merge_workspace_deps`). +// +// The entries only: `[xlings.overrides]` is a root-position table +// (`inherit_workspace_xlings_overrides`). `reportPosition` is false for a +// member reached as a dependency of a build rooted in the same workspace, whose +// root already received these entries (R5). export void inherit_workspace_xlings(mcpp::manifest::Manifest& member, - const mcpp::manifest::Manifest& workspace) { + const mcpp::manifest::Manifest& workspace, + bool reportPosition = true) { static const std::vector none; + using detail::XlingsPart; if (auto const* layer = workspace.layer.get()) { auto const& tables = workspace.workspace.layerTables; detail::merge_xlings(member, detail::lists(tables, "xlings") ? &layer->xlings : nullptr, - detail::lists(tables, "target") ? layer->conditionalConfigs : none); + detail::lists(tables, "target") ? layer->conditionalConfigs : none, + XlingsPart::Entries); } // COMPAT(workspace-position): the root's own [xlings] and [target..xlings]. const auto position = mcpp::pm::compat::position_tables(workspace); @@ -364,8 +406,36 @@ export void inherit_workspace_xlings(mcpp::manifest::Manifest& member, const bool rows = detail::lists(position, "target"); if (!top && !rows) return; const bool received = detail::merge_xlings(member, top ? &workspace.xlings : nullptr, - rows ? workspace.conditionalConfigs : none); - if (top) detail::warn_position(workspace, member, "xlings", received); + rows ? workspace.conditionalConfigs : none, + XlingsPart::Entries); + if (top && !workspace.xlings.deps.empty()) + detail::warn_position(workspace, member, "xlings", received, reportPosition); +} + +// `[workspace.xlings.overrides]` and `[workspace.target..xlings.overrides]`, +// and by position the root's own `[xlings.overrides]`, for a member that is +// the root of a build (mcpp#797, R4). +export void inherit_workspace_xlings_overrides(mcpp::manifest::Manifest& member, + const mcpp::manifest::Manifest& workspace) { + static const std::vector none; + using detail::XlingsPart; + if (auto const* layer = workspace.layer.get()) { + auto const& tables = workspace.workspace.layerTables; + detail::merge_xlings(member, + detail::lists(tables, "xlings") ? &layer->xlings : nullptr, + detail::lists(tables, "target") ? layer->conditionalConfigs : none, + XlingsPart::Overrides); + } + // COMPAT(workspace-position) + const auto position = mcpp::pm::compat::position_tables(workspace); + const bool top = detail::lists(position, "xlings"); + const bool rows = detail::lists(position, "target"); + if (!top && !rows) return; + const bool received = detail::merge_xlings(member, top ? &workspace.xlings : nullptr, + rows ? workspace.conditionalConfigs : none, + XlingsPart::Overrides); + if (top && !workspace.xlings.overrides.empty()) + detail::warn_position(workspace, member, "xlings.overrides", received); } // The rows of `[workspace.target.]` -- `.build`, `.abi`, `.runtime` -- @@ -405,6 +475,7 @@ export void inherit_workspace_root_position(mcpp::manifest::Manifest& member, const std::filesystem::path& wsRoot) { if (auto const* layer = workspace.layer.get()) detail::merge_layer_root_position(member, *layer, wsRoot, workspace.sourcePath); + inherit_workspace_xlings_overrides(member, workspace); // COMPAT(workspace-position) const auto position = mcpp::pm::compat::position_tables(workspace); if (position.empty()) return; @@ -445,8 +516,13 @@ void inherit_workspace_indices(mcpp::manifest::Manifest& member, // compiles the same way in every selection. Its own `[toolchain]`, // `[target.*]`, ... are its own declarations and come first; its // `[workspace.dependencies]` are merged explicitly by the caller. +// +// `asBuildRoot` is false when the package is reached as a dependency of a +// member's build: it then takes the workspace's entries and not its +// `[xlings.overrides]`, which belong to the build's root (R4). export void inherit_as_root_package(mcpp::manifest::Manifest& root, - const std::filesystem::path& wsRoot) { + const std::filesystem::path& wsRoot, + bool asBuildRoot = true) { if (!root.workspace.present || root.package.name.empty()) return; if (root.inheritedAsRootPackage) return; root.inheritedAsRootPackage = true; @@ -460,7 +536,13 @@ export void inherit_as_root_package(mcpp::manifest::Manifest& root, inherit_workspace_layer_rows(root, workspace, wsRoot); detail::merge_xlings(root, detail::lists(tables, "xlings") ? &layer->xlings : nullptr, - detail::lists(tables, "target") ? layer->conditionalConfigs : none); + detail::lists(tables, "target") ? layer->conditionalConfigs : none, + detail::XlingsPart::Entries); + if (asBuildRoot) + detail::merge_xlings(root, + detail::lists(tables, "xlings") ? &layer->xlings : nullptr, + detail::lists(tables, "target") ? layer->conditionalConfigs : none, + detail::XlingsPart::Overrides); } } From 74324fe6ddc8567ac01d1aedecec5dac90cac41b Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sun, 11 Oct 2026 06:32:30 +0900 Subject: [PATCH 03/12] Build programs are compiled by their own graph: host instances, engine package, plugin roots; remove the single-unit host module path (#797) --- modules/manifest/src/xpkg.cppm | 5 +- modules/source-kind/src/source_kind.cppm | 13 + src/build/build_program.cppm | 1029 ++-------------------- src/build/engine_package.cppm | 162 ++++ src/build/host_module_compile.cppm | 622 ------------- src/build/host_module_store.cppm | 381 -------- src/build/hostprogram.cppm | 5 +- src/build/ninja_backend.cppm | 7 + src/build/plan.cppm | 6 +- src/build/prepare.cppm | 13 + src/build/prepare/features.cpp | 364 ++------ src/build/prepare/graph.cpp | 45 + src/build/prepare/host_graph.cpp | 863 ++++++++++++++++++ src/build/prepare/local_toolchain.cpp | 15 +- src/build/prepare/plan.cpp | 1 + src/build/prepare/scan.cpp | 43 + src/build/prepare/sources.cpp | 8 +- src/build/prepare/state.cppm | 47 +- src/build/prepare/target_side.cpp | 130 +-- src/build/prepare/toolchain.cpp | 12 + src/build/source_digest.cppm | 126 +++ src/pack/library_pipeline.cppm | 9 + tests/unit/test_host_module_store.cpp | 348 -------- 23 files changed, 1489 insertions(+), 2765 deletions(-) create mode 100644 src/build/engine_package.cppm delete mode 100644 src/build/host_module_compile.cppm delete mode 100644 src/build/host_module_store.cppm create mode 100644 src/build/prepare/host_graph.cpp create mode 100644 src/build/source_digest.cppm delete mode 100644 tests/unit/test_host_module_store.cpp diff --git a/modules/manifest/src/xpkg.cppm b/modules/manifest/src/xpkg.cppm index 782accd9..3b960e47 100644 --- a/modules/manifest/src/xpkg.cppm +++ b/modules/manifest/src/xpkg.cppm @@ -1861,7 +1861,7 @@ synthesize_from_xpkg_lua(std::string_view luaContent, // original comment was describing. std::string dver; std::vector dtools; - bool dhostModule = false, dreexport = false; + bool dhostModule = false, dreexport = false, dhostModuleDeclared = false; if (cur.peek() == '{') { cur.consume('{'); cur.skip_ws_and_comments(); @@ -1889,7 +1889,7 @@ synthesize_from_xpkg_lua(std::string_view luaContent, // two spellings of ONE schema, so a key accepted in // one must not be an unknown-key error in the other. bool v = cur.read_bareword() == "true"; - if (dk == "host-module") dhostModule = v; + if (dk == "host-module") { dhostModule = v; dhostModuleDeclared = true; } else dreexport = v; } else { // Record rather than swallow — a descriptor author @@ -1911,6 +1911,7 @@ synthesize_from_xpkg_lua(std::string_view luaContent, spec.version = dver; spec.tools = std::move(dtools); spec.hostModule = dhostModule; + spec.hostModuleDeclared = dhostModuleDeclared; spec.reexport = dreexport; auto selector = parse_xpkg_dependency_selector( dname, m.sourcePath); diff --git a/modules/source-kind/src/source_kind.cppm b/modules/source-kind/src/source_kind.cppm index b59459c5..1d2f43eb 100644 --- a/modules/source-kind/src/source_kind.cppm +++ b/modules/source-kind/src/source_kind.cppm @@ -157,6 +157,14 @@ validate_module_extensions(std::span extras); SourceKind classify(const std::filesystem::path& p, const ExtensionTable& t); +// A BUILD PROGRAM'S SOURCE (`build.mcpp`) is a C++ translation unit whose +// extension no driver knows (mcpp#797). It is classified `Cxx`, and its +// compile and scan state the language explicitly (`-x c++`, `/TP`), which a +// driver would otherwise hand the file to the linker as a script. Only a +// plan's entry point can be one: the default source globs never match it. +inline constexpr std::string_view kBuildProgramExtension = ".mcpp"; +bool needs_explicit_cxx_language(const std::filesystem::path& p); + // ─── Predicates derived from the kind ──────────────────────────────────── // // These exist so that a consumer interested in ONE axis does not write its own @@ -458,6 +466,7 @@ SourceKind classify(const std::filesystem::path& p, const ExtensionTable& t) { for (auto const& m : t.moduleInterface) if (ext == m) return SourceKind::ModuleInterface; if (contains(kCxxExtensions, ext)) return SourceKind::Cxx; + if (ext == kBuildProgramExtension) return SourceKind::Cxx; if (contains(kCExtensions, ext)) return SourceKind::C; if (contains(kGasExtensions, ext)) return SourceKind::GasAsm; if (contains(kNasmExtensions, ext)) return SourceKind::NasmAsm; @@ -472,6 +481,10 @@ SourceKind classify(const std::filesystem::path& p, const ExtensionTable& t) { return SourceKind::Other; } +bool needs_explicit_cxx_language(const std::filesystem::path& p) { + return p.extension() == kBuildProgramExtension; +} + bool produces_bmi(SourceKind k) { return k == SourceKind::ModuleInterface; } bool links_unconditionally(SourceKind k) { return k == SourceKind::ModuleInterface; } diff --git a/src/build/build_program.cppm b/src/build/build_program.cppm index 14cdd8c8..fa7e3765 100644 --- a/src/build/build_program.cppm +++ b/src/build/build_program.cppm @@ -28,7 +28,6 @@ import mcpp.build.directives; // the directive definition table (own modu import mcpp.build.progress; // the program's line (build progress design 2026-09-29) import mcpp.build.refusal; // the machine-readable identity of a refusal import mcpp.build.hostprogram; // the bundled `mcpp` module's text (own module: see its header) -import mcpp.build.host_module_compile; // the bundled module and the host modules, compiled once per key (#748) import mcpp.build.resources; // compile_utf8_manifest — the build program speaks UTF-8 (#693) import mcpp.toolchain.hostflags; // the shared host-compile flag producer import mcpp.toolchain.linkmodel; // shared C-library / clang-cfg-bypass model @@ -122,16 +121,6 @@ struct BuildProgramEnv { // same producers the engine's own command lines read; a key absent here is // emitted empty. std::map buildInfo; - // #734 E7: the features of this program's host-module providers that are - // NOT enabled, with the files each would add. Read only when build.mcpp - // imports a module nothing provides, to name the feature that would; the - // files are not opened on a successful plan. - struct DormantFeature { - std::string package; - std::string feature; - std::vector files; - }; - std::vector dormantFeatures; // WHICH COMPILER RESOLVED — "gcc" | "clang" | "msvc" | "". // // A package should never have to guess this, and until this field existed @@ -368,63 +357,17 @@ struct BuildProgramEnv { // PREPENDED, NOT SUBSTITUTED. A build program legitimately calls `git`, // `python3` or a shell, none of which arrive this way. std::string toolsBin; - // #355 step 5: dependency-provided modules to compile FOR THE HOST and make - // importable from this build.mcpp — reusable build rules distributed as - // ordinary mcpp packages (`import mcpp.rules.protobuf;`) instead of a - // second, non-C++ rule DSL. - // - // Compiled with the SAME flags as build.mcpp itself, in the same directory, - // which is what makes the BMI usable at all — see DependencySpec::hostModule. - // - // ORDER IS LOAD-BEARING. The compile loop accumulates `moduleFlags` as it - // goes, so every entry sees the BMIs of the entries before it. That is the - // whole mechanism by which a rule can import another rule: the caller - // topologically sorts this list, and nothing else is required. - struct HostModuleRef { - std::string logical; - std::filesystem::path interface; - // False when this module is present ONLY because another rule imports - // it. Its BMI must exist for that rule to compile, but this build.mcpp - // never declared it, and a package's build-time provisions cross one - // further edge only on a `reexport = true` edge. GCC cannot enforce - // that — its BMIs are implicit under gcm.cache and reachable by name - // whatever the flags say — so mcpp enforces it, and does so on every - // platform rather than leaving the rule true only where the compiler - // happens to help. - bool importable = true; - // WHERE THE COMPILED MODULE MAY BE KEPT (#748, B1). An entry from an - // index package whose sources are in the immutable store goes to the - // global cache, where every project of the machine reuses it; everything - // else (a path or git dependency, a workspace member) is kept under the - // workspace's `target/`, because its sources can change without its name - // and version changing. The rule is the dependency cache's own - // (plan.cpp), and a host module is compiled alone, so the local taint it - // asks about is its own package's. - // - // The provider's identity is part of the entry's key either way: the - // index, name and version for an index package, and the manifest's own - // name and version otherwise. - std::string providerIndex; - std::string providerName; - std::string providerVersion; - bool immutableSource = false; - // The provider's package root. For a module whose sources can change in - // place, the tree below it is part of the entry's key: an edit to a header - // the interface includes is an edit to what was compiled. - std::filesystem::path providerRoot; - }; - std::vector hostModules; - // WHERE THE MODULES THIS PROGRAM IMPORTS ARE COMPILED TO (#748, B1). - // - // `moduleCacheRoot` is the global cache root, or empty when the build's cache - // mode (`--cache local` or `off`) asks for nothing to be written there. - // `moduleStore` is `/target/.build-mcpp/host-modules`, which the - // programs of one workspace share, so that a host module several members - // import is compiled once for all of them. Empty means the program's own - // artifacts directory, which is what a program that is not part of a - // workspace had before. - std::filesystem::path moduleCacheRoot; - std::filesystem::path moduleStore; + // THE PROGRAM, COMPILED (mcpp#797). The build programs' graph compiles + // every build program of a build in one plan, with its libraries, and + // hands each run its binary and the identity it was compiled from; the + // run cache keys on the identity, so a program whose graph changed runs + // again. `programCompile` is the compile's duration when this build made + // it. `programError` is a compile that failed, reported at the program's + // turn so that the first failure in run order is the one reported. + std::filesystem::path programBinary; + std::string programIdentity; + std::chrono::milliseconds programCompile{0}; + std::string programError; }; // The env-var name `hostprogram::xpkg_dir` reads back. One spelling of the @@ -546,61 +489,21 @@ bool mentions_missing_mcpp_api(std::string_view compilerOutput); std::vector> install_hook_env(const BuildProgramEnv& env); -// THE COMPILE OF A PROGRAM, MADE AHEAD OF ITS RUN (#748, B2). -// -// A workspace's programs run in an order: a member's program reads the directives -// of the programs of the members it depends on. Their compiles have none. This is -// what the compile leaves behind, so that `step9_member_build_programs` can -// compile every program that needs it at the same time, and then run the programs -// in the order they always ran in, each taking its compile from here. -// -// NOTHING IS REPORTED OR APPLIED BY THE COMPILE. A compile that failed leaves its -// message here, and the program's turn reports it, so the failure that reaches -// the user is the first in the order the programs run in, whichever compile -// finished first. The warnings and the refusal the compile produced are kept for -// the same reason: they are said, and recorded, by the thread that runs the -// programs, in that order. -struct PrecompiledProgram { - // A compile was attempted. False when the program's result was still valid - // when it was asked (nothing to compile), or when a check that comes before - // the compile refused it (which the program's turn refuses again, in order). - bool compiled = false; - // What the compile was made for: the program's text, the host compiler, the - // host modules' text. The program's turn uses the compile only when its own - // computation of this agrees, so a compile made for something that changed in - // the meantime is made again rather than trusted. - std::string stamp; - std::filesystem::path bin; - std::chrono::milliseconds compile{0}; - // The compile's own failure, in the words a serial build gives it. - std::string error; - std::optional refusal; - std::vector warnings; -}; +// The program its rules describe, for a package with device sources and no +// `build.mcpp` of its own: it imports each rule module and calls its +// `compile()`. Written by the build programs' graph (mcpp#797). +std::string synthesised_rule_program(const std::vector& modules); -// Everything `run_build_program` does before it runs the program, and stops there: -// the checks, the cache, and, for a program that needs it, the compile. Safe to -// call from several threads at once for different programs. A program that -// compiles shows as compiling; nothing else is shown. -PrecompiledProgram precompile_build_program( - mcpp::manifest::Manifest& m, - const std::filesystem::path& root, - const std::filesystem::path& hostCompiler, - const mcpp::toolchain::Toolchain& tc, - const mcpp::manifest::CppStandardConfig& cppStandard, - const BuildProgramEnv& env); - -// `precompiled`, when given, is what `precompile_build_program` made for this -// program. The program is compiled here only when it is absent or made for -// something else. +// Runs the build program of the package at `root` -- `env.programBinary`, as +// the build programs' graph compiled it -- or replays its cached directives +// when nothing it depends on changed, and applies them to `m`. std::expected run_build_program( mcpp::manifest::Manifest& m, const std::filesystem::path& root, const std::filesystem::path& hostCompiler, const mcpp::toolchain::Toolchain& tc, const mcpp::manifest::CppStandardConfig& cppStandard, - const BuildProgramEnv& env, - const PrecompiledProgram* precompiled = nullptr); + const BuildProgramEnv& env); // Has any recorded build-program input changed since its build.mcpp cache was // written: a glob's path SET (#359), a declared file's CONTENT, or a declared @@ -694,71 +597,6 @@ std::string env_value(const std::string& name) { return v ? std::string(v) : std::string(); } -// The host subset of flags.cppm's sysroot/runtime handling — enough to compile + -// link a freestanding host program on a fresh sandbox (where bare `g++ file -o x` -// can't find crt/libc). `tc` is always a HOST-targeting toolchain: under a cross -// --target, prepare.cppm resolves the spec a second time without the target axis -// (host_tc_for_build_program) and passes that here, so the native cases are the -// only ones needed. Passed as separate argv tokens (no shell). -std::vector host_base_flags(const mcpp::toolchain::Toolchain& tc, - std::string_view macosDeploymentTarget) { - // One driver invocation compiles AND links build.mcpp, so it needs both - // sides. Both come from mcpp.toolchain.hostflags — the same producer - // flags.cppm and the std module build use. This function used to hand-write - // the whole assembly, which is how it kept missing what the main build - // already knew (quoting, the macOS deployment target, the MSVC dialect); - // see 2026-08-02-host-compile-single-producer-design.md. - mcpp::toolchain::HostFlagOptions opt; - // The host helper keeps TRUSTING clang's cfg on macOS/Windows: the macOS - // link needs the libc++abi/unwind handling the main build's - // needs_explicit_libcxx path owns, and duplicating it here produced - // undefined __cxa_* / __gxx_personality_v0. - opt.cfgBypass = mcpp::toolchain::HostFlagOptions::CfgBypass::LinuxOnly; - opt.clangStdlibSelect = true; - // `cAbiPrebuilt` is left at its default (true), and that is a statement - // rather than an omission: `tc` here is always HOST-targeting (see this - // function's header), so the helper's C library is the payload's whatever - // the project's target side turns out to be. - // - // It also corrects a latent defect. The predicate this replaced was - // `!tc.crossTargetFlag.empty()`, and a host toolchain resolved for a cross - // build could carry one — in which case the helper lost the payload's own - // headers for a reason that had nothing to do with it. - // binutils -B so the driver finds ld/as (GCC; musl and MinGW ship their own). - opt.binutilsPrefix = !mcpp::toolchain::is_musl_target(tc) - && !mcpp::toolchain::is_mingw_target(tc); - // The helper is exec'd outside anything mcpp controls, so it must be able - // to find the toolchain's private runtime libs itself. - opt.runtimeLibDirs = true; - opt.macosDeploymentTarget = std::string(macosDeploymentTarget); - - const mcpp::toolchain::PathEscape plain = mcpp::toolchain::no_escape; - auto f = mcpp::toolchain::host_compile_tokens(tc, opt, plain); - for (auto& w : mcpp::toolchain::apple_float_macro_words(tc)) f.push_back(std::move(w)); - for (auto& t : mcpp::toolchain::host_link_tokens(tc, opt, plain)) - f.push_back(t); - return f; -} - -// ── Cache (line-based; one record per line, internal format) ─────────────── -// epoch -// program -// compiler -// ctx -// in -// env -// d -// The leading epoch/program/compiler/ctx/in/env lines are the re-run key; the -// `d` lines are the directives to reapply on a hit. The `d` tag vocabulary is -// owned by mcpp.build.directives::kTable and is NOT spelled here — that list -// used to be duplicated in four places and drifted. - -// build.mcpp artifacts live under target/ (the build output tree), not in the -// project: target/.build-mcpp/{build.mcpp.bin, build.mcpp.cache}. A stable subdir -// (not the fingerprint-keyed one — build.mcpp runs before the fingerprint exists) -// so the binary + cache survive across builds and aren't rebuilt needlessly. -// A dependency's artifacts are redirected into the CONSUMING project's tree -// via BuildProgramEnv::artifactsDir (a registry root may be read-only). fs::path build_dir(const fs::path& root, const BuildProgramEnv& env) { return env.artifactsDir.empty() ? root / "target" / ".build-mcpp" : env.artifactsDir; @@ -1126,7 +964,7 @@ bool cache_fresh(const fs::path& root, const fs::path& bdir, const CacheRecord& // The namespace comes from the module name with `.` exchanged for `::`, which // is the convention `mcpp.rules.` already follows and the one a third-party // rule opts into by naming its module. Nothing here knows what any rule does. -std::string synthesised_rule_program(const std::vector& modules) { +std::string synthesised_rule_program_impl(const std::vector& modules) { std::string s = "// Generated by mcpp from the build rules this package's dependencies\n" "// declare. Do not edit. To take it over, copy this file to `build.mcpp`\n" @@ -1169,68 +1007,20 @@ install_hook_env(const BuildProgramEnv& env) { }; } -// WHICH PART OF A PROGRAM'S LIFE ONE CALL RUNS. `Whole` is the program as it has -// always been run: check, compile if the cache is stale, run, apply. `Compile` -// stops at the end of the compile and leaves its outcome in `*made`; it reports, -// records and applies nothing. `Run` is `Whole` with a compile taken from `*made`. -enum class ProgramMode { Whole, Compile, Run }; - std::expected run_build_program_impl( - ProgramMode mode, mcpp::manifest::Manifest& m, const fs::path& root, - const fs::path& hostCompiler, const mcpp::toolchain::Toolchain& tc, - const mcpp::manifest::CppStandardConfig& cppStandard, - const BuildProgramEnv& env, - const PrecompiledProgram* given, - PrecompiledProgram* made) { + const BuildProgramEnv& env) { - fs::path src = root / "build.mcpp"; + // A package with neither a `build.mcpp` nor rules that describe one has no + // program (the synthesised one is written by the build programs' graph). std::error_code ec; - if (!fs::exists(src, ec)) { - // The layer above this file: when a dependency's rules claimed device - // sources in this package and the package wrote no program, mcpp writes - // the program those rules describe. - // - // SYNTHESISED ONLY IN THE ABSENCE. A project with its own `build.mcpp` - // keeps it, because the two would otherwise both submit the same - // actions and the second submission is one nobody asked for. That is - // also what makes the descent safe: copy the generated file into the - // project root, edit it, and synthesis stops. - if (env.ruleModules.empty()) return {}; // nothing to do - src = build_dir(root, env) / "build.mcpp"; - fs::create_directories(src.parent_path(), ec); - const std::string text = synthesised_rule_program(env.ruleModules); - // Written only when it differs, so a package whose rule set did not - // change does not rebuild its build program on every configure. - bool same = false; - if (std::ifstream in(src, std::ios::binary); in) { - std::string prev((std::istreambuf_iterator(in)), - std::istreambuf_iterator()); - same = (prev == text); - } - if (!same) { - // REFUSED, not skipped. Returning "nothing to do" here would leave - // the device sources uncompiled and the build otherwise successful, - // and the first symptom would be an unresolved name in a consumer - // that imported the interface this program was to generate. - std::ofstream out(src, std::ios::binary | std::ios::trunc); - if (!out) - return std::unexpected(std::format( - "cannot write the build program the rules describe: {}", - src.string())); - out << text; - out.close(); - if (!out) - return std::unexpected(std::format( - "failed while writing the build program: {}", src.string())); - } - } + if (!fs::exists(root / "build.mcpp", ec) && env.ruleModules.empty()) return {}; fs::path bdir = build_dir(root, env); fs::path outDir = env.outDir.empty() ? bdir / "out" : env.outDir; - auto childEnv = contract_env(root, outDir, env, mode != ProgramMode::Compile); + auto childEnv = contract_env(root, outDir, env); std::string ctxHash = contract_hash(childEnv); // THE GRAPH DOCUMENT'S CONTENT, NOT ITS PATH. The path is the same on // every run; what the document says is what the program's answer depends @@ -1238,193 +1028,21 @@ std::expected run_build_program_impl( if (!env.graphDigest.empty()) ctxHash = mcpp::toolchain::hash_string(ctxHash + "\ngraph=" + env.graphDigest); - // ── Helper self-containment (the single decision point) ───────────────── - // The compiled helper is exec'd by the host OS, outside anything mcpp - // controls — no wrapper, no injected LD_LIBRARY_PATH, no PATH guarantee. - // Making it runnable is a per-platform mechanism, and only two of the four - // need a static link: - // Linux + glibc payload — host_base_flags bakes absolute payload paths - // into -Wl,-rpath; that already closes it, so leave it dynamic. - // Linux + musl (#295) — rpath cannot reach PT_INTERP, an absolute - // /lib/ld-musl-.so.1 that no payload installs and glibc distros - // do not ship. Only a static link removes the interpreter entirely. - // Windows PE (#299) — PE has no rpath, so a dynamic helper resolves - // libstdc++-6 / libgcc_s / libwinpthread through the process PATH and - // dies with STATUS_DLL_NOT_FOUND unless the user put the toolchain bin - // there by hand. A static link is the only PATH-independent answer. - // macOS — the system libc++ is always present. - // Keep this the ONLY place that decides; the flag is appended to the final - // link argv further down, never to `base`. - const bool muslStaticHelper = mcpp::platform::supports_full_static - && mcpp::toolchain::is_musl_target(tc); - const bool mingwStaticHelper = mcpp::toolchain::is_mingw_target(tc); - const bool staticHostHelper = muslStaticHelper || mingwStaticHelper; - - // Fold the policy into the compiler identity: a helper produced under an - // older link policy must be rebuilt, not reused from the cache. - std::string compilerIdentity = hostCompiler.string(); - // Host modules change what the helper links, so they belong in the identity - // the cache keys on — otherwise adding or removing a rule package would - // replay a cached run compiled without it. - for (auto const& hm : env.hostModules) { - compilerIdentity += "\nhost-module="; - compilerIdentity += hm.logical; - compilerIdentity += "@"; - compilerIdentity += mcpp::toolchain::hash_file(hm.interface); - } - compilerIdentity += "\nbuild-program-link="; - compilerIdentity += muslStaticHelper ? "musl-static-v1" - : mingwStaticHelper ? "mingw-static-v1" - : "default-v2"; // v2: DT_RPATH on Linux - std::string programHash = mcpp::toolchain::hash_file(src); - std::string compilerHash = mcpp::toolchain::hash_string(compilerIdentity); - - // Read once, here, because the check below has to run BEFORE the cache - // fast path returns. - std::string srcText; - { std::ifstream is(src); std::ostringstream ss; ss << is.rdbuf(); srcText = ss.str(); } - - // A module that is present only as another rule's prerequisite is not part - // of this package's declared surface. Refusing the import rather than - // letting it work is the difference between a rule that travels and one - // that happens to build on whoever's machine: the same source stops - // compiling the moment the intermediate rule stops depending on it, or the - // moment a Clang user tries it, since only GCC makes the BMI reachable - // without the flags. - // - // AHEAD OF THE FAST PATH ON PURPOSE. Withdrawing `reexport = true` from - // an edge leaves the module SET and every interface hash identical, so - // nothing in `compilerIdentity` moves. Measured: the replay is prevented - // today only because `ctxHash` happens to change too — an incidental - // coupling, and the shape this codebase keeps paying for. Asking the - // question before the cache is consulted needs no key at all. - for (auto const& hm : env.hostModules) { - if (hm.importable) continue; - if (!imports_module(srcText, hm.logical)) continue; - return std::unexpected(std::format( - "build.mcpp imports '{}', which reaches this build only as a " - "prerequisite of another build rule.\n" - " A rule's build-time provisions cross one further edge only " - "when that edge says so.\n" - " Either depend on the package providing '{}' directly, or " - "have the rule that owns it re-export it:\n" - " = {{ ..., host-module = true, " - "reexport = true }}", - hm.logical, hm.logical)); - } + // A compile that failed is reported at the program's turn, so the failure + // that reaches the user is the first in the order the programs run. + if (!env.programError.empty()) return std::unexpected(env.programError); + if (env.programBinary.empty()) + return std::unexpected(std::string( + "the build program was not compiled: no binary reached its run " + "(an mcpp defect; please report it with this line)")); - // …and the complementary case: an import NOTHING provides. - // - // Left to the compiler this is a raw `failed to read compiled module` with - // a note that imports must be built before being imported -- true, and it - // names neither the package that would provide the module nor the key that - // would make it importable. Measured on a project that declared its rule - // package correctly except for `host-module`: - // - // mcpp.rules.spirv: error: failed to read compiled module: No such - // file or directory - // mcpp.rules.spirv: note: imports must be built before being imported - // - // The set of names that CAN compile here is closed -- `std`, `std.compat`, - // the bundled `mcpp` module, and the importable host modules -- so a name - // outside it cannot become valid later and is refused rather than warned - // about. Header units (`import ;`, `import "x.h";`) name no module - // and `imported_module_names` does not report them. - // - // ONLY build.mcpp's own imports. A rule interface compiled alongside it may - // import a module that is present as a prerequisite and not importable - // here, which is the case the check above states. - { - std::set available{"std", "std.compat", "mcpp", "mcpp.core"}; - for (auto const& hm : env.hostModules) - if (hm.importable) available.insert(hm.logical); - for (auto const& want : mcpp::pm::imported_module_names(srcText)) { - if (available.contains(want)) continue; - // A dependency that could plausibly be meant: one declared without - // `host-module = true`. Named as a hint, not as a claim -- the - // engine has not resolved this package's lib root here and does not - // know which module it would declare. - // - // Ranked by the one relation available without resolving anything: - // a rule package's modules are expected to open with its namespace, - // which is what `reserved_prefix_warning` is about. A dependency - // whose key shares the imported name's first segment is listed - // alone; when none does, all of them are listed rather than none, - // because the convention is a convention and a package may declare - // a module outside its own prefix. - const auto dot = want.find('.'); - const auto head = dot == std::string::npos ? want : want.substr(0, dot); - std::string near, all; - auto scan = [&](const std::map& deps, - std::string_view section) { - for (auto const& [k, s] : deps) { - if (s.hostModule) continue; - const auto one = std::format("{} (in [{}])", k, section); - auto& dst = k.starts_with(head + ".") || k == head ? near : all; - if (!dst.empty()) dst += ", "; - dst += one; - } - }; - scan(m.buildDependencies, "build-dependencies"); - scan(m.dependencies, "dependencies"); - const std::string candidates = near.empty() ? all : near; - std::string importable; - for (auto const& hm : env.hostModules) - if (hm.importable) { - if (!importable.empty()) importable += ", "; - importable += hm.logical; - } - // #734 E7: the module may be one a provider offers behind a - // feature this build has not enabled. The files were listed while - // planning; they are read only now, on the way to an error. - for (auto const& df : env.dormantFeatures) { - for (auto const& f : df.files) { - std::ifstream in(f, std::ios::binary); - std::string text{std::istreambuf_iterator(in), {}}; - std::size_t at = 0; - bool declares = false; - while ((at = text.find("export module", at)) != std::string::npos) { - std::size_t i = at + std::string_view("export module").size(); - while (i < text.size() && (text[i] == ' ' || text[i] == '\t')) ++i; - std::size_t j = i; - while (j < text.size() && text[j] != ';' && text[j] != ' ' - && text[j] != '\t' && text[j] != '\n' && text[j] != '\r') ++j; - if (text.compare(i, j - i, want) == 0) { declares = true; break; } - at = j; - } - if (!declares) continue; - mcpp::build::refusal::record( - mcpp::build::refusal::Code::HostModuleMissing); - return std::unexpected(std::format( - "build.mcpp imports '{}'\n" - " provided by: {}, feature \"{}\" (not enabled)\n" - " hint: enable it on the dependency edge: {} = {{ ..., " - "features = [\"{}\"] }}", - want, df.package, df.feature, df.package, df.feature)); - } - } - mcpp::build::refusal::record( - mcpp::build::refusal::Code::HostModuleMissing); - return std::unexpected(std::format( - "build.mcpp imports '{}', and no dependency provides it as a " - "host module.\n" - " A package's module is compiled for the build program " - "only when the dependency\n" - " edge asks for it, which is a separate question from " - "whether the package reaches\n" - " the target:\n" - " [build-dependencies.]\n" - " = {{ version = \"...\", host-module = true }}\n" - " importable here: {}\n" - "{}", - want, - importable.empty() ? "(none)" : importable, - candidates.empty() - ? " This package declares no dependency that could provide it." - : std::format(" declared without `host-module = true`: {}", - candidates))); - } - } + // THE IDENTITY THE PROGRAM WAS COMPILED FROM (mcpp#797) keys the run + // cache: the program's text, the engine, the toolchain, and every package + // of its graph with the content of each tree that can change in place. A + // program whose graph changed is compiled again and run again; one whose + // graph stands is replayed. The second field is the record's own shape. + const std::string programHash = env.programIdentity; + const std::string compilerHash = mcpp::toolchain::hash_string("build-programs-graph-v1"); // The package the status lines name: several programs run in one // workspace plan, and a line without its package cannot be attributed. @@ -1435,9 +1053,6 @@ std::expected run_build_program_impl( // directives, no run. CacheRecord cache = read_cache(bdir); if (cache_fresh(root, bdir, cache, programHash, compilerHash, ctxHash)) { - // Nothing to compile. Applying, advising and reporting the cached result - // belong to the program's turn, which does them in order. - if (mode == ProgramMode::Compile) return {}; if (auto terr = dirs::target_directive_error(m, cache.directives); !terr.empty()) return std::unexpected(terr); // #622 A4. Checked on the cache-hit path too, so a `deploy` a fresh @@ -1462,16 +1077,14 @@ std::expected run_build_program_impl( return {}; } - // From here on the program is compiled or run, and every return reports - // its outcome: `ran` when the directives were applied, `failed` otherwise. + // From here on the program is run, and every return reports its outcome: + // `ran` when the directives were applied, `failed` otherwise. using ProgressClock = std::chrono::steady_clock; struct ProgramReport { const std::string& who; bool requested; - ProgressClock::time_point compileStart{}, runStart{}; - // The compile was made elsewhere, by the thread that compiled it: its - // duration is stated, and is not read off this thread's clock. - std::optional compileDuration; + ProgressClock::time_point runStart{}; + std::chrono::milliseconds compile{0}; bool ran = false; bool reported = false; ~ProgramReport() { report(); } @@ -1479,542 +1092,26 @@ std::expected run_build_program_impl( if (reported) return; reported = true; using namespace std::chrono; - const auto now = ProgressClock::now(); const auto zero = ProgressClock::time_point{}; - const auto compile = compileDuration ? *compileDuration - : compileStart == zero ? milliseconds{0} - : duration_cast((runStart == zero ? now : runStart) - compileStart); const auto run = runStart == zero ? milliseconds{0} - : duration_cast(now - runStart); + : duration_cast(ProgressClock::now() - runStart); mcpp::build::progress::program_finished( who, requested, ran ? mcpp::build::progress::ProgramOutcome::Ran : mcpp::build::progress::ProgramOutcome::Failed, - compile, run, /*compiledAside=*/compileDuration.has_value()); + compile, run, /*compiledAside=*/compile.count() > 0); } } programReport{who, env.requested}; - // The compile phase says nothing: what it made is reported, or refused, when - // the program's turn comes. - if (mode == ProgramMode::Compile) programReport.reported = true; + programReport.compile = env.programCompile; // Both: the output directory is per configuration and need not lie - // below `bdir`, where the program binary and its cache are (D7). + // below `bdir`, where the program's cache is (D7). fs::create_directories(bdir, ec); fs::create_directories(outDir, ec); - // #230: on Windows the capture_exec shell is cmd.exe, which can only launch - // a PE by an executable extension — a bare `.bin` is not in PATHEXT and - // fails to run. Name the compiled program `.exe` there; other platforms keep - // `.bin` (cosmetic — bdir is separate from the `.mcpp` source). Surfaces - // once a workspace build.mcpp member is reached on Windows, after the - // scanner symlink-escape crash fix (df985df) stops masking it as exit 127. - fs::path bin = bdir / (mcpp::platform::is_windows - ? "build.mcpp.exe" : "build.mcpp.bin"); - - // The driver's dialect: what the compile is spelled in, and how the program's - // output is read (`accept_output`). Used on both sides of the compile. - // - // The host compile has always been spelled in GNU driver syntax with no - // dialect branch at all — `grep -i msvc` over this file used to hit only - // comments. Under cl.exe every one of `-O0` / `-x c++` / `-static` / `-o` - // is wrong, so the whole build.mcpp path was unusable on a native MSVC - // toolchain regardless of what else was fixed. + const fs::path bin = env.programBinary; + // The driver's dialect decides how the program's output is read + // (`accept_output`). const auto& dial = mcpp::toolchain::dialect_for(tc); - const bool msvcHost = dial.id == std::string_view("msvc"); - - // THE COMPILE, AS ONE STEP. The preparation of what the program imports and - // the compile of the program itself are one piece of work with one duration, - // which is what `ran` reports; it used to report the compile of the program - // alone and leave the preparation (seven seconds a program, measured on - // #748's runner) to be counted in `plan`. It is a lambda so that it can be - // made ahead of the program's run, on another thread (`precompile_build_ - // program`), and used by the run without being made again. - ProgressClock::time_point compileBegan{}, compileEnded{}; - std::vector compileWarnings; - std::optional compileRefusal; - auto compile_binary = [&]() -> std::expected { - compileBegan = ProgressClock::now(); - mcpp::build::progress::program_compiling(who, env.requested); - mcpp::log::verbose("buildmcpp-host", std::format("build.mcpp {}: compile start", who)); - // ── Compile build.mcpp with the host toolchain ────────────────────────── - // Spelled by the dialect layer, not concatenated here: the canonical of - // `standard = "c++fly"` / `"c++latest"` is not a valid -std= spelling, so - // the old `"-std=" + canonical` produced `-std=c++fly` and the host compile - // died on an unknown dialect. cppfly::std_flag resolves those against the - // toolchain that will actually run the compile — the host one, here. - std::string std_flag = mcpp::toolchain::cppfly::std_flag( - tc, cppStandard.canonical.empty() ? std::string_view("c++23") - : std::string_view(cppStandard.canonical), - cppStandard.level); - // One resolution of the deployment target, used by every compile below - // and by the std module it asks stdmod to build — they must agree or - // clang rejects the BMI. - // - // LEGITIMATELY HOST-KEYED, unlike the main build's readers (#685). `tc` - // here is always the HOST toolchain (see this function's own doc above - // `run_build_program` / `host_base_flags`'s header) -- build.mcpp is - // compiled AND run on the machine doing the build, so "the target" this - // one compile is for IS the host, and `tc.targetTriple` already names - // it. Asking `tc`'s own resolved target keeps this correct without - // reading a compile-time `__APPLE__`/`is_macos` constant, which would - // silently disagree with `tc` the day build.mcpp gains a host toolchain - // resolved for something other than the machine mcpp itself runs on. - const bool buildProgramTargetIsMacos = [&] { - auto bpTt = mcpp::toolchain::triple::parse(tc.targetTriple); - return bpTt && bpTt->os == "macos"; - }(); - const std::string macosDeploymentTarget = - mcpp::platform::macos::deployment_target( - buildProgramTargetIsMacos, m.buildConfig.macosDeploymentTarget); - auto base = host_base_flags(tc, macosDeploymentTarget); - - // Only wire the bundled `mcpp` module when build.mcpp actually imports it — - // so the common `#include`-based program compiles exactly as before (no - // -fmodules, cwd = project root). When it does `import mcpp;`, compile the - // module, link its object, and run the build.mcpp compile from `bdir` so GCC - // finds gcm.cache/mcpp.gcm. - // `srcText` was read above the cache fast path, which the prerequisite - // check needs to run ahead of. - bool usesModule = srcText.find("import mcpp") != std::string::npos; - bool usesStdCompat = imports_module(srcText, "std.compat"); - bool usesStd = usesStdCompat || imports_module(srcText, "std"); - - // A rule package's interface is compiled by this same function, so what IT - // imports decides what has to be built just as much as what build.mcpp - // imports. Scanning only build.mcpp made a rule that said `import std;` - // fail with `module 'std' not found` — the std module was never built, - // because the program that triggers the build did not mention it. - for (auto const& hm : env.hostModules) { - std::ifstream is(hm.interface); - if (!is) continue; // a missing interface is diagnosed by provide_host_module - std::ostringstream ss; ss << is.rdbuf(); - const std::string t = ss.str(); - if (t.find("import mcpp") != std::string::npos) usesModule = true; - if (imports_module(t, "std.compat")) usesStdCompat = true; - if (imports_module(t, "std")) usesStd = true; - } - - usesStd = usesStd || usesStdCompat; - - // The toolchain's own environment (MSVC's INCLUDE / LIB / VSLANG, which - // detection synthesized from the located VC tools + Windows SDK). Needed - // by every compile below, the module precompile included. - std::vector> compileEnv; - for (auto const& ev : tc.envOverrides) - compileEnv.emplace_back(ev.key, ev.value); - - // Named modules dispatch on the same BmiTraits/CommandDialect rows the - // main build uses, so there is no per-family gate here: cl.exe's - // .ifc + /reference works because the table already describes it, not - // because build.mcpp grew a second implementation of it. - std::vector moduleFlags; - fs::path mcppModuleObject; - fs::path mcppCoreObject; // `mcpp.core`, the same interface (#734, E8) - // The BMIs a compile can see, in the order they were made: what GCC finds by - // name under `gcm.cache`, and what the other two families are told by flag. - // A host module is compiled with every one before it. - std::vector visibleBmis; - // Where what this program imports is kept: the engine's module globally when - // the cache mode allows it, and the workspace's store for everything that is - // the project's own (#748, B1). - mcpp::build::ModuleStores stores; - stores.cacheRoot = env.moduleCacheRoot; - stores.workspaceStore = env.moduleStore.empty() ? bdir / "host-modules" : env.moduleStore; - if (usesModule) { - auto mf = mcpp::build::provide_mcpp_module(stores, bdir, hostCompiler, base, - std_flag, tc, compileEnv); - if (!mf) return std::unexpected(mf.error()); - moduleFlags = std::move(mf->useFlags); - mcppModuleObject = std::move(mf->object); - mcppCoreObject = std::move(mf->aliasObject); - for (auto& b : mf->bmis) visibleBmis.push_back(std::move(b)); - } - - // ── `import std;` in build.mcpp ───────────────────────────────────────── - // - // mcpp asks projects to `import std;` everywhere and then made their build - // script fall back to `#include` — the bundled `mcpp` module even says so - // in its own header comment. The std module the main build already uses is - // reusable verbatim: stdmod::ensure_built caches on - // (toolchain × standard × dialect), so for a native build this is a cache - // HIT on the very artifact the project's own TUs import. Only a cross - // build pays for a second one, which is unavoidable — see below. - // - // `tc` here is the HOST toolchain: prepare.cppm's - // host_tc_for_build_program() resolves the spec WITHOUT the --target axis - // and hands it in. That is load-bearing. build.mcpp is compiled AND run on - // the machine doing the build, so a std BMI built for the target would - // produce a helper that cannot execute — the same host≠target mistake the - // mingw-cross work had to fix in four separate places. - std::vector stdFlags; - std::vector stdObjects; - // GCC finds staged BMIs by cwd; Clang/MSVC get an explicit path flag. - bool stdStagedInBdir = false; - if (usesStd) { - if (!tc.hasImportStd) { - return std::unexpected(std::format( - "build.mcpp uses `import std;` but the host toolchain ({}) " - "ships no std module.\n" - " Use #include in build.mcpp, or switch to a toolchain " - "that provides one.", tc.label())); - } - // One thread builds the std module, and the others find it: the first - // build in a fresh home would otherwise have every program's compile - // write the same cache directory. - std::optional smValue; - std::optional smError; - { - static std::mutex stdModuleMutex; - std::lock_guard stdLock(stdModuleMutex); - auto built = mcpp::toolchain::ensure_built( - tc, cppStandard.canonical, std_flag, macosDeploymentTarget, - mcpp::toolchain::default_cache_root(), {}, usesStdCompat); - if (built) smValue = std::move(*built); - else smError = built.error().message; - } - if (!smValue) { - // The second branch of the same refusal, and it is named for the - // same reason: both `run_build_program` call sites in prepare.cppm - // return `std::unexpected` unconditionally, so this error always - // reaches the layer that reads the code. An unnamed one here would - // reproduce, for the host std module, exactly the gap the target - // std module had. - compileRefusal = refusal::Code::StdModulePrecompile; - return std::unexpected(std::format( - "build.mcpp uses `import std;` but the std module could not be " - "built for the host toolchain: {}", *smError)); - } - auto sm = std::move(smValue); - - auto traits = mcpp::toolchain::bmi_traits(tc); - if (traits.stdBmiUsePrefix.empty()) { - // GCC: BMIs are found implicitly under /gcm.cache, so stage - // the cached ones where the compile will look. Copy rather than - // symlink — this mirrors the main build's staging edge, and a - // stale copy is caught by ensure_built's own cache key. - std::error_code ec; - fs::path gcmDir = bdir / traits.bmiDir; - fs::create_directories(gcmDir, ec); - auto stage = [&](const fs::path& from, std::string_view name) - -> std::expected { - if (from.empty() || !fs::exists(from)) return {}; - fs::path to = gcmDir / std::format("{}{}", name, traits.bmiExt); - fs::copy_file(from, to, fs::copy_options::overwrite_existing, ec); - if (ec) return std::unexpected(std::format( - "staging {} for build.mcpp failed: {}", name, ec.message())); - return {}; - }; - if (auto r = stage(sm->bmiPath, "std"); !r) - return std::unexpected(r.error()); - if (usesStdCompat) { - if (auto r = stage(sm->compatBmiPath, "std.compat"); !r) - return std::unexpected(r.error()); - } - // -fmodules may already be present from the `mcpp` module path; - // GCC tolerates the repeat, but keep the argv honest. - if (!usesModule) stdFlags.push_back("-fmodules"); - stdStagedInBdir = true; - } else { - // Through bmi_reference_tokens, not string concatenation: the - // traits spell these for the ninja STRING channel, where - // `-fmodule-file=std=

` (one word) and `/reference std=

` - // (two) are indistinguishable. Concatenating produced a single - // argv element with a space inside it, and cl answered - // "C2230: could not find module 'std'". - for (auto& t : mcpp::toolchain::bmi_reference_tokens( - traits.stdBmiUsePrefix, sm->bmiPath)) - stdFlags.push_back(t); - if (usesStdCompat && !sm->compatBmiPath.empty()) - for (auto& t : mcpp::toolchain::bmi_reference_tokens( - traits.stdCompatBmiUsePrefix, sm->compatBmiPath)) - stdFlags.push_back(t); - } - if (!sm->objectPath.empty() && fs::exists(sm->objectPath)) - stdObjects.push_back(sm->objectPath.string()); - if (usesStdCompat && !sm->compatObjectPath.empty() - && fs::exists(sm->compatObjectPath)) - stdObjects.push_back(sm->compatObjectPath.string()); - // The std cache's directory is named by its own identity hash, which is - // what a dependent entry's key records for the BMIs inside it. - { - const auto id8 = sm->cacheDir.filename().generic_u8string(); - const std::string stdIdentity(reinterpret_cast(id8.data()), id8.size()); - visibleBmis.push_back({"std", sm->bmiPath, stdIdentity}); - if (usesStdCompat && !sm->compatBmiPath.empty()) - visibleBmis.push_back({"std.compat", sm->compatBmiPath, stdIdentity}); - } - } - - // #355 step 5: dependency-provided host modules (reusable build rules - // shipped as ordinary packages). Compiled HERE, with `base` and `std_flag` - // — the same flags the build.mcpp compile below gets — because a BMI is - // only usable by a compile that agrees with it. Doing this in a separate - // sub-build would leave that agreement to chance, and disagreement shows - // up as `module X CRC mismatch`, not as a clear error. - // - // AFTER the std block, and that ordering is load-bearing: a rule may - // `import std;` just as build.mcpp may, and it can only do so once the std - // BMI exists and `stdFlags` names it. Compiling rules first — which is what - // 2026.8.5.1 did — handed them an empty `stdFlags` and failed with - // `module 'std' not found`. - std::vector hostModuleObjects; - for (auto const& ref : env.hostModules) { - std::vector use = moduleFlags; - use.insert(use.end(), stdFlags.begin(), stdFlags.end()); - mcpp::build::HostModuleSource source; - source.logical = ref.logical; - source.interface = ref.interface; - source.index = ref.providerIndex; - source.package = ref.providerName; - source.version = ref.providerVersion; - source.immutableSource = ref.immutableSource; - source.sourceRoot = ref.providerRoot; - auto hm = mcpp::build::provide_host_module(stores, source, bdir, hostCompiler, - base, std_flag, tc, compileEnv, use, - visibleBmis); - if (!hm) return std::unexpected(hm.error()); - for (auto& b : hm->bmis) visibleBmis.push_back(b); - // APPENDED VERBATIM, and nothing is de-duplicated. - // - // This filtered per TOKEN, and it was written for the one family whose - // marker is a single idempotent word: GCC's `-fmodules`, already - // present because the bundled `mcpp` module put it there. The other - // two families do not have that shape. - // - // GCC `-fmodules` 1 token, idempotent - // Clang `-fmodule-file==` 1 token, unique - // MSVC `/reference`, `=` 2 tokens, FIRST REPEATS - // - // So on `windows = "msvc@system"` the pair arrived, `/reference` was - // found already in the list, and only the pair's second half was - // appended. cl.exe received `=.ifc` with no switch in - // front of it and read it as a source file name: - // - // c1xx: fatal error C1083: Cannot open source file: - // 'huxerui.rules.sources=...\huxerui.rules.sources.ifc' - // - // Clang was immune by construction -- one word, never equal to an - // existing element -- which is why the defect was specific to the one - // toolchain selection that reaches `import std;` at c++20 on Windows. - // - // The comment this replaces stated the whole value of the filter: - // "repeating it is harmless but noisy". It bought a tidier argv and - // paid with a broken command line. De-duplicating by logical module - // name, or by contiguous subsequence, would both be correct -- and - // would both be a new rule kept for the same cosmetic reason. The rule - // is gone instead. - for (auto& f : hm->useFlags) - moduleFlags.push_back(f); - hostModuleObjects.push_back(std::move(hm->object)); - } - - // `-x c++` is required: the `.mcpp` extension is unknown to the compiler, so - // without it the driver hands build.mcpp to the linker as a linker script. - std::vector compileArgv = { hostCompiler.string() }; - if (msvcHost) { - // /nologo /EHsc /utf-8 — cl.exe needs these to behave like the other - // two drivers do by default (quiet, exceptions on, UTF-8 sources). - for (auto f : dial.alwaysFlagsArgv) compileArgv.emplace_back(f); - } - compileArgv.push_back(std_flag); - // No optimization: this program runs once per build and its compile time - // is on the critical path. MSVC spells "off" /Od, not /O0. - compileArgv.push_back(msvcHost ? std::string("/Od") - : std::string(dial.optPrefix) + "0"); - for (auto& bf : base) compileArgv.push_back(bf); - for (auto& mf : moduleFlags) compileArgv.push_back(mf); - for (auto& sf : stdFlags) compileArgv.push_back(sf); - // The `.mcpp` extension is unknown to every driver, so without this the - // file is handed to the linker as a linker script. - // Per-file where the driver has that form (cl's /Tp), positional - // otherwise. Object files follow on this same command line, and cl's - // global /TP would compile them as C++ source. - if (!dial.perFileCxxPrefix.empty()) { - compileArgv.push_back(std::string(dial.perFileCxxPrefix) + src.string()); - } else { - for (auto f : dial.forceCxxLangArgv) compileArgv.emplace_back(f); - compileArgv.push_back(src.string()); - } - if (usesModule || !stdObjects.empty() || !hostModuleObjects.empty()) { - // Link the module objects. GNU drivers need the input language reset - // first, or the .o that follows `-x c++` is handed to the frontend as - // C++ source; cl.exe has no `-x` at all and infers from the extension. - // This used to be unconditional and was only harmless while MSVC could - // not reach it — removing that gate made the dead branch live, and cl - // answered with `D9002: ignoring unknown option '-x'`. - if (!msvcHost) { compileArgv.push_back("-x"); compileArgv.push_back("none"); } - if (usesModule) compileArgv.push_back(mcppModuleObject.string()); - if (usesModule && !mcppCoreObject.empty()) compileArgv.push_back(mcppCoreObject.string()); - for (auto& hmo : hostModuleObjects) compileArgv.push_back(hmo.string()); - for (auto& so : stdObjects) compileArgv.push_back(so); - } - // THE BUILD PROGRAM SPEAKS THE ENCODING mcpp SPEAKS (#693, D4). - // - // mcpp hands a build program its paths through the environment and reads - // its directives from stdout, and on Windows mcpp runs with a UTF-8 process - // code page. A build program without the same application manifest reads - // the environment through the machine's ANSI code page instead. Measured on - // a cp1252 runner: a narrow build.mcpp in `C:\w\caf` printed its - // path in cp1252 bytes that mcpp could not decode, and in a directory - // outside the code page it received `??` in place of the name. So the - // program carries the manifest; where no resource compiler stands beside - // the compiler, it is built without one and the build says so. - if constexpr (mcpp::platform::is_windows) { - auto manifestRes = mcpp::build::resources::compile_utf8_manifest( - tc, dial.id, bdir, "build.mcpp.utf8"); - if (manifestRes) { - if (!msvcHost) { compileArgv.push_back("-x"); compileArgv.push_back("none"); } - compileArgv.push_back(manifestRes->string()); - } else { - compileWarnings.push_back(std::format( - "build.mcpp: built without the UTF-8 code page ({}); a path outside " - "this machine's ANSI code page will not reach it intact", - manifestRes.error())); - } - } - // Self-contained helper link — see the staticHostHelper doctrine above. - // Deliberately NOT in `base`: that also feeds the bundled module's - // compile/precompile commands, where a link flag has no business (and for - // Clang would perturb the default PIC/PIE codegen of mcpp.o). - if (staticHostHelper) compileArgv.push_back(std::string(dial.staticRuntime)); - // A DYNAMIC HELPER ON LINUX GETS `DT_RPATH`, NOT `DT_RUNPATH`. - // - // The driver's default is the new tag, and a runpath is consulted only for - // the helper's OWN needed libraries. A build program that opens a host - // library at run time -- a rule package reading a driver's version through - // the driver itself -- then fails one hop later, because that library's - // own dependencies (`libdl.so.2`, `libpthread.so.0`) are looked up without - // the helper's search path and the payload loader has no default that - // reaches them. Measured: `dlopen("/lib/libcuda.so.1")` from a - // build.mcpp answered `libdl.so.2: cannot open shared object file` while - // the very same directories sat in the helper's RUNPATH. The artifacts - // mcpp links carry DT_RPATH for this reason (loader_contract's Rpath tag); - // the helper now does too. Driver-only spelling: the helper is always - // linked through the compiler driver, never through the linker directly. - if (!staticHostHelper && !msvcHost - && !mcpp::platform::is_windows && !mcpp::platform::is_macos) - compileArgv.push_back("-Wl,--disable-new-dtags"); - if (msvcHost) { - // /Fe: takes its value attached, not as a separate argv token. - compileArgv.push_back(std::string(dial.outputExePrefix) + bin.string()); - } else { - compileArgv.push_back("-o"); compileArgv.push_back(bin.string()); - } - // A `=` with no switch in front of it, checked before the - // command runs rather than diagnosed from cl.exe's answer to it. cl reports - // such a token as `C1083: Cannot open source file`, which names the module - // and the BMI and never names the missing flag -- so it reads as a broken - // build tree rather than as a broken command line. See - // mcpp::toolchain::orphaned_reference; this is the reader that makes the - // rule enforced rather than merely stated. - if (auto orphan = mcpp::toolchain::orphaned_reference(compileArgv)) { - return std::unexpected(std::format( - "build.mcpp: the module reference '{}' reached the compiler with no " - "switch in front of it.\n" - " This is an mcpp defect, not a problem with the project: the " - "reference is\n" - " assembled as a pair (`/reference =` on MSVC) and " - "only one half\n" - " arrived. Please report it with the toolchain name and this " - "line.", *orphan)); - } - // GCC resolves imported BMIs via gcm.cache/ relative to the compile cwd, so - // any compile that imports a module — `mcpp`, `std`, a build rule's host - // module, or any mix — has to run from bdir, where they were staged or - // compiled. One condition: a build.mcpp that imports only a rule needs - // exactly the same cwd as one that imports only mcpp (a rule-only program - // compiled in the project root and failed with "failed to read compiled - // module", e2e 807 under GCC). Otherwise the project root is fine. - const bool needsBmiCwd = usesModule || stdStagedInBdir || !env.hostModules.empty(); - std::string compileCwd = needsBmiCwd ? bdir.string() : root.string(); - // A COMMAND THAT OUTGREW ITS CHANNEL GOES THROUGH A RESPONSE FILE. - // - // The engine's rule for an unbounded payload is a response file - // (mcpp.build.cmdlimits, and the architecture record it names), and this - // command carries one: a `-fmodule-file==` for every host - // module the program imports, with absolute paths. A collection that - // offers many rules through features is the case -- mcpp-plugins' - // `all-rules-compile` imports fifteen -- and on Windows - // `capture_exec` reaches the shell, whose 8191 bytes this crossed the - // moment that collection gained one more module: - // - // The command line is too long. - // build.mcpp failed to compile (exit 1) - // - // Every driver mcpp supports reads `@file`, one argument per line, and - // the file is written beside the program it compiles, so a failed - // compile leaves it to read. - { - std::string flat; - for (auto const& a : compileArgv) { flat += a; flat.push_back(' '); } - if (mcpp::build::cmdlimits::check_inline( - flat, mcpp::platform::is_windows, /*needsShell=*/true)) { - const auto rsp = bdir / "build.mcpp.compile.rsp"; - const auto body = response_file_body( - std::span(compileArgv).subspan(1), !msvcHost); - std::ofstream out(rsp, std::ios::binary | std::ios::trunc); - out << body; - out.close(); - if (out) { - mcpp::log::verbose("buildmcpp-host", std::format( - "build.mcpp {}: the compile command is {} bytes and goes through {}", - who, flat.size(), rsp.string())); - compileArgv = { compileArgv.front(), "@" + rsp.string() }; - } - } - } - auto cres = mcpp::platform::process::capture_exec(compileArgv, compileEnv, - compileCwd); - mcpp::log::verbose("buildmcpp-host", std::format("build.mcpp {}: compile end", who)); - if (cres.exit_code != 0) { - std::string msg = std::format("build.mcpp failed to compile (exit {}):\n{}", - cres.exit_code, cres.output); - if (mentions_missing_mcpp_api(cres.output)) { - msg += std::format( - "\n The `mcpp` build module this engine bundles does not have " - "that name.\n" - " Either the package was written for a newer mcpp (try " - "`mcpp self update`;\n" - " this is mcpp {}), or the name is misspelled — the compiler " - "cannot tell\n" - " the two apart, because the module is generated by whichever " - "mcpp is running.", - mcpp::MCPP_VERSION); - } - return std::unexpected(std::move(msg)); - } - return {}; - }; - - // The compile is taken from the phase that made it ahead of this run, or made - // here. It is taken only when it was made for what this run computes: the - // program's text, the host compiler and the text of the host modules. - const std::string stamp = programHash + '|' + compilerHash; - if (mode == ProgramMode::Run && given && given->compiled && given->stamp == stamp) { - for (auto const& w : given->warnings) mcpp::ui::warning(w); - programReport.compileDuration = given->compile; - if (!given->error.empty()) { - if (given->refusal) mcpp::build::refusal::record(*given->refusal); - return std::unexpected(given->error); - } - } else { - auto compiled = compile_binary(); - compileEnded = ProgressClock::now(); - if (mode == ProgramMode::Compile) { - made->compiled = true; - made->stamp = stamp; - made->bin = bin; - made->compile = std::chrono::duration_cast( - compileEnded - compileBegan); - made->warnings = std::move(compileWarnings); - made->refusal = compileRefusal; - if (!compiled) made->error = compiled.error(); - return {}; - } - for (auto const& w : compileWarnings) mcpp::ui::warning(w); - if (compileRefusal) mcpp::build::refusal::record(*compileRefusal); - if (!compiled) return std::unexpected(compiled.error()); - programReport.compileStart = compileBegan; - } // ── Run it; capture stdout(+stderr) and parse directives ──────────────── // Run with cwd = package root so the program's relative file writes (e.g. @@ -2200,32 +1297,18 @@ std::expected run_build_program_impl( return {}; } -PrecompiledProgram precompile_build_program( - mcpp::manifest::Manifest& m, - const fs::path& root, - const fs::path& hostCompiler, - const mcpp::toolchain::Toolchain& tc, - const mcpp::manifest::CppStandardConfig& cppStandard, - const BuildProgramEnv& env) { - PrecompiledProgram made; - // A refusal that comes before the compile is not carried: the program's turn - // runs the same checks and refuses in its own order. - (void)run_build_program_impl(ProgramMode::Compile, m, root, hostCompiler, tc, - cppStandard, env, nullptr, &made); - return made; +std::string synthesised_rule_program(const std::vector& modules) { + return synthesised_rule_program_impl(modules); } std::expected run_build_program( mcpp::manifest::Manifest& m, const fs::path& root, - const fs::path& hostCompiler, + const fs::path& /*hostCompiler*/, const mcpp::toolchain::Toolchain& tc, - const mcpp::manifest::CppStandardConfig& cppStandard, - const BuildProgramEnv& env, - const PrecompiledProgram* precompiled) { - return run_build_program_impl(precompiled ? ProgramMode::Run : ProgramMode::Whole, - m, root, hostCompiler, tc, cppStandard, env, - precompiled, nullptr); + const mcpp::manifest::CppStandardConfig& /*cppStandard*/, + const BuildProgramEnv& env) { + return run_build_program_impl(m, root, tc, env); } std::vector diff --git a/src/build/engine_package.cppm b/src/build/engine_package.cppm new file mode 100644 index 00000000..a85c9a97 --- /dev/null +++ b/src/build/engine_package.cppm @@ -0,0 +1,162 @@ +// mcpp.build.engine_package — the engine interface (`mcpp`, `mcpp.core`) as a +// package of the build program's graph (mcpp#797). +// +// A build program imports `mcpp.core` (spelled `mcpp` too, SPEC-007 R9.1). +// The interface is the engine's own text: it follows this mcpp's directive +// protocol, so it is generated from the binary rather than shipped as a file +// that could disagree with it. It is written once per mcpp version, as an +// ordinary package with a manifest, into +// +// $MCPP_HOME/engine//modules/ +// +// and the build program's graph depends on it like on any library. That is +// what lets the interface be compiled by the same plan, with the same flags, +// as every unit importing it, and what lets an editor read it: the directory +// is stated by `mcpp env` (`engineModules`) and by the build database's +// `mcpp:core` collection (SPEC-005). +// +// THE DIRECTORY IS IMMUTABLE once written, like a payload in the package +// store: its content is a function of the mcpp version alone. The dependency +// cache therefore keeps its compiled entry globally, under the address it has +// always had (`pkg/_engine/mcpp-build-module@`), and `mcpp cache` +// lists, verifies and collects it as before. + +export module mcpp.build.engine_package; + +import std; +import mcpp.build.directives; // kProtocolVersion -- the announced value has ONE source +import mcpp.build.hostprogram; // the interface text +import mcpp.home; +import mcpp.version; + +export namespace mcpp::build::engine { + +// The package's identity in a manifest, and the address its compiled entry +// has in the dependency cache. +inline constexpr std::string_view kNamespace = "mcpp"; +inline constexpr std::string_view kName = "core"; +inline constexpr std::string_view kCacheIndex = "_engine"; +inline constexpr std::string_view kCachePackage = "mcpp-build-module"; + +// `$MCPP_HOME/engine`: every version's package lives below it. +std::filesystem::path root(); +// The package directory of this mcpp version. +std::filesystem::path package_dir(); +// Whether `dir` is an engine package directory (of any version). +bool is_engine_package(const std::filesystem::path& dir); + +// The interface and its alias, as they are written: `mcpp.cppm` and +// `mcpp_core.cppm`. +std::pair module_texts(); + +// Writes the package of this version when it is absent or differs, atomically +// (a sibling directory renamed into place), and returns its directory. +std::expected ensure_package(); + +} // namespace mcpp::build::engine + +namespace mcpp::build::engine { + +namespace fs = std::filesystem; + +namespace { + +std::string substituted(std::string_view text, std::string_view placeholder, + std::string_view value) { + std::string out(text); + if (auto p = out.find(placeholder); p != std::string::npos) + out.replace(p, placeholder.size(), value); + return out; +} + +std::string manifest_text() { + return std::format( + "# Written by mcpp {0}: the engine interface a build program imports.\n" + "# Do not edit; it is replaced when it differs from what mcpp {0} writes.\n" + "[package]\n" + "namespace = \"{1}\"\n" + "name = \"{2}\"\n" + "version = \"{0}\"\n" + "\n" + "[language]\n" + "import_std = false\n" + "\n" + "[lib]\n" + "path = \"mcpp.cppm\"\n" + "\n" + "[build]\n" + "sources = [\"mcpp.cppm\", \"mcpp_core.cppm\"]\n" + "\n" + "[targets.core]\n" + "kind = \"lib\"\n", + mcpp::MCPP_VERSION, kNamespace, kName); +} + +std::string read_all(const fs::path& p) { + std::ifstream in(p, std::ios::binary); + if (!in) return {}; + return {std::istreambuf_iterator(in), {}}; +} + +} // namespace + +fs::path root() { return mcpp::home::root() / "engine"; } + +fs::path package_dir() { return root() / std::string(mcpp::MCPP_VERSION) / "modules"; } + +bool is_engine_package(const fs::path& dir) { + auto rel = dir.lexically_normal().lexically_relative(root().lexically_normal()); + return !rel.empty() && *rel.begin() != ".."; +} + +std::pair module_texts() { + auto mod = substituted(kMcppModuleSource, "@MODULE@", "export module"); + // Substituted rather than hardcoded so the announced version can never + // drift from the one the engine checks against. + mod = substituted(mod, "@PROTOCOL@", + std::to_string(mcpp::build::directives::kProtocolVersion)); + auto alias = substituted(kMcppCoreAliasSource, "@MODULE@", "export module"); + alias = substituted(alias, "@EXPORT@", "export"); + return {std::move(mod), std::move(alias)}; +} + +std::expected ensure_package() { + const auto dir = package_dir(); + const auto [mod, alias] = module_texts(); + const auto manifest = manifest_text(); + if (read_all(dir / "mcpp.toml") == manifest && read_all(dir / "mcpp.cppm") == mod + && read_all(dir / "mcpp_core.cppm") == alias) + return dir; + + std::error_code ec; + fs::create_directories(dir.parent_path(), ec); + static std::atomic serial{0}; + const auto scratch = dir.parent_path() / std::format( + ".modules.{}.{}", std::chrono::steady_clock::now().time_since_epoch().count(), + serial.fetch_add(1)); + fs::create_directories(scratch, ec); + if (ec) + return std::unexpected(std::format("cannot create {}: {}", scratch.string(), ec.message())); + for (auto const& [name, text] : {std::pair{"mcpp.toml", manifest}, + std::pair{"mcpp.cppm", mod}, + std::pair{"mcpp_core.cppm", alias}}) { + std::ofstream out(scratch / name, std::ios::binary | std::ios::trunc); + out << text; + out.close(); + if (!out) + return std::unexpected(std::format("cannot write {}", (scratch / name).string())); + } + // Another process may have written the same content meanwhile; either + // copy is this version's package. + fs::remove_all(dir, ec); + fs::rename(scratch, dir, ec); + if (ec) { + fs::remove_all(scratch, ec); + if (read_all(dir / "mcpp.cppm") != mod) + return std::unexpected(std::format("cannot place the engine interface at {}", + dir.string())); + } + return dir; +} + +} // namespace mcpp::build::engine diff --git a/src/build/host_module_compile.cppm b/src/build/host_module_compile.cppm deleted file mode 100644 index 4aa6f5fa..00000000 --- a/src/build/host_module_compile.cppm +++ /dev/null @@ -1,622 +0,0 @@ -// mcpp.build.host_module_compile — the bundled `mcpp` module and the host modules -// a build program imports, compiled once per key and kept by provenance (#748, B1). -// -// Split out of hostprogram.cppm, where `build_mcpp_module` and `build_host_module` -// compiled into each program's own directory. That file's header states why it -// must not grow (a clang 22 miscompile that followed the growth of the module it -// was in); the compile moved here, and the store it reads and writes is -// mcpp.build.host_module_store, which states what an entry is and where it lives. -// -// THE COMPILE IS UNCHANGED, AND ITS PLACE IS NOT. The argument vectors are the -// ones the two functions built, per compiler family, from the same BmiTraits and -// CommandDialect rows. What moved is the directory they run in and write to: a -// scratch directory of the store, where the files are made under the names the -// entry will have, and which is renamed into place when the compile succeeds. A -// consumer then names the entry's files: -// -// GCC finds BMIs by name under `/gcm.cache`, so a consumer is staged a -// copy of each BMI it imports in its own `gcm.cache`, as bmi_cache's -// consumers are. The compile of a host module is given the same staging -// in its scratch directory, so it finds the BMIs it imports the way its -// consumer will. -// Clang names a BMI with `-fmodule-file==`, and the path is the -// entry's. -// MSVC names it with `/reference =`. -// -// WHAT A KEY HOLDS. The store records the inputs and compares them field by field -// on a hit; this file decides what they are (`inputs_for`): the host compiler's -// identity, the standard flag, the flags of the compile, the flags that name the -// BMIs it imports and which BMIs those are, the digest of the text it compiles, -// and the provenance (the engine's version for the bundled module, the providing -// package for a host module). For GCC, whose use flags name nothing, the BMIs a -// compile can see are recorded by identity: an entry's key, or the std cache's. -// The compile of a host module is given every BMI made before it, in order, as -// it always was; that is the mechanism by which a rule can import another rule. -// -// A host module is compiled ALONE: its interface may import `std` and the bundled -// `mcpp` module and not a third package. A rule package is a leaf by construction. - -export module mcpp.build.host_module_compile; - -import std; -import mcpp.build.directives; // kProtocolVersion — the announced value has ONE source -import mcpp.build.host_module_store; -import mcpp.build.hostprogram; // the bundled module's text -import mcpp.libs.json; -import mcpp.log; -import mcpp.platform; -import mcpp.platform.process; -import mcpp.toolchain.dialect; -import mcpp.toolchain.fingerprint; // hash_file / hash_string -import mcpp.toolchain.hostflags; // bmi_reference_tokens -import mcpp.toolchain.model; -import mcpp.version; // MCPP_VERSION — the bundled module's text follows it - -export namespace mcpp::build { - -namespace fs = std::filesystem; - -// A BMI a compile can see: what GCC would find under `gcm.cache`, and what the -// other two families are told by flag. `identity` is what a dependent entry's -// key records for it: the key of the entry it came from, or, for `std`, the name -// of the std cache's directory (which is its own identity hash). -struct VisibleBmi { - std::string name; - fs::path path; - std::string identity; -}; - -// What one of these modules contributes to the build.mcpp compile. -struct McppModule { - std::vector useFlags; // how the consumer names the BMIs - fs::path object; // linked alongside build.mcpp - // `mcpp.core` (#734, E8): the same interface under the layer's name, a unit - // whose whole body re-exports `mcpp`. Empty for a host module. - fs::path aliasObject; - // The BMIs it made visible, for the compile after it. - std::vector bmis; - std::string entryKey; - bool reused = false; // found in the store, not compiled here - bool global = false; // kept in the global cache -}; - -// Where a build program's compiled imports may be kept. -struct ModuleStores { - // The global cache root; empty when `--cache local` or `off` asked for no - // global entry, in which case everything is kept in the workspace store. - fs::path cacheRoot; - // `/target/.build-mcpp/host-modules`. - fs::path workspaceStore; -}; - -// One host module to provide, and where its text came from (D4 of the plan). -struct HostModuleSource { - std::string logical; // the module name its source declares - fs::path interface; - // The provider's identity: the index, package and version for an index - // package, and the manifest's own name and version otherwise. - std::string index; - std::string package; - std::string version; - // True when the provider is an index package whose sources are in the - // immutable store. Only then may the entry be kept globally. - bool immutableSource = false; - // The provider's package root; the interface's directory when empty. - fs::path sourceRoot; -}; - -using CompileEnv = std::vector>; - -// The bundled `mcpp` module and its `mcpp.core` alias, for the build.mcpp -// compile in `consumerDir`. -std::expected -provide_mcpp_module(const ModuleStores& stores, const fs::path& consumerDir, - const fs::path& compiler, const std::vector& base, - const std::string& stdFlag, const mcpp::toolchain::Toolchain& tc, - const CompileEnv& env); - -// One host module. `useFlags` and `visible` are what the compile of the program -// would carry before this module: the flags that name the BMIs made so far, and -// those BMIs. -std::expected -provide_host_module(const ModuleStores& stores, const HostModuleSource& source, - const fs::path& consumerDir, const fs::path& compiler, - const std::vector& base, const std::string& stdFlag, - const mcpp::toolchain::Toolchain& tc, const CompileEnv& env, - const std::vector& useFlags, - const std::vector& visible); - -} // namespace mcpp::build - -namespace mcpp::build { - -namespace { - -namespace hm = mcpp::build::hostmods; - -constexpr int kStoreEpoch = 1; - -std::string join(const std::vector& argv) { - std::string joined; - for (auto const& a : argv) { if (!joined.empty()) joined += ' '; joined += a; } - return joined; -} - -// The host compiler's identity, as the std module's metadata states it: the -// compiler, its version, its driver (the declared identity, else a hash of the -// binary), the target, and the standard library. -// -// The driver's bytes are read once per path and process. A workspace of N -// programs asks N times and the binary is megabytes. -std::string driver_identity(const mcpp::toolchain::Toolchain& tc) { - if (!tc.driverIdent.empty()) return mcpp::toolchain::hash_string(tc.driverIdent); - if (tc.binaryPath.empty()) return {}; - static std::mutex m; - static std::map seen; - // UTF-8, never the code page: the name is an identity. - const auto u8 = tc.binaryPath.generic_u8string(); - const std::string key(reinterpret_cast(u8.data()), u8.size()); - std::lock_guard lock(m); - if (auto it = seen.find(key); it != seen.end()) return it->second; - return seen.emplace(key, mcpp::toolchain::hash_file(tc.binaryPath)).first->second; -} - -nlohmann::json toolchain_identity(const mcpp::toolchain::Toolchain& tc) { - return { - {"compiler", std::string(tc.compiler_name())}, - {"compiler_version", tc.version}, - {"driver_identity", driver_identity(tc)}, - {"target_triple", tc.targetTriple}, - {"stdlib", tc.stdlibId}, - {"stdlib_version", tc.stdlibVersion}, - }; -} - -// The inputs that hold for every entry of this file: what the compile is run -// with, apart from the text it compiles and where the text came from. -nlohmann::json common_inputs(std::string_view role, const mcpp::toolchain::Toolchain& tc, - const std::vector& base, const std::string& stdFlag, - const CompileEnv& env, const std::vector& use, - const std::vector& visible) { - nlohmann::json j; - j["epoch"] = kStoreEpoch; - j["role"] = std::string(role); - j["toolchain"] = toolchain_identity(tc); - j["std_flag"] = stdFlag; - j["base"] = base; - j["use"] = use; - // The toolchain's own environment (MSVC's INCLUDE and LIB) names the headers - // the compile reads, and two installs of one cl.exe can differ in it. - nlohmann::json envj = nlohmann::json::array(); - for (auto const& [k, v] : env) envj.push_back({k, v}); - j["env"] = std::move(envj); - nlohmann::json imports = nlohmann::json::array(); - for (auto const& b : visible) imports.push_back({{"module", b.name}, {"identity", b.identity}}); - j["imports"] = std::move(imports); - return j; -} - -// `-fmodule-file==` and `/reference =`; nothing for GCC, -// which finds the BMI by name under `gcm.cache`. -std::vector use_flags(const mcpp::toolchain::Toolchain& tc, std::string_view name, - const fs::path& bmi) { - if (tc.compiler == mcpp::toolchain::CompilerId::MSVC) - return mcpp::toolchain::bmi_reference_tokens(std::format(" /reference {}=", name), bmi); - if (mcpp::toolchain::is_clang(tc)) - return mcpp::toolchain::bmi_reference_tokens(std::format("-fmodule-file={}=", name), bmi); - return {"-fmodules"}; -} - -// What GCC finds by name: copies of the BMIs a compile imports, where the -// compile will look. A copy rather than a link, as the std module is staged, and -// for the same reason: a BMI that is rebuilt in place must not change under a -// compile that is reading it. -std::expected -stage_for_gcc(const fs::path& cwd, std::string_view bmiDir, - const std::vector>& bmis) { - std::error_code ec; - const fs::path dir = cwd / std::string(bmiDir); - fs::create_directories(dir, ec); - for (auto const& [file, from] : bmis) { - if (from.empty() || !fs::exists(from, ec)) continue; - fs::copy_file(from, dir / file, fs::copy_options::overwrite_existing, ec); - if (ec) - return std::unexpected(std::format("staging {} for a build program failed: {}", - file, ec.message())); - } - return {}; -} - -std::string substituted(std::string_view text, std::string_view placeholder, - std::string_view value) { - std::string out(text); - if (auto p = out.find(placeholder); p != std::string::npos) - out.replace(p, placeholder.size(), value); - return out; -} - -// The text of the bundled module and of its alias, as they are written and -// compiled. The placeholders stand for what the engine's own scanner would read -// as a second module of hostprogram.cppm (see kMcppModuleSource). -std::pair module_texts() { - auto mod = substituted(kMcppModuleSource, "@MODULE@", "export module"); - // Substituted rather than hardcoded so the announced version can never - // drift from the one the engine checks against. - mod = substituted(mod, "@PROTOCOL@", - std::to_string(mcpp::build::directives::kProtocolVersion)); - auto alias = substituted(kMcppCoreAliasSource, "@MODULE@", "export module"); - alias = substituted(alias, "@EXPORT@", "export"); - return {std::move(mod), std::move(alias)}; -} - -hm::Home home_for(const ModuleStores& stores, const mcpp::toolchain::Toolchain& tc, - bool global, std::string_view index, std::string_view package, - std::string_view version) { - const auto traits = mcpp::toolchain::bmi_traits(tc); - hm::Home h; - h.bmiDirName = std::string(traits.bmiDir); - h.manifestTag = std::string(traits.manifestPrefix); - if (global && !stores.cacheRoot.empty()) { - h.kind = hm::Home::Kind::Global; - h.cacheRoot = stores.cacheRoot; - h.index = std::string(index); - h.package = std::string(package); - h.version = std::string(version); - } else { - h.kind = hm::Home::Kind::Workspace; - h.workspaceStore = stores.workspaceStore; - } - return h; -} - -// Runs one compile of the store's producer. `what` names the step in the log -// and in the failure. The command is logged, always: it is the only place the -// argv of a SUCCESSFUL compile is observable (the `-isystem` rows that say -// whether a host toolchain's C library was attached, #622). -std::expected -run_step(std::string_view subject, const fs::path& cwd, const CompileEnv& env, - std::vector argv, const char* what) { - mcpp::log::verbose("buildmcpp-host", - std::format("{} {}: {}", subject, what, join(argv))); - auto r = mcpp::platform::process::capture_exec(argv, env, cwd.string()); - if (r.exit_code != 0) - return std::unexpected(std::format("{} {} failed (exit {}):\n{}", - subject, what, r.exit_code, r.output)); - return {}; -} - -// Moves what GCC wrote under `gcm.cache` to the entry's `bmi/`, and removes -// what is left there: the copies of the imported BMIs `stage_for_gcc` put in -// place for the compile (the std module's among them, tens of megabytes). They -// are inputs of the compile, not part of the entry, and an entry that kept them -// would carry a copy of `std` for every host module. -std::expected -collect_gcc_bmis(const fs::path& scratch, std::string_view bmiDir, - const std::vector& files) { - std::error_code ec; - for (auto const& f : files) { - fs::rename(scratch / std::string(bmiDir) / f, scratch / "bmi" / f, ec); - if (ec) - return std::unexpected(std::format("the compiler wrote no {}: {}", f, ec.message())); - } - fs::remove_all(scratch / std::string(bmiDir), ec); - return {}; -} - -} // namespace - -std::expected -provide_mcpp_module(const ModuleStores& stores, const fs::path& consumerDir, - const fs::path& compiler, const std::vector& base, - const std::string& stdFlag, const mcpp::toolchain::Toolchain& tc, - const CompileEnv& env) -{ - const auto traits = mcpp::toolchain::bmi_traits(tc); - const auto& dial = mcpp::toolchain::dialect_for(tc); - const bool msvc = tc.compiler == mcpp::toolchain::CompilerId::MSVC; - const bool clang = mcpp::toolchain::is_clang(tc); - const std::string ext(traits.bmiExt), obj(dial.objExt); - - const auto texts = module_texts(); - const std::string& moduleSrc = texts.first; - const std::string& aliasSrc = texts.second; - - // What the entry holds, under the names a consumer will use. - hm::Files files; - files.bmi = {"mcpp" + ext, "mcpp.core" + ext}; - files.obj = {"mcpp" + obj, "mcpp_core" + obj}; - - auto inputs = common_inputs("build-module", tc, base, stdFlag, env, {}, {}); - // The text the entry is compiled from: the module and its alias, exactly as - // written. The version is there too, for the reader of entry.json, and because - // the text follows it. - inputs["mcpp_version"] = std::string(mcpp::MCPP_VERSION); - inputs["interface_sha256"] = hm::sha256_hex(moduleSrc + '\x1f' + aliasSrc); - - // The bundled module is the engine's own text: identical in every project for - // one mcpp version and one host compiler, so it is kept globally (D4). - const auto home = home_for(stores, tc, /*global=*/true, "_engine", "mcpp-build-module", - std::string(mcpp::MCPP_VERSION)); - - auto entry = hm::obtain(home, inputs, files, - [&](const fs::path& S) -> std::expected { - constexpr std::string_view kSubject = "mcpp module"; - std::error_code ec; - { - std::ofstream os(S / "mcpp.cppm", std::ios::trunc); - os << moduleSrc; - if (!os) return std::unexpected(std::string("could not write mcpp module source")); - } - { - std::ofstream os(S / "mcpp_core.cppm", std::ios::trunc); - os << aliasSrc; - if (!os) return std::unexpected(std::string("could not write the mcpp.core unit")); - } - const fs::path bmiOut = S / "bmi", objOut = S / "obj"; - auto with_base = [&](std::vector head) { - for (auto const& b : base) head.push_back(b); - return head; - }; - auto step = [&](std::vector argv, const char* what) { - return run_step(kSubject, S, env, with_base(std::move(argv)), what); - }; - - if (msvc) { - // cl produces the .ifc and the .obj in one step. - const fs::path ifc = bmiOut / ("mcpp" + ext), coreIfc = bmiOut / ("mcpp.core" + ext); - const fs::path o1 = objOut / ("mcpp" + obj), o2 = objOut / ("mcpp_core" + obj); - std::vector argv{compiler.string()}; - for (auto f : dial.alwaysFlagsArgv) argv.emplace_back(f); - argv.push_back(stdFlag); - argv.push_back("/interface"); - for (auto f : dial.forceCxxLangArgv) argv.emplace_back(f); - argv.push_back(dial.compileOnly == std::string_view("/c") ? "/c" : "-c"); - argv.push_back("mcpp.cppm"); - argv.push_back("/ifcOutput"); argv.push_back(ifc.string()); - argv.push_back(std::string(dial.outputObjPrefix) + o1.string()); - if (auto r = step(std::move(argv), "compile"); !r) return r; - auto use = mcpp::toolchain::bmi_reference_tokens(" /reference mcpp=", ifc); - std::vector av{compiler.string()}; - for (auto f : dial.alwaysFlagsArgv) av.emplace_back(f); - av.push_back(stdFlag); - av.push_back("/interface"); - for (auto f : dial.forceCxxLangArgv) av.emplace_back(f); - av.push_back(dial.compileOnly == std::string_view("/c") ? "/c" : "-c"); - av.push_back("mcpp_core.cppm"); - av.push_back("/ifcOutput"); av.push_back(coreIfc.string()); - av.push_back(std::string(dial.outputObjPrefix) + o2.string()); - for (auto& f : use) av.push_back(f); - return step(std::move(av), "mcpp.core compile"); - } - - if (clang) { - const fs::path pcm = bmiOut / ("mcpp" + ext), corePcm = bmiOut / ("mcpp.core" + ext); - const fs::path o1 = objOut / ("mcpp" + obj), o2 = objOut / ("mcpp_core" + obj); - if (auto r = step({compiler.string(), stdFlag, "--precompile", - "mcpp.cppm", "-o", pcm.string()}, "precompile"); !r) return r; - if (auto r = step({compiler.string(), stdFlag, "-c", - pcm.string(), "-o", o1.string()}, "object"); !r) return r; - auto use = mcpp::toolchain::bmi_reference_tokens("-fmodule-file=mcpp=", pcm); - std::vector pre{compiler.string(), stdFlag, "--precompile", - "mcpp_core.cppm", "-o", corePcm.string()}; - for (auto& f : use) pre.push_back(f); - if (auto r = step(std::move(pre), "mcpp.core precompile"); !r) return r; - std::vector ob{compiler.string(), stdFlag, "-c", - corePcm.string(), "-o", o2.string()}; - for (auto& f : use) ob.push_back(f); - return step(std::move(ob), "mcpp.core object"); - } - - // GCC: BMIs are implicit under /gcm.cache, so nothing to name. - const fs::path o1 = objOut / ("mcpp" + obj), o2 = objOut / ("mcpp_core" + obj); - if (auto r = step({compiler.string(), stdFlag, "-fmodules", "-c", - "mcpp.cppm", "-o", o1.string()}, "compile"); !r) return r; - if (auto r = step({compiler.string(), stdFlag, "-fmodules", "-c", - "mcpp_core.cppm", "-o", o2.string()}, "mcpp.core compile"); !r) return r; - return collect_gcc_bmis(S, traits.bmiDir, files.bmi); - }); - if (!entry) return std::unexpected(entry.error()); - mcpp::log::verbose("buildmcpp-host", - std::format("bundled module mcpp: entry {} {} ({})", entry->key, - entry->global ? "global" : "workspace", - entry->reused ? "reused" : "compiled")); - - McppModule out; - out.entryKey = entry->key; - out.reused = entry->reused; - out.global = entry->global; - out.object = entry->obj(files.obj[0]); - out.aliasObject = entry->obj(files.obj[1]); - out.bmis = {{"mcpp", entry->bmi(files.bmi[0]), entry->key}, - {"mcpp.core", entry->bmi(files.bmi[1]), entry->key}}; - if (msvc || clang) { - out.useFlags = use_flags(tc, "mcpp", out.bmis[0].path); - for (auto& f : use_flags(tc, "mcpp.core", out.bmis[1].path)) out.useFlags.push_back(f); - } else { - out.useFlags = {"-fmodules"}; - if (auto r = stage_for_gcc(consumerDir, traits.bmiDir, - {{files.bmi[0], out.bmis[0].path}, - {files.bmi[1], out.bmis[1].path}}); !r) - return std::unexpected(r.error()); - } - return out; -} - -std::expected -provide_host_module(const ModuleStores& stores, const HostModuleSource& source, - const fs::path& consumerDir, const fs::path& compiler, - const std::vector& base, const std::string& stdFlag, - const mcpp::toolchain::Toolchain& tc, const CompileEnv& env, - const std::vector& useFlags, - const std::vector& visible) -{ - std::error_code ec; - if (!fs::exists(source.interface, ec)) { - return std::unexpected(std::format( - "host module '{}': no interface unit at {}\n" - " A package offering build rules must have a lib root " - "(src/.cppm or [lib] path).", - source.logical, source.interface.string())); - } - const auto traits = mcpp::toolchain::bmi_traits(tc); - const auto& dial = mcpp::toolchain::dialect_for(tc); - const bool msvc = tc.compiler == mcpp::toolchain::CompilerId::MSVC; - const bool clang = mcpp::toolchain::is_clang(tc); - const std::string ext(traits.bmiExt), objExt(dial.objExt); - - // A filesystem-safe stem. Partition separators and any path separator that - // sneaks into a logical name would otherwise create directories that do - // not exist. Dots are left ALONE on purpose: `a.b.rules.o` is a legal - // filename, GCC's own gcm.cache uses the dotted module name verbatim, and - // rewriting them would make the object name disagree with the BMI name for - // no gain. - std::string stem(source.logical); - for (auto& c : stem) if (c == ':' || c == '/' || c == '\\') c = '-'; - - hm::Files files; - files.bmi = {stem + ext}; - files.obj = {stem + objExt}; - - const auto digest = hm::sha256_file(source.interface); - if (digest.empty()) - return std::unexpected(std::format("host module '{}': cannot read {}", - source.logical, source.interface.string())); - - auto inputs = common_inputs("host-module", tc, base, stdFlag, env, useFlags, visible); - inputs["module"] = source.logical; - inputs["interface_sha256"] = digest; - inputs["provider"] = {{"index", source.index}, {"name", source.package}, - {"version", source.version}}; - // A package whose sources can change in place: what the interface includes is - // part of what was compiled, and it may sit anywhere in the package, so the - // package's tree is part of the key. - const bool global = source.immutableSource && !stores.cacheRoot.empty(); - inputs["source"] = global ? "immutable" : "workspace"; - if (!global) { - auto tree = hm::tree_digest(source.sourceRoot.empty() ? source.interface.parent_path() - : source.sourceRoot); - inputs["source_tree"] = tree.complete ? tree.hex : ("unbounded:" + hm::process_nonce()); - } - - const auto home = home_for(stores, tc, global, source.index, source.package, source.version); - - // The interface's LANGUAGE, stated rather than inferred from its extension. - // - // Measured on macOS CI: a rule package whose lib root is `rulepkg.ixx` - // made `clang++ --precompile rulepkg.ixx -o rulepkg.pcm` EXIT 0 AND WRITE - // NOTHING — clang's driver does not recognise `.ixx`, so it treated the file - // as a linker input, warned that it was unused, and succeeded. The failure - // surfaced one step later as `no such file or directory: …/rulepkg.pcm`, - // naming an output rather than the input that was never read. - // - // Every other module compile in mcpp already says this (BmiTraits:: - // moduleInterfaceLangFlag — `/interface /TP`, `-x c++-module`, `-x c++`); - // the host-module path was the one place that still let the driver guess. - // It is positional on GNU-style drivers, so it goes immediately before the - // input. - std::vector langArgv; - { - std::string_view lang = traits.moduleInterfaceLangFlag; - for (std::size_t i = 0; i < lang.size(); ) { - while (i < lang.size() && lang[i] == ' ') ++i; - auto j = lang.find(' ', i); - if (j == std::string_view::npos) j = lang.size(); - if (j > i) langArgv.emplace_back(lang.substr(i, j - i)); - i = j; - } - } - - auto entry = hm::obtain(home, inputs, files, - [&](const fs::path& S) -> std::expected { - const std::string subject = std::format("host module '{}'", source.logical); - const fs::path bmiOut = S / "bmi", objOut = S / "obj"; - auto with_base = [&](std::vector head) { - for (auto const& b : base) head.push_back(b); - for (auto const& f : useFlags) head.push_back(f); - return head; - }; - auto step = [&](std::vector argv, const char* what) { - return run_step(subject, S, env, with_base(std::move(argv)), what); - }; - const fs::path o = objOut / files.obj[0]; - - if (msvc) { - const fs::path ifc = bmiOut / files.bmi[0]; - std::vector argv{compiler.string()}; - for (auto f : dial.alwaysFlagsArgv) argv.emplace_back(f); - argv.push_back(stdFlag); - argv.push_back("/interface"); - for (auto f : dial.forceCxxLangArgv) argv.emplace_back(f); - argv.push_back("/c"); - argv.push_back(source.interface.string()); - argv.push_back("/ifcOutput"); argv.push_back(ifc.string()); - argv.push_back(std::string(dial.outputObjPrefix) + o.string()); - return step(std::move(argv), "compile"); - } - - if (clang) { - const fs::path pcm = bmiOut / files.bmi[0]; - std::vector pre{compiler.string(), stdFlag, "--precompile"}; - for (auto const& l : langArgv) pre.push_back(l); - pre.push_back(source.interface.string()); - pre.push_back("-o"); pre.push_back(pcm.string()); - if (auto r = step(std::move(pre), "precompile"); !r) return r; - // The precompile can succeed and write nothing when the driver - // ignored the input, which is exactly what happened above. - // Checked here so the diagnostic names the interface rather than - // a missing output. - std::error_code e2; - if (!fs::exists(pcm, e2)) { - return std::unexpected(std::format( - "host module '{}': the compiler accepted '{}' and produced no " - "BMI.\n" - " The interface's language is passed explicitly, so this " - "is not an extension\n" - " the driver failed to recognise — check that the file " - "really is a module interface.", - source.logical, source.interface.string())); - } - return step({compiler.string(), stdFlag, "-c", pcm.string(), "-o", o.string()}, - "object"); - } - - // GCC: BMIs are implicit under /gcm.cache, so nothing to name — - // which is also why the compile has to find every BMI it imports - // there. - { - std::vector> staged; - for (auto const& b : visible) - staged.emplace_back(b.path.filename().string(), b.path); - if (auto r = stage_for_gcc(S, traits.bmiDir, staged); !r) return r; - } - std::vector gccArgv{compiler.string(), stdFlag, "-fmodules", "-c"}; - for (auto const& l : langArgv) gccArgv.push_back(l); - gccArgv.push_back(source.interface.string()); - gccArgv.push_back("-o"); gccArgv.push_back(o.string()); - if (auto r = step(std::move(gccArgv), "compile"); !r) return r; - return collect_gcc_bmis(S, traits.bmiDir, files.bmi); - }); - if (!entry) return std::unexpected(entry.error()); - mcpp::log::verbose("buildmcpp-host", - std::format("host module '{}': entry {} {} ({})", source.logical, entry->key, - entry->global ? "global" : "workspace", - entry->reused ? "reused" : "compiled")); - - McppModule out; - out.entryKey = entry->key; - out.reused = entry->reused; - out.global = entry->global; - out.object = entry->obj(files.obj[0]); - out.bmis = {{source.logical, entry->bmi(files.bmi[0]), entry->key}}; - if (msvc || clang) { - out.useFlags = use_flags(tc, source.logical, out.bmis[0].path); - } else { - out.useFlags = {"-fmodules"}; - if (auto r = stage_for_gcc(consumerDir, traits.bmiDir, - {{files.bmi[0], out.bmis[0].path}}); !r) - return std::unexpected(r.error()); - } - return out; -} - -} // namespace mcpp::build diff --git a/src/build/host_module_store.cppm b/src/build/host_module_store.cppm deleted file mode 100644 index 268d9dcf..00000000 --- a/src/build/host_module_store.cppm +++ /dev/null @@ -1,381 +0,0 @@ -// mcpp.build.host_module_store — where what a build program imports is kept once -// it is compiled (#748, B1). -// -// A build program that imports `mcpp`, or a host module of a rule package, needs -// that module's BMI and object before its own compile. Until this module, each -// program compiled them into its own directory, before its own compile and -// whether or not an identical copy sat in the directory next door: four members -// that import one host module compiled the `mcpp` module four times and the host -// module four times. #748 measured 7.8 s per program outside the program's own -// `ran`, repeated for every program and every invocation. -// -// AN ENTRY IS ADDRESSED BY ITS INPUTS, AND THE INPUTS ARE RECORDED. The key is a -// hash of everything that reaches the compile: the host compiler's identity, the -// standard flag, the flags the compile carries, the BMIs it imports, and the text -// it compiles. The same inputs are written to entry.json, and a hit compares -// them field by field (mcpp.bmi_cache::probe_cached), never the hash alone. The -// flags of one compile are the flags of every consumer of its BMI, so a BMI is -// shared only between compiles that agree with it (hostprogram.cppm, P6): the -// agreement is a consequence of the key and is not checked afterwards. -// -// WHERE AN ENTRY LIVES DEPENDS ON WHERE ITS TEXT CAME FROM, and on nothing else. -// -// Global the text comes from the engine (the bundled `mcpp` module) or from -// an index package whose sources sit in the immutable store. The -// same name and version are the same bytes, so one entry serves every -// project of this machine. It lives in the global cache, in the -// layout of a dependency's entry, so `mcpp cache gc` collects it. -// Workspace the text comes from a path or git dependency or from a workspace -// member. Its sources can change without its name and version -// changing, which is the rule the dependency cache applies -// (plan.cpp, the admission of a package to the cache), so it is kept -// under the workspace's `target/` and is never written to the global -// cache. -// -// A host module is compiled ALONE, against `std` and `mcpp` only, so the local -// taint the dependency cache walks the closure for is, for a host module, its -// own package's alone. -// -// AN ENTRY IS PUBLISHED WHOLE. It is compiled into a staging directory and -// renamed into place with entry.json written last (mcpp.bmi_cache:: -// publish_staged), so it is never seen half-written. Threads of one process that -// want one key take one lock, so the key is compiled once, and the others find -// it; two processes that race both compile, and the second to publish finds the -// first's entry in place and discards its own. - -export module mcpp.build.host_module_store; - -import std; -import mcpp.bmi_cache; -import mcpp.libs.json; -import mcpp.toolchain.fingerprint; // hash_string — the one key hash of the build cache - -export namespace mcpp::build::hostmods { - -namespace fs = std::filesystem; - -// ─── SHA-256 ───────────────────────────────────────────────────────────── -// -// The digest of the text an entry was compiled from. The build's other content -// identities are 64-bit FNV-1a, which answers "has this changed"; an entry that -// other projects and other users of the machine reuse is held to "is this the -// same text" and records a digest that answers that. -std::string sha256_hex(std::string_view bytes); -// Empty when the file cannot be read: an unreadable file has no identity, and a -// caller that keyed on the empty string would key every unreadable file alike. -std::string sha256_file(const fs::path& p); - -// The digest of every regular file below `dir`: the relative name (UTF-8) and -// the content of each, in name order. A host module is compiled from one -// interface file, and the file may include what sits beside it; for a package -// whose sources can change in place, the interface's digest alone would not see -// an edit to an included file. `target`, `.git` and `.mcpp` are not entered. -// -// A tree of more than `kTreeFileLimit` files, or of more than `kTreeByteLimit` -// bytes, is not digested: `complete` is false, and the caller keys the entry on -// something that cannot match a later invocation, so the entry is reused only -// within this one. -inline constexpr std::size_t kTreeFileLimit = 4096; -inline constexpr std::uintmax_t kTreeByteLimit = 64ull * 1024 * 1024; -struct TreeDigest { - std::string hex; - bool complete = true; -}; -TreeDigest tree_digest(const fs::path& dir); - -// A stand-in identity that matches within this process and nowhere else. -std::string process_nonce(); - -// ─── Where an entry lives ──────────────────────────────────────────────── - -struct Home { - enum class Kind { Global, Workspace }; - Kind kind = Kind::Workspace; - - // Global: the cache root (mcpp::home::cache_root()) and the package address - // below it, `pkg//@//`. - fs::path cacheRoot; - std::string index; - std::string package; - std::string version; - - // Workspace: `/target/.build-mcpp/host-modules`; an entry is - // `//`. - fs::path workspaceStore; - - // What entry.json records about the BMI family, as for a dependency's entry. - std::string bmiDirName = "gcm.cache"; - std::string manifestTag = "gcm"; -}; - -// What an entry holds: file names below its `bmi/` and `obj/`. -struct Files { - std::vector bmi; - std::vector obj; -}; - -struct Entry { - fs::path dir; - std::string key; // 16 hex; what a dependent entry records as its import - bool reused = false; // found, and not compiled by this call - bool global = false; - - fs::path bmi(std::string_view name) const { return dir / "bmi" / std::string(name); } - fs::path obj(std::string_view name) const { return dir / "obj" / std::string(name); } -}; - -// Writes `files` into `scratch/bmi` and `scratch/obj`. The scratch directory is -// the producer's own, and is gone when the call returns. -using Producer = std::function(const fs::path& scratch)>; - -// The entry for `inputs` in `home`: found, or produced once and published. -// -// `inputs` is the complete description of what `produce` will compile. Two calls -// whose inputs are equal get one entry, and `produce` runs at most once for them -// in this process. `files` names what it writes. -std::expected -obtain(const Home& home, const nlohmann::json& inputs, const Files& files, - const Producer& produce); - -// The key `obtain` derives for `inputs`. -std::string key_of(const nlohmann::json& inputs); - -} // namespace mcpp::build::hostmods - -namespace mcpp::build::hostmods { - -// ─── SHA-256 (FIPS 180-4) ──────────────────────────────────────────────── - -namespace { - -constexpr std::array kSha256K = { - 0x428a2f98u, 0x71374491u, 0xb5c0fbcfu, 0xe9b5dba5u, 0x3956c25bu, 0x59f111f1u, - 0x923f82a4u, 0xab1c5ed5u, 0xd807aa98u, 0x12835b01u, 0x243185beu, 0x550c7dc3u, - 0x72be5d74u, 0x80deb1feu, 0x9bdc06a7u, 0xc19bf174u, 0xe49b69c1u, 0xefbe4786u, - 0x0fc19dc6u, 0x240ca1ccu, 0x2de92c6fu, 0x4a7484aau, 0x5cb0a9dcu, 0x76f988dau, - 0x983e5152u, 0xa831c66du, 0xb00327c8u, 0xbf597fc7u, 0xc6e00bf3u, 0xd5a79147u, - 0x06ca6351u, 0x14292967u, 0x27b70a85u, 0x2e1b2138u, 0x4d2c6dfcu, 0x53380d13u, - 0x650a7354u, 0x766a0abbu, 0x81c2c92eu, 0x92722c85u, 0xa2bfe8a1u, 0xa81a664bu, - 0xc24b8b70u, 0xc76c51a3u, 0xd192e819u, 0xd6990624u, 0xf40e3585u, 0x106aa070u, - 0x19a4c116u, 0x1e376c08u, 0x2748774cu, 0x34b0bcb5u, 0x391c0cb3u, 0x4ed8aa4au, - 0x5b9cca4fu, 0x682e6ff3u, 0x748f82eeu, 0x78a5636fu, 0x84c87814u, 0x8cc70208u, - 0x90befffau, 0xa4506cebu, 0xbef9a3f7u, 0xc67178f2u, -}; - -constexpr std::uint32_t rotr(std::uint32_t x, int n) { return (x >> n) | (x << (32 - n)); } - -class Sha256 { -public: - void update(const unsigned char* data, std::size_t len) { - total_ += len; - while (len > 0) { - const std::size_t take = std::min(len, 64 - fill_); - std::memcpy(block_.data() + fill_, data, take); - fill_ += take; data += take; len -= take; - if (fill_ == 64) { compress(); fill_ = 0; } - } - } - - std::string finish() { - const std::uint64_t bits = total_ * 8; - const unsigned char one = 0x80; - update(&one, 1); - const unsigned char zero = 0; - while (fill_ != 56) update(&zero, 1); - unsigned char len[8]; - for (int i = 0; i < 8; ++i) len[i] = static_cast(bits >> (56 - 8 * i)); - update(len, 8); - std::string out; - out.reserve(64); - for (auto w : h_) - for (int i = 3; i >= 0; --i) - out += std::format("{:02x}", static_cast((w >> (8 * i)) & 0xffu)); - return out; - } - -private: - void compress() { - std::array w{}; - for (int i = 0; i < 16; ++i) - w[i] = (std::uint32_t{block_[4 * i]} << 24) | (std::uint32_t{block_[4 * i + 1]} << 16) - | (std::uint32_t{block_[4 * i + 2]} << 8) | std::uint32_t{block_[4 * i + 3]}; - for (int i = 16; i < 64; ++i) { - const auto s0 = rotr(w[i - 15], 7) ^ rotr(w[i - 15], 18) ^ (w[i - 15] >> 3); - const auto s1 = rotr(w[i - 2], 17) ^ rotr(w[i - 2], 19) ^ (w[i - 2] >> 10); - w[i] = w[i - 16] + s0 + w[i - 7] + s1; - } - auto [a, b, c, d, e, f, g, h] = h_; - for (int i = 0; i < 64; ++i) { - const auto S1 = rotr(e, 6) ^ rotr(e, 11) ^ rotr(e, 25); - const auto ch = (e & f) ^ (~e & g); - const auto t1 = h + S1 + ch + kSha256K[i] + w[i]; - const auto S0 = rotr(a, 2) ^ rotr(a, 13) ^ rotr(a, 22); - const auto maj = (a & b) ^ (a & c) ^ (b & c); - const auto t2 = S0 + maj; - h = g; g = f; f = e; e = d + t1; d = c; c = b; b = a; a = t1 + t2; - } - h_[0] += a; h_[1] += b; h_[2] += c; h_[3] += d; - h_[4] += e; h_[5] += f; h_[6] += g; h_[7] += h; - } - - std::array h_ = { - 0x6a09e667u, 0xbb67ae85u, 0x3c6ef372u, 0xa54ff53au, - 0x510e527fu, 0x9b05688cu, 0x1f83d9abu, 0x5be0cd19u, - }; - std::array block_{}; - std::size_t fill_ = 0; - std::uint64_t total_ = 0; -}; - -// The file name as UTF-8, never through the code page (the path narrowing rule: -// a name that is an identity is `u8string()`). -std::string utf8(const fs::path& p) { - const auto u8 = p.generic_u8string(); - return std::string(reinterpret_cast(u8.data()), u8.size()); -} - -} // namespace - -std::string sha256_hex(std::string_view bytes) { - Sha256 h; - h.update(reinterpret_cast(bytes.data()), bytes.size()); - return h.finish(); -} - -std::string sha256_file(const fs::path& p) { - std::ifstream is(p, std::ios::binary); - if (!is) return {}; - Sha256 h; - std::array buf{}; - while (is.read(buf.data(), buf.size()) || is.gcount() > 0) - h.update(reinterpret_cast(buf.data()), - static_cast(is.gcount())); - return h.finish(); -} - -std::string process_nonce() { - static const std::string nonce = [] { - std::random_device rd; - return std::format("{:08x}{:08x}", rd(), rd()); - }(); - return nonce; -} - -TreeDigest tree_digest(const fs::path& dir) { - TreeDigest out; - std::vector> rows; // (relative name, file digest) - std::uintmax_t bytes = 0; - std::error_code ec; - fs::recursive_directory_iterator it(dir, fs::directory_options::skip_permission_denied, ec); - if (ec) { out.complete = false; return out; } - for (; it != fs::recursive_directory_iterator(); it.increment(ec)) { - if (ec) { out.complete = false; break; } - const auto name = utf8(it->path().filename()); - std::error_code tec; - if (it->is_directory(tec)) { - if (name == "target" || name == ".git" || name == ".mcpp") it.disable_recursion_pending(); - continue; - } - if (!it->is_regular_file(tec)) continue; - if (rows.size() >= kTreeFileLimit) { out.complete = false; break; } - if (const auto size = it->file_size(tec); !tec) bytes += size; - if (bytes > kTreeByteLimit) { out.complete = false; break; } - rows.emplace_back(utf8(it->path().lexically_relative(dir)), sha256_file(it->path())); - } - std::ranges::sort(rows); - Sha256 h; - for (auto const& [name, digest] : rows) { - const auto line = std::format("{}\t{}\n", name, digest); - h.update(reinterpret_cast(line.data()), line.size()); - } - out.hex = h.finish(); - return out; -} - -// ─── The entries ───────────────────────────────────────────────────────── - -std::string key_of(const nlohmann::json& inputs) { - // `dump()` is canonical: an object's members are kept in key order. - return mcpp::toolchain::hash_string("mcpp-host-module-store-v1\x1f" + inputs.dump()); -} - -namespace { - -mcpp::bmi_cache::CacheKey cache_key_of(const Home& home, const nlohmann::json& inputs) { - mcpp::bmi_cache::CacheKey ck; - ck.keyHex = key_of(inputs); - ck.inputs = inputs; - ck.bmiDirName = home.bmiDirName; - ck.manifestTag = home.manifestTag; - if (home.kind == Home::Kind::Global) { - ck.cacheRoot = home.cacheRoot; - ck.indexName = home.index; - ck.packageName = home.package; - ck.version = home.version; - } else { - ck.cacheRoot = home.workspaceStore; - ck.directDir = home.workspaceStore / ck.keyHex; - ck.indexName = "workspace"; - ck.packageName = "host-modules"; - ck.version = ""; - } - return ck; -} - -mcpp::bmi_cache::DepArtifacts artifacts_of(const Files& files) { - mcpp::bmi_cache::DepArtifacts a; - a.bmiFiles = files.bmi; - for (auto const& o : files.obj) a.objFiles.push_back({o, {}}); - return a; -} - -// One lock per entry address, for the threads of this process. A lock is never -// removed: there are a handful of entries, and a thread still waiting on a -// removed lock would be waiting on a destroyed one. -std::mutex& lock_for(const fs::path& dir) { - static std::mutex mapMutex; - static std::map> locks; - std::lock_guard guard(mapMutex); - auto& slot = locks[utf8(dir)]; - if (!slot) slot = std::make_unique(); - return *slot; -} - -} // namespace - -std::expected -obtain(const Home& home, const nlohmann::json& inputs, const Files& files, - const Producer& produce) -{ - const auto ck = cache_key_of(home, inputs); - const auto want = artifacts_of(files); - Entry entry; - entry.dir = ck.dir(); - entry.key = ck.keyHex; - entry.global = home.kind == Home::Kind::Global; - - // One thread at a time per address: the thread that finds nothing compiles, - // and the threads behind it find the entry. - std::lock_guard guard(lock_for(entry.dir)); - - if (mcpp::bmi_cache::probe_cached(ck, want).ok) { - mcpp::bmi_cache::touch_accessed(ck); - entry.reused = true; - return entry; - } - - auto staged = mcpp::bmi_cache::stage_entry(ck); - if (!staged) return std::unexpected(staged.error()); - if (auto r = produce(*staged); !r) { - std::error_code ec; - fs::remove_all(*staged, ec); - return std::unexpected(r.error()); - } - auto placed = mcpp::bmi_cache::publish_staged(ck, *staged, want); - if (!placed) return std::unexpected(placed.error()); - // Another process published first: its entry is the one in place. - entry.reused = !*placed; - return entry; -} - -} // namespace mcpp::build::hostmods diff --git a/src/build/hostprogram.cppm b/src/build/hostprogram.cppm index fc2aa0da..c5b91c98 100644 --- a/src/build/hostprogram.cppm +++ b/src/build/hostprogram.cppm @@ -12,9 +12,8 @@ // // WHAT IS LEFT HERE IS THE TEXT, AND THE COMPILE OF IT IS ELSEWHERE. The bundled // module's source and the question "does this program import that module" stay; -// `build_mcpp_module` and `build_host_module`, which compiled the module and the -// host modules into each program's own directory, moved to -// mcpp.build.host_module_compile (#748), which keeps them in a store by key. +// the text is written as a package of the build programs' graph by +// mcpp.build.engine_package, and compiled by that graph's plan (mcpp#797). export module mcpp.build.hostprogram; diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index bafbe132..fecf5ab0 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -2816,6 +2816,13 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, while (!lang.empty() && lang.front() == ' ') lang.remove_prefix(1); append(std::format(" unit_lang = {}\n", lang)); + } else if (mcpp::needs_explicit_cxx_language(cu.source) + && !traits.moduleImplLangFlag.empty()) { + // `build.mcpp`: an extension no driver knows (mcpp#797). + auto lang = traits.moduleImplLangFlag; + while (!lang.empty() && lang.front() == ' ') + lang.remove_prefix(1); + append(std::format(" unit_lang = {}\n", lang)); } } append("\n"); diff --git a/src/build/plan.cppm b/src/build/plan.cppm index e4f1aa87..9883d30d 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -89,9 +89,13 @@ struct CompileUnit { bool is_implementation_partition() const { return declaration == mcpp::modgraph::ModuleDeclaration::ImplementationPartition; } + // A build program's `.mcpp` source takes the module rule too: that rule + // states the unit's language per edge (`module_lang`), which a `.mcpp` + // file needs whether or not it imports anything (mcpp#797). bool uses_module_rule() const { return kind == mcpp::SourceKind::ModuleInterface - || (kind == mcpp::SourceKind::Cxx && !providesModule.empty()); + || (kind == mcpp::SourceKind::Cxx && !providesModule.empty()) + || mcpp::needs_explicit_cxx_language(source); } // This unit's outputs are already in the global cache: the backend emits // `stage_file` edges from the cache instead of a compile edge (and skips diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 82211163..c9bc5dc1 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -431,6 +431,9 @@ export struct DepSourceRoot { }; export struct BuildContext { + // The root is a build plugin, planned in the build-program configuration + // (mcpp#797, SPEC-007 P3). `mcpp pack` refuses it (P5). + bool buildPluginRoot = false; // THE PER-MACHINE JOB DEFAULT, carried so it is read once. // // `[build] default_jobs` in `$MCPP_HOME/config.toml` is the machine's @@ -780,6 +783,16 @@ export struct BuildOverrides { // mcpp.build.execute imported it). A pointer keeps the exported layout // trivial, and it also avoids copying the manifest per tool build. std::shared_ptr preloaded_manifest; + // THE BUILD PROGRAMS' GRAPH (mcpp#797). Set on the sub-build that compiles + // a build's programs (prepare/host_graph.cpp). Every package it shares + // with the build that asked is taken at the version that build resolved + // (`.` -> version): one package has one version across + // the two graphs, as the lock records. `build_program_graph` marks every + // package the sub-build loads as a build-program instance, which is what + // the compatibility reading `plugin-imports-from-build-dependencies` + // applies to. + std::map resolution_pins; + bool build_program_graph = false; // Nested source/tool builds inherit the consumer root's local development // OS. A dependency's own [xlings].subos is never consulted or propagated. std::shared_ptr diff --git a/src/build/prepare/features.cpp b/src/build/prepare/features.cpp index 023c9dbf..867d8e09 100644 --- a/src/build/prepare/features.cpp +++ b/src/build/prepare/features.cpp @@ -993,158 +993,8 @@ static std::expected step6_xlings_workspace_from_graph(Prepar return {}; } -// Every host module one package contributes, the lib root first. -// -// The lib root is what a rule package has always been: one unit, -// compiled alone, registered under the name it declares. A package -// that offers several rules through features (mcpp 2026.9.5.3+) -// lists their sources under `[features.] sources`, and those -// globs have been folded into `buildConfig.sources` by now for -// exactly the features the consumer activated. Every module -// INTERFACE unit among them is therefore a host module of its own, -// under its own declared name, and nothing else in the host-module -// path assumes one unit per package: `build_host_module` is per -// unit and the compile loop accumulates BMIs in list order, so a -// feature unit may import the lib root, which precedes it. -// -// Only sources the manifest LISTS take part. The inferred `src/**` -// of a package with no `sources` is not consulted, so a rule -// package published before this round exposes exactly what it -// exposed then; widening that implicitly would compile units that -// were written to be part of an ordinary library, alone. -// -// Split out of step6_host_module_registration (#734). -static std::vector host_module_units(const PrepareState& st, std::size_t p) { - auto identity = [&](std::size_t q) { - auto const& pkg = st.packages[q].manifest.package; - return pkg.namespace_.empty() ? pkg.name : pkg.namespace_ + "." + pkg.name; - }; - auto const& depPkg = st.packages[p]; - auto const& pkg = depPkg.manifest.package; - std::vector out; - auto push = [&](std::filesystem::path iface, std::string name) { - prov::HostModule hm; - hm.module = std::move(name); - hm.package = identity(p); - hm.nameSpace = pkg.namespace_; - hm.interface = std::move(iface); - out.push_back(std::move(hm)); - }; - // PROBING form: a host-module dependency whose interface is - // `.ixx` resolves to a `src/.cppm` that does not exist, - // and the consumer's build.mcpp is then handed a path to - // nothing. - auto rel = mcpp::manifest::resolve_lib_root_path( - depPkg.manifest, depPkg.root); - auto iface = depPkg.root / rel; - auto rootName = prov::host_module_name(iface, pkg.name); - // A missing lib root is reported as such by build_host_module, - // and that has to stay the diagnostic. Enumerating the listed - // units first would let one of them collide with the missing - // root's fallback name and report a collision between a file - // and a file that does not exist. - std::error_code ec; - if (!std::filesystem::exists(iface, ec)) { - push(iface, std::move(rootName)); - return out; - } - - std::set matched, dropped; - for (auto const& g : depPkg.manifest.buildConfig.sources) { - if (g.empty()) continue; - if (g[0] == '!') { - for (auto& f : mcpp::modgraph::expand_glob(depPkg.root, g.substr(1))) - dropped.insert(f.lexically_normal()); - } else { - for (auto& f : mcpp::modgraph::expand_glob(depPkg.root, g)) - matched.insert(f.lexically_normal()); - } - } - const auto root = iface.lexically_normal(); - // ORDERED BY WHAT THEY IMPORT, NOT BY WHERE THEY SIT. - // - // The compile loop accumulates BMIs in list order, so each - // entry sees only what precedes it. Path order was the previous - // rule and it is not a valid one: `rules/spirv.cppm` sorts - // before `src/surface.cppm`, so a member importing a unit its - // package shares was compiled first and failed with "failed to - // read compiled module ... imports must be built before being - // imported". Reproduced, and reproduced in both directions -- - // renaming the shared unit so its path sorted first made the - // same package build, which is what says the cause is the sort - // and nothing else. - // - // A package that works today is ordered IDENTICALLY: the sort - // below keeps path order wherever no import constrains it, so - // it differs only where the old order was already broken. - struct Unit { - std::filesystem::path path; - std::string name; - std::vector imports; - }; - // The lib root is the first node of the same sort (mcpp#720). - // Placing it ahead of the sort assumed that it imports no - // other unit of its package; a root that does was compiled - // before the unit it imports and failed with "module not - // found". As the first node it is still emitted first whenever - // it imports nothing of its own package, so the order of every - // package that built before is unchanged. - std::vector pending; - { - std::ifstream is(root); - std::stringstream buf; - if (is) buf << is.rdbuf(); - pending.push_back({root, std::move(rootName), - prov::declared_imports(buf.str())}); - } - for (auto const& f : matched) { // std::set: sorted - if (dropped.contains(f)) continue; - if (std::filesystem::equivalent(f, root, ec)) continue; - std::ifstream is(f); - if (!is) continue; - std::stringstream buf; - buf << is.rdbuf(); - auto text = buf.str(); - auto name = prov::declared_interface_name(text); - if (name.empty()) continue; - pending.push_back({f, std::move(name), prov::declared_imports(text)}); - } - - // Only names this package itself declares constrain anything. - // `import std;` is ahead of every entry here, and a name from - // another package is ordered by the cross-package DFS below - // rather than by this sort. - std::map byName; - for (std::size_t i = 0; i < pending.size(); ++i) - byName.emplace(pending[i].name, i); - - // deps[u] = the pending-list indices `u` imports BY NAME within this - // same package, in import order. A depth-first order from every unit in - // path order emits the first unit that can be emitted, which is what - // preserves path order in the unconstrained case, and is the order the - // build program's objects are linked in. - mcpp::graph::AdjacencyList deps(pending.size()); - for (std::size_t u = 0; u < pending.size(); ++u) - for (auto const& want : pending[u].imports) - if (auto it = byName.find(want); it != byName.end()) - deps[u].push_back(it->second); - std::vector roots(pending.size()); - std::iota(roots.begin(), roots.end(), std::size_t{0}); - - // A CYCLE IS LEFT TO THE COMPILER, ON PURPOSE. It is ill-formed C++ and - // the compiler says so with the two units named; refusing here would - // report the same fact in a worse place, and getting the ordering wrong - // is no longer possible either way -- so the edge that closes a cycle is - // skipped. - std::vector order; - order = *mcpp::graph::depth_first_order(deps, roots, mcpp::graph::Cycles::Skip); - - for (auto i : order) push(pending[i].path, std::move(pending[i].name)); - return out; -} - static std::expected>, std::string> -step6_host_module_registration(PrepareState& state) { +step6_tool_requests(PrepareState& state) { // ── #355: HOST tool provisioning ──────────────────────────────────── // // Runs AFTER feature activation (a tool target's gate is a feature) and @@ -1173,10 +1023,13 @@ step6_host_module_registration(PrepareState& state) { // store key are the edge-requested tool's in every respect. for (auto& edge : state.dependencyEdges) { const auto d = edge.dependencyPackageIndex; - if (d >= state.packages.size() || d >= state.activeFeaturesByPackage.size()) continue; + // The features of the instance this edge reaches (mcpp#797). + const auto& byInstance = edge.buildTable ? state.buildProgramFeaturesByPackage + : state.activeFeaturesByPackage; + if (d >= state.packages.size() || d >= byInstance.size()) continue; auto const& ft = state.packages[d].manifest.featureTools; if (ft.empty()) continue; - for (auto const& f : state.activeFeaturesByPackage[d]) + for (auto const& f : byInstance[d]) if (auto it = ft.find(f); it != ft.end()) for (auto const& t : it->second) if (std::ranges::find(edge.requestedTools, t) @@ -1195,154 +1048,25 @@ step6_host_module_registration(PrepareState& state) { // that asked for it rather than the project that needs it. state.provisionGraph = prov::propagate(state.dependencyEdges, state.packages.size()); - // #355 step 5: dependencies offering HOST build rules. Nothing is - // compiled here — the interface is handed to build_program.cppm, - // which compiles it in the SAME command as build.mcpp so the BMI - // and its consumer agree on standard, dialect and compiler by - // construction rather than by luck. - // - // Driven off the visible set rather than the root manifest, so a - // rule a library re-exports is importable from the consumer's - // build.mcpp without the consumer naming it. The name matching the - // old loop needed is gone with it: the edge already knows which - // package it points at. - // - // The registered name is the one the rule's SOURCE declares, not - // the package's name. See provisions::host_module_name for why the - // two had drifted apart and what that cost on Clang and MSVC. - std::set prefixWarned; - // Providers of host modules that THIS package sees directly. - auto directHostProviders = [&](std::size_t p) { - std::vector out; - if (p >= state.provisionGraph.visible.size()) return out; - for (auto const& pr : state.provisionGraph.visible[p]) { - if (pr.kind != prov::Kind::HostModule) continue; - if (pr.provider >= state.packages.size()) continue; - out.push_back(pr.provider); - } - return out; - }; - auto identity = [&](std::size_t p) { - auto const& pkg = state.packages[p].manifest.package; - return pkg.namespace_.empty() - ? pkg.name : pkg.namespace_ + "." + pkg.name; - }; - auto units = [&](std::size_t p) { return host_module_units(state, p); }; - // Host-module provider edges over EVERY package, built once: - // pkgHostDeps[p] is what directHostProviders(p) computes, reused - // below instead of re-querying provisionGraph per consumer. - mcpp::graph::AdjacencyList pkgHostDeps(state.packages.size()); - for (std::size_t p = 0; p < state.packages.size(); ++p) - pkgHostDeps[p] = directHostProviders(p); + // Each consumer's build program's graph: the libraries it uses and + // everything they reach (mcpp#797). for (std::size_t c = 0; c < state.provisionGraph.visible.size(); ++c) { - const auto direct = directHostProviders(c); - if (direct.empty()) continue; - std::set isDirect(direct.begin(), direct.end()); - - // #734 E7: what each direct provider could offer with a feature - // it does not have enabled here. Globs are expanded, nothing is - // read: the module names are looked up only if build.mcpp - // imports something no provider offers. - for (auto p : direct) { - auto const& dm = state.packages[p].manifest; - const auto& active = p < state.activeFeaturesByPackage.size() - ? state.activeFeaturesByPackage[p] : std::vector{}; - for (auto const& [fname, globs] : dm.buildConfig.featureSources) { - if (std::ranges::find(active, fname) != active.end()) continue; - mcpp::build::BuildProgramEnv::DormantFeature df; - df.package = identity(p); - df.feature = fname; - for (auto const& g : globs) - for (auto& f : mcpp::modgraph::expand_glob(state.packages[p].root, g)) - df.files.push_back(f); - if (!df.files.empty()) - state.dormantFeaturesByConsumer[c].push_back(std::move(df)); - } - } - - // Post-order DFS, so a rule's own host modules are compiled - // BEFORE it. That ordering is the entire mechanism: the - // compile loop in build_program.cppm accumulates the module - // flags as it goes, so each entry sees the BMIs of everything - // ahead of it, and "a rule may import another rule" needs no - // second machinery — only this sort. It walks only what `c` - // reaches, so a cycle elsewhere never fails `c`'s build. - auto topo = mcpp::graph::depth_first_order(pkgHostDeps, direct); - if (!topo) { - // A cycle, reported AS a cycle and naming the packages - // on it. A depth limit would answer a different - // question and would answer it later. - std::string ring; - bool first = true; - for (auto q : topo.error().cycle) { - if (!first) ring += " -> "; - first = false; - ring += identity(q); - } - return std::unexpected(std::format( - "build rules form an import cycle: {}\n" - " A rule's host modules are compiled before " - "it, so a cycle has no order that could satisfy " - "all of them.", ring)); - } - - std::vector ordered; - // The package each entry of `ordered` came from, for where its - // compiled module may be kept (#748, B1). - std::vector orderedProvider; - for (auto p : *topo) { - for (auto& hm : units(p)) { - hm.importable = isDirect.contains(p); - ordered.push_back(std::move(hm)); - orderedProvider.push_back(p); - } - } - - // Every provider on this consumer's rule closure, transitive - // ones included, and a rule imported by another rule declares - // payloads just as directly. - std::set reached(topo->begin(), topo->end()); - state.hostModuleProvidersByConsumer[c].assign(reached.begin(), reached.end()); - - if (auto clash = prov::host_module_collision(ordered)) - return std::unexpected(*clash); - for (std::size_t k = 0; k < ordered.size(); ++k) { - auto const& hm = ordered[k]; - // Warned once per (package, module), not once per consumer: - // a rule re-exported down a chain is visible to every - // package on it, and repeating one naming remark N times - // reads as N problems. - if (auto w = prov::reserved_prefix_warning( - hm.module, hm.nameSpace, hm.package)) { - if (prefixWarned.insert(hm.package + "\x1e" + hm.module).second) - mcpp::diag::warning("build/rule-namespace", *w); - } - mcpp::build::BuildProgramEnv::HostModuleRef ref{ - hm.module, hm.interface, hm.importable}; - // Where the compiled module may be kept (#748, B1): in the - // global cache only when the provider is an index package - // whose sources are in the immutable store, the rule the - // dependency cache applies to a package (plan.cpp). The - // module is compiled alone, so nothing it was built against - // can be local: the provider's own location decides. - { - const auto pi = orderedProvider[k]; - const auto& provider = state.packages[pi]; - const auto* ident = pi >= 1 && pi - 1 < state.dep_cache_identities.size() - ? &state.dep_cache_identities[pi - 1] : nullptr; - ref.providerName = identity(pi); - ref.providerVersion = provider.manifest.package.version; - ref.providerRoot = provider.root; - if (ident && ident->sourceKind == "version" - && mcpp::build::path_is_under_any(provider.root, state.storeRoots)) { - ref.immutableSource = true; - ref.providerIndex = ident->indexName; - ref.providerName = ident->packageName; - if (!ident->version.empty()) ref.providerVersion = ident->version; - } - } - state.hostModulesByConsumer[c].push_back(std::move(ref)); + std::vector work; + for (auto const& pr : state.provisionGraph.visible[c]) + if (pr.kind == prov::Kind::HostModule && pr.provider < state.packages.size()) + work.push_back(pr.provider); + if (work.empty()) continue; + std::set seen(work.begin(), work.end()); + while (!work.empty()) { + const auto p = work.back(); + work.pop_back(); + for (auto const& e : state.dependencyEdges) + if (e.consumerPackageIndex == p + && e.dependencyPackageIndex < state.packages.size() + && seen.insert(e.dependencyPackageIndex).second) + work.push_back(e.dependencyPackageIndex); } + state.buildProgramLibrariesByConsumer[c].assign(seen.begin(), seen.end()); } if (state.overrides.tool_depth >= mcpp::build::tool_store::kMaxDepth @@ -1899,6 +1623,25 @@ static std::expected step6_dependency_build_programs(PrepareS // files live in the CONSUMING project's tree — a registry package // root is shared across projects and may be read-only; it is never // written to. + // EVERY PROGRAM THIS PLAN RUNS IS COMPILED IN ONE BUILD of the build + // programs' graph, the dependencies', the members' and the root's + // (mcpp#797). They run one at a time, in their order, below and in + // phase 9; their compiles have no order and share their libraries. + // A failure here is not reported here: each program asks for its + // binary at its turn, alone, so the failure that reaches the user is + // the first in the order the programs run. + { + std::vector programs; + for (std::size_t i = 0; i < state.packages.size(); ++i) { + if (i == 0 && state.m->package.virtualRoot) continue; + if (i > 0 && !state.compilesHere(i)) continue; + programs.push_back(i); + } + if (auto r = ensure_build_programs(state, programs); !r) + mcpp::log::verbose("host-graph", std::format( + "the build programs' graph failed; each program is compiled at its turn: {}", + r.error())); + } for (std::size_t i = 1; i < state.packages.size(); ++i) { auto& pkg = state.packages[i]; // A package of programs runs its build program in its own tool @@ -1960,11 +1703,6 @@ static std::expected step6_dependency_build_programs(PrepareS / (dirSafe(pkg.manifest.package.name) + "@" + pkg.manifest.package.version); bpEnv.outDir = state.programOutDir(state.workRoot, pkg.manifest, bpEnv.features); bpEnv.genBase = bpEnv.outDir; - // What the program imports is kept once for the workspace, and in the - // global cache where it comes from the engine or the index (#748). - bpEnv.moduleStore = state.workRoot / "target" / ".build-mcpp" / "host-modules"; - if (state.cacheMode == CacheMode::Global) - bpEnv.moduleCacheRoot = mcpp::home::cache_root(); // mcpp#241: this package's resolved dependencies as // MCPP_DEP__DIR, from the authoritative edge graph (no // name-guessing); covers feature-activated deps too @@ -1978,12 +1716,16 @@ static std::expected step6_dependency_build_programs(PrepareS // #355: the host tools THIS package requested (resolved above). if (auto tit = state.toolEnvByConsumer.find(i); tit != state.toolEnvByConsumer.end()) bpEnv.toolPaths = tit->second; - bpEnv.hostModules = state.hostModulesByConsumer.count(i) - ? state.hostModulesByConsumer.at(i) - : decltype(bpEnv.hostModules){}; - bpEnv.dormantFeatures = state.dormantFeaturesByConsumer.count(i) - ? state.dormantFeaturesByConsumer.at(i) - : decltype(bpEnv.dormantFeatures){}; + // The program, as the build programs' graph compiled it (mcpp#797). + auto program = build_program_binary(state, i); + if (!program && !state.overrides.plan_only) return std::unexpected(program.error()); + if (program) { + bpEnv.programBinary = program->path; + bpEnv.programIdentity = program->identity; + bpEnv.programCompile = program->compile; + } else { + bpEnv.programError = program.error(); + } auto& bcDep = pkg.manifest.buildConfig; const auto mark = state.markDirectiveTail(pkg.manifest); const auto ldN = bcDep.ldflags.size(); @@ -2276,7 +2018,7 @@ std::expected phase6_features_and_host_tools(PrepareState& st if (auto r = step6_device_extensions_and_rules(state); !r) return std::unexpected(r.error()); if (auto r = step6_xlings_workspace_from_graph(state); !r) return std::unexpected(r.error()); - auto toolRequests = step6_host_module_registration(state); + auto toolRequests = step6_tool_requests(state); if (!toolRequests) return std::unexpected(toolRequests.error()); // The root build program's toolchain phase (mcpp#755): its host modules // are registered, and nothing that depends on the build toolchain has diff --git a/src/build/prepare/graph.cpp b/src/build/prepare/graph.cpp index d3bc9a67..7ecadf17 100644 --- a/src/build/prepare/graph.cpp +++ b/src/build/prepare/graph.cpp @@ -9,6 +9,7 @@ import mcpp.build.prepare_inputs; import std; import mcpp.diag; import mcpp.log; +import mcpp.build.engine_package; import mcpp.build.refusal; import mcpp.xlings.address_set; import mcpp.build.version_floor; @@ -818,6 +819,16 @@ step4b_resolve_identity(PrepareState& state, WorklistItemCtx& ctx) { } } + // THE BUILD PROGRAMS' GRAPH TAKES EACH PACKAGE AT THE VERSION THE BUILD + // THAT ASKED RESOLVED (mcpp#797): one package, one version, across the + // two graphs, as the lock records it. + if (!state.overrides.resolution_pins.empty() && !spec.isPath() && !spec.isGit()) { + const auto shortName = spec.shortName.empty() ? name : spec.shortName; + const auto k = spec.namespace_.empty() ? shortName : spec.namespace_ + "." + shortName; + if (auto pin = state.overrides.resolution_pins.find(k); + pin != state.overrides.resolution_pins.end()) + spec.version = "=" + pin->second; + } // Pin SemVer constraint before dedup/fetch. if (auto r = state.resolveSemver(spec, name); !r) { return std::unexpected(r.error()); @@ -1824,10 +1835,27 @@ step4b_finalize_dependency(PrepareState& state, WorklistItemCtx& ctx) { : sourceKind == "path" ? ctx.dep_root.string() : std::string{}, }); + // The engine interface (mcpp#797) is addressed in the dependency cache + // as it always was: by the engine's name and version, in the global + // cache, because its directory is immutable for that version. + if (sourceKind == "path" && mcpp::build::engine::is_engine_package(ctx.dep_root)) { + auto& ident = state.dep_cache_identities.back(); + ident.indexName = std::string(mcpp::build::engine::kCacheIndex); + ident.packageName = std::string(mcpp::build::engine::kCachePackage); + ident.version = state.dep_manifests.back()->package.version; + ident.sourceKind = "version"; + ident.sourceRef.clear(); + } const auto depPackageIndex = state.packages.size(); auto depPackage = makePackageRoot(state, ctx.dep_root, *state.dep_manifests.back()); if (!depPackage) return std::unexpected(depPackage.error()); if (selectedMember) { + // P3 (mcpp#797): a selected member that is a build plugin. + if (auto r = configure_build_plugin_root( + state, *state.dep_manifests.back(), ctx.dep_root, + feature_closure(*state.dep_manifests.back(), {})); + !r) + return std::unexpected(r.error()); depPackage->selectedMember = true; depPackage->memberProducts = state.selectedMembers.at(*selectedMember); } @@ -1934,6 +1962,23 @@ step4b_finalize_dependency(PrepareState& state, WorklistItemCtx& ctx) { childReq.version, selfIdx, ctx.dep_root, item.devOnly, /*buildOnly=*/true, /*fromBuildTable=*/true}); + // COMPAT(plugin-imports-from-build-dependencies): before mcpp#797 a + // build plugin's library was compiled inside its consumer's build + // program, so a package it listed in its own + // `[build-dependencies]` with `host-module = true` was importable + // from the library too. In the build programs' graph that library + // is an ordinary package, and its library's dependencies are its + // `[dependencies]`: the entry is read as one as well. Reported at + // the scan, where a unit of the library is seen to import it. + if (state.overrides.build_program_graph && child_spec.hostModuleDeclared + && child_spec.hostModule + && !state.dep_manifests.back()->dependencies.contains(child_name)) { + state.compatLibraryEdges.push_back({selfIdx, child_name}); + state.worklist.push_back({child_name, childReq, + thisDepLabel + " (build-dep, read as a dependency)", + childReq.version, selfIdx, ctx.dep_root, + item.devOnly, item.buildOnly}); + } } return {}; } diff --git a/src/build/prepare/host_graph.cpp b/src/build/prepare/host_graph.cpp new file mode 100644 index 00000000..2df83e4c --- /dev/null +++ b/src/build/prepare/host_graph.cpp @@ -0,0 +1,863 @@ +// host_graph.cpp -- the build programs' graph (mcpp#797). Declared in `:state`. +// +// A build program is a program of the build-program configuration: its main +// file is `build.mcpp` (or the program a rule feature synthesises), and its +// dependencies are the libraries its package asks for in +// `[build-dependencies]` -- and those a dependency hands on with +// `reexport = true`. The programs of one build that share a host toolchain and +// a C++ standard are the members of one workspace, written to +// +// /target/.build-mcpp/host// +// +// and planned and built by the engine itself, in a sub-build (the same entry +// the host tools use). Each dependency is compiled as the whole package it is +// -- its include directories, defines, features, its own dependencies, its +// implementation units -- and `build.mcpp` is compiled by the same plan, so a +// BMI and the program importing it come from one flag producer by +// construction. The workspace is written to disk so that `mcpp build` in that +// directory reproduces the compile of the build programs. +// +// A PROGRAM IS COMPILED WHEN ITS IDENTITY MOVED. The identity is computed from +// this build's resolution, without planning the sub-build: the engine +// version, the toolchain, the standard, the program's text, and every package +// of its graph -- name, version, source, the features of its build-program +// instance, and the content of a source tree that can change in place +// (`source_digest`). Every program of a build whose identity stands is run, or +// replayed, without the sub-build being planned at all. + +module mcpp.build.prepare; +import :state; + +import std; +import mcpp.diag; +import mcpp.log; +import mcpp.build.backend; +import mcpp.build.build_program; +import mcpp.build.engine_package; +import mcpp.build.ninja; +import mcpp.build.plan; +import mcpp.build.progress; +import mcpp.build.refusal; +import mcpp.build.provisions; +import mcpp.build.source_digest; +import mcpp.build.tool_store; +import mcpp.libs.json; +import mcpp.manifest; +import mcpp.modgraph.scanner; +import mcpp.platform.fs; +import mcpp.pm.mangle; +import mcpp.toolchain.fingerprint; +import mcpp.toolchain.triple; +import mcpp.ui; +import mcpp.version; + +namespace mcpp::build { + +namespace fs = std::filesystem; + +namespace { + +std::string qualified(const mcpp::manifest::Manifest& m) { + return m.package.namespace_.empty() ? m.package.name + : m.package.namespace_ + "." + m.package.name; +} + +const mcpp::manifest::Manifest& manifest_of(const PrepareState& state, std::size_t i) { + return i == 0 ? *state.m : state.packages[i].manifest; +} + +// A TOML basic string. +std::string toml_string(std::string_view s) { + std::string out = "\""; + for (char c : s) { + if (c == '"' || c == '\\') out.push_back('\\'); + out.push_back(c); + } + out.push_back('"'); + return out; +} + +// A bare key when it is one, a quoted key otherwise. +std::string key(std::string_view s) { + const bool bare = !s.empty() && std::ranges::all_of(s, [](char c) { + return std::isalnum(static_cast(c)) || c == '-' || c == '_'; + }); + return bare ? std::string(s) : toml_string(s); +} + +std::string path_text(const fs::path& p) { + const auto u = p.lexically_normal().generic_u8string(); + return std::string(reinterpret_cast(u.data()), u.size()); +} + +std::string list(const std::vector& v) { + std::string out = "["; + for (auto const& e : v) { + if (out.size() > 1) out += ", "; + out += toml_string(e); + } + return out + "]"; +} + +// The program text of package `i`: its `build.mcpp`, or the program its +// rules describe when it has none. +struct ProgramSource { + fs::path file; // empty when synthesised + std::string synthesised; // the text, when synthesised + bool exists() const { return !file.empty() || !synthesised.empty(); } +}; + +ProgramSource program_source(const PrepareState& state, std::size_t i) { + const auto root = i == 0 ? *state.root : state.packages[i].root; + std::error_code ec; + if (fs::is_regular_file(root / "build.mcpp", ec)) return {root / "build.mcpp", {}}; + auto const& rules = manifest_of(state, i).buildConfig.ruleModules; + if (rules.empty()) return {}; + return {{}, mcpp::build::synthesised_rule_program(rules)}; +} + +// The member a program is in the build programs' workspace: its package's +// qualified name, made a directory name, and a digest of its root (two +// packages of one name do not meet in one build, but a name says nothing +// about where it came from). +std::string member_name(const PrepareState& state, std::size_t i) { + std::string n = qualified(manifest_of(state, i)); + for (auto& c : n) + if (!std::isalnum(static_cast(c)) && c != '-' && c != '_') c = '-'; + const auto root = i == 0 ? *state.root : state.packages[i].root; + return std::format("{}-{}", n, mcpp::toolchain::hash_string(path_text(root)).substr(0, 6)); +} + +// The libraries program `i` uses: the providers of a build-program library +// this package sees, its own `[build-dependencies]` and what a dependency +// re-exports to it. +std::vector program_libraries(const PrepareState& state, std::size_t i) { + std::vector out; + if (i < state.provisionGraph.visible.size()) + for (auto const& pr : state.provisionGraph.visible[i]) + if (pr.kind == mcpp::build::provisions::Kind::HostModule + && pr.provider < state.packages.size() + && std::ranges::find(out, pr.provider) == out.end()) + out.push_back(pr.provider); + std::ranges::sort(out); + return out; +} + +// Every package below `roots` in this build's graph, over every edge: the +// packages the build programs' graph will contain. +std::vector closure_of(const PrepareState& state, std::vector roots) { + std::set seen(roots.begin(), roots.end()); + while (!roots.empty()) { + const auto p = roots.back(); + roots.pop_back(); + for (auto const& e : state.dependencyEdges) + if (e.consumerPackageIndex == p && e.dependencyPackageIndex < state.packages.size() + && seen.insert(e.dependencyPackageIndex).second) + roots.push_back(e.dependencyPackageIndex); + } + return {seen.begin(), seen.end()}; +} + +// How the build programs' graph names package `d`: the version this build +// resolved for an index package, the directory otherwise (a `path` package, +// a git checkout, a workspace member). +struct DependencyLine { + std::string ns, name, source; // `source`: the inline table's location keys +}; + +DependencyLine dependency_line(const PrepareState& state, std::size_t d) { + auto const& m = state.packages[d].manifest; + DependencyLine line{m.package.namespace_, m.package.name, {}}; + const auto* ident = d >= 1 && d - 1 < state.dep_cache_identities.size() + ? &state.dep_cache_identities[d - 1] : nullptr; + if (ident && ident->sourceKind == "version" && !engine::is_engine_package(state.packages[d].root)) { + line.source = std::format("version = {}", toml_string("=" + (ident->version.empty() + ? m.package.version + : ident->version))); + if (!ident->packageName.empty()) { + // The index address the build resolved, which a Form B package's + // manifest may spell differently. + const auto dot = ident->packageName.find('.'); + if (line.ns.empty() && dot != std::string::npos) { + line.ns = ident->packageName.substr(0, dot); + line.name = ident->packageName.substr(dot + 1); + } + } + } else { + line.source = std::format("path = {}", toml_string(path_text(state.packages[d].root))); + } + return line; +} + +std::string features_of(const PrepareState& state, std::size_t d) { + const auto& f = d < state.buildProgramFeaturesByPackage.size() + ? state.buildProgramFeaturesByPackage[d] : std::vector{}; + std::vector named; + for (auto const& x : f) + if (x != "default") named.push_back(x); + return std::format("features = {}, default-features = {}", list(named), + std::ranges::find(f, std::string("default")) != f.end() ? "true" : "false"); +} + +// The toolchain the build programs are compiled with, as a sub-build is told +// it (the host tools' rule, SPEC-004 §10.1), and its identity. +struct HostToolchain { + std::string spec; + std::string identity; +}; + +std::expected host_toolchain(PrepareState& state) { + auto host = state.host_tc_for_build_program(); + if (!host) return std::unexpected(host.error()); + HostToolchain t; + t.spec = state.host_spec_for_build_program(); + t.identity = std::format("{}|{}|{}|{}", t.spec, path_text(host->first), + host->second.version, host->second.targetTriple); + return t; +} + +fs::path host_root(const PrepareState& state) { + return state.workRoot / "target" / ".build-mcpp" / "host"; +} + +std::string standard_of(const PrepareState& state, std::size_t i) { + auto const& s = manifest_of(state, i).cppStandard.canonical; + return s.empty() ? std::string("c++23") : s; +} + +// The identity of program `i`: what its compile and link read. +std::string program_identity(PrepareState& state, std::size_t i, const ProgramSource& src, + const HostToolchain& tc) { + std::string text = std::format("mcpp-build-program-v1\nengine {}\ntoolchain {}\nstandard {}\n", + mcpp::MCPP_VERSION, tc.identity, standard_of(state, i)); + text += src.file.empty() ? "program synthesised " + mcpp::toolchain::hash_string(src.synthesised) + : "program " + mcpp::toolchain::hash_file(src.file); + text += '\n'; + const auto libs = program_libraries(state, i); + for (auto d : libs) text += std::format("uses {}\n", qualified(state.packages[d].manifest)); + for (auto d : closure_of(state, libs)) { + auto const& pkg = state.packages[d]; + const auto* ident = d >= 1 && d - 1 < state.dep_cache_identities.size() + ? &state.dep_cache_identities[d - 1] : nullptr; + std::string source = ident ? ident->sourceKind + ":" + ident->sourceRef : ""; + std::string tree; + // A tree that can change in place: anything but an index package or + // a pinned git checkout. + if (!ident || (ident->sourceKind != "version" && ident->sourceKind != "git")) { + const auto cache = host_root(state) / "digests" + / (mcpp::toolchain::hash_string(path_text(pkg.root)) + ".tsv"); + tree = mcpp::build::source_tree_digest(pkg.root, cache); + } + const auto& f = d < state.buildProgramFeaturesByPackage.size() + ? state.buildProgramFeaturesByPackage[d] : std::vector{}; + text += std::format("package {} {} {} [{}] {}\n", qualified(pkg.manifest), + ident ? ident->version : pkg.manifest.package.version, source, + list(f), tree); + } + return mcpp::toolchain::hash_string(text); +} + +// The primary modules package `d` exports in its build-program instance: the +// units its sources select, with the sources of the features that instance +// has and the target instance's manifest may not have folded in. +std::set exported_modules(const PrepareState& state, std::size_t d, + const std::vector& features) { + auto const& pkg = state.packages[d]; + std::set files; + for (auto const& f : mcpp::modgraph::package_source_files(pkg.root, pkg.manifest)) + files.insert(f); + for (auto const& f : features) + if (auto it = pkg.manifest.buildConfig.featureSources.find(f); + it != pkg.manifest.buildConfig.featureSources.end()) + for (auto const& g : it->second) + for (auto const& file : mcpp::modgraph::expand_glob(pkg.root, g)) + files.insert(file); + std::set out; + for (auto const& file : files) { + std::ifstream in(file, std::ios::binary); + const std::string text{std::istreambuf_iterator(in), {}}; + for (auto const& m : mcpp::pm::declared_module_roots(text)) out.insert(m); + } + return out; +} + +const std::vector& program_features(const PrepareState& state, std::size_t d) { + static const std::vector none; + return d < state.buildProgramFeaturesByPackage.size() ? state.buildProgramFeaturesByPackage[d] + : none; +} + +// What a program may import, refused by name before anything is compiled: a +// module of a package that reaches the program only as another library's +// dependency, a module a library offers behind a feature that is not enabled +// (SPEC-007 R9.7), and a module no library provides. Left to the compiler, +// each is `failed to read compiled module`, naming neither the package nor +// the key that would change the answer. +std::expected check_program_imports(PrepareState& state, std::size_t i, + const ProgramSource& src) { + std::string text = src.synthesised; + if (!src.file.empty()) { + std::ifstream in(src.file, std::ios::binary); + text.assign(std::istreambuf_iterator(in), {}); + } + const auto wants = mcpp::pm::imported_module_names(text); + if (wants.empty()) return {}; + const auto libs = program_libraries(state, i); + std::set available{"std", "std.compat", "mcpp", "mcpp.core"}; + std::map providerOf; + for (auto d : libs) + for (auto const& m : exported_modules(state, d, program_features(state, d))) { + available.insert(m); + providerOf.emplace(m, d); + } + std::set prerequisites; + for (auto d : closure_of(state, libs)) + if (std::ranges::find(libs, d) == libs.end()) + for (auto const& m : exported_modules(state, d, program_features(state, d))) + prerequisites.insert(m); + + auto const& m = manifest_of(state, i); + for (auto const& want : wants) { + if (available.contains(want)) continue; + if (prerequisites.contains(want)) { + refusal::record(refusal::Code::HostModuleMissing); + return std::unexpected(std::format( + "build.mcpp imports '{}', which reaches this build only as a " + "dependency of another build dependency.\n" + " A library's own dependencies are not importable from the " + "program unless that library hands them on.\n" + " Either depend on the package providing '{}' directly, or " + "have the library that owns it re-export it:\n" + " = {{ ..., reexport = true }}", + want, want)); + } + // #734 E7: a module a library offers behind a feature this build has + // not enabled. Its files are read only now, on the way to an error. + for (auto d : libs) { + auto const& pm = state.packages[d].manifest; + const auto& active = program_features(state, d); + for (auto const& [feature, globs] : pm.buildConfig.featureSources) { + if (std::ranges::find(active, feature) != active.end()) continue; + for (auto const& g : globs) + for (auto const& file : mcpp::modgraph::expand_glob(state.packages[d].root, g)) { + std::ifstream in(file, std::ios::binary); + const std::string t{std::istreambuf_iterator(in), {}}; + const auto decl = mcpp::pm::declared_module_roots(t); + if (std::ranges::find(decl, want) == decl.end()) continue; + refusal::record(refusal::Code::HostModuleMissing); + return std::unexpected(std::format( + "build.mcpp imports '{}'\n" + " provided by: {}, feature \"{}\" (not enabled)\n" + " hint: enable it on the dependency edge: {} = {{ ..., " + "features = [\"{}\"] }}", + want, qualified(pm), feature, qualified(pm), feature)); + } + } + } + // A dependency that could plausibly be meant: one the program does not + // use as a library. Ranked by the one relation available without + // resolving anything: a key sharing the imported name's first segment. + const auto dot = want.find('.'); + const auto head = dot == std::string::npos ? want : want.substr(0, dot); + std::string near, all; + auto scan = [&](const std::map& deps, + std::string_view section, bool buildTable) { + for (auto const& [k, s] : deps) { + const bool library = buildTable + && (s.hostModuleDeclared ? s.hostModule : s.tools.empty() && s.artifacts.empty()); + if (library) continue; + const auto one = std::format("{} (in [{}])", k, section); + auto& dst = k.starts_with(head + ".") || k == head ? near : all; + if (!dst.empty()) dst += ", "; + dst += one; + } + }; + scan(m.buildDependencies, "build-dependencies", true); + scan(m.dependencies, "dependencies", false); + const std::string candidates = near.empty() ? all : near; + std::string importable; + for (auto const& a : available) { + if (a == "std" || a == "std.compat" || a == "mcpp" || a == "mcpp.core") continue; + if (!importable.empty()) importable += ", "; + importable += a; + } + refusal::record(refusal::Code::HostModuleMissing); + return std::unexpected(std::format( + "build.mcpp imports '{}', and no build dependency provides it.\n" + " A build program imports the public modules of the libraries in its\n" + " package's [build-dependencies]; an entry that lists only `tools`, or\n" + " states `host-module = false`, asks for no library:\n" + " [build-dependencies.]\n" + " = {{ version = \"...\" }} # or `host-module = true` beside `tools`\n" + " importable here: {}\n" + "{}", + want, importable.empty() ? "(none)" : importable, + candidates.empty() + ? " This package declares no dependency that could provide it." + : std::format(" declared without its library: {}", candidates))); + } + return {}; +} + +// SPEC-007 R9.6: a module whose name opens with `mcpp.` comes only +// from a package of namespace `mcpp`. Warned once per (package, module). +void warn_reserved_prefixes(const PrepareState& state, std::size_t i) { + static std::set warned; + for (auto d : program_libraries(state, i)) { + auto const& pm = state.packages[d].manifest; + for (auto const& m : exported_modules(state, d, program_features(state, d))) + if (auto w = mcpp::build::provisions::reserved_prefix_warning(m, pm.package.namespace_, + qualified(pm))) + if (warned.insert(qualified(pm) + "\x1e" + m).second) + mcpp::diag::warning("build/rule-namespace", *w); + } +} + +// What `ensure_build_programs` learned about one program. +struct Program { + std::size_t index = 0; + ProgramSource source; + std::string member; + std::string identity; + std::string group; // the group directory's name +}; + +// The record of the programs a group directory holds. +nlohmann::json read_record(const fs::path& file) { + std::ifstream in(file); + auto j = in ? nlohmann::json::parse(in, nullptr, false) : nlohmann::json{}; + return j.is_object() ? j : nlohmann::json::object(); +} + +void write_if_changed(const fs::path& file, const std::string& text) { + std::error_code ec; + { + std::ifstream in(file, std::ios::binary); + if (in && std::string{std::istreambuf_iterator(in), {}} == text) return; + } + fs::create_directories(file.parent_path(), ec); + const auto tmp = file.string() + ".tmp"; + { + std::ofstream out(tmp, std::ios::binary | std::ios::trunc); + out << text; + } + fs::rename(tmp, file, ec); +} + +// The manifests of one group: the workspace and a member per program. +void write_group(PrepareState& state, const fs::path& dir, const std::vector& programs, + const std::string& standard, const fs::path& enginePackage) { + std::string root = std::format( + "# Written by mcpp {}: the build programs of {} (mcpp#797).\n" + "# `mcpp build` here reproduces their compile; it is rewritten on every build.\n" + "[workspace]\nmembers = [", + mcpp::MCPP_VERSION, path_text(state.workRoot)); + for (std::size_t k = 0; k < programs.size(); ++k) + root += (k ? ", " : "") + toml_string(programs[k].member); + // The engine that wrote it is its floor: the workspace layer below is read + // by 2026.10.10.1 and later. + root += std::format("]\n\n[workspace.package]\nmcpp = \">={}\"\n", mcpp::MCPP_VERSION); + root += "\n[workspace.profile.build-program]\nopt = 0\ndebug = false\n"; + if (!state.m->indices.empty()) { + root += "\n[workspace.indices]\n"; + for (auto const& [name, idx] : state.m->indices) { + std::vector fields; + if (idx.is_local()) fields.push_back("path = " + toml_string(path_text(idx.path))); + if (!idx.url.empty()) fields.push_back("url = " + toml_string(idx.url)); + if (!idx.rev.empty()) fields.push_back("rev = " + toml_string(idx.rev)); + if (!idx.tag.empty()) fields.push_back("tag = " + toml_string(idx.tag)); + if (!idx.branch.empty()) fields.push_back("branch = " + toml_string(idx.branch)); + if (!idx.artifact.empty()) fields.push_back("artifact = " + toml_string(idx.artifact)); + if (!idx.source.empty()) fields.push_back("source = " + toml_string(idx.source)); + std::string row; + for (auto const& f : fields) row += (row.empty() ? "" : ", ") + f; + root += std::format("{} = {{ {} }}\n", key(name.empty() ? "default" : name), row); + } + } + write_if_changed(dir / "mcpp.toml", root); + + for (auto const& p : programs) { + const auto memberDir = dir / p.member; + fs::path main = p.source.file; + if (main.empty()) { + main = memberDir / "build.mcpp"; + write_if_changed(main, p.source.synthesised); + } + std::string m = std::format( + "# The build program of {} ({}).\n" + "[package]\nname = {}\nversion = \"0.0.0\"\nstandard = {}\n\n" + "[build]\nsources = []\n\n", + qualified(manifest_of(state, p.index)), path_text(main), toml_string(p.member), + toml_string(standard)); + std::map> tables; // namespace -> lines + tables[std::string(engine::kNamespace)].push_back( + std::format("{} = {{ path = {} }}", key(engine::kName), toml_string(path_text(enginePackage)))); + for (auto d : program_libraries(state, p.index)) { + const auto line = dependency_line(state, d); + tables[line.ns].push_back(std::format("{} = {{ {}, {} }}", key(line.name), + line.source, features_of(state, d))); + } + if (auto it = tables.find(""); it != tables.end()) { + m += "[dependencies]\n"; + for (auto const& l : it->second) m += l + "\n"; + m += "\n"; + } + for (auto const& [ns, lines] : tables) { + if (ns.empty()) continue; + m += std::format("[dependencies.{}]\n", key(ns)); + for (auto const& l : lines) m += l + "\n"; + m += "\n"; + } + m += std::format("[targets.build-program]\nkind = \"bin\"\nmain = {}\n" + "windows_code_page = \"utf-8\"\n", toml_string(path_text(main))); + write_if_changed(memberDir / "mcpp.toml", m); + } +} + +// The packages of this build that the build programs' graph shares, at the +// version this build resolved. +std::map resolution_pins(const PrepareState& state) { + std::map pins; + for (auto const& [k, rec] : state.resolved) { + if (rec.depIndex >= state.dep_cache_identities.size()) continue; + auto const& ident = state.dep_cache_identities[rec.depIndex]; + if (ident.sourceKind != "version" || rec.version.empty()) continue; + pins[k.ns.empty() ? k.shortName : k.ns + "." + k.shortName] = rec.version; + } + return pins; +} + +// Build the group's programs `goals`: one sub-build, one ninja. +std::expected, std::string> +build_group(PrepareState& state, const fs::path& dir, const std::vector& all, + const std::vector& goals, const HostToolchain& tc) { + BuildOverrides sub; + sub.project_root = dir; + sub.work_dir = dir; + sub.target_triple = ""; + sub.toolchain = tc.spec; + sub.profile = "build-program"; + sub.cache_mode = state.overrides.cache_mode; + sub.strict = state.overrides.strict; + sub.tool_depth = state.overrides.tool_depth + 1; + sub.tool_chain_sources = state.overrides.tool_chain_sources; + sub.tool_chain = state.overrides.tool_chain.empty() + ? std::string("root → build programs") + : state.overrides.tool_chain + " → build programs"; + sub.inherited_runtime_selection = + std::make_shared(state.runtimeSelection); + sub.inherited_runtime_binding = + std::make_shared(state.runtimeBindingSnapshot); + sub.resolution_pins = resolution_pins(state); + sub.build_program_graph = true; + for (auto const& p : all) sub.workspace_members.push_back(p.member); + + auto reproduce = [&] { + return std::format("\n the build programs' workspace: {}\n" + " re-run it directly: cd {} && mcpp build --profile build-program\n" + " (MCPP_TOOL_BUILD_VERBOSE=1 shows the inner build's output unfiltered)", + path_text(dir), path_text(dir)); + }; + if (state.overrides.tool_depth >= mcpp::build::tool_store::kMaxDepth) + return std::unexpected(std::format( + "build programs nested more than {} levels deep — a build program's library " + "is its own build program's library.\n chain: {}", + mcpp::build::tool_store::kMaxDepth, sub.tool_chain)); + + mcpp::log::ScopedTimer timer("host-graph", std::format("build programs in {}", path_text(dir))); + // THE SUB-BUILD SAYS NOTHING ON THE TERMINAL. Its toolchain, its members + // and its compiles are the build programs' line ("build.mcpp + // compiled · ran"); what it would narrate is in the log, and a failure + // comes back as this function's error. + struct Quiet { + bool prev = mcpp::ui::is_quiet(); + Quiet() { mcpp::ui::set_quiet(true); } + ~Quiet() { mcpp::ui::set_quiet(prev); } + } quiet; + auto ctx = prepare_build(/*print_fingerprint=*/false, /*includeDevDeps=*/false, {}, sub); + if (!ctx) + return std::unexpected(std::format("planning the build programs failed: {}{}", + ctx.error(), reproduce())); + timer.mark("planned"); + + std::map binaries; + std::vector ninjaGoals; + for (auto const& lu : ctx->plan.linkUnits) { + if (lu.kind != mcpp::build::LinkUnit::Binary || lu.targetName != "build-program") continue; + for (auto const* g : goals) + if (lu.memberOf == g->member) { + binaries[g->member] = ctx->plan.outputDir / lu.output; + ninjaGoals.push_back(lu.output.generic_string()); + } + } + for (auto const* g : goals) + if (!binaries.contains(g->member)) + return std::unexpected(std::format( + "the build programs' plan has no program for '{}'{}", + qualified(manifest_of(state, g->index)), reproduce())); + + auto be = mcpp::build::make_ninja_backend(); + mcpp::build::BuildOptions bopt; + bopt.ninjaTargets = ninjaGoals; + bopt.report = mcpp::build::BuildOptions::Report::Caller; + if (const char* v = std::getenv("MCPP_TOOL_BUILD_VERBOSE"); + v && *v && std::string_view(v) != "0") + bopt.verbose = true; + auto br = be->build(ctx->plan, bopt); + timer.mark("built"); + if (!br || br->exitCode != 0) { + std::string who; + for (auto const* g : goals) + who += (who.empty() ? "" : ", ") + qualified(manifest_of(state, g->index)); + const auto diag = br ? std::string{} : br.error().diagnosticOutput; + const auto msg = br ? std::format("exit {}", br->exitCode) : br.error().message; + return std::unexpected(std::format("build.mcpp of {} failed to compile ({}){}\n{}", + who, msg, reproduce(), diag)); + } + return binaries; +} + +// ── Build plugins (mcpp#797, SPEC-007 P1-P5) ────────────────────────────── +// +// A unit that imports the engine interface (`mcpp.core`, or `mcpp`) is a +// build-plugin unit: the interface exists only in the build-program +// configuration. Nothing declares it; the import is the fact. + +bool imports_engine(std::string_view text) { + for (auto const& m : mcpp::pm::imported_module_names(text)) + if (m == "mcpp" || m == "mcpp.core") return true; + return false; +} + +std::vector build_plugin_units(const mcpp::manifest::Manifest& m, const fs::path& root, + const std::vector& features) { + std::set files; + for (auto const& f : mcpp::modgraph::package_source_files(root, m)) files.insert(f); + for (auto const& f : features) + if (auto it = m.buildConfig.featureSources.find(f); it != m.buildConfig.featureSources.end()) + for (auto const& g : it->second) + for (auto const& file : mcpp::modgraph::expand_glob(root, g)) files.insert(file); + std::vector units; + bool declaresEngine = false; + for (auto const& file : files) { + std::ifstream in(file, std::ios::binary); + const std::string text{std::istreambuf_iterator(in), {}}; + for (auto const& d : mcpp::pm::declared_module_roots(text)) + if (d == "mcpp" || d == "mcpp.core") declaresEngine = true; + if (imports_engine(text)) units.push_back(file); + } + // A package that is itself named `mcpp` provides that module; its units + // import their own package. + if (declaresEngine) units.clear(); + return units; +} + +} // namespace + +// P3: a package whose LIBRARY has build-plugin units, built as the root of a +// build (`mcpp build`, `mcpp test`, a selected member), is planned in the +// build-program configuration: for this machine, with the engine interface a +// dependency of it. The library's own `[dependencies]` are then build-program +// instances, which is what a plugin built on other plugins needs. +std::expected +configure_build_plugin_root(PrepareState& state, mcpp::manifest::Manifest& m, + const fs::path& root, const std::vector& features) { + if (state.overrides.build_program_graph) return {}; + if (!mcpp::manifest::has_lib_target(m)) return {}; + const auto units = build_plugin_units(m, root, features); + if (units.empty()) return {}; + const auto who = qualified(m); + if (!state.overrides.target_triple.empty()) { + const auto want = mcpp::toolchain::triple::parse(state.overrides.target_triple); + const auto host = mcpp::toolchain::triple::host_triple(); + if (!want || want->arch != host.arch || want->os != host.os) + return std::unexpected(std::format( + "'{}' is a build plugin: '{}' imports mcpp.core, which exists only in a build\n" + " program, and a build program runs on the machine doing the build. It is\n" + " built for this machine; --target {} cannot apply to it.", + who, path_text(units.front()), state.overrides.target_triple)); + } + auto engineDir = engine::ensure_package(); + if (!engineDir) return std::unexpected(engineDir.error()); + const std::string key = std::format("{}.{}", engine::kNamespace, engine::kName); + if (!m.dependencies.contains(key)) { + mcpp::manifest::DependencySpec spec; + spec.namespace_ = std::string(engine::kNamespace); + spec.shortName = std::string(engine::kName); + spec.path = path_text(*engineDir); + spec.declaredIn = "[dependencies]"; + spec.candidates = {{.namespace_ = spec.namespace_, .shortName = spec.shortName}}; + m.dependencies.emplace(key, std::move(spec)); + } + state.buildPluginRoots.insert(path_text(root)); + mcpp::log::info("build-program", std::format( + "{}: a build plugin ({} imports mcpp.core); planned in the build-program configuration", + who, path_text(units.front()))); + mcpp::ui::status("Configuration", std::format("build-program (host {}) for {}", + mcpp::toolchain::triple::host_triple().str(), who)); + return {}; +} + +// P1, P4: a build-plugin unit in a target instance. Called after the scan. +std::expected refuse_plugin_units_in_target(PrepareState& state) { + if (state.overrides.build_program_graph) return {}; + bool engineInGraph = false; + for (auto const& pkg : state.packages) + if (engine::is_engine_package(pkg.root)) engineInGraph = true; + if (engineInGraph) return {}; + for (auto const& u : state.scan.graph.units) { + bool importsEngine = false; + for (auto const& req : u.requires_) + if (req.logicalName == "mcpp" || req.logicalName == "mcpp.core") importsEngine = true; + if (!importsEngine) continue; + std::size_t p = state.packages.size(); + for (std::size_t i = 0; i < state.packages.size(); ++i) + if (qualified(state.packages[i].manifest) == u.packageName) { p = i; break; } + if (p == state.packages.size()) continue; + refusal::record(refusal::Code::HostModuleMissing); + if (p == 0 || state.packages[p].selectedMember) + return std::unexpected(std::format( + "{}: imports mcpp.core, the engine interface, which exists only in a build\n" + " program. A project's build program is its `build.mcpp`; a library that\n" + " other build programs import is a build plugin, built as such when its\n" + " library imports mcpp.core (this unit belongs to a program target).", + path_text(u.path))); + // The chain of ordinary edges that brought it into the target. + std::vector parent(state.packages.size(), state.packages.size()); + std::vector work{0}; + std::vector seen(state.packages.size(), false); + seen[0] = true; + while (!work.empty()) { + const auto c = work.back(); + work.pop_back(); + for (auto const& e : state.dependencyEdges) + if (e.consumerPackageIndex == c && !e.buildTable + && e.dependencyPackageIndex < seen.size() && !seen[e.dependencyPackageIndex]) { + seen[e.dependencyPackageIndex] = true; + parent[e.dependencyPackageIndex] = c; + work.push_back(e.dependencyPackageIndex); + } + } + std::string chain = qualified(state.packages[p].manifest); + for (auto q = parent[p]; q < state.packages.size(); q = parent[q]) + chain = qualified(manifest_of(state, q)) + " → " + chain; + const auto consumer = parent[p] < state.packages.size() + ? qualified(manifest_of(state, parent[p])) : std::string("the project"); + return std::unexpected(std::format( + "'{}' is a build plugin: {} imports mcpp.core, which exists only in a build program,\n" + " and it reaches this target through [dependencies]: {}\n" + " A build plugin serves the build program. In {}, write it in\n" + " [build-dependencies] instead of [dependencies].", + qualified(state.packages[p].manifest), path_text(u.path), chain, consumer)); + } + return {}; +} + +std::expected +ensure_build_programs(PrepareState& state, const std::vector& packages) { + std::vector wanted; + for (auto i : packages) + if (!state.buildProgramBinaries.contains(i) && program_source(state, i).exists()) + wanted.push_back(i); + if (wanted.empty()) return {}; + + auto tc = host_toolchain(state); + if (!tc) return std::unexpected(tc.error()); + auto enginePackage = engine::ensure_package(); + if (!enginePackage) return std::unexpected(enginePackage.error()); + + // One group per standard: a BMI is used only by a compile that agrees on it. + std::map> groups; + for (auto i : wanted) { + Program p; + p.index = i; + p.source = program_source(state, i); + p.member = member_name(state, i); + p.identity = program_identity(state, i, p.source, *tc); + const auto standard = standard_of(state, i); + p.group = mcpp::toolchain::hash_string( + std::format("{}|{}|{}", tc->identity, standard, mcpp::MCPP_VERSION)).substr(0, 12); + groups[standard].push_back(std::move(p)); + } + + for (auto& [standard, programs] : groups) { + const auto dir = host_root(state) / programs.front().group; + std::error_code ec; + fs::create_directories(dir, ec); + // Two configuration groups of one command may share a host toolchain + // and a standard: one builds, the other waits and finds it built. + std::optional lock; + for (int waited = 0; !(lock = mcpp::platform::fs::FileLock::try_acquire(dir)); ++waited) { + if (waited == 0) + mcpp::log::verbose("host-graph", std::format("waiting for {}", path_text(dir))); + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } + + auto record = read_record(dir / "programs.json"); + std::vector stale; + for (auto const& p : programs) { + auto it = record.find(p.member); + const bool current = it != record.end() && it->is_object() + && it->value("identity", std::string{}) == p.identity + && fs::exists(it->value("binary", std::string{}), ec); + if (current) { + state.buildProgramBinaries[p.index] = {fs::path(it->value("binary", std::string{})), + p.identity}; + mcpp::log::info("build-program", std::format( + "{}: identity {} current", qualified(manifest_of(state, p.index)), p.identity)); + continue; + } + mcpp::log::info("build-program", std::format( + "{}: identity {} -> {}, compiled", qualified(manifest_of(state, p.index)), + it != record.end() && it->is_object() ? it->value("identity", std::string("none")) + : std::string("none"), + p.identity)); + stale.push_back(&p); + } + if (stale.empty()) continue; + for (auto const* p : stale) { + warn_reserved_prefixes(state, p->index); + if (auto r = check_program_imports(state, p->index, p->source); !r) + return std::unexpected(r.error()); + } + + write_group(state, dir, programs, standard, *enginePackage); + const auto began = std::chrono::steady_clock::now(); + for (auto const* p : stale) + mcpp::build::progress::program_compiling(qualified(manifest_of(state, p->index)), + p->index == 0 || state.packages[p->index].selectedMember); + auto built = build_group(state, dir, programs, stale, *tc); + if (!built) return std::unexpected(built.error()); + const auto took = std::chrono::duration_cast( + std::chrono::steady_clock::now() - began); + for (auto const* p : stale) { + const auto bin = built->at(p->member); + state.buildProgramBinaries[p->index] = {bin, p->identity, took}; + nlohmann::json row; + row["identity"] = p->identity; + row["binary"] = path_text(bin); + row["package"] = qualified(manifest_of(state, p->index)); + row["source"] = p->source.file.empty() ? std::string("synthesised") + : path_text(p->source.file); + std::vector libs; + for (auto d : program_libraries(state, p->index)) + libs.push_back(qualified(state.packages[d].manifest)); + row["libraries"] = libs; + record[p->member] = std::move(row); + } + write_if_changed(dir / "programs.json", record.dump(2) + "\n"); + } + return {}; +} + +std::expected +build_program_binary(PrepareState& state, std::size_t i) { + if (auto it = state.buildProgramBinaries.find(i); it != state.buildProgramBinaries.end()) + return it->second; + if (auto r = ensure_build_programs(state, {i}); !r) return std::unexpected(r.error()); + if (auto it = state.buildProgramBinaries.find(i); it != state.buildProgramBinaries.end()) + return it->second; + return BuildProgramBinary{}; +} + +} // namespace mcpp::build diff --git a/src/build/prepare/local_toolchain.cpp b/src/build/prepare/local_toolchain.cpp index 9088b9b9..7b8c025b 100644 --- a/src/build/prepare/local_toolchain.cpp +++ b/src/build/prepare/local_toolchain.cpp @@ -342,15 +342,16 @@ std::expected step6_toolchain_phase(PrepareState& state) { // one program with two contracts, and one record would make each // invalidate the other on every build. bpEnv.artifactsDir = state.workRoot / "target" / ".build-mcpp" / "toolchain-phase"; - bpEnv.moduleStore = state.workRoot / "target" / ".build-mcpp" / "host-modules"; - if (state.cacheMode == CacheMode::Global) - bpEnv.moduleCacheRoot = mcpp::home::cache_root(); bpEnv.features = feature_closure(*state.m, parse_feature_request(state.overrides.features)); state.fillDepDirs(bpEnv, 0, nullptr); - bpEnv.hostModules = state.hostModulesByConsumer.count(0u) - ? state.hostModulesByConsumer.at(0u) : decltype(bpEnv.hostModules){}; - bpEnv.dormantFeatures = state.dormantFeaturesByConsumer.count(0u) - ? state.dormantFeaturesByConsumer.at(0u) : decltype(bpEnv.dormantFeatures){}; + // The program, compiled by the build programs' graph with the bootstrap + // toolchain (mcpp#797). + auto program = build_program_binary(state, 0); + if (!program) return std::unexpected(std::format("the root build program's toolchain phase: {}", + program.error())); + bpEnv.programBinary = program->path; + bpEnv.programIdentity = program->identity; + bpEnv.programCompile = program->compile; bpEnv.phase = "toolchain"; // A copy: the first pass is discarded, and what the phase states is the // whole of what is carried to the second. diff --git a/src/build/prepare/plan.cpp b/src/build/prepare/plan.cpp index 0253197f..70691c2e 100644 --- a/src/build/prepare/plan.cpp +++ b/src/build/prepare/plan.cpp @@ -1867,6 +1867,7 @@ static std::expected step13_empty_link_check(PrepareState& st std::expected phase13_finish(PrepareState& state) { BuildContext ctx; + ctx.buildPluginRoot = !state.buildPluginRoots.empty(); ctx.strict = state.overrides.strict; ctx.manifest = *state.m; ctx.tc = *state.tc; diff --git a/src/build/prepare/scan.cpp b/src/build/prepare/scan.cpp index c7e33bb9..167cd4e9 100644 --- a/src/build/prepare/scan.cpp +++ b/src/build/prepare/scan.cpp @@ -994,9 +994,52 @@ static void step11_public_module_check(PrepareState& state) { } } +// COMPAT(plugin-imports-from-build-dependencies): a build plugin's library +// imports a package it lists in its own `[build-dependencies]` with +// `host-module = true` (mcpp#797). The worklist read the entry as a dependency +// too; the use is reported here, where a unit of the library is seen to import +// a module of that package, and only then. +static void step11_compat_library_edges(PrepareState& state) { + if (state.compatLibraryEdges.empty()) return; + const auto& g = state.scan.graph; + for (auto const& ce : state.compatLibraryEdges) { + const auto consumer = ce.consumerDepIndex + 1; + if (consumer >= state.packages.size()) continue; + std::size_t dep = state.packages.size(); + for (auto const& r : state.graphRequests) + if (r.consumerPackageIndex == consumer && r.key == ce.key) { + dep = r.dependencyPackageIndex; + break; + } + if (dep >= state.packages.size()) continue; + auto qualified = [](const mcpp::manifest::Manifest& m) { + return m.package.namespace_.empty() ? m.package.name + : m.package.namespace_ + "." + m.package.name; + }; + const auto consumerName = qualified(state.packages[consumer].manifest); + const auto depName = qualified(state.packages[dep].manifest); + std::set provided; + for (auto const& u : g.units) + if (u.packageName == depName && u.provides) + provided.insert(u.provides->logicalName.substr(0, u.provides->logicalName.find(':'))); + bool imports = false; + for (auto const& u : g.units) { + if (u.packageName != consumerName) continue; + for (auto const& req : u.requires_) + if (provided.contains(req.logicalName.substr(0, req.logicalName.find(':')))) + imports = true; + } + if (imports) + state.reportCompatUse("plugin-imports-from-build-dependencies", consumer, + std::format("[build-dependencies] {}", ce.key)); + } +} + std::expected phase11_scan(PrepareState& state) { auto needsStdModule = step11_scan_sources(state); if (!needsStdModule) return std::unexpected(needsStdModule.error()); + step11_compat_library_edges(state); + if (auto r = refuse_plugin_units_in_target(state); !r) return std::unexpected(r.error()); step11_public_module_check(state); if (auto r = step11_dependency_standard_scope_check(state); !r) diff --git a/src/build/prepare/sources.cpp b/src/build/prepare/sources.cpp index db33906e..0fc58160 100644 --- a/src/build/prepare/sources.cpp +++ b/src/build/prepare/sources.cpp @@ -359,8 +359,8 @@ void fill_xpkg_env(PrepareState& state, mcpp::build::BuildProgramEnv& e, // Their own active features, not the consumer's: the consumer asked for // `features = ["rules-cuda"]` on the edge, and that is what decides which // of the rule's `[feature-xlings]` tables apply. - if (auto pit = state.hostModuleProvidersByConsumer.find(consumer); - pit != state.hostModuleProvidersByConsumer.end()) { + if (auto pit = state.buildProgramLibrariesByConsumer.find(consumer); + pit != state.buildProgramLibrariesByConsumer.end()) { for (auto q : pit->second) { if (q >= state.packages.size()) continue; auto const& pm = state.packages[q].manifest; @@ -369,8 +369,8 @@ void fill_xpkg_env(PrepareState& state, mcpp::build::BuildProgramEnv& e, declared.push_back(address); }; for (auto const& address : pm.xlings.deps) want(address); - const auto& pf = q < state.activeFeaturesByPackage.size() - ? state.activeFeaturesByPackage[q] : std::vector{}; + const auto& pf = q < state.buildProgramFeaturesByPackage.size() + ? state.buildProgramFeaturesByPackage[q] : std::vector{}; for (auto const& f : pf) if (auto it = pm.xlings.featureDeps.find(f); it != pm.xlings.featureDeps.end()) for (auto const& address : it->second) want(address); diff --git a/src/build/prepare/state.cppm b/src/build/prepare/state.cppm index bbd62bcb..0f15066a 100644 --- a/src/build/prepare/state.cppm +++ b/src/build/prepare/state.cppm @@ -243,6 +243,15 @@ constexpr std::size_t kMainConsumer = static_cast(-1); std::string url, refKind, ref; }; +// A build program as its graph compiled it (host_graph.cpp, mcpp#797): where +// the binary is, the identity it was compiled from -- which the program's run +// cache keys on -- and how long the compile took when this build made it. +struct BuildProgramBinary { + std::filesystem::path path; + std::string identity; + std::chrono::milliseconds compile{0}; +}; + // PrepareState carries prepare_build's working state across the phases it // decomposes into (see the layout comment at the top of prepare.cppm). Each // phase is an ordinary function taking `PrepareState&`; the state itself is @@ -580,9 +589,6 @@ struct PrepareState { // has none). The build program's graph activates the same set; payloads a // feature declares serve this build in either instance. std::vector> buildProgramFeaturesByPackage; - // #734 E7: per consumer, the dormant features of its host-module providers. - std::map> - dormantFeaturesByConsumer; std::map xlingsWinner; // ── Sources (mcpp#755): see sources.cpp ───────────────────────────────── @@ -628,7 +634,10 @@ struct PrepareState { std::vector dependencyEdges; std::vector graphRequests; std::map dependencyLinkForms; - std::map> hostModuleProvidersByConsumer; + // Every package of a consumer's build program's graph: the libraries the + // program uses and everything they reach (mcpp#797). Their payloads are + // what the program asks `xpkg_dir` about. + std::map> buildProgramLibrariesByConsumer; std::string runnerProvider; std::map namedRunnerProvider; std::vector rootReq; @@ -661,6 +670,17 @@ struct PrepareState { std::function(std::size_t)> bareBindingsFor; std::function compilesHere; + // The build programs this prepare compiled or found current, by package + // index (`ensure_build_programs`). + std::map buildProgramBinaries; + // COMPAT(plugin-imports-from-build-dependencies): `[build-dependencies]` + // entries a build-program instance's library also reads as dependencies, + // by the consumer's dep_manifests slot and the key it wrote. The scan + // reports those its units import. + struct CompatLibraryEdge { std::size_t consumerDepIndex; std::string key; }; + std::vector compatLibraryEdges; + // The roots (package directories) planned as build plugins (P3). + std::set buildPluginRoots; // A use of a deprecated reading (mcpp.pm.compat.registry) by the manifest // of package `i`: warned where that manifest can be edited -- the root, a // workspace member, a path package outside the package store -- and @@ -735,8 +755,6 @@ struct PrepareState { std::map resolved; std::map>> toolEnvByConsumer; - std::map> - hostModulesByConsumer; std::function foldDirectiveTailIntoPrivateBuild; @@ -915,6 +933,23 @@ provision_xlings_addresses(const mcpp::config::GlobalConfig& cfg, std::string_view label); std::string with_index_cause(std::string msg); +// ── The build programs' graph (mcpp#797), defined in host_graph.cpp ─────── +// Compile, in one build of the build programs' graph, every program of +// `packages` whose identity moved; a current one is recorded and not planned. +std::expected +ensure_build_programs(PrepareState& state, const std::vector& packages); +// The program of package `i`, compiled first when it is not current. An empty +// path: the package has no program. +std::expected +build_program_binary(PrepareState& state, std::size_t i); +// P3: plan a build plugin built as a root in the build-program configuration. +std::expected +configure_build_plugin_root(PrepareState& state, mcpp::manifest::Manifest& m, + const std::filesystem::path& root, + const std::vector& features); +// P1, P4: refuse a build-plugin unit in a target instance (after the scan). +std::expected refuse_plugin_units_in_target(PrepareState& state); + // ── Sources (mcpp#755), defined in sources.cpp ───────────────────────────── // Record (or replace) the decision for `d.subject`; a non-default source is // announced once with its `Using` line. diff --git a/src/build/prepare/target_side.cpp b/src/build/prepare/target_side.cpp index d7de9958..246bcb5d 100644 --- a/src/build/prepare/target_side.cpp +++ b/src/build/prepare/target_side.cpp @@ -1617,11 +1617,6 @@ static std::expected step9_root_build_program(PrepareState& s // helper straight into it. Same value as the default when work_dir is // unset, so an ordinary build is unchanged. bpEnv.artifactsDir = state.workRoot / "target" / ".build-mcpp"; - // What the program imports is kept once for the workspace, and in the - // global cache where it comes from the engine or the index (#748). - bpEnv.moduleStore = bpEnv.artifactsDir / "host-modules"; - if (state.cacheMode == CacheMode::Global) - bpEnv.moduleCacheRoot = mcpp::home::cache_root(); // Root mode keeps genBase empty: a relative `generated=` from the ROOT // package resolves against the package root (the documented contract), // not against OUT_DIR. @@ -1652,12 +1647,16 @@ static std::expected step9_root_build_program(PrepareState& s // #355: the host tools the ROOT package requested (consumer index 0). if (auto tit = state.toolEnvByConsumer.find(0u); tit != state.toolEnvByConsumer.end()) bpEnv.toolPaths = tit->second; - bpEnv.hostModules = state.hostModulesByConsumer.count(0u) - ? state.hostModulesByConsumer.at(0u) - : decltype(bpEnv.hostModules){}; - bpEnv.dormantFeatures = state.dormantFeaturesByConsumer.count(0u) - ? state.dormantFeaturesByConsumer.at(0u) - : decltype(bpEnv.dormantFeatures){}; + // The program, as the build programs' graph compiled it (mcpp#797). + if (auto program = build_program_binary(state, 0); program) { + bpEnv.programBinary = program->path; + bpEnv.programIdentity = program->identity; + bpEnv.programCompile = program->compile; + } else if (!state.overrides.plan_only) { + return std::unexpected(program.error()); + } else { + bpEnv.programError = program.error(); + } // #647 E1: THE RESOLVED GRAPH, FOR THE ROOT'S PROGRAM ONLY. // // Every package, dependencies before the packages that request them @@ -2044,110 +2043,27 @@ static std::expected step9_member_build_programs(PrepareState state.fillXpkgDirs(bpEnv, pkg.manifest, i); if (auto tit = state.toolEnvByConsumer.find(i); tit != state.toolEnvByConsumer.end()) bpEnv.toolPaths = tit->second; - bpEnv.hostModules = state.hostModulesByConsumer.count(i) - ? state.hostModulesByConsumer.at(i) : decltype(bpEnv.hostModules){}; - bpEnv.dormantFeatures = state.dormantFeaturesByConsumer.count(i) - ? state.dormantFeaturesByConsumer.at(i) : decltype(bpEnv.dormantFeatures){}; + // The program, as the build programs' graph compiled it (mcpp#797). + if (auto program = build_program_binary(state, i); program) { + bpEnv.programBinary = program->path; + bpEnv.programIdentity = program->identity; + bpEnv.programCompile = program->compile; + } else { + bpEnv.programError = program.error(); + } auto graph = write_graph_document(state, i, bpEnv.artifactsDir); if (!graph) return std::unexpected(graph.error()); bpEnv.graphFile = graph->first; bpEnv.graphDigest = graph->second; - // What the program imports is kept once for the workspace, so that a host - // module several members import is compiled once for all of them, and in - // the global cache where it comes from the engine or the index (#748). - bpEnv.moduleStore = state.workRoot / "target" / ".build-mcpp" / "host-modules"; - if (state.cacheMode == CacheMode::Global) - bpEnv.moduleCacheRoot = mcpp::home::cache_root(); return bpEnv; }; - // THE COMPILES COME FIRST, AT THE SAME TIME (#748, B2). What a program needs - // before it runs is a compile, and a compile depends on no other program's - // run: only the runs have an order. So every program that needs compiling is - // compiled now, up to the build's job count at once, and the programs are then - // run in the order above, each taking the compile made for it. - // - // What the compile phase leaves out is what keeps the plan the serial build's - // plan: it applies no directive, reports no outcome, states no warning and - // writes no program cache. (It writes what the compile reads: the graph - // document and a program synthesised from rules.) The runs below do the - // rest, in the same order as before, so `build.ninja` and the directives - // applied are byte for byte the serial build's. A program's environment is computed again at its turn, from - // the plan as the programs before it left it; the compile is used only when - // it was made for what that computes, and is made again otherwise. - std::vector turns; // the programs, in the order they run - for (auto const i : order) - if (hasProgram(i)) turns.push_back(i); - std::vector precompiled(turns.size()); - if (turns.size() >= 2) { - if (auto host = state.host_tc_for_build_program()) { - std::vector> envs(turns.size()); - for (std::size_t k = 0; k < turns.size(); ++k) - if (auto e = program_env(turns[k])) envs[k] = std::move(*e); - - int globalDefaultJobs = 0; - if (auto c = state.get_cfg(/*requireBootstrap=*/false)) - globalDefaultJobs = static_cast((*c)->defaultJobs); - int jobs = mcpp::build::schedule::resolve_jobs(*state.m, {}, globalDefaultJobs); - if (jobs <= 0) - jobs = mcpp::platform::capacity::recommended_jobs( - mcpp::platform::capacity::host_capacity()); - const std::size_t workers = std::min( - turns.size(), static_cast(std::max(jobs, 1))); - - std::atomic next{0}; - // The first program, in the order they run, whose compile failed. A - // program after it is never reached by a build that stops at the - // first failure, so its compile is not started. - std::atomic firstFailure{turns.size()}; - auto work = [&] { - for (;;) { - const auto k = next.fetch_add(1); - if (k >= turns.size()) return; - if (k > firstFailure.load() || !envs[k]) continue; - auto& pkg = state.packages[turns[k]]; - try { - precompiled[k] = mcpp::build::precompile_build_program( - pkg.manifest, pkg.root, host->first, host->second, - pkg.manifest.cppStandard, *envs[k]); - } catch (const std::exception& ex) { - // A compile that threw is a compile that failed. Its - // empty stamp matches nothing, so the program's turn - // compiles it again, alone, and reports what that - // compile says. - precompiled[k] = {}; - precompiled[k].compiled = true; - precompiled[k].stamp = {}; - precompiled[k].error = std::format( - "build.mcpp failed to compile: {}", ex.what()); - } - if (precompiled[k].compiled && !precompiled[k].error.empty()) { - auto seen = firstFailure.load(); - while (k < seen && !firstFailure.compare_exchange_weak(seen, k)) {} - } - } - }; - const auto phaseBegan = std::chrono::steady_clock::now(); - std::vector pool; - for (std::size_t w = 1; w < workers; ++w) { - // A thread the system cannot start is one fewer worker, and the - // compiles are shared by the ones that did. - try { pool.emplace_back(work); } catch (const std::system_error&) { break; } - } - work(); - for (auto& t : pool) t.join(); - // The compiles overlapped, so the time they took is the phase's wall - // time, stated once; each program's line still shows its own. - mcpp::build::progress::programs_compiled( - std::chrono::duration_cast( - std::chrono::steady_clock::now() - phaseBegan)); - } - } - - std::size_t turn = 0; + // THE COMPILES WERE MADE TOGETHER: every program this plan runs is compiled + // in one build of the build programs' graph, before the dependencies' + // programs run (features.cpp, mcpp#797). The programs run here, one at a + // time, in the order above. for (auto const i : order) { if (!hasProgram(i)) continue; - const auto& made = precompiled[turn++]; auto& pkg = state.packages[i]; auto host = state.host_tc_for_build_program(); if (!host) return std::unexpected(host.error()); @@ -2165,7 +2081,7 @@ static std::expected step9_member_build_programs(PrepareState pkg.manifest.package.name, [&] { return mcpp::build::run_build_program( pkg.manifest, pkg.root, host->first, host->second, - pkg.manifest.cppStandard, bpEnv, made.compiled ? &made : nullptr); + pkg.manifest.cppStandard, bpEnv); }); if (!bp) { if (!state.overrides.plan_only) diff --git a/src/build/prepare/toolchain.cpp b/src/build/prepare/toolchain.cpp index ec270835..062b32ad 100644 --- a/src/build/prepare/toolchain.cpp +++ b/src/build/prepare/toolchain.cpp @@ -8,6 +8,7 @@ import :state; import mcpp.build.prepare_inputs; import std; +import mcpp.build.engine_package; import mcpp.diag; import mcpp.build.refusal; import mcpp.build.version_floor; @@ -297,6 +298,9 @@ static std::expected step1_define_early_toolchain_closures(Pr if (auto c = state.get_cfg(true)) roots.push_back((*c)->xlingsHome() / "data" / "xpkgs"); for (auto& d : mcpp::config::project_xlings_data_roots(state.workRoot)) roots.push_back(d / "xpkgs"); + // The engine interface's package: written once per mcpp version and + // never edited, like a payload (mcpp#797). + roots.push_back(mcpp::build::engine::root()); return roots; }(); @@ -1217,6 +1221,14 @@ static std::expected step1_device_axis_and_layer_merge(Prepar } std::expected phase1_toolchain_spec_and_axes(PrepareState& state) { + // P3 (mcpp#797): a build plugin built as the root is planned for this + // machine with the engine interface, decided before the toolchain is. + if (!state.m->package.virtualRoot) + if (auto r = configure_build_plugin_root( + state, *state.m, *state.root, + feature_closure(*state.m, parse_feature_request(state.overrides.features))); + !r) + return std::unexpected(r.error()); if (auto r = step1_define_early_toolchain_closures(state); !r) return std::unexpected(r.error()); if (auto r = step1_target_and_static_overrides(state); !r) return std::unexpected(r.error()); if (auto r = step1_device_axis_and_layer_merge(state); !r) return std::unexpected(r.error()); diff --git a/src/build/source_digest.cppm b/src/build/source_digest.cppm new file mode 100644 index 00000000..988d6403 --- /dev/null +++ b/src/build/source_digest.cppm @@ -0,0 +1,126 @@ +// mcpp.build.source_digest — the content digest of a package's source tree, +// recomputed only where a file's size or modification time moved (mcpp#797). +// +// A build program depends on every source of the packages in its graph, +// headers a module unit includes among them, and for a `path` package those +// can change without its name or version changing. Its identity therefore +// holds the digest of each such tree (R6: hashing only the interface file left +// an edited header unseen, and the program was neither rebuilt nor rerun). +// +// Hashing every file on every build is what checking one host module entry +// used to cost (52 ms for a one-file package, measured). The digest is a +// function of the contents, and a file whose `(size, mtime)` is what the +// previous walk recorded is taken to have the contents recorded with it -- +// ninja's own assumption. A file whose stat moved is read again, so a touched +// file whose bytes did not change leaves the digest as it was. +// +// WHAT IS WALKED. Every regular file below the root, except `target`, `.git` +// and `.mcpp`, and except a subdirectory holding its own `mcpp.toml`: that is +// another package (a workspace member), with its own digest where something +// depends on it -- the tool store's rule (SPEC-004 §10.1). + +export module mcpp.build.source_digest; + +import std; +import mcpp.toolchain.fingerprint; // hash_file / hash_string + +export namespace mcpp::build { + +// The digest of the tree below `root`. `statCache` is where the previous walk's +// `(path, size, mtime, hash)` rows are kept; it is read when present and +// rewritten when anything moved. An unreadable cache is a cold walk. +std::string source_tree_digest(const std::filesystem::path& root, + const std::filesystem::path& statCache); + +} // namespace mcpp::build + +namespace mcpp::build { + +namespace fs = std::filesystem; + +namespace { + +struct Row { + std::uintmax_t size = 0; + long long mtime = 0; + std::string hash; +}; + +std::string utf8(const fs::path& p) { + const auto u = p.generic_u8string(); + return std::string(reinterpret_cast(u.data()), u.size()); +} + +std::map read_cache(const fs::path& file) { + std::map rows; + std::ifstream in(file, std::ios::binary); + std::string line; + if (!std::getline(in, line) || line != "mcpp-source-digest-v1") return rows; + while (std::getline(in, line)) { + // path \t size \t mtime \t hash + const auto a = line.find('\t'); + const auto b = a == std::string::npos ? a : line.find('\t', a + 1); + const auto c = b == std::string::npos ? b : line.find('\t', b + 1); + if (c == std::string::npos) continue; + Row r; + std::from_chars(line.data() + a + 1, line.data() + b, r.size); + std::from_chars(line.data() + b + 1, line.data() + c, r.mtime); + r.hash = line.substr(c + 1); + rows.emplace(line.substr(0, a), std::move(r)); + } + return rows; +} + +} // namespace + +std::string source_tree_digest(const fs::path& root, const fs::path& statCache) { + const auto previous = read_cache(statCache); + std::map now; + bool moved = false; + std::error_code ec; + fs::recursive_directory_iterator it(root, fs::directory_options::skip_permission_denied, ec); + for (; !ec && it != fs::recursive_directory_iterator(); it.increment(ec)) { + std::error_code tec; + if (it->is_directory(tec)) { + const auto name = utf8(it->path().filename()); + if (name == "target" || name == ".git" || name == ".mcpp" + || fs::exists(it->path() / "mcpp.toml", tec)) + it.disable_recursion_pending(); + continue; + } + if (!it->is_regular_file(tec)) continue; + const auto rel = utf8(it->path().lexically_relative(root)); + Row r; + r.size = it->file_size(tec); + r.mtime = static_cast( + it->last_write_time(tec).time_since_epoch().count()); + if (auto p = previous.find(rel); + p != previous.end() && p->second.size == r.size && p->second.mtime == r.mtime) { + r.hash = p->second.hash; + } else { + r.hash = mcpp::toolchain::hash_file(it->path()); + moved = true; + } + now.emplace(rel, std::move(r)); + } + if (now.size() != previous.size()) moved = true; + + std::string text; + for (auto const& [rel, r] : now) text += std::format("{}\t{}\n", rel, r.hash); + const auto digest = mcpp::toolchain::hash_string(text); + + if (moved && !statCache.empty()) { + fs::create_directories(statCache.parent_path(), ec); + const auto tmp = statCache.string() + ".tmp"; + { + std::ofstream out(tmp, std::ios::binary | std::ios::trunc); + out << "mcpp-source-digest-v1\n"; + for (auto const& [rel, r] : now) + out << std::format("{}\t{}\t{}\t{}\n", rel, r.size, r.mtime, r.hash); + } + fs::rename(tmp, statCache, ec); + } + return digest; +} + +} // namespace mcpp::build diff --git a/src/pack/library_pipeline.cppm b/src/pack/library_pipeline.cppm index d12b16cb..0f1ae3e0 100644 --- a/src/pack/library_pipeline.cppm +++ b/src/pack/library_pipeline.cppm @@ -163,6 +163,15 @@ export int build_and_pack_library(const std::string& targetName, ov.features = opts.features; auto ctx = mcpp::build::prepare_build(false, /*includeDevDeps=*/false, {}, ov); if (!ctx) { mcpp::ui::error(ctx.error()); return 2; } + // P5 (mcpp#797): a build plugin is distributed as source; its BMIs + // are the build program's toolchain's and travel nowhere. + if (ctx->buildPluginRoot) { + mcpp::ui::error(std::format( + "'{}' is a build plugin (its library imports mcpp.core) and is distributed as " + "source: build programs compile it with their own toolchain. Publish it " + "(`mcpp publish`) instead of packing it.", ctx->manifest.package.name)); + return 2; + } // ── the target, and what its kind means here ────────────────── // In a workspace plan the package is its one selected member diff --git a/tests/unit/test_host_module_store.cpp b/tests/unit/test_host_module_store.cpp deleted file mode 100644 index e0d7996c..00000000 --- a/tests/unit/test_host_module_store.cpp +++ /dev/null @@ -1,348 +0,0 @@ -#include - -import std; -import mcpp.bmi_cache; -import mcpp.build.host_module_store; -import mcpp.libs.json; - -// The store a build program's compiled imports live in (#748, B1): what an entry -// is addressed by, where it lives by provenance, what a hit compares, and that a -// key is compiled once however many threads ask for it. - -namespace hm = mcpp::build::hostmods; -namespace fs = std::filesystem; - -namespace { - -struct Tmp { - fs::path path; - Tmp() { - path = fs::temp_directory_path() - / std::format("mcpp_host_module_store_test_{}", std::random_device{}()); - fs::create_directories(path); - } - ~Tmp() { std::error_code ec; fs::remove_all(path, ec); } -}; - -nlohmann::json inputs(std::string_view stdFlag = "-std=c++23", std::string_view digest = "abc") { - nlohmann::json j; - j["epoch"] = 1; - j["role"] = "host-module"; - j["module"] = "acme.rules"; - j["std_flag"] = std::string(stdFlag); - j["interface_sha256"] = std::string(digest); - j["toolchain"] = {{"compiler", "gcc"}, {"compiler_version", "16.1.0"}}; - return j; -} - -hm::Files files() { return {{"acme.rules.gcm"}, {"acme.rules.o"}}; } - -// What a producer writes: the two files an entry is listed with. -hm::Producer writing(std::atomic& calls, std::string_view body = "x") { - return [&calls, body = std::string(body)](const fs::path& scratch) -> std::expected { - ++calls; - std::ofstream(scratch / "bmi" / "acme.rules.gcm") << body; - std::ofstream(scratch / "obj" / "acme.rules.o") << body; - return {}; - }; -} - -hm::Home workspaceHome(const fs::path& root) { - hm::Home h; - h.kind = hm::Home::Kind::Workspace; - h.workspaceStore = root / "target" / ".build-mcpp" / "host-modules"; - return h; -} - -hm::Home globalHome(const fs::path& root) { - hm::Home h; - h.kind = hm::Home::Kind::Global; - h.cacheRoot = root / "cache"; - h.index = "mcpplibs"; - h.package = "acme.rules"; - h.version = "1.2.3"; - return h; -} - -} // namespace - -// Known answers for SHA-256 (FIPS 180-2, appendix B and the empty message). -TEST(HostModuleStoreSha256, KnownAnswers) { - EXPECT_EQ(hm::sha256_hex(""), - "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"); - EXPECT_EQ(hm::sha256_hex("abc"), - "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"); - EXPECT_EQ(hm::sha256_hex("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq"), - "248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1"); - // A message that spans several blocks. - EXPECT_EQ(hm::sha256_hex(std::string(1000, 'a')), - "41edece42d63e8d9bf515a9ba6932e1c20cbc9f5a5d134645adb5db1b9737ea3"); -} - -TEST(HostModuleStoreSha256, AFileHasTheDigestOfItsBytes) { - Tmp t; - const auto p = t.path / "f.cppm"; - std::ofstream(p, std::ios::binary) << "abc"; - EXPECT_EQ(hm::sha256_file(p), hm::sha256_hex("abc")); - EXPECT_EQ(hm::sha256_file(t.path / "absent"), ""); -} - -TEST(HostModuleStoreTree, TheDigestFollowsNamesAndContentsAndSkipsTarget) { - Tmp t; - fs::create_directories(t.path / "src"); - std::ofstream(t.path / "src" / "a.cppm") << "a"; - std::ofstream(t.path / "src" / "a.h") << "h"; - const auto first = hm::tree_digest(t.path); - EXPECT_TRUE(first.complete); - EXPECT_EQ(first.hex, hm::tree_digest(t.path).hex); - - // An included header edited: the digest moves although the interface did not. - std::ofstream(t.path / "src" / "a.h") << "changed"; - const auto edited = hm::tree_digest(t.path); - EXPECT_NE(first.hex, edited.hex); - - // What the build wrote below `target` is not the package's text. - fs::create_directories(t.path / "target" / "x"); - std::ofstream(t.path / "target" / "x" / "out.o") << "bytes"; - EXPECT_EQ(edited.hex, hm::tree_digest(t.path).hex); - - // A file renamed, its bytes kept: another tree. - fs::rename(t.path / "src" / "a.h", t.path / "src" / "b.h"); - EXPECT_NE(edited.hex, hm::tree_digest(t.path).hex); -} - -TEST(HostModuleStore, TheKeyIsAFunctionOfTheInputs) { - EXPECT_EQ(hm::key_of(inputs()), hm::key_of(inputs())); - EXPECT_NE(hm::key_of(inputs("-std=c++23")), hm::key_of(inputs("-std=c++20"))); - EXPECT_NE(hm::key_of(inputs("-std=c++23", "abc")), hm::key_of(inputs("-std=c++23", "abd"))); - EXPECT_EQ(hm::key_of(inputs()).size(), 16u); -} - -TEST(HostModuleStore, AnEntryOfAWorkspaceLivesInTheWorkspacesStore) { - Tmp t; - std::atomic calls{0}; - auto e = hm::obtain(workspaceHome(t.path), inputs(), files(), writing(calls)); - ASSERT_TRUE(e) << e.error(); - EXPECT_FALSE(e->reused); - EXPECT_FALSE(e->global); - EXPECT_EQ(e->dir, t.path / "target" / ".build-mcpp" / "host-modules" / hm::key_of(inputs())); - EXPECT_TRUE(fs::exists(e->bmi("acme.rules.gcm"))); - EXPECT_TRUE(fs::exists(e->obj("acme.rules.o"))); - EXPECT_TRUE(fs::exists(e->dir / "entry.json")); - // Nothing is left beside it: the staging directory went with the rename. - const auto staging = t.path / "target" / ".build-mcpp" / "host-modules" / ".tmp"; - EXPECT_TRUE(!fs::exists(staging) || fs::is_empty(staging)); -} - -TEST(HostModuleStore, AnEntryOfTheEngineOrTheIndexLivesBelowThePackageAddress) { - Tmp t; - std::atomic calls{0}; - auto e = hm::obtain(globalHome(t.path), inputs(), files(), writing(calls)); - ASSERT_TRUE(e) << e.error(); - EXPECT_TRUE(e->global); - // The layout of a dependency's entry, which is what lets `mcpp cache gc`, - // `list` and `verify` walk it. - EXPECT_EQ(e->dir, t.path / "cache" / "pkg" / "mcpplibs" / "acme.rules@1.2.3" / hm::key_of(inputs())); - // Nothing of a workspace entry is written to the cache root, and the other - // way round. - Tmp u; - auto w = hm::obtain(workspaceHome(u.path), inputs(), files(), writing(calls)); - ASSERT_TRUE(w) << w.error(); - EXPECT_FALSE(fs::exists(u.path / "cache")); -} - -TEST(HostModuleStore, TheEntryRecordsItsInputsAndItsFiles) { - Tmp t; - std::atomic calls{0}; - auto e = hm::obtain(workspaceHome(t.path), inputs(), files(), writing(calls)); - ASSERT_TRUE(e) << e.error(); - std::ifstream is(e->dir / "entry.json"); - nlohmann::json j; - is >> j; - EXPECT_EQ(j["key"], hm::key_of(inputs())); - EXPECT_EQ(j["inputs"], inputs()); - EXPECT_EQ(j["bmi"], nlohmann::json::array({"acme.rules.gcm"})); - EXPECT_EQ(j["obj"], nlohmann::json::array({"acme.rules.o"})); -} - -TEST(HostModuleStore, AnEntryFoundIsNotProducedAgain) { - Tmp t; - std::atomic calls{0}; - auto first = hm::obtain(workspaceHome(t.path), inputs(), files(), writing(calls)); - ASSERT_TRUE(first) << first.error(); - auto second = hm::obtain(workspaceHome(t.path), inputs(), files(), writing(calls)); - ASSERT_TRUE(second) << second.error(); - EXPECT_EQ(calls.load(), 1); - EXPECT_FALSE(first->reused); - EXPECT_TRUE(second->reused); - EXPECT_EQ(first->dir, second->dir); -} - -TEST(HostModuleStore, DifferentInputsAreDifferentEntries) { - Tmp t; - std::atomic calls{0}; - auto a = hm::obtain(workspaceHome(t.path), inputs("-std=c++23"), files(), writing(calls)); - auto b = hm::obtain(workspaceHome(t.path), inputs("-std=c++20"), files(), writing(calls)); - ASSERT_TRUE(a) << a.error(); - ASSERT_TRUE(b) << b.error(); - EXPECT_EQ(calls.load(), 2); - EXPECT_NE(a->dir, b->dir); -} - -// A hit compares the recorded inputs, not the address: an entry whose recorded -// inputs disagree is a miss, and is replaced. -TEST(HostModuleStore, AnEntryWhoseRecordedInputsDisagreeIsReplaced) { - Tmp t; - std::atomic calls{0}; - auto first = hm::obtain(workspaceHome(t.path), inputs(), files(), writing(calls, "old")); - ASSERT_TRUE(first) << first.error(); - - nlohmann::json j; - { std::ifstream is(first->dir / "entry.json"); is >> j; } - j["inputs"]["std_flag"] = "-std=c++98"; - { std::ofstream os(first->dir / "entry.json"); os << j.dump(2); } - - auto again = hm::obtain(workspaceHome(t.path), inputs(), files(), writing(calls, "new")); - ASSERT_TRUE(again) << again.error(); - EXPECT_EQ(calls.load(), 2); - EXPECT_FALSE(again->reused); - std::ifstream is(again->bmi("acme.rules.gcm")); - std::string body; - std::getline(is, body); - EXPECT_EQ(body, "new"); - std::ifstream es(again->dir / "entry.json"); - nlohmann::json after; - es >> after; - EXPECT_EQ(after["inputs"]["std_flag"], "-std=c++23"); -} - -// An entry missing a file it lists is a miss too: nothing half-there is served. -TEST(HostModuleStore, AnEntryMissingAFileIsAMiss) { - Tmp t; - std::atomic calls{0}; - auto first = hm::obtain(workspaceHome(t.path), inputs(), files(), writing(calls)); - ASSERT_TRUE(first) << first.error(); - fs::remove(first->obj("acme.rules.o")); - auto again = hm::obtain(workspaceHome(t.path), inputs(), files(), writing(calls)); - ASSERT_TRUE(again) << again.error(); - EXPECT_EQ(calls.load(), 2); - EXPECT_TRUE(fs::exists(again->obj("acme.rules.o"))); -} - -TEST(HostModuleStore, AProducerThatFailsLeavesNoEntryAndNoStaging) { - Tmp t; - auto e = hm::obtain(workspaceHome(t.path), inputs(), files(), - [](const fs::path&) -> std::expected { - return std::unexpected(std::string("the compiler said no")); - }); - ASSERT_FALSE(e); - EXPECT_EQ(e.error(), "the compiler said no"); - const auto store = t.path / "target" / ".build-mcpp" / "host-modules"; - EXPECT_FALSE(fs::exists(store / hm::key_of(inputs()))); - const auto staging = store / ".tmp"; - EXPECT_TRUE(!fs::exists(staging) || fs::is_empty(staging)); -} - -TEST(HostModuleStore, AProducerThatWritesLessThanItListedIsRefused) { - Tmp t; - auto e = hm::obtain(workspaceHome(t.path), inputs(), files(), - [](const fs::path& scratch) -> std::expected { - std::ofstream(scratch / "bmi" / "acme.rules.gcm") << "x"; // no object - return {}; - }); - ASSERT_FALSE(e); - EXPECT_NE(e.error().find("acme.rules.o"), std::string::npos) << e.error(); - EXPECT_FALSE(fs::exists(t.path / "target" / ".build-mcpp" / "host-modules" / hm::key_of(inputs()))); -} - -// B2: the programs of a workspace are compiled at the same time, and a host -// module they share is compiled once. Many threads ask for one key; one of them -// produces it, and every thread is handed the same entry. -TEST(HostModuleStore, ManyThreadsAskingForOneKeyProduceItOnce) { - Tmp t; - std::atomic calls{0}; - constexpr int kThreads = 16; - std::vector> results(kThreads, std::unexpected(std::string("unset"))); - std::atomic gate{false}; - std::vector threads; - for (int i = 0; i < kThreads; ++i) - threads.emplace_back([&, i] { - while (!gate.load()) std::this_thread::yield(); - results[i] = hm::obtain(workspaceHome(t.path), inputs(), files(), - [&](const fs::path& scratch) -> std::expected { - ++calls; - // Long enough for the other threads to arrive at the lock. - std::this_thread::sleep_for(std::chrono::milliseconds(50)); - std::ofstream(scratch / "bmi" / "acme.rules.gcm") << "x"; - std::ofstream(scratch / "obj" / "acme.rules.o") << "x"; - return {}; - }); - }); - gate.store(true); - for (auto& th : threads) th.join(); - - EXPECT_EQ(calls.load(), 1); - int produced = 0; - for (auto& r : results) { - ASSERT_TRUE(r) << r.error(); - EXPECT_EQ(r->dir, results[0]->dir); - if (!r->reused) ++produced; - } - EXPECT_EQ(produced, 1); -} - -TEST(HostModuleStore, ThreadsAskingForDifferentKeysProduceTheirOwn) { - Tmp t; - std::atomic calls{0}; - constexpr int kThreads = 6; - std::vector threads; - std::vector> results(kThreads, std::unexpected(std::string("unset"))); - for (int i = 0; i < kThreads; ++i) - threads.emplace_back([&, i] { - results[i] = hm::obtain(workspaceHome(t.path), inputs("-std=c++23", std::format("d{}", i)), - files(), writing(calls)); - }); - for (auto& th : threads) th.join(); - EXPECT_EQ(calls.load(), kThreads); - std::set dirs; - for (auto& r : results) { ASSERT_TRUE(r) << r.error(); dirs.insert(r->dir); } - EXPECT_EQ(dirs.size(), static_cast(kThreads)); -} - -// A second producer that finishes after the first one published finds the entry -// in place and discards its own: the staged directory is removed. -TEST(HostModuleStore, APublishThatFindsTheEntryInPlaceKeepsTheFirst) { - Tmp t; - mcpp::bmi_cache::CacheKey key; - key.cacheRoot = t.path / "store"; - key.directDir = t.path / "store" / "abcd"; - key.keyHex = "abcd"; - key.indexName = "workspace"; - key.packageName = "host-modules"; - key.inputs = inputs(); - mcpp::bmi_cache::DepArtifacts arts; - arts.bmiFiles = {"m.gcm"}; - arts.objFiles = {{"m.o", {}}}; - - auto stage = [&](std::string_view body) { - auto s = mcpp::bmi_cache::stage_entry(key); - EXPECT_TRUE(s); - std::ofstream(*s / "bmi" / "m.gcm") << body; - std::ofstream(*s / "obj" / "m.o") << body; - return *s; - }; - const auto a = stage("first"); - const auto b = stage("second"); - EXPECT_NE(a, b); - auto pa = mcpp::bmi_cache::publish_staged(key, a, arts); - ASSERT_TRUE(pa); - EXPECT_TRUE(*pa); - auto pb = mcpp::bmi_cache::publish_staged(key, b, arts); - ASSERT_TRUE(pb); - EXPECT_FALSE(*pb); - EXPECT_FALSE(fs::exists(b)); - std::ifstream is(key.directDir / "bmi" / "m.gcm"); - std::string body; - std::getline(is, body); - EXPECT_EQ(body, "first"); -} From 01a94c74a3cf61dacb6c43ee9c07d97635d3c366 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sun, 11 Oct 2026 06:39:32 +0900 Subject: [PATCH 04/12] Host graph: module collision and import checks before the compile, dependency cache, quiet sub-build; tests follow the build programs' graph (#797) --- .edited | 0 src/build/prepare/graph.cpp | 11 ++- src/build/prepare/host_graph.cpp | 54 +++++++++++- tests/e2e/189_host_module_rules.sh | 13 +-- tests/e2e/309_host_module_identity.sh | 32 +++---- .../610_feature_controlled_rule_collection.sh | 11 ++- ...624_a_missing_host_module_names_the_key.sh | 4 +- ...rule_applies_through_a_source_it_claims.sh | 2 +- ...several_members_import_is_compiled_once.sh | 87 +++++++++---------- ..._entry_is_reused_while_its_inputs_agree.sh | 79 ++++++++--------- 10 files changed, 168 insertions(+), 125 deletions(-) create mode 100644 .edited diff --git a/.edited b/.edited new file mode 100644 index 00000000..e69de29b diff --git a/src/build/prepare/graph.cpp b/src/build/prepare/graph.cpp index 7ecadf17..4cbfca28 100644 --- a/src/build/prepare/graph.cpp +++ b/src/build/prepare/graph.cpp @@ -282,7 +282,10 @@ static void recordDependencyEdge( // program and not the target before the section decided the // configuration (mcpp#797). It keeps that meaning, reported where the // manifest can be edited. - if (!buildTable && ((spec.hostModuleDeclared && spec.hostModule) || ruleFeature)) { + // Not in the build programs' graph: every package there is a build-program + // instance already, and a library's rule features are what its edge asks. + if (!buildTable && !state.overrides.build_program_graph + && ((spec.hostModuleDeclared && spec.hostModule) || ruleFeature)) { buildTable = true; state.reportCompatUse("host-module-in-dependencies", consumerPackageIndex, std::format("[dependencies] {}", writtenKey)); @@ -1974,7 +1977,11 @@ step4b_finalize_dependency(PrepareState& state, WorklistItemCtx& ctx) { && child_spec.hostModule && !state.dep_manifests.back()->dependencies.contains(child_name)) { state.compatLibraryEdges.push_back({selfIdx, child_name}); - state.worklist.push_back({child_name, childReq, + // A library dependency now: the request is what the edge is. + auto asDependency = childReq; + asDependency.hostModule = false; + asDependency.hostModuleDeclared = false; + state.worklist.push_back({child_name, asDependency, thisDepLabel + " (build-dep, read as a dependency)", childReq.version, selfIdx, ctx.dep_root, item.devOnly, item.buildOnly}); diff --git a/src/build/prepare/host_graph.cpp b/src/build/prepare/host_graph.cpp index 2df83e4c..78737aff 100644 --- a/src/build/prepare/host_graph.cpp +++ b/src/build/prepare/host_graph.cpp @@ -399,6 +399,47 @@ std::expected check_program_imports(PrepareState& state, std: return {}; } +// ONE MODULE NAME, ONE PROVIDER, IN ONE PROGRAM. Two packages of a program's +// graph that declare one module name would both be compiled and linked, and +// the failure would be a duplicate module initializer naming one object twice. +// The two package identities and the two interface paths are the content of +// the refusal: with one name shared they are the only way to tell them apart. +std::expected check_module_collisions(const PrepareState& state, std::size_t i) { + const auto libs = program_libraries(state, i); + std::map> seen; + for (auto d : closure_of(state, libs)) { + if (engine::is_engine_package(state.packages[d].root)) continue; + auto const& pkg = state.packages[d]; + std::set files; + for (auto const& f : mcpp::modgraph::package_source_files(pkg.root, pkg.manifest)) + files.insert(f); + for (auto const& f : program_features(state, d)) + if (auto it = pkg.manifest.buildConfig.featureSources.find(f); + it != pkg.manifest.buildConfig.featureSources.end()) + for (auto const& g : it->second) + for (auto const& file : mcpp::modgraph::expand_glob(pkg.root, g)) + files.insert(file); + for (auto const& file : files) { + std::ifstream in(file, std::ios::binary); + const std::string text{std::istreambuf_iterator(in), {}}; + for (auto const& m : mcpp::pm::declared_module_roots(text)) { + auto [it, fresh] = seen.try_emplace(m, d, file); + if (fresh || it->second.first == d) continue; + return std::unexpected(std::format( + "two libraries of the build program of '{}' declare the module '{}':\n" + " {} ({})\n" + " {} ({})\n" + " A module name identifies one module within a program. Depend on one " + "of them, or rename the module in one.", + qualified(manifest_of(state, i)), m, + qualified(state.packages[it->second.first].manifest), + path_text(it->second.second), qualified(pkg.manifest), path_text(file))); + } + } + } + return {}; +} + // SPEC-007 R9.6: a module whose name opens with `mcpp.` comes only // from a package of namespace `mcpp`. Warned once per (package, module). void warn_reserved_prefixes(const PrepareState& state, std::size_t i) { @@ -480,7 +521,9 @@ void write_group(PrepareState& state, const fs::path& dir, const std::vector bopt.verbose = true; auto br = be->build(ctx->plan, bopt); timer.mark("built"); + // What an index package and the engine interface compiled to is kept in + // the global dependency cache, as for any build (`--cache` decides). + if (br && br->exitCode == 0) mcpp::build::populate_dependency_cache(*ctx); if (!br || br->exitCode != 0) { std::string who; for (auto const* g : goals) @@ -803,11 +849,11 @@ ensure_build_programs(PrepareState& state, const std::vector& packa if (current) { state.buildProgramBinaries[p.index] = {fs::path(it->value("binary", std::string{})), p.identity}; - mcpp::log::info("build-program", std::format( + mcpp::log::verbose("build-program", std::format( "{}: identity {} current", qualified(manifest_of(state, p.index)), p.identity)); continue; } - mcpp::log::info("build-program", std::format( + mcpp::log::verbose("build-program", std::format( "{}: identity {} -> {}, compiled", qualified(manifest_of(state, p.index)), it != record.end() && it->is_object() ? it->value("identity", std::string("none")) : std::string("none"), @@ -817,6 +863,8 @@ ensure_build_programs(PrepareState& state, const std::vector& packa if (stale.empty()) continue; for (auto const* p : stale) { warn_reserved_prefixes(state, p->index); + if (auto r = check_module_collisions(state, p->index); !r) + return std::unexpected(r.error()); if (auto r = check_program_imports(state, p->index, p->source); !r) return std::unexpected(r.error()); } diff --git a/tests/e2e/189_host_module_rules.sh b/tests/e2e/189_host_module_rules.sh index ebe943d7..80e0b0c2 100755 --- a/tests/e2e/189_host_module_rules.sh +++ b/tests/e2e/189_host_module_rules.sh @@ -140,14 +140,15 @@ cd "$TMP" rm -f rules/src/rules.cppm mv rules/src/rules.cppm.bak rules/src/rules.cppm -# A missing lib root must say so, not fail three edges later. +# The interface root is not what makes a library usable by a build program +# (mcpp#797): the module is found by the name its unit declares, wherever the +# file is. A unit renamed away from the conventional lib root still serves. cd "$TMP" mv rules/src/rules.cppm rules/src/elsewhere.cppm cd app && rm -rf target -if "$MCPP" build > b3.log 2>&1; then - cat b3.log; echo "FAIL: a host module with no interface unit was accepted"; exit 1 -fi -grep -q "no interface unit" b3.log || { - cat b3.log; echo "FAIL: unhelpful diagnostic for a missing rule interface"; exit 1; } +"$MCPP" build > b3.log 2>&1 || { + cat b3.log; echo "FAIL: a library whose interface is not at the conventional lib root was refused"; exit 1; } +out="$("$MCPP" run 2>&1 | grep '^ANSWER=' | tail -1)" +[[ "$out" == "ANSWER=43" ]] || { cat b3.log; echo "FAIL: the renamed interface's directive did not land: $out"; exit 1; } echo "OK" diff --git a/tests/e2e/309_host_module_identity.sh b/tests/e2e/309_host_module_identity.sh index e52a1d7e..44ca9246 100755 --- a/tests/e2e/309_host_module_identity.sh +++ b/tests/e2e/309_host_module_identity.sh @@ -80,30 +80,22 @@ out="$("$MCPP" run 2>&1 | grep '^DIVERGENT=' | tail -1)" [[ "$out" == "DIVERGENT=1" ]] || { echo "FAIL: the divergently-named rule's directive did not land: $out"; exit 1; } -# THE IMPORT ABOVE DOES NOT DISCRIMINATE ON GCC, AND THAT IS THE WHOLE -# POINT OF THIS FEATURE. Measured: with the pre-I1 code — which registered the -# PACKAGE name — this build still succeeded under GCC, because GCC's BMIs are -# implicit under gcm.cache and keyed by the name the SOURCE declares, so -# nothing consulted the registered name at all. Under Clang and MSVC the same -# fixture fails, since both are handed an explicit `=` mapping. +# THE IMPORT ABOVE DOES NOT DISCRIMINATE ON GCC: GCC's BMIs are implicit under +# gcm.cache and keyed by the name the SOURCE declares, so a build that +# registered the PACKAGE name succeeded there and failed under Clang and MSVC, +# which are handed an explicit `=` mapping. # -# So the criterion below is the one that works everywhere: `build_host_module` -# derives the BMI and OBJECT filenames from the REGISTERED name. Under the old -# rule the object was `protobufgen.o`; under this one it is named for what the -# source declares. A file listing is platform-independent evidence about which -# name the engine used, where the successful import is not. -# -# A host module is kept as an entry of the workspace's store, keyed by what it -# was compiled from (#748), and the entry names its files as before: the BMI -# under `bmi/` and the object under `obj/`. -BM=target/.build-mcpp/host-modules +# The build programs' graph (mcpp#797) compiles the library as a package of +# its plan, and the plan names a BMI for the module the source declares. The +# BMI is the platform-independent evidence of which name the engine used. +BM=target/.build-mcpp/host [ -n "$(find "$BM" -name 'acme.rules.protobuf.*' 2>/dev/null)" ] || { echo "FAIL: the host module was not named for its DECLARED module name" - find "$BM" 2>/dev/null + find "$BM" 2>/dev/null | head -40 exit 1; } -if [ -n "$(find "$BM" -name 'protobufgen.*' 2>/dev/null)" ]; then - find "$BM" - echo "FAIL: the host module is still named for the PACKAGE name" +if [ -n "$(find "$BM" \( -name 'protobufgen.gcm' -o -name 'protobufgen.pcm' -o -name 'protobufgen.ifc' \) 2>/dev/null)" ]; then + find "$BM" | head -40 + echo "FAIL: a BMI is named for the PACKAGE name" exit 1 fi diff --git a/tests/e2e/610_feature_controlled_rule_collection.sh b/tests/e2e/610_feature_controlled_rule_collection.sh index 685ec6b6..1e8b143f 100755 --- a/tests/e2e/610_feature_controlled_rule_collection.sh +++ b/tests/e2e/610_feature_controlled_rule_collection.sh @@ -186,12 +186,17 @@ version = "0.1.0" [dependencies] plugins = { path = "../plugins", host-module = true } EOT -echo 'import mcpp; int main() {}' > app/build.mcpp +printf 'import mcpp;\nint main() {}\n' > app/build.mcpp echo 'int main() {}' > app/src/main.cpp cd app if "$MCPP" build > b5.log 2>&1; then - echo "FAIL: a host-module package with no interface unit built"; cat b5.log; exit 1; fi -grep -q 'no interface unit at' b5.log || { echo "FAIL: the missing-lib-root diagnostic changed"; cat b5.log; exit 1; } + echo "FAIL: a library whose only unit implements a module no unit declares built"; cat b5.log; exit 1; fi +# The library is compiled as the package it is (mcpp#797), so its own defect is +# reported as it would be anywhere: the unit, the module, and where the build +# programs' workspace is to reproduce it. +grep -q 'nothing.exported' b5.log || { echo "FAIL: the diagnostic does not name the module"; cat b5.log; exit 1; } +grep -q 'rules/impl.cppm' b5.log || { echo "FAIL: the diagnostic does not name the unit"; cat b5.log; exit 1; } +grep -q "the build programs' workspace" b5.log || { echo "FAIL: the diagnostic does not say where to reproduce it"; cat b5.log; exit 1; } echo "PASS: a package with neither a lib root nor a listed interface unit keeps its diagnostic" echo "PASS: feature-controlled rule collection" diff --git a/tests/e2e/624_a_missing_host_module_names_the_key.sh b/tests/e2e/624_a_missing_host_module_names_the_key.sh index 90be3147..3317c996 100755 --- a/tests/e2e/624_a_missing_host_module_names_the_key.sh +++ b/tests/e2e/624_a_missing_host_module_names_the_key.sh @@ -65,7 +65,7 @@ standard = "c++23" modules = true import_std = true [build-dependencies] -rule = { path = "../rule" } +rule = { path = "../rule", host-module = false } [targets.consumer] kind = "bin" main = "src/main.cpp" @@ -81,7 +81,7 @@ echo "ok: refused by name, before the compiler" # The same project with the key added must build -- otherwise the check could # be refusing something that was always going to work. -sed -i 's|rule = { path = "../rule" }|rule = { path = "../rule", host-module = true }|' mcpp.toml +sed -i 's|rule = { path = "../rule", host-module = false }|rule = { path = "../rule", host-module = true }|' mcpp.toml "$MCPP" build >/dev/null 2>&1 || { echo "FAIL: adding host-module = true did not make it build"; exit 1; } echo "ok: the fix the message names is the fix that works" diff --git a/tests/e2e/798_a_rule_applies_through_a_source_it_claims.sh b/tests/e2e/798_a_rule_applies_through_a_source_it_claims.sh index 45187c8e..0db72bc7 100755 --- a/tests/e2e/798_a_rule_applies_through_a_source_it_claims.sh +++ b/tests/e2e/798_a_rule_applies_through_a_source_it_claims.sh @@ -33,7 +33,7 @@ rm -f build.mcpp "$MCPP" build > b1.log 2>&1 || { cat b1.log; echo "FAIL: leg 1 did not build"; exit 1; } grep -q 'Rules.*example.rules.toy' b1.log || { cat b1.log; echo "FAIL: leg 1 did not report the rule it applied"; exit 1; } -find target -name build.mcpp | grep -q . || { +find target -name synthesised.mcpp | grep -q . || { echo "FAIL: leg 1 synthesised no build program"; exit 1; } out="$("$MCPP" run 2>&1 | tail -1)" [[ "$out" == *"= 42"* ]] || { diff --git a/tests/e2e/857_a_host_module_several_members_import_is_compiled_once.sh b/tests/e2e/857_a_host_module_several_members_import_is_compiled_once.sh index 487e23bb..8f78245d 100755 --- a/tests/e2e/857_a_host_module_several_members_import_is_compiled_once.sh +++ b/tests/e2e/857_a_host_module_several_members_import_is_compiled_once.sh @@ -1,31 +1,29 @@ #!/usr/bin/env bash # requires: -# 857 -- a host module that several members import is compiled once for all of -# them, and a host module of a path dependency is never kept globally. +# 857 -- a library that several members' build programs use is compiled once +# for all of them, and a library of a path dependency is never kept globally. # -# #748 (B1, D4 of the plan for member selection and build programs). Every -# member of a workspace has a build program, and each imports the same host -# module, which imports the bundled `mcpp` module. Until this change each -# program compiled the `mcpp` module and every host module it imports into its -# own `target/.build-mcpp`, before its own compile and again for every program -# and every invocation: 7.8 s per program on #748's runner. +# #748 (B1), and mcpp#797. Every member of a workspace has a build program, +# and each uses the same library, which imports the engine interface. The +# programs of one build that share a host toolchain and a C++ standard are the +# members of one build programs' graph, below the workspace root, so the +# library is compiled once for them; a program of another standard is in +# another graph and never takes the others' BMI. # -# The host module here is a path dependency, so its sources can change without -# its name and version changing, and it is kept where the project keeps -# everything that is its own: under the workspace's `target/`. The engine's own -# `mcpp` module is identical for every project and is kept in the global cache. +# The library is a path dependency, so its sources can change without its name +# and version changing, and it is kept where the project keeps everything that +# is its own: under the workspace's `target/`. The engine interface is +# identical for every project and is kept in the global cache. # # Criteria: -# A. `m1` and `m2` import one host module and agree on every flag; the module -# is compiled once for the two, and the second program reuses the entry. -# (The compile commands are logged under `buildmcpp-host`.) -# B. `m3` states another standard. Its flags differ, so it gets an entry of -# its own and never takes the BMI of the others. -# C. The entries are under the workspace root's `target/.build-mcpp/ -# host-modules/`, each recording the inputs it was compiled from, and not -# under any member's own `target/`. -# D. With `--cache global`, no entry of the host module is written below the -# global cache root; the engine's `mcpp` module is. +# A. `m1` and `m2` agree on every flag: their programs are in one graph, +# which compiles the library once. +# B. `m3` states another standard. Its program is in another graph, with a +# compile of the library of its own, at its own standard. +# C. The graphs are under the workspace root's `target/.build-mcpp/host/`, +# and not under any member's own `target/`. +# D. With `--cache global`, nothing of the path dependency is written below +# the global cache root; the engine interface is. set -e source "$(dirname "$0")/_host_path.sh" @@ -96,38 +94,37 @@ for m in m1 m2 m3; do grep -qE "^ *build\.mcpp $m .* ran " a.log || fail "the program of $m did not run" a.log done -STORE=target/.build-mcpp/host-modules +HOST=target/.build-mcpp/host +# The compiles of the library: one entry per graph in that graph's database. +entries() { cat $HOST/*/compile_commands.json 2>/dev/null | grep -c '"file": ".*rules/src/rule857.cppm"' || true; } -# A and B. Three programs, two flag sets: two compiles, and one reuse. -compiles=$(grep -cE "host module 't857\.rules' (precompile|compile):" a.log || true) -[ "$compiles" = 2 ] || fail "A/B: the host module was compiled $compiles times for three programs of two flag sets (want 2)" a.log -reused=$(grep -cE "host module 't857\.rules': entry [0-9a-f]+ workspace \(reused\)" a.log || true) -[ "$reused" = 1 ] || fail "A: $reused programs reused the host module's entry (want 1: m1 and m2 share one)" a.log -entries=$(find "$STORE" -name entry.json | wc -l | tr -d ' ') -[ "$entries" = 2 ] || fail "B: the store holds $entries entries of the host module (want 2: one per standard)" a.log -# The two entries state different standards, which is what keeps them apart. -flags=$(for e in $(find "$STORE" -name entry.json); do - grep -h '"std_flag"' "$e"; done | sort -u | wc -l | tr -d ' ') -[ "$flags" = 2 ] || fail "B: the two entries record $flags distinct standard flags (want 2)" +# A and B. Three programs, two standards: two graphs, two compiles. +graphs=$(ls $HOST/*/programs.json 2>/dev/null | wc -l | tr -d ' ') +[ "$graphs" = 2 ] || fail "A/B: $graphs build programs' graphs for three programs of two standards (want 2)" a.log +[ "$(entries)" = 2 ] || fail "A/B: the library was compiled $(entries) times for three programs of two standards (want 2)" a.log +for g in $HOST/*/programs.json; do + if grep -q '"m3-' "$g"; then + grep -q '"m1-' "$g" && fail "B: m3's program shares a graph with m1's" "$g" + else + { grep -q '"m1-' "$g" && grep -q '"m2-' "$g"; } || fail "A: m1 and m2 are not in one graph" "$g" + fi +done +stds=$(cat $HOST/*/compile_commands.json | grep -o -- '-std=c++2[03]' | sort -u | wc -l | tr -d ' ') +[ "$stds" = 2 ] || fail "B: the two graphs compile at $stds distinct standards (want 2)" -# C. One store for the workspace, below the workspace root; none per member. +# C. One place for the workspace, below the workspace root; none per member. for m in m1 m2 m3; do - [ ! -d "$m/target/.build-mcpp/host-modules" ] \ - || fail "C: $m keeps its own host-module store beside the workspace's" -done -for e in $(find "$STORE" -name entry.json); do - grep -q '"module": "t857.rules"' "$e" || fail "C: an entry does not record its module name" "$e" - grep -q '"interface_sha256"' "$e" || fail "C: an entry does not record the digest of its interface" "$e" - grep -q '"imports"' "$e" || fail "C: an entry does not record the BMIs it imports" "$e" + [ ! -d "$m/target/.build-mcpp/host" ] \ + || fail "C: $m keeps its own build programs' graph beside the workspace's" done # D. The path dependency's module is not below the global cache root; the -# engine's module is. +# engine's interface is. CACHE_ROOT="$("$MCPP" cache dir | head -1)" if [ -d "$CACHE_ROOT/pkg" ] && grep -rl 't857.rules' "$CACHE_ROOT/pkg" --include=entry.json 2>/dev/null | grep -q .; then - fail "D: a host module of a path dependency was written below the global cache root" + fail "D: a module of a path dependency was written below the global cache root" fi [ -n "$(find "$CACHE_ROOT/pkg/_engine" -name entry.json 2>/dev/null)" ] \ - || fail "D: the engine's bundled module is not in the global cache" a.log + || fail "D: the engine interface is not in the global cache" a.log echo "PASS: 857_a_host_module_several_members_import_is_compiled_once" diff --git a/tests/e2e/859_a_host_module_entry_is_reused_while_its_inputs_agree.sh b/tests/e2e/859_a_host_module_entry_is_reused_while_its_inputs_agree.sh index 64621c25..dfcf95b9 100755 --- a/tests/e2e/859_a_host_module_entry_is_reused_while_its_inputs_agree.sh +++ b/tests/e2e/859_a_host_module_entry_is_reused_while_its_inputs_agree.sh @@ -1,24 +1,23 @@ #!/usr/bin/env bash # requires: -# 859 -- an entry of a host module is reused across invocations, and only while -# the inputs it records agree with the ones the compile would have. +# 859 -- a build program's library is compiled again exactly when what it is +# compiled from changed, and the program never runs on a stale compile. # -# #748 (B1). A host module from a path dependency is kept in the workspace's -# store, keyed by what it is compiled from. A hit compares the recorded inputs -# field by field (mcpp.bmi_cache::probe_cached), and never the hash alone. The -# sources of such a package can change without its name and version changing, -# so the key holds the interface's digest and the digest of the tree the -# interface sits in: an included header beside it is part of what was compiled. +# #748 (B1), mcpp#797 (R6). A library of a path dependency can change without +# its name and version changing, so a program's identity holds the content +# digest of that package's tree: an included header beside the interface is +# part of what was compiled. Hashing the interface file alone left an edited +# header unseen, and the program was neither compiled nor run again (R6). # # Criteria: -# A. A second run of a stale program reuses the entry: nothing is compiled. -# B. An edit to the interface compiles a new entry. +# A. A second run of a stale program (its declared input changed, its +# graph did not) compiles nothing. +# B. An edit to the interface compiles the library and the program again. # C. An edit to a header the interface includes, and not to the interface -# itself, compiles a new entry. The program of the project that imports -# the module re-runs either way; what must not happen is that it takes the -# old BMI. -# D. An entry whose recorded inputs disagree with the compile's (here, its -# standard flag edited in place) is a miss, and it is replaced. +# itself, does too, and the program reads the new value. +# D. A record that disagrees with the identity the program would have +# (here, edited in place) is a miss: the program is compiled again and +# the record replaced. set -e source "$(dirname "$0")/_host_path.sh" @@ -76,23 +75,24 @@ int main() { EOF echo 0 > marker.txt -STORE=target/.build-mcpp/host-modules -compile_lines() { grep -cE "host module 't859\.rules' (precompile|compile):" "$1" || true; } -entry_count() { find "$STORE" -name entry.json | wc -l | tr -d ' '; } +HOST=target/.build-mcpp/host +lib_object() { find $HOST -name 'rule859.m.o' | head -1; } +stamp() { stat -c %Y.%s "$(lib_object)" 2>/dev/null || stat -f %m.%z "$(lib_object)"; } +compiled() { grep -cE "build-program: app859: identity [0-9a-f]+ -> [0-9a-f]+, compiled" "$1" || true; } # A stale program: its declared input changed, the sources of its imports did not. rerun() { echo "$1" > marker.txt; MCPP_VERBOSE=1 "$MCPP" build > "$2" 2>&1 || fail "the build failed ($2)" "$2"; } MCPP_VERBOSE=1 "$MCPP" build > first.log 2>&1 || fail "the first build failed" first.log -[ "$(compile_lines first.log)" = 1 ] || fail "the first build compiled the host module $(compile_lines first.log) times" first.log -[ "$(entry_count)" = 1 ] || fail "the first build left $(entry_count) entries" -grep -q "T859 value one" first.log || fail "the program did not read the host module's value" first.log +[ -n "$(lib_object)" ] || fail "the first build compiled no library object" first.log +grep -q "T859 value one" first.log || fail "the program did not read the library's value" first.log +s0=$(stamp) # A +sleep 1 rerun 1 a.log -[ "$(compile_lines a.log)" = 0 ] || fail "A: a stale program compiled its host module again" a.log -grep -qE "host module 't859\.rules': entry [0-9a-f]+ workspace \(reused\)" a.log \ - || fail "A: the log does not say the entry was reused" a.log -grep -q "T859 value one" a.log || fail "A: the reused BMI gave another value" a.log +[ "$(compiled a.log)" = 0 ] || fail "A: a stale program was compiled again" a.log +[ "$(stamp)" = "$s0" ] || fail "A: the library was compiled again" a.log +grep -q "T859 value one" a.log || fail "A: the program gave another value" a.log # B sleep 1 @@ -104,31 +104,24 @@ import mcpp; export int rule_value() { return T859_VALUE + 0; } EOF rerun 2 b.log -[ "$(compile_lines b.log)" = 1 ] || fail "B: an edit to the interface did not compile a new entry" b.log -[ "$(entry_count)" = 2 ] || fail "B: expected two entries, found $(entry_count)" b.log +[ "$(compiled b.log)" = 1 ] || fail "B: an edit to the interface did not compile the program again" b.log +[ "$(stamp)" != "$s0" ] || fail "B: an edit to the interface did not compile the library" b.log +s1=$(stamp) # C sleep 1 printf '#define T859_VALUE 2\n' > "$TMP/rules/src/value.h" rerun 3 c.log -[ "$(compile_lines c.log)" = 1 ] || fail "C: an edit to an included header did not compile a new entry" c.log -[ "$(entry_count)" = 3 ] || fail "C: expected three entries, found $(entry_count)" c.log -grep -q "T859 value two" c.log || fail "C: the program took the BMI compiled before the header changed" c.log +[ "$(compiled c.log)" = 1 ] || fail "C: an edit to an included header did not compile the program again" c.log +[ "$(stamp)" != "$s1" ] || fail "C: an edit to an included header did not compile the library" c.log +grep -q "T859 value two" c.log || fail "C: the program took the compile made before the header changed" c.log # D -# The entry the last build used is the one the tree as it stands maps to: the -# most recently written or read. Its recorded standard flag is edited in place, -# so that the key still names it and what it records no longer agrees. -entry=$(ls -t $(find "$STORE" -name entry.json) | head -1) -grep -q '"std_flag"' "$entry" || fail "D: the entry records no standard flag" "$entry" -sed 's/"std_flag": "[^"]*"/"std_flag": "-std=c++98"/' "$entry" > "$entry.edited" -mv "$entry.edited" "$entry" +rec=$(ls $HOST/*/programs.json | head -1) +sed 's/"identity": "[0-9a-f]*"/"identity": "0000000000000000"/' "$rec" > "$rec.edited" && mv "$rec.edited" "$rec" rerun 4 d.log -[ "$(compile_lines d.log)" = 1 ] || fail "D: an entry whose recorded inputs disagree was used without compiling" d.log -if grep -q -- '-std=c++98' "$entry"; then - fail "D: the entry that disagreed was not replaced" "$entry" -fi -[ "$(entry_count)" = 3 ] || fail "D: expected the entry to be replaced in place, found $(entry_count) entries" d.log -grep -q "T859 value two" d.log || fail "D: the replaced entry gave another value" d.log +[ "$(compiled d.log)" = 1 ] || fail "D: a record that disagrees was used without compiling" d.log +grep -q '"identity": "0000000000000000"' "$rec" && fail "D: the record that disagreed was not replaced" "$rec" +grep -q "T859 value two" d.log || fail "D: the replaced compile gave another value" d.log echo "PASS: 859_a_host_module_entry_is_reused_while_its_inputs_agree" From 122c4f6c127f28b144988204f9c34797883706c1 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sun, 11 Oct 2026 06:46:14 +0900 Subject: [PATCH 05/12] One library interface; Form B lib; rule_module checked; the build database describes build programs, host instances and the engine interface (#797) --- modules/manifest/src/toml.cppm | 13 +++ modules/manifest/src/xpkg.cppm | 12 ++- src/build/build_database.cppm | 39 +++++++-- src/build/compile_commands.cppm | 6 ++ src/build/prepare.cppm | 15 ++++ src/build/prepare/features.cpp | 26 ++++++ src/build/prepare/host_graph.cpp | 22 ++++- src/build/prepare/plan.cpp | 1 + src/build/prepare/scan.cpp | 26 +++--- src/build/prepare/state.cppm | 2 + src/cli/cmd_build.cppm | 9 +++ src/cli/cmd_self.cppm | 5 ++ src/modgraph/library_interface.cppm | 121 ++++++++++++++++++++++++++++ src/modgraph/validate.cppm | 29 ++++--- 14 files changed, 286 insertions(+), 40 deletions(-) create mode 100644 src/modgraph/library_interface.cppm diff --git a/modules/manifest/src/toml.cppm b/modules/manifest/src/toml.cppm index d611d74f..538c4f6b 100644 --- a/modules/manifest/src/toml.cppm +++ b/modules/manifest/src/toml.cppm @@ -5243,6 +5243,19 @@ std::filesystem::path resolve_lib_root_path(const Manifest& manifest, const std::filesystem::path& projectRoot) { auto candidates = lib_root_candidates(manifest); std::error_code ec; + // A Form B `lib` names the file below the archive's wrapping directory + // the way its `sources` and `include_dirs` do: `*/` stands for that one + // directory, whatever the archive calls it (mcpp#797). + if (!manifest.lib.path.empty()) { + auto it = manifest.lib.path.begin(); + if (it != manifest.lib.path.end() && *it == "*") { + std::filesystem::path rest; + for (++it; it != manifest.lib.path.end(); ++it) rest /= *it; + for (auto const& d : std::filesystem::directory_iterator(projectRoot, ec)) + if (d.is_directory(ec) && std::filesystem::is_regular_file(d.path() / rest, ec)) + return d.path().lexically_relative(projectRoot) / rest; + } + } for (auto const& rel : candidates) if (std::filesystem::is_regular_file(projectRoot / rel, ec)) return rel; // None on disk: hand back the conventional first candidate so the caller's diff --git a/modules/manifest/src/xpkg.cppm b/modules/manifest/src/xpkg.cppm index 3b960e47..c6c1139e 100644 --- a/modules/manifest/src/xpkg.cppm +++ b/modules/manifest/src/xpkg.cppm @@ -257,7 +257,7 @@ namespace mcpp::manifest { // the parser accepts. inline constexpr std::string_view kKnownXpkgKeys[] = { "cflags", "c_standard", "cxxflags", "defines", "deps", "features", "flags", - "generated_files", "import_std", "include_dirs", "include_dirs_after", + "generated_files", "import_std", "include_dirs", "include_dirs_after", "lib", "private_include_dirs", "language", "ldflags", "linux", "macosx", "modules", "provides", "requires", "runtime", @@ -1342,6 +1342,16 @@ synthesize_from_xpkg_lua(std::string_view luaContent, auto v = cur.read_bareword(); m.language.importStd = (v == "true"); } + // `lib = ""` (mcpp#797): the interface root, as `[lib] path` states it + // in a manifest. Relative to the version directory; `*/` stands for the + // archive's wrapping directory, as in `sources`. + else if (key == "lib") { + auto v = cur.read_string(); + if (v.empty()) + return std::unexpected(ManifestError{ + "expected a path string after `lib =`", m.sourcePath, 0, 0}); + m.lib.path = v; + } else if (key == "modules") { // `{ "a", "b", ... }` if (!cur.consume('{')) { diff --git a/src/build/build_database.cppm b/src/build/build_database.cppm index 4ebf4d6b..60c7592d 100644 --- a/src/build/build_database.cppm +++ b/src/build/build_database.cppm @@ -58,6 +58,15 @@ struct Member { // per package the plan tests (each member of a group plan): a new file // matching it is a new test target, and so a new unit in the document. std::vector>> tests; + // A build programs' graph (mcpp#797, SPEC-005 v1.8): its sets are named by + // the reserved suffixes -- `:build` for a build program (from + // `programOf`, member to package), `:host` for a library of the + // build-program configuration, `mcpp:core` for the engine interface, + // `mcpp:std:host` for its standard library -- carry the configuration + // `build-program`, and enter the S1 document only, never the JSON + // Compilation Database. + bool buildPrograms = false; + std::map programOf; }; struct Rendered { @@ -414,8 +423,21 @@ Rendered render(std::span members, const std::string kind = isTest ? "test" : subject != subjectByName.end() ? target_kind(*subject->second) : "library"; - auto& set = set_for(member.setPrefix + package + (isTest ? ":test" : ""), - package, kind); + std::string setName = member.setPrefix + package + (isTest ? ":test" : ""); + std::string setKind = kind; + if (member.buildPrograms) { + if (auto it = member.programOf.find(package); it != member.programOf.end()) { + setName = it->second + ":build"; + setKind = "executable"; + } else if (package == "mcpp.core") { + setName = "mcpp:core"; + setKind = "library"; + } else { + setName = package + ":host"; + setKind = "library"; + } + } + auto& set = set_for(setName, package, setKind); for (auto const* dirs : {&cu.localIncludeDirs, &cu.localIncludeDirsAfter}) for (auto const& dir : *dirs) if (relative_to(dir, generatedTree)) @@ -424,7 +446,8 @@ Rendered render(std::span members, if (!cu.providesModule.empty()) provides[cu.providesModule] = ""; nlohmann::json requires_ = nlohmann::json::array(); for (auto const& name : cu.imports) requires_.push_back(name); - if (describedCommands.insert(inv.file + '\x1f' + inv.output).second) + if (!member.buildPrograms + && describedCommands.insert(inv.file + '\x1f' + inv.output).second) r.compileCommands.push_back(nlohmann::json{ {"directory", inv.directory}, {"file", inv.file}, @@ -451,13 +474,16 @@ Rendered render(std::span members, // (P1). S1-12-1: they are translation units of the build like any // other, so `--spec compile-commands` lists them too. if (!ctx.plan.stdModuleUnits.empty()) { - auto& set = set_for(member.setPrefix + std::string(kStdSetName), + auto& set = set_for(member.buildPrograms + ? std::string(kStdSetName) + ":host" + : member.setPrefix + std::string(kStdSetName), std::string(kStdSetName), "library"); for (auto const& unit : ctx.plan.stdModuleUnits) { const auto sourceStr = native_string(unit.source); const auto objectStr = native_string(unit.object); const auto workDirStr = native_string(unit.workDirectory); - if (describedCommands.insert(sourceStr + '\x1f' + objectStr).second) + if (!member.buildPrograms + && describedCommands.insert(sourceStr + '\x1f' + objectStr).second) r.compileCommands.push_back(nlohmann::json{ {"directory", workDirStr}, {"file", sourceStr}, @@ -547,7 +573,8 @@ Rendered render(std::span members, {"translation-units", std::move(set.units)}, {"ide", { {"toolchain", tcId}, - {"configuration", ctx.profile}, + {"configuration", member.buildPrograms ? std::string("build-program") + : ctx.profile}, {"kind", set.kind}, }}, }; diff --git a/src/build/compile_commands.cppm b/src/build/compile_commands.cppm index 39f9ba24..6720ec27 100644 --- a/src/build/compile_commands.cppm +++ b/src/build/compile_commands.cppm @@ -505,6 +505,12 @@ std::vector unit_invocations(const BuildPlan& plan, for (auto& w : split_flags(traits.moduleInterfaceLangFlag)) inv.arguments.push_back(std::move(w)); } + // A build program's `.mcpp` source: an extension no driver knows, so + // its language is stated, as the compile states it (mcpp#797). + else if (mcpp::needs_explicit_cxx_language(cu.source)) { + for (auto& w : split_flags(traits.moduleImplLangFlag)) + inv.arguments.push_back(std::move(w)); + } inv.arguments.push_back("-c"); inv.arguments.push_back(inv.file); inv.arguments.push_back("-o"); diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index c9bc5dc1..581255d9 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -430,7 +430,22 @@ export struct DepSourceRoot { bool operator==(const DepSourceRoot&) const = default; }; +export struct BuildContext; + +// A BUILD PROGRAMS' GRAPH a prepare planned (mcpp#797), for the readers that +// describe it: `mcpp emit build-database` lists its units (SPEC-005 v1.8). +// `programOf` maps each member of the graph to the package whose build program +// it is. +export struct BuildProgramGraph { + std::shared_ptr ctx; + std::filesystem::path dir; + std::map programOf; +}; + export struct BuildContext { + // The build programs' graphs this prepare planned to describe them + // (`plan_only`; empty otherwise). + std::vector buildProgramGraphs; // The root is a build plugin, planned in the build-program configuration // (mcpp#797, SPEC-007 P3). `mcpp pack` refuses it (P5). bool buildPluginRoot = false; diff --git a/src/build/prepare/features.cpp b/src/build/prepare/features.cpp index 867d8e09..3559910e 100644 --- a/src/build/prepare/features.cpp +++ b/src/build/prepare/features.cpp @@ -733,6 +733,32 @@ static std::expected step6_device_extensions_and_rules(Prepar // anything is compiled and a build that scanned a // dependency to decide what to write would order the two // the wrong way round. + // `rule_module` names one of the modules the feature's + // own sources declare (SPEC-008 I2 as amended, mcpp#797): + // a name nothing declares would be imported by the + // synthesised program and fail as an unknown module. + if (auto mit = dep.manifest.featureRuleModule.find(f); + mit != dep.manifest.featureRuleModule.end()) { + std::vector declared; + if (auto fs = dep.manifest.buildConfig.featureSources.find(f); + fs != dep.manifest.buildConfig.featureSources.end()) + for (auto const& g : fs->second) + for (auto const& file : mcpp::modgraph::expand_glob(dep.root, g)) { + std::ifstream in(file, std::ios::binary); + const std::string text{std::istreambuf_iterator(in), {}}; + for (auto& d : mcpp::pm::declared_module_roots(text)) + declared.push_back(std::move(d)); + } + if (std::ranges::find(declared, mit->second) == declared.end()) { + std::string list; + for (auto const& d : declared) list += (list.empty() ? "" : ", ") + d; + return std::unexpected(std::format( + "{}: feature \"{}\" states `rule_module = \"{}\"`, and no unit " + "among its sources declares that module (they declare: {})", + dep.manifest.package.name, f, mit->second, + list.empty() ? "nothing" : list)); + } + } if (auto mit = dep.manifest.featureRuleModule.find(f); mit != dep.manifest.featureRuleModule.end() && std::ranges::find(ruleModules, mit->second) == ruleModules.end()) { diff --git a/src/build/prepare/host_graph.cpp b/src/build/prepare/host_graph.cpp index 78737aff..192502ea 100644 --- a/src/build/prepare/host_graph.cpp +++ b/src/build/prepare/host_graph.cpp @@ -574,6 +574,15 @@ std::map resolution_pins(const PrepareState& state) { std::expected, std::string> build_group(PrepareState& state, const fs::path& dir, const std::vector& all, const std::vector& goals, const HostToolchain& tc) { + // The graph `mcpp emit build-database` describes, beside the one it builds. + auto describe = [&](const BuildContext& ctx) { + if (!state.overrides.plan_only) return; + BuildProgramGraph g; + g.ctx = std::make_shared(ctx); + g.dir = dir; + for (auto const& p : all) g.programOf[p.member] = qualified(manifest_of(state, p.index)); + state.buildProgramGraphs.push_back(std::move(g)); + }; BuildOverrides sub; sub.project_root = dir; sub.work_dir = dir; @@ -622,6 +631,8 @@ build_group(PrepareState& state, const fs::path& dir, const std::vector return std::unexpected(std::format("planning the build programs failed: {}{}", ctx.error(), reproduce())); timer.mark("planned"); + describe(*ctx); + if (goals.empty()) return std::map{}; std::map binaries; std::vector ninjaGoals; @@ -860,7 +871,16 @@ ensure_build_programs(PrepareState& state, const std::vector& packa p.identity)); stale.push_back(&p); } - if (stale.empty()) continue; + if (stale.empty()) { + // Described, not built: `mcpp emit build-database` lists the graph + // of programs that are current too. + if (state.overrides.plan_only) { + write_group(state, dir, programs, standard, *enginePackage); + if (auto r = build_group(state, dir, programs, {}, *tc); !r) + mcpp::log::verbose("host-graph", r.error()); + } + continue; + } for (auto const* p : stale) { warn_reserved_prefixes(state, p->index); if (auto r = check_module_collisions(state, p->index); !r) diff --git a/src/build/prepare/plan.cpp b/src/build/prepare/plan.cpp index 70691c2e..012bdc81 100644 --- a/src/build/prepare/plan.cpp +++ b/src/build/prepare/plan.cpp @@ -1868,6 +1868,7 @@ static std::expected step13_empty_link_check(PrepareState& st std::expected phase13_finish(PrepareState& state) { BuildContext ctx; ctx.buildPluginRoot = !state.buildPluginRoots.empty(); + ctx.buildProgramGraphs = state.buildProgramGraphs; ctx.strict = state.overrides.strict; ctx.manifest = *state.m; ctx.tc = *state.tc; diff --git a/src/build/prepare/scan.cpp b/src/build/prepare/scan.cpp index 167cd4e9..dd866538 100644 --- a/src/build/prepare/scan.cpp +++ b/src/build/prepare/scan.cpp @@ -18,6 +18,7 @@ import mcpp.modgraph.glob; import mcpp.modgraph.graph; import mcpp.modgraph.scanner; import mcpp.modgraph.validate; +import mcpp.modgraph.library_interface; import mcpp.toolchain.hostflags; // the compile-token producer the package std module reuses import mcpp.toolchain.cppfly; import mcpp.toolchain.detect; @@ -951,23 +952,14 @@ static void step11_public_module_check(PrepareState& state) { for (std::size_t i = 1; i < state.packages.size(); ++i) { auto const& pr = state.packages[i]; if (!workspace_member_of(state, pr.root).empty()) continue; - const auto rootFile = (pr.root / mcpp::manifest::resolve_lib_root_path(pr.manifest, pr.root)) - .lexically_normal(); - std::error_code ec; - if (!std::filesystem::is_regular_file(rootFile, ec)) continue; - const mcpp::modgraph::SourceUnit* rootUnit = nullptr; - for (auto const& u : g.units) - if (u.path.lexically_normal() == rootFile && u.provides) { rootUnit = &u; break; } - if (!rootUnit) continue; - std::set pub{primary(rootUnit->provides->logicalName)}; - std::vector work(pub.begin(), pub.end()); - while (!work.empty()) { - auto mod = work.back(); work.pop_back(); - for (auto const* u : unitsOf[mod]) - for (auto const& re : u->reexports) - if (auto p = primary(re.logicalName); pub.insert(p).second) work.push_back(p); - } - publicOf[qualified(pr.manifest)] = std::move(pub); + // One definition of what a library offers (mcpp.modgraph.library_interface). + const auto iface = mcpp::modgraph::library_interface( + g, pr.manifest, pr.root, qualified(pr.manifest), + i < state.activeFeaturesByPackage.size() ? state.activeFeaturesByPackage[i] + : std::vector{}); + // A dependency without an interface root states no interface. + if (iface.rootModule.empty()) continue; + publicOf[qualified(pr.manifest)] = iface.publicModules; } std::set warned; diff --git a/src/build/prepare/state.cppm b/src/build/prepare/state.cppm index 0f15066a..a5fdf526 100644 --- a/src/build/prepare/state.cppm +++ b/src/build/prepare/state.cppm @@ -681,6 +681,8 @@ struct PrepareState { std::vector compatLibraryEdges; // The roots (package directories) planned as build plugins (P3). std::set buildPluginRoots; + // The build programs' graphs planned for description (`plan_only`). + std::vector buildProgramGraphs; // A use of a deprecated reading (mcpp.pm.compat.registry) by the manifest // of package `i`: warned where that manifest can be edited -- the root, a // workspace member, a path package outside the package store -- and diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index 83b2b77a..cd31cb82 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -746,6 +746,15 @@ export int cmd_emit_build_database(const mcpplibs::cmdline::ParsedArgs& parsed) for (std::size_t i = 0; i < contexts.size(); ++i) { members.push_back({&contexts[i], configurationPrefix(contexts[i]), workDirs[i], testDiscovery[i]}); + // The build programs' graphs of this plan (mcpp#797, SPEC-005 v1.8). + for (auto const& g : contexts[i].buildProgramGraphs) { + mcpp::build::database::Member m; + m.ctx = g.ctx.get(); + m.workDir = g.dir; + m.buildPrograms = true; + m.programOf = g.programOf; + members.push_back(std::move(m)); + } if (!mcpp::build::declared_program_inputs(workDirs[i]).empty()) ranBuildPrograms = true; for (auto const& sp : contexts[i].sourcePackages) { diff --git a/src/cli/cmd_self.cppm b/src/cli/cmd_self.cppm index 84b8ee20..405b4b08 100644 --- a/src/cli/cmd_self.cppm +++ b/src/cli/cmd_self.cppm @@ -8,6 +8,7 @@ module; export module mcpp.cli.cmd_self; import std; +import mcpp.build.engine_package; import mcpplibs.cmdline; import mcpp.doctor; import mcpp.toolchain.fingerprint; // MCPP_VERSION @@ -58,6 +59,10 @@ nlohmann::json env_data_readonly() { ("xlings" + std::string(mcpp::platform::exe_suffix)))}, {"config", s(config)}, {"buildCache", s(mcpp::home::cache_root())}, + // The engine interface a build program imports, as a package directory + // (mcpp#797): written by the first build program of this version, read + // by an editor (`mcpp:core` in the build database). + {"engineModules", s(mcpp::build::engine::package_dir())}, {"mcppVersion", std::string(mcpp::toolchain::MCPP_VERSION)}, // What a build with nothing configured resolves on this host (WS8): // the one function the first run uses. Probing for a usable MSVC is diff --git a/src/modgraph/library_interface.cppm b/src/modgraph/library_interface.cppm new file mode 100644 index 00000000..42929109 --- /dev/null +++ b/src/modgraph/library_interface.cppm @@ -0,0 +1,121 @@ +// mcpp.modgraph.library_interface — what a library offers, defined once +// (SPEC-008 v0.2, mcpp#797). +// +// Before this module the answer had two definitions and depended on who asked. +// An ordinary consumer could import every `export module` unit of a +// dependency's sources, and the interface root was read only for warnings; the +// build program's path required the root and took the listed interface units +// beside it. `mcpp pack` and the warning W3 computed the public modules a third +// and a fourth time. Every reader now asks this function: +// +// the interface root `[lib] path` (a Form B descriptor's `lib`), else the +// convention `src/.` when +// the file exists; at most one. +// public modules the root's module and what it re-exports with +// `export import`, transitively; and, for each active +// feature, the primary interface units its `sources` +// add, with what they re-export. A feature is enabled by +// the consumer, and `export import` cannot say "only when +// enabled", so a feature's units are that feature's +// interface (SPEC-008 I2, as amended). +// exported modules every primary module interface the package compiles: +// what a consumer can import. One that is exported and +// not public draws W3. +// +// A package with no interface root states no interface: its exported modules +// are its public modules (SPEC-008 phase one). + +export module mcpp.modgraph.library_interface; + +import std; +import mcpp.manifest; +import mcpp.modgraph.graph; +import mcpp.modgraph.scanner; + +export namespace mcpp::modgraph { + +struct LibraryInterface { + std::filesystem::path root; // absolute; empty when the package has none + bool rootDeclared = false; + std::string rootModule; + std::set publicModules; // primary module names + std::set exportedModules; // primary module names +}; + +// `packageName` is the name the scan recorded on the package's units (its +// qualified name). `activeFeatures` are the features of the instance asked about. +LibraryInterface library_interface(const Graph& g, + const mcpp::manifest::Manifest& manifest, + const std::filesystem::path& packageRoot, + std::string_view packageName, + const std::vector& activeFeatures); + +} // namespace mcpp::modgraph + +namespace mcpp::modgraph { + +namespace { +std::string primary(std::string_view name) { return std::string(name.substr(0, name.find(':'))); } +} + +LibraryInterface library_interface(const Graph& g, const mcpp::manifest::Manifest& manifest, + const std::filesystem::path& packageRoot, + std::string_view packageName, + const std::vector& activeFeatures) { + LibraryInterface out; + std::map> unitsOf; + for (auto const& u : g.units) { + if (u.packageName != packageName || !u.provides) continue; + const auto p = primary(u.provides->logicalName); + unitsOf[p].push_back(&u); + if (u.provides->logicalName.find(':') == std::string::npos) out.exportedModules.insert(p); + } + auto closeOver = [&](std::string start) { + std::vector work{start}; + if (!out.publicModules.insert(start).second) return; + while (!work.empty()) { + auto mod = work.back(); + work.pop_back(); + for (auto const* u : unitsOf[mod]) + for (auto const& re : u->reexports) + if (auto p = primary(re.logicalName); out.publicModules.insert(p).second) + work.push_back(p); + } + }; + + out.rootDeclared = !manifest.lib.path.empty(); + const auto rootFile = (packageRoot / mcpp::manifest::resolve_lib_root_path(manifest, packageRoot)) + .lexically_normal(); + std::error_code ec; + if (std::filesystem::is_regular_file(rootFile, ec)) { + out.root = rootFile; + for (auto const& u : g.units) + if (u.packageName == packageName && u.provides + && u.path.lexically_normal() == rootFile) { + out.rootModule = primary(u.provides->logicalName); + break; + } + } + + if (out.rootModule.empty()) { + out.publicModules = out.exportedModules; + } else { + closeOver(out.rootModule); + } + for (auto const& f : activeFeatures) { + auto it = manifest.buildConfig.featureSources.find(f); + if (it == manifest.buildConfig.featureSources.end()) continue; + std::set files; + for (auto const& glob : it->second) + for (auto const& file : expand_glob(packageRoot, glob)) + files.insert(file.lexically_normal()); + for (auto const& u : g.units) + if (u.packageName == packageName && u.provides + && u.provides->logicalName.find(':') == std::string::npos + && files.contains(u.path.lexically_normal())) + closeOver(primary(u.provides->logicalName)); + } + return out; +} + +} // namespace mcpp::modgraph diff --git a/src/modgraph/validate.cppm b/src/modgraph/validate.cppm index 358fada8..7eaf35a5 100644 --- a/src/modgraph/validate.cppm +++ b/src/modgraph/validate.cppm @@ -6,6 +6,7 @@ import std; import mcpp.manifest; import mcpp.modgraph.graph; import mcpp.modgraph.scanner; +import mcpp.modgraph.library_interface; export namespace mcpp::modgraph { @@ -87,31 +88,29 @@ ValidateReport validate(const Graph& g, } } - // 2. modules.exports vs scanner result. - // Only checks the primary manifest's package — dep packages have their - // own [modules].exports validated by their own builds. + // 2. `[modules] exports` (a Form B descriptor's `modules`) is an + // ASSERTION about the package's public modules (mcpp#797, SPEC-008): + // the interface root and what it re-exports, or, with no root, every + // exported module. Only the primary manifest's package is checked -- + // a dependency's assertion is its own build's to check. if (!manifest.modules.exports_.empty()) { std::set declared(manifest.modules.exports_.begin(), manifest.modules.exports_.end()); - std::set actual; - for (auto& u : g.units) { - if (u.provides && u.packageName == manifest.package.name) { - auto base = u.provides->logicalName.substr(0, u.provides->logicalName.find(':')); - actual.insert(base); - } - } - // Any actual module that isn't in declared is a violation. - for (auto& m : actual) { + const auto iface = library_interface(g, manifest, projectRoot, + manifest.package.name, {}); + for (auto& m : iface.publicModules) { if (!declared.contains(m)) { r.errors.push_back({{}, - std::format("module '{}' is exported by code but not listed in [modules].exports", m)}); + std::format("module '{}' is a public module of this package but not listed " + "in [modules].exports", m)}); } } if (manifest.modules.strict) { for (auto& m : declared) { - if (!actual.contains(m)) { + if (!iface.publicModules.contains(m)) { r.errors.push_back({{}, - std::format("module '{}' declared in [modules].exports but never exported (strict)", m)}); + std::format("module '{}' declared in [modules].exports but not a public " + "module of this package (strict)", m)}); } } } From ce8f675db52e2aae0fc28d1bb30726e7764d7e43 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sun, 11 Oct 2026 06:49:05 +0900 Subject: [PATCH 06/12] Observability: mcpp why build-programs, mcpp why compat, instances in the resolution record (#797) --- src/build/prepare/target_side.cpp | 9 +++ src/cli.cppm | 2 +- src/cli/cmd_self.cppm | 8 +- src/doctor.cppm | 126 +++++++++++++++++++++++++++++- src/wire.cppm | 9 ++- 5 files changed, 147 insertions(+), 7 deletions(-) diff --git a/src/build/prepare/target_side.cpp b/src/build/prepare/target_side.cpp index 246bcb5d..3e935ff7 100644 --- a/src/build/prepare/target_side.cpp +++ b/src/build/prepare/target_side.cpp @@ -1511,6 +1511,15 @@ static void step9_define_graph_package_entry_closure(PrepareState& state) { }); } entry["requested_by"] = std::move(requests); + // Which instances the package has in this build (mcpp#797): a target + // instance, compiled in this plan, and a build-program instance, + // compiled in the build programs' graph. + { + nlohmann::json instances = nlohmann::json::array(); + if (i == 0 || !state.packages[i].buildTimeOnly) instances.push_back("target"); + if (state.packages[i].hostInstance) instances.push_back("build-program"); + entry["instances"] = std::move(instances); + } if (auto form = state.dependencyLinkForms.find(i); form != state.dependencyLinkForms.end() && form->second.recorded) entry["link"] = { diff --git a/src/cli.cppm b/src/cli.cppm index a7e3bbd7..2d766eef 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -572,7 +572,7 @@ int run(int argc, char** argv) { .action(wrap_rc(cmd_clean))) .subcommand(cl::App("why") .description("Explain how the toolchain / runtime / deps / runners / sources were resolved") - .arg(cl::Arg("topic").help("toolchain | runtime | deps | runners | sources | tool | payload (default: all)")) + .arg(cl::Arg("topic").help("toolchain | runtime | deps | runners | sources | tool | payload | build-programs | compat (default: all)")) .arg(cl::Arg("subject").help("for `tool` and `payload`: the name to look up")) // `--target` / `--toolchain` make this a QUERY rather than a // report on the current directory's default: "what would a build diff --git a/src/cli/cmd_self.cppm b/src/cli/cmd_self.cppm index 405b4b08..f0a65902 100644 --- a/src/cli/cmd_self.cppm +++ b/src/cli/cmd_self.cppm @@ -116,10 +116,14 @@ export int cmd_why(const mcpplibs::cmdline::ParsedArgs& parsed) { if (topic == "sources" || topic == "tool" || topic == "payload") return mcpp::doctor::why_sources_json(topic, parsed.positional(1), parsed.option_or_empty("features").value()); + if (topic == "build-programs" || topic == "compat") + return mcpp::doctor::why_build_programs_or_compat_json( + topic, parsed.option_or_empty("features").value()); if (!topic.empty() && topic != "toolchain") { std::println(stderr, - "error: --format json is defined for `mcpp why toolchain` and " - "`mcpp why sources|tool|payload`; '{}' has no machine-readable " + "error: --format json is defined for `mcpp why toolchain`, " + "`mcpp why sources|tool|payload`, `mcpp why build-programs` and " + "`mcpp why compat`; '{}' has no machine-readable " "shape yet", topic); return 2; } diff --git a/src/doctor.cppm b/src/doctor.cppm index c633f975..946d0327 100644 --- a/src/doctor.cppm +++ b/src/doctor.cppm @@ -1283,16 +1283,90 @@ export int why_sources_json(std::string_view topic, std::string_view subject, return 0; } +// ── `mcpp why build-programs` and `mcpp why compat` (mcpp#797) ─────────── + +// The build programs' graphs of the project at `root`, read from the records +// each graph writes beside its workspace (`programs.json`): which program is in +// which graph, what it uses, the identity it was compiled from, and its binary. +nlohmann::json build_programs_json(const std::filesystem::path& root) { + nlohmann::json graphs = nlohmann::json::array(); + std::error_code ec; + const auto host = root / "target" / ".build-mcpp" / "host"; + if (!std::filesystem::is_directory(host, ec)) return graphs; + for (auto const& d : std::filesystem::directory_iterator(host, ec)) { + const auto rec = d.path() / "programs.json"; + if (!std::filesystem::is_regular_file(rec, ec)) continue; + std::ifstream in(rec); + auto j = nlohmann::json::parse(in, nullptr, false); + if (!j.is_object()) continue; + nlohmann::json programs = nlohmann::json::array(); + for (auto it = j.begin(); it != j.end(); ++it) { + auto r = it.value(); + r["member"] = it.key(); + programs.push_back(std::move(r)); + } + graphs.push_back({{"workspace", d.path().generic_string()}, {"programs", programs}}); + } + return graphs; +} + +// Every use of a deprecated reading this process recorded, with the row of +// the deprecation registry it belongs to. +nlohmann::json compat_uses_json() { + nlohmann::json uses = nlohmann::json::array(); + for (auto const& u : mcpp::pm::compat::recorded_uses()) { + nlohmann::json row{{"id", u.id}, {"where", u.where}, {"detail", u.detail}, + {"warned", u.warned}}; + if (auto const* d = mcpp::pm::compat::find_deprecation(u.id)) { + row["since"] = std::string(d->since); + row["floor"] = std::string(d->floor); + row["canonical"] = std::string(d->canonical); + row["docs"] = std::string(d->docs); + } + uses.push_back(std::move(row)); + } + return uses; +} + +export int why_build_programs_or_compat_json(std::string_view topic, const std::string& features) { + const bool wasQuiet = mcpp::ui::is_quiet(); + mcpp::ui::set_quiet(true); + struct QuietGuard { + bool prev; + ~QuietGuard() { mcpp::ui::set_quiet(prev); } + } quietGuard{wasQuiet}; + mcpp::build::BuildOverrides ov; + ov.features = features; + auto ctx = mcpp::build::prepare_build(/*print_fingerprint=*/false, + /*includeDevDeps=*/false, {}, ov); + nlohmann::json data; + std::vector diags; + data["status"] = ctx ? "ok" : "refused"; + if (!ctx) + diags.push_back({.code = "prepare.failed", .severity = mcpp::wire::Severity::Error, + .message = ctx.error()}); + if (topic == "compat") data["uses"] = compat_uses_json(); + else data["graphs"] = ctx ? build_programs_json(ctx->projectRoot) : nlohmann::json::array(); + mcpp::wire::emit({ + .kind = topic == "compat" ? "mcpp.why.compat" : "mcpp.why.build-programs", + .effects = { mcpp::wire::Effect::ReadProject }, + .data = data, + .diagnostics = diags, + }); + return ctx ? 0 : 2; +} + export int why_report(const std::string& topic, const std::string& features, const std::string& subject = {}) { const bool all = topic.empty() || topic == "all"; static constexpr std::string_view kTopics[] = { "all", "toolchain", "runtime", "deps", "runners", "sources", "tool", "payload", + "build-programs", "compat", }; if (!topic.empty() && std::ranges::find(kTopics, std::string_view(topic)) == std::end(kTopics)) { std::println(stderr, "error: '{}' is not a topic of `mcpp why`; the topics are " - "toolchain, runtime, deps, runners, sources, tool and " - "payload ", topic); + "toolchain, runtime, deps, runners, sources, tool , " + "payload , build-programs and compat", topic); return 2; } @@ -1350,6 +1424,39 @@ export int why_report(const std::string& topic, const std::string& features, if (all || topic == "sources" || topic == "tool" || topic == "payload") print_sources(ctx->sources, all ? std::string_view("sources") : std::string_view(topic), subject, ctx->projectRoot); + // mcpp#797: which build programs' graph compiled each program, from what. + if (all || topic == "build-programs") { + std::println("build programs:"); + const auto graphs = build_programs_json(ctx->projectRoot); + if (graphs.empty()) std::println(" (none)"); + for (auto const& g : graphs) { + std::println(" graph {}", g.value("workspace", std::string{})); + for (auto const& p : g["programs"]) { + std::println(" {} ({})", p.value("package", std::string("?")), + p.value("source", std::string("?"))); + std::string libs; + for (auto const& l : p.value("libraries", nlohmann::json::array())) + libs += (libs.empty() ? "" : ", ") + l.get(); + std::println(" uses: {}", libs.empty() ? std::string("(no library)") : libs); + std::println(" identity {} binary {}", p.value("identity", std::string("?")), + p.value("binary", std::string("?"))); + } + } + } + // The deprecated readings this project and its dependencies use, with + // when each is removed (the deprecation registry, docs/51). + if (all || topic == "compat") { + std::println("deprecated readings:"); + const auto uses = compat_uses_json(); + if (uses.empty()) std::println(" (none)"); + for (auto const& u : uses) + std::println(" {}: {} ({})\n deprecated since {}, removed from {}; write {}{}", + u.value("id", std::string{}), u.value("detail", std::string{}), + u.value("where", std::string{}), u.value("since", std::string{}), + u.value("floor", std::string{}), u.value("canonical", std::string{}), + u.value("warned", false) ? std::string{} + : "\n (in a package this build does not edit: not warned)"); + } if (all) (void)print_stored_runtime_resolution(); // WHERE A NAMED RUNNER BECOMES DISCOVERABLE. // @@ -1413,7 +1520,20 @@ export int why_report(const std::string& topic, const std::string& features, r.value("requester", std::string("?")), r.value("key", std::string("?")), r.value("table", std::string("?"))); - if (auto link = entry.find("link"); link != entry.end()) + // mcpp#797: the target instance is compiled in this plan, the + // build-program instance in the build programs' graph. + if (auto inst = entry.find("instances"); + inst != entry.end() && inst->is_array() && !inst->empty()) { + std::string list; + for (auto const& x : *inst) + list += (list.empty() ? "" : ", ") + x.get(); + std::println(" instances: {}", list); + } + // A package with no target instance is linked into nothing here. + const auto inst = entry.value("instances", nlohmann::json::array()); + const bool linked = !inst.is_array() || inst.empty() + || std::ranges::any_of(inst, [](auto const& x) { return x == "target"; }); + if (auto link = entry.find("link"); linked && link != entry.end()) std::println(" linked {} ({})", link->value("form", std::string("?")), link->value("reason", std::string("?"))); diff --git a/src/wire.cppm b/src/wire.cppm index 9b384ab7..f548923c 100644 --- a/src/wire.cppm +++ b/src/wire.cppm @@ -68,7 +68,7 @@ inline constexpr int kEnvelopeVersion = 1; // from what comes back. struct KindVersion { std::string_view kind; int version; }; -inline constexpr std::array kKinds{{ +inline constexpr std::array kKinds{{ {"mcpp.env", 1}, {"mcpp.xpkg", 1}, {"mcpp.cache", 1}, @@ -81,6 +81,13 @@ inline constexpr std::array kKinds{{ // record of a prepare -- where the toolchain, each payload and each plugin // tool came from, its class and origin -- filtered to the topic. {"mcpp.why.sources", 1}, + // `mcpp why build-programs --format json` (mcpp#797): the build programs' + // graphs of the project, each program's member, libraries, identity and + // binary. + {"mcpp.why.build-programs", 1}, + // `mcpp why compat --format json` (mcpp#797): every use of a deprecated + // reading, with its row of the deprecation registry. + {"mcpp.why.compat", 1}, // `mcpp toolchain list --format json`: which toolchains are installed and // which target rows this host serves, with their status. {"mcpp.toolchain.list", 1}, From 16cc5556bf81b54b5ee15e6b3448a9942c4959f5 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sun, 11 Oct 2026 09:01:15 +0900 Subject: [PATCH 07/12] Tests, docs, specs and examples follow the build programs' graph; deprecations dated by CalVer; 2026.10.11.1 (#797) - e2e 902-913: libraries of a build program, two instances of one package, features per instance, whole libraries, program identity, the library interface, the request of a build dependency, the deprecation registry, build plugins (P1-P5), workspace tables (R4, R5, W1, W2), shared build programs' graph across configuration groups, the build database. - Existing tests read the build programs' graph under target/.build-mcpp/host instead of the removed host module store and its log lines. - The host graph carries the build root's [xlings.overrides] and the project whose .mcpp holds the custom indices' packages; its directory states both. - A root without [package] gives its members [workspace.xlings.overrides], and its schema warnings are reported when its members are built. - A path package is editable wherever it lies; the source digest re-reads a racily clean file; a CalVer month has a four-digit year and a month 1-12. - graph.cpp's lookups move to graph_closures.cpp (file length limit). - Examples 08, 09, 10 and 12 use [build-dependencies] without host-module. - docs (en, zh), SPEC-004/005/007/008, CHANGELOG; version 2026.10.11.1. --- ...-build-dependencies-one-pipeline-design.md | 15 + .github/tools/check_compat_registry.py | 98 ++++++ .github/workflows/ci.yml | 3 + CHANGELOG.md | 92 ++++++ docs/04-mcpp-toml.md | 34 +- docs/05-dependencies.md | 79 +++-- docs/07-workspace.md | 43 ++- docs/09-commands-by-scenario.md | 13 +- docs/20-toolchains.md | 2 +- docs/30-build-mcpp.md | 290 ++++++++---------- docs/31-authoring-a-rule-package.md | 39 ++- docs/50-machine-output.md | 22 +- docs/51-supported-versions.md | 26 +- docs/specs/README.md | 8 +- docs/specs/build-database.md | 24 +- docs/specs/build-plugins.md | 42 ++- docs/specs/library-interface.md | 36 ++- docs/specs/manifest-semantics.md | 59 +++- docs/zh/04-mcpp-toml.md | 22 +- docs/zh/05-dependencies.md | 60 ++-- docs/zh/07-workspace.md | 32 +- docs/zh/09-commands-by-scenario.md | 8 +- docs/zh/20-toolchains.md | 2 +- docs/zh/30-build-mcpp.md | 214 ++++++------- docs/zh/31-authoring-a-rule-package.md | 27 +- docs/zh/50-machine-output.md | 20 +- docs/zh/51-supported-versions.md | 21 +- examples/08-build-rules/README.md | 10 +- examples/08-build-rules/app/mcpp.toml | 9 +- examples/08-build-rules/rules-tidy/mcpp.toml | 12 +- examples/09-heterogeneous/README.md | 2 +- examples/09-heterogeneous/boundary/mcpp.toml | 4 +- examples/09-heterogeneous/cann/app/mcpp.toml | 2 +- examples/09-heterogeneous/cuda/README.md | 2 +- examples/09-heterogeneous/cuda/app/mcpp.toml | 2 +- examples/09-heterogeneous/hip/app/mcpp.toml | 2 +- .../09-heterogeneous/multi-backend/README.md | 9 +- .../09-heterogeneous/multi-backend/mcpp.toml | 8 +- examples/09-heterogeneous/sycl/app/mcpp.toml | 2 +- .../09-heterogeneous/vulkan/app/mcpp.toml | 2 +- examples/10-graphics/README.md | 2 +- examples/10-graphics/offscreen/mcpp.toml | 2 +- examples/12-a-new-device-language/README.md | 9 +- .../12-a-new-device-language/app/mcpp.toml | 8 +- .../rules-toy/mcpp.toml | 4 +- mcpp.toml | 2 +- modules/manifest/src/compat.cppm | 19 +- modules/manifest/src/compat/legacy.cppm | 4 +- modules/manifest/src/compat/registry.cppm | 4 +- .../manifest/tests/test_compat_registry.cpp | 78 +++++ modules/versioning/src/version.cppm | 2 +- src/build/build_database.cppm | 6 +- src/build/prepare.cppm | 16 +- src/build/prepare/driver.cpp | 49 +++ src/build/prepare/features.cpp | 66 ++++ src/build/prepare/graph.cpp | 195 ------------ src/build/prepare/graph_closures.cpp | 196 ++++++++++++ src/build/prepare/graph_load.cpp | 12 +- src/build/prepare/host_graph.cpp | 78 ++++- src/build/prepare/manifest.cpp | 6 + src/build/prepare/scan.cpp | 54 +--- src/build/prepare/sources.cpp | 10 +- src/build/prepare/state.cppm | 15 +- src/build/prepare/toolchain.cpp | 3 + src/build/prepare/xlings.cpp | 14 +- src/build/source_digest.cppm | 20 +- src/fallback/legacy_dirs.cppm | 2 +- src/modgraph/validate.cppm | 7 +- src/pm/package_fetcher.cppm | 8 +- src/pm/resolver.cppm | 4 +- src/project.cppm | 3 + tests/e2e/168_build_mcpp_musl_host_static.sh | 3 +- ...ed_build_program_hides_no_device_source.sh | 2 +- ...r_several_members_plans_and_builds_once.sh | 4 +- ...ngines_module_is_compiled_once_per_home.sh | 79 +++-- ...paces_programs_compile_at_the_same_time.sh | 29 +- ...compiles_plan_what_serial_compiles_plan.sh | 2 +- ...ers_plans_builds_and_runs_programs_once.sh | 4 +- ..._is_refused_before_anything_is_compiled.sh | 6 +- ...figuration_has_its_own_generated_output.sh | 2 +- ...ency_library_is_compiled_as_its_package.sh | 81 +++++ ...ackage_in_both_tables_has_two_instances.sh | 138 +++++++++ ...endency_feature_stays_out_of_the_target.sh | 44 +++ ...build_program_links_its_libraries_whole.sh | 35 +++ ...gram_is_compiled_when_its_graph_changes.sh | 44 +++ ...07_the_library_interface_is_stated_once.sh | 64 ++++ .../908_what_a_build_dependency_asks_for.sh | 53 ++++ ...ecated_readings_are_registered_and_kept.sh | 45 +++ ...lugin_is_a_package_of_the_build_program.sh | 98 ++++++ .../911_a_workspace_states_each_table_once.sh | 90 ++++++ ...n_groups_share_the_build_programs_graph.sh | 47 +++ ...d_database_describes_the_build_programs.sh | 76 +++++ tests/unit/test_library_interface.cpp | 112 +++++++ tests/unit/test_manifest.cpp | 32 ++ tests/unit/test_provisions.cpp | 33 +- tests/unit/test_source_digest.cpp | 70 +++++ 96 files changed, 2654 insertions(+), 834 deletions(-) create mode 100644 .github/tools/check_compat_registry.py create mode 100644 modules/manifest/tests/test_compat_registry.cpp create mode 100644 src/build/prepare/graph_closures.cpp create mode 100755 tests/e2e/902_a_build_dependency_library_is_compiled_as_its_package.sh create mode 100755 tests/e2e/903_one_package_in_both_tables_has_two_instances.sh create mode 100755 tests/e2e/904_a_build_dependency_feature_stays_out_of_the_target.sh create mode 100755 tests/e2e/905_a_build_program_links_its_libraries_whole.sh create mode 100755 tests/e2e/906_a_build_program_is_compiled_when_its_graph_changes.sh create mode 100755 tests/e2e/907_the_library_interface_is_stated_once.sh create mode 100755 tests/e2e/908_what_a_build_dependency_asks_for.sh create mode 100755 tests/e2e/909_deprecated_readings_are_registered_and_kept.sh create mode 100755 tests/e2e/910_a_build_plugin_is_a_package_of_the_build_program.sh create mode 100755 tests/e2e/911_a_workspace_states_each_table_once.sh create mode 100755 tests/e2e/912_two_configuration_groups_share_the_build_programs_graph.sh create mode 100755 tests/e2e/913_the_build_database_describes_the_build_programs.sh create mode 100644 tests/unit/test_library_interface.cpp create mode 100644 tests/unit/test_source_digest.cpp diff --git a/.agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md b/.agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md index 37386684..4271d040 100644 --- a/.agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md +++ b/.agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md @@ -727,3 +727,18 @@ mcpp-index、mcpp-plugins 各一个 PR（发布之后）；mcppls 一个 issue 关键路径是 T3 → T4 → T7 → T8/T9/T10 → T11。T1、T2、T5、T6 与关键路径并行。 宿主图的合成工作空间**写到磁盘**（修订 3 写作"不落盘"）：它由引擎生成，落盘之后可以在该目录直接运行 `mcpp build` 复现宿主图的构建，这是可观测性要求的一部分；内容不变时不重写，ninja 保持增量。 + +## 21. 实施记录（实现中确定的细节） + +| 项 | 结论 | +|---|---| +| 宿主图目录的自描述 | 目录中写 `.mcpp-build-programs`（`toolchain=`、`profile=`、`pin=`、`index-store=`），`prepare_build` 在其所在目录读取它，使 `cd

&& mcpp build` 与子构建得到相同的覆盖项 | +| 宿主图的 `[xlings.overrides]` | 构建根生效的 overrides 写入合成工作空间的 `[workspace.xlings.overrides]`（相对路径改为绝对路径），子构建以同一来源声明供给宿主图中的包所声明的载荷（e2e 873） | +| 项目级 index 存储 | 自定义 index 的包装在发起构建的项目的 `.mcpp/` 中；子构建以 `BuildOverrides::project_index_root` 指向它，不在宿主图目录重新安装（e2e 858 E） | +| `plugin-imports-from-build-dependencies` | 判据是两包的源码（插件库 import 的模块由 `[build-dependencies]` 中 `host-module = true` 的包声明），在 phase 6 的 `step6_compat_library_edges` 中改写为普通边并写入清单的 `[dependencies]`，使链接闭包包含它；与读取它的图无关 | +| 可编辑的包 | 兼容读法的警告给"不在包存储（index 存储、`$MCPP_HOME`）中的包"，`path` 依赖无论位置都算可编辑（与 Cargo 对 lint 的划分一致） | +| 失败报告的确定性 | 一次构建的全部程序先在宿主图中一次编译；失败时各程序按运行顺序单独请求自己的二进制，报告的是运行顺序中第一个失败的程序（e2e 860 C） | +| 虚拟根的 overrides | `virtual_workspace_root` 原本没有从首个成员复制 `xlings.overrides`，`[workspace.xlings.overrides]` 在不带 `[package]` 的根上不生效；本 PR 修正（e2e 911 V） | +| 工作空间根的 schema 警告 | 不带 `[package]` 的根在成员构建时其 `schemaWarnings`（含层表与 W2 的请求键）此前不报告；`report_workspace_root` 报告它们 | +| 文件长度 | `graph.cpp` 的查询闭包拆到 `graph_closures.cpp`（P4b 的阶段边界） | +| 本机限制 | 代理会话运行在 xlings 沙箱 subos 中，`xlings self init` 被拒绝，依赖全新 `MCPP_HOME` 的 e2e（618、622、625、627–630、657、669、686、737、779、790、884 等）在本机与基线二进制同样失败；它们由 CI 验证 | diff --git a/.github/tools/check_compat_registry.py b/.github/tools/check_compat_registry.py new file mode 100644 index 00000000..4d253bd3 --- /dev/null +++ b/.github/tools/check_compat_registry.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""The deprecation registry, the code and docs/51 agree (mcpp#797). + +modules/manifest/src/compat/registry.cppm is the one list of readings a user +can write and that are to be removed. Three things are derived from it and can +drift from it, so each is checked here: + + 1. Every `COMPAT()` marker in the code names a row, and every row has at + least one marker -- the reading is somewhere, and it is registered. + `COMPAT(internal:)` names engine-internal compatibility that no + manifest can spell; it has no row and is not checked. + 2. docs/51 (English and Chinese) lists every row with its `since` and its + `floor`. + 3. Each row's fields are present. + +The removal itself -- that no reading outlives its floor -- is the unit test +modules/manifest/tests/test_compat_registry.cpp. +""" +import pathlib +import re +import sys + +ROOT = pathlib.Path(__file__).resolve().parents[2] +REGISTRY = ROOT / "modules/manifest/src/compat/registry.cppm" +DOCS = [ROOT / "docs/51-supported-versions.md", ROOT / "docs/zh/51-supported-versions.md"] + + +def rows(): + text = REGISTRY.read_text(encoding="utf-8") + floor = re.search(r'kRemovalFloor\s*=\s*"([^"]+)"', text).group(1) + body = text[text.index("kDeprecations[] = {"):] + body = body[:body.index("\n};")] + out = [] + for block in re.findall(r"\{(.*?)\n \}", body, re.S): + fields = re.findall(r'"((?:[^"\\]|\\.)*)"|(kRemovalFloor)', block) + values = [] + pending = "" + # Adjacent string literals concatenate, as in C++. + tokens = re.findall(r'"((?:[^"\\]|\\.)*)"\s*|(kRemovalFloor)|(,)', block) + for lit, name, comma in tokens: + if comma: + values.append(pending) + pending = "" + elif name: + pending = floor + else: + pending += lit + if pending: + values.append(pending) + values = [v for v in values] + out.append(values) + return out + + +def main() -> int: + failures = [] + registered = {} + for r in rows(): + if len(r) < 7 or not all(r[:7]): + failures.append(f"registry: a row has an empty field: {r}") + continue + registered[r[0]] = (r[1], r[2]) + + markers = {} + for path in list(ROOT.glob("src/**/*.cpp*")) + list(ROOT.glob("modules/**/*.cpp*")): + text = path.read_text(encoding="utf-8", errors="replace") + for m in re.finditer(r"COMPAT\(([a-z0-9:<>-]+)\)", text): + ident = m.group(1) + if ident.startswith("internal:") or ident.startswith("<"): + continue + markers.setdefault(ident, []).append(str(path.relative_to(ROOT))) + for ident, where in sorted(markers.items()): + if ident not in registered: + failures.append(f"COMPAT({ident}) in {where[0]} names no row of {REGISTRY.relative_to(ROOT)}") + for ident in sorted(registered): + if ident not in markers: + failures.append(f"the row '{ident}' has no COMPAT({ident}) marker in the code") + + for doc in DOCS: + text = doc.read_text(encoding="utf-8") + for ident, (since, floor) in sorted(registered.items()): + line = next((l for l in text.splitlines() if f"`{ident}`" in l), None) + if line is None: + failures.append(f"{doc.relative_to(ROOT)} does not list `{ident}`") + elif since not in line or floor not in line: + failures.append(f"{doc.relative_to(ROOT)}: the row of `{ident}` does not state " + f"{since} and {floor}") + + for f in failures: + print(f"FAIL: {f}") + if failures: + return 1 + print(f"ok: {len(registered)} deprecations, their markers and docs/51 agree") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a61cf442..d6aca258 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -134,6 +134,9 @@ jobs: - name: The key registry and the field reference agree run: python3 .github/tools/check_manifest_key_classes.py + - name: The deprecation registry, the code and docs/51 agree + run: python3 .github/tools/check_compat_registry.py + - name: Documented target tiers agree with the table run: python3 .github/tools/check_target_tiers.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c67a736..3802d5e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,98 @@ > Each `## []` section is that release's notes. Entries are written in English > from 2026.9.28.3 on; earlier entries remain as written. +## [2026.10.11.1] - 2026-10-11 + +### Added + +- Compile build programs by their own graph (#797). The build programs of one + build that share a host toolchain and a C++ standard are the members of one + workspace under `target/.build-mcpp/host//`, planned and built like + any workspace: a library is compiled as its package, with its include + directories, defines, features, dependencies and implementation units, and + `build.mcpp` by the same plan. `mcpp build` in that directory reproduces the + compile. +- Give a `[build-dependencies]` entry the package's library unless it names + `tools` or `artifacts` only; `host-module = false` declines it. Any library + is importable from `build.mcpp`. +- Plan a package once per configuration that reaches it: a package in + `[dependencies]` and `[build-dependencies]` has a target instance and a + build-program instance, each with the features its own edges request, and a + build-only package's usage requirements and link flags stay out of the + target. +- Write the engine interface (`mcpp`, `mcpp.core`) as a package under + `$MCPP_HOME/engine//modules/`, compiled into the global dependency + cache once per version and host toolchain. `mcpp self env` states the + directory as `engineModules`. +- Read a unit that imports `mcpp.core` as a build-plugin unit (SPEC-007 §10). A + plugin's `[dependencies]` are compiled with it for the build program, so a + plugin may build on plugins and libraries; `mcpp build` and `mcpp test` in a + plugin plan it in the build-program configuration. A plugin reached through + a target's `[dependencies]`, a program target that imports `mcpp`, `--target` + for another machine on a plugin, and `mcpp pack` of a plugin are refused, + naming the cause. +- Define a library's interface once (SPEC-008 v0.2): the interface root (`[lib] + path`, the new `lib` key of a Form B descriptor, or `src/.cppm`), the + public modules it re-exports together with the interface units of active + features, and the exported modules. `[modules] exports` is checked against + the public modules, and a feature's `rule_module` must be declared by one of + its units. +- Describe the build programs in `mcpp emit build-database`: the sets + `:build`, `:host`, `mcpp:core` and `mcpp:std:host`, in the + configuration `build-program`. `compile_commands.json` is unchanged. +- Add `mcpp why build-programs` and `mcpp why compat` (JSON kinds + `mcpp.why.build-programs` and `mcpp.why.compat`). `resolution.json` and + `mcpp why deps` state each package's instances. +- Keep every deprecated reading in one registry with its id, the release that + deprecated it and the CalVer month from which it is removed. The warning + states both releases, `mcpp why compat` lists the uses, `--strict` refuses + them, and docs/51 lists the rows. + +### Changed + +- Compile a build program again when its identity changes: the engine version, + the host toolchain, the standard, the program text and every package of its + graph, including the source tree of a `path` package or workspace member. An + edited header that a library includes compiles and runs the program again. +- Remove the `workspace-position` reading from mcpp 2027.1 instead of 1.0.0, + which CalVer does not reach. +- Give `[xlings.overrides]` and `[workspace.xlings.overrides]` to the root of a + build only; a member reached as a dependency does not receive them. A member + reached as a dependency is told about a position reading only when the + build's root lies outside its workspace. +- Pass `features` and `default-features` of a `[workspace.dependencies]` entry + to the members that inherit it, and report a request key written there + (`host-module`, `tools`, `artifacts`, `reexport`, `visibility`, `linkage`). + +### Fixed + +- Import from `build.mcpp` a module library whose interface includes the + package's own headers (#797). +- Keep the target's library of a package listed in both `[dependencies]` and + `[build-dependencies]` (#797). +- Keep a feature requested only by a build edge out of the target. +- Build a build plugin as the root of its own build; the engine interface was + absent. +- Give the members of a workspace root without `[package]` its + `[workspace.xlings.overrides]`. +- Report a `build.mcpp` beside a workspace root without `[package]`, and the + schema warnings of a workspace root when its members are built. + +### Deprecated + +- `host-module-in-dependencies`: a `[dependencies]` entry that asks for + `host-module = true` or for a feature declaring `rule_module` is read as a + build dependency and warned about. Removed from mcpp 2027.1. +- `plugin-imports-from-build-dependencies`: a plugin library that imports a + package from its own `[build-dependencies]` reads that entry as a dependency + too. Removed from mcpp 2027.1. + +### Removed + +- The single-unit host module compile and its store + `target/.build-mcpp/host-modules/`. Build programs are compiled by their + graph. + ## [2026.10.10.2] - 2026-10-10 ### Fixed diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index ed0edb8c..88a2413d 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -1129,9 +1129,25 @@ path = "src/capi/lua.cppm" # Override the default lib-root location Default convention: `src/.cppm` (e.g. package name `mcpplibs.cmdline` → `src/cmdline.cppm`). `path` is the only key of the table; any other key is reported, as in `[build]` -(mcpp 2026.9.28.3+). The lib root is the interface root of SPEC-008: the module a -packed library publishes, with what it re-exports, and the module a build -program imports from a host-module dependency. +(mcpp 2026.9.28.3+). The lib root is the interface root of SPEC-008, and it states +what the library offers to every consumer alike (mcpp 2026.10.11.1+): its module +and what that module re-exports with `export import`, together with the +interface units an enabled feature adds, are the **public modules**. A target that +imports the library, a build program that uses it from `[build-dependencies]`, +and `mcpp pack` read the same set; a module that is exported and not public draws +a warning (SPEC-008 W3). A package without a lib root states no interface, and +every module it exports is public. + +A path that `[lib] path` states and that names no file is an error in the +package's own build and a warning where the package is a dependency. A Form B +descriptor states the same thing as `lib = ""` beside `sources`, relative to +the version directory, `*/` standing for the archive's wrapping directory. + +`[modules] exports` (a Form B descriptor's `modules = { ... }`) lists the public +modules as an assertion: in the package's own build, a public module it does not +list is an error, and with `[modules] strict = true` so is a listed name that is +not public. It does not choose the interface root. + ### 2.5 `[dependencies]`, `[dev-dependencies]`, `[build-dependencies]` Moved to [05 — Dependencies and Resolution](05-dependencies.md). @@ -1481,12 +1497,12 @@ What is **not** cached: `path` and `git` dependencies, at any depth, and workspace members. Their sources can change without their `name@version` changing, so no key over that identity could notice the change. -The modules a build program imports follow the same rule (2026.10.1.1+). The -bundled `mcpp` module and the host modules of index packages are kept in the -global cache, and a host module of a `path` or `git` dependency or of a workspace -member is kept under `target/.build-mcpp/host-modules/`, never in the global -cache. `--cache local` and `--cache off` keep all of them in the project. See -[30 — Placement of compiled host modules](30-build-mcpp.md). +The libraries a build program imports follow the same rule (2026.10.11.1+). The +engine interface (`mcpp.core`) and index packages are kept in the global cache; +a `path` or `git` package or a workspace member is compiled in the build +programs' graph under `target/.build-mcpp/host/`, never in the global cache. +`--cache local` and `--cache off` keep all of them in the project. See +[30 — The build programs' graph](30-build-mcpp.md). Inspection and reclamation: diff --git a/docs/05-dependencies.md b/docs/05-dependencies.md index a3bdfb54..42cece9e 100644 --- a/docs/05-dependencies.md +++ b/docs/05-dependencies.md @@ -402,36 +402,53 @@ gtest = "1.15.2" ```toml [build-dependencies] -protobuf = { version = "35.1", tools = ["protoc"] } +tomlplusplus = "3.4.0" # a library build.mcpp imports +protobuf = { version = "35.1", tools = ["protoc"] } # a program the build runs ``` -The section and the per-edge request answer **different questions**, and -conflating them is a modelling error rather than a matter of spelling. - -- The **section** says whether the package itself reaches the target. - `[dependencies]` means it does; `[build-dependencies]` means it never does, - and neither does anything reachable only through it. -- The **request on the edge** says which build-time product is wanted. - `tools = [...]` asks for a host executable; `host-module = true` asks for a - module the build program can import. - -A package takes a value on both axes at once, and protobuf is the case that -proves the axes must stay separate: a project links `libprotobuf` *and* needs -`protoc` during the build. It is written once, in `[dependencies]`: - -```toml -[dependencies] -protobuf = { version = "35.1", tools = ["protoc"] } -``` - -`[build-dependencies]` is for the combination the first axis cannot otherwise -express — a package whose library must not reach the target while its tool or -its rule is still wanted. Naming one package in both tables is not an error: -the ordinary declaration wins, because a `[build-dependencies]` line must not -quietly drop a library the target needs, and what each declaration requests -(`tools`, `features`, `host-module`, `reexport`) is requested of the one edge -(mcpp 2026.9.16.1+; before that release the second declaration's requests were -dropped). +A dependency is described on two axes (mcpp 2026.10.11.1+, SPEC-004 §12), and +each axis has its own place in the manifest. + +- The **section** decides the configuration the dependency serves. + `[dependencies]` and `[dev-dependencies]` serve the target: the package is + compiled for it and linked into the project. `[build-dependencies]` serves the + project's build program (`build.mcpp`): the package is compiled for the machine + that runs the build and linked into that program. A configuration is inherited + along `[dependencies]`, so the dependencies of a build dependency serve the + build program as well. +- The **request on the edge** decides which products are wanted. A + `[build-dependencies]` entry asks for the package's library unless it lists + `tools` or `artifacts`; `host-module = true` asks for the library beside them, + and `host-module = false` declines it. `tools = [...]` asks for programs built + for the machine that runs the build, from either section. + +| Entry | Effect | +|---|---| +| `[build-dependencies] x = "1.0"` | `build.mcpp` imports the public modules of `x`, includes its public headers and links its objects | +| `[build-dependencies] x = { tools = ["gen"] }` | the program `gen`; no library | +| `[build-dependencies] x = { tools = ["gen"], host-module = true }` | the program and the library | +| `[build-dependencies] x = { host-module = false }` | neither; the directory of `x` (`mcpp::dep_dir`) and its payloads | +| `[dependencies] x = { tools = ["gen"] }` | the library in the target, and the program | + +The name `host-module` is older than its meaning: it was introduced when a build +program received the lib-root module of a dependency and nothing else, and a key +that has reached published descriptors is not renamed (SPEC-004 §5.2). A library +used by a build program is compiled as the package it is: its include +directories, defines, features, dependencies and implementation units take part, +and the build program is compiled by the same plan (30 — `build.mcpp`). + +**One package in both tables.** A package written in `[dependencies]` and in +`[build-dependencies]` of one manifest has two instances, one compiled for the +target and one for the build program. Both have the version the lock records. +Their features are activated separately: a feature requested only by a +`[build-dependencies]` edge does not reach the target instance. + +**A reading kept for compatibility.** A `[dependencies]` entry that asks for +`host-module = true`, or for a feature that declares `rule_module`, is read as a +`[build-dependencies]` entry: the package serves the build program and not the +target, as it did before mcpp 2026.10.11.1. The reading is deprecated since +mcpp 2026.10.11.1 and removed from mcpp 2027.1 (`host-module-in-dependencies`, +51 — Supported Versions); `mcpp why compat` lists its uses. **A package of programs contributes only its programs (mcpp 2026.9.16.1+).** A dependency whose declared `[targets]` are all programs (`bin`, `app`, `test`) @@ -448,9 +465,9 @@ A cycle among packages is refused where the graph is resolved, naming its edges, under every cache mode; a tool whose own sub-build requests it again is refused at that first repetition, naming the chain. -Unlike `[dev-dependencies]`, these **are** walked transitively: a build -dependency's own dependencies are what make it work, and they inherit its -build-only nature. +Unlike `[dev-dependencies]`, build dependencies **are** walked transitively and +resolved with the project, one version per package: a build dependency's own +dependencies are what make it work, and they serve the build program with it. A feature scopes build-time requests without a second declaration site. A `[feature-deps.]` entry may restate a dependency already declared diff --git a/docs/07-workspace.md b/docs/07-workspace.md index e1f26925..be3eac54 100644 --- a/docs/07-workspace.md +++ b/docs/07-workspace.md @@ -126,6 +126,24 @@ mbedtls.workspace = true # inherits version → "3.6.1" gtest.workspace = true # inherits version → "1.15.2" ``` +The features an entry of `[workspace.dependencies]` requests are inherited +with its source, ahead of the member's own; `default-features` is the +member's when it states one (mcpp 2026.10.11.1+). The other requests of an +edge -- `host-module`, `tools`, `artifacts`, `reexport`, `visibility`, +`linkage` -- belong to the member's own entry, because one workspace entry may +be inherited into `[dependencies]` and into `[build-dependencies]`, where they +mean different things; written in `[workspace.dependencies]` each is warned +about, and `--strict` refuses it. + +```toml +[workspace.dependencies] +hl = { path = "libs/hl", features = ["x"] } + +# member mcpp.toml +[dependencies] +hl = { workspace = true, features = ["y"] } # requests x and y +``` + A member can override an inherited version: ```toml @@ -208,7 +226,12 @@ them where it is the root of a build: built from the workspace, with `-p`, or as a host tool of another package. A member reached as a dependency takes them from that build's root. The rows of `[workspace.target.]` -- `.build`, `.abi`, `.runtime`, `.xlings.workspace` -- and `[workspace.xlings]` -reach a member in every position (§4.1). +reach a member in every position (§4.1). `[workspace.xlings.overrides]` -- +where a payload comes from on this machine -- is a table of the build's root, +like `[workspace.toolchain]`: a member takes it where it is the root of a build, +and a member reached as a dependency does not (mcpp 2026.10.11.1+; before that +release a member reached as a dependency received it and was refused for +stating an override). A conditional row counts as it evaluates for the target being built: a `dialect_cxxflags` or `.abi` row for another target does not separate a member @@ -235,14 +258,19 @@ name replaces the root's whole, and `[indices]` reaches only a member that declares none. Each use prints a `manifest/workspace-position` warning with the `[workspace.X]` spelling; on a root with `[package]` the tables are the root package's own and the warning appears only for a member that received a value -from them. This reading is removed in mcpp 1.0.0. On a root without +from them, and for a member reached as a dependency only when the build's root +lies outside the workspace (the root's own build already holds the entries). +The reading is deprecated since mcpp 2026.10.10.1 and removed from mcpp 2027.1 +(`workspace-position`, 51 — Supported Versions). On a root without `[package]`, writing a table both ways is refused. **A root without `[package]` holds no package.** `[build]`, `[targets]`, `[dependencies]`, `[features]`, `[resources]`, `[test]`, `[hooks]`, `[runtime]` and a selector's `.build`, `.abi`, `.runtime`, `.targets` and `.dependencies` written there act on nothing; each prints a warning naming the shared spelling -where there is one, and `--strict` refuses. +where there is one, and `--strict` refuses. A `build.mcpp` beside such a root +belongs to no package and does not run; it is reported the same way +(mcpp 2026.10.11.1+). ### 4.1 `[workspace.package]` and `[workspace.build]` @@ -352,7 +380,7 @@ two kinds of tables: | Tables | Owner | Effect on the root package | Effect on other members | |---|---|---|---| | `[workspace]` and every `[workspace.X]` | the workspace | received, its own declarations first | received; `[workspace.dependencies]` through `x.workspace = true` | -| every other table (`[package]`, `[build]`, `[toolchain]`, `[target.]`, `[profile.]`, ...) | the root package | its own | none (the position reading above excepted, until 1.0.0) | +| every other table (`[package]`, `[build]`, `[toolchain]`, `[target.]`, `[profile.]`, ...) | the root package | its own | none (the position reading above excepted, until mcpp 2027.1) | The root package's `[build] ldflags` are its own as well: they reach its own images, and neither the members' images nor a dependency's shared library. @@ -678,9 +706,10 @@ myproject/ - `mcpp.lock` at the workspace root records the resolution of every member. `mcpp build --workspace` writes the whole record; `mcpp build -p X` updates the entries of X's graph. -- A member's build program writes to `/target/.build-mcpp/`. The host - modules the programs import are compiled into the workspace's own - `target/.build-mcpp/host-modules/`, once for all of them. +- A member's build program writes to `/target/.build-mcpp/`. The + members' build programs are compiled in the workspace's build programs' + graph, `target/.build-mcpp/host/`, and a library several of them import is + compiled once. - Build directories a member held under its own `target/` with an earlier mcpp are not read; `mcpp clean --stale` removes them. diff --git a/docs/09-commands-by-scenario.md b/docs/09-commands-by-scenario.md index 6e4c1040..a0058963 100644 --- a/docs/09-commands-by-scenario.md +++ b/docs/09-commands-by-scenario.md @@ -168,8 +168,15 @@ dependency graph: requested by mcpplibs.app@0.1.0 as 'huxdemo.fw' in [dependencies] requested by huxdemo.comp@0.1.0 as 'fw' in [dependencies] linked static (default) + instances: target ``` +`instances` states whether the package is compiled for the target, for the +build programs, or for both (2026.10.11.1+). `mcpp why build-programs` lists the +build programs' graphs: each program, the libraries it uses, the identity it +was compiled from and its binary. `mcpp why compat` lists every use of a +deprecated reading with the release that removes it (51 — Supported Versions). + The same graph is recorded under `graph` in `target///resolution.json`, one entry per package with the root first: `package` (canonical identity, namespace, name, version, source), @@ -682,9 +689,9 @@ mcpp is involved and its documentation will not help. ## Current limitations -- `mcpp why --format json` is defined for the `toolchain`, `sources`, `tool` and - `payload` topics. The other topics report `'' has no machine-readable - shape yet` and exit non-zero. +- `mcpp why --format json` is defined for the `toolchain`, `sources`, `tool`, + `payload`, `build-programs` and `compat` topics. The other topics report + `'' has no machine-readable shape yet` and exit non-zero. - `mcpp search` matches a substring; there is no field selector, and no way to restrict a search to one namespace. - `mcpp clean --stale` reads `target/.build_cache`, which holds a bounded number diff --git a/docs/20-toolchains.md b/docs/20-toolchains.md index cde7604b..a7e763f3 100644 --- a/docs/20-toolchains.md +++ b/docs/20-toolchains.md @@ -462,7 +462,7 @@ default = { path = "/opt/llvm-trunk" } bootstrap = "llvm@23.1.3" ``` -Build programs (`build.mcpp`), host tools and host modules are compiled and run +Build programs (`build.mcpp`), their libraries and host tools are compiled and run on the machine doing the build. `bootstrap` names the toolchain that does that when it should not be the one building the project — a toolchain under development, say, which has to be able to fail without stopping the program that diff --git a/docs/30-build-mcpp.md b/docs/30-build-mcpp.md index 4dc39e12..1aa904ae 100644 --- a/docs/30-build-mcpp.md +++ b/docs/30-build-mcpp.md @@ -144,9 +144,9 @@ interface and belongs in the declarative manifest/descriptor Instead of printing raw strings, `build.mcpp` can be written **modules-first** — `import mcpp;`, no `#include` needed. The `mcpp` module is bundled in the -mcpp binary (so it always matches that mcpp's protocol) and is compiled on demand, -once per mcpp version and host compiler, into the build cache (see *Where compiled -host modules are kept* below); its functions just emit the directives above: +mcpp binary (so it always matches that mcpp's protocol), written once per mcpp +version as a package and compiled like any library of the build program (see +*The build programs' graph* below); its functions just emit the directives above: ```cpp // build.mcpp @@ -295,7 +295,7 @@ neither one's. **Within one build program, the tokens are one argv per name.** Every `mcpp::runner(tok)` call appends to the default runner, and every `mcpp::runner(name, tok)` call appends to the runner `name`, in emission order, -whichever host module compiled into the program makes the call. A program that +whichever library of the program makes the call. A program that chooses between two runners chooses before it emits either. A runner named after a pack format reaches that format's distributable under `mcpp run --format ` ([41 — Reaching a Device](41-devices.md)). @@ -1407,9 +1407,9 @@ and when to want one; `examples/07-project-subos/` is a working project. A rule — "run protoc over these `.proto` files", "run clang-tidy over these sources" — belongs in a package, not copy-pasted into every consumer's -`build.mcpp`. The mechanism is -[`host-module = true`](04-mcpp-toml.md); this section is about the shape of -what goes inside. +`build.mcpp`. The mechanism is a library in the consumer's +`[build-dependencies]` (*Libraries of a build program*); this section is about +the shape of what goes inside. The guidance below generalises from `mcpplibs.grpcgen`, the first such package, with each of its traits judged individually. It is guidance and not a rule @@ -1443,24 +1443,23 @@ this build names no backend it serves — a build program with several rules calls them all. **An import nothing provides is refused by name.** A build program may import -`std`, `std.compat`, the bundled `mcpp`, and the host modules its dependency -edges asked for. Anything else is refused before the compiler is reached, with -the key that would have made it importable: +`std`, `std.compat`, the engine interface `mcpp`, and the public modules of the +libraries its package uses. Anything else is refused before the compiler is +reached, with the entry that would make it importable: - error: build.mcpp imports 'mcpp.rules.spirv', and no dependency provides it - as a host module. + error: build.mcpp imports 'mcpp.rules.spirv', and no build dependency provides it. ... [build-dependencies.] - = { version = "...", host-module = true } - declared without `host-module = true`: mcpp.plugins (in [build-dependencies]) + = { version = "..." } # or `host-module = true` beside `tools` + declared without its library: mcpp.plugins (in [build-dependencies]) **The module name is declared by the rule's source, and `mcpp.*` is reserved.** -A host module is registered under the name its interface unit declares, not +A module is imported under the name its interface unit declares, not under the package name, so `export module mcpp.rules.spirv;` is what a consumer then imports. Official plugins live in one package, `mcpp:plugins` (repository `mcpp-community/mcpp-plugins`): rule packages are named `mcpp.rules.`, build-time utilities `mcpp.tools.`, and each member is selected by a feature -of that package (see [`host-module = true`](04-mcpp-toml.md)). `mcpp.build.*` +of that package (*Several modules behind features*). `mcpp.build.*` is the engine's own module family and is not used for plugins. The engine cannot tell who is official, so it keys the check on the package *namespace* and warns when the two disagree — @@ -1705,109 +1704,125 @@ Qt's `QT_HOST_PATH`), and for the same reason: a tool that cannot be built from source on this machine must not be a dead end. It is deliberately **not** part of the cache key — an override is an escape hatch, not a reproducible input. -### `host-module = true` — reusable build rules as packages +### Libraries of a build program: `[build-dependencies]` (mcpp 2026.10.11.1+) -A rule (say "run protoc over these `.proto` files") should be written once, not -copy-pasted into every consumer's `build.mcpp`. Ship it as an ordinary mcpp -library package and import it: +A rule (say "run protoc over these `.proto` files") is written once, as an +ordinary mcpp library package, and so is any library a build program needs — +a TOML parser, a formatter. A build program uses a library its package names +in `[build-dependencies]`: ```toml -[dependencies] -protobufgen = { version = "0.1.0", host-module = true } +[build-dependencies] +tomlplusplus = "3.4.0" +protobufgen = { version = "0.1.0" } ``` ```cpp // build.mcpp +import std; import mcpp; +import tomlplusplus; import protobufgen; -int main() { return protobufgen::generate({"schema"}) ? 0 : 1; } +int main() { + auto cfg = toml::parse_file("codegen.toml"); + return protobufgen::generate({cfg["schema"].value_or("schema")}) ? 0 : 1; +} ``` -mcpp compiles that package's lib-root module **for the host, with the same -flags as `build.mcpp`** — which is what makes the BMI usable at all, since a -module interface is only importable by a compile that agrees with it on -standard, dialect and compiler identity. The compiled module is kept by key, so -it is compiled once for every program that agrees on those flags (see -*Placement of compiled host modules*). +The entry asks for the library unless it lists `tools` or `artifacts`; +`host-module = true` asks for it beside them, and `host-module = false` +declines it (05 — Dependencies and Resolution). A library a dependency hands on +with `reexport = true` is used the same way. + +#### The build programs' graph + +A build program is a program of the build-program configuration. Its main file +is `build.mcpp`, and its dependencies are the libraries above. The build +programs of one build that share a host toolchain and a C++ standard are the +members of one workspace, written to + + /target/.build-mcpp/host// + +and planned and built by mcpp like any workspace: each library is compiled as +the package it is — its include directories, defines, features, its own +dependencies and its implementation units — and `build.mcpp` is compiled by +the same plan, so a BMI and the program importing it agree on standard, +dialect and compiler by construction. A library several programs use is +compiled once for them, and a program of another standard is in another graph. +`mcpp build` in that directory reproduces the compile of the build programs. + +A program is compiled when its **identity** changes: the engine version, the +host toolchain, the standard, the program's text, and every package of its +graph — name, version, source, the features of its build-program instance, and +the content of a source tree that can change in place (a `path` package or a +workspace member). An edit to a header a library includes therefore compiles +the program again and runs it again. When the identity of every program stands, +the graph is not planned. A program's run is cached on the identity and on its +declared inputs (*Incremental: declared inputs*). + +Where the compiled packages are kept follows the dependency cache's rule +([04 §2.10](04-mcpp-toml.md)): + +| Package | Kept in | Reason | +|---|---|---| +| the engine interface, `mcpp` and its `mcpp.core` alias | the global cache: `$MCPP_HOME/build-cache/v1/pkg/_engine/mcpp-build-module@//` | its text is the engine's, written once per version to `$MCPP_HOME/engine//modules/` | +| an index package whose sources are in the immutable store | the global cache: `…/pkg//@//` | a name and version identify the bytes | +| a `path` or `git` package, or a workspace member | the build programs' graph under `/target/` | its sources can change without its name and version changing | -Rules are therefore versioned, testable and distributable through the package -manager already in use, written in **C++** — no second language, which is the -whole point of `build.mcpp` existing. +`mcpp why build-programs` lists each graph, its programs, the libraries each +uses, the identity it was compiled from and its binary; `--verbose` states, +for each program, whether its identity stood. `mcpp emit build-database` +describes the graph to an editor (SPEC-005: the sets `:build`, +`:host` and `mcpp:core`). -**The module name is what the rule's source declares** (mcpp 2026.8.29.1+). +#### Module names + +**The module name is what the library's source declares** (mcpp 2026.8.29.1+). `export module acme.rules.protobuf;` is imported as `acme.rules.protobuf`, -whatever the package is called. Module names are authored API and do not mirror -package identity — the rule ordinary library packages have always followed. - -Until 2026.8.29.1 the host-module path registered the bare `package.name` -instead, which made a divergent name build under GCC and fail under Clang and -MSVC: GCC's BMIs are implicit under `gcm.cache` and keyed by the declared name, -while the other two are handed an explicit `=` mapping. Package names -carry no C++ naming constraint as a result, and `grpc-rules` is a legal package -name again. - -**Two rules may not declare one module name.** `import` addresses the module, -so two such packages are indistinguishable to the compiler, and their BMIs and -objects share a filename — the second overwrites the first and the surviving -object reaches the link twice. mcpp refuses this, naming both packages and both -interface paths. The check covers the rules one `build.mcpp` can see; it is not -an index-wide uniqueness rule, which `path` dependencies and private registries -would escape anyway. - -**`mcpp.` is reserved for rules maintained by the mcpp project.** A module name -under that prefix from a package outside the `mcpp` namespace produces a -warning naming both, and the build proceeds. It is a warning because the engine -cannot decide who is official: a `path` dependency, a private mirror and an -internal fork are all legitimate and indistinguishable from here. - -The lib root must be at `src/.cppm` (or wherever `[lib] path` points); a -missing one is reported as *"host module 'x': no interface unit at …"*. - -#### Placement of compiled host modules (2026.10.1.1+) - -The bundled `mcpp` module and the host modules a program imports are not -compiled into each program's own directory. Each is an entry of a store, -addressed by everything that reaches its compile: the host compiler's identity, -the standard flag, the flags the compile carries, the BMIs it imports, the -SHA-256 of the interface file, the mcpp version (for the bundled module) and the -providing package (for a host module). The inputs are recorded in the entry's -`entry.json`, and a hit compares them field by field, never the hash alone. Two -programs whose compiles agree on all of these share one entry, and a program -whose standard or flags differ has an entry of its own, so a BMI is never taken -by a compile that does not agree with it. - -Where an entry lives depends on where its text comes from: - -| Module | Kept in | Reason | -|---|---|---| -| the bundled `mcpp` module and its `mcpp.core` alias | the global cache: `$MCPP_HOME/build-cache/v1/pkg/_engine/mcpp-build-module@//` | the engine owns its text, which is identical in every project for one mcpp version and one host compiler | -| a host module of an index package whose sources are in the immutable store | the global cache: `…/pkg//@//` | the dependency cache's rule: a name and version identify the bytes | -| a host module of a `path` or `git` dependency, or of a workspace member | `/target/.build-mcpp/host-modules//` | its sources can change without its name and version changing | - -The cache mode applies as it does to dependencies (see -[04 §2.10](04-mcpp-toml.md)). With `--cache global`, the default, an entry goes to -the global cache where the table says so. With `--cache local` or `--cache off` -every entry is kept in the workspace's store, and the programs of one invocation -still share them. A host module of a `path` dependency is never written to the -global cache. The key of an entry in the workspace's store also holds a digest of -the package's source tree, so an edit to a header that the interface includes -compiles a new entry, and the old one is left where it is. - -An entry in the global cache is an entry like a dependency's. `mcpp cache list` -shows the bundled module as `_engine/mcpp-build-module@`, `mcpp cache -verify` checks it, and `mcpp cache gc` collects it by last use; an upgrade of mcpp -leaves the previous version's entry for `gc`. The workspace's store is removed -with its `target/` by `mcpp clean`. A program sees what it saw before: the same -modules, compiled with the same flags. - -**A package may offer several rules, selected by features** (mcpp 2026.9.5.3+). -Every module interface unit among the package's resolved `[build] sources` — -including the sources a feature adds — is compiled as a host module under the -name it declares, the lib root first. A feature unit may import the lib root; -units are otherwise compiled alone, so they import `std`, `mcpp` and nothing -else. Only listed sources take part: the inferred `src/**` of a package that -declares no `sources` is not consulted, so a rule package published before this -release exposes exactly what it exposed then. +whatever the package is called. A build program imports the public modules of +the libraries it uses (SPEC-008): the interface root and what it re-exports, +and the units an active feature adds; another exported module draws the +warning a target build gives (W3). + +**Two libraries of one program may not declare one module name.** `import` +addresses the module, so the two are indistinguishable to the compiler. mcpp +refuses this before compiling, naming both packages and both interface paths. + +**`mcpp.` is reserved for modules maintained by the mcpp project** +(SPEC-007 R9.6). A module name under that prefix from a package outside the +`mcpp` namespace produces a warning naming both, and the build proceeds. + +A build program that imports a module no library provides, a module of a +library that reaches it only as another library's dependency, or a module +behind a feature that is not enabled, is refused by name before anything is +compiled. + +#### Build plugins + +A unit that imports the engine interface (`mcpp.core`, or `mcpp`) is a +**build-plugin unit**; nothing else declares it. The engine interface exists +only in the build-program configuration, so: + +- a plugin may depend on other plugins and on ordinary libraries in its + **`[dependencies]`**: when a build program uses the plugin, those + dependencies are build-program instances too; +- a package whose library has plugin units, built as the root of a build + (`mcpp build`, `mcpp test`, a selected member), is planned in the + build-program configuration, for this machine. Its unit tests are ordinary + tests under `tests/` that may import `mcpp`; `--target` naming another machine + is refused; +- a plugin unit in the target — a plugin reached through `[dependencies]` of a + project — is refused when the build is planned, naming the chain of + dependencies and the section to move it to; +- `mcpp pack` refuses a plugin: it is distributed as source, and a build + program compiles it with its own toolchain. + +#### Several modules behind features + +A package may offer several rules, each behind a feature (mcpp 2026.9.5.3+). +The units a feature adds are that feature's public modules (SPEC-008 I2), and +`rule_module` names the one the feature's synthesised program imports; it must +be declared by one of the feature's units: ```toml # the collection's manifest @@ -1815,69 +1830,32 @@ release exposes exactly what it exposed then. sources = ["src/plugins.cppm"] # export module mcpp.plugins; [features] -rules-cuda = { sources = ["rules/cuda.cppm"] } # export module mcpp.rules.cuda; -rules-spirv = { sources = ["rules/spirv.cppm"] } # export module mcpp.rules.spirv; +rules-cuda = { sources = ["rules/cuda.cppm"], rule_module = "mcpp.rules.cuda" } +rules-spirv = { sources = ["rules/spirv.cppm"], rule_module = "mcpp.rules.spirv" } ``` ```toml # a consumer [build-dependencies.mcpp] -plugins = { version = "0.3.0", features = ["rules-spirv"], host-module = true } +plugins = { version = "0.3.0", features = ["rules-spirv"] } ``` -**`[build-dependencies]`, not `[dependencies]`** — a rule package is the case -[04 §2.6.1](04-mcpp-toml.md) describes exactly: its library must never reach the target while its -rule is still wanted. The two axes are separate, so `host-module = true` says -*which build-time product* is wanted and the section says *whether the package -reaches the target*; a rule package answers "no" on the second axis, and the -section is where that is said. Written in `[dependencies]` it still works, and -that is precisely why the distinction has to be stated rather than enforced by -a failure. - The module set is the feature set: a unit whose feature is not active is not -compiled, and importing it fails as an unknown module. `mcpp:plugins` is the -collection the mcpp project maintains (repository `mcpp-community/mcpp-plugins`); -its members are named `mcpp.rules.` for rule packages and `mcpp.tools.` -for build-time utilities. +compiled, and importing it is refused naming the feature. `mcpp:plugins` is the +collection the mcpp project maintains (repository `mcpp-community/mcpp-plugins`). -*Build-time only:* a `host-module = true` dependency is **not** compiled into -or linked with the target, and neither is anything it depends on. It exists to -run during `build.mcpp` and nowhere else. (Before 2026.8.5.2 it was also built -as an ordinary library, which made `import mcpp;` inside a rule fail: the -bundled module does not exist in that second compile. Until 2026.8.29.1 the -rule itself was excluded but its own `[dependencies]` were not, so they were -compiled and linked into the consumer's binary while the rule could not import -them.) +#### Readings kept for compatibility -### A rule that depends on another rule (mcpp 2026.8.29.1+) +Two spellings from before mcpp 2026.10.11.1 keep their meaning and are +deprecated (51 — Supported Versions): -A rule declares what it needs in its own `[build-dependencies]`, and may import -any entry there marked `host-module = true`: - -```toml -# inside the rule package's manifest -[build-dependencies] -globbing = { path = "../globbing", host-module = true } -``` - -```cpp -// the rule's own interface -export module tidyrule; -import std; -import mcpp; -import globbing; -``` - -mcpp compiles the inner rule first, in the same command and with the same -flags, so BMI agreement stays structural rather than checked. - -The consumer may **not** import `globbing`: build-time provisions cross one -further edge only on a `reexport = true` edge, and mcpp enforces that rather -than leaving it to the compiler, which on GCC would allow the import and then -fail on someone else's machine. +| Spelling | Read as | Removed from | +|---|---|---| +| `[dependencies] x = { host-module = true }`, or a `[dependencies]` entry requesting a `rule_module` feature | the same entry in `[build-dependencies]` (`host-module-in-dependencies`) | mcpp 2027.1 | +| a plugin whose library imports a package listed in its own `[build-dependencies]` with `host-module = true` | that entry in the plugin's `[dependencies]` as well (`plugin-imports-from-build-dependencies`) | mcpp 2027.1 | -*Limit:* one interface unit per host module. A library with implementation -units or several modules cannot yet be a rule's build dependency. +Each use is warned about where the manifest can be edited, and every use is +listed by `mcpp why compat`. ### `reexport = true` — a library standing up a toolchain for its user (2026.8.6.2+) diff --git a/docs/31-authoring-a-rule-package.md b/docs/31-authoring-a-rule-package.md index 8fbf521a..8b28c388 100644 --- a/docs/31-authoring-a-rule-package.md +++ b/docs/31-authoring-a-rule-package.md @@ -122,12 +122,12 @@ device_extensions = [".toy"] | key | effect on a consumer | |---|---| | `sources` | the module is compiled only when the feature is active | -| `rule_module` *(2026.9.7.1+)* | the module the consumer's build program imports. It implies `host-module = true` | +| `rule_module` *(2026.9.7.1+)* | the module the consumer's build program imports; requesting the feature asks for the package's library. One of the feature's units must declare it (2026.10.11.1+) | | `device_extensions` *(2026.9.7.1+)* | those extensions are classified as **device sources**: never scanned for imports, never producing a BMI, and refused if no action claims them | A rule that generates or checks rather than compiling a language declares no -`device_extensions`; `08-build-rules` is that shape and its consumer writes -`host-module = true` on the dependency edge itself. +`device_extensions`; `08-build-rules` is that shape, and its consumer lists the +package in `[build-dependencies]`. Device extensions are not in the default source glob. A consumer opts in by naming the files: @@ -169,6 +169,31 @@ and it holds the package's **whole** device set. A build with two backends puts both backends' sources in that one list, and every rule in the build program reads the same value. +## A rule package's own dependencies, build and tests (mcpp 2026.10.11.1+) + +A rule's module imports `mcpp`, the engine interface, and that makes it a +**build plugin** (30 — `build.mcpp`, *Build plugins*): a package that exists in +the build-program configuration only. Three consequences follow for its author. + +- **What the rule's library imports is in its `[dependencies]`.** A rule built + on another rule (`mcpp:plugins`' `plugins-core`, say) or on an ordinary + library (a TOML parser) names it there. When a consumer's build program uses + the rule, those dependencies are compiled for the build program with it. The + rule's `[build-dependencies]` serve the rule package's own `build.mcpp`, as + any package's do. +- **The package builds and tests as the root of a build.** `mcpp build` in the + package compiles its library for this machine, with the engine interface, and + `mcpp test` runs its tests under `tests/` the same way: a test may import the + rule's modules and `mcpp`, and checks the rule's logic without a consumer + project around it. +- **It is published as source.** `mcpp pack` refuses a build plugin; a consumer's + build compiles it with the toolchain of its own build programs. + +A rule package written before this release that imports a package from its own +`[build-dependencies]` (`host-module = true`) keeps building: the entry is read +as a dependency of the rule as well, and the author is told to move it +(`plugin-imports-from-build-dependencies`, 51 — Supported Versions). + ## Declaring work: `mcpp::action` ```cpp @@ -357,10 +382,10 @@ consumer's build program, so a member that runs a payload tool declares it itself — and says so when the lookup returns empty, rather than pointing at a manifest the reader does not own. -A host module's declaration is the exception, and it is visible in every build -program it is compiled into (`tests/e2e/622`), because a host module's own -code runs as part of each of those build programs rather than as a dependency -seen only through the graph. This is why a member declares its own payload on +A build program's library's declaration is the exception, and it is visible in +every build program that uses the library (`tests/e2e/622`), because the +library's own code runs as part of each of those build programs rather than as +a dependency seen only through the graph. This is why a member declares its own payload on itself, and a project that pulls the member in never repeats the declaration. A dist member's payload therefore belongs on the member — on the target axis when the tool serves one target and not others, exactly as any other diff --git a/docs/50-machine-output.md b/docs/50-machine-output.md index bbb849bc..9d9fd50f 100644 --- a/docs/50-machine-output.md +++ b/docs/50-machine-output.md @@ -308,7 +308,8 @@ mcpp self env --format json "config": "/home/u/.mcpp/config.toml", "buildCache": "/home/u/.mcpp/build-cache/v1", "mcppVersion": "2026.8.8.3", - "defaultToolchain": "gcc@16.1.0" // 2026.9.28.2+ + "defaultToolchain": "gcc@16.1.0", // 2026.9.28.2+ + "engineModules": "/home/u/.mcpp/engine/2026.10.11.1/modules" // 2026.10.11.1+ } ``` @@ -389,6 +390,25 @@ Each entry is `{subject, value, class, origin, decidedFor, considered[]}`: Every build writes the same record into `target///resolution.json` under `sources`. +### `mcpp.why.build-programs` and `mcpp.why.compat` *(mcpp 2026.10.11.1+)* + +``` +mcpp why build-programs --format json +mcpp why compat --format json +``` + +Both prepare the build as `mcpp why` does and report what it recorded; `data` +carries `status` (`ok` or `refused`) and: + +| kind | field | content | +|---|---|---| +| `mcpp.why.build-programs` | `graphs[]` | one entry per build programs' graph: `workspace` (its directory) and `programs[]`, each `{member, package, source, libraries[], identity, binary}` | +| `mcpp.why.compat` | `uses[]` | one entry per use of a deprecated reading: `{id, where, detail, warned, since, floor, canonical, docs}`; `warned` is false in a manifest the build does not edit | + +`engineModules` in `mcpp.env` is the directory of the engine interface package +of this mcpp version, which a build program's graph depends on (30 — +`build.mcpp`); it exists once a build program has been compiled. + ### `mcpp.toolchain.list` — the installed toolchains and the targets this host serves ``` diff --git a/docs/51-supported-versions.md b/docs/51-supported-versions.md index f39fbd36..cf1ebf4d 100644 --- a/docs/51-supported-versions.md +++ b/docs/51-supported-versions.md @@ -41,7 +41,7 @@ worth stating precisely. | Surface | Stability | |---|---| -| `mcpp.toml` keys | Additive. An existing key keeps its meaning; an unrecognised key is reported, never silently ignored | +| `mcpp.toml` keys | Additive. An existing key keeps its meaning; an unrecognised key is reported, never silently ignored. The exceptions are the deprecations below | | CLI commands and flags | Additive. A removed spelling is kept as an alias | | Machine output (`--message-format json`) | Versioned by `schemaVersion`; see [50](50-machine-output.md) | | `build.mcpp` directive protocol | Versioned; see `kProtocolVersion`. An engine refuses a program declaring a **higher** version rather than guessing | @@ -55,6 +55,30 @@ error. The protocol number governs directives on the wire; the typed API is governed by which engine is installed. Both are stated here because the failure a package author sees depends on which one they crossed. +## Deprecations + +A spelling or a reading of a manifest that is to be removed is deprecated +first, and the exceptions to "an existing key keeps its meaning" are exactly +the rows of this table (mcpp 2026.10.11.1+). + +- A deprecated spelling keeps working. Each use in a manifest the build can + edit (the root, a workspace member, a path package of the workspace) is + warned about, naming the spelling that replaces it, the release that + deprecated it and the release from which it is removed. `--strict` refuses + it. `mcpp why compat` lists every use, in dependencies too. +- The removal is a CalVer month. At least two whole calendar months separate + the release that deprecates a reading and that month, and the reading is + removed in the first release of that month or later. +- The table is generated from one registry in the engine + (`modules/manifest/src/compat/registry.cppm`); the engine's own tests fail + once a release reaches a row's month while the reading is still present. + +| Reading | Since | Removed from | Write instead | +|---|---|---|---| +| `workspace-position`: a workspace root's own `[toolchain]`, `[indices]`, `[profile.]`, `[target.]` scalars or `[xlings]` reaching members by position | 2026.10.10.1 | 2027.1 | `[workspace.toolchain]`, `[workspace.indices]`, `[workspace.profile.]`, `[workspace.target.]`, `[workspace.xlings]` | +| `host-module-in-dependencies`: a `[dependencies]` entry asking for `host-module = true` or for a `rule_module` feature, read as a build dependency | 2026.10.11.1 | 2027.1 | the same entry in `[build-dependencies]` | +| `plugin-imports-from-build-dependencies`: a build plugin's library importing a package it lists in its own `[build-dependencies]` with `host-module = true` | 2026.10.11.1 | 2027.1 | the package in the plugin's `[dependencies]` | + ## Reproducing a build `mcpp.lock` records what a build resolved. `--locked` asserts that a resolution diff --git a/docs/specs/README.md b/docs/specs/README.md index ca2a7af5..b39d6dd3 100644 --- a/docs/specs/README.md +++ b/docs/specs/README.md @@ -33,11 +33,11 @@ | [SPEC-001](package-identity.md) | 包身份(`package.namespace` / `package.name`)、`[dependencies]` 选择器与匹配机制 | 评审中 v1.1 | 2026-08-03 | mcpp >= 0.0.106 | | [SPEC-002](target-side.md) | 目标侧模型与能力声明(`mcpp:` 保留命名空间、五层、三条规则) | 评审中 v1.0 | 2026-08-24 | mcpp >= 2026.8.24.2 | | [SPEC-003](exit-codes.md) | 退出码契约(分类、语义、稳定性承诺) | 评审中 v1.0 | 2026-09-01 | mcpp >= 2026.9.1.1 | -| [SPEC-004](manifest-semantics.md) | `mcpp.toml` 的平面划分、条件化形状、解析轴与命名规约 | 草案 v1.14 | 2026-10-10 | 条件化形状 mcpp >= 2026.8.29.1;目标轴 mcpp >= 2026.9.6.4;`linkage` 默认值 mcpp >= 2026.9.15.2;链接 flag 的词读法 mcpp >= 2026.9.26.2;条件化的 `dialect_cxxflags` 与 `-p` 的包身份 mcpp >= 2026.9.28.1;条件表按具体程度生效 mcpp >= 2026.9.28.2;平台前缀规约 mcpp >= 2026.10.5.1;根包的继承、根位置的 profile、链接 flag 的作用域、链接期优化与导出发现 mcpp >= 2026.10.5.2;架构段的 `x86` 即 `i686` mcpp >= 2026.10.5.3;前缀即语义与工作空间层 mcpp >= 2026.10.10.1 | -| [SPEC-005](build-database.md) | 构建数据库:`mcpp emit build-database` 的内容、取值规则与不写工程目录的保证 | 评审中 v1.6 | 2026-09-29 | mcpp >= 2026.9.15.1;v1.3 条款 mcpp >= 2026.9.26.2;v1.4 条款 mcpp >= 2026.9.27.1;v1.5 条款 mcpp >= 2026.9.28.1;v1.6 条款 mcpp >= 2026.9.29.5 | +| [SPEC-004](manifest-semantics.md) | `mcpp.toml` 的平面划分、条件化形状、解析轴与命名规约 | 草案 v1.15 | 2026-10-11 | 条件化形状 mcpp >= 2026.8.29.1;目标轴 mcpp >= 2026.9.6.4;`linkage` 默认值 mcpp >= 2026.9.15.2;链接 flag 的词读法 mcpp >= 2026.9.26.2;条件化的 `dialect_cxxflags` 与 `-p` 的包身份 mcpp >= 2026.9.28.1;条件表按具体程度生效 mcpp >= 2026.9.28.2;平台前缀规约 mcpp >= 2026.10.5.1;根包的继承、根位置的 profile、链接 flag 的作用域、链接期优化与导出发现 mcpp >= 2026.10.5.2;架构段的 `x86` 即 `i686` mcpp >= 2026.10.5.3;前缀即语义与工作空间层 mcpp >= 2026.10.10.1;依赖的库与配置(§12) mcpp >= 2026.10.11.1 | +| [SPEC-005](build-database.md) | 构建数据库:`mcpp emit build-database` 的内容、取值规则与不写工程目录的保证 | 评审中 v1.8 | 2026-10-11 | mcpp >= 2026.9.15.1;v1.3 条款 mcpp >= 2026.9.26.2;v1.4 条款 mcpp >= 2026.9.27.1;v1.5 条款 mcpp >= 2026.9.28.1;v1.6 条款 mcpp >= 2026.9.29.5;v1.7 条款 mcpp >= 2026.9.30.2;v1.8 条款 mcpp >= 2026.10.11.1 | | [SPEC-006](toolchain-management.md) | 工具链管理:身份、来源、选择与载荷契约 | 草案 v0.7 | 2026-10-09 | 逐条标注;已实现条款 mcpp >= 2026.9.24.1;§3.7 mcpp >= 2026.9.28.1;§3.7.1 mcpp >= 2026.9.28.2;§2.2.1 与 §3.3 的非缺省来源 mcpp >= 2026.10.1.3;§3.8 mcpp >= 2026.10.8.1 | -| [SPEC-007](build-plugins.md) | 构建插件:配置、施工与校验的分工,运行时与规划期的义务 | 草案 v0.8 | 2026-10-10 | 逐条标注;mcpp >= 2026.9.26.2;v0.3 条款 mcpp >= 2026.9.27.1;v0.4 条款 mcpp >= 2026.9.28.1;v0.5 条款 mcpp >= 2026.9.28.2;v0.6(§9)mcpp >= 2026.9.28.3;R3.9、R3.10 mcpp >= 2026.10.10.1 | -| [SPEC-008](library-interface.md) | 库的接口:公开模块、发布闭包与两种形态的一致 | 草案 v0.1 | 2026-09-28 | 第一阶段(只警告)mcpp >= 2026.9.28.3 | +| [SPEC-007](build-plugins.md) | 构建插件:配置、施工与校验的分工,运行时与规划期的义务 | 草案 v0.9 | 2026-10-11 | 逐条标注;mcpp >= 2026.9.26.2;v0.3 条款 mcpp >= 2026.9.27.1;v0.4 条款 mcpp >= 2026.9.28.1;v0.5 条款 mcpp >= 2026.9.28.2;v0.6(§9)mcpp >= 2026.9.28.3;R3.9、R3.10 mcpp >= 2026.10.10.1;§10 mcpp >= 2026.10.11.1 | +| [SPEC-008](library-interface.md) | 库的接口:公开模块、发布闭包与两种形态的一致 | 草案 v0.2 | 2026-10-11 | 第一阶段(只警告)mcpp >= 2026.9.28.3;v0.2 mcpp >= 2026.10.11.1 | | [SPEC-009](toolchain-maintenance.md) | 工具链的支持与维护:版本线、默认值、来源、移动与退役 | 草案 v0.6 | 2026-10-09 | 逐条标注;本版只有规范,多数条款未实现;§6.4 mcpp >= 2026.10.8.1 | ## 文档约定 diff --git a/docs/specs/build-database.md b/docs/specs/build-database.md index 99a52cc1..a67f0e9c 100644 --- a/docs/specs/build-database.md +++ b/docs/specs/build-database.md @@ -4,12 +4,12 @@ |---|---| | 规范编号 | SPEC-005 | | 标题 | mcpp 输出的构建数据库:内容、取值规则与不写工程目录的保证 | -| 状态 | 评审中 v1.7 | -| 版本 | 1.7 | -| 最后修改 | 2026-09-30 | -| 对应实现 | mcpp >= 2026.9.15.1;v1.3 修改的 R2.5、R3.7、R3.8、R4.1、R5.2 为 mcpp >= 2026.9.26.2;v1.4 修改的 R2.5 为 mcpp >= 2026.9.27.1;v1.5 修改的 R3.7、R3.12、R5.1、R5.2 为 mcpp >= 2026.9.28.1;v1.6 修改的 R2.1、R3.3、R3.4、R3.5、R4.1、R5.2 为 mcpp >= 2026.9.29.5;v1.7 新增的 R3.8a 为 mcpp >= 2026.9.30.2 | +| 状态 | 评审中 v1.8 | +| 版本 | 1.8 | +| 最后修改 | 2026-10-11 | +| 对应实现 | mcpp >= 2026.9.15.1;v1.3 修改的 R2.5、R3.7、R3.8、R4.1、R5.2 为 mcpp >= 2026.9.26.2;v1.4 修改的 R2.5 为 mcpp >= 2026.9.27.1;v1.5 修改的 R3.7、R3.12、R5.1、R5.2 为 mcpp >= 2026.9.28.1;v1.6 修改的 R2.1、R3.3、R3.4、R3.5、R4.1、R5.2 为 mcpp >= 2026.9.29.5;v1.7 新增的 R3.8a 为 mcpp >= 2026.9.30.2;v1.8 新增的 R3.13 为 mcpp >= 2026.10.11.1 | | 相关设计文档 | `.agents/docs/2026-09-14-636-build-database-and-the-latest-xlings.md`
`.agents/docs/2026-09-26-compile-database-and-issue-699-design.md` | -| 相关 issue | #636, #648, #655, #699, #702, #707, #732 | +| 相关 issue | #636, #648, #655, #699, #702, #707, #732, #797 | | 依据的外部规范 | S1「C++ Build Database: IDE Profile」profile 0.3.0(§7.2 的 `generated`,Sunrisepeak/mcpp-language-server#28;此前为 0.2.0)与 S2 0.2.0 §3.4,取自 https://github.com/Sunrisepeak/lsp-mcpp-private 提交 `b82859d`(schema 自提交 `28ecd6e` 起未变);S2 0.3.0 §3.4 的部分回答(S2-3.4-12、S2-3.4-13,Sunrisepeak/mcpp-language-server#25);JSON Compilation Database | ## 0. 适用范围 @@ -183,6 +183,19 @@ mcpp 输出的 S1 文档满足 S1 等级 2,不输出 `ide.options`。等级 3 (R2.5);由消费方决定是否在其用户同意时运行 `generator`。**已实现** (mcpp >= 2026.9.28.1,mcpp#724) +### 3.6 构建程序 + +- **R3.13** 计划中的构建程序由构建程序图编译(SPEC-004 §12),文档以保留的集合名描述该图的每个 + 单元:构建程序(`build.mcpp` 或合成的规则程序)归入 `<包>:build`,构建程序实例的库归入 + `<包>:host`,引擎接口归入 `mcpp:core`,构建程序图的标准库模块归入 `mcpp:std:host`。`<包>:build` + 的 `family-name` 是该包的名字。这些集合的 `ide.configuration` 为 `build-program`,`visible-sets` 只列出同一张图的集合;构建程序配置不计为 + R3.3 的多个配置,不改变其余集合的名字。单元的 `arguments` 取自构建程序图计划中该单元的编译 + 记录,`.mcpp` 单元带有语言参数(GCC、Clang 为 `-x c++`)。`mcpp:core` 的单元提供 `mcpp` 与 + `mcpp.core`,其 `source` 位于 `mcpp self env` 的 `engineModules` 目录。这些单元**禁止**进入 + `--spec compile-commands` 的文档与 `mcpp build` 写出的 `compile_commands.json`:`build.mcpp` 与 + 引擎接口是 mcpp 生态内部的约定,由 mcppls 消费。构建程序全部新鲜时,命令仍规划构建程序图以描述 + 它,而不编译。**已实现** + ## 4. `--spec compile-commands` - **R4.1** 文档为 `mcpp build --configure-only` 在同一组选择器下写入 @@ -247,3 +260,4 @@ mcpp 输出的 S1 文档满足 S1 等级 2,不输出 `ide.options`。等级 3 | 1.5 | 2026-09-28 | R3.7:规则声明的设备源不是编译单元,不进入 S1 与 `compile_commands.json`(#724)。新增 R3.12:集合的 `ide.generated` 列出规则生成的文件与目录,给出构建写入的路径与生成它的步骤,S1 0.3.0(#724,Sunrisepeak/mcpp-language-server#28)。R5.1:S1 版本为 0.3.0。R5.2:以构建程序的指令为前提的检查不对其构建程序已失败的包运行,失败路径保留已记录的说明(#724)。 | | 1.6 | 2026-09-29 | 工作区按配置规划,与 `mcpp build` 相同(R2.1、R3.3、R3.4、R5.2):成员共用的包在一个配置中只描述一次;集合名的前缀由 `<成员>/` 改为只在文档描述多个配置时出现的 `<配置>/`;一个配置的规划失败时逐成员规划。R3.5:被选成员的集合按其目标给出 `ide.kind`。R4.1:同一文件与输出一条条目。 | | 1.7 | 2026-09-30 | 新增 R3.8a:同一配置中两个包提供同一个模块名时,两个单元都列出它,可能导入它的单元的 `arguments` 带有构建所用的绑定(#732)。 | +| 1.8 | 2026-10-11 | 新增 R3.13:构建程序、构建程序实例的库与引擎接口以 `<包>:build`、`<包>:host`、`mcpp:core`、`mcpp:std:host` 集合描述,`ide.configuration` 为 `build-program`,只进入 S1 文档(#797)。 | diff --git a/docs/specs/build-plugins.md b/docs/specs/build-plugins.md index 7be5911b..7aff95cb 100644 --- a/docs/specs/build-plugins.md +++ b/docs/specs/build-plugins.md @@ -4,12 +4,12 @@ |---|---| | 规范编号 | SPEC-007 | | 标题 | 构建插件:配置、施工与校验的分工,运行时与规划期的义务 | -| 状态 | 草案 v0.8 | -| 版本 | 0.8 | -| 最后修改 | 2026-10-10 | -| 对应实现 | 逐条标注。未注明版本的「已实现」条款对应 mcpp >= 2026.9.26.1;注明 mcpp#702 的条款对应 mcpp >= 2026.9.26.2;注明 mcpp#707、#708、#709、#711 的条款对应 mcpp >= 2026.9.27.1;注明 mcpp#723 的条款对应 mcpp >= 2026.9.28.1;注明 mcpp 2026.9.28.2 的条款对应该版本;§9 与注明 mcpp#734 的条款对应 mcpp >= 2026.9.28.3;注明 mcpp#778 的条款对应 mcpp >= 2026.10.10.1 | -| 相关设计文档 | `.agents/docs/2026-09-26-compile-database-and-issue-699-design.md`(§5)、`.agents/docs/2026-09-28-ecosystem-design-and-optimisation-plan.md`(WS1、WS3)、`.agents/docs/2026-09-28-build-cost-foreign-toolsets-and-library-surface-design.md`(§3、§4)、`.agents/docs/2026-10-10-one-source-one-meaning-design.md`(§4) | -| 相关 issue | mcpp#699、mcpp#701、mcpp#702、mcpp#703、mcpp#707、mcpp#708、mcpp#709、mcpp#711、mcpp#734、mcpp#755、mcpp#778 | +| 状态 | 草案 v0.9 | +| 版本 | 0.9 | +| 最后修改 | 2026-10-11 | +| 对应实现 | 逐条标注。未注明版本的「已实现」条款对应 mcpp >= 2026.9.26.1;注明 mcpp#702 的条款对应 mcpp >= 2026.9.26.2;注明 mcpp#707、#708、#709、#711 的条款对应 mcpp >= 2026.9.27.1;注明 mcpp#723 的条款对应 mcpp >= 2026.9.28.1;注明 mcpp 2026.9.28.2 的条款对应该版本;§9 与注明 mcpp#734 的条款对应 mcpp >= 2026.9.28.3;注明 mcpp#778 的条款对应 mcpp >= 2026.10.10.1;§10 对应 mcpp >= 2026.10.11.1 | +| 相关设计文档 | `.agents/docs/2026-09-26-compile-database-and-issue-699-design.md`(§5)、`.agents/docs/2026-09-28-ecosystem-design-and-optimisation-plan.md`(WS1、WS3)、`.agents/docs/2026-09-28-build-cost-foreign-toolsets-and-library-surface-design.md`(§3、§4)、`.agents/docs/2026-10-10-one-source-one-meaning-design.md`(§4)、`.agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md` | +| 相关 issue | mcpp#699、mcpp#701、mcpp#702、mcpp#703、mcpp#707、mcpp#708、mcpp#709、mcpp#711、mcpp#734、mcpp#755、mcpp#778、mcpp#797 | | 使用文档 | [docs/30 - build.mcpp](../30-build-mcpp.md)、[docs/31 - 编写规则包](../31-authoring-a-rule-package.md) | 本规范规定构建插件对引擎和对消费方承担的义务,以及引擎为此提供的机制。docs/31 说明怎样编写 @@ -288,7 +288,34 @@ 提供构造这些陈述的函数。编译并运行构建程序的是 bootstrap 工具链,因此一个无法使用的 自定义工具链**必须**仍能让构建程序运行并报告原因。(**已实现**,mcpp 2026.10.1.3) -## 10. 变更记录 +## 10. 构建程序的库与构建插件(mcpp#797) + +- **R10.1** 构建程序的依赖是其包 `[build-dependencies]` 中请求库的条目(SPEC-004 §12 第 5 条), + 以及依赖以 `reexport = true` 转交的库。每个库按完整的包编译,其包含目录、宏定义、特性、依赖与 + 实现单元都参与。(**已实现**) +- **R10.2** 一次构建中宿主工具链与 C++ 标准相同的构建程序是同一张构建程序图的成员;图由引擎写到 + `/target/.build-mcpp/host//`,由子构建规划,由 ninja 构建,`build.mcpp` 由同一个 + 计划编译。在该目录执行 `mcpp build` 即复现构建程序的编译。多个程序使用的库只编译一次。(**已实现**) +- **R10.3** 程序在其身份变化时编译;身份覆盖引擎版本、工具链、标准、程序文本,以及图中每个包的 + 名字、版本、来源、构建程序实例的特性与可原地变化的源码树的内容。程序的运行缓存以身份为键。 + 所有程序身份未变时,不规划构建程序图。(**已实现**) +- **R10.4** 引擎接口(`mcpp`、`mcpp.core`)是一个引擎包,按 mcpp 版本写到 + `$MCPP_HOME/engine/<版本>/modules/`,只读;只在图中有单元导入它时成为依赖。它的编译产物在全局 + 依赖缓存中的位置为 `pkg/_engine/mcpp-build-module@<版本>`。(**已实现**) +- **R10.5** 构建程序导入的模块**必须**由其使用的库公开(SPEC-008),否则在编译之前按名字拒绝:没有 + 库提供的模块写出候选条目;仅作为另一个库的依赖到达的模块写出 `reexport`;未启用特性之后的模块 + 写出特性(R9.7)。一个程序的两个库声明同一个模块名时,在编译之前拒绝,写出两个包与两个文件。 + (**已实现**) +- **R10.6** 导入引擎接口的单元是构建插件单元(P1–P5): + - **P1** 插件单元只出现在构建程序实例中;目标实例中出现时在规划期拒绝,写出依赖链与应写入的段。 + - **P2** 插件以 `[dependencies]` 依赖其他插件与普通库,它们随配置成为构建程序实例。 + - **P3** 库含插件单元的包作为构建的根时,以构建程序配置、为构建机器规划;`--target` 指向其他 + 机器时拒绝。 + - **P4** 程序目标的单元导入引擎接口时拒绝,并指向 `build.mcpp`。 + - **P5** `mcpp pack` 拒绝插件:插件以源码分发。 + (**已实现**) + +## 11. 变更记录 | 版本 | 日期 | 变更 | |---|---|---| @@ -300,3 +327,4 @@ | 0.6 | 2026-09-28 | 随 mcpp 2026.9.28.3:新增 §9(mcpp#734),即 `mcpp.core` 与 `mcpp` 的永久等价、接口的稳定性与协议表、构建信息、结构化诊断、批量放置、插件模块的名字、缺失模块指出 feature、包的版本下限;R7.1 的版本写在清单中;变更记录移为 §10。 | | 0.8 | 2026-10-10 | 随 mcpp 2026.10.10.1(mcpp#778):新增 R3.9(action 的输出只有一个写入者,生成的翻译单元以声明单元进入规划,生成的模块接口须声明 `.provides()`)与 R3.10(构建程序的输出目录按配置分开,`resolution.json` 记录 `outDir`)。 | | 0.2 | 2026-09-26 | 随 mcpp 2026.9.26.2 落地:R1.3 的警告、R2.1 的 `runtime_search_dir`、R2.4、R3.3 的 `prepare`(目录须含文件;链接边等待所有 `prepare`)、R3.5、R3.6、R4.1、R4.3、R5.2、R5.3 标为已实现。 | +| 0.9 | 2026-10-11 | 随 mcpp 2026.10.11.1(mcpp#797):新增 §10 构建程序的库与构建插件(R10.1–R10.6,P1–P5);变更记录移为 §11。 | diff --git a/docs/specs/library-interface.md b/docs/specs/library-interface.md index fce11dcd..23734c67 100644 --- a/docs/specs/library-interface.md +++ b/docs/specs/library-interface.md @@ -4,17 +4,17 @@ |---|---| | 规范编号 | SPEC-008 | | 标题 | 库的接口:公开模块、发布闭包与两种形态的一致 | -| 状态 | 草案 v0.1 | -| 版本 | 0.1 | -| 最后修改 | 2026-09-28 | -| 对应实现 | 第一阶段(只警告)mcpp >= 2026.9.28.3;第二阶段未实现 | +| 状态 | 草案 v0.2 | +| 版本 | 0.2 | +| 最后修改 | 2026-10-11 | +| 对应实现 | 第一阶段(只警告)mcpp >= 2026.9.28.3;v0.2 的唯一定义、特性接口与 Form B `lib` mcpp >= 2026.10.11.1;第二阶段未实现 | | 相关设计文档 | `.agents/docs/2026-09-28-build-cost-foreign-toolsets-and-library-surface-design.md`(§5) | -| 相关 issue | mcpp#734 | +| 相关 issue | mcpp#734、mcpp#797 | | 使用文档 | [docs/04 - mcpp.toml](../04-mcpp-toml.md)(`[lib]`)、[docs/12 - 二进制分发](../12-binary-distribution.md) | 本规范规定一个库包对消费方公开什么、`mcpp pack` 发布什么,以及同一个包以源码形态和打包形态 -到达消费方时接口保持一致的条件。库包提供给构建程序的模块(构建插件的模块)由 SPEC-007 §9 -规定,不在本规范范围内。 +到达消费方时接口保持一致的条件。消费方包括目标与构建程序:构建程序使用的库(SPEC-007 §10) +的接口由本规范规定。 规范用语与实现状态标记见 [规范索引](README.md)。 @@ -31,10 +31,12 @@ ## 2. 定义 -- **接口根**:`[lib].path` 所指的单元;未写时按约定为 `src/<包名最后一段>.<模块接口扩展名>`。 - 它必须声明主模块接口(`export module ;`),不能是分区。 -- **公开模块**:接口根的模块,以及它用 `export import` 传递地转出的模块与分区。消费方可以 - 导入的就是这些名字。 +- **接口根**:`[lib].path`(Form B 描述符的 `lib`)所指的单元;未写时按约定为 + `src/<包名最后一段>.<模块接口扩展名>`。它必须声明主模块接口(`export module ;`),不能是分区。 +- **公开模块**:接口根的模块,以及它用 `export import` 传递地转出的模块与分区;加上每个已启用 + 特性的 `sources` 加入的主模块接口单元及其转出闭包。消费方可以依赖的就是这些名字。没有接口根 + 的包不陈述接口,它导出的每个主模块都是公开模块。 +- **可导入模块**:包在该配置中编译的全部导出主模块。可导入而不公开的模块引起 W3。 - **公开头文件**:`include/` 下的文件(docs/12)。 - **接口**:公开模块与公开头文件的合集。 - **发布闭包**:公开模块的接口传递地导入的每个单元,无论是否转出。消费方构建公开模块的 BMI @@ -45,9 +47,15 @@ ## 3. 规则 -- **I1** 一个包的接口与它到达消费方的形态无关。(第一阶段:由 W3 警告陈述,源码构建不受限制) +- **I1** 一个包的接口与它到达消费方的形态无关,也与消费方是目标还是构建程序无关。引擎以一个 + 函数(`mcpp.modgraph.library_interface`)回答,所有读者调用它。(第一阶段:由 W3 警告陈述, + 源码构建不受限制;对构建程序同样适用,v0.2) - **I2** 一个包至多有一个接口根,因此它有一个与身份 `(namespace, name)` 对应的可导入名字。代码 - 组织为多个模块的包,由接口根以 `export import` 转出它们(门面形式)。不设接口根的列表。 + 组织为多个模块的包,由接口根以 `export import` 转出它们(门面形式)。不设接口根的列表。特性是 + 例外(v0.2):特性是否启用由消费方决定,`export import` 无法表达"只在特性启用时转出",因此特性 + 加入的接口单元就是该特性的接口;`rule_module` 必须是其中之一。 +- **I6** `[modules] exports`(Form B 的 `modules`)是对公开模块的断言,不指定接口根:本包的构建中 + 未列出的公开模块是错误,`strict` 时列出而不公开的名字也是错误。(v0.2) - **I3** 公开模块的名字**应当**以包的身份为前缀:`.` 或其下的 `..`,因为模块名在一个程序中是全局的。(第一阶段:建议;引擎已拒绝同一 图中的同名模块) @@ -64,6 +72,7 @@ | W1 | `mcpp build`,只对当前构建的包 | `lib` 目标导出模块且没有接口根 | `mcpp pack` 发布的库不带模块接口 | | W2 | `mcpp pack` | 有未进入发布闭包的导出主模块,逐个写出 | 使用打包形态的消费方无法导入它们 | | W3 | `mcpp build`,只对当前构建的包 | 包导入了某个有接口根的依赖的非公开模块;该依赖不是本包所在工作区的成员(同一工作区的成员总是与本包一起从源码构建) | 从源码构建成功,对打包形态会失败 | +| W4 | `mcpp build`,对依赖 | 依赖以 `[lib] path` 写出的接口根不存在(本包自己的构建中为错误) | 它不向消费方陈述接口根 | `mcpp pack` 的 "Withheld" 一行列出每个未发布的单元,包括包没有接口根的情形。(**已实现**) @@ -83,3 +92,4 @@ W1 与 W2 在三个条件同时成立时成为错误,并提供声明"只发布 | 版本 | 日期 | 变更 | |---|---|---| | 0.1 | 2026-09-28 | 首版草案(mcpp#734):定义、I1 至 I5、第一阶段的 W1 至 W3 与完整的 Withheld 报告。 | +| 0.2 | 2026-10-11 | mcpp#797:接口由一个函数定义,目标、构建程序、`mcpp pack` 与校验共用(I1);特性接口(I2);`exports` 为断言(I6);Form B `lib`;W4。 | diff --git a/docs/specs/manifest-semantics.md b/docs/specs/manifest-semantics.md index 4c85042d..052b1d6d 100644 --- a/docs/specs/manifest-semantics.md +++ b/docs/specs/manifest-semantics.md @@ -5,11 +5,11 @@ | **规范编号** | SPEC-004 | | **标题** | `mcpp.toml` 的平面划分、条件化形状、解析轴与命名规约 | | **状态** | **草案(Draft)** | -| **版本** | 1.14 | -| **最后修改** | 2026-10-10 | +| **版本** | 1.15 | +| **最后修改** | 2026-10-11 | | **最低实现版本** | 条件化形状:mcpp **2026.8.29.1**(`[target..build-dependencies]` 起齐备);目标轴:mcpp **2026.9.6.4** | | **作者/维护** | mcpp-community | -| **相关设计文档** | `.agents/docs/2026-09-07-mcpp-toml-unified-semantics-design.md`
`.agents/docs/2026-06-04-manifest-schema-ownership.md`
`.agents/docs/2026-09-03-xlings-workspace-as-the-one-table.md`
`.agents/docs/2026-09-25-issue-690-workspace-build-inheritance-consistency.md`
`.agents/docs/2026-09-27-eight-reports-by-home-and-one-optimisation-plan.md`
`.agents/docs/2026-10-05-std-module-pair-msvc-lto-and-export-discovery-design.md`
`.agents/docs/2026-10-06-windows-x86-arch-vocabulary-and-rc-follow-ups-design.md`
`.agents/docs/2026-10-10-one-source-one-meaning-design.md` | +| **相关设计文档** | `.agents/docs/2026-09-07-mcpp-toml-unified-semantics-design.md`
`.agents/docs/2026-06-04-manifest-schema-ownership.md`
`.agents/docs/2026-09-03-xlings-workspace-as-the-one-table.md`
`.agents/docs/2026-09-25-issue-690-workspace-build-inheritance-consistency.md`
`.agents/docs/2026-09-27-eight-reports-by-home-and-one-optimisation-plan.md`
`.agents/docs/2026-10-05-std-module-pair-msvc-lto-and-export-discovery-design.md`
`.agents/docs/2026-10-06-windows-x86-arch-vocabulary-and-rc-follow-ups-design.md`
`.agents/docs/2026-10-10-one-source-one-meaning-design.md`
`.agents/docs/2026-10-11-build-dependencies-one-pipeline-design.md` | | **相关使用文档** | [docs/04 —— mcpp.toml 字段参考](../04-mcpp-toml.md) | ## 规范用语 @@ -593,17 +593,28 @@ mcpp 2026.9.26.2,#703)。** `[runtime]`,选择器的 `.build`、`.abi`、`.runtime`、`.targets`、`.dependencies`)不作用于 任何包,实现**必须**给出警告,`--strict` 下**必须**拒绝;同一张表在这样的根上同时写在根位置 与 `[workspace.X]` **必须**被拒绝。 -13. **按位置读取(兼容,至 1.0.0)。** 工作空间根上没有 `[workspace.X]` 写法的 `[toolchain]`、 +13. **按位置读取(兼容,自 mcpp 2026.10.10.1 弃用,自 mcpp 2027.1 移除;docs/51 的弃用登记表, + `workspace-position`)。** 工作空间根上没有 `[workspace.X]` 写法的 `[toolchain]`、 `[indices]`、`[profile.]`、`[target.]` 标量行与 `[xlings]`、 `[target..xlings]` 条目,仍按 mcpp 2026.10.8.1 的规则到达成员(成员的 `[toolchain]` 整体替换根的,同名的行与 profile 整体替换,`[indices]` 只到达未声明任何索引 的成员)。每次使用**必须**给出警告 `manifest/workspace-position` 并写出 `[workspace.X]` - 写法;在带 `[package]` 的根上,只在成员确实经由这一读法得到值时给出。 + 写法;在带 `[package]` 的根上,只在成员确实经由这一读法得到值时给出;对作为依赖到达的成员, + 只在构建的根位于该工作空间之外时给出。 +14. **`[xlings.overrides]` 属于构建的根。** `[workspace.xlings.overrides]`(与按位置读取的根自己的 + `[xlings.overrides]`)只给作为构建之根的成员,不给作为依赖到达的成员,与 `[workspace.toolchain]` + 相同。 +15. **`[workspace.dependencies]` 的继承。** 成员以 `x.workspace = true` 继承条目的来源与它请求的 + `features`(工作空间的在前,成员的追加);`default-features` 在成员写出时以成员为准。边上的其余 + 请求(`host-module`、`tools`、`artifacts`、`reexport`、`visibility`、`linkage`)属于成员自己的边, + 写在 `[workspace.dependencies]` 中时实现**必须**给出警告,`--strict` 下**必须**拒绝。 +16. **虚拟根旁的 `build.mcpp`。** 不带 `[package]` 的工作空间根旁的 `build.mcpp` 不属于任何包, + 不运行;实现**必须**给出警告,`--strict` 下**必须**拒绝。 **状态:已实现(第 1 至 7 条 mcpp 2026.9.25.1;第 8 至 10 条 mcpp 2026.9.27.1,mcpp#713、 #714、#710;第 10 条的 `dialect_cxxflags` 为 mcpp 2026.9.28.1,#717;第 1 条的根包、第 10 条的 `[profile.]` 与第 11 条为 mcpp 2026.10.5.2,#771;第 7、8、10 条的工作空间层与第 12、13 条为 -mcpp 2026.10.10.1,#785、#786)。** +mcpp 2026.10.10.1,#785、#786;第 14 至 16 条与第 13 条的依赖位置为 mcpp 2026.10.11.1,#797)。** ## 10. 依赖的程序 @@ -660,6 +671,41 @@ mcpp 2026.10.10.1,#785、#786)。** **状态:已实现(mcpp 2026.10.5.2,#770)。** +## 12. 依赖的库与配置 + +一条依赖由两条轴描述:**段**决定它服务的配置,**边上的请求**决定需要的产物。 + +1. **配置。** `[dependencies]` 与 `[dev-dependencies]` 的条目服务目标配置;`[build-dependencies]` + 的条目服务声明它的包的构建程序配置。配置沿 `[dependencies]` 继承:一个服务构建程序的包,其 + `[dependencies]` 同样服务构建程序。边的配置取自该条声明所在的表,不随依赖链传递;一个包在 + 某配置中是否有实例,由从根出发、经过该配置的边的可达性决定。 +2. **实例。** 一个包在一次构建中可以有目标实例与构建程序实例。一份清单在两张表中写同一个包, + 得到两条边、两个实例,实现**禁止**把它们合并为一条边。 +3. **一个包一个版本。** 两个实例的版本相同,是锁文件记录的版本;构建程序图**必须**以主图的 + 解析为准。 +4. **按配置激活特性。** 目标实例的特性只由服务目标的边请求;构建程序实例的特性只由服务构建程序 + 的边请求。特性声明的载荷(`feature-xlings`)服务这次构建,取两者的并集。构建程序实例的 + 公开 include 目录、接口宏定义与链接 flag **禁止**进入目标的编译与链接。 +5. **产物。** `[build-dependencies]` 的条目在没有列出 `tools` 或 `artifacts` 时请求包的库; + `host-module = true` 在它们之外再请求库,`host-module = false` 不要库;请求声明了 `rule_module` + 的特性即请求库。`host-module` 的名字来自它只表示接口根模块的时期,按 §5.2 不改名;它的含义 + 是"构建程序使用这个依赖的库":公开模块可导入、公开头文件可见、对象参与链接。在 + `[dependencies]` 中,库就是这条边的含义。 +6. **构建程序图。** 一次构建中宿主工具链与 C++ 标准相同的构建程序,是同一张图的成员;图中每个库 + 按完整的包编译(包含目录、宏定义、特性、依赖、实现单元),构建程序本身由同一个计划编译。程序 + 在其身份变化时编译;身份**必须**覆盖引擎版本、工具链、标准、程序文本,以及图中每个包的名字、 + 版本、来源、构建程序实例的特性与可原地变化的源码树的内容。 +7. **构建插件。** 导入引擎接口(`mcpp.core` 或 `mcpp`)的单元是构建插件单元。插件单元只能出现在 + 构建程序实例中;目标实例中出现插件单元时实现**必须**在规划期拒绝并写出依赖链。库中含有插件 + 单元的包作为构建的根时,以构建程序配置、为构建机器规划。 +8. **兼容读法(docs/51 的弃用登记表)。** `[dependencies]` 中请求 `host-module = true` 或 + `rule_module` 特性的条目按 `[build-dependencies]` 读取(`host-module-in-dependencies`); + 构建插件的库导入其自身 `[build-dependencies]` 中以 `host-module = true` 写出的包时,该条目同时 + 读作插件的 `[dependencies]`(`plugin-imports-from-build-dependencies`)。两者自 mcpp + 2026.10.11.1 弃用,自 mcpp 2027.1 移除;每次使用在可编辑的清单处**必须**给出警告。 + +**状态:已实现(mcpp 2026.10.11.1,#797)。** + ## 变更记录 | 版本 | 日期 | 变更 | @@ -679,3 +725,4 @@ mcpp 2026.10.10.1,#785、#786)。** | 1.12 | 2026-10-05 | mcpp 2026.10.5.2:§9 第 1 条补上带 `[package]` 的工作空间根自己的包;第 10 条把 `[profile.]` 列为根位置的键;新增第 11 条链接 flag 的作用域(#771);新增 §11 链接期优化与导出发现(#770)与 `[test] windows_code_page`;§7 补第 21 至 23 条判据。 | | 1.13 | 2026-10-06 | mcpp 2026.10.5.3:§4.6 陈述架构段的等同拼写,`x86` 即 `i686`;`i386` 至 `i586` 不与 `i686` 相互替代。 | | 1.14 | 2026-10-10 | mcpp 2026.10.10.1(#785、#786):§2 陈述键的作用域与来源;§9 新增第 12 条前缀即语义(工作空间层 `[workspace.X]`、按键合并、W7)与第 13 条按位置读取的兼容期;第 7 条发布写出工作空间层;第 8 条改为 `[workspace.xlings]`;第 10 条按求值后的配置分组。 | +| 1.15 | 2026-10-11 | mcpp 2026.10.11.1(#797):新增 §12 依赖的库与配置(两条轴、实例、一个包一个版本、按配置激活特性、`host-module` 的含义与默认值、构建程序图、构建插件、兼容读法);§9 第 13 条的移除版本改为 mcpp 2027.1,并补依赖位置;新增 §9 第 14 至 16 条(`[xlings.overrides]` 属于构建的根、`[workspace.dependencies]` 的继承、虚拟根旁的 `build.mcpp`)。 | diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index 659917b3..8e0a29b9 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -1056,8 +1056,18 @@ path = "src/capi/lua.cppm" # Override the default lib-root location 默认约定：`src/<包名的最后一段>.cppm`（例如包名 `mcpplibs.cmdline` → `src/cmdline.cppm`）。 `path` 是这张表唯一的键；其他键像 `[build]` 中一样被报告（mcpp 2026.9.28.3+）。lib root 即 -SPEC-008 的接口根：打包后的库发布的模块（连同它转出的模块），以及构建程序从 host-module -依赖中导入的模块。 +SPEC-008 的接口根，它对每一种消费方给出同一个陈述（mcpp 2026.10.11.1+）：它的模块及其以 +`export import` 转出的模块，连同已启用特性加入的接口单元，是**公开模块**。导入该库的目标、 +在 `[build-dependencies]` 中使用它的构建程序与 `mcpp pack` 读同一个集合；导出而不公开的模块 +引起警告（SPEC-008 W3）。没有 lib root 的包不陈述接口，它导出的每个模块都是公开模块。 + +`[lib] path` 写出而不存在的路径，在本包的构建中是错误，在本包作为依赖时是警告。Form B +描述符以 `lib = ""` 写出同一件事，与 `sources` 并列，相对于版本目录，`*/` 代表归档的 +外层目录。 + +`[modules] exports`（Form B 描述符的 `modules = { ... }`）把公开模块列为断言：在本包的构建 +中，未列出的公开模块是错误；`[modules] strict = true` 时，列出而不公开的名字也是错误。它不 +选择接口根。 ### 2.5 `[dependencies]`、`[dev-dependencies]`、`[build-dependencies]` @@ -1392,10 +1402,10 @@ cache = "global" # "global" (default) | "local" | "off" workspace 成员。它们的源码可以在 `name@version` 不变的情况下改变， 所以没有任何基于那个身份的键能察觉到变化。 -构建程序 import 的模块遵循同一条规则（2026.10.1.1+）。内置的 `mcpp` 模块与索引包的 host -模块保存在全局缓存中；`path` 或 `git` 依赖、或 workspace 成员的 host 模块保存在 -`target/.build-mcpp/host-modules/` 下，绝不进入全局缓存。`--cache local` 与 `--cache off` -把它们全部保存在工程内。见 [30 — 已编译 host 模块的存放位置](30-build-mcpp.md)。 +构建程序 import 的库遵循同一条规则（2026.10.11.1+）。引擎接口（`mcpp.core`）与索引包保存在 +全局缓存中；`path` 或 `git` 包、或 workspace 成员在 `target/.build-mcpp/host/` 下的构建程序图 +中编译，绝不进入全局缓存。`--cache local` 与 `--cache off` 把它们全部保存在工程内。见 +[30 — 构建程序图](30-build-mcpp.md)。 查看与回收： diff --git a/docs/zh/05-dependencies.md b/docs/zh/05-dependencies.md index 0954003b..c28d4e80 100644 --- a/docs/zh/05-dependencies.md +++ b/docs/zh/05-dependencies.md @@ -374,32 +374,44 @@ gtest = "1.15.2" ```toml [build-dependencies] -protobuf = { version = "35.1", tools = ["protoc"] } +tomlplusplus = "3.4.0" # build.mcpp 导入的库 +protobuf = { version = "35.1", tools = ["protoc"] } # 构建期间运行的程序 ``` -段与边上的请求回答的是**两个不同的问题**，把它们混为一谈是建模上的错误， -不是写法之争。 +一条依赖由两条轴描述（mcpp 2026.10.11.1+，SPEC-004 §12），每条轴在清单中有自己的位置。 -- **段**回答：这个包本身进不进目标。`[dependencies]` 表示进； - `[build-dependencies]` 表示永不进，只能经由它到达的东西同样不进。 -- **边上的请求**回答：想要它的哪一种构建期产物。`tools = [...]` 要一个宿主 - 可执行文件，`host-module = true` 要一个构建程序可以 import 的模块。 +- **段**决定依赖服务的配置。`[dependencies]` 与 `[dev-dependencies]` 服务目标： + 包为目标编译，链接进工程。`[build-dependencies]` 服务工程的构建程序 + （`build.mcpp`）：包为执行构建的机器编译，链接进该程序。配置沿 `[dependencies]` + 继承，因此构建依赖自己的依赖同样服务构建程序。 +- **边上的请求**决定需要哪些产物。`[build-dependencies]` 的条目在没有列出 `tools` + 或 `artifacts` 时请求包的库；`host-module = true` 在它们之外再请求库， + `host-module = false` 不要库。`tools = [...]` 请求为执行构建的机器构建的程序， + 两个段都可以写。 -一个包可以同时在这两个轴上取值，而 protobuf 正是证明这两个轴必须分开的 -例子：一个工程既链接 `libprotobuf`，构建期又需要 `protoc`。它只写一次， -写在 `[dependencies]` 里： - -```toml -[dependencies] -protobuf = { version = "35.1", tools = ["protoc"] } -``` - -`[build-dependencies]` 用于第一个轴无法表达的那种组合 —— 某个包的库不得 -进入目标，而它的工具或规则仍然需要。同一个包同时出现在两张表里不是错误： -普通声明胜出，因为一行 `[build-dependencies]` 不应该悄悄拿掉目标真正需要 -的库；两条声明各自请求的内容（`tools`、`features`、`host-module`、 -`reexport`）都作用在同一条边上（mcpp 2026.9.16.1+；该版本之前，第二条声明 -的请求会被丢弃）。 +| 条目 | 效果 | +|---|---| +| `[build-dependencies] x = "1.0"` | `build.mcpp` 导入 `x` 的公开模块、包含其公开头文件并链接其对象 | +| `[build-dependencies] x = { tools = ["gen"] }` | 程序 `gen`；不要库 | +| `[build-dependencies] x = { tools = ["gen"], host-module = true }` | 程序与库 | +| `[build-dependencies] x = { host-module = false }` | 都不要；只有 `x` 的目录（`mcpp::dep_dir`）与其载荷 | +| `[dependencies] x = { tools = ["gen"] }` | 目标中的库，以及该程序 | + +`host-module` 这个名字早于它现在的含义：引入它时，构建程序只得到依赖的接口根模块； +已经进入已发布描述符的键不改名（SPEC-004 §5.2）。构建程序使用的库按完整的包编译： +它的包含目录、宏定义、特性、依赖与实现单元都参与，构建程序本身也由同一个计划编译 +（30 —— `build.mcpp`）。 + +**同一个包写在两张表中。** 一份清单在 `[dependencies]` 与 `[build-dependencies]` +中写同一个包时，这个包有两个实例，一个为目标编译，一个为构建程序编译。两者的版本都是 +锁文件记录的版本。它们的特性分别激活：只由 `[build-dependencies]` 边请求的特性 +不进入目标实例。 + +**为兼容保留的读法。** `[dependencies]` 中请求 `host-module = true`、或请求声明了 +`rule_module` 的特性的条目，按 `[build-dependencies]` 的条目读取：这个包服务构建程序 +而不进入目标，与 mcpp 2026.10.11.1 之前相同。该读法自 mcpp 2026.10.11.1 弃用， +自 mcpp 2027.1 移除（`host-module-in-dependencies`，51 —— 支持的版本）； +`mcpp why compat` 列出它的使用。 **只含程序的包，只贡献它的程序（mcpp 2026.9.16.1+）。** 一个声明的 `[targets]` 全部是程序（`bin`、`app`、`test`）的依赖，没有可链接的东西。 @@ -413,8 +425,8 @@ protobuf = { version = "35.1", tools = ["protoc"] } 一个工具子构建再次请求正在构建的同一个工具时，在第一次重复处被拒绝，并给出 请求链。 -与 `[dev-dependencies]` 不同，这些依赖**会**被传递遍历：一个构建期依赖自己 -的依赖，正是让它能工作的东西，并且继承它"只服务构建"的性质。 +与 `[dev-dependencies]` 不同，构建期依赖**会**被传递遍历，并与工程一起解析， +一个包一个版本：构建期依赖自己的依赖正是让它能工作的东西，它们与它一起服务构建程序。 一个 feature 可以为构建期请求划定范围，而无需另开一个声明处。 `[feature-deps.]` 的条目可以用相同的源，重述一条已经无条件声明的 diff --git a/docs/zh/07-workspace.md b/docs/zh/07-workspace.md index 68253fbc..c80761db 100644 --- a/docs/zh/07-workspace.md +++ b/docs/zh/07-workspace.md @@ -129,6 +129,21 @@ mbedtls.workspace = true # 继承版本 → "3.6.1" gtest.workspace = true # 继承版本 → "1.15.2" ``` +`[workspace.dependencies]` 条目请求的特性随其来源一起继承，排在成员自己的特性之前； +成员写出 `default-features` 时以成员为准（mcpp 2026.10.11.1+）。边上的其余请求—— +`host-module`、`tools`、`artifacts`、`reexport`、`visibility`、`linkage`——属于成员自己的 +条目，因为同一个工作空间条目可以被继承进 `[dependencies]` 与 `[build-dependencies]`， +两处含义不同；写在 `[workspace.dependencies]` 中时逐个给出警告，`--strict` 下拒绝。 + +```toml +[workspace.dependencies] +hl = { path = "libs/hl", features = ["x"] } + +# 成员 mcpp.toml +[dependencies] +hl = { workspace = true, features = ["y"] } # 请求 x 与 y +``` + 成员可以覆盖继承来的版本： ```toml @@ -203,7 +218,9 @@ default = "llvm@23.1.3" 成员在作为一次构建的根时接收它们：从工作空间构建、以 `-p` 选中，或作为另一个包的宿主工具 构建。作为依赖到达的成员从该次构建的根取得它们。`[workspace.target.]` 的各行 （`.build`、`.abi`、`.runtime`、`.xlings.workspace`）与 `[workspace.xlings]` 在每种位置 -都到达成员（§4.1）。 +都到达成员（§4.1）。`[workspace.xlings.overrides]`（载荷在这台机器上从哪里来）与 +`[workspace.toolchain]` 一样属于构建的根：成员作为一次构建的根时接收它，作为依赖到达时 +不接收（mcpp 2026.10.11.1+；此前作为依赖到达的成员会收到它，随后因声明了 override 被拒绝）。 条件行按所构建的目标求值后计入：其他目标上的 `dialect_cxxflags` 或 `.abi` 行不会把成员与 其他成员分开（§5.4）；对当前目标成立的行，在成员于工作空间的根下构建时生效。 @@ -223,13 +240,16 @@ default = "llvm@23.1.3" `[toolchain]` 整体替换根的，成员同名的行或配置档整体替换根的，`[indices]` 只到达未声明 任何索引的成员。每次使用打印一条 `manifest/workspace-position` 警告，并给出 `[workspace.X]` 写法；在带 `[package]` 的根上，这些表首先属于根包，警告只对确实经由它们 -得到值的成员给出。这种读法在 mcpp 1.0.0 移除。在不带 `[package]` 的根上，同一张表两种写法 +得到值的成员给出；对作为依赖到达的成员，只在构建的根位于该工作空间之外时给出（根自己的构建 +已经包含这些条目）。该读法自 mcpp 2026.10.10.1 弃用，自 mcpp 2027.1 移除 +（`workspace-position`，51 —— 支持的版本）。在不带 `[package]` 的根上，同一张表两种写法 并存会被拒绝。 **不带 `[package]` 的根上没有包。** 写在那里的 `[build]`、`[targets]`、`[dependencies]`、 `[features]`、`[resources]`、`[test]`、`[hooks]`、`[runtime]`，以及选择器的 `.build`、 `.abi`、`.runtime`、`.targets`、`.dependencies` 不作用于任何包；每张表打印一条警告，有共享 -写法时一并给出，`--strict` 下拒绝。 +写法时一并给出，`--strict` 下拒绝。这样的根旁边的 `build.mcpp` 不属于任何包，不会运行， +以同样的方式报告（mcpp 2026.10.11.1+）。 ### 4.1 `[workspace.package]` 与 `[workspace.build]` @@ -326,7 +346,7 @@ workspace 键。 | 表 | 归属 | 对根包的作用 | 对其它成员的作用 | |---|---|---|---| | `[workspace]` 与每一张 `[workspace.X]` | 工作空间 | 接收，根包自己的声明优先 | 接收；`[workspace.dependencies]` 通过 `x.workspace = true` | -| 其余每一张表（`[package]`、`[build]`、`[toolchain]`、`[target.]`、`[profile.]`……） | 根包 | 本身的值 | 无（上文的按位置读取除外，直到 1.0.0） | +| 其余每一张表（`[package]`、`[build]`、`[toolchain]`、`[target.]`、`[profile.]`……） | 根包 | 本身的值 | 无（上文的按位置读取除外，直到 mcpp 2027.1） | 根包的 `[build] ldflags` 同样属于它自己：它们到达根包自己的镜像，不到达成员的镜像， 也不到达依赖的共享库。 @@ -588,8 +608,8 @@ myproject/ - 工作空间根的 `compile_commands.json` 覆盖所有已构建或已配置的成员。 - 工作空间根的 `mcpp.lock` 记录全体成员的解析结果。`mcpp build --workspace` 写入完整记录； `mcpp build -p X` 更新 X 所在图中的条目。 -- 成员的构建程序写入 `/target/.build-mcpp/`。各程序 import 的 host 模块编译进 - workspace 自己的 `target/.build-mcpp/host-modules/`，为全体程序只编一次。 +- 成员的构建程序写入 `/target/.build-mcpp/`。各成员的构建程序在 workspace 的构建程序图 + `target/.build-mcpp/host/` 中编译，多个程序 import 的库只编一次。 - 早期版本的 mcpp 在成员自己的 `target/` 下留下的构建目录不再被读取；`mcpp clean --stale` 会删除它们。 diff --git a/docs/zh/09-commands-by-scenario.md b/docs/zh/09-commands-by-scenario.md index 9b0c4772..c01f92b3 100644 --- a/docs/zh/09-commands-by-scenario.md +++ b/docs/zh/09-commands-by-scenario.md @@ -149,8 +149,13 @@ dependency graph: requested by mcpplibs.app@0.1.0 as 'huxdemo.fw' in [dependencies] requested by huxdemo.comp@0.1.0 as 'fw' in [dependencies] linked static (default) + instances: target ``` +`instances` 说明这个包是为目标编译、为构建程序编译，还是两者都有（2026.10.11.1+）。 +`mcpp why build-programs` 列出构建程序图：每个程序、它使用的库、编译所依据的身份与其二进制。 +`mcpp why compat` 列出每次使用弃用读法的位置与移除它的版本（51 —— 支持的版本）。 + 同一张图记录在 `target///resolution.json` 的 `graph` 下， 每个包一条，根在最前：`package`（规范身份、命名空间、名字、版本、来源）、 `root`、`requested_by`(`requester`、`key`、`table`)，对库还有 `link` @@ -566,8 +571,7 @@ side_effect = false ## 当前边界 -- `mcpp why --format json` 对 `toolchain`、`sources`、`tool` 与 `payload` 话题有定义。 - 其余话题报 `'' has no machine-readable shape yet` 并以非零退出。 +- `mcpp why --format json` 对 `toolchain`、`sources`、`tool`、`payload`、`build-programs` 与 `compat` 话题有定义。其余话题报告 `'' has no machine-readable shape yet` 并以非零退出。 - `mcpp search` 按子串匹配；没有字段选择器，也没有把搜索限定到单个命名 空间的办法。 - `mcpp clean --stale` 读 `target/.build_cache`，它保存的近期条目数量 diff --git a/docs/zh/20-toolchains.md b/docs/zh/20-toolchains.md index f6bde748..9614b113 100644 --- a/docs/zh/20-toolchains.md +++ b/docs/zh/20-toolchains.md @@ -424,7 +424,7 @@ default = { path = "/opt/llvm-trunk" } bootstrap = "llvm@23.1.3" ``` -构建程序（`build.mcpp`）、宿主工具与宿主模块在执行构建的那台机器上编译并运行。 +构建程序（`build.mcpp`）、它们的库与宿主工具在执行构建的那台机器上编译并运行。 `bootstrap` 命名做这件事的工具链 —— 当它不应当是构建工程的那一个时，例如一个正在开发的 工具链，它必须能在不妨碍选择它的那个程序运行的前提下失败。不写 `bootstrap` 时行为不变： 原生构建用自己的工具链，交叉构建解析一个宿主的。 diff --git a/docs/zh/30-build-mcpp.md b/docs/zh/30-build-mcpp.md index 39ec3634..21affa4e 100644 --- a/docs/zh/30-build-mcpp.md +++ b/docs/zh/30-build-mcpp.md @@ -129,8 +129,8 @@ manifest/描述符里（`[build] include_dirs`），而不是构建期程序里 除打印裸字符串外，`build.mcpp` 也可以写成**模块优先**形式——`import mcpp;`，不需要 `#include`。`mcpp` 模块**内置在 mcpp 二进制里**（因此始终与当前这版 mcpp -的协议匹配），按需编译——每个 mcpp 版本与 host 编译器组合只编一次，存入构建缓存 -（见下文*已编译 host 模块的存放位置*）；它的函数只是 emit 上面那些指令： +的协议匹配），每个 mcpp 版本写成一个包，像构建程序的任何库一样编译（见下文*构建程序图*）； +它的函数只是 emit 上面那些指令： ```cpp // build.mcpp @@ -259,7 +259,7 @@ for (auto a : {"-machine","virt","-nographic","-no-reboot","-kernel"}) **在一个构建程序之内，每个名字的 token 组成一个 argv。** 每次 `mcpp::runner(tok)` 调用追加到默认 runner，每次 `mcpp::runner(name, tok)` 调用追加到名为 `name` 的 -runner，按发出顺序，无论调用来自编译进这个程序的哪个 host module。要在两个 runner +runner，按发出顺序，无论调用来自这个程序的哪个库。要在两个 runner 之间选择的程序，在发出任何一个之前完成选择。以打包格式命名的 runner 在 `mcpp run --format ` 下抵达该格式的可分发物（[41 —— 抵达一台设备](41-devices.md)）。 @@ -1202,7 +1202,7 @@ shim，而可用的那份就在项目自己的环境里，根本不在 `PATH` 一条规则 ——「对这些 `.proto` 跑 protoc」「对这些源码跑 clang-tidy」—— 属于一个包， 而不该被复制到每个消费者的 `build.mcpp` 里。机制是 -[`host-module = true`](../04-mcpp-toml.md)；本节讲的是它里面应该长什么样。 +消费方 `[build-dependencies]` 中的库（*构建程序的库*）；本节讲的是它里面应该长什么样。 下面这些从第一个规则包 `mcpplibs.grpcgen` 归纳而来，每一条特征都单独判过是必然还是偶然。 它们是指引而非规则，因为其中没有一条能给出引擎可以检查的判据。 @@ -1228,21 +1228,20 @@ shim，而可用的那份就在项目自己的环境里，根本不在 `PATH` 全部调用一遍。 **没有任何东西提供的 import 会被点名拒绝。** 一个构建程序可以 import 的是：`std`、 -`std.compat`、内置的 `mcpp`，以及依赖边要来的 host 模块。此外的名字在编译器被调用之前 -就被拒绝，并给出那个本该让它可导入的键： +`std.compat`、引擎接口 `mcpp`，以及其包所用库的公开模块。此外的名字在编译器被调用之前 +就被拒绝，并给出能让它可导入的条目： - error: build.mcpp imports 'mcpp.rules.spirv', and no dependency provides it - as a host module. + error: build.mcpp imports 'mcpp.rules.spirv', and no build dependency provides it. ... [build-dependencies.] - = { version = "...", host-module = true } - declared without `host-module = true`: mcpp.plugins (in [build-dependencies]) + = { version = "..." } # or `host-module = true` beside `tools` + declared without its library: mcpp.plugins (in [build-dependencies]) -**模块名由规则的源码声明，`mcpp.*` 是保留前缀。** host 模块以其接口单元声明的名字注册， -而不是以包名注册，所以 `export module mcpp.rules.spirv;` 就是消费者 import 的那个名字。 +**模块名由规则的源码声明，`mcpp.*` 是保留前缀。** 模块以其接口单元声明的名字导入， +而不是以包名导入，所以 `export module mcpp.rules.spirv;` 就是消费者 import 的那个名字。 官方插件集中在一个包里，`mcpp:plugins`（仓库 `mcpp-community/mcpp-plugins`）：规则包命名为 `mcpp.rules.`，构建期工具命名为 `mcpp.tools.`，每个成员由该包的一个 feature 选择 -（见 [`host-module = true`](../04-mcpp-toml.md)）。`mcpp.build.*` 是引擎自己的模块族， +（*特性之后的多个模块*）。`mcpp.build.*` 是引擎自己的模块族， 不用于插件。引擎判定不了谁是官方，所以检查以包的**命名空间**为键，两者不一致时告警 —— warning: build rule 'mcpplibs.plugins' declares the module @@ -1429,148 +1428,127 @@ MCPP_TOOL_PROTOBUF_PROTOC=/usr/bin/protoc mcpp build 理由一样：一个在本机构建不出来的工具**不能是死路**。它**刻意不进** cache key —— 逃生舱不是可复现输入。 -### `host-module = true` —— 可复用的构建规则以包分发 +### 构建程序的库：`[build-dependencies]`（mcpp 2026.10.11.1+） -一条规则（比如「对这些 `.proto` 跑 protoc」）应该**写一次**，而不是复制进每个 -消费者的 `build.mcpp`。把它做成普通的 mcpp 库包再 import: +一条规则（例如"对这些 `.proto` 文件运行 protoc"）只写一次，作为普通的 mcpp 库包发布； +构建程序需要的任何库（TOML 解析器、格式化库）也是如此。构建程序使用其包在 +`[build-dependencies]` 中写出的库： ```toml -[dependencies] -protobufgen = { version = "0.1.0", host-module = true } +[build-dependencies] +tomlplusplus = "3.4.0" +protobufgen = { version = "0.1.0" } ``` ```cpp // build.mcpp +import std; import mcpp; +import tomlplusplus; import protobufgen; -int main() { return protobufgen::generate({"schema"}) ? 0 : 1; } +int main() { + auto cfg = toml::parse_file("codegen.toml"); + return protobufgen::generate({cfg["schema"].value_or("schema")}) ? 0 : 1; +} ``` -mcpp 会把该包的 lib 根模块**为 host 编译，且与 `build.mcpp` 使用相同的标志** —— -这正是 BMI 能用的前提：一个模块接口只对「在 standard / dialect / 编译器身份上与 -它一致」的编译可导入。编译结果按键保存，因此对每组在这些标志上一致的程序只编一次 -（见*已编译 host 模块的存放位置*）。 +条目在没有列出 `tools` 或 `artifacts` 时请求库；`host-module = true` 在它们之外再请求库， +`host-module = false` 不要库（05 —— 依赖与解析）。依赖以 `reexport = true` 转交的库， +以同样的方式使用。 -于是规则**有版本、能测试、能通过既有的包管理器分发**，而且是用 **C++** 写的 -—— 不引入第二门语言，这正是 `build.mcpp` 存在的理由。 +#### 构建程序图 -**模块名是规则源码自己声明的那个**(mcpp 2026.8.29.1+)。`export module -acme.rules.protobuf；` 就以 `acme.rules.protobuf` 被 import，与包叫什么无关。 -模块名是作者定义的 API，不镜像包身份 —— 普通库包一直遵循的就是这条规则。 +构建程序是构建程序配置中的一个程序。它的主文件是 `build.mcpp`，它的依赖是上面的库。 +一次构建中宿主工具链与 C++ 标准相同的构建程序，是同一个工作空间的成员，写到 -2026.8.29.1 之前 host 模块这条路径注册的是裸 `package.name`，于是声明名与包名 -分叉的规则包在 GCC 上能构建、在 Clang 与 MSVC 上失败：GCC 的 BMI 隐式落在 -`gcm.cache` 且按**声明名**索引，而另外两者拿到的是显式的 `=` 映射。 -因此包名不再承担任何 C++ 命名约束，`grpc-rules` 重新是合法包名。 + /target/.build-mcpp/host// -**两个规则不得声明同一个模块名。** `import` 寻址的是模块，所以两个这样的包对编译器 -不可区分，而它们的 BMI 与对象文件同名 —— 后者覆盖前者，存活的那个对象被送进链接两次。 -mcpp 拒绝这种情形，并点名两个包与各自的 interface 路径。检查的范围是一次 `build.mcpp` -能看见的那些规则，不是索引级的全局唯一性 —— `path` 依赖与私有 registry 本来就绕得开。 +并像任何工作空间一样由 mcpp 规划与构建：每个库按完整的包编译——它的包含目录、宏定义、 +特性、自己的依赖与实现单元都参与——`build.mcpp` 由同一个计划编译，因此 BMI 与导入它的 +程序在标准、方言与编译器上的一致由构造保证。多个程序使用的库只为它们编译一次，标准不同的 +程序在另一张图中。在该目录中执行 `mcpp build` 即可复现构建程序的编译。 -**`mcpp.` 前缀保留给由 mcpp 项目维护的规则。** 不在 `mcpp` 命名空间下的包声明该前缀 -的模块名时给出一条同时点名两者的警告，构建继续。之所以是警告：引擎判定不了谁是官方， -`path` 依赖、私有镜像与内部 fork 都合法，而且从这里看都一样。 +程序在其**身份**变化时编译：引擎版本、宿主工具链、标准、程序文本，以及其图中的每个包—— +名字、版本、来源、构建程序实例的特性，以及可以原地变化的源码树（`path` 包或工作空间成员） +的内容。因此，库所包含的头文件被修改时，程序会重新编译并重新运行。所有程序的身份都未变时， +不规划这张图。程序的运行按身份与它声明的输入缓存（*增量：声明的输入*）。 -lib 根必须在 `src/.cppm`（或 `[lib] path` 指向的位置）；缺失时报 -*"host module 'x': no interface unit at …"*。 +编译产物的存放位置遵循依赖缓存的规则（[04 §2.10](04-mcpp-toml.md)）： -#### 已编译 host 模块的存放位置（2026.10.1.1+） +| 包 | 存放位置 | 依据 | +|---|---|---| +| 引擎接口 `mcpp` 及其别名 `mcpp.core` | 全局缓存：`$MCPP_HOME/build-cache/v1/pkg/_engine/mcpp-build-module@//` | 文本属于引擎，每个版本写一次到 `$MCPP_HOME/engine/<版本>/modules/` | +| 源码位于不可变存储中的索引包 | 全局缓存：`…/pkg//@//` | 名字与版本确定其内容 | +| `path` 或 `git` 包、工作空间成员 | `/target/` 下的构建程序图 | 源码可以在名字与版本不变的情况下变化 | -内置的 `mcpp` 模块与程序 import 的 host 模块不再编译进每个程序自己的目录。每一个都是存储中 -的一个条目，条目的地址由进入它那次编译的一切决定：host 编译器的身份、standard 标志、该次 -编译携带的标志、它 import 的 BMI、interface 文件的 SHA-256、mcpp 版本（对内置模块）与提供者 -包（对 host 模块）。这些输入记录在条目的 `entry.json` 中，命中时逐字段比较，而不只比较哈希。 -两个程序的编译在所有这些上一致时共用一个条目；standard 或标志不同的程序有自己的条目，因此 -BMI 不会被与它不一致的编译取用。 +`mcpp why build-programs` 列出每张图、其中的程序、每个程序使用的库、编译所依据的身份与其 +二进制；`--verbose` 写出每个程序的身份是否未变。`mcpp emit build-database` 向编辑器描述 +这张图（SPEC-005：集合 `:build`、`:host` 与 `mcpp:core`）。 -条目放在哪里取决于它的文本从哪里来： +#### 模块名 -| 模块 | 存放位置 | 依据 | -|---|---|---| -| 内置的 `mcpp` 模块及其 `mcpp.core` 别名 | 全局缓存：`$MCPP_HOME/build-cache/v1/pkg/_engine/mcpp-build-module@//` | 引擎拥有这段文本，对同一 mcpp 版本与同一 host 编译器，它在每个工程中都相同 | -| 源码位于不可变 store 的索引包的 host 模块 | 全局缓存：`…/pkg//@//` | 依赖缓存的规则：名称与版本确定了字节 | -| `path` 或 `git` 依赖、或 workspace 成员的 host 模块 | `/target/.build-mcpp/host-modules//` | 它的源码可以在名称与版本不变的情况下改变 | - -缓存模式的作用与对依赖相同（见 [04 §2.10](04-mcpp-toml.md)）。`--cache global`（默认）下， -条目按上表放入全局缓存。`--cache local` 或 `--cache off` 下，所有条目都保存在 workspace 的 -存储里，同一次调用中的各程序仍共用它们。`path` 依赖的 host 模块绝不写入全局缓存。workspace -存储中条目的键还包含包源码树的摘要，因此修改 interface 所 include 的头文件会编出新条目， -旧条目原样保留。 - -全局缓存中的条目与依赖的条目一样：`mcpp cache list` 把内置模块显示为 -`_engine/mcpp-build-module@`，`mcpp cache verify` 检查它，`mcpp cache gc` 按最近 -使用回收它；升级 mcpp 会留下上一版本的条目，交给 `gc`。workspace 的存储随 `target/` 由 -`mcpp clean` 一并删除。程序看到的与以前相同：同样的模块，用同样的标志编译。 - -**一个包可以提供多条规则，由 feature 选择**(mcpp 2026.9.5.3+)。包解析后的 -`[build] sources` 里 —— 含 feature 加入的源文件 —— 每一个模块接口单元都以它自己声明的 -名字编成一个 host 模块，lib 根排在最前。feature 单元可以 import lib 根；除此之外每个 -单元单独编译，因此只 import `std` 与 `mcpp`。只有写在清单里的源文件参与：未声明 -`sources` 的包所推断出的 `src/**` 不被读取，所以此前发布的规则包暴露的仍是它当时暴露 -的那一个模块。 +**模块名是库的源码所声明的名字**（mcpp 2026.8.29.1+）。`export module acme.rules.protobuf;` +以 `acme.rules.protobuf` 导入，与包名无关。构建程序导入其所用库的公开模块（SPEC-008）： +接口根及其转出的模块，以及已启用特性加入的单元；导入其他导出模块时给出与目标构建相同的警告（W3）。 -```toml -# the collection's manifest -[build] -sources = ["src/plugins.cppm"] # export module mcpp.plugins; +**一个程序的两个库不得声明同一个模块名。** `import` 按模块寻址，两者对编译器不可区分。 +mcpp 在编译之前拒绝，写出两个包与两个接口文件的路径。 -[features] -rules-cuda = { sources = ["rules/cuda.cppm"] } # export module mcpp.rules.cuda; -rules-spirv = { sources = ["rules/spirv.cppm"] } # export module mcpp.rules.spirv; -``` +**`mcpp.` 前缀保留给 mcpp 项目维护的模块**（SPEC-007 R9.6）。命名空间不是 `mcpp` +的包使用该前缀时给出警告，写出两者，构建继续。 -```toml -# a consumer -[build-dependencies.mcpp] -plugins = { version = "0.3.0", features = ["rules-spirv"], host-module = true } -``` +构建程序导入没有库提供的模块、仅作为另一个库的依赖到达的库的模块，或未启用特性之后的模块时， +在任何编译之前按名字拒绝。 -**用 `[build-dependencies]` 而不是 `[dependencies]`** —— 规则包正是 [04 §2.6.1](04-mcpp-toml.md) 描述的那种 -情形：它的库绝不该到达目标，而它的规则仍然被需要。两条轴是分开的： -`host-module = true` 说的是**要哪一种构建期产物**，而 section 说的是**这个包是否到达 -目标**；规则包在第二条轴上的答案是"否"，而 section 就是说这件事的地方。写在 -`[dependencies]` 里同样能工作 —— 这恰恰是为什么这条区分必须被**陈述**，而不能指望由 -一次失败来教会。 +#### 构建插件 -模块集合就是 feature 集合：feature 未激活的单元不编译，import 它会以未知模块失败。 -`mcpp:plugins` 是 mcpp 项目维护的集合（仓库 `mcpp-community/mcpp-plugins`）；其成员 -命名为 `mcpp.rules.`（规则包）与 `mcpp.tools.`（构建期工具）。 +导入引擎接口（`mcpp.core` 或 `mcpp`）的单元是**构建插件单元**，不需要另行声明。 +引擎接口只存在于构建程序配置中，因此： -*仅构建期：* `host-module = true` 的依赖**不会**被编进、也不会被链进本工程的 target, -它所依赖的东西也不会。它只在 `build.mcpp` 期间运行，别处都不出现。（2026.8.5.2 之前 -它还会被当作普通库再编一遍，这正是规则里 `import mcpp;` 失败的原因：在那第二次编译里 -内置模块并不存在。2026.8.29.1 之前被排除的只有规则本身，它自己的 `[dependencies]` -仍会被编译并链进消费者的二进制，而规则却 import 不到它们。） +- 插件可以在 **`[dependencies]`** 中依赖其他插件与普通库：构建程序使用该插件时， + 这些依赖同样是构建程序实例； +- 库中含有插件单元的包作为构建的根（`mcpp build`、`mcpp test`、被选中的成员）时， + 以构建程序配置、为本机规划。它的单元测试是 `tests/` 下可以导入 `mcpp` 的普通测试； + 指向他处的 `--target` 被拒绝； +- 目标中出现插件单元（工程经 `[dependencies]` 到达插件）时，在规划期拒绝，写出依赖链 + 与应当移入的段； +- `mcpp pack` 拒绝插件：插件以源码分发，由构建程序用自己的工具链编译。 -### 依赖另一个规则的规则（mcpp 2026.8.29.1+） +#### 特性之后的多个模块 -规则在自己的 `[build-dependencies]` 里声明所需之物，并可以 import 其中标了 -`host-module = true` 的条目： +一个包可以提供多条规则，每条位于一个特性之后（mcpp 2026.9.5.3+）。特性加入的单元是该 +特性的公开模块（SPEC-008 I2），`rule_module` 写出该特性合成程序所导入的模块，它必须由 +该特性的某个单元声明： ```toml -# inside the rule package's manifest -[build-dependencies] -globbing = { path = "../globbing", host-module = true } +# 集合的清单 +[build] +sources = ["src/plugins.cppm"] # export module mcpp.plugins; + +[features] +rules-cuda = { sources = ["rules/cuda.cppm"], rule_module = "mcpp.rules.cuda" } +rules-spirv = { sources = ["rules/spirv.cppm"], rule_module = "mcpp.rules.spirv" } ``` -```cpp -// the rule's own interface -export module tidyrule; -import std; -import mcpp; -import globbing; +```toml +# 消费方 +[build-dependencies.mcpp] +plugins = { version = "0.3.0", features = ["rules-spirv"] } ``` -mcpp 先编译内层规则，同一条命令、同一套 flag，因此 BMI 的一致性仍是结构性事实而不是 -需要事后校验的性质。 +模块集合即特性集合：特性未启用的单元不编译，导入它时按特性名拒绝。`mcpp:plugins` 是 +mcpp 项目维护的集合（仓库 `mcpp-community/mcpp-plugins`）。 -消费者**不可以** import `globbing`：构建期的 provision 只在 `reexport = true` 的边上 -再跨一跳，而 mcpp 自己执行这条规则，不交给编译器 —— 在 GCC 上那个 import 会成功， -然后在别人的机器上失败。 +#### 为兼容保留的读法 + +mcpp 2026.10.11.1 之前的两种写法保留其含义并被弃用（51 —— 支持的版本）： + +| 写法 | 读作 | 移除版本 | +|---|---|---| +| `[dependencies] x = { host-module = true }`，或请求 `rule_module` 特性的 `[dependencies]` 条目 | `[build-dependencies]` 中的同一条目（`host-module-in-dependencies`） | mcpp 2027.1 | +| 插件的库导入其自身 `[build-dependencies]` 中以 `host-module = true` 写出的包 | 该条目同时作为插件的 `[dependencies]`（`plugin-imports-from-build-dependencies`） | mcpp 2027.1 | -*限制：* 每个 host 模块只有一个接口单元。带实现单元或多个模块的库还不能作为规则的 -构建期依赖。 +每次使用在清单可编辑处给出警告，`mcpp why compat` 列出所有使用。 ### `reexport = true` —— 由库替用户拉起整条工具链（2026.8.6.2+） diff --git a/docs/zh/31-authoring-a-rule-package.md b/docs/zh/31-authoring-a-rule-package.md index f7efc736..0334c52e 100644 --- a/docs/zh/31-authoring-a-rule-package.md +++ b/docs/zh/31-authoring-a-rule-package.md @@ -112,12 +112,11 @@ device_extensions = [".toy"] | 键 | 对消费方的作用 | |---|---| | `sources` | 只有 feature 激活时才编译该模块 | -| `rule_module`(2026.9.7.1+) | 消费方构建程序 import 的模块。它隐含 `host-module = true` | +| `rule_module`(2026.9.7.1+) | 消费方构建程序 import 的模块；请求该特性即请求包的库。它必须由该特性的某个单元声明（2026.10.11.1+） | | `device_extensions`(2026.9.7.1+) | 这些扩展名被归类为**设备源**：不做 import 扫描、不产出 BMI，且没有 action 认领时被拒绝 | 只做生成或检查、不编译某种语言的规则不声明 `device_extensions`; -`08-build-rules` 就是这个形状，它的消费方在依赖边上自己写 -`host-module = true`。 +`08-build-rules` 就是这个形状，它的消费方把这个包写在 `[build-dependencies]` 中。 设备扩展名不在默认的 source glob 里。消费方通过点名这些文件来选入： @@ -157,6 +156,24 @@ bool compile(options opt = {}); 后端的构建把两边的源都放进同一个列表，而构建程序里的每一条规则读到的是 同一个值。 +## 规则包自己的依赖、构建与测试（mcpp 2026.10.11.1+） + +规则的模块导入引擎接口 `mcpp`，这使它成为**构建插件**（30 —— `build.mcpp`，*构建插件*）： +只存在于构建程序配置中的包。对作者而言有三个推论。 + +- **规则的库所导入的东西写在它的 `[dependencies]` 中。** 基于另一条规则（例如 + `mcpp:plugins` 的 `plugins-core`）或普通库（例如 TOML 解析器）的规则在那里写出它。消费方的 + 构建程序使用这条规则时，这些依赖与它一起为构建程序编译。规则的 `[build-dependencies]` + 服务规则包自己的 `build.mcpp`，与任何包相同。 +- **规则包可以作为构建的根构建与测试。** 在包中执行 `mcpp build` 会为本机编译它的库并带上 + 引擎接口，`mcpp test` 以同样的方式运行 `tests/` 下的测试：测试可以导入规则的模块与 `mcpp`， + 在没有消费方工程的情况下检查规则的逻辑。 +- **它以源码发布。** `mcpp pack` 拒绝构建插件；消费方的构建用其构建程序的工具链编译它。 + +本版本之前编写、从自己的 `[build-dependencies]`（`host-module = true`）导入包的规则包照常构建： +该条目同时被读作规则的依赖，并提示作者移动它（`plugin-imports-from-build-dependencies`， +51 —— 支持的版本）。 + ## 声明工作：`mcpp::action` ```cpp @@ -328,8 +345,8 @@ mcpp::floor("cuda.driver", runtime_needs); 声明它 —— 并且在查找返回空时把这件事说出来，而不是指向一份读者并不拥有的 manifest。 -host module 是这条规则的例外，它的声明在它被编入的**每一个**构建程序里 -都可见（`tests/e2e/622`），因为 host module 自己的代码，是作为那些构建 +构建程序的库是这条规则的例外，它的声明在使用它的**每一个**构建程序里 +都可见（`tests/e2e/622`），因为库自己的代码，是作为那些构建 程序各自的一部分运行的，而不是作为一个只经由依赖图才能看到的依赖。这正是 一个成员要在自己身上声明自己的载荷、而拉入这个成员的项目从不重复这条声明 的原因。dist 成员的载荷因此也归属于该成员 —— 当工具服务于某一个 target diff --git a/docs/zh/50-machine-output.md b/docs/zh/50-machine-output.md index 8cd9ef91..6826244b 100644 --- a/docs/zh/50-machine-output.md +++ b/docs/zh/50-machine-output.md @@ -271,7 +271,8 @@ mcpp self env --format json "config": "/home/u/.mcpp/config.toml", "buildCache": "/home/u/.mcpp/build-cache/v1", "mcppVersion": "2026.8.8.3", - "defaultToolchain": "gcc@16.1.0" // 2026.9.28.2+ + "defaultToolchain": "gcc@16.1.0", // 2026.9.28.2+ + "engineModules": "/home/u/.mcpp/engine/2026.10.11.1/modules" // 2026.10.11.1+ } ``` @@ -350,6 +351,23 @@ mcpp why sources|tool |payload --format json 每次构建把同一份记录写入 `target///resolution.json` 的 `sources`。 +### `mcpp.why.build-programs` 与 `mcpp.why.compat` *(mcpp 2026.10.11.1+)* + +``` +mcpp why build-programs --format json +mcpp why compat --format json +``` + +两者都像 `mcpp why` 一样准备构建并报告它记录的内容；`data` 带 `status`（`ok` 或 `refused`），以及： + +| kind | 字段 | 内容 | +|---|---|---| +| `mcpp.why.build-programs` | `graphs[]` | 每张构建程序图一项：`workspace`（其目录）与 `programs[]`，每个程序为 `{member, package, source, libraries[], identity, binary}` | +| `mcpp.why.compat` | `uses[]` | 每次使用弃用读法一项：`{id, where, detail, warned, since, floor, canonical, docs}`；在构建不编辑的清单中 `warned` 为 false | + +`mcpp.env` 的 `engineModules` 是本版本 mcpp 的引擎接口包的目录，构建程序图依赖它（30 —— +`build.mcpp`）；第一次编译构建程序之后它才存在。 + ### `mcpp.toolchain.list` —— 已安装的工具链，以及这台宿主能服务的目标 ``` diff --git a/docs/zh/51-supported-versions.md b/docs/zh/51-supported-versions.md index 11df5ecf..26055ad8 100644 --- a/docs/zh/51-supported-versions.md +++ b/docs/zh/51-supported-versions.md @@ -35,7 +35,7 @@ schema，那是 [50 —— 机器可读输出](50-machine-output.md)。 | 面 | 稳定性 | |---|---| -| `mcpp.toml` 的键 | 只增。既有键保持原有含义；无法识别的键会被报告，不会被静默忽略 | +| `mcpp.toml` 的键 | 只增。既有键保持原有含义；无法识别的键会被报告，不会被静默忽略。例外是下文的弃用 | | CLI 命令与旗标 | 只增。被替换的拼写保留为别名 | | 机器输出（`--message-format json`） | 由 `schemaVersion` 版本化，见 [50](50-machine-output.md) | | `build.mcpp` 指令协议 | 版本化，见 `kProtocolVersion`。引擎遇到声明版本**高于**自身的程序会拒绝，而不是猜测 | @@ -47,6 +47,25 @@ schema，那是 [50 —— 机器可读输出](50-machine-output.md)。 构建程序的**编译**阶段，而不是通过协议错误。协议号管的是线上的指令；类型化 API 由所装的引擎决定。两者都写在这里，是因为包作者看到的失败取决于他越过了哪一条边界。 +## 弃用 + +清单中将被移除的写法或读法先被弃用；"已有的键保持其含义"的例外，恰好是本表的各行 +（mcpp 2026.10.11.1+）。 + +- 被弃用的写法照常工作。构建可以编辑的清单（根、工作空间成员、工作空间内的 path 包） + 中的每次使用都给出警告，写出取代它的写法、弃用它的版本与移除它的版本；`--strict` 下拒绝。 + `mcpp why compat` 列出所有使用，包括依赖中的使用。 +- 移除以 CalVer 月份表示。弃用的版本与该月份之间至少相隔两个完整的日历月，读法在该月的 + 第一个版本或之后移除。 +- 本表由引擎中的唯一登记表（`modules/manifest/src/compat/registry.cppm`）生成；某个版本 + 到达某一行的月份而读法仍在时，引擎自己的测试失败。 + +| 读法 | 弃用版本 | 移除版本 | 应写作 | +|---|---|---|---| +| `workspace-position`：工作空间根自己的 `[toolchain]`、`[indices]`、`[profile.]`、`[target.]` 标量或 `[xlings]` 按位置到达成员 | 2026.10.10.1 | 2027.1 | `[workspace.toolchain]`、`[workspace.indices]`、`[workspace.profile.]`、`[workspace.target.]`、`[workspace.xlings]` | +| `host-module-in-dependencies`：`[dependencies]` 中请求 `host-module = true` 或 `rule_module` 特性的条目，按构建期依赖读取 | 2026.10.11.1 | 2027.1 | `[build-dependencies]` 中的同一条目 | +| `plugin-imports-from-build-dependencies`：构建插件的库导入其自身 `[build-dependencies]` 中以 `host-module = true` 写出的包 | 2026.10.11.1 | 2027.1 | 插件 `[dependencies]` 中的该包 | + ## 复现一次构建 `mcpp.lock` 记录一次构建解析出的结果。`--locked` 断言当前解析与它一致，不一致 diff --git a/examples/08-build-rules/README.md b/examples/08-build-rules/README.md index b4ee7bbb..f055b42a 100644 --- a/examples/08-build-rules/README.md +++ b/examples/08-build-rules/README.md @@ -39,7 +39,9 @@ behind a linter costs more than it saves, and a failing check fails the build either way; `blocking = true` is for the case where a failure means the compile was wasted. -**`rules-tidy` depends on `rules-embed`** through its own -`[build-dependencies]`, which is what lets one rule reuse another instead of -copying it. The consumer cannot import `rules-embed` through that edge — build -time provisions cross one further edge only when the edge says `reexport`. +**`rules-tidy` depends on `rules-embed`** through its own `[dependencies]`, +which is what lets one rule reuse another instead of copying it: the library of +a build plugin imports what its dependencies provide, and they are compiled +with it for each build program that uses it. The consumer's `build.mcpp` +imports `rules-embed` as well, so the consumer names it in its own +`[build-dependencies]`. diff --git a/examples/08-build-rules/app/mcpp.toml b/examples/08-build-rules/app/mcpp.toml index f7798182..7da632c4 100644 --- a/examples/08-build-rules/app/mcpp.toml +++ b/examples/08-build-rules/app/mcpp.toml @@ -3,7 +3,8 @@ name = "app" version = "0.1.0" # Both rules are BUILD-time only: neither reaches this binary, and neither does -# anything they depend on. -[dependencies] -example.rules-tidy = { path = "../rules-tidy", host-module = true } -example.rules-embed = { path = "../rules-embed", host-module = true } +# anything they depend on. A [build-dependencies] entry gives build.mcpp the +# package's library. +[build-dependencies] +example.rules-tidy = { path = "../rules-tidy" } +example.rules-embed = { path = "../rules-embed" } diff --git a/examples/08-build-rules/rules-tidy/mcpp.toml b/examples/08-build-rules/rules-tidy/mcpp.toml index 6615f1e7..4a142650 100644 --- a/examples/08-build-rules/rules-tidy/mcpp.toml +++ b/examples/08-build-rules/rules-tidy/mcpp.toml @@ -5,9 +5,9 @@ version = "0.1.0" description = "Per-file source analysis as build-graph nodes (role = check)" license = "Apache-2.0" -# This rule reuses another rule rather than copying it. `[build-dependencies]` -# is the only declaration site a package owns: `tools` and `host-module` are -# written by the CONSUMER on an edge, so without this a rule could not request -# anything on its own behalf — which is why rules used to be leaves. -[build-dependencies] -example.rules-embed = { path = "../rules-embed", host-module = true } +# This rule reuses another rule rather than copying it. The rule's library +# imports `example.rules.embed`, so the package is one of its [dependencies]: +# a build plugin's dependencies are compiled with it for the build program +# that uses it. +[dependencies] +example.rules-embed = { path = "../rules-embed" } diff --git a/examples/09-heterogeneous/README.md b/examples/09-heterogeneous/README.md index 9d53ba72..815f781a 100644 --- a/examples/09-heterogeneous/README.md +++ b/examples/09-heterogeneous/README.md @@ -104,7 +104,7 @@ selects it and the accelerator it serves, so a project writes one edge and no ```toml [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-cuda", "tools-island"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-cuda", "tools-island"] } ``` `multi-backend/` is the one example here that also pins a version, and it does diff --git a/examples/09-heterogeneous/boundary/mcpp.toml b/examples/09-heterogeneous/boundary/mcpp.toml index 1d1eebe1..06a42a3c 100644 --- a/examples/09-heterogeneous/boundary/mcpp.toml +++ b/examples/09-heterogeneous/boundary/mcpp.toml @@ -6,9 +6,9 @@ version = "0.1.0" standard = "c++23" # `tools-island` is not a device rule: it claims no extension and names no rule -# module, so the edge states `host-module = true` itself. +# module. A [build-dependencies] entry asks for the library all the same. [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["tools-island"], host-module = true } +plugins = { version = "0.5.2", features = ["tools-island"] } [build] sources = ["src/*.cpp", "src/kernels/**/*.c"] diff --git a/examples/09-heterogeneous/cann/app/mcpp.toml b/examples/09-heterogeneous/cann/app/mcpp.toml index 93505dcc..14f0b7e6 100644 --- a/examples/09-heterogeneous/cann/app/mcpp.toml +++ b/examples/09-heterogeneous/cann/app/mcpp.toml @@ -14,7 +14,7 @@ import_std = true # imports it as `mcpp.rules.ascendc`. `[build-dependencies]`, because a rule # package's library must never reach the target while its rule is wanted. [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-ascendc"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-ascendc"] } # NO [xlings.workspace]. `mcpp.rules.ascendc` declares `xim:cann-toolkit` # itself, gated on this accelerator and on the feature that selects the rule -- diff --git a/examples/09-heterogeneous/cuda/README.md b/examples/09-heterogeneous/cuda/README.md index 766b8c04..dcfa2a67 100644 --- a/examples/09-heterogeneous/cuda/README.md +++ b/examples/09-heterogeneous/cuda/README.md @@ -98,7 +98,7 @@ The rule names it. This project writes one edge and no payload list at all: ```toml [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-cuda", "tools-island"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-cuda", "tools-island"] } ``` `mcpp.rules.cuda` declares nvcc, cudart, cuRAND's headers, CCCL and the driver diff --git a/examples/09-heterogeneous/cuda/app/mcpp.toml b/examples/09-heterogeneous/cuda/app/mcpp.toml index ca89df37..3882992a 100644 --- a/examples/09-heterogeneous/cuda/app/mcpp.toml +++ b/examples/09-heterogeneous/cuda/app/mcpp.toml @@ -23,7 +23,7 @@ default = "llvm@23.1.3" # `extern "C"` boundary the island's compiler reads and the module the seam # imports, so neither signature is stated twice. [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-cuda", "tools-island"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-cuda", "tools-island"] } # The driver's userspace library, reached through an index package that owns # the one hop mcpp needs: a directory on the artifact's runtime search path. diff --git a/examples/09-heterogeneous/hip/app/mcpp.toml b/examples/09-heterogeneous/hip/app/mcpp.toml index 1bf1b164..be1938ea 100644 --- a/examples/09-heterogeneous/hip/app/mcpp.toml +++ b/examples/09-heterogeneous/hip/app/mcpp.toml @@ -20,7 +20,7 @@ import_std = true default = "llvm@23.1.3" [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-hip"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-hip"] } # The driver's userspace library. HIP reaches the device through the CUDA # runtime here, so this is the same one hop the CUDA consumer needs: mcpp's diff --git a/examples/09-heterogeneous/multi-backend/README.md b/examples/09-heterogeneous/multi-backend/README.md index 4a9cfc08..de09e1e8 100644 --- a/examples/09-heterogeneous/multi-backend/README.md +++ b/examples/09-heterogeneous/multi-backend/README.md @@ -93,7 +93,7 @@ saying it after the vocabulary grows. ```toml [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-cuda", "rules-spirv"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-cuda", "rules-spirv"] } ``` Two rules, in one build program, which is what an additive-backend package @@ -106,6 +106,7 @@ action, naming the file. `[build-dependencies]`, not `[dependencies]`: a rule package's library must never reach the target while its rule is still wanted, which is the case -docs/05 section 2.6.1 exists for. `host-module = true` says which build-time -product is wanted; the section says whether the package reaches the target. -Two axes, and a rule package answers "no" on the second. +docs/05 section 2.6.1 exists for. The section says whether the package reaches +the target; the entry says which build-time products are wanted, and an entry +that names none asks for the library. Two axes, and a rule package answers "no" +on the first. diff --git a/examples/09-heterogeneous/multi-backend/mcpp.toml b/examples/09-heterogeneous/multi-backend/mcpp.toml index 2b440aa1..4953a062 100644 --- a/examples/09-heterogeneous/multi-backend/mcpp.toml +++ b/examples/09-heterogeneous/multi-backend/mcpp.toml @@ -34,13 +34,13 @@ import_std = true [toolchain] default = "llvm@23.1.3" -# BOTH rules, in one build program. `host-module = true` compiles their module -# interfaces for the build program to import; `[build-dependencies]` keeps the -# package out of the target, which is the case docs/05 section 2.6.1 exists for. +# BOTH rules, in one build program. A `[build-dependencies]` entry gives the +# build program the package's library and keeps the package out of the target, +# which is the case docs/05 section 2.6.1 exists for. # The rules are declared unconditionally because `build.mcpp` imports them # unconditionally -- each returns immediately when its own backend is absent. [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-cuda", "rules-spirv"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-cuda", "rules-spirv"] } # ── the payloads, gated on the device they are for ────────────────────────── # diff --git a/examples/09-heterogeneous/sycl/app/mcpp.toml b/examples/09-heterogeneous/sycl/app/mcpp.toml index cc69a753..38720353 100644 --- a/examples/09-heterogeneous/sycl/app/mcpp.toml +++ b/examples/09-heterogeneous/sycl/app/mcpp.toml @@ -20,7 +20,7 @@ import_std = true default = "llvm@23.1.3" [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-sycl", "tools-island"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-sycl", "tools-island"] } # The SYCL runtime, on the artifact's runtime search path. mcpp's private # loader does not consult /usr/lib, so `libsycl.so.9` -- which the rule diff --git a/examples/09-heterogeneous/vulkan/app/mcpp.toml b/examples/09-heterogeneous/vulkan/app/mcpp.toml index 627ce12c..913b2376 100644 --- a/examples/09-heterogeneous/vulkan/app/mcpp.toml +++ b/examples/09-heterogeneous/vulkan/app/mcpp.toml @@ -13,7 +13,7 @@ import_std = true # The rule that compiles the shaders lives in the official plugin collection, # selected by its feature; `build.mcpp` imports it as `mcpp.rules.spirv`. [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-spirv"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-spirv"] } # The Khronos loader, built by the index rather than taken from the host, and # the adapter that makes the host's own ICDs reachable from a binary running diff --git a/examples/10-graphics/README.md b/examples/10-graphics/README.md index 6c80891b..2f5308d0 100644 --- a/examples/10-graphics/README.md +++ b/examples/10-graphics/README.md @@ -53,7 +53,7 @@ tells the two apart, which is why it is printed. ```toml [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-spirv"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-spirv"] } [build] accel = "vulkan1.2" diff --git a/examples/10-graphics/offscreen/mcpp.toml b/examples/10-graphics/offscreen/mcpp.toml index 46468dc1..40d80f9c 100644 --- a/examples/10-graphics/offscreen/mcpp.toml +++ b/examples/10-graphics/offscreen/mcpp.toml @@ -13,7 +13,7 @@ import_std = true # One edge. `mcpp.rules.spirv` declares the shader compiler it drives, so this # project names no payload for it. [build-dependencies.mcpp] -plugins = { version = "0.5.2", features = ["rules-spirv"], host-module = true } +plugins = { version = "0.5.2", features = ["rules-spirv"] } # The Khronos loader. Unconditional, and the reason is a rule of the engine # rather than a preference: A DEPENDENCY CANNOT BE CONDITIONED ON A LAYER. diff --git a/examples/12-a-new-device-language/README.md b/examples/12-a-new-device-language/README.md index daf4703c..52aa6f67 100644 --- a/examples/12-a-new-device-language/README.md +++ b/examples/12-a-new-device-language/README.md @@ -122,13 +122,13 @@ device_extensions = [".toy"] | key | effect | |---|---| | `device_extensions` | a consumer that activates the feature gets `.toy` classified as a **device source**: never scanned for imports, never producing a BMI, and refused if no action claims it | -| `rule_module` | the module the consumer's build program imports. It implies `host-module = true`, so the consumer writes the feature and nothing else | +| `rule_module` | the module the consumer's build program imports; one of the feature's units declares it. Requesting the feature asks for the package's library, so the consumer writes the feature and nothing else | Device extensions are not in the default source glob. A `.toy` is compiled because the manifest names it: ```toml -[dependencies] +[build-dependencies] example.rules-toy = { path = "../rules-toy", features = ["rules-toy"] } [build] @@ -159,8 +159,9 @@ That gating was measured. With `features = ["rules-toy"]` removed from the consumer, the build stops before any tool is built: ``` -error: build.mcpp imports 'example.rules.toy', and no dependency provides it as a host module. - declared without `host-module = true`: rules-toy (in [dependencies]) +error: build.mcpp imports 'example.rules.toy' + provided by: example.rules-toy, feature "rules-toy" (not enabled) + hint: enable it on the dependency edge: example.rules-toy = { ..., features = ["rules-toy"] } ``` and the tool store holds no `toyc` entry afterwards. diff --git a/examples/12-a-new-device-language/app/mcpp.toml b/examples/12-a-new-device-language/app/mcpp.toml index 0db8696f..da06d146 100644 --- a/examples/12-a-new-device-language/app/mcpp.toml +++ b/examples/12-a-new-device-language/app/mcpp.toml @@ -5,10 +5,10 @@ version = "0.1.0" [language] standard = "c++23" -# ONE EDGE. `rule_module` on the feature implies `host-module = true`, and the -# feature's own `[feature-deps]` brings the compiler, so the consumer names the -# feature and nothing else. -[dependencies] +# ONE EDGE. A [build-dependencies] entry gives the build program the package's +# library, and the feature's own `[feature-deps]` brings the compiler, so the +# consumer names the feature and nothing else. +[build-dependencies] example.rules-toy = { path = "../rules-toy", features = ["rules-toy"] } # `.toy` IS NOT IN THE DEFAULT SOURCE GLOB. Device sources are opted into by diff --git a/examples/12-a-new-device-language/rules-toy/mcpp.toml b/examples/12-a-new-device-language/rules-toy/mcpp.toml index 70bfa76c..e13f857f 100644 --- a/examples/12-a-new-device-language/rules-toy/mcpp.toml +++ b/examples/12-a-new-device-language/rules-toy/mcpp.toml @@ -13,8 +13,8 @@ license = "Apache-2.0" # it. # # `rule_module` says which module the consumer's build program imports to reach -# the rule. It also implies `host-module = true`, so a consumer writes the -# feature and nothing else. +# the rule; one of the feature's units declares it. Requesting the feature asks +# for the package's library, so a consumer writes the feature and nothing else. # # `.toy` is not in the engine's built-in table and never will be. The engine # knows what a device source IS; it does not know that `.toy` is anything. diff --git a/mcpp.toml b/mcpp.toml index 3b925192..707abb72 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.10.10.2" +version = "2026.10.11.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/manifest/src/compat.cppm b/modules/manifest/src/compat.cppm index ebe915fb..dd8832e2 100644 --- a/modules/manifest/src/compat.cppm +++ b/modules/manifest/src/compat.cppm @@ -3,11 +3,14 @@ // explicit retirement boundary. // // DEPRECATION SCHEDULE: -// The shims in this module are slated for removal in mcpp 1.0.0. -// Projects should migrate to the canonical forms before that point: +// A reading a user can write and that is to be removed is a row of the +// deprecation registry (compat/registry.cppm), which states the release that +// deprecated it and the CalVer month from which it is removed. The shims +// below that no manifest can spell carry `COMPAT(internal:)` and +// promise nothing. The readings this module serves: // // ┌─────────────────────────────────────┬──────────────────────────────┐ -// │ Deprecated (works until 1.0) │ Canonical (use now) │ +// │ Deprecated │ Canonical (use now) │ // ├─────────────────────────────────────┼──────────────────────────────┤ // │ name = "mcpplibs.cmdline" │ namespace = "mcpplibs" │ // │ (namespace embedded in name) │ name = "cmdline" │ @@ -158,9 +161,9 @@ inline std::string xpkg_dir_name(std::string_view indexName, // The FIRST candidate is the canonical form; subsequent candidates are // backward-compatibility fallbacks. // -// DEPRECATION SCHEDULE (fallback candidates): -// Fallback candidates are slated for removal in mcpp 1.0.0. -// Package indices should use the canonical filename form by then. +// Fallback candidates (COMPAT(internal:xpkg-fallback-candidates)): index +// files written before the canonical filename form. Package indices should +// use the canonical form. inline std::vector xpkg_lua_candidates(std::string_view ns, std::string_view shortName) @@ -179,7 +182,7 @@ inline std::vector xpkg_lua_candidates(std::string_view ns, candidates.push_back(fqname + ".lua"); } - // ── Fallback candidates (COMPAT, remove in 1.0.0) ────────────── + // ── Fallback candidates (COMPAT(internal:xpkg-fallback-candidates)) ────────────── // Fallback: full qualified name when it differs from qname // (e.g. default-ns "mcpplibs": qname="cmdline", fqname="mcpplibs.cmdline") @@ -245,7 +248,7 @@ inline std::vector install_dir_candidates(std::string_view ns, candidates.push_back(std::format("{}-x-{}", ns, fqname)); } - // ── Fallback candidates (COMPAT, remove in 1.0.0) ────────────── + // ── Fallback candidates (COMPAT(internal:xpkg-fallback-candidates)) ────────────── // Old index-prefixed layout: -x-. candidates.push_back(std::format("{}-x-{}", indexName, fqname)); diff --git a/modules/manifest/src/compat/legacy.cppm b/modules/manifest/src/compat/legacy.cppm index 3db0574a..628177e4 100644 --- a/modules/manifest/src/compat/legacy.cppm +++ b/modules/manifest/src/compat/legacy.cppm @@ -1,8 +1,8 @@ // mcpp.pm.compat.legacy - legacy dependency-key compatibility. // // COMPAT: This module exists only to keep pre-namespace dependency syntax -// working during migration. It is slated for removal in mcpp 1.0.0. -// TODO(mcpp 1.0.0): remove this module and the legacy dotted-key parser. +// working during migration: COMPAT(internal:legacy-dotted-keys), which no +// deprecation registry row schedules yet. // // Deprecated: // diff --git a/modules/manifest/src/compat/registry.cppm b/modules/manifest/src/compat/registry.cppm index 26fffb3d..66d52d23 100644 --- a/modules/manifest/src/compat/registry.cppm +++ b/modules/manifest/src/compat/registry.cppm @@ -81,7 +81,8 @@ inline std::string deprecation_clause(const Deprecation& d) { } // The `(year, month)` of a version or of a floor: the first two dot-separated -// numbers. Both are CalVer, so their order is the order of these pairs. +// numbers, a four-digit year and a month 1-12. Both are CalVer, so their order +// is the order of these pairs; anything else (a SemVer string) has none. inline std::optional> calver_month(std::string_view v) { int parts[2] = {0, 0}; for (int i = 0; i < 2; ++i) { @@ -93,6 +94,7 @@ inline std::optional> calver_month(std::string_view v) { if (i == 0 && dot == std::string_view::npos) return std::nullopt; v = dot == std::string_view::npos ? std::string_view{} : v.substr(dot + 1); } + if (parts[0] < 1000 || parts[0] > 9999 || parts[1] < 1 || parts[1] > 12) return std::nullopt; return std::pair{parts[0], parts[1]}; } diff --git a/modules/manifest/tests/test_compat_registry.cpp b/modules/manifest/tests/test_compat_registry.cpp new file mode 100644 index 00000000..55c781a1 --- /dev/null +++ b/modules/manifest/tests/test_compat_registry.cpp @@ -0,0 +1,78 @@ +#include + +import std; +import mcpp.pm.compat.registry; +import mcpp.version; + +// THE DEPRECATION REGISTRY (mcpp#797, docs/51 §Deprecation). Every reading a +// user can write and that is to be removed is a row; the warning, `mcpp why +// compat`, docs/51 and the removal itself are derived from it. These tests +// are the part of that derivation the compiler can hold the code to. + +using mcpp::pm::compat::calver_month; +using mcpp::pm::compat::kDeprecations; + +// THE FORCING FUNCTION. Once mcpp reaches a row's floor, the reading must be +// gone from the code, not merely past its date: this fails the build of the +// release that would otherwise ship it, naming the row. +TEST(CompatRegistry, NoReadingOutlivesItsFloor) { + for (auto const& d : kDeprecations) + EXPECT_FALSE(mcpp::pm::compat::past_floor(d, mcpp::MCPP_VERSION)) + << "mcpp " << mcpp::MCPP_VERSION << " is at or past the floor " << d.floor + << " of COMPAT(" << d.id << "): remove the reading and its row"; +} + +// docs/51: at least two whole calendar months between the release that +// deprecates a reading and the floor from which it is removed. +TEST(CompatRegistry, EveryFloorLeavesTwoWholeMonths) { + for (auto const& d : kDeprecations) { + auto since = calver_month(d.since); + auto floor = calver_month(d.floor); + ASSERT_TRUE(since && floor) << d.id; + const int months = (floor->first - since->first) * 12 + (floor->second - since->second); + // The month of `since` itself is not whole: two more must follow it. + EXPECT_GE(months, 3) << d.id << ": " << d.since << " -> " << d.floor; + } +} + +TEST(CompatRegistry, RowsAreCompleteAndUnique) { + std::set ids; + for (auto const& d : kDeprecations) { + EXPECT_TRUE(ids.insert(d.id).second) << "duplicate id " << d.id; + EXPECT_FALSE(d.what.empty()) << d.id; + EXPECT_FALSE(d.canonical.empty()) << d.id; + EXPECT_FALSE(d.domain.empty()) << d.id; + EXPECT_TRUE(d.docs.starts_with("docs/")) << d.id; + EXPECT_NE(mcpp::pm::compat::find_deprecation(d.id), nullptr); + } + EXPECT_EQ(mcpp::pm::compat::find_deprecation("no-such-reading"), nullptr); +} + +TEST(CompatRegistry, TheClauseNamesBothVersions) { + auto const* d = mcpp::pm::compat::find_deprecation("workspace-position"); + ASSERT_NE(d, nullptr); + EXPECT_EQ(mcpp::pm::compat::deprecation_clause(*d), + "deprecated since mcpp 2026.10.10.1 and removed from mcpp 2027.1"); +} + +TEST(CompatRegistry, CalverMonthOrdersVersionsAndFloors) { + EXPECT_EQ(calver_month("2026.10.11.1"), (std::pair{2026, 10})); + EXPECT_EQ(calver_month("2027.1"), (std::pair{2027, 1})); + EXPECT_FALSE(calver_month("1.0.0-rc").has_value()); + mcpp::pm::compat::Deprecation d{"x", "2026.10.1.1", "2027.1", "w", "c", "d", "docs/x"}; + EXPECT_FALSE(mcpp::pm::compat::past_floor(d, "2026.12.31.9")); + EXPECT_TRUE(mcpp::pm::compat::past_floor(d, "2027.1.2.1")); + EXPECT_TRUE(mcpp::pm::compat::past_floor(d, "2028.3.1.1")); +} + +TEST(CompatRegistry, UsesAreRecordedOnceAndKeepTheWarning) { + mcpp::pm::compat::record_use("workspace-position", "/w/mcpp.toml", "[toolchain]", false); + mcpp::pm::compat::record_use("workspace-position", "/w/mcpp.toml", "[toolchain]", true); + std::size_t n = 0; + for (auto const& u : mcpp::pm::compat::recorded_uses()) + if (u.where == "/w/mcpp.toml" && u.detail == "[toolchain]") { + ++n; + EXPECT_TRUE(u.warned); + } + EXPECT_EQ(n, 1u); +} diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index 0736d4ac..7dd9bb7b 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.10.10.2"; +inline constexpr std::string_view MCPP_VERSION = "2026.10.11.1"; } // namespace mcpp diff --git a/src/build/build_database.cppm b/src/build/build_database.cppm index 60c7592d..5ee0132d 100644 --- a/src/build/build_database.cppm +++ b/src/build/build_database.cppm @@ -425,10 +425,14 @@ Rendered render(std::span members, : "library"; std::string setName = member.setPrefix + package + (isTest ? ":test" : ""); std::string setKind = kind; + std::string family = package; if (member.buildPrograms) { if (auto it = member.programOf.find(package); it != member.programOf.end()) { + // The program belongs to the package whose build.mcpp it is, + // not to the member the graph names it by. setName = it->second + ":build"; setKind = "executable"; + family = it->second; } else if (package == "mcpp.core") { setName = "mcpp:core"; setKind = "library"; @@ -437,7 +441,7 @@ Rendered render(std::span members, setKind = "library"; } } - auto& set = set_for(setName, package, setKind); + auto& set = set_for(setName, family, setKind); for (auto const* dirs : {&cu.localIncludeDirs, &cu.localIncludeDirsAfter}) for (auto const& dir : *dirs) if (relative_to(dir, generatedTree)) diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 581255d9..8d64eaa0 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -20,6 +20,9 @@ // xlings.cpp P3 -- xlings payloads before the graph. // graph_load.cpp P4a -- loading one git, path or version dependency. // graph.cpp P4b -- the worklist, the graph, the cycle check. +// graph_closures.cpp P4b -- the lookups the later phases ask of the graph. +// host_graph.cpp the build programs' graph: its workspace, its sub-build, +// the programs' identities and the build-plugin checks. // toolchain_decision.cpp P5 -- the toolchain, decided once the graph exists. // features.cpp P6-P8 -- features, capabilities, host tools, and the // dependencies' build programs. @@ -802,12 +805,17 @@ export struct BuildOverrides { // a build's programs (prepare/host_graph.cpp). Every package it shares // with the build that asked is taken at the version that build resolved // (`.` -> version): one package has one version across - // the two graphs, as the lock records. `build_program_graph` marks every - // package the sub-build loads as a build-program instance, which is what - // the compatibility reading `plugin-imports-from-build-dependencies` - // applies to. + // the two graphs, as the lock records. `build_program_graph` marks the + // sub-build as that graph: its members' `[dependencies]` are the build + // programs' libraries, so the reading `host-module-in-dependencies` does + // not apply to them. std::map resolution_pins; bool build_program_graph = false; + // The project whose `.mcpp/` holds the packages of its custom indices. + // Empty = the project being built. The build programs' graph names the + // build that asked, whose indices it shares and whose store already holds + // what they installed. + std::filesystem::path project_index_root; // Nested source/tool builds inherit the consumer root's local development // OS. A dependency's own [xlings].subos is never consulted or propagated. std::shared_ptr diff --git a/src/build/prepare/driver.cpp b/src/build/prepare/driver.cpp index d4871e4a..51ca3a7c 100644 --- a/src/build/prepare/driver.cpp +++ b/src/build/prepare/driver.cpp @@ -22,6 +22,7 @@ import mcpp.build.backend; // BuildOptions for the tool sub-build import mcpp.build.ninja; // make_ninja_backend — driving that sub-build import mcpp.platform; import mcpp.log; +import mcpp.project; import mcpp.ui; namespace mcpp::build { @@ -43,6 +44,22 @@ std::vector take_notes_on_failure() { } namespace { +// `--strict` refuses a deprecated reading the build can edit (docs/51 +// §Deprecations): each was warned about where it was read, and is refused +// here, after the phase that reads it, naming every use. +std::expected refuse_deprecated_readings_under_strict(const PrepareState& state) { + if (!state.overrides.strict) return {}; + std::string uses; + for (auto const& u : mcpp::pm::compat::recorded_uses()) { + if (!u.warned) continue; + auto const* d = mcpp::pm::compat::find_deprecation(u.id); + uses += std::format("\n {}: {} ({}; write {})", u.where, u.detail, u.id, + d ? std::string(d->canonical) : std::string("the canonical spelling")); + } + if (uses.empty()) return {}; + return std::unexpected("--strict refuses the deprecated readings this build uses:" + uses); +} + std::expected prepare_build_pass(bool print_fingerprint, bool includeDevDeps, @@ -80,11 +97,41 @@ prepare_build(bool print_fingerprint, } namespace { +// A BUILD PROGRAMS' GRAPH IS SELF-DESCRIBING (mcpp#797). The engine writes +// `.mcpp-build-programs` beside the graph's workspace manifest: the toolchain, +// the profile and the version pins its sub-build was given. A command run in +// that directory reads them, so `mcpp build` there reproduces the compile of +// the build programs exactly, as the build's own error message says it does. +void read_build_programs_marker(BuildOverrides& ov) { + namespace fs = std::filesystem; + std::error_code ec; + auto root = ov.project_root; + if (root.empty()) + if (auto r = mcpp::project::find_manifest_root(fs::current_path(ec))) root = *r; + if (root.empty()) return; + std::ifstream in(root / ".mcpp-build-programs"); + if (!in) return; + ov.build_program_graph = true; + std::string line; + while (std::getline(in, line)) { + const auto eq = line.find('='); + if (eq == std::string::npos) continue; + const auto key = line.substr(0, eq), value = line.substr(eq + 1); + if (key == "toolchain" && ov.toolchain.empty()) ov.toolchain = value; + else if (key == "profile" && ov.profile.empty()) ov.profile = value; + else if (key == "index-store" && ov.project_index_root.empty()) ov.project_index_root = value; + else if (key == "pin") + if (auto p = value.find('='); p != std::string::npos) + ov.resolution_pins.try_emplace(value.substr(0, p), value.substr(p + 1)); + } +} + std::expected prepare_build_pass(bool print_fingerprint, bool includeDevDeps, std::vector extraTargets, BuildOverrides overrides) { + read_build_programs_marker(overrides); PrepareState state(print_fingerprint, includeDevDeps, std::move(extraTargets), std::move(overrides)); pending_flag_words_notes().clear(); @@ -142,6 +189,7 @@ prepare_build_pass(bool print_fingerprint, if (auto r = timed("graph", [&] { return phase4b_graph_worklist(state); }); !r) return fail(r.error()); if (auto r = check_engine_floors(state, /*rootOnly=*/false); !r) return fail(r.error()); + if (auto r = refuse_deprecated_readings_under_strict(state); !r) return fail(r.error()); if (auto r = timed("toolchain", [&] { return phase5_toolchain_after_graph(state); }); !r) return fail(r.error()); if (auto r = timed("features and host tools", [&] { return phase6_features_and_host_tools(state); }); !r) @@ -150,6 +198,7 @@ prepare_build_pass(bool print_fingerprint, return fail(r.error()); if (auto r = timed("scan", [&] { return phase11_scan(state); }); !r) return fail(r.error()); + if (auto r = refuse_deprecated_readings_under_strict(state); !r) return fail(r.error()); g_notesOnFailure.clear(); return timed("finish", [&] { return phase13_finish(state); }); diff --git a/src/build/prepare/features.cpp b/src/build/prepare/features.cpp index 3559910e..cad3974d 100644 --- a/src/build/prepare/features.cpp +++ b/src/build/prepare/features.cpp @@ -101,6 +101,71 @@ aggregatedRequest(PrepareState& state, std::size_t depPkgIndex, Instance instanc return { std::move(feats), anyEdge ? anyDefault : true }; } +// COMPAT(plugin-imports-from-build-dependencies). Before mcpp#797 a build +// plugin's library was compiled inside its consumer's build program, so a +// package it listed in its own `[build-dependencies]` with `host-module = true` +// was importable from the library as well. In the new model a library's +// dependencies are its `[dependencies]`: where the plugin's library imports a +// module that entry provides, the entry is read as a dependency too -- an +// ordinary edge, and an entry of the manifest's `[dependencies]`, which the +// link closure follows. The condition is a fact of the two packages' sources, +// so the reading is the same in every graph that loads them. +static void step6_compat_library_edges(PrepareState& state) { + auto declared = [&](std::size_t d) { + std::set out; + for (auto const& f : mcpp::modgraph::package_source_files(state.packages[d].root, + state.packages[d].manifest)) { + std::ifstream in(f, std::ios::binary); + const std::string text{std::istreambuf_iterator(in), {}}; + for (auto& m : mcpp::pm::declared_module_roots(text)) out.insert(std::move(m)); + } + return out; + }; + auto imported = [&](std::size_t p) { + std::set out; + for (auto const& f : mcpp::modgraph::package_source_files(state.packages[p].root, + state.packages[p].manifest)) { + std::ifstream in(f, std::ios::binary); + const std::string text{std::istreambuf_iterator(in), {}}; + for (auto& m : mcpp::pm::imported_module_names(text)) out.insert(std::move(m)); + } + return out; + }; + std::vector added; + for (auto const& r : state.graphRequests) { + const auto p = r.consumerPackageIndex, d = r.dependencyPackageIndex; + if (p == 0 || p >= state.packages.size() || d >= state.packages.size()) continue; + if (r.table.find("build-dependencies") == std::string::npos) continue; + auto& pm = state.packages[p].manifest; + auto spec = pm.buildDependencies.find(r.key); + if (spec == pm.buildDependencies.end() || !spec->second.hostModuleDeclared + || !spec->second.hostModule || pm.dependencies.contains(r.key)) + continue; + const auto provides = declared(d); + const auto wants = imported(p); + if (std::ranges::none_of(wants, [&](auto const& m) { return provides.contains(m); })) + continue; + auto asDependency = spec->second; + asDependency.hostModule = false; + asDependency.hostModuleDeclared = false; + asDependency.declaredIn = "[dependencies]"; + pm.dependencies.emplace(r.key, asDependency); + if (p - 1 < state.dep_manifests.size() && state.dep_manifests[p - 1]) + state.dep_manifests[p - 1]->dependencies.emplace(r.key, asDependency); + for (auto const& e : state.dependencyEdges) + if (e.consumerPackageIndex == p && e.dependencyPackageIndex == d && e.buildTable) { + auto copy = e; + copy.buildTable = false; + copy.wantsLibrary = false; + added.push_back(std::move(copy)); + break; + } + state.reportCompatUse("plugin-imports-from-build-dependencies", p, + std::format("[build-dependencies] {}", r.key)); + } + for (auto& e : added) state.dependencyEdges.push_back(std::move(e)); +} + // WHICH INSTANCES EACH PACKAGE HAS (mcpp#797). // // A package has a target instance when a chain of ordinary edges reaches it @@ -2038,6 +2103,7 @@ static std::expected step6_capability_binding(PrepareState& s std::expected phase6_features_and_host_tools(PrepareState& state) { step6_check_version_floors_closure(state); + step6_compat_library_edges(state); step6_instances(state); if (auto r = step6_activate_features(state); !r) return std::unexpected(r.error()); diff --git a/src/build/prepare/graph.cpp b/src/build/prepare/graph.cpp index 4cbfca28..ca6942bb 100644 --- a/src/build/prepare/graph.cpp +++ b/src/build/prepare/graph.cpp @@ -1965,27 +1965,6 @@ step4b_finalize_dependency(PrepareState& state, WorklistItemCtx& ctx) { childReq.version, selfIdx, ctx.dep_root, item.devOnly, /*buildOnly=*/true, /*fromBuildTable=*/true}); - // COMPAT(plugin-imports-from-build-dependencies): before mcpp#797 a - // build plugin's library was compiled inside its consumer's build - // program, so a package it listed in its own - // `[build-dependencies]` with `host-module = true` was importable - // from the library too. In the build programs' graph that library - // is an ordinary package, and its library's dependencies are its - // `[dependencies]`: the entry is read as one as well. Reported at - // the scan, where a unit of the library is seen to import it. - if (state.overrides.build_program_graph && child_spec.hostModuleDeclared - && child_spec.hostModule - && !state.dep_manifests.back()->dependencies.contains(child_name)) { - state.compatLibraryEdges.push_back({selfIdx, child_name}); - // A library dependency now: the request is what the edge is. - auto asDependency = childReq; - asDependency.hostModule = false; - asDependency.hostModuleDeclared = false; - state.worklist.push_back({child_name, asDependency, - thisDepLabel + " (build-dep, read as a dependency)", - childReq.version, selfIdx, ctx.dep_root, - item.devOnly, item.buildOnly}); - } } return {}; } @@ -2023,180 +2002,6 @@ static std::expected step4b_cycle_check(PrepareState& state) return {}; } -static void step4b_define_lookup_closures(PrepareState& state) { - state.bareBindingsFor = [&](std::size_t consumer) { - std::vector fqns; - if (consumer < state.provisionGraph.visible.size()) - for (auto const& pr : state.provisionGraph.visible[consumer]) { - if (pr.provider >= state.packages.size()) continue; - auto const& n = state.packages[pr.provider].manifest.package.name; - if (std::find(fqns.begin(), fqns.end(), n) == fqns.end()) - fqns.push_back(n); - } - return prov::bind_bare_names(fqns); - }; - // THE NAMES UNDER WHICH ONE PROVIDER IS PUBLISHED TO ONE CONSUMER, derived - // once for every channel (#647 E4.3). The manifest's `name`, the qualified - // `namespace.name` when the manifest writes the two apart, and the bare - // tail where the namespace ladder binds it to this provider for this - // consumer. `dep_dir`/`dep_linkage` and `dep_bin` used to derive this list - // separately; #642 added the qualified spelling to the first and the second - // kept publishing `MCPP_DEP_INSTALLER_BIN_*` alone for a package written - // `namespace = "spike"`, `name = "installer"`, so - // `dep_bin("spike.installer", ...)` read nothing. - state.publishedNamesFor = - [&](std::size_t provider, - const std::map& bind) { - std::vector out; - auto const& manifest = state.packages[provider].manifest; - auto const& canon = manifest.package.name; - out.push_back(canon); - if (auto qualified = mcpp::build::qualified_package_name(manifest); - qualified != canon) - out.push_back(std::move(qualified)); - if (auto tail = prov::tail_of(canon); tail != canon) { - auto it = bind.find(tail); - if (it != bind.end() && it->second.owner == canon) - out.push_back(std::move(tail)); - } - return out; - }; - - // A package whose DECLARED targets are all programs (#649 E6). See the - // worklist, where such a package is not walked into a consumer's graph. - state.isProgramOnlyPackage = [](const mcpp::manifest::Manifest& pm) { - if (pm.targetsInferred || pm.targets.empty()) return false; - return std::ranges::none_of(pm.targets, [](const mcpp::manifest::Target& t) { - return t.kind == mcpp::manifest::Target::Library - || t.kind == mcpp::manifest::Target::SharedLibrary; - }); - }; - // A package some edge asked for programs to SHIP (mcpp#711). Its programs - // are linked in this plan, so it is scanned and configured here like any - // library dependency, even when every target it declares is a program. - state.isArtifactPackage = [&](std::size_t i) { - return std::ranges::any_of(state.dependencyEdges, [&](const DependencyEdge& e) { - return e.dependencyPackageIndex == i && !e.requestedArtifacts.empty(); - }); - }; - // Compiled in this plan: not a package of programs, or one whose programs - // this plan ships. - state.isWorkspaceMemberPackage = [&](std::size_t i) { - if (i == 0 || i >= state.packages.size() || !state.workspacePlan()) return false; - if (state.packages[i].selectedMember) return true; - const auto& r = state.packages[i].root; - if (r.lexically_normal() == state.runtimeWorkspaceRoot.lexically_normal()) return true; - return !workspace_member_of(state, r).empty(); - }; - // A selected workspace member is compiled here whatever its targets are - // (workspace design 2026-09-29 §15). - // - // NOT a package with no target instance (mcpp#797): one reached only - // through `[build-dependencies]` is compiled in the build program's graph, - // not here. `targetInstance` is decided by `step6_instances`; until then - // every package counts. - state.compilesHere = [&](std::size_t i) { - if (i < state.packages.size() && i > 0 && state.packages[i].buildTimeOnly) - return false; - return i == 0 || !state.isProgramOnlyPackage(state.packages[i].manifest) - || state.isArtifactPackage(i) || state.packages[i].selectedMember; - }; - state.editablePackage = [&](std::size_t i) { - if (i == 0 || i >= state.packages.size()) return true; - auto const& pkg = state.packages[i]; - if (pkg.selectedMember || state.isWorkspaceMemberPackage(i)) return true; - if (mcpp::build::path_is_under_any(pkg.root, state.storeRoots)) return false; - const auto home = !state.runtimeWorkspaceRoot.empty() ? state.runtimeWorkspaceRoot - : *state.root; - auto rel = pkg.root.lexically_normal().lexically_relative(home.lexically_normal()); - return !rel.empty() && *rel.begin() != ".."; - }; - state.reportCompatUse = [&](std::string_view id, std::size_t i, std::string detail) { - auto const* d = mcpp::pm::compat::find_deprecation(id); - if (!d) return; - const bool warn = state.editablePackage(i); - const auto where = i < state.packages.size() - ? (state.packages[i].root / "mcpp.toml").string() : std::string{}; - mcpp::pm::compat::record_use(id, where, detail, warn); - mcpp::log::info("compat", std::format("{} {}: {}{}", id, where, detail, - warn ? "" : " (not editable here; not warned)")); - if (!warn) return; - mcpp::diag::warning(d->domain, std::format( - "{}: {} is {}. Write {} instead.", where, detail, mcpp::pm::compat::deprecation_clause(*d), - d->canonical)); - }; - - - state.appendUniquePath = - [](std::vector& dirs, - const std::filesystem::path& dir) -> bool - { - if (std::find(dirs.begin(), dirs.end(), dir) != dirs.end()) return false; - dirs.push_back(dir); - return true; - }; - - state.appendUniquePaths = - [&](std::vector& dirs, - const std::vector& additions) -> bool - { - bool changed = false; - for (auto const& dir : additions) { - changed = state.appendUniquePath(dirs, dir) || changed; - } - return changed; - }; - - // "Which compile-visible channels a build.mcpp directive lands in" is a - // property of the DIRECTIVE TABLE, not of this call site, so both the mark - // and the fold now live with the table in mcpp.build.directives. This pair - // used to be defined here and was already incomplete — the comment it - // replaced admitted that link/source residues stayed at the call sites, - // which is the #242 two-derivations shape. - // - // The fold is PRIVATE by design (Cargo discipline — a build-time program - // must not widen the package's public interface): privateBuild only, never - // publicUsage. The after-dirs ride the typed #249 channel, which owns the - // per-dialect degradations (cl.exe /I, NASM -I). - using DirectiveMark = mcpp::build::directives::Mark; - state.markDirectiveTail = [](const mcpp::manifest::Manifest& mm) { - return mcpp::build::directives::mark(mm); - }; - state.foldDirectiveTailIntoPrivateBuild = - [](mcpp::modgraph::PackageRoot& pkg, const mcpp::manifest::Manifest& ran, - const DirectiveMark& t) - { - mcpp::build::directives::fold_private_tail(pkg.privateBuild, ran, t); - }; - - // mcpp#241: the (name → dir) pairs a package's build.mcpp receives as - // MCPP_DEP__DIR. ONE owner: the dependency loop and the root call - // site had drifted into two near-identical copies of this, and #355 was - // about to add a third. Each dependency is emitted under BOTH its - // canonical name and its namespace-stripped tail, so - // `mcpp::dep_dir("compat.zlib")` and `mcpp::dep_dir("zlib")` both resolve - // regardless of which spelling the author used in `deps`. - // - // #359: the set is now the consumer's VISIBLE provisions rather than its - // direct edges, so a re-exported dependency's directory reaches it too. - // That is what makes a rule package able to find data files belonging to a - // dependency the user never declared — protoc's well-known .proto files - // are exactly such a directory, and `grpcgen` reads them through dep_dir. - // - // The bare tail is emitted only when the namespace ladder binds it here. - // Emitting it unconditionally was safe while only the root's own - // declarations reached build.mcpp; with re-export, two packages that never - // heard of each other can share a tail and the later emplace_back would - // silently win. - // The xlings half of fillDepDirs. Same question ("where did my declared - // dependency's payload land"), different namespace and store layout, so it - // cannot ride the mcpp dependency channel — but it must be an INTERFACE on - // the build.mcpp side for the same reason that one is: a program that - // reconstructs the store path is coupled to internals mcpp is free to - // change. See mcpp::build::hostprogram::xpkg_dir. - -} - static std::expected step4b_define_provisioning_closures(PrepareState& state) { // Which dependency supplied the runner, for the exactly-one-provider diff --git a/src/build/prepare/graph_closures.cpp b/src/build/prepare/graph_closures.cpp new file mode 100644 index 00000000..b4fecb34 --- /dev/null +++ b/src/build/prepare/graph_closures.cpp @@ -0,0 +1,196 @@ +// graph_closures.cpp -- P4b: the lookups the phases after the graph ask of it +// (which providers a consumer sees, which package a user may edit, how a +// deprecated reading is reported). Defined once the graph exists; split from +// graph.cpp at that boundary. + +module mcpp.build.prepare; +import :state; + +import mcpp.build.prepare_inputs; + +import std; +import mcpp.diag; +import mcpp.log; +import mcpp.home; +import mcpp.manifest; +import mcpp.modgraph.scanner; +import mcpp.build.plan; // path_is_under_any, qualified_package_name +import mcpp.build.directives; // directive table: mark / fold_private_tail +import mcpp.pm.compat; + +namespace mcpp::build { + +void step4b_define_lookup_closures(PrepareState& state) { + state.bareBindingsFor = [&](std::size_t consumer) { + std::vector fqns; + if (consumer < state.provisionGraph.visible.size()) + for (auto const& pr : state.provisionGraph.visible[consumer]) { + if (pr.provider >= state.packages.size()) continue; + auto const& n = state.packages[pr.provider].manifest.package.name; + if (std::find(fqns.begin(), fqns.end(), n) == fqns.end()) + fqns.push_back(n); + } + return prov::bind_bare_names(fqns); + }; + // THE NAMES UNDER WHICH ONE PROVIDER IS PUBLISHED TO ONE CONSUMER, derived + // once for every channel (#647 E4.3). The manifest's `name`, the qualified + // `namespace.name` when the manifest writes the two apart, and the bare + // tail where the namespace ladder binds it to this provider for this + // consumer. `dep_dir`/`dep_linkage` and `dep_bin` used to derive this list + // separately; #642 added the qualified spelling to the first and the second + // kept publishing `MCPP_DEP_INSTALLER_BIN_*` alone for a package written + // `namespace = "spike"`, `name = "installer"`, so + // `dep_bin("spike.installer", ...)` read nothing. + state.publishedNamesFor = + [&](std::size_t provider, + const std::map& bind) { + std::vector out; + auto const& manifest = state.packages[provider].manifest; + auto const& canon = manifest.package.name; + out.push_back(canon); + if (auto qualified = mcpp::build::qualified_package_name(manifest); + qualified != canon) + out.push_back(std::move(qualified)); + if (auto tail = prov::tail_of(canon); tail != canon) { + auto it = bind.find(tail); + if (it != bind.end() && it->second.owner == canon) + out.push_back(std::move(tail)); + } + return out; + }; + + // A package whose DECLARED targets are all programs (#649 E6). See the + // worklist, where such a package is not walked into a consumer's graph. + state.isProgramOnlyPackage = [](const mcpp::manifest::Manifest& pm) { + if (pm.targetsInferred || pm.targets.empty()) return false; + return std::ranges::none_of(pm.targets, [](const mcpp::manifest::Target& t) { + return t.kind == mcpp::manifest::Target::Library + || t.kind == mcpp::manifest::Target::SharedLibrary; + }); + }; + // A package some edge asked for programs to SHIP (mcpp#711). Its programs + // are linked in this plan, so it is scanned and configured here like any + // library dependency, even when every target it declares is a program. + state.isArtifactPackage = [&](std::size_t i) { + return std::ranges::any_of(state.dependencyEdges, [&](const DependencyEdge& e) { + return e.dependencyPackageIndex == i && !e.requestedArtifacts.empty(); + }); + }; + // Compiled in this plan: not a package of programs, or one whose programs + // this plan ships. + state.isWorkspaceMemberPackage = [&](std::size_t i) { + if (i == 0 || i >= state.packages.size() || !state.workspacePlan()) return false; + if (state.packages[i].selectedMember) return true; + const auto& r = state.packages[i].root; + if (r.lexically_normal() == state.runtimeWorkspaceRoot.lexically_normal()) return true; + return !workspace_member_of(state, r).empty(); + }; + // A selected workspace member is compiled here whatever its targets are + // (workspace design 2026-09-29 §15). + // + // NOT a package with no target instance (mcpp#797): one reached only + // through `[build-dependencies]` is compiled in the build program's graph, + // not here. `targetInstance` is decided by `step6_instances`; until then + // every package counts. + state.compilesHere = [&](std::size_t i) { + if (i < state.packages.size() && i > 0 && state.packages[i].buildTimeOnly) + return false; + return i == 0 || !state.isProgramOnlyPackage(state.packages[i].manifest) + || state.isArtifactPackage(i) || state.packages[i].selectedMember; + }; + state.editablePackage = [&](std::size_t i) { + if (i == 0 || i >= state.packages.size()) return true; + auto const& pkg = state.packages[i]; + if (pkg.selectedMember || state.isWorkspaceMemberPackage(i)) return true; + // A package fetched into a store is not; a `path` package is, wherever + // it lies (Cargo draws the same line for its lints). + return !mcpp::build::path_is_under_any(pkg.root, state.storeRoots) + && !mcpp::build::path_is_under_any(pkg.root, std::vector{mcpp::home::root()}); + }; + state.reportCompatUse = [&](std::string_view id, std::size_t i, std::string detail) { + auto const* d = mcpp::pm::compat::find_deprecation(id); + if (!d) return; + const bool warn = state.editablePackage(i); + const auto where = i < state.packages.size() + ? (state.packages[i].root / "mcpp.toml").string() : std::string{}; + mcpp::pm::compat::record_use(id, where, detail, warn); + mcpp::log::info("compat", std::format("{} {}: {}{}", id, where, detail, + warn ? "" : " (not editable here; not warned)")); + if (!warn) return; + mcpp::diag::warning(d->domain, std::format( + "{}: {} is {}. Write {} instead.", where, detail, mcpp::pm::compat::deprecation_clause(*d), + d->canonical)); + }; + + + state.appendUniquePath = + [](std::vector& dirs, + const std::filesystem::path& dir) -> bool + { + if (std::find(dirs.begin(), dirs.end(), dir) != dirs.end()) return false; + dirs.push_back(dir); + return true; + }; + + state.appendUniquePaths = + [&](std::vector& dirs, + const std::vector& additions) -> bool + { + bool changed = false; + for (auto const& dir : additions) { + changed = state.appendUniquePath(dirs, dir) || changed; + } + return changed; + }; + + // "Which compile-visible channels a build.mcpp directive lands in" is a + // property of the DIRECTIVE TABLE, not of this call site, so both the mark + // and the fold now live with the table in mcpp.build.directives. This pair + // used to be defined here and was already incomplete — the comment it + // replaced admitted that link/source residues stayed at the call sites, + // which is the #242 two-derivations shape. + // + // The fold is PRIVATE by design (Cargo discipline — a build-time program + // must not widen the package's public interface): privateBuild only, never + // publicUsage. The after-dirs ride the typed #249 channel, which owns the + // per-dialect degradations (cl.exe /I, NASM -I). + using DirectiveMark = mcpp::build::directives::Mark; + state.markDirectiveTail = [](const mcpp::manifest::Manifest& mm) { + return mcpp::build::directives::mark(mm); + }; + state.foldDirectiveTailIntoPrivateBuild = + [](mcpp::modgraph::PackageRoot& pkg, const mcpp::manifest::Manifest& ran, + const DirectiveMark& t) + { + mcpp::build::directives::fold_private_tail(pkg.privateBuild, ran, t); + }; + + // mcpp#241: the (name → dir) pairs a package's build.mcpp receives as + // MCPP_DEP__DIR. ONE owner: the dependency loop and the root call + // site had drifted into two near-identical copies of this, and #355 was + // about to add a third. Each dependency is emitted under BOTH its + // canonical name and its namespace-stripped tail, so + // `mcpp::dep_dir("compat.zlib")` and `mcpp::dep_dir("zlib")` both resolve + // regardless of which spelling the author used in `deps`. + // + // #359: the set is now the consumer's VISIBLE provisions rather than its + // direct edges, so a re-exported dependency's directory reaches it too. + // That is what makes a rule package able to find data files belonging to a + // dependency the user never declared — protoc's well-known .proto files + // are exactly such a directory, and `grpcgen` reads them through dep_dir. + // + // The bare tail is emitted only when the namespace ladder binds it here. + // Emitting it unconditionally was safe while only the root's own + // declarations reached build.mcpp; with re-export, two packages that never + // heard of each other can share a tail and the later emplace_back would + // silently win. + // The xlings half of fillDepDirs. Same question ("where did my declared + // dependency's payload land"), different namespace and store layout, so it + // cannot ride the mcpp dependency channel — but it must be an INTERFACE on + // the build.mcpp side for the same reason that one is: a program that + // reconstructs the store path is coupled to internals mcpp is free to + // change. See mcpp::build::hostprogram::xpkg_dir. + +} + +} // namespace mcpp::build diff --git a/src/build/prepare/graph_load.cpp b/src/build/prepare/graph_load.cpp index 6ab48918..81e0b977 100644 --- a/src/build/prepare/graph_load.cpp +++ b/src/build/prepare/graph_load.cpp @@ -293,7 +293,7 @@ static void step4a_define_split_and_identity_closures(PrepareState& state) { // rebuilt on demand because `root` moves when a workspace member is // selected above. state.index_route = [&](mcpp::config::GlobalConfig* cfg = nullptr) { - return mcpp::pm::IndexRoute{ &state.m->indices, *state.root, cfg }; + return mcpp::pm::IndexRoute{ &state.m->indices, state.projectIndexRoot(), cfg }; }; state.findIndexForNs = [&](const std::string& ns) -> const mcpp::pm::IndexSpec* @@ -701,7 +701,7 @@ step4a_read_lua_content(PrepareState& state, LoadVersionDepCtx& ctx) { } if (ctx.idxSpec && !ctx.idxSpec->is_builtin()) { return mcpp::fetcher::Fetcher::read_xpkg_lua_from_project_data( - *state.root, ctx.ns, ctx.shortName); + state.projectIndexRoot(), ctx.ns, ctx.shortName); } return fetcher.read_xpkg_lua(ctx.ns, ctx.shortName); } @@ -716,7 +716,7 @@ step4a_find_raw_installed(PrepareState& state, LoadVersionDepCtx& ctx) { mcpp::fetcher::Fetcher fetcher(**cfg); if (ctx.useProjectEnv) { if (auto p = mcpp::fetcher::Fetcher::install_path_from_project_data( - *state.root, ctx.ns, ctx.shortName, ctx.version)) { + state.projectIndexRoot(), ctx.ns, ctx.shortName, ctx.version)) { return p; } } @@ -965,7 +965,7 @@ step4a_load_version_dep_fetch(PrepareState& state, LoadVersionDepCtx& ctx) { // Using the interface (rather than the silenced direct CLI) // restores the live `Downloading … [bar] X/Y Z/s` UI here, // matching the toolchain and builtin-index paths. - auto projEnv = mcpp::config::make_project_xlings_env(**cfg, *state.root); + auto projEnv = mcpp::config::make_project_xlings_env(**cfg, state.projectIndexRoot()); auto argsJson = std::format( R"({{"targets":["{}"],"yes":true}})", target); mcpp::fetcher::InstallProgressHandler progress; @@ -1070,7 +1070,9 @@ step4a_load_version_dep_fetch(PrepareState& state, LoadVersionDepCtx& ctx) { // a hint about the known ≥2-repo xlings resolution gap. The // real fix lives in openxlings/xlings; this only surfaces WHY. auto xlingsJson = (ctx.useProjectEnv - ? (state.workRoot / ".mcpp") + ? ((state.overrides.project_index_root.empty() ? state.workRoot + : state.overrides.project_index_root) + / ".mcpp") : (*cfg)->xlingsHome()) / ".xlings.json"; auto indexRepos = mcpp::pm::read_seeded_index_repos(xlingsJson); diff --git a/src/build/prepare/host_graph.cpp b/src/build/prepare/host_graph.cpp index 192502ea..3114f41a 100644 --- a/src/build/prepare/host_graph.cpp +++ b/src/build/prepare/host_graph.cpp @@ -66,6 +66,10 @@ const mcpp::manifest::Manifest& manifest_of(const PrepareState& state, std::size return i == 0 ? *state.m : state.packages[i].manifest; } +bool imports_engine(std::string_view text); +std::vector build_plugin_units(const mcpp::manifest::Manifest& m, const fs::path& root, + const std::vector& features); + // A TOML basic string. std::string toml_string(std::string_view s) { std::string out = "\""; @@ -486,8 +490,11 @@ void write_if_changed(const fs::path& file, const std::string& text) { } // The manifests of one group: the workspace and a member per program. +std::map resolution_pins(const PrepareState& state); + void write_group(PrepareState& state, const fs::path& dir, const std::vector& programs, - const std::string& standard, const fs::path& enginePackage) { + const std::string& standard, const fs::path& enginePackage, + const std::string& toolchainSpec) { std::string root = std::format( "# Written by mcpp {}: the build programs of {} (mcpp#797).\n" "# `mcpp build` here reproduces their compile; it is rewritten on every build.\n" @@ -515,7 +522,47 @@ void write_group(PrepareState& state, const fs::path& dir, const std::vector rows; + const mcpp::manifest::Manifest* stating[] = {state.m ? &*state.m : nullptr, + state.runtimeOwnerManifest}; + for (auto const* man : stating) { + if (!man) continue; + for (auto const& [k, o] : man->xlings.overrides) { + if (rows.contains(k)) continue; + using Kind = mcpp::manifest::XlingsOverride::Kind; + auto value = o.value; + if (fs::path v(value); (o.kind != Kind::Program || names_a_file_path(value)) + && v.is_relative()) + value = path_text((man->sourcePath.parent_path() / v).lexically_normal()); + const bool isRoot = o.kind == Kind::Root + || (o.kind == Kind::Path && fs::is_directory(value)); + std::vector fields; + fields.push_back(std::format("{} = {}", isRoot ? "root" : "program", toml_string(value))); + if (!o.version.empty()) fields.push_back("version = " + toml_string(o.version)); + std::string row; + for (auto const& f : fields) row += (row.empty() ? "" : ", ") + f; + rows.emplace(k, std::format("{} = {{ {} }}\n", toml_string(k), row)); + } + } + if (!rows.empty()) { + root += "\n[workspace.xlings.overrides]\n"; + for (auto const& [k, row] : rows) root += row; + } + } write_if_changed(dir / "mcpp.toml", root); + // What the sub-build is given, for a command run in this directory + // (driver.cpp, `read_build_programs_marker`). + { + std::string marker = std::format("toolchain={}\nprofile=build-program\nindex-store={}\n", + toolchainSpec, path_text(state.projectIndexRoot())); + for (auto const& [k, v] : resolution_pins(state)) marker += std::format("pin={}={}\n", k, v); + write_if_changed(dir / ".mcpp-build-programs", marker); + } for (auto const& p : programs) { const auto memberDir = dir / p.member; @@ -533,8 +580,26 @@ void write_group(PrepareState& state, const fs::path& dir, const std::vector> tables; // namespace -> lines - tables[std::string(engine::kNamespace)].push_back( - std::format("{} = {{ path = {} }}", key(engine::kName), toml_string(path_text(enginePackage)))); + // THE ENGINE INTERFACE ONLY WHERE SOMETHING IMPORTS IT. A module's + // object is linked whether or not anything references it, and the + // interface announces the directive protocol when the program starts: + // a hand-written `printf` program that imports nothing must stay one, + // whose unknown directives are warned about rather than refused. + bool needsEngine = !p.source.synthesised.empty(); + if (!needsEngine) { + std::ifstream in(p.source.file, std::ios::binary); + needsEngine = imports_engine(std::string{std::istreambuf_iterator(in), {}}); + } + if (!needsEngine) + for (auto d : closure_of(state, program_libraries(state, p.index))) + if (!build_plugin_units(state.packages[d].manifest, state.packages[d].root, + program_features(state, d)).empty()) { + needsEngine = true; + break; + } + if (needsEngine) + tables[std::string(engine::kNamespace)].push_back( + std::format("{} = {{ path = {} }}", key(engine::kName), toml_string(path_text(enginePackage)))); for (auto d : program_libraries(state, p.index)) { const auto line = dependency_line(state, d); tables[line.ns].push_back(std::format("{} = {{ {}, {} }}", key(line.name), @@ -602,11 +667,12 @@ build_group(PrepareState& state, const fs::path& dir, const std::vector std::make_shared(state.runtimeBindingSnapshot); sub.resolution_pins = resolution_pins(state); sub.build_program_graph = true; + sub.project_index_root = state.projectIndexRoot(); for (auto const& p : all) sub.workspace_members.push_back(p.member); auto reproduce = [&] { return std::format("\n the build programs' workspace: {}\n" - " re-run it directly: cd {} && mcpp build --profile build-program\n" + " re-run it directly: cd {} && mcpp build\n" " (MCPP_TOOL_BUILD_VERBOSE=1 shows the inner build's output unfiltered)", path_text(dir), path_text(dir)); }; @@ -875,7 +941,7 @@ ensure_build_programs(PrepareState& state, const std::vector& packa // Described, not built: `mcpp emit build-database` lists the graph // of programs that are current too. if (state.overrides.plan_only) { - write_group(state, dir, programs, standard, *enginePackage); + write_group(state, dir, programs, standard, *enginePackage, tc->spec); if (auto r = build_group(state, dir, programs, {}, *tc); !r) mcpp::log::verbose("host-graph", r.error()); } @@ -889,7 +955,7 @@ ensure_build_programs(PrepareState& state, const std::vector& packa return std::unexpected(r.error()); } - write_group(state, dir, programs, standard, *enginePackage); + write_group(state, dir, programs, standard, *enginePackage, tc->spec); const auto began = std::chrono::steady_clock::now(); for (auto const* p : stale) mcpp::build::progress::program_compiling(qualified(manifest_of(state, p->index)), diff --git a/src/build/prepare/manifest.cpp b/src/build/prepare/manifest.cpp index 4b487e03..6a22653d 100644 --- a/src/build/prepare/manifest.cpp +++ b/src/build/prepare/manifest.cpp @@ -170,6 +170,12 @@ report_workspace_root(const mcpp::manifest::Manifest& ws, const std::filesystem: if (strict) return std::unexpected(w); mcpp::diag::warning("manifest/schema", w); } + // The root's own statements, its layer's among them: a build of members + // reads this manifest as much as theirs (mcpp#797, W2). + for (auto const& w : ws.schemaWarnings) { + if (strict) return std::unexpected(w); + mcpp::diag::warning("manifest/schema", w); + } if (ws.package.name.empty()) { std::error_code ec; if (std::filesystem::is_regular_file(wsRoot / "build.mcpp", ec)) { diff --git a/src/build/prepare/scan.cpp b/src/build/prepare/scan.cpp index dd866538..86dbe07e 100644 --- a/src/build/prepare/scan.cpp +++ b/src/build/prepare/scan.cpp @@ -951,6 +951,18 @@ static void step11_public_module_check(PrepareState& state) { std::map> publicOf; for (std::size_t i = 1; i < state.packages.size(); ++i) { auto const& pr = state.packages[i]; + // A declared interface root that names no file: the package's own build + // refuses it (modgraph::validate); here, where the package is a + // dependency, it is reported (mcpp#797, SPEC-008 D14). + if (!pr.manifest.lib.path.empty()) { + std::error_code lec; + const auto declared = pr.root / mcpp::manifest::resolve_lib_root_path(pr.manifest, pr.root); + if (!std::filesystem::is_regular_file(declared, lec)) + mcpp::diag::warning("build/interface", std::format( + "'{}' states [lib] path = '{}', which names no file; it states no " + "interface root to its consumers", qualified(pr.manifest), + pr.manifest.lib.path.generic_string())); + } if (!workspace_member_of(state, pr.root).empty()) continue; // One definition of what a library offers (mcpp.modgraph.library_interface). const auto iface = mcpp::modgraph::library_interface( @@ -986,51 +998,9 @@ static void step11_public_module_check(PrepareState& state) { } } -// COMPAT(plugin-imports-from-build-dependencies): a build plugin's library -// imports a package it lists in its own `[build-dependencies]` with -// `host-module = true` (mcpp#797). The worklist read the entry as a dependency -// too; the use is reported here, where a unit of the library is seen to import -// a module of that package, and only then. -static void step11_compat_library_edges(PrepareState& state) { - if (state.compatLibraryEdges.empty()) return; - const auto& g = state.scan.graph; - for (auto const& ce : state.compatLibraryEdges) { - const auto consumer = ce.consumerDepIndex + 1; - if (consumer >= state.packages.size()) continue; - std::size_t dep = state.packages.size(); - for (auto const& r : state.graphRequests) - if (r.consumerPackageIndex == consumer && r.key == ce.key) { - dep = r.dependencyPackageIndex; - break; - } - if (dep >= state.packages.size()) continue; - auto qualified = [](const mcpp::manifest::Manifest& m) { - return m.package.namespace_.empty() ? m.package.name - : m.package.namespace_ + "." + m.package.name; - }; - const auto consumerName = qualified(state.packages[consumer].manifest); - const auto depName = qualified(state.packages[dep].manifest); - std::set provided; - for (auto const& u : g.units) - if (u.packageName == depName && u.provides) - provided.insert(u.provides->logicalName.substr(0, u.provides->logicalName.find(':'))); - bool imports = false; - for (auto const& u : g.units) { - if (u.packageName != consumerName) continue; - for (auto const& req : u.requires_) - if (provided.contains(req.logicalName.substr(0, req.logicalName.find(':')))) - imports = true; - } - if (imports) - state.reportCompatUse("plugin-imports-from-build-dependencies", consumer, - std::format("[build-dependencies] {}", ce.key)); - } -} - std::expected phase11_scan(PrepareState& state) { auto needsStdModule = step11_scan_sources(state); if (!needsStdModule) return std::unexpected(needsStdModule.error()); - step11_compat_library_edges(state); if (auto r = refuse_plugin_units_in_target(state); !r) return std::unexpected(r.error()); step11_public_module_check(state); diff --git a/src/build/prepare/sources.cpp b/src/build/prepare/sources.cpp index 0fc58160..da91c7fc 100644 --- a/src/build/prepare/sources.cpp +++ b/src/build/prepare/sources.cpp @@ -76,11 +76,6 @@ std::string override_env_var(std::string_view key) { return out; } -bool names_a_file_path(std::string_view v) { - return v.contains('/') || v.contains('\\') || v.starts_with(".") - || (v.size() > 1 && v[1] == ':'); -} - // The root a program implies: its directory, or that directory's parent when // it is a `bin/` -- so `/bin/`, the layout every payload // plugin already reads, holds for `/usr/bin/cmake` as it does for a payload. @@ -92,6 +87,11 @@ fs::path root_of_program(const fs::path& program) { } // namespace +bool names_a_file_path(std::string_view v) { + return v.contains('/') || v.contains('\\') || v.starts_with(".") + || (v.size() > 1 && v[1] == ':'); +} + std::string source_tag(const SourceDecision& d, const fs::path& relativeTo) { const auto cls = std::string(source_class_name(d.cls)); if (d.originKind == "env") diff --git a/src/build/prepare/state.cppm b/src/build/prepare/state.cppm index a5fdf526..312eb707 100644 --- a/src/build/prepare/state.cppm +++ b/src/build/prepare/state.cppm @@ -507,6 +507,11 @@ struct PrepareState { std::filesystem::path runtimeWorkspaceRoot; mcpp::xlings::runtime::RuntimeSelection runtimeSelection; std::filesystem::path workRoot; + // The project whose `.mcpp/` holds the packages of the custom indices: + // `overrides.project_index_root`, else the project being built. + std::filesystem::path projectIndexRoot() const { + return overrides.project_index_root.empty() ? *root : overrides.project_index_root; + } std::vector planNotes; std::map gitLockAnchors; std::map packageIdentityLockAnchors; @@ -673,12 +678,6 @@ struct PrepareState { // The build programs this prepare compiled or found current, by package // index (`ensure_build_programs`). std::map buildProgramBinaries; - // COMPAT(plugin-imports-from-build-dependencies): `[build-dependencies]` - // entries a build-program instance's library also reads as dependencies, - // by the consumer's dep_manifests slot and the key it wrote. The scan - // reports those its units import. - struct CompatLibraryEdge { std::size_t consumerDepIndex; std::string key; }; - std::vector compatLibraryEdges; // The roots (package directories) planned as build plugins (P3). std::set buildPluginRoots; // The build programs' graphs planned for description (`plan_only`). @@ -835,6 +834,8 @@ std::expected phase2_define_toolchain_resolver(PrepareState& std::expected phase3_xlings_before_graph(PrepareState& state); std::expected phase4a_graph_load(PrepareState& state); std::expected phase4b_graph_worklist(PrepareState& state); +// The lookups the later phases ask of the graph (graph_closures.cpp). +void step4b_define_lookup_closures(PrepareState& state); std::expected phase5_toolchain_after_graph(PrepareState& state); std::expected phase6_features_and_host_tools(PrepareState& state); std::expected phase9_target_side(PrepareState& state); @@ -956,6 +957,8 @@ std::expected refuse_plugin_units_in_target(PrepareState& sta // Record (or replace) the decision for `d.subject`; a non-default source is // announced once with its `Using` line. void record_source(PrepareState& state, SourceDecision d); +// Whether an override value names a file path rather than a program on PATH. +bool names_a_file_path(std::string_view v); // The override for a package key, resolved and cached; nullptr when none. std::expected payload_override(PrepareState& state, std::string_view key); diff --git a/src/build/prepare/toolchain.cpp b/src/build/prepare/toolchain.cpp index 062b32ad..5ddebc69 100644 --- a/src/build/prepare/toolchain.cpp +++ b/src/build/prepare/toolchain.cpp @@ -298,6 +298,9 @@ static std::expected step1_define_early_toolchain_closures(Pr if (auto c = state.get_cfg(true)) roots.push_back((*c)->xlingsHome() / "data" / "xpkgs"); for (auto& d : mcpp::config::project_xlings_data_roots(state.workRoot)) roots.push_back(d / "xpkgs"); + if (!state.overrides.project_index_root.empty()) + for (auto& d : mcpp::config::project_xlings_data_roots(state.overrides.project_index_root)) + roots.push_back(d / "xpkgs"); // The engine interface's package: written once per mcpp version and // never edited, like a payload (mcpp#797). roots.push_back(mcpp::build::engine::root()); diff --git a/src/build/prepare/xlings.cpp b/src/build/prepare/xlings.cpp index 7a2a806c..7ace3eee 100644 --- a/src/build/prepare/xlings.cpp +++ b/src/build/prepare/xlings.cpp @@ -530,15 +530,19 @@ std::expected phase3_xlings_before_graph(PrepareState& state) // Under `plan_only` (`emit build-database`) nothing is written // into the project (SPEC-005 R2.1, mcpp#724 side finding B, e2e // 817), so the owner's half goes to the planning directory too. + // The build programs' graph shares the indices of the build that + // asked, whose project already holds them (`project_index_root`). const auto& runtimeRoot = state.overrides.plan_only ? state.workRoot : state.runtimeSelection.ownerRoot; + const auto indices = state.overrides.project_index_root.empty() + ? state.m->indices : decltype(state.m->indices){}; if (runtimeRoot == state.workRoot) { mcpp::config::ensure_project_index_dir( - **cfg2, state.workRoot, state.m->indices, penv); + **cfg2, state.workRoot, indices, penv); } else { - if (!state.m->indices.empty()) + if (!indices.empty()) mcpp::config::ensure_project_index_dir( - **cfg2, state.workRoot, state.m->indices, {}); + **cfg2, state.workRoot, indices, {}); if (materializeRootRuntime) mcpp::config::ensure_project_index_dir( **cfg2, runtimeRoot, {}, penv); @@ -604,7 +608,7 @@ std::expected phase3_xlings_before_graph(PrepareState& state) } } if (hasCustomIndices) { - bool needsClone = !mcpp::config::project_index_data_initialized(*state.root); + bool needsClone = !mcpp::config::project_index_data_initialized(state.projectIndexRoot()); if (needsClone) { bool needsRemoteUpdate = false; for (auto& [idxName, spec] : state.m->indices) { @@ -637,7 +641,7 @@ std::expected phase3_xlings_before_graph(PrepareState& state) } if (needsRemoteUpdate && !refreshPolicy.offline) { mcpp::ui::status("Fetching", "custom index repos (first use)"); - auto projEnv = mcpp::config::make_project_xlings_env(**cfg2, *state.root); + auto projEnv = mcpp::config::make_project_xlings_env(**cfg2, state.projectIndexRoot()); int rc = mcpp::xlings::update_index(projEnv, /*quiet=*/true); if (rc != 0) { return std::unexpected( diff --git a/src/build/source_digest.cppm b/src/build/source_digest.cppm index 988d6403..3ea667c9 100644 --- a/src/build/source_digest.cppm +++ b/src/build/source_digest.cppm @@ -14,6 +14,12 @@ // ninja's own assumption. A file whose stat moved is read again, so a touched // file whose bytes did not change leaves the digest as it was. // +// A RACILY CLEAN FILE IS READ AGAIN (git's rule). A file written in the same +// timestamp tick as the walk that recorded it can be written again in that +// tick with the same size, and its stat does not move. A row whose mtime is +// not at least `kSettled` older than the walk that recorded it is therefore +// recorded without a hash, and the next walk reads the file. +// // WHAT IS WALKED. Every regular file below the root, except `target`, `.git` // and `.mcpp`, and except a subdirectory holding its own `mcpp.toml`: that is // another package (a workspace member), with its own digest where something @@ -40,6 +46,10 @@ namespace fs = std::filesystem; namespace { +// Coarser than the timestamp granularity of every file system mcpp builds on +// (FAT's is two seconds). +constexpr auto kSettled = std::chrono::seconds(3); + struct Row { std::uintmax_t size = 0; long long mtime = 0; @@ -94,8 +104,8 @@ std::string source_tree_digest(const fs::path& root, const fs::path& statCache) r.size = it->file_size(tec); r.mtime = static_cast( it->last_write_time(tec).time_since_epoch().count()); - if (auto p = previous.find(rel); - p != previous.end() && p->second.size == r.size && p->second.mtime == r.mtime) { + if (auto p = previous.find(rel); p != previous.end() && !p->second.hash.empty() + && p->second.size == r.size && p->second.mtime == r.mtime) { r.hash = p->second.hash; } else { r.hash = mcpp::toolchain::hash_file(it->path()); @@ -109,6 +119,12 @@ std::string source_tree_digest(const fs::path& root, const fs::path& statCache) for (auto const& [rel, r] : now) text += std::format("{}\t{}\n", rel, r.hash); const auto digest = mcpp::toolchain::hash_string(text); + // The rows a later walk may trust: those settled before this one. + const auto settled = static_cast( + (fs::file_time_type::clock::now() - kSettled).time_since_epoch().count()); + for (auto& [rel, r] : now) + if (r.mtime > settled) { r.hash.clear(); moved = true; } + if (moved && !statCache.empty()) { fs::create_directories(statCache.parent_path(), ec); const auto tmp = statCache.string() + ".tmp"; diff --git a/src/fallback/legacy_dirs.cppm b/src/fallback/legacy_dirs.cppm index 528334bf..9b6c2ff2 100644 --- a/src/fallback/legacy_dirs.cppm +++ b/src/fallback/legacy_dirs.cppm @@ -1,6 +1,6 @@ // mcpp.fallback.legacy_dirs — legacy xpkg directory scan. // -// Last-resort fallback scan (COMPAT, remove in 1.0.0): walk xpkgs/ for a +// Last-resort fallback scan (COMPAT(internal:legacy-xpkgs-scan)): walk xpkgs/ for a // directory that holds the requested package under an older naming layout. // // WHY THE SCAN IS NAMESPACE-BOUND diff --git a/src/modgraph/validate.cppm b/src/modgraph/validate.cppm index 7eaf35a5..a2a22bc1 100644 --- a/src/modgraph/validate.cppm +++ b/src/modgraph/validate.cppm @@ -96,8 +96,11 @@ ValidateReport validate(const Graph& g, if (!manifest.modules.exports_.empty()) { std::set declared(manifest.modules.exports_.begin(), manifest.modules.exports_.end()); - const auto iface = library_interface(g, manifest, projectRoot, - manifest.package.name, {}); + // The name the scan records on the package's units. + const auto unitsName = manifest.package.namespace_.empty() + ? manifest.package.name + : manifest.package.namespace_ + "." + manifest.package.name; + const auto iface = library_interface(g, manifest, projectRoot, unitsName, {}); for (auto& m : iface.publicModules) { if (!declared.contains(m)) { r.errors.push_back({{}, diff --git a/src/pm/package_fetcher.cppm b/src/pm/package_fetcher.cppm index 7586c01b..923e3b1e 100644 --- a/src/pm/package_fetcher.cppm +++ b/src/pm/package_fetcher.cppm @@ -182,7 +182,7 @@ public: // dirs). Used to feed `scan_short_name_matches` on the failure path. std::vector builtin_index_roots() const; - // ─── Legacy overloads (COMPAT, remove in 1.0.0) ───────────── + // ─── Legacy overloads (COMPAT(internal:legacy-fetcher-api)) ───────────── // // Accept a raw package name string and infer namespace from it. // New code should use the (ns, shortName) overloads above. @@ -967,7 +967,7 @@ Fetcher::builtin_index_roots() const return roots; } -// ─── Legacy read_xpkg_lua (COMPAT, remove in 1.0.0) ───────────────── +// ─── Legacy read_xpkg_lua (COMPAT(internal:legacy-fetcher-api)) ───────────────── // // Infers namespace from the raw package_name string and delegates to the // namespace-aware overload. @@ -1321,7 +1321,7 @@ Fetcher::install_path(std::string_view ns, std::string_view shortName, if (auto p = try_dir(dirName)) return *p; } - // Last-resort fallback scan (COMPAT, remove in 1.0.0): walk xpkgs/ for a + // Last-resort fallback scan (COMPAT(internal:legacy-fetcher-api)): walk xpkgs/ for a // directory holding this package under an older layout. // // The bare `-x-` arm is bound to the namespaces the caller @@ -1342,7 +1342,7 @@ Fetcher::install_path(std::string_view ns, std::string_view shortName, return std::nullopt; } -// ─── Legacy install_path (COMPAT, remove in 1.0.0) ────────────────── +// ─── Legacy install_path (COMPAT(internal:legacy-fetcher-api)) ────────────────── // // Infers namespace from the raw name string and delegates to the // namespace-aware overload. diff --git a/src/pm/resolver.cppm b/src/pm/resolver.cppm index b7df0b24..101894f8 100644 --- a/src/pm/resolver.cppm +++ b/src/pm/resolver.cppm @@ -74,7 +74,7 @@ try_merge_semver(std::string_view ns, std::string_view shortName, const mcpp::pm::IndexRoute& route, const mcpp::platform::PlatformKey& platform); -// ─── Legacy overloads (COMPAT, remove in 1.0.0) ───────────────────── +// ─── Legacy overloads (COMPAT(internal:legacy-resolver-api)) ───────────────────── std::expected resolve_semver(std::string_view name, @@ -287,7 +287,7 @@ try_merge_semver(std::string_view ns, std::string_view shortName, return resolve_semver(ns, shortName, merged, route, platform); } -// ─── Legacy overloads (COMPAT, remove in 1.0.0) ───────────────────── +// ─── Legacy overloads (COMPAT(internal:legacy-resolver-api)) ───────────────────── std::expected resolve_semver(std::string_view name, diff --git a/src/project.cppm b/src/project.cppm index 9176c708..6fac1eb0 100644 --- a/src/project.cppm +++ b/src/project.cppm @@ -1110,6 +1110,9 @@ virtual_workspace_root(const mcpp::manifest::Manifest& workspace, v.workspace = workspace.workspace; v.xlings.subos = workspace.xlings.subos; v.xlings.subosDeclared = workspace.xlings.subosDeclared; + // Where a payload comes from is a root-position statement, received from + // `[workspace.xlings.overrides]` as the tables above are (mcpp#797, R4). + v.xlings.overrides = first.xlings.overrides; auto& b = v.buildConfig; auto const& f = first.buildConfig; b.sourcesDeclared = true; diff --git a/tests/e2e/168_build_mcpp_musl_host_static.sh b/tests/e2e/168_build_mcpp_musl_host_static.sh index f6b0e696..c5b7b708 100755 --- a/tests/e2e/168_build_mcpp_musl_host_static.sh +++ b/tests/e2e/168_build_mcpp_musl_host_static.sh @@ -96,7 +96,8 @@ EOF exit 1 } -helper=target/.build-mcpp/build.mcpp.bin +# The program as the build programs' graph linked it (mcpp#797). +helper=$(find target/.build-mcpp/host -path '*/bin/*' -name build-program -type f | head -1) test -f helper-ran || { cat build.log; echo "FAIL: build.mcpp did not run"; exit 1; } test -x "$helper" || { cat build.log; echo "FAIL: helper binary missing"; exit 1; } diff --git a/tests/e2e/809_failed_build_program_hides_no_device_source.sh b/tests/e2e/809_failed_build_program_hides_no_device_source.sh index 0d4e69cd..df59d9a6 100755 --- a/tests/e2e/809_failed_build_program_hides_no_device_source.sh +++ b/tests/e2e/809_failed_build_program_hides_no_device_source.sh @@ -58,7 +58,7 @@ version = "0.1.0" [language] standard = "c++23" -[dependencies] +[build-dependencies] t809.rules = { path = "../rules", features = ["x"] } [build] diff --git a/tests/e2e/854_a_test_over_several_members_plans_and_builds_once.sh b/tests/e2e/854_a_test_over_several_members_plans_and_builds_once.sh index 585af862..c81e2edf 100755 --- a/tests/e2e/854_a_test_over_several_members_plans_and_builds_once.sh +++ b/tests/e2e/854_a_test_over_several_members_plans_and_builds_once.sh @@ -93,7 +93,7 @@ done runs() { find . -name runs.log -exec cat {} + | wc -l | tr -d ' '; } core_edges() { awk -F'\t' '$4 ~ /(^|\/)obj\/core\/src\/core\.m\.o$/' "$1/.ninja_log" | wc -l | tr -d ' '; } -build_dir() { find target -name build.ninja -exec dirname {} \; | head -1; } +build_dir() { find target -name build.ninja -not -path '*/.build-mcpp/*' -exec dirname {} \; | head -1; } reset() { rm -rf target core/target a/target b/target; } # ── F ────────────────────────────────────────────────────────────────────── @@ -105,7 +105,7 @@ grep -q "Workspace building 2 members: a, b" f.log || fail "F: the plan does not grep -q "member 'a' (2/3) ok — 1 passed" f.log && grep -q "member 'b' (3/3) ok — 1 passed" f.log \ || fail "F: a and b did not both run their test" f.log grep -q "workspace result ok. 3 member(s); 2 passed" f.log || fail "F: the workspace result is not that of the two tests" f.log -[ "$(find target -name build.ninja | wc -l | tr -d ' ')" = 1 ] || fail "F: the members were built in more than one directory" f.log +[ "$(find target -name build.ninja -not -path '*/.build-mcpp/*' | wc -l | tr -d ' ')" = 1 ] || fail "F: the members were built in more than one directory" f.log [ "$(core_edges "$(build_dir)")" = 1 ] || fail "F: core was compiled more than once" "$(build_dir)/.ninja_log" echo "ok: F, the members' shared build program ran once, in one plan, and each member's tests ran" diff --git a/tests/e2e/858_the_engines_module_is_compiled_once_per_home.sh b/tests/e2e/858_the_engines_module_is_compiled_once_per_home.sh index 8892c61f..a061d927 100755 --- a/tests/e2e/858_the_engines_module_is_compiled_once_per_home.sh +++ b/tests/e2e/858_the_engines_module_is_compiled_once_per_home.sh @@ -1,27 +1,27 @@ #!/usr/bin/env bash # requires: -# 858 -- the bundled `mcpp` module is compiled once per mcpp version and host -# compiler, for every project that shares a home. +# 858 -- the engine interface (`mcpp.core`) is compiled once per mcpp version and +# host compiler, for every project that shares a home. # -# #748 (B1). The module is the engine's own text: identical in every project -# for one mcpp version and one host compiler. Each build program used to compile -# it into its own directory. It is now an entry of the global cache, in the -# layout of a dependency's entry, so `mcpp cache list`, `info`, `verify` and `gc` -# treat it as they treat one. +# #748 (B1), mcpp#797. The interface is the engine's own text, written as a +# package under $MCPP_HOME/engine//modules, and the build programs' +# graph depends on it like on any library. Its compiled entry is kept in the +# global cache under `_engine/mcpp-build-module@`, so `mcpp cache list`, +# `info`, `verify` and `gc` treat it as they treat a dependency's entry. # # Criteria: -# A. The first project compiles the module (the compile commands are logged -# under `buildmcpp-host`); a second project with another name in another -# directory, sharing the home, compiles nothing of it. +# A. The first project's build programs' graph compiles the interface; a +# second project with another name in another directory, sharing the home, +# places the cached entry and compiles nothing of it. # B. The entry is one of `mcpp cache list`, is complete to `mcpp cache verify`, # and records inputs that name no project: the flags it was compiled with # are the toolchain's, so they hold for every project. # C. `mcpp cache gc` collects it, and the next build compiles it again. # D. `--cache local` writes nothing below the global cache's package tree; the -# module is then kept in the project's own store. -# E. A host module of an index package, whose sources are in the immutable -# store, is kept in the global cache too, under its package address, and a -# second project that imports it compiles nothing of it. +# interface is compiled in the project's build programs' graph. +# E. A library of an index package that a build program imports is kept in +# the global cache too, under its package address, and a second project +# that imports it compiles nothing of it. set -e source "$(dirname "$0")/_host_path.sh" @@ -56,19 +56,21 @@ project "$TMP/beta" beta858 ALPHA_HOST="$(host_path "$TMP/alpha")" BETA_HOST="$(host_path "$TMP/beta")" -compiled_module() { grep -cE "mcpp module (precompile|compile|object|mcpp\.core (precompile|compile|object)):" "$1" || true; } +# The build programs' graph of project $1, and how it obtains object $2. +host_ninja() { find "$1/target/.build-mcpp/host" -name build.ninja | head -1; } +compiles() { grep -qE "^build obj/[^ ]*/$2( [|] [^:]*)? : cxx_module " "$(host_ninja "$1")"; } +places() { grep -qE "^build obj/[^ ]*/$2 : stage_file [^ ]*/build-cache/v1/pkg/$3" "$(host_ninja "$1")"; } # A ( cd "$TMP/alpha" && MCPP_VERBOSE=1 "$MCPP" build > "$TMP/alpha.log" 2>&1 ) \ || fail "the first project did not build" "$TMP/alpha.log" -[ "$(compiled_module "$TMP/alpha.log")" -ge 2 ] \ - || fail "A: the first project did not compile the bundled module" "$TMP/alpha.log" +compiles "$TMP/alpha" 'mcpp\.m\.o' \ + || fail "A: the first project did not compile the engine interface" "$TMP/alpha.log" "$(host_ninja "$TMP/alpha")" ( cd "$TMP/beta" && MCPP_VERBOSE=1 "$MCPP" build > "$TMP/beta.log" 2>&1 ) \ || fail "the second project did not build" "$TMP/beta.log" -[ "$(compiled_module "$TMP/beta.log")" = 0 ] \ - || fail "A: the second project compiled the bundled module again" "$TMP/beta.log" -grep -qE "bundled module mcpp: entry [0-9a-f]+ global \(reused\)" "$TMP/beta.log" \ - || fail "A: the second project's log does not say it reused the entry" "$TMP/beta.log" +places "$TMP/beta" 'mcpp\.m\.o' '_engine/mcpp-build-module@' \ + || fail "A: the second project did not place the cached entry" "$TMP/beta.log" "$(host_ninja "$TMP/beta")" +compiles "$TMP/beta" 'mcpp\.m\.o' && fail "A: the second project compiled the engine interface again" "$(host_ninja "$TMP/beta")" # B "$MCPP" cache list > "$TMP/list.log" 2>&1 @@ -77,7 +79,7 @@ grep -qE "bundled module mcpp: entry [0-9a-f]+ global \(reused\)" "$TMP/beta.log "$MCPP" cache verify > "$TMP/verify.log" 2>&1 || fail "B: mcpp cache verify failed" "$TMP/verify.log" grep -q "all complete" "$TMP/verify.log" || fail "B: the entry is not complete" "$TMP/verify.log" "$MCPP" cache info _engine/mcpp-build-module > "$TMP/info.log" 2>&1 -grep -q '"interface_sha256"' "$TMP/info.log" || fail "B: the entry does not record the digest of its text" "$TMP/info.log" +grep -q 'mcpp.cppm' "$TMP/info.log" || fail "B: the entry does not record its sources" "$TMP/info.log" if grep -qF "$ALPHA_HOST" "$TMP/info.log" || grep -qF "$BETA_HOST" "$TMP/info.log"; then fail "B: the recorded inputs name a project, so the entry could not serve another" "$TMP/info.log" fi @@ -89,8 +91,8 @@ grep -q "Collected 1 entries" "$TMP/gc.log" || fail "C: gc did not collect the e ( cd "$TMP/alpha" && touch build.mcpp && echo '// edited' >> build.mcpp \ && MCPP_VERBOSE=1 "$MCPP" build > "$TMP/alpha2.log" 2>&1 ) \ || fail "C: the build after gc failed" "$TMP/alpha2.log" -[ "$(compiled_module "$TMP/alpha2.log")" -ge 2 ] \ - || fail "C: the module was not compiled again after gc collected it" "$TMP/alpha2.log" +compiles "$TMP/alpha" 'mcpp\.m\.o' \ + || fail "C: the interface was not compiled again after gc collected it" "$TMP/alpha2.log" # D export MCPP_HOME="$TMP/mcpp-home-local" @@ -101,10 +103,8 @@ project "$TMP/gamma" gamma858 if [ -n "$(find "$MCPP_HOME/build-cache/v1/pkg" -name entry.json 2>/dev/null)" ]; then fail "D: --cache local wrote an entry below the global cache's package tree" "$TMP/gamma.log" fi -[ -n "$(find "$TMP/gamma/target/.build-mcpp/host-modules" -name entry.json 2>/dev/null)" ] \ - || fail "D: the module is not in the project's own store" "$TMP/gamma.log" -grep -qE "bundled module mcpp: entry [0-9a-f]+ workspace \(compiled\)" "$TMP/gamma.log" \ - || fail "D: the log does not say the module went to the workspace store" "$TMP/gamma.log" +compiles "$TMP/gamma" 'mcpp\.m\.o' \ + || fail "D: the interface was not compiled in the project's build programs' graph" "$TMP/gamma.log" # E. The package is in a local index, and its payload is where the store keeps # installed packages: the two facts the cache admits a package on. @@ -117,7 +117,7 @@ cat > "$INDEX_DIR/pkgs/r/rule-dep.lua" <<'LUA' package = { spec = "1", name = "rule-dep", - description = "A package that offers a build rule as a host module", + description = "A library a build program imports", licenses = {"MIT"}, type = "package", xpm = { @@ -175,25 +175,18 @@ BUILDMCPP indexed "$TMP/delta" delta858 indexed "$TMP/epsilon" epsilon858 ( cd "$TMP/delta" && MCPP_VERBOSE=1 "$MCPP" build > "$TMP/delta.log" 2>&1 ) \ - || fail "E: the first project that imports the index package's host module did not build" "$TMP/delta.log" -grep -qE "host module 't858\.rule' (precompile|compile):" "$TMP/delta.log" \ - || fail "E: the first project did not compile the host module" "$TMP/delta.log" -entry=$(grep -rl '"module": "t858.rule"' "$MCPP_HOME/build-cache/v1/pkg" --include=entry.json 2>/dev/null | head -1) -[ -n "$entry" ] || fail "E: the host module of an index package is not in the global cache" "$TMP/delta.log" + || fail "E: the first project that imports the index package's library did not build" "$TMP/delta.log" +compiles "$TMP/delta" 'rule-dep\.m\.o' \ + || fail "E: the first project did not compile the library" "$TMP/delta.log" "$(host_ninja "$TMP/delta")" +entry=$(find "$MCPP_HOME/build-cache/v1/pkg" -path '*local-dev.rule-dep@1.0.0*' -name entry.json 2>/dev/null | head -1) +[ -n "$entry" ] || fail "E: the library of an index package is not in the global cache" "$TMP/delta.log" case "$entry" in */pkg/local-dev/local-dev.rule-dep@1.0.0/*/entry.json) ;; *) fail "E: the entry is not at its package address: $entry" ;; esac -if [ -d "$TMP/delta/target/.build-mcpp/host-modules" ] \ - && grep -rl 't858.rule' "$TMP/delta/target/.build-mcpp/host-modules" --include=entry.json 2>/dev/null | grep -q .; then - fail "E: a host module of an index package was also written to the project's store" -fi ( cd "$TMP/epsilon" && MCPP_VERBOSE=1 "$MCPP" build > "$TMP/epsilon.log" 2>&1 ) \ || fail "E: the second project did not build" "$TMP/epsilon.log" -if grep -qE "host module 't858\.rule' (precompile|compile):" "$TMP/epsilon.log"; then - fail "E: the second project compiled the index package's host module again" "$TMP/epsilon.log" -fi -grep -qE "host module 't858\.rule': entry [0-9a-f]+ global \(reused\)" "$TMP/epsilon.log" \ - || fail "E: the second project's log does not say it reused the entry" "$TMP/epsilon.log" +places "$TMP/epsilon" 'rule-dep\.m\.o' 'local-dev/local-dev.rule-dep@1.0.0' \ + || fail "E: the second project did not place the cached library" "$TMP/epsilon.log" "$(host_ninja "$TMP/epsilon")" echo "PASS: 858_the_engines_module_is_compiled_once_per_home" diff --git a/tests/e2e/860_a_workspaces_programs_compile_at_the_same_time.sh b/tests/e2e/860_a_workspaces_programs_compile_at_the_same_time.sh index 8417d56c..1838f374 100755 --- a/tests/e2e/860_a_workspaces_programs_compile_at_the_same_time.sh +++ b/tests/e2e/860_a_workspaces_programs_compile_at_the_same_time.sh @@ -4,14 +4,15 @@ # the job count, and run one after another; the failure reported is the first # in the order they run in. # -# #748 (B2). A member's program runs after the programs of the members it -# depends on, and that order is kept. Their compiles have no order, and each -# used to wait for the one before it. +# #748 (B2), mcpp#797. A member's program runs after the programs of the +# members it depends on, and that order is kept. Their compiles have no order: +# every program of the build is compiled in one build of the build programs' +# graph, whose ninja runs up to the job count at once. # # Criteria: # A. With `-j 4`, the compile intervals of three independent programs -# overlap. The test compares the timestamps the verbose log records at the -# start and the end of each compile, not the time the build took. +# overlap. The test compares the start and the end ninja records for each +# program's object in the graph's `.ninja_log`, not the time the build took. # B. With `-j 1`, they do not: each compile ends before the next begins. # C. When two programs fail to compile, the failure reported is the first in # the order the programs run in, whichever compile finished first. The @@ -64,21 +65,21 @@ int main() { EOF done -# Milliseconds since midnight of the first log line that matches $2 in $1. -at() { - awk -v pat="$2" '$0 ~ pat { split($3, t, /[]:.]/); print ((t[1] * 60 + t[2]) * 60 + t[3]) * 1000 + t[4]; exit }' "$1" -} +# The start and the end, in milliseconds, ninja recorded for the object of +# member $2's program in the build programs' graph. ($1, the build's log, is +# kept for the failure message.) +host_log() { find target/.build-mcpp/host -name .ninja_log | head -1; } # The names of the programs that ran, one per line, in the order the lines came. ran_lines() { grep -E "^ *build\.mcpp p[abc] .*ran [0-9]" "$1" | sed -E 's/^ *build\.mcpp (p[abc]) .*/\1/'; } -span() { echo "$(at "$1" "build\\.mcpp $2: compile start") $(at "$1" "build\\.mcpp $2: compile end")"; } +span() { awk -F'\t' -v m="$2" '$4 ~ ("^obj/" m "-[0-9a-f]+/.*build\\.mcpp\\.o$") { print $1, $2; exit }' "$(host_log)"; } # A -MCPP_VERBOSE=1 "$MCPP" build --workspace -j 4 > a.log 2>&1 || fail "A: the workspace did not build at -j 4" a.log +"$MCPP" build --workspace -j 4 > a.log 2>&1 || fail "A: the workspace did not build at -j 4" a.log read -r sa ea <<< "$(span a.log pa)" read -r sb eb <<< "$(span a.log pb)" read -r sc ec <<< "$(span a.log pc)" for v in "$sa" "$ea" "$sb" "$eb" "$sc" "$ec"; do - [ -n "$v" ] || fail "A: the log does not record the start and the end of every compile" a.log + [ -n "$v" ] || fail "A: the build programs' .ninja_log does not record every compile" a.log "$(host_log)" done latest_start=$sa; [ "$sb" -gt "$latest_start" ] && latest_start=$sb; [ "$sc" -gt "$latest_start" ] && latest_start=$sc earliest_end=$ea; [ "$eb" -lt "$earliest_end" ] && earliest_end=$eb; [ "$ec" -lt "$earliest_end" ] && earliest_end=$ec @@ -91,11 +92,11 @@ order=$(ran_lines a.log | tr '\n' ' ') # B rm -rf target pa/target pb/target pc/target -MCPP_VERBOSE=1 "$MCPP" build --workspace -j 1 > b.log 2>&1 || fail "B: the workspace did not build at -j 1" b.log +"$MCPP" build --workspace -j 1 > b.log 2>&1 || fail "B: the workspace did not build at -j 1" b.log starts=() for m in pa pb pc; do read -r s e <<< "$(span b.log $m)" - [ -n "$s" ] && [ -n "$e" ] || fail "B: the log does not record the compile of $m" b.log + [ -n "$s" ] && [ -n "$e" ] || fail "B: the build programs' .ninja_log does not record the compile of $m" b.log "$(host_log)" starts+=("$s:$e") done sorted=$(printf '%s\n' "${starts[@]}" | sort -n) diff --git a/tests/e2e/861_concurrent_compiles_plan_what_serial_compiles_plan.sh b/tests/e2e/861_concurrent_compiles_plan_what_serial_compiles_plan.sh index 6060fdae..c6e6e248 100755 --- a/tests/e2e/861_concurrent_compiles_plan_what_serial_compiles_plan.sh +++ b/tests/e2e/861_concurrent_compiles_plan_what_serial_compiles_plan.sh @@ -94,7 +94,7 @@ done snapshot() { local out="$1" mkdir -p "$out" - find target -name build.ninja | sort > "$out/ninja.list" + find target -name build.ninja -not -path '*/.build-mcpp/*' | sort > "$out/ninja.list" [ -s "$out/ninja.list" ] || fail "no build.ninja was written" local n=0 while read -r f; do diff --git a/tests/e2e/867_a_pack_over_several_members_plans_builds_and_runs_programs_once.sh b/tests/e2e/867_a_pack_over_several_members_plans_builds_and_runs_programs_once.sh index 0df6865a..009f8950 100644 --- a/tests/e2e/867_a_pack_over_several_members_plans_builds_and_runs_programs_once.sh +++ b/tests/e2e/867_a_pack_over_several_members_plans_builds_and_runs_programs_once.sh @@ -192,12 +192,12 @@ echo "ok: B, each member's tree and distributable are those of mcpp pack -p c-build.log 2>&1 || fail "C: the workspace did not build" c-build.log -dirs=$(find target -name build.ninja -exec dirname {} \;) +dirs=$(find target -name build.ninja -not -path '*/.build-mcpp/*' -exec dirname {} \;) [ "$(echo "$dirs" | wc -l | tr -d ' ')" = 1 ] || fail "C: expected one build directory, got: $dirs" c-build.log before=$(objects "$dirs") [ "$before" -gt 0 ] || fail "C: the build compiled nothing" "$dirs/.ninja_log" "$MCPP" pack --workspace --format zap > c-pack.log 2>&1 || fail "C: the pack failed" c-pack.log -[ "$(find target -name build.ninja | wc -l | tr -d ' ')" = 1 ] || fail "C: the pack used another build directory" c-pack.log +[ "$(find target -name build.ninja -not -path '*/.build-mcpp/*' | wc -l | tr -d ' ')" = 1 ] || fail "C: the pack used another build directory" c-pack.log after=$(objects "$dirs") [ "$after" = "$before" ] || fail "C: the pack compiled $((after - before)) object(s) after the build" "$dirs/.ninja_log" c-pack.log echo "ok: C, after a build of the workspace the pack compiles nothing" diff --git a/tests/e2e/868_a_pack_over_several_members_is_refused_before_anything_is_compiled.sh b/tests/e2e/868_a_pack_over_several_members_is_refused_before_anything_is_compiled.sh index 25279811..069d48ea 100644 --- a/tests/e2e/868_a_pack_over_several_members_is_refused_before_anything_is_compiled.sh +++ b/tests/e2e/868_a_pack_over_several_members_is_refused_before_anything_is_compiled.sh @@ -35,9 +35,11 @@ MCPP="${MCPP:-mcpp}" export MCPP_HOME="$TMP/mcpp-home" source "$(dirname "$0")/_inherit_toolchain.sh" -# No object was compiled in $1, and no build was driven. +# No object of the members was compiled in $1, and no build of them was driven. +# The build programs are compiled while planning (they declare the providers), +# in the build programs' graph under target/.build-mcpp/host (mcpp#797). compiled_nothing() { - [ -z "$(find "$1" -name .ninja_log 2>/dev/null)" ] + [ -z "$(find "$1" -name .ninja_log -not -path '*/.build-mcpp/host/*' 2>/dev/null)" ] } # A program member named $1, optionally using the packages in $2. diff --git a/tests/e2e/897_each_configuration_has_its_own_generated_output.sh b/tests/e2e/897_each_configuration_has_its_own_generated_output.sh index 64e6e759..5899f374 100755 --- a/tests/e2e/897_each_configuration_has_its_own_generated_output.sh +++ b/tests/e2e/897_each_configuration_has_its_own_generated_output.sh @@ -69,7 +69,7 @@ done # Each configuration directory: what its binary prints, and what its outDir's # generated file says. They must agree, and the two must be dev and release. seen="" -for r in $(find target -name resolution.json); do +for r in $(find target -name resolution.json -not -path '*/.build-mcpp/*'); do d=$(dirname "$r") out=$(outdir "$d") [ -n "$out" ] || fail "resolution.json records no outDir: $r" diff --git a/tests/e2e/902_a_build_dependency_library_is_compiled_as_its_package.sh b/tests/e2e/902_a_build_dependency_library_is_compiled_as_its_package.sh new file mode 100755 index 00000000..af9f49c0 --- /dev/null +++ b/tests/e2e/902_a_build_dependency_library_is_compiled_as_its_package.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# requires: +# 902 -- a library in [build-dependencies] is usable from build.mcpp as the +# package it is (mcpp#797, R1). +# +# The shape is a module wrapper over a header library (toml++, nlohmann.json, +# fmt in mcpp-index): the interface unit's global module fragment includes the +# library's own header through the package's `include_dirs`. Before the build +# programs' graph, the interface was compiled alone with the build program's +# flags, and the include failed (`fatal error: hl/hl.hpp: No such file`). +# +# A a plain `[build-dependencies]` entry, with no `host-module`, gives the +# build program the library: `` resolves, the package's +# `defines` apply, and the value reaches the target through a directive; +# B `[lib] path` names an interface root outside `src/`, and the module is +# imported under the name it declares; +# C the library is not part of the target (no object of it in the target's +# plan), since only [build-dependencies] names it. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +mkdir -p "$TMP/hl/include/hl" "$TMP/hl/modules" "$TMP/app/src" +cat > "$TMP/hl/include/hl/hl.hpp" <<'EOF' +#pragma once +namespace hl { inline int answer() { return HL_BASE + 2; } } +EOF +cat > "$TMP/hl/modules/hl.cppm" <<'EOF' +module; +#include +export module acme.hl; +export namespace hl { using hl::answer; } +EOF +cat > "$TMP/hl/mcpp.toml" <<'EOF' +[package] +namespace = "acme" +name = "hl" +version = "0.1.0" +[lib] +path = "modules/hl.cppm" +[build] +sources = ["modules/hl.cppm"] +include_dirs = ["include"] +defines = ["HL_BASE=40"] +[targets.hl] +kind = "lib" +EOF +cat > "$TMP/app/mcpp.toml" <<'EOF' +[package] +name = "app" +version = "0.1.0" +[build-dependencies.acme] +hl = { path = "../hl" } +[targets.app] +kind = "bin" +main = "src/main.cpp" +EOF +cat > "$TMP/app/src/main.cpp" <<'EOF' +#include +int main() { std::printf("V=%d\n", V); } +EOF +cat > "$TMP/app/build.mcpp" <<'EOF' +import std; +import mcpp; +import acme.hl; +int main() { mcpp::cxxflag(std::format("-DV={}", hl::answer()).c_str()); } +EOF + +cd "$TMP/app" +"$MCPP" build > b.log 2>&1 || fail "A: a header-backed module library in [build-dependencies] did not build" b.log +out="$("$MCPP" run 2>/dev/null | tr -d '\r' | grep '^V=' | tail -1)" +[ "$out" = "V=42" ] || fail "A/B: the library's value did not reach the target: $out" b.log +grep -q "host module" b.log && fail "A: the old host-module path was reported" b.log +# C: the target's own plan compiles no unit of the library. +if find target -path '*/obj/*' -name 'hl.m.o*' -not -path '*/.build-mcpp/*' | grep -q .; then + fail "C: the build dependency was compiled into the target's plan" +fi +echo "PASS: 902_a_build_dependency_library_is_compiled_as_its_package" diff --git a/tests/e2e/903_one_package_in_both_tables_has_two_instances.sh b/tests/e2e/903_one_package_in_both_tables_has_two_instances.sh new file mode 100755 index 00000000..7d31a502 --- /dev/null +++ b/tests/e2e/903_one_package_in_both_tables_has_two_instances.sh @@ -0,0 +1,138 @@ +#!/usr/bin/env bash +# requires: elf python3 +# 903 -- one package in [dependencies] and [build-dependencies] has two +# instances, each correct (mcpp#797, R2, R3; design §3.5, M1-M3, M5-M7, M9, M10). +# +# The two declarations used to merge into one edge that carried the +# build-program request, and the target-reachability test read that request as +# "build time only": the package's sources were cleared and the target's +# `import` failed with `failed to read compiled module`. +# +# M1 same consumer, both tables: the target and build.mcpp both import it; +# M2 `host-module = true` written beside it changes nothing; +# M3 a feature requested only by the build dependency reaches the +# build-program instance and not the target instance; +# M5 a workspace: one member links it, another's build program uses it; +# M6 the target reaches it through another library, build.mcpp directly; +# M7 a dependency's own build.mcpp uses it while the root links it; +# M9 the build database has a target set and a `:host` set for it, and +# compile_commands.json describes the target units only; +# M10 an edit to its header recompiles both instances and reruns the program. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +mkdir -p "$TMP/hl/include/hl" "$TMP/hl/src" +cat > "$TMP/hl/include/hl/hl.hpp" <<'EOF' +#pragma once +namespace hl { +inline int answer() { return 42; } +inline int x_on() { +#ifdef HL_X + return 1; +#else + return 0; +#endif +} +} +EOF +cat > "$TMP/hl/src/hl.cppm" <<'EOF' +module; +#include +export module hl; +export namespace hl { using hl::answer; using hl::x_on; } +EOF +cat > "$TMP/hl/mcpp.toml" <<'EOF' +[package] +name = "hl" +version = "0.1.0" +[build] +sources = ["src/hl.cppm"] +include_dirs = ["include"] +[features] +x = { defines = ["HL_X"] } +[targets.hl] +kind = "lib" +EOF +program='import std; +import mcpp; +import hl; +int main() { + mcpp::cxxflag(std::format("-DV={}", hl::answer()).c_str()); + mcpp::cxxflag(std::format("-DX_IN_PROGRAM={}", hl::x_on()).c_str()); +}' +target_main='#include +import hl; +int main() { std::printf("V=%d target=%d xprog=%d xtarget=%d\n", V, hl::answer(), X_IN_PROGRAM, hl::x_on()); }' + +app() { # $1 dir, $2 extra manifest + mkdir -p "$1/src" + printf '[package]\nname = "%s"\nversion = "0.1.0"\n%s\n[targets.%s]\nkind = "bin"\nmain = "src/main.cpp"\n' \ + "$(basename "$1")" "$2" "$(basename "$1")" > "$1/mcpp.toml" + printf '%s\n' "$target_main" > "$1/src/main.cpp" + printf '%s\n' "$program" > "$1/build.mcpp" +} +run_line() { "$MCPP" run 2>/dev/null | grep '^V=' | tail -1; } + +# M1, M3 +app "$TMP/m1" $'[dependencies]\nhl = { path = "../hl" }\n[build-dependencies]\nhl = { path = "../hl", features = ["x"] }' +cd "$TMP/m1" +"$MCPP" build > b.log 2>&1 || fail "M1: one package in both tables did not build" b.log +[ "$(run_line)" = "V=42 target=42 xprog=1 xtarget=0" ] || fail "M1/M3: $(run_line)" b.log + +# M2 +sed -i 's/features = \["x"\] }/features = ["x"], host-module = true }/' mcpp.toml +"$MCPP" build > b2.log 2>&1 || fail "M2: host-module = true beside the entry broke the build" b2.log +grep -q "deprecated" b2.log && fail "M2: stating the default was reported" b2.log +[ "$(run_line)" = "V=42 target=42 xprog=1 xtarget=0" ] || fail "M2: $(run_line)" b2.log + +# M9 +"$MCPP" emit build-database --format json > db.json 2>/dev/null || fail "M9: emit build-database failed" +python3 - db.json <<'PY' || fail "M9: the build database does not describe both instances" db.json +import json, sys +doc = json.load(open(sys.argv[1]))["data"]["database"] +sets = {s["name"]: s for s in doc["sets"]} +assert "hl" in sets or any(n.endswith(".hl") for n in sets), sets.keys() +host = [n for n in sets if n.endswith("hl:host")] +assert host, sets.keys() +assert sets[host[0]]["ide"]["configuration"] == "build-program" +assert any(n.endswith(":build") for n in sets), sets.keys() +PY +"$MCPP" emit build-database --spec compile-commands > cc.json 2>/dev/null +grep -q 'build.mcpp' cc.json && fail "M9: compile_commands describes build.mcpp" cc.json + +# M10 +sleep 1 +sed -i 's/return 42;/return 43;/' "$TMP/hl/include/hl/hl.hpp" +"$MCPP" build > b3.log 2>&1 || fail "M10: the build after a header edit failed" b3.log +[ "$(run_line)" = "V=43 target=43 xprog=1 xtarget=0" ] || fail "M10: an instance kept the old header: $(run_line)" b3.log +sed -i 's/return 43;/return 42;/' "$TMP/hl/include/hl/hl.hpp" + +# M5 +mkdir -p "$TMP/ws" +printf '[workspace]\nmembers = ["a", "b"]\n' > "$TMP/ws/mcpp.toml" +app "$TMP/ws/a" $'[dependencies]\nhl = { path = "../../hl" }' +printf '#include \nimport hl;\nint main() { std::printf("V=%%d\\n", hl::answer()); }\n' > "$TMP/ws/a/src/main.cpp" +rm "$TMP/ws/a/build.mcpp" +app "$TMP/ws/b" $'[build-dependencies]\nhl = { path = "../../hl" }' +printf '#include \nint main() { std::printf("V=%%d\\n", V); }\n' > "$TMP/ws/b/src/main.cpp" +cd "$TMP/ws" +"$MCPP" build --workspace > w.log 2>&1 || fail "M5: the workspace did not build" w.log +[ "$("$MCPP" run -p a 2>/dev/null | grep '^V=' | tail -1)" = "V=42" ] || fail "M5: member a" w.log +[ "$("$MCPP" run -p b 2>/dev/null | grep '^V=' | tail -1)" = "V=42" ] || fail "M5: member b" w.log + +# M6, M7 +mkdir -p "$TMP/mid/src" +printf '[package]\nname = "mid"\nversion = "0.1.0"\n[dependencies]\nhl = { path = "../hl" }\n[build-dependencies]\nhl = { path = "../hl" }\n[targets.mid]\nkind = "lib"\n' > "$TMP/mid/mcpp.toml" +printf 'export module mid;\nimport hl;\nexport int via_mid() { return hl::answer() + MID; }\n' > "$TMP/mid/src/mid.cppm" +printf 'import std;\nimport mcpp;\nimport hl;\nint main() { mcpp::cxxflag(std::format("-DMID={}", hl::answer() - 42).c_str()); }\n' > "$TMP/mid/build.mcpp" +app "$TMP/m6" $'[dependencies]\nmid = { path = "../mid" }\n[build-dependencies]\nhl = { path = "../hl" }' +printf '#include \nimport mid;\nint main() { std::printf("V=%%d mid=%%d\\n", V, via_mid()); }\n' > "$TMP/m6/src/main.cpp" +cd "$TMP/m6" +"$MCPP" build > b.log 2>&1 || fail "M6/M7: a library reached through another one, used by two programs, did not build" b.log +[ "$(run_line)" = "V=42 mid=42" ] || fail "M6/M7: $(run_line)" b.log + +echo "PASS: 903_one_package_in_both_tables_has_two_instances" diff --git a/tests/e2e/904_a_build_dependency_feature_stays_out_of_the_target.sh b/tests/e2e/904_a_build_dependency_feature_stays_out_of_the_target.sh new file mode 100755 index 00000000..0eced0d7 --- /dev/null +++ b/tests/e2e/904_a_build_dependency_feature_stays_out_of_the_target.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# requires: elf +# 904 -- features are activated per instance (mcpp#797, R3). +# +# A feature one package requests of a library in its [build-dependencies] is +# a request of the library's build-program instance. It used to be unioned +# with every other request of the package, so it reached the target instance: +# the target compile of the library carried the feature's define. +# +# A the target instance of `hl` does not have feature `x`, which only +# `mid`'s [build-dependencies] requests; +# B the build-program instance has it: mid's build program sees `x`. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +mkdir -p "$TMP/hl/src" "$TMP/mid/src" "$TMP/app/src" +cat > "$TMP/hl/src/hl.cppm" <<'EOF' +export module hl; +export namespace hl { +inline int x_on() { +#ifdef HL_X + return 1; +#else + return 0; +#endif +} +} +EOF +printf '[package]\nname = "hl"\nversion = "0.1.0"\n[features]\nx = { defines = ["HL_X"] }\n[targets.hl]\nkind = "lib"\n' > "$TMP/hl/mcpp.toml" +printf '[package]\nname = "mid"\nversion = "0.1.0"\n[build-dependencies]\nhl = { path = "../hl", features = ["x"] }\n[targets.mid]\nkind = "lib"\n' > "$TMP/mid/mcpp.toml" +printf 'export module mid;\nexport int mid_x() { return MID_X; }\n' > "$TMP/mid/src/mid.cppm" +printf 'import std;\nimport mcpp;\nimport hl;\nint main() { mcpp::cxxflag(std::format("-DMID_X={}", hl::x_on()).c_str()); }\n' > "$TMP/mid/build.mcpp" +printf '[package]\nname = "app"\nversion = "0.1.0"\n[dependencies]\nhl = { path = "../hl" }\nmid = { path = "../mid" }\n[targets.app]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/app/mcpp.toml" +printf '#include \nimport hl;\nimport mid;\nint main() { std::printf("target=%%d program=%%d\\n", hl::x_on(), mid_x()); }\n' > "$TMP/app/src/main.cpp" + +cd "$TMP/app" +"$MCPP" build > b.log 2>&1 || fail "the build failed" b.log +out="$("$MCPP" run 2>/dev/null | grep '^target=' | tail -1)" +[ "$out" = "target=0 program=1" ] || fail "A/B: $out (want target=0 program=1)" b.log +echo "PASS: 904_a_build_dependency_feature_stays_out_of_the_target" diff --git a/tests/e2e/905_a_build_program_links_its_libraries_whole.sh b/tests/e2e/905_a_build_program_links_its_libraries_whole.sh new file mode 100755 index 00000000..0209ea44 --- /dev/null +++ b/tests/e2e/905_a_build_program_links_its_libraries_whole.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# requires: elf +# 905 -- a library a build program uses is compiled and linked whole +# (mcpp#797, #797 Q4/Q5): its implementation units, its C sources and its own +# dependencies reach the build program. The single-unit path compiled one +# object per interface unit and nothing else, so a library whose functions are +# defined out of line could not be used at all. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +mkdir -p "$TMP/base/src" "$TMP/lib/src" "$TMP/app/src" +printf 'export module acme.base;\nexport int base_two();\n' > "$TMP/base/src/base.cppm" +printf 'module acme.base;\nint base_two() { return 2; }\n' > "$TMP/base/src/base_impl.cpp" +printf '[package]\nname = "base"\nversion = "0.1.0"\n[targets.base]\nkind = "lib"\n' > "$TMP/base/mcpp.toml" +printf 'export module lib;\nexport int lib_value();\n' > "$TMP/lib/src/lib.cppm" +cat > "$TMP/lib/src/lib_impl.cpp" <<'EOF' +module lib; +import acme.base; +extern "C" int c_forty(); +int lib_value() { return c_forty() + base_two(); } +EOF +printf 'int c_forty(void) { return 40; }\n' > "$TMP/lib/src/forty.c" +printf '[package]\nname = "lib"\nversion = "0.1.0"\n[dependencies]\nbase = { path = "../base" }\n[targets.lib]\nkind = "lib"\n' > "$TMP/lib/mcpp.toml" +printf '[package]\nname = "app"\nversion = "0.1.0"\n[build-dependencies]\nlib = { path = "../lib" }\n[targets.app]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/app/mcpp.toml" +printf '#include \nint main() { std::printf("V=%%d\\n", V); }\n' > "$TMP/app/src/main.cpp" +printf 'import std;\nimport mcpp;\nimport lib;\nint main() { mcpp::cxxflag(std::format("-DV={}", lib_value()).c_str()); }\n' > "$TMP/app/build.mcpp" + +cd "$TMP/app" +"$MCPP" build > b.log 2>&1 || fail "a library with implementation units, C sources and a dependency did not link into the build program" b.log +[ "$("$MCPP" run 2>/dev/null | grep '^V=' | tail -1)" = "V=42" ] || fail "the program computed another value" b.log +echo "PASS: 905_a_build_program_links_its_libraries_whole" diff --git a/tests/e2e/906_a_build_program_is_compiled_when_its_graph_changes.sh b/tests/e2e/906_a_build_program_is_compiled_when_its_graph_changes.sh new file mode 100755 index 00000000..af2554a6 --- /dev/null +++ b/tests/e2e/906_a_build_program_is_compiled_when_its_graph_changes.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# requires: elf +# 906 -- a build program is compiled again exactly when its identity moves, and +# a build whose programs stand does not plan their graph (mcpp#797, R6). +# +# A an edit to a header a path library includes reruns the program with the +# new value (the identity used to hash the interface file only); +# B a build in which nothing changed states that the identity is current +# and plans no build programs' graph; +# C the identity and the reason are in the verbose log. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +mkdir -p "$TMP/hl/src" "$TMP/app/src" +printf '#define HL_VALUE 42\n' > "$TMP/hl/src/value.h" +printf 'module;\n#include "value.h"\nexport module hl;\nexport int hl_value() { return HL_VALUE; }\n' > "$TMP/hl/src/hl.cppm" +printf '[package]\nname = "hl"\nversion = "0.1.0"\n[targets.hl]\nkind = "lib"\n' > "$TMP/hl/mcpp.toml" +printf '[package]\nname = "app"\nversion = "0.1.0"\n[build-dependencies]\nhl = { path = "../hl" }\n[targets.app]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/app/mcpp.toml" +printf '#include \nint main() { std::printf("V=%%d\\n", V); }\n' > "$TMP/app/src/main.cpp" +printf 'import std;\nimport mcpp;\nimport hl;\nint main() { mcpp::cxxflag(std::format("-DV={}", hl_value()).c_str()); }\n' > "$TMP/app/build.mcpp" + +cd "$TMP/app" +"$MCPP" build > b1.log 2>&1 || fail "the first build failed" b1.log +[ "$("$MCPP" run 2>/dev/null | grep '^V=' | tail -1)" = "V=42" ] || fail "first value" b1.log + +# B, C: nothing changed. A source the fast path sweeps is touched so that +# planning runs; the program's identity stands. +sleep 1 +touch src/main.cpp +"$MCPP" build --verbose > b2.log 2>&1 || fail "B: the second build failed" b2.log +grep -qE "build-program: app: identity [0-9a-f]+ current" b2.log || fail "C: the log does not state the identity stood" b2.log +grep -q "host-graph: build programs in" b2.log && fail "B: the build programs' graph was planned although nothing changed" b2.log + +# A +sleep 1 +printf '#define HL_VALUE 43\n' > "$TMP/hl/src/value.h" +"$MCPP" build --verbose > b3.log 2>&1 || fail "A: the build after the header edit failed" b3.log +grep -qE "build-program: app: identity [0-9a-f]+ -> [0-9a-f]+, compiled" b3.log || fail "C: the log does not state why the program was compiled" b3.log +[ "$("$MCPP" run 2>/dev/null | grep '^V=' | tail -1)" = "V=43" ] || fail "A: the program ran on the old header" b3.log +echo "PASS: 906_a_build_program_is_compiled_when_its_graph_changes" diff --git a/tests/e2e/907_the_library_interface_is_stated_once.sh b/tests/e2e/907_the_library_interface_is_stated_once.sh new file mode 100755 index 00000000..01d5e9b2 --- /dev/null +++ b/tests/e2e/907_the_library_interface_is_stated_once.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# requires: elf +# 907 -- a library's interface has one definition (SPEC-008 v0.2, mcpp#797). +# +# A `[lib] path` naming no file fails the package's own build; +# B where the package is a dependency, the same statement is a warning +# (its consumer cannot edit it), and the dependency's modules are still +# imported by the names they declare; +# C `[modules] exports` is an assertion about the public modules: a public +# module it does not list fails the package's own build, and a module +# that is exported but not public is not one of them; +# D a consumer importing an exported module that is not public is warned +# (W3), from build.mcpp as from the target. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +mkdir -p "$TMP/lib/src" "$TMP/app/src" +printf 'export module acme.lib;\nexport import acme.lib.part;\nexport int lib_value() { return 40; }\n' > "$TMP/lib/src/lib.cppm" +printf 'export module acme.lib.part;\nexport int part_value() { return 2; }\n' > "$TMP/lib/src/part.cppm" +printf 'export module acme.lib.detail;\nexport int detail_value() { return 0; }\n' > "$TMP/lib/src/detail.cppm" +cat > "$TMP/lib/mcpp.toml" <<'EOF' +[package] +namespace = "acme" +name = "lib" +version = "0.1.0" +[lib] +path = "src/missing.cppm" +[targets.lib] +kind = "lib" +EOF + +# A +cd "$TMP/lib" +"$MCPP" build > a.log 2>&1 && fail "A: a [lib] path naming no file was accepted in the package's own build" a.log +grep -q "missing.cppm" a.log || fail "A: the error does not name the path" a.log + +# B +printf '[package]\nname = "app"\nversion = "0.1.0"\n[dependencies.acme]\nlib = { path = "../lib" }\n[targets.app]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/app/mcpp.toml" +printf '#include \nimport acme.lib;\nint main() { std::printf("V=%%d\\n", lib_value() + part_value()); }\n' > "$TMP/app/src/main.cpp" +cd "$TMP/app" +"$MCPP" build > b.log 2>&1 || fail "B: a dependency's missing [lib] path failed its consumer" b.log +grep -q "names no file" b.log || fail "B: the consumer was not told" b.log +[ "$("$MCPP" run 2>/dev/null | grep '^V=' | tail -1)" = "V=42" ] || fail "B: value" b.log + +# C +cd "$TMP/lib" +sed -i 's|path = "src/missing.cppm"|path = "src/lib.cppm"|' mcpp.toml +printf '[modules]\nexports = ["acme.lib"]\n' >> mcpp.toml +"$MCPP" build > c1.log 2>&1 && fail "C: a public module missing from [modules] exports was accepted" c1.log +grep -q "acme.lib.part" c1.log || fail "C: the error does not name the public module" c1.log +sed -i 's|exports = \["acme.lib"\]|exports = ["acme.lib", "acme.lib.part"]|' mcpp.toml +"$MCPP" build > c2.log 2>&1 || fail "C: exports listing the public modules, and not the private one, was refused" c2.log + +# D +cd "$TMP/app" +printf '#include \nimport acme.lib;\nimport acme.lib.detail;\nint main() { std::printf("V=%%d\\n", lib_value() + part_value() + detail_value()); }\n' > src/main.cpp +"$MCPP" build > d.log 2>&1 || fail "D: the build failed" d.log +grep -q "acme.lib.detail" d.log && grep -qi "not one of that package's public modules" d.log \ + || fail "D: importing a module that is not public was not warned about" d.log +echo "PASS: 907_the_library_interface_is_stated_once" diff --git a/tests/e2e/908_what_a_build_dependency_asks_for.sh b/tests/e2e/908_what_a_build_dependency_asks_for.sh new file mode 100755 index 00000000..4dea3032 --- /dev/null +++ b/tests/e2e/908_what_a_build_dependency_asks_for.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# requires: elf +# 908 -- what a [build-dependencies] entry asks for (SPEC-004 §12 item 5). +# +# A with neither `tools` nor `host-module`, the library: build.mcpp imports it; +# B with only `tools`, the program and no library: an import of the library +# is refused by name, before the compiler, pointing at `host-module`; +# C `host-module = true` beside `tools` gives both; +# D `host-module = false` declines the library. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +mkdir -p "$TMP/gen/src" "$TMP/gen/tools" "$TMP/app/src" +printf 'export module acme.gen;\nexport int gen_value() { return 42; }\n' > "$TMP/gen/src/gen.cppm" +printf '#include \nint main() { std::puts("generated"); }\n' > "$TMP/gen/tools/gen_main.cpp" +cat > "$TMP/gen/mcpp.toml" <<'EOF' +[package] +namespace = "acme" +name = "gen" +version = "0.1.0" +[build] +sources = ["src/gen.cppm"] +[targets.gen] +kind = "lib" +[targets.gentool] +kind = "bin" +main = "tools/gen_main.cpp" +EOF +printf '#include \nint main() { std::printf("V=%%d\\n", V); }\n' > "$TMP/app/src/main.cpp" +printf 'import std;\nimport mcpp;\nimport acme.gen;\nint main() { mcpp::cxxflag(std::format("-DV={}", gen_value()).c_str()); }\n' > "$TMP/app/build.mcpp" +manifest() { printf '[package]\nname = "app"\nversion = "0.1.0"\n[build-dependencies.acme]\ngen = { path = "../gen"%s }\n[targets.app]\nkind = "bin"\nmain = "src/main.cpp"\n' "$1" > "$TMP/app/mcpp.toml"; } +cd "$TMP/app" + +manifest "" +"$MCPP" build > a.log 2>&1 || fail "A: the default entry did not give the library" a.log + +manifest ', tools = ["gentool"]' +"$MCPP" build > b.log 2>&1 && fail "B: a tools-only entry gave the library" b.log +grep -q "imports 'acme.gen'" b.log || fail "B: the refusal does not name the module" b.log +grep -q "host-module = true" b.log || fail "B: the refusal does not name the key" b.log +grep -qi "failed to read compiled module" b.log && fail "B: the compiler was reached" b.log + +manifest ', tools = ["gentool"], host-module = true' +"$MCPP" build > c.log 2>&1 || fail "C: tools and host-module = true did not give both" c.log + +manifest ', host-module = false' +"$MCPP" build > d.log 2>&1 && fail "D: host-module = false still gave the library" d.log +grep -q "imports 'acme.gen'" d.log || fail "D: the refusal does not name the module" d.log +echo "PASS: 908_what_a_build_dependency_asks_for" diff --git a/tests/e2e/909_deprecated_readings_are_registered_and_kept.sh b/tests/e2e/909_deprecated_readings_are_registered_and_kept.sh new file mode 100755 index 00000000..11c2e6e4 --- /dev/null +++ b/tests/e2e/909_deprecated_readings_are_registered_and_kept.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# requires: elf +# 909 -- the deprecation registry (docs/51 §Deprecations, mcpp#797). +# +# A `[dependencies] x = { host-module = true }` keeps its meaning -- the +# package serves the build program and not the target -- and is warned +# about with the release that deprecated it and the one that removes it; +# B `mcpp why compat` lists the use, and `--format json` states its row; +# C `--strict` refuses it; +# D a plugin whose library imports a package from its own +# [build-dependencies] (`host-module = true`) keeps building, and is told. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +mkdir -p "$TMP/inner/src" "$TMP/rule/src" "$TMP/app/src" +printf 'export module inner;\nexport int inner_value() { return 40; }\n' > "$TMP/inner/src/inner.cppm" +printf '[package]\nname = "inner"\nversion = "0.1.0"\n[targets.inner]\nkind = "lib"\n' > "$TMP/inner/mcpp.toml" +printf 'export module rule;\nimport std;\nimport mcpp;\nimport inner;\nexport namespace rule { inline void go() { mcpp::cxxflag(std::format("-DV={}", inner_value() + 2).c_str()); } }\n' > "$TMP/rule/src/rule.cppm" +printf '[package]\nname = "rule"\nversion = "0.1.0"\n[build-dependencies]\ninner = { path = "../inner", host-module = true }\n[targets.rule]\nkind = "lib"\n' > "$TMP/rule/mcpp.toml" +printf '[package]\nname = "app"\nversion = "0.1.0"\n[dependencies]\nrule = { path = "../rule", host-module = true }\n[targets.app]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/app/mcpp.toml" +printf '#include \nint main() { std::printf("V=%%d\\n", V); }\n' > "$TMP/app/src/main.cpp" +printf 'import mcpp;\nimport rule;\nint main() { rule::go(); }\n' > "$TMP/app/build.mcpp" + +cd "$TMP/app" +"$MCPP" build > a.log 2>&1 || fail "A/D: the deprecated spellings stopped building" a.log +[ "$("$MCPP" run 2>/dev/null | grep '^V=' | tail -1)" = "V=42" ] || fail "A/D: value" a.log +grep -q "deprecated since mcpp 2026.10.11.1 and removed from mcpp 2027.1" a.log || fail "A: the warning does not state both releases" a.log +grep -q "\[build-dependencies\]" a.log || fail "A: the warning does not name the canonical spelling" a.log +grep -q "plugin-imports-from-build-dependencies\|\[build-dependencies\] inner" a.log || fail "D: the plugin's reading was not reported" a.log +if find target -path '*/obj/*' -name 'rule.m.o' -not -path '*/.build-mcpp/*' | grep -q .; then + fail "A: the package reached the target" +fi + +"$MCPP" why compat > why.log 2>&1 || fail "B: mcpp why compat failed" why.log +grep -q "host-module-in-dependencies" why.log || fail "B: the use is not listed" why.log +"$MCPP" why compat --format json > why.json 2>/dev/null || fail "B: --format json failed" +grep -q '"floor": "2027.1"' why.json || fail "B: the JSON does not state the floor" why.json + +"$MCPP" build --strict > s.log 2>&1 && fail "C: --strict accepted a deprecated reading" s.log +grep -q "refuses the deprecated readings" s.log || fail "C: the refusal does not say why" s.log +echo "PASS: 909_deprecated_readings_are_registered_and_kept" diff --git a/tests/e2e/910_a_build_plugin_is_a_package_of_the_build_program.sh b/tests/e2e/910_a_build_plugin_is_a_package_of_the_build_program.sh new file mode 100755 index 00000000..452add2b --- /dev/null +++ b/tests/e2e/910_a_build_plugin_is_a_package_of_the_build_program.sh @@ -0,0 +1,98 @@ +#!/usr/bin/env bash +# requires: elf +# 910 -- build plugins (SPEC-007 §10, mcpp#797). A unit that imports +# `mcpp.core` (or `mcpp`) is a build-plugin unit; nothing else declares it. +# +# A a plugin built on another plugin and on an ordinary library, each named +# in its [dependencies], serves a consumer's build program (S1, S2); +# B P3: the plugin builds as the root of a build, in the build-program +# configuration, and its tests run; +# C P3: `--target` for another machine is refused, naming the plugin; +# D P1: a plugin reached through [dependencies] of a target is refused, +# naming the chain and [build-dependencies]; +# E P4: a program target importing mcpp is refused; +# F P5: `mcpp pack` refuses a plugin. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +lib() { # $1 = name, $2 = dependencies table body + mkdir -p "$TMP/$1/src" + printf '[package]\nnamespace = "acme"\nname = "%s"\nversion = "0.1.0"\n[dependencies.acme]\n%s\n[targets.%s]\nkind = "lib"\n' \ + "$1" "$2" "$1" > "$TMP/$1/mcpp.toml" +} +lib util "" +printf 'export module acme.util;\nexport int util_base() { return 40; }\n' > "$TMP/util/src/util.cppm" +lib core-rule "" +cat > "$TMP/core-rule/src/core-rule.cppm" <<'EOF' +export module acme.core_rule; +import std; +import mcpp; +export namespace acme::core_rule { +inline void define(std::string_view key, int value) { mcpp::define(std::format("{}={}", key, value).c_str()); } +} +EOF +lib rule 'core-rule = { path = "../core-rule" } +util = { path = "../util" }' +cat > "$TMP/rule/src/rule.cppm" <<'EOF' +export module acme.rule; +import mcpp; +import acme.core_rule; +import acme.util; +export namespace acme::rule { +inline int value() { return util_base() + 2; } +inline void go() { acme::core_rule::define("V", value()); } +} +EOF + +# A +mkdir -p "$TMP/app/src" +printf '[package]\nname = "app"\nversion = "0.1.0"\n[build-dependencies.acme]\nrule = { path = "../rule" }\n[targets.app]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/app/mcpp.toml" +printf '#include \nint main() { std::printf("V=%%d\\n", V); }\n' > "$TMP/app/src/main.cpp" +printf 'import acme.rule;\nint main() { acme::rule::go(); }\n' > "$TMP/app/build.mcpp" +cd "$TMP/app" +"$MCPP" build > a.log 2>&1 || fail "A: a plugin built on a plugin and a library did not serve the build program" a.log +[ "$("$MCPP" run 2>/dev/null | grep '^V=' | tail -1)" = "V=42" ] || fail "A: value" a.log +if find target -path '*/obj/*' -name '*.m.o' -not -path '*/.build-mcpp/*' | grep -E 'rule|util' | grep -q .; then + fail "A: a package of the build program reached the target" +fi + +# B +mkdir -p "$TMP/rule/tests" +printf 'import mcpp;\nimport acme.rule;\nint main() { return acme::rule::value() == 42 ? 0 : 1; }\n' > "$TMP/rule/tests/value.cpp" +cd "$TMP/rule" +"$MCPP" build > b1.log 2>&1 || fail "B: the plugin did not build as the root of a build" b1.log +grep -q "build-program" b1.log || fail "B: the build does not state its configuration" b1.log +"$MCPP" test > b2.log 2>&1 || fail "B: the plugin's tests did not run" b2.log +grep -q "1 passed" b2.log || fail "B: the test did not pass" b2.log + +# C +"$MCPP" build --target aarch64-linux-gnu > c.log 2>&1 && fail "C: --target for another machine was accepted" c.log +grep -q "acme.rule' is a build plugin" c.log && grep -q "cannot apply" c.log \ + || fail "C: the refusal does not name the plugin" c.log + +# D +mkdir -p "$TMP/app2/src" +printf '[package]\nname = "app2"\nversion = "0.1.0"\n[dependencies.acme]\nrule = { path = "../rule" }\n[targets.app2]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/app2/mcpp.toml" +printf 'int main() { return 0; }\n' > "$TMP/app2/src/main.cpp" +cd "$TMP/app2" +"$MCPP" build > d.log 2>&1 && fail "D: a plugin in [dependencies] of a target was accepted" d.log +grep -q "is a build plugin" d.log && grep -q "app2 → acme.rule" d.log && grep -q "\[build-dependencies\]" d.log \ + || fail "D: the refusal does not name the chain and the table" d.log + +# E +mkdir -p "$TMP/app3/src" +printf '[package]\nname = "app3"\nversion = "0.1.0"\n[targets.app3]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/app3/mcpp.toml" +printf 'import mcpp;\nint main() { return 0; }\n' > "$TMP/app3/src/main.cpp" +cd "$TMP/app3" +"$MCPP" build > e.log 2>&1 && fail "E: a program importing mcpp was accepted" e.log +grep -q "exists only in a build" e.log || fail "E: the refusal does not say why" e.log + +# F +cd "$TMP/rule" +"$MCPP" pack > f.log 2>&1 && fail "F: mcpp pack accepted a plugin" f.log +grep -q "is a build plugin" f.log || fail "F: the refusal does not say why" f.log +echo "PASS: 910_a_build_plugin_is_a_package_of_the_build_program" diff --git a/tests/e2e/911_a_workspace_states_each_table_once.sh b/tests/e2e/911_a_workspace_states_each_table_once.sh new file mode 100755 index 00000000..3e249e84 --- /dev/null +++ b/tests/e2e/911_a_workspace_states_each_table_once.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# requires: unix-shell +# 911 -- what a workspace root states, and to whom (docs/07, mcpp#797). +# +# R4 `[workspace.xlings.overrides]` on a root with [package] that depends on +# a member by path: the overrides belong to the root of the build, and +# the member reached as a dependency does not receive them; +# R5 the root's own `[xlings.overrides]` serves its own build without a +# position warning; a member built alone still receives it by position, +# and is told the reading is deprecated; +# V `[workspace.xlings.overrides]` on a root without [package] serves its +# members: the declared payload is not provisioned; +# W1 a build.mcpp beside a root without [package] is reported; +# W2 `features` of a `[workspace.dependencies]` entry reach the member that +# inherits it, and a request key there is reported. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export NO_COLOR=1 +MCPP="${MCPP:-mcpp}" + +mkdir -p "$TMP/opt/bin" +printf '#!/bin/sh\necho tool\n' > "$TMP/opt/bin/tool"; chmod +x "$TMP/opt/bin/tool" + +rooted() { # $1 = dir, $2 = tables of the root + mkdir -p "$1/src" "$1/sub/src" + printf '[package]\nname = "root"\nversion = "0.1.0"\n[dependencies]\nsub = { path = "sub" }\n[targets.root]\nkind = "bin"\nmain = "src/main.cpp"\n[workspace]\nmembers = ["sub"]\n%s\n' "$2" > "$1/mcpp.toml" + printf 'import sub;\nint main() { return f() - 1; }\n' > "$1/src/main.cpp" + printf '[package]\nname = "sub"\nversion = "0.1.0"\n[targets.sub]\nkind = "lib"\n' > "$1/sub/mcpp.toml" + printf 'export module sub;\nexport int f() { return 1; }\n' > "$1/sub/src/sub.cppm" +} + +# R4 +rooted "$TMP/r4" "$(printf '[workspace.xlings.overrides]\n"xim:mcpp-e2e-absent-tool" = "%s"' "$TMP/opt/bin/tool")" +( cd "$TMP/r4" && "$MCPP" build > r4.log 2>&1 ) || fail "R4: the workspace's overrides failed its own build" "$TMP/r4/r4.log" +grep -q "is a dependency of this build" "$TMP/r4/r4.log" && fail "R4: the member received the overrides" "$TMP/r4/r4.log" + +# R5 +rooted "$TMP/r5" "$(printf '[xlings.overrides]\n"xim:mcpp-e2e-absent-tool" = "%s"' "$TMP/opt/bin/tool")" +( cd "$TMP/r5" && "$MCPP" build > r5.log 2>&1 ) || fail "R5: the root's own overrides failed its own build" "$TMP/r5/r5.log" +grep -q "workspace-position\|deprecated since" "$TMP/r5/r5.log" \ + && fail "R5: the root's own build was told about a position reading" "$TMP/r5/r5.log" +( cd "$TMP/r5/sub" && "$MCPP" build > r5b.log 2>&1 ) || fail "R5: the member did not build alone" "$TMP/r5/sub/r5b.log" +grep -q "deprecated since mcpp 2026.10.10.1 and removed from mcpp 2027.1" "$TMP/r5/sub/r5b.log" \ + || fail "R5: the member built alone was not told the position reading is deprecated" "$TMP/r5/sub/r5b.log" + +# V +mkdir -p "$TMP/v/m/src" +printf '[workspace]\nmembers = ["m"]\n[workspace.xlings.overrides]\n"xim:mcpp-e2e-absent-tool" = { program = "%s", version = "2.1" }\n' "$TMP/opt/bin/tool" > "$TMP/v/mcpp.toml" +printf '[package]\nname = "m"\nversion = "0.1.0"\n[xlings.workspace]\n"xim:mcpp-e2e-absent-tool" = ">=2.0"\n[targets.m]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/v/m/mcpp.toml" +printf 'int main() { return 0; }\n' > "$TMP/v/m/src/main.cpp" +( cd "$TMP/v" && MCPP_NO_AUTO_INSTALL=1 "$MCPP" build > v.log 2>&1 ) \ + || fail "V: [workspace.xlings.overrides] did not serve a member of a root without [package]" "$TMP/v/v.log" +grep -q "custom: xim:mcpp-e2e-absent-tool" "$TMP/v/v.log" || fail "V: the build does not state the source" "$TMP/v/v.log" +( cd "$TMP/v/m" && MCPP_NO_AUTO_INSTALL=1 "$MCPP" build > v2.log 2>&1 ) \ + || fail "V: the member built from its own directory was not served" "$TMP/v/m/v2.log" + +# W1 +mkdir -p "$TMP/w1/m/src" +printf '[workspace]\nmembers = ["m"]\n' > "$TMP/w1/mcpp.toml" +printf 'int main() { return 1; }\n' > "$TMP/w1/build.mcpp" +printf '[package]\nname = "m"\nversion = "0.1.0"\n[targets.m]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/w1/m/mcpp.toml" +printf 'int main() { return 0; }\n' > "$TMP/w1/m/src/main.cpp" +( cd "$TMP/w1" && "$MCPP" build > w1.log 2>&1 ) || fail "W1: the workspace did not build" "$TMP/w1/w1.log" +grep -q "build.mcpp is written beside a workspace root without \[package\]" "$TMP/w1/w1.log" \ + || fail "W1: the build.mcpp beside the root was not reported" "$TMP/w1/w1.log" + +# W2 +mkdir -p "$TMP/w2/hl/src" "$TMP/w2/m/src" +printf '[package]\nname = "hl"\nversion = "0.1.0"\n[features]\nx = []\n[targets.hl]\nkind = "lib"\n' > "$TMP/w2/hl/mcpp.toml" +cat > "$TMP/w2/hl/src/hl.cppm" <<'EOF' +export module hl; +export int feature_x() { +#ifdef MCPP_FEATURE_X + return 1; +#else + return 0; +#endif +} +EOF +printf '[workspace]\nmembers = ["m"]\n[workspace.dependencies]\nhl = { path = "hl", features = ["x"], linkage = "static" }\n' > "$TMP/w2/mcpp.toml" +printf '[package]\nname = "m"\nversion = "0.1.0"\n[dependencies]\nhl.workspace = true\n[targets.m]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/w2/m/mcpp.toml" +printf '#include \nimport hl;\nint main() { std::printf("X=%%d\\n", feature_x()); }\n' > "$TMP/w2/m/src/main.cpp" +( cd "$TMP/w2" && "$MCPP" build > w2.log 2>&1 ) || fail "W2: the workspace did not build" "$TMP/w2/w2.log" +[ "$(cd "$TMP/w2" && "$MCPP" run -p m 2>/dev/null | grep '^X=' | tail -1)" = "X=1" ] \ + || fail "W2: the feature stated in [workspace.dependencies] did not reach the member" "$TMP/w2/w2.log" +grep -q "'linkage' is a request of a member's own edge" "$TMP/w2/w2.log" \ + || fail "W2: the request key in [workspace.dependencies] was not reported" "$TMP/w2/w2.log" +echo "PASS: 911_a_workspace_states_each_table_once" diff --git a/tests/e2e/912_two_configuration_groups_share_the_build_programs_graph.sh b/tests/e2e/912_two_configuration_groups_share_the_build_programs_graph.sh new file mode 100755 index 00000000..a0bf51e3 --- /dev/null +++ b/tests/e2e/912_two_configuration_groups_share_the_build_programs_graph.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# requires: unix-shell +# 912 -- two configuration groups of one command whose build programs have one +# host configuration share the build programs' graph (mcpp#797). +# +# The groups of a workspace command build at the same time. Their build +# programs are compiled for the machine doing the build, with one toolchain and +# one standard, so they are planned in one directory: one group compiles, the +# other waits for its lock and finds the libraries compiled. +# +# A members `a` and `b` differ in [profile.dev], so they are two groups; both +# build, and each program sees the library both import; +# B the library is compiled once, in one build programs' graph. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" + +mkdir -p "$TMP/shared/src" "$TMP/ws" +printf '[package]\nname = "shared"\nversion = "0.1.0"\n[targets.shared]\nkind = "lib"\n' > "$TMP/shared/mcpp.toml" +printf 'export module shared;\nexport int shared_value() { return 42; }\n' > "$TMP/shared/src/shared.cppm" +cd "$TMP/ws" +printf '[workspace]\nmembers = ["a", "b"]\n' > mcpp.toml +for m in a b; do + mkdir -p $m/src + printf '[package]\nname = "%s"\nversion = "0.1.0"\n[build-dependencies]\nshared = { path = "../../shared" }\n[targets.%s]\nkind = "bin"\nmain = "src/main.cpp"\n' $m $m > $m/mcpp.toml + printf '#include \nint main() { std::printf("V=%%d\\n", V); }\n' > $m/src/main.cpp + printf 'import std;\nimport mcpp;\nimport shared;\nint main() { mcpp::cxxflag(std::format("-DV={}", shared_value()).c_str()); }\n' > $m/build.mcpp +done +printf '[profile.dev]\nopt = 1\n' >> a/mcpp.toml + +# A +"$MCPP" build --workspace > a.log 2>&1 || fail "A: the workspace did not build" a.log +[ "$(find target -name build.ninja -not -path '*/.build-mcpp/*' | wc -l | tr -d ' ')" = 2 ] \ + || fail "A: the members were not two configuration groups" a.log +for m in a b; do + [ "$("$MCPP" run -p $m 2>/dev/null | grep '^V=' | tail -1)" = "V=42" ] || fail "A: $m's value" a.log +done + +# B +[ "$(find target/.build-mcpp/host -mindepth 1 -maxdepth 1 -type d -not -name digests | wc -l | tr -d ' ')" = 1 ] \ + || fail "B: the groups' build programs were planned in more than one directory" a.log +compiles=$(find target/.build-mcpp/host -name .ninja_log -exec awk -F'\t' '$4 ~ /shared\.m\.o$/' {} + | wc -l | tr -d ' ') +[ "$compiles" = 1 ] || fail "B: the library was compiled $compiles times" a.log +echo "PASS: 912_two_configuration_groups_share_the_build_programs_graph" diff --git a/tests/e2e/913_the_build_database_describes_the_build_programs.sh b/tests/e2e/913_the_build_database_describes_the_build_programs.sh new file mode 100755 index 00000000..4a2edb5c --- /dev/null +++ b/tests/e2e/913_the_build_database_describes_the_build_programs.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# requires: python3 +# 913 -- `mcpp emit build-database` describes the build programs' graph +# (SPEC-005 R3.13, mcpp#797), and compile_commands.json does not. +# +# A the sets `app:build` (the program, family `app`), `:host` (a library +# of the program), `mcpp:core` (the engine interface) and `mcpp:std:host` +# are present, in the configuration `build-program`; the target's set +# keeps its name and its configuration; +# B the arguments of build.mcpp are those of the graph's compile, up to the +# directory a planning pass writes in; +# C compile_commands.json, written by the build and by +# `--spec compile-commands`, names no unit of the build programs' graph; +# D `mcpp self env` states `engineModules`, the directory of the units of +# `mcpp:core`, and it holds the interface. +set -e +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +export MCPP_HOME="$TMP/mcpp-home" +source "$(dirname "$0")/_inherit_toolchain.sh" +PY=python3 + +mkdir -p "$TMP/lib/src" "$TMP/app/src" +printf '[package]\nname = "lib"\nversion = "0.1.0"\n[targets.lib]\nkind = "lib"\n' > "$TMP/lib/mcpp.toml" +printf 'export module lib;\nexport int lib_value() { return 42; }\n' > "$TMP/lib/src/lib.cppm" +printf '[package]\nname = "app"\nversion = "0.1.0"\n[build-dependencies]\nlib = { path = "../lib" }\n[targets.app]\nkind = "bin"\nmain = "src/main.cpp"\n' > "$TMP/app/mcpp.toml" +printf '#include \nint main() { std::printf("V=%%d\\n", V); }\n' > "$TMP/app/src/main.cpp" +printf 'import std;\nimport mcpp;\nimport lib;\nint main() { mcpp::cxxflag(std::format("-DV={}", lib_value()).c_str()); }\n' > "$TMP/app/build.mcpp" +cd "$TMP/app" + +"$MCPP" build > build.log 2>&1 || fail "the project did not build" build.log +"$MCPP" emit build-database --format json > db.json 2> db.err || fail "emit build-database failed" db.err +"$MCPP" emit build-database --spec compile-commands > cc.json 2> cc.err || fail "--spec compile-commands failed" cc.err +"$MCPP" self env --format json > env.json 2> env.err || fail "mcpp self env failed" env.err +host_cc=$(find target/.build-mcpp/host -maxdepth 2 -name compile_commands.json | head -1) +[ -n "$host_cc" ] || fail "the build programs' graph wrote no compile_commands.json" build.log + +"$PY" - db.json cc.json compile_commands.json env.json "$host_cc" <<'EOF' || fail "A-D" db.json +import json, os, sys +db, cc, root_cc, env, host_cc = (json.load(open(p)) for p in sys.argv[1:6]) +sets = {s["name"]: s for s in db["data"]["database"]["sets"]} + +# A +for name, kind in (("app:build", "executable"), ("lib:host", "library"), + ("mcpp:core", "library"), ("mcpp:std:host", None)): + candidates = [n for n in sets if n == name or n.endswith("." + name)] + assert candidates, f"A: no set {name}: {sorted(sets)}" + s = sets[candidates[0]] + assert s["ide"]["configuration"] == "build-program", (name, s["ide"]) + if kind: assert s["ide"]["kind"] == kind, (name, s["ide"]) +assert sets["app:build"]["family-name"] == "app", sets["app:build"]["family-name"] +assert "app" in sets and sets["app"]["ide"]["configuration"] == "dev", sets.get("app", {}).get("ide") + +# B +unit = next(u for u in sets["app:build"]["translation-units"] if u["source"].endswith("build.mcpp")) +built = next(e for e in host_cc if e["file"].endswith("build.mcpp")) +norm = lambda args, wd: [a.replace(wd, "") for a in args] +assert norm(unit["arguments"], unit["work-directory"]) == norm(built["arguments"], built["directory"]), \ + ("B", unit["arguments"], built["arguments"]) + +# C +for entries in (cc, root_cc): + names = [os.path.basename(e["file"]) for e in entries] + assert "main.cpp" in names, ("C", names) + for unwanted in ("build.mcpp", "lib.cppm", "mcpp.cppm", "mcpp_core.cppm"): + assert unwanted not in names, ("C", unwanted, names) + +# D +engine = env["data"]["engineModules"] +assert os.path.isfile(os.path.join(engine, "mcpp.cppm")), ("D", engine) +core = sets["mcpp:core"]["translation-units"] +assert all(os.path.dirname(u["source"]) == os.path.normpath(engine) for u in core), \ + ("D", engine, [u["source"] for u in core]) +EOF +echo "PASS: 913_the_build_database_describes_the_build_programs" diff --git a/tests/unit/test_library_interface.cpp b/tests/unit/test_library_interface.cpp new file mode 100644 index 00000000..93b6ae56 --- /dev/null +++ b/tests/unit/test_library_interface.cpp @@ -0,0 +1,112 @@ +#include + +import std; +import mcpp.manifest; +import mcpp.modgraph.graph; +import mcpp.modgraph.library_interface; + +// ONE DEFINITION OF WHAT A LIBRARY OFFERS (SPEC-008 v0.2, mcpp#797): the +// interface root and what it re-exports, the interface units of the active +// features, and -- with no root -- every exported module. The target build, +// the build programs' graph, `mcpp pack` and the `exports` assertion all read +// this function. + +using namespace mcpp::modgraph; + +namespace { + +struct Tree { + std::filesystem::path root; + explicit Tree(std::string_view name) + : root(std::filesystem::temp_directory_path() + / std::format("mcpp-libiface-{}-{}", name, std::random_device{}())) { + std::filesystem::create_directories(root / "src"); + std::filesystem::create_directories(root / "rules"); + } + ~Tree() { std::error_code ec; std::filesystem::remove_all(root, ec); } + std::filesystem::path touch(std::string_view rel) { + auto p = root / rel; + std::ofstream(p) << "// unit\n"; + return p.lexically_normal(); + } +}; + +SourceUnit unit(const std::filesystem::path& p, std::string module, + std::vector reexports = {}) { + SourceUnit u; + u.path = p; + u.packageName = "acme.lib"; + u.provides = ModuleId{std::move(module)}; + for (auto& r : reexports) u.reexports.push_back(ModuleId{std::move(r)}); + return u; +} + +} // namespace + +TEST(LibraryInterface, TheRootAndWhatItReExports) { + Tree t("root"); + Graph g; + g.units.push_back(unit(t.touch("src/lib.cppm"), "acme.lib", {"acme.lib.part"})); + g.units.push_back(unit(t.touch("src/part.cppm"), "acme.lib.part")); + g.units.push_back(unit(t.touch("src/detail.cppm"), "acme.lib.detail")); + mcpp::manifest::Manifest m; + m.package.name = "lib"; + auto iface = library_interface(g, m, t.root, "acme.lib", {}); + EXPECT_EQ(iface.rootModule, "acme.lib"); + EXPECT_EQ(iface.publicModules, (std::set{"acme.lib", "acme.lib.part"})); + EXPECT_EQ(iface.exportedModules, + (std::set{"acme.lib", "acme.lib.part", "acme.lib.detail"})); +} + +TEST(LibraryInterface, NoRootStatesNoInterface) { + Tree t("noroot"); + Graph g; + g.units.push_back(unit(t.touch("src/a.cppm"), "acme.a")); + g.units.push_back(unit(t.touch("src/b.cppm"), "acme.b")); + mcpp::manifest::Manifest m; + m.package.name = "lib"; + auto iface = library_interface(g, m, t.root, "acme.lib", {}); + EXPECT_TRUE(iface.rootModule.empty()); + EXPECT_EQ(iface.publicModules, iface.exportedModules); +} + +TEST(LibraryInterface, ADeclaredRootIsTheOneUsed) { + Tree t("declared"); + Graph g; + g.units.push_back(unit(t.touch("src/lib.cppm"), "acme.conventional")); + g.units.push_back(unit(t.touch("rules/facade.cppm"), "acme.facade")); + mcpp::manifest::Manifest m; + m.package.name = "lib"; + m.lib.path = "rules/facade.cppm"; + auto iface = library_interface(g, m, t.root, "acme.lib", {}); + EXPECT_TRUE(iface.rootDeclared); + EXPECT_EQ(iface.rootModule, "acme.facade"); + EXPECT_EQ(iface.publicModules, (std::set{"acme.facade"})); +} + +TEST(LibraryInterface, AnActiveFeatureAddsItsInterfaceUnits) { + Tree t("features"); + Graph g; + g.units.push_back(unit(t.touch("src/lib.cppm"), "acme.lib")); + g.units.push_back(unit(t.touch("rules/cuda.cppm"), "acme.rules.cuda")); + g.units.push_back(unit(t.touch("rules/spirv.cppm"), "acme.rules.spirv")); + mcpp::manifest::Manifest m; + m.package.name = "lib"; + m.buildConfig.featureSources["rules-cuda"] = {"rules/cuda.cppm"}; + m.buildConfig.featureSources["rules-spirv"] = {"rules/spirv.cppm"}; + auto iface = library_interface(g, m, t.root, "acme.lib", {"rules-cuda"}); + EXPECT_EQ(iface.publicModules, (std::set{"acme.lib", "acme.rules.cuda"})); +} + +TEST(LibraryInterface, OnlyThePackagesOwnUnitsCount) { + Tree t("own"); + Graph g; + g.units.push_back(unit(t.touch("src/lib.cppm"), "acme.lib")); + auto other = unit(t.touch("src/other.cppm"), "other.mod"); + other.packageName = "other.pkg"; + g.units.push_back(other); + mcpp::manifest::Manifest m; + m.package.name = "lib"; + auto iface = library_interface(g, m, t.root, "acme.lib", {}); + EXPECT_FALSE(iface.exportedModules.contains("other.mod")); +} diff --git a/tests/unit/test_manifest.cpp b/tests/unit/test_manifest.cpp index 88ad09e6..3a6328f9 100644 --- a/tests/unit/test_manifest.cpp +++ b/tests/unit/test_manifest.cpp @@ -6803,3 +6803,35 @@ TEST(Manifest, TestWindowsCodePageIsAClosedSetOfTwo) { ASSERT_FALSE(bad.has_value()); EXPECT_NE(bad.error().format().find("windows_code_page"), std::string::npos); } + +// A Form B descriptor states its interface root with `lib`, below the +// archive's wrapping directory as its `sources` are (SPEC-008 v0.2, mcpp#797). +TEST(SynthesizeFromXpkgLua, LibStatesTheInterfaceRootBelowTheWrappingDirectory) { + constexpr auto src = R"( +package = { + spec = "1", + name = "tomlpp", + xpm = { linux = { ["3.4.0"] = { url = "u", sha256 = "h" } } }, + mcpp = { + lib = "*/modules/tomlplusplus.cppm", + sources = { "*/modules/tomlplusplus.cppm" }, + include_dirs = { "*/include" }, + targets = { ["tomlpp"] = { kind = "lib" } }, + }, +} +)"; + auto m = mcpp::manifest::synthesize_from_xpkg_lua(src, "tomlpp", "3.4.0", + mcpp::platform::HostPlatform::current()); + ASSERT_TRUE(m.has_value()) << m.error().format(); + EXPECT_EQ(m->lib.path.generic_string(), "*/modules/tomlplusplus.cppm"); + EXPECT_TRUE(m->xpkgUnknownKeys.empty()); + + const auto root = std::filesystem::temp_directory_path() + / std::format("mcpp-lib-root-{}", std::random_device{}()); + std::filesystem::create_directories(root / "tomlplusplus-3.4.0" / "modules"); + std::ofstream(root / "tomlplusplus-3.4.0" / "modules" / "tomlplusplus.cppm") << "export module tomlpp;\n"; + EXPECT_EQ(mcpp::manifest::resolve_lib_root_path(*m, root).generic_string(), + "tomlplusplus-3.4.0/modules/tomlplusplus.cppm"); + std::error_code ec; + std::filesystem::remove_all(root, ec); +} diff --git a/tests/unit/test_provisions.cpp b/tests/unit/test_provisions.cpp index 907bf503..9955f4a3 100644 --- a/tests/unit/test_provisions.cpp +++ b/tests/unit/test_provisions.cpp @@ -23,7 +23,8 @@ struct Edge { std::size_t consumerPackageIndex = 0; std::size_t dependencyPackageIndex = 0; std::vector requestedTools; - bool hostModule = false; + bool buildTable = false; // the edge is a [build-dependencies] entry + bool wantsLibrary = false; // and it asks for the package's library bool reexport = false; }; @@ -67,7 +68,7 @@ TEST(Provisions, EveryKindAnswersBothQuestions) { TEST(Provisions, ToolIsVisibleToTheRequesterWithoutAnyReexport) { // 0=root, 1=protobuf. root --tools=[protoc]--> protobuf - std::vector edges{ { 0, 1, { "protoc" }, false, false } }; + std::vector edges{ { 0, 1, { "protoc" }, false, false, false } }; auto p = prov::propagate(edges, 2); EXPECT_TRUE(sees_tool(p, 0, 1, "protoc")); // Nothing to export: the root has no consumers, and it never said it @@ -78,8 +79,8 @@ TEST(Provisions, ToolIsVisibleToTheRequesterWithoutAnyReexport) { TEST(Provisions, WithoutReexportALibrarysToolStaysWithTheLibrary) { // 0=app, 1=lib, 2=protobuf. app -> lib --tools=[protoc]--> protobuf std::vector edges{ - { 0, 1, {}, false, false }, - { 1, 2, { "protoc" }, false, /*reexport=*/false }, + { 0, 1, {}, false, false, false }, + { 1, 2, { "protoc" }, false, false, /*reexport=*/false }, }; auto p = prov::propagate(edges, 3); EXPECT_TRUE(sees_tool(p, 1, 2, "protoc")); @@ -91,8 +92,8 @@ TEST(Provisions, WithoutReexportALibrarysToolStaysWithTheLibrary) { TEST(Provisions, ReexportHandsTheToolToTheLibrarysConsumer) { std::vector edges{ - { 0, 1, {}, false, false }, - { 1, 2, { "protoc" }, false, /*reexport=*/true }, + { 0, 1, {}, false, false, false }, + { 1, 2, { "protoc" }, false, false, /*reexport=*/true }, }; auto p = prov::propagate(edges, 3); EXPECT_TRUE(sees_tool(p, 0, 2, "protoc")); @@ -103,9 +104,9 @@ TEST(Provisions, ReexportTravelsFurtherOnlyWhenEachHopSaysSo) { // 0=app, 1=mid, 2=lib, 3=protobuf. // lib re-exports protoc to mid; mid does NOT re-export to app. std::vector edges{ - { 0, 1, {}, false, /*reexport=*/false }, - { 1, 2, {}, false, /*reexport=*/false }, - { 2, 3, { "protoc" }, false, /*reexport=*/true }, + { 0, 1, {}, false, false, /*reexport=*/false }, + { 1, 2, {}, false, false, /*reexport=*/false }, + { 2, 3, { "protoc" }, false, false, /*reexport=*/true }, }; auto p = prov::propagate(edges, 4); EXPECT_TRUE(sees_tool(p, 2, 3, "protoc")); @@ -122,8 +123,8 @@ TEST(Provisions, ReexportTravelsFurtherOnlyWhenEachHopSaysSo) { TEST(Provisions, HostModuleFollowsTheSameRule) { std::vector edges{ - { 0, 1, {}, false, false }, - { 1, 2, {}, /*hostModule=*/true, /*reexport=*/true }, + { 0, 1, {}, false, false, false }, + { 1, 2, {}, /*buildTable=*/true, /*wantsLibrary=*/true, /*reexport=*/true }, }; auto p = prov::propagate(edges, 3); const prov::Provision rule{ prov::Kind::HostModule, 2, {} }; @@ -135,9 +136,9 @@ TEST(Provisions, EveryEdgeExposesTheDependencyDirectory) { // dep_dir() has always worked for a DIRECT dependency without anyone // declaring anything; that stays true. Re-export is what extends it. std::vector edges{ - { 0, 1, {}, false, false }, - { 1, 2, {}, false, /*reexport=*/true }, - { 1, 3, {}, false, /*reexport=*/false }, + { 0, 1, {}, false, false, false }, + { 1, 2, {}, false, false, /*reexport=*/true }, + { 1, 3, {}, false, false, /*reexport=*/false }, }; auto p = prov::propagate(edges, 4); EXPECT_TRUE(sees_dir(p, 0, 1)); @@ -150,8 +151,8 @@ TEST(Provisions, ACycleTerminates) { // Resolution should not produce one, but the fixpoint must not depend on // that: sets only grow and are bounded, so a cycle simply stops changing. std::vector edges{ - { 0, 1, { "a" }, false, true }, - { 1, 0, { "b" }, false, true }, + { 0, 1, { "a" }, false, false, true }, + { 1, 0, { "b" }, false, false, true }, }; auto p = prov::propagate(edges, 2); EXPECT_TRUE(sees_tool(p, 0, 1, "a")); diff --git a/tests/unit/test_source_digest.cpp b/tests/unit/test_source_digest.cpp new file mode 100644 index 00000000..fdc3c050 --- /dev/null +++ b/tests/unit/test_source_digest.cpp @@ -0,0 +1,70 @@ +#include + +import std; +import mcpp.build.source_digest; + +// THE CONTENT DIGEST OF A SOURCE TREE, RE-READ ONLY WHERE A STAT MOVED +// (mcpp#797, R6). A build program depends on every source of the packages in +// its graph, an included header among them. + +namespace { +struct Tree { + std::filesystem::path root, cache; + Tree() : root(std::filesystem::temp_directory_path() + / std::format("mcpp-digest-{}", std::random_device{}())) { + std::filesystem::create_directories(root / "src"); + cache = root.string() + ".tsv"; + } + ~Tree() { + std::error_code ec; + std::filesystem::remove_all(root, ec); + std::filesystem::remove(cache, ec); + } + void write(std::string_view rel, std::string_view text) { + std::filesystem::create_directories((root / rel).parent_path()); + std::ofstream(root / rel, std::ios::binary | std::ios::trunc) << text; + } + std::string digest() { return mcpp::build::source_tree_digest(root, cache); } +}; +} // namespace + +TEST(SourceDigest, AnEditedHeaderChangesIt) { + Tree t; + t.write("src/lib.cppm", "module;\n#include \"v.h\"\nexport module lib;\n"); + t.write("src/v.h", "#define V 1\n"); + const auto a = t.digest(); + EXPECT_EQ(t.digest(), a) << "a tree that did not change keeps its digest"; + t.write("src/v.h", "#define V 2\n"); + EXPECT_NE(t.digest(), a); +} + +TEST(SourceDigest, ATouchWithoutAnEditKeepsIt) { + Tree t; + t.write("src/v.h", "#define V 1\n"); + const auto a = t.digest(); + std::filesystem::last_write_time(t.root / "src/v.h", + std::filesystem::last_write_time(t.root / "src/v.h") + std::chrono::seconds(5)); + EXPECT_EQ(t.digest(), a); +} + +TEST(SourceDigest, BuildOutputAndNestedPackagesAreNotPartOfIt) { + Tree t; + t.write("src/v.h", "#define V 1\n"); + const auto a = t.digest(); + t.write("target/obj/x.o", "binary"); + t.write(".git/HEAD", "ref"); + t.write("member/mcpp.toml", "[package]\n"); + t.write("member/src/m.cppm", "export module m;\n"); + EXPECT_EQ(t.digest(), a); +} + +TEST(SourceDigest, AnAddedOrRemovedFileChangesIt) { + Tree t; + t.write("src/a.cppm", "export module a;\n"); + const auto a = t.digest(); + t.write("src/b.cppm", "export module b;\n"); + const auto b = t.digest(); + EXPECT_NE(b, a); + std::filesystem::remove(t.root / "src/b.cppm"); + EXPECT_EQ(t.digest(), a); +} From ea05a875121c0dd2d6c252dea468608ef460f7dc Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sun, 11 Oct 2026 09:34:28 +0900 Subject: [PATCH 08/12] A host sub-build targets the host; one directory, every spelling; tests find the target's build directory (#797) - `BuildOverrides::host_target`: the build programs' graph and a host tool's sub-build target the machine doing the build; `[build] target` and the configured default target name the target of the build that asked (e2e 97 on a Windows host whose default target is x86_64-windows-gnu). - The engine package and a workspace member are recognised through any spelling of their directory (macOS /var and /private/var, Windows 8.3 and long names): a member reached through another spelling was named with the default namespace and its program was not found (e2e 857, 858, 913). - The verbose log states why a build program's cached run is not used. - The error for a missing program names the programs the plan has. - Tests that look up a build directory under target/ exclude the build programs' graph under target/.build-mcpp/host; 92 and 857 read the new object names and either path separator. --- src/build/build_program.cppm | 42 ++++++++++++------- src/build/engine_package.cppm | 13 +++++- src/build/prepare.cppm | 4 ++ src/build/prepare/driver.cpp | 1 + src/build/prepare/features.cpp | 1 + src/build/prepare/host_graph.cpp | 14 +++++-- src/build/prepare/toolchain.cpp | 5 ++- src/build/prepare/windows_resources.cpp | 16 +++++-- tests/e2e/03_multi_module.sh | 2 +- tests/e2e/07_static_library.sh | 2 +- tests/e2e/08_shared_library.sh | 2 +- tests/e2e/09_path_dependency.sh | 2 +- tests/e2e/100_cppfly_reflection.sh | 4 +- tests/e2e/100_feature_sources_test_mode.sh | 6 +-- tests/e2e/101_cppfly_llvm_soft.sh | 2 +- tests/e2e/102_mingw_cross_wine.sh | 2 +- tests/e2e/104_asm_sources_gas.sh | 2 +- tests/e2e/105_asm_sources_nasm.sh | 2 +- tests/e2e/117_object_path_collision.sh | 2 +- tests/e2e/120_ws_root_indices.sh | 4 +- tests/e2e/123_same_named_main_across_dep.sh | 2 +- .../124_gen_source_out_of_root_object_path.sh | 2 +- tests/e2e/141_include_dirs_after.sh | 2 +- tests/e2e/144_build_mcpp_include_dir.sh | 2 +- tests/e2e/164_ninja_goal_command_length.sh | 2 +- tests/e2e/170_bmi_staging_no_cascade.sh | 2 +- tests/e2e/171_bmi_staging_locked_dest.sh | 2 +- tests/e2e/173_build_profile_isolation.sh | 2 +- tests/e2e/174_cache_modes_and_commands.sh | 16 +++---- tests/e2e/175_cpp20_import_std.sh | 4 +- tests/e2e/179_spaced_paths.sh | 2 +- tests/e2e/183_cxx_runtime_contract.sh | 8 ++-- tests/e2e/188_build_actions.sh | 2 +- tests/e2e/190_link_rspfile_newlines.sh | 2 +- tests/e2e/191_link_scale.sh | 2 +- tests/e2e/198_windows_resources_cross.sh | 2 +- tests/e2e/207_runtime_contract_provenance.sh | 2 +- .../208_private_libc_stays_in_the_binary.sh | 2 +- tests/e2e/217_module_extensions.sh | 4 +- .../e2e/218_module_extensions_graph_shape.sh | 4 +- tests/e2e/231_jobs_option.sh | 6 +-- tests/e2e/239_msvc_managed_toolset.sh | 2 +- .../e2e/241_windows_ucrt_runtime_identity.sh | 2 +- .../e2e/247_bare_triple_conditional_native.sh | 2 +- tests/e2e/257_shared_library_pe.sh | 4 +- tests/e2e/258_shared_library_msvc_auto_def.sh | 2 +- .../260_module_extension_is_configuration.sh | 2 +- tests/e2e/26_c_language_support.sh | 2 +- ..._naming_the_host_target_changes_nothing.sh | 2 +- ...report_names_is_what_the_link_line_uses.sh | 2 +- tests/e2e/307_symbol_provision.sh | 4 +- .../315_blocking_check_gates_compilation.sh | 4 +- tests/e2e/323_post_link_record_durability.sh | 2 +- tests/e2e/324_fast_path_sees_path_deps.sh | 2 +- tests/e2e/40_llvm_bmi_cache.sh | 2 +- tests/e2e/48_build_error_output.sh | 2 +- tests/e2e/49_bmi_cache_nested_custom_index.sh | 2 +- tests/e2e/50_package_owned_build_flags.sh | 2 +- tests/e2e/54_package_owned_ldflags.sh | 2 +- tests/e2e/59_cpp_standard_config.sh | 4 +- .../617_build_program_knows_the_cxx_stdlib.sh | 2 +- tests/e2e/62_runtime_library_dirs.sh | 2 +- ...vate_module_fragment_is_not_a_partition.sh | 2 +- ...xtension_file_need_not_provide_a_module.sh | 2 +- tests/e2e/638_pack_format_dispatch.sh | 2 +- ...time_deploy_places_files_in_a_directory.sh | 2 +- ...abi_threads_is_one_switch_for_the_graph.sh | 2 +- tests/e2e/650_wasm_row_names_its_launcher.sh | 2 +- ...build_program_deploys_what_it_generated.sh | 2 +- ...653_exceptions_is_the_second_abi_member.sh | 8 ++-- .../654_a_requirement_on_the_target_axis.sh | 2 +- ...an_action_declaration_has_no_size_limit.sh | 4 +- ...tion_names_the_engine_through_mcpp_self.sh | 2 +- tests/e2e/66_runtime_provides.sh | 2 +- .../682_why_deps_prints_the_resolved_graph.sh | 2 +- ..._on_the_msvc_abi_records_its_static_crt.sh | 2 +- tests/e2e/70_per_target_flags.sh | 2 +- ...age_contributes_nothing_to_its_consumer.sh | 2 +- ...714_features_names_a_dependency_feature.sh | 2 +- ...ds_reach_the_compiler_and_the_databases.sh | 2 +- ...lied_target_closes_the_hosts_own_search.sh | 2 +- tests/e2e/73_issue131_per_target_cxxflag.sh | 2 +- tests/e2e/760_msvc_toolset_is_chosen_once.sh | 6 +-- ...d_program_declares_a_runtime_search_dir.sh | 2 +- tests/e2e/781_deleted_root_cdb_is_restored.sh | 4 +- tests/e2e/78_test_main_combinations.sh | 2 +- ...e_action_populates_an_unknown_directory.sh | 2 +- .../e2e/79_gtest_regular_dep_feature_main.sh | 4 +- ...808_device_source_is_not_a_compile_unit.sh | 2 +- ...kages_deploy_the_same_bytes_to_one_name.sh | 2 +- ..._conditional_dialect_cxxflags_root_only.sh | 14 +++---- ...he_llvm_row_defaults_to_the_dynamic_crt.sh | 2 +- ...an_edge_states_its_advice_on_both_paths.sh | 2 +- .../827_many_placements_run_as_one_process.sh | 4 +- ...ed_bmi_waits_for_a_module_compiled_here.sh | 2 +- ...mber_is_compiled_once_per_configuration.sh | 4 +- ...ed_dash_p_selects_every_member_it_names.sh | 4 +- ..._one_member_and_exclude_removes_members.sh | 2 +- ...several_members_import_is_compiled_once.sh | 2 +- ...members_names_each_member_in_its_report.sh | 2 +- tests/e2e/86_llvm_hermetic_link.sh | 2 +- ...ild_and_every_drive_takes_the_job_count.sh | 2 +- ...er_compiles_once_whatever_the_selection.sh | 4 +- ...ed_engine_does_not_replay_the_old_graph.sh | 2 +- ...workspace_profiles_and_the_root_package.sh | 4 +- .../e2e/887_msvc_lto_and_export_discovery.sh | 2 +- ...nfiguration_rows_count_as_they_evaluate.sh | 4 +- ...d_database_describes_the_build_programs.sh | 3 +- tests/e2e/92_build_mcpp_import.sh | 4 +- tests/e2e/_cxx_identity_across_images_body.sh | 2 +- tests/e2e/_windows_resources_body.sh | 2 +- 111 files changed, 218 insertions(+), 173 deletions(-) diff --git a/src/build/build_program.cppm b/src/build/build_program.cppm index fa7e3765..b9c509ce 100644 --- a/src/build/build_program.cppm +++ b/src/build/build_program.cppm @@ -919,27 +919,27 @@ CacheRecord read_cache(const fs::path& bdir) { return r; } -// Decide whether the cached run is still valid (so we can skip recompiling/running). -bool cache_fresh(const fs::path& root, const fs::path& bdir, const CacheRecord& c, - const std::string& programHash, const std::string& compilerHash, - const std::string& ctxHash) { - if (!c.loaded) return false; - if (c.epoch != dirs::kCacheEpoch) return false; // pre-epoch entries rerun once - if (c.unknownRecord) return false; - if (c.programHash != programHash) return false; - if (c.compilerHash != compilerHash) return false; - if (c.ctxHash != ctxHash) return false; // pre-G3 caches (no ctx line) rerun once +// Why the cached run is not valid, for the log; empty when it is. +std::string cache_stale_reason(const fs::path& root, const fs::path& bdir, const CacheRecord& c, + const std::string& programHash, const std::string& compilerHash, + const std::string& ctxHash) { + if (!c.loaded) return "no record"; + if (c.epoch != dirs::kCacheEpoch) return "record epoch"; // pre-epoch entries rerun once + if (c.unknownRecord) return "unknown record line"; + if (c.programHash != programHash) return "program identity " + c.programHash + " -> " + programHash; + if (c.compilerHash != compilerHash) return "record shape"; + if (c.ctxHash != ctxHash) return "environment contract"; // pre-G3 caches rerun once for (auto const& [h, path] : c.inputs) - if (mcpp::toolchain::hash_file(abs_against_root(root, path)) != h) return false; + if (mcpp::toolchain::hash_file(abs_against_root(root, path)) != h) return "input " + path; for (auto const& [h, name] : c.envs) - if (mcpp::toolchain::hash_string(env_value(name)) != h) return false; + if (mcpp::toolchain::hash_string(env_value(name)) != h) return "environment variable " + name; // #359: the path SET behind each declared glob. A file appearing or // disappearing changes it; editing one does not (that is what the `in` // entries above are for). if (!c.globs.empty()) { auto outName = output_dir_name(root, bdir); for (auto const& [h, pattern] : c.globs) - if (dirs::glob_fingerprint(root, pattern, outName) != h) return false; + if (dirs::glob_fingerprint(root, pattern, outName) != h) return "glob " + pattern; } // A declared output that vanished invalidates the cache. Driven off the // table's mustExistAfterRun so a future output-shaped directive is covered @@ -947,9 +947,16 @@ bool cache_fresh(const fs::path& root, const fs::path& bdir, const CacheRecord& for (auto const& def : dirs::kTable) { if (!def.mustExistAfterRun) continue; for (auto const& p : c.directives.at(def.slot)) - if (!fs::exists(abs_against_root(root, p))) return false; + if (!fs::exists(abs_against_root(root, p))) return "output " + p; } - return true; + return {}; +} + +// Decide whether the cached run is still valid (so we can skip recompiling/running). +bool cache_fresh(const fs::path& root, const fs::path& bdir, const CacheRecord& c, + const std::string& programHash, const std::string& compilerHash, + const std::string& ctxHash) { + return cache_stale_reason(root, bdir, c, programHash, compilerHash, ctxHash).empty(); } @@ -1052,7 +1059,10 @@ std::expected run_build_program_impl( // Fast path: declared inputs + contract unchanged → reapply cached // directives, no run. CacheRecord cache = read_cache(bdir); - if (cache_fresh(root, bdir, cache, programHash, compilerHash, ctxHash)) { + const auto stale = cache_stale_reason(root, bdir, cache, programHash, compilerHash, ctxHash); + if (!stale.empty()) + mcpp::log::verbose("build-program", std::format("{}: runs ({})", who, stale)); + if (stale.empty()) { if (auto terr = dirs::target_directive_error(m, cache.directives); !terr.empty()) return std::unexpected(terr); // #622 A4. Checked on the cache-hit path too, so a `deploy` a fresh diff --git a/src/build/engine_package.cppm b/src/build/engine_package.cppm index a85c9a97..9d4b10a1 100644 --- a/src/build/engine_package.cppm +++ b/src/build/engine_package.cppm @@ -105,8 +105,17 @@ fs::path root() { return mcpp::home::root() / "engine"; } fs::path package_dir() { return root() / std::string(mcpp::MCPP_VERSION) / "modules"; } bool is_engine_package(const fs::path& dir) { - auto rel = dir.lexically_normal().lexically_relative(root().lexically_normal()); - return !rel.empty() && *rel.begin() != ".."; + auto inside = [](const fs::path& a, const fs::path& b) { + auto rel = a.lexically_normal().lexically_relative(b.lexically_normal()); + return !rel.empty() && *rel.begin() != ".."; + }; + if (inside(dir, root())) return true; + // One directory has more than one spelling: `/var` is `/private/var` on + // macOS, and Windows has HOME and USERPROFILE, drive-letter case, 8.3 names. + std::error_code e1, e2; + const auto cd = fs::weakly_canonical(dir, e1); + const auto cr = fs::weakly_canonical(root(), e2); + return !e1 && !e2 && inside(cd, cr); } std::pair module_texts() { diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 8d64eaa0..ecd39680 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -811,6 +811,10 @@ export struct BuildOverrides { // not apply to them. std::map resolution_pins; bool build_program_graph = false; + // The build targets the machine doing it: a sub-build of build programs + // or of a host tool. Neither `[build] target` nor the configured default + // target applies, since both name the target of the build that asked. + bool host_target = false; // The project whose `.mcpp/` holds the packages of its custom indices. // Empty = the project being built. The build programs' graph names the // build that asked, whose indices it shares and whose store already holds diff --git a/src/build/prepare/driver.cpp b/src/build/prepare/driver.cpp index 51ca3a7c..d0f59746 100644 --- a/src/build/prepare/driver.cpp +++ b/src/build/prepare/driver.cpp @@ -112,6 +112,7 @@ void read_build_programs_marker(BuildOverrides& ov) { std::ifstream in(root / ".mcpp-build-programs"); if (!in) return; ov.build_program_graph = true; + ov.host_target = true; std::string line; while (std::getline(in, line)) { const auto eq = line.find('='); diff --git a/src/build/prepare/features.cpp b/src/build/prepare/features.cpp index cad3974d..151d2c2a 100644 --- a/src/build/prepare/features.cpp +++ b/src/build/prepare/features.cpp @@ -1480,6 +1480,7 @@ step6_build_tool(PrepareState& state, HostToolCtx& ctx) { sub.work_dir = mcpp::build::tool_store::scratch_dir( ctx.cacheRoot, ctx.entry, state.workRoot); sub.target_triple = ""; // HOST — the whole point + sub.host_target = true; sub.toolchain = ctx.toolTcSpec; sub.profile = "release"; sub.cache_mode = state.overrides.cache_mode; diff --git a/src/build/prepare/host_graph.cpp b/src/build/prepare/host_graph.cpp index 3114f41a..c983db6d 100644 --- a/src/build/prepare/host_graph.cpp +++ b/src/build/prepare/host_graph.cpp @@ -652,6 +652,7 @@ build_group(PrepareState& state, const fs::path& dir, const std::vector sub.project_root = dir; sub.work_dir = dir; sub.target_triple = ""; + sub.host_target = true; sub.toolchain = tc.spec; sub.profile = "build-program"; sub.cache_mode = state.overrides.cache_mode; @@ -711,10 +712,17 @@ build_group(PrepareState& state, const fs::path& dir, const std::vector } } for (auto const* g : goals) - if (!binaries.contains(g->member)) + if (!binaries.contains(g->member)) { + std::string planned; + for (auto const& lu : ctx->plan.linkUnits) + planned += std::format("{}{}:{}", planned.empty() ? "" : ", ", + lu.memberOf.empty() ? lu.package : lu.memberOf, + lu.targetName); return std::unexpected(std::format( - "the build programs' plan has no program for '{}'{}", - qualified(manifest_of(state, g->index)), reproduce())); + "the build programs' plan has no program for '{}' (member {}; planned: {}){}", + qualified(manifest_of(state, g->index)), g->member, + planned.empty() ? std::string("nothing") : planned, reproduce())); + } auto be = mcpp::build::make_ninja_backend(); mcpp::build::BuildOptions bopt; diff --git a/src/build/prepare/toolchain.cpp b/src/build/prepare/toolchain.cpp index 5ddebc69..7e92359f 100644 --- a/src/build/prepare/toolchain.cpp +++ b/src/build/prepare/toolchain.cpp @@ -1010,12 +1010,13 @@ static std::expected step1_target_and_static_overrides(Prepar // Target-axis default resolution when no --target flag was passed: // [build] target (project default, ≙ cargo build.target) > // [toolchain] default_target (global config) > host. - if (state.overrides.target_triple.empty() && !state.m->buildConfig.target.empty()) + if (state.overrides.target_triple.empty() && !state.m->buildConfig.target.empty() + && !state.overrides.host_target) state.overrides.target_triple = state.m->buildConfig.target; // Remembered, not requested: this one came out of the global config, so // it must not outrank anything the user wrote down (see the pin below). bool targetFromGlobalDefault = false; - if (state.overrides.target_triple.empty()) { + if (state.overrides.target_triple.empty() && !state.overrides.host_target) { if (auto cfg = state.get_cfg(true); cfg && !(*cfg)->defaultTarget.empty()) { state.overrides.target_triple = (*cfg)->defaultTarget; targetFromGlobalDefault = true; diff --git a/src/build/prepare/windows_resources.cpp b/src/build/prepare/windows_resources.cpp index ab202249..96624b1d 100644 --- a/src/build/prepare/windows_resources.cpp +++ b/src/build/prepare/windows_resources.cpp @@ -526,9 +526,19 @@ std::expected step13_windows_resources(PrepareState& state, B // Read by W3 (a member's non-public modules). std::string workspace_member_of(const PrepareState& state, const std::filesystem::path& root) { if (!state.wsManifest || state.runtimeWorkspaceRoot.empty()) return {}; - const auto rel = root.lexically_normal() - .lexically_relative(state.runtimeWorkspaceRoot.lexically_normal()) - .generic_string(); + auto relative = [](const std::filesystem::path& a, const std::filesystem::path& b) { + return a.lexically_normal().lexically_relative(b.lexically_normal()).generic_string(); + }; + auto rel = relative(root, state.runtimeWorkspaceRoot); + // One directory has more than one spelling (`/var` is `/private/var` on + // macOS; a symlinked home; HOME and USERPROFILE on Windows): a member + // reached through the other spelling is the same member. + if (rel.empty() || rel.starts_with("..")) { + std::error_code e1, e2; + const auto cr = std::filesystem::weakly_canonical(root, e1); + const auto cw = std::filesystem::weakly_canonical(state.runtimeWorkspaceRoot, e2); + if (!e1 && !e2) rel = relative(cr, cw); + } if (rel.empty() || rel == "." || rel.starts_with("..")) return {}; for (auto const& m : state.wsManifest->workspace.members) { if (m == rel) return rel; diff --git a/tests/e2e/03_multi_module.sh b/tests/e2e/03_multi_module.sh index 42b8d95e..af23803f 100755 --- a/tests/e2e/03_multi_module.sh +++ b/tests/e2e/03_multi_module.sh @@ -36,7 +36,7 @@ out=$("$MCPP" run 2>&1) [[ "$out" == *"multi-module build OK"* ]] || { echo "main println missing: $out"; exit 1; } # Verify build.ninja contents -build_ninja="$(find target -name build.ninja | head -1)" +build_ninja="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -q "gcm.cache/multi.greeter.gcm" "$build_ninja" || { echo "ninja missing greeter BMI"; exit 1; } grep -q "gcm.cache/std.gcm" "$build_ninja" || { echo "ninja missing std BMI"; exit 1; } diff --git a/tests/e2e/07_static_library.sh b/tests/e2e/07_static_library.sh index 98af0def..32eb1f0a 100755 --- a/tests/e2e/07_static_library.sh +++ b/tests/e2e/07_static_library.sh @@ -45,7 +45,7 @@ AR="${SANDBOX_AR:-ar}" "$AR" t "$archive" | grep -q '\.o$' || { echo ".a contains no objects"; exit 1; } # build.ninja must use cxx_archive rule for the lib target -ninja_file="$(find target -name build.ninja)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*')" grep -q 'bin/libmathlib.a : cxx_archive' "$ninja_file" || { echo "build.ninja did not use cxx_archive for the lib target"; exit 1; } diff --git a/tests/e2e/08_shared_library.sh b/tests/e2e/08_shared_library.sh index 2b91755a..e36a0f41 100755 --- a/tests/e2e/08_shared_library.sh +++ b/tests/e2e/08_shared_library.sh @@ -39,7 +39,7 @@ so="$(find target -name 'libmathlib.so' | head -1)" file "$so" | grep -q 'ELF.*shared object' || { echo "not a valid .so"; exit 1; } # build.ninja must use cxx_shared rule -ninja_file="$(find target -name build.ninja)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*')" grep -q 'bin/libmathlib.so : cxx_shared' "$ninja_file" || { echo "build.ninja did not use cxx_shared for the shared lib target"; exit 1; } diff --git a/tests/e2e/09_path_dependency.sh b/tests/e2e/09_path_dependency.sh index e023286c..59f818d3 100755 --- a/tests/e2e/09_path_dependency.sh +++ b/tests/e2e/09_path_dependency.sh @@ -75,7 +75,7 @@ out=$("$MCPP" run 2>&1) [[ "$out" == *"path dep build OK"* ]] || { echo "main println missing: $out"; exit 1; } # build.ninja should compile sources from BOTH packages -ninja_file="$(find target -name build.ninja)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*')" grep -q 'mylibA.*greet.cppm' "$ninja_file" || { echo "ninja missing dep package source"; exit 1; } # A dependency's BMI lies below its package's directory (pack drive and # selection design 2026-10-01, B1); the root's would lie at its name. diff --git a/tests/e2e/100_cppfly_reflection.sh b/tests/e2e/100_cppfly_reflection.sh index 313edd6d..2e04dc6f 100755 --- a/tests/e2e/100_cppfly_reflection.sh +++ b/tests/e2e/100_cppfly_reflection.sh @@ -80,7 +80,7 @@ grep -q 'enabled: .*contracts' "$TMP/build.log" || { } # Latest level + gate flags reach the global cxxflags (every TU, deps incl.). -build_ninja="$(find target -name build.ninja | head -1)" +build_ninja="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE '^cxxflags = -std=c\+\+26 -freflection' "$build_ninja" || { grep '^cxxflags' "$build_ninja" || true echo "FAIL: build.ninja cxxflags lacks -std=c++26 -freflection" @@ -137,7 +137,7 @@ EOF echo "FAIL: c++latest build failed" exit 1 } -latest_ninja="$(find target -name build.ninja | head -1)" +latest_ninja="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE '^cxxflags = -std=c\+\+26' "$latest_ninja" || { grep '^cxxflags' "$latest_ninja" || true echo "FAIL: c++latest did not resolve to -std=c++26" diff --git a/tests/e2e/100_feature_sources_test_mode.sh b/tests/e2e/100_feature_sources_test_mode.sh index f0c89fd2..4accb365 100755 --- a/tests/e2e/100_feature_sources_test_mode.sh +++ b/tests/e2e/100_feature_sources_test_mode.sh @@ -55,7 +55,7 @@ EOF "$MCPP" test 2>&1 | tail -5 exit 1 } -nj=$(find target -name build.ninja | xargs ls -t 2>/dev/null | head -1) +nj=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | xargs ls -t 2>/dev/null | head -1) grep -q 'cJSON_Utils' "$nj" || { echo "FAIL: cJSON_Utils.c absent from the test-mode build graph"; exit 1 } @@ -71,7 +71,7 @@ int main() { } EOF "$MCPP" build > /dev/null || { echo "FAIL: build with feature sources failed"; exit 1; } -nj=$(find target -name build.ninja | xargs ls -t 2>/dev/null | head -1) +nj=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | xargs ls -t 2>/dev/null | head -1) grep -q 'cJSON_Utils' "$nj" || { echo "FAIL: cJSON_Utils.c absent from build graph"; exit 1; } # (3) The drop still works: without the feature, the gated source stays OUT of a @@ -89,7 +89,7 @@ cat > src/main.cpp <<'EOF' int main() { cJSON* r = cJSON_Parse("{\"a\":1}"); cJSON_Delete(r); return 0; } EOF "$MCPP" build > /dev/null || { echo "FAIL: default (no feature) build failed"; exit 1; } -nj=$(find target -name build.ninja | xargs ls -t 2>/dev/null | head -1) +nj=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | xargs ls -t 2>/dev/null | head -1) if grep -q 'cJSON_Utils' "$nj"; then echo "FAIL: cJSON_Utils.c compiled without the \`utils\` feature"; exit 1 fi diff --git a/tests/e2e/101_cppfly_llvm_soft.sh b/tests/e2e/101_cppfly_llvm_soft.sh index a304f873..d846caea 100755 --- a/tests/e2e/101_cppfly_llvm_soft.sh +++ b/tests/e2e/101_cppfly_llvm_soft.sh @@ -52,7 +52,7 @@ grep -q 'skipped: .*reflection' "$TMP/build.log" || { exit 1 } -build_ninja="$(find target -name build.ninja | head -1)" +build_ninja="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE '^cxxflags = -std=c\+\+2c' "$build_ninja" || { grep '^cxxflags' "$build_ninja" || true echo "FAIL: clang c++fly did not resolve to -std=c++2c" diff --git a/tests/e2e/102_mingw_cross_wine.sh b/tests/e2e/102_mingw_cross_wine.sh index 590ac7ca..2216e5a5 100644 --- a/tests/e2e/102_mingw_cross_wine.sh +++ b/tests/e2e/102_mingw_cross_wine.sh @@ -101,7 +101,7 @@ out="$(wine "$EXE" 2>/dev/null)" [[ "$out" == *"import-std + multi-module OK"* ]] || { echo "main output missing: $out"; exit 1; } # ── ninja sanity: std BMI + module BMI were built for the cross target ────── -build_ninja="$(find target/$TRIPLE -name build.ninja | head -1)" +build_ninja="$(find target/$TRIPLE -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -q "gcm.cache/std.gcm" "$build_ninja" || { echo "ninja missing std BMI"; exit 1; } grep -q "gcm.cache/crosswin.banner.gcm" "$build_ninja" || { echo "ninja missing module BMI"; exit 1; } diff --git a/tests/e2e/104_asm_sources_gas.sh b/tests/e2e/104_asm_sources_gas.sh index bb0bf1be..7ad4ff66 100755 --- a/tests/e2e/104_asm_sources_gas.sh +++ b/tests/e2e/104_asm_sources_gas.sh @@ -53,7 +53,7 @@ EOF "$MCPP" build > build.log 2>&1 || { cat build.log; echo "build failed"; exit 1; } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "no build.ninja generated"; exit 1; } grep -q '^rule asm_object' "$ninja_file" || { diff --git a/tests/e2e/105_asm_sources_nasm.sh b/tests/e2e/105_asm_sources_nasm.sh index d8ddbd6d..ad50a5fd 100755 --- a/tests/e2e/105_asm_sources_nasm.sh +++ b/tests/e2e/105_asm_sources_nasm.sh @@ -54,7 +54,7 @@ EOF "$MCPP" build > build.log 2>&1 || { cat build.log; echo "build failed"; exit 1; } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "no build.ninja generated"; exit 1; } grep -q '^rule nasm_object' "$ninja_file" || { diff --git a/tests/e2e/117_object_path_collision.sh b/tests/e2e/117_object_path_collision.sh index e1aad4f7..56716975 100755 --- a/tests/e2e/117_object_path_collision.sh +++ b/tests/e2e/117_object_path_collision.sh @@ -55,7 +55,7 @@ EOF exit 1 } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "no build.ninja generated"; exit 1; } # Both util.cpp compile edges must be present, each with a DISTINCT object diff --git a/tests/e2e/120_ws_root_indices.sh b/tests/e2e/120_ws_root_indices.sh index e2aa5c23..de19eb89 100755 --- a/tests/e2e/120_ws_root_indices.sh +++ b/tests/e2e/120_ws_root_indices.sh @@ -186,9 +186,9 @@ grep -q '\[workspace.indices\]' build.log || { echo "FAIL: --workspace" exit 1 } -n=$(find target -name build.ninja | wc -l) +n=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | wc -l) [[ "$n" == 1 ]] || { - find target -name build.ninja + find target -name build.ninja -not -path '*/.build-mcpp/*' echo "FAIL: members naming one index tree are $n configurations, expected 1" exit 1 } diff --git a/tests/e2e/123_same_named_main_across_dep.sh b/tests/e2e/123_same_named_main_across_dep.sh index 238ebe96..878f0d9e 100755 --- a/tests/e2e/123_same_named_main_across_dep.sh +++ b/tests/e2e/123_same_named_main_across_dep.sh @@ -62,7 +62,7 @@ cd consumer exit 1 } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "no build.ninja generated"; exit 1; } # THE invariant: every object the link edge names must be PRODUCED by an edge in diff --git a/tests/e2e/124_gen_source_out_of_root_object_path.sh b/tests/e2e/124_gen_source_out_of_root_object_path.sh index 0581a013..717c1b3b 100755 --- a/tests/e2e/124_gen_source_out_of_root_object_path.sh +++ b/tests/e2e/124_gen_source_out_of_root_object_path.sh @@ -82,7 +82,7 @@ cd consumer exit 1 } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "no build.ninja generated"; exit 1; } # No object output may escape obj/ via `..` or an absolute root. diff --git a/tests/e2e/141_include_dirs_after.sh b/tests/e2e/141_include_dirs_after.sh index 8c781283..ee85dbf5 100755 --- a/tests/e2e/141_include_dirs_after.sh +++ b/tests/e2e/141_include_dirs_after.sh @@ -85,7 +85,7 @@ out=$("$MCPP" run 2>&1) [[ "$out" == *"answer = 42"* ]] || { echo "dep header not found via -idirafter: $out"; exit 1; } # The -idirafter flag (not -I) must carry the dep's include root. -ninja_file="$(find target -name build.ninja)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*')" grep -q -- '-idirafter[^ ]*poisonlib[^ ]*inc' "$ninja_file" || { echo "ninja missing -idirafter for dep include root"; exit 1; } diff --git a/tests/e2e/144_build_mcpp_include_dir.sh b/tests/e2e/144_build_mcpp_include_dir.sh index eedfde95..3da15f7b 100755 --- a/tests/e2e/144_build_mcpp_include_dir.sh +++ b/tests/e2e/144_build_mcpp_include_dir.sh @@ -66,7 +66,7 @@ out="$("$MCPP" run 2>&1 | grep '^ANSWER=' | tail -1)" # include-dir-after reaches the compile edges as -idirafter (typed #249 # channel: privateBuild.includeDirsAfter → localIncludeDirsAfter). -ninja_file=$(find target -name build.ninja | head -1) +ninja_file=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) grep -q -- "-idirafter.*vendor/sysinc" "$ninja_file" || { echo "FAIL: include-dir-after not emitted as -idirafter in build.ninja"; grep -n "local_includes" "$ninja_file" | head; exit 1; } diff --git a/tests/e2e/164_ninja_goal_command_length.sh b/tests/e2e/164_ninja_goal_command_length.sh index edf2c557..cb054fb3 100755 --- a/tests/e2e/164_ninja_goal_command_length.sh +++ b/tests/e2e/164_ninja_goal_command_length.sh @@ -45,7 +45,7 @@ cd proj "$MCPP" test > test.log 2>&1 || { cat test.log; echo "FAIL: mcpp test"; exit 1; } grep -q "sanity ... ok" test.log || { cat test.log; exit 1; } -NINJA=$(find target -name build.ninja | head -1) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [ -n "$NINJA" ] || { echo "FAIL: no build.ninja"; exit 1; } # The phony edge is how the goal set reaches ninja. diff --git a/tests/e2e/170_bmi_staging_no_cascade.sh b/tests/e2e/170_bmi_staging_no_cascade.sh index 0e01b5de..628cd542 100755 --- a/tests/e2e/170_bmi_staging_no_cascade.sh +++ b/tests/e2e/170_bmi_staging_no_cascade.sh @@ -43,7 +43,7 @@ EOF cd app "$MCPP" build > build1.log 2>&1 || { cat build1.log; echo "FAIL: initial build"; exit 1; } -NINJA=$(find target -name build.ninja | head -1) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [[ -n "$NINJA" ]] || { echo "FAIL: no build.ninja produced"; exit 1; } # Parse the staging edge, NOT the rule declaration: `awk '{print $NF}'` over a diff --git a/tests/e2e/171_bmi_staging_locked_dest.sh b/tests/e2e/171_bmi_staging_locked_dest.sh index 6bd7998c..b90bbb0a 100755 --- a/tests/e2e/171_bmi_staging_locked_dest.sh +++ b/tests/e2e/171_bmi_staging_locked_dest.sh @@ -53,7 +53,7 @@ EOF cd app "$MCPP" build > build1.log 2>&1 || { cat build1.log; echo "FAIL: initial build"; exit 1; } -NINJA=$(find target -name build.ninja | head -1) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) EDGE=$(grep -E '^build [^ ]+ : stage_file ' "$NINJA" | head -1) [[ -n "$EDGE" ]] || { echo "FAIL: no stage_file edge for the std BMI"; exit 1; } DST="$(dirname "$NINJA")/$(unescape_ninja "$(echo "$EDGE" | awk '{print $2}')")" diff --git a/tests/e2e/173_build_profile_isolation.sh b/tests/e2e/173_build_profile_isolation.sh index 516903cf..a637a4c5 100644 --- a/tests/e2e/173_build_profile_isolation.sh +++ b/tests/e2e/173_build_profile_isolation.sh @@ -58,7 +58,7 @@ dir_for_opt() { "$MCPP" build --release > release.log 2>&1 || { cat release.log; exit 1; } "$MCPP" build --profile dist > dist.log 2>&1 || { cat dist.log; exit 1; } -count=$(find target -name build.ninja | wc -l) +count=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | wc -l) [[ "$count" -eq 3 ]] || { echo "FAIL: expected one build dir per profile, found $count" for d in target/*/*/; do echo " $d -> $(opt_of "$d")"; done diff --git a/tests/e2e/174_cache_modes_and_commands.sh b/tests/e2e/174_cache_modes_and_commands.sh index 5c8b34d3..b3914bdf 100644 --- a/tests/e2e/174_cache_modes_and_commands.sh +++ b/tests/e2e/174_cache_modes_and_commands.sh @@ -98,7 +98,7 @@ out=$("$MCPP" cache dir 2>&1) exit 1 } # The dependency still has to be built, in-project. -NINJA="$(find target -name build.ninja | head -1)" +NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { echo "FAIL: --cache=local did not compile the dependency locally" grep -n 'mode-lib' "$NINJA" | head @@ -167,7 +167,7 @@ mv "$victim.bak" "$victim" # ── off: clears this build's build dir and touches no cache ──────────────── # The build DIRECTORY (target///), not all of target/: sibling dirs # for other profiles and other targets are not this invocation's business. -builddir="$(dirname "$(find target -name build.ninja | head -1)")" +builddir="$(dirname "$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)")" marker="$builddir/_off_marker" touch "$marker" before="$(entry_count)" @@ -177,7 +177,7 @@ before="$(entry_count)" echo "FAIL: --cache=off changed the cache ($before -> $(entry_count))" exit 1 } -NINJA="$(find target -name build.ninja | head -1)" +NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { echo "FAIL: --cache=off served the dependency from the cache" exit 1 @@ -197,7 +197,7 @@ grep -qi 'cache mode' bogus.log || { cat bogus.log; echo "FAIL: unhelpful error" # MCPP_BUILD_CACHE is the env-level equivalent. rm -rf target MCPP_BUILD_CACHE=local "$MCPP" build > env.log 2>&1 || { cat env.log; exit 1; } -NINJA="$(find target -name build.ninja | head -1)" +NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { echo "FAIL: MCPP_BUILD_CACHE=local was ignored" exit 1 @@ -211,14 +211,14 @@ cache = "local" EOF rm -rf target "$MCPP" build > manifest.log 2>&1 || { cat manifest.log; exit 1; } -NINJA="$(find target -name build.ninja | head -1)" +NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA" || { echo "FAIL: [build] cache = local was ignored" exit 1 } rm -rf target "$MCPP" build --cache=global > override.log 2>&1 || { cat override.log; exit 1; } -NINJA="$(find target -name build.ninja | head -1)" +NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" if grep -qE ': (cxx_module|cxx_object) .*mode-lib' "$NINJA"; then echo "FAIL: --cache=global did not override [build] cache = local" exit 1 @@ -261,7 +261,7 @@ cache = "local" EOF rm -rf target "$MCPP" build --cache=global > modeglobal.log 2>&1 || { cat modeglobal.log; exit 1; } -NINJA="$(find target -name build.ninja | head -1)" +NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" staged_before=$(grep -c ': stage_file .*mode-lib' "$NINJA" || true) [[ "$staged_before" -gt 0 ]] || { echo "FAIL: --cache=global did not produce stage edges (cache should be warm here)" @@ -272,7 +272,7 @@ staged_before=$(grep -c ': stage_file .*mode-lib' "$NINJA" || true) # global graph. Note mcpp.toml is untouched between these two builds, so its # mtime cannot be what saves us — only the recorded mode can. "$MCPP" build > modebare.log 2>&1 || { cat modebare.log; exit 1; } -NINJA="$(find target -name build.ninja | head -1)" +NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" staged_after=$(grep -c ': stage_file .*mode-lib' "$NINJA" || true) [[ "$staged_after" -eq 0 ]] || { echo "FAIL: bare build replayed the global-mode graph ($staged_after stage edges)" diff --git a/tests/e2e/175_cpp20_import_std.sh b/tests/e2e/175_cpp20_import_std.sh index 84b04d89..ee388c6d 100755 --- a/tests/e2e/175_cpp20_import_std.sh +++ b/tests/e2e/175_cpp20_import_std.sh @@ -69,7 +69,7 @@ out=$("$binary") } # The level reaches the module-graph-global cxxflags (every TU, deps included). -build_ninja="$(find target -name build.ninja | head -1)" +build_ninja="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE '^cxxflags = -std=c\+\+20' "$build_ninja" || { grep '^cxxflags' "$build_ninja" || true echo "FAIL: build.ninja cxxflags is not -std=c++20" @@ -112,7 +112,7 @@ EOF echo "FAIL: gnu++20 build failed" exit 1 } -gnu_ninja="$(find target -name build.ninja | head -1)" +gnu_ninja="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE '^cxxflags = -std=gnu\+\+20' "$gnu_ninja" || { grep '^cxxflags' "$gnu_ninja" || true echo "FAIL: gnu++20 did not reach cxxflags" diff --git a/tests/e2e/179_spaced_paths.sh b/tests/e2e/179_spaced_paths.sh index 39a49464..c70661d1 100755 --- a/tests/e2e/179_spaced_paths.sh +++ b/tests/e2e/179_spaced_paths.sh @@ -69,7 +69,7 @@ run_out=$("$MCPP" run 2>&1) || { echo "FAIL: run: $run_out"; exit 1; } # The include flag must reach the compiler as ONE shell word. Assert on the # generated ninja file rather than on the build succeeding by luck: a split # flag can still compile if the header happens to be findable another way. -ninja_file=$(find target -name build.ninja | head -1) +ninja_file=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [[ -n "$ninja_file" ]] || { echo "FAIL: no build.ninja found"; exit 1; } grep -q "vendor" "$ninja_file" || { echo "FAIL: include dir absent from build.ninja"; exit 1; } diff --git a/tests/e2e/183_cxx_runtime_contract.sh b/tests/e2e/183_cxx_runtime_contract.sh index 47659d19..ab1bb754 100755 --- a/tests/e2e/183_cxx_runtime_contract.sh +++ b/tests/e2e/183_cxx_runtime_contract.sh @@ -96,7 +96,7 @@ echo "$test_out" | grep -q "test result ok" || { # there and actually first — a silently-absent shim would leave this test # passing by luck on a future toolchain and failing for users on another. if [[ "$HOST" == macos ]]; then - NINJA=$(find target -name build.ninja | head -1) + NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) grep -q "mcpp_ios_init" "$NINJA" || { echo "FAIL: no initializer-ordering shim in the macOS link"; exit 1; } # `build : cxx_link ...` @@ -112,7 +112,7 @@ fi echo "== 3. cxx_runtime = host-coupled reaches BOTH roles (#336 part one) ==" # Capture the self-contained shape first so the comparison is against this # machine's actual output rather than a hard-coded flag string. -NINJA=$(find target -name build.ninja | head -1) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) self_units=$(grep -c "unit_ldflags" "$NINJA" || true) cat >> mcpp.toml <<'EOF' @@ -123,7 +123,7 @@ EOF "$MCPP" build >/dev/null 2>&1 || { echo "FAIL: host-coupled build"; exit 1; } "$MCPP" test >/dev/null 2>&1 || { echo "FAIL: host-coupled test"; exit 1; } -NINJA=$(find target -name build.ninja | head -1) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) # Whatever the platform's self-contained mechanism is (-static-libstdc++, # -static, -load_hidden archives), host-coupled must not carry it — for the # test target as much as for the binary. @@ -138,7 +138,7 @@ echo "== 4. the older spelling means the same thing ==" sed 's/cxx_runtime = "host-coupled"/static_stdlib = false/' mcpp.toml > mcpp.toml.tmp mv mcpp.toml.tmp mcpp.toml "$MCPP" build >/dev/null 2>&1 || { echo "FAIL: static_stdlib=false build"; exit 1; } -NINJA=$(find target -name build.ninja | head -1) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) if grep -E "unit_ldflags.*(-static-libstdc\+\+|-load_hidden|-nostdlib\+\+)" "$NINJA" >/dev/null; then echo "FAIL: static_stdlib = false is not an alias of host-coupled:" grep "unit_ldflags" "$NINJA" diff --git a/tests/e2e/188_build_actions.sh b/tests/e2e/188_build_actions.sh index 4cb79a84..54e0393b 100755 --- a/tests/e2e/188_build_actions.sh +++ b/tests/e2e/188_build_actions.sh @@ -297,7 +297,7 @@ version = "0.1.0" EOF printf 'int main() { return 0; }\n' > src/main.cpp "$MCPP" build > o0.log 2>&1 || { cat o0.log; echo "FAIL: probe build failed"; exit 1; } -OBJ_NINJA=$(find target -name build.ninja | head -1) +OBJ_NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) # A command line, written into mkobj.sh as one: it may start with # `env LD_LIBRARY_PATH=...` for a compiler without its own search path. OBJ_CXX=$(sed -n 's/^cxx *= *//p' "$OBJ_NINJA" | head -1) diff --git a/tests/e2e/190_link_rspfile_newlines.sh b/tests/e2e/190_link_rspfile_newlines.sh index 41fa677f..94a14002 100755 --- a/tests/e2e/190_link_rspfile_newlines.sh +++ b/tests/e2e/190_link_rspfile_newlines.sh @@ -46,7 +46,7 @@ printf 'int main() { return 0; }\n' > multi/src/main.cpp cd multi "$MCPP" build > b.log 2>&1 || { cat b.log; echo "FAIL: build"; exit 1; } -ninja_file=$(find target -name build.ninja | head -1) +ninja_file=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [ -n "$ninja_file" ] || { echo "FAIL: no build.ninja"; exit 1; } # 1. Structural: no link rule may write its response file on one line. Under diff --git a/tests/e2e/191_link_scale.sh b/tests/e2e/191_link_scale.sh index d68414af..735796b7 100755 --- a/tests/e2e/191_link_scale.sh +++ b/tests/e2e/191_link_scale.sh @@ -94,7 +94,7 @@ cd scale "$MCPP" build > b.log 2>&1 || { tail -40 b.log; echo "FAIL: build $N objects"; exit 1; } echo " ok: built $((N + 1)) objects" -ninja_file=$(find target -name build.ninja | head -1) +ninja_file=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [ -n "$ninja_file" ] || { echo "FAIL: no build.ninja"; exit 1; } bdir=$(dirname "$ninja_file") # The program by its name: on an MSVC-ABI row bin/ also holds the toolset's diff --git a/tests/e2e/198_windows_resources_cross.sh b/tests/e2e/198_windows_resources_cross.sh index 24f51dff..2f49a7a2 100755 --- a/tests/e2e/198_windows_resources_cross.sh +++ b/tests/e2e/198_windows_resources_cross.sh @@ -37,7 +37,7 @@ main = "src/main.cpp" TOML "$MCPP" build > host.log 2>&1 || { cat host.log; echo "FAIL: host build with [resources] failed"; exit 1; } grep -qi 'resource' host.log && { cat host.log; echo "FAIL: a non-PE build must say nothing about resources"; exit 1; } -HOST_DIR=$(dirname "$(find target -name 'build.ninja' -print | xargs grep -L 'rc_object' | head -1)") +HOST_DIR=$(dirname "$(find target -name 'build.ninja' -not -path '*/.build-mcpp/*' -print | xargs grep -L 'rc_object' | head -1)") [ -d "$HOST_DIR/res" ] && { echo "FAIL: a non-PE build must not emit resource units"; exit 1; } # ── A4. "inapplicable" stops at COMPILATION, not at validation ──────────── diff --git a/tests/e2e/207_runtime_contract_provenance.sh b/tests/e2e/207_runtime_contract_provenance.sh index fe5e005f..77f8ca3c 100755 --- a/tests/e2e/207_runtime_contract_provenance.sh +++ b/tests/e2e/207_runtime_contract_provenance.sh @@ -75,7 +75,7 @@ cd "$TMP/app" exit 1 } -RES=$(find target -name resolution.json | head -1) +RES=$(find target -name resolution.json -not -path '*/.build-mcpp/*' | head -1) test -n "$RES" || { echo "resolution.json missing"; exit 1; } python3 - "$RES" <<'PY' import json, sys diff --git a/tests/e2e/208_private_libc_stays_in_the_binary.sh b/tests/e2e/208_private_libc_stays_in_the_binary.sh index f86b8e12..189617cc 100755 --- a/tests/e2e/208_private_libc_stays_in_the_binary.sh +++ b/tests/e2e/208_private_libc_stays_in_the_binary.sh @@ -97,7 +97,7 @@ EOF cd app "$MCPP" build > build.log 2>&1 || { cat build.log; echo "build failed"; exit 1; } -NINJA=$(find target -name build.ninja | head -1) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) GLIBC_LIB=$(grep -oE '/[^ ",]*/xim-x-glibc/[0-9.]+/lib(64)?' "$NINJA" | head -1 || true) if [[ -z "$GLIBC_LIB" ]]; then echo "SKIP: this toolchain has no private glibc payload" diff --git a/tests/e2e/217_module_extensions.sh b/tests/e2e/217_module_extensions.sh index 998d0d9c..a8b86659 100755 --- a/tests/e2e/217_module_extensions.sh +++ b/tests/e2e/217_module_extensions.sh @@ -59,7 +59,7 @@ cd nocfg "$MCPP" build > b.log 2>&1 || { cat b.log; echo "FAIL: default build broke"; exit 1; } grep -q 'sources \[src/\*\*/\*\.{cppm,cpp,cc,c,S,s,asm}\]' b.log || { cat b.log; echo "FAIL: default source glob changed"; exit 1; } -nj="$(find target -name build.ninja | head -1)" +nj="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" for stray in b.ccm c.cxxm d.ixx; do grep -q "$stray" "$nj" && { echo "FAIL: $stray compiled without opt-in"; exit 1; } done @@ -90,7 +90,7 @@ $2" [[ "$out" == *"$want"* ]] || { echo "FAIL[$1]: missing '$want' in: $out"; exit 1; } done - local nj; nj="$(find target -name build.ninja | head -1)" + local nj; nj="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" # Every module interface must use the MODULE rule. Asserting on the rule — # not on the BMI path — is the point: the BMI output line is emitted from diff --git a/tests/e2e/218_module_extensions_graph_shape.sh b/tests/e2e/218_module_extensions_graph_shape.sh index 2be95cfb..eb155200 100755 --- a/tests/e2e/218_module_extensions_graph_shape.sh +++ b/tests/e2e/218_module_extensions_graph_shape.sh @@ -53,7 +53,7 @@ printf 'import std;\nimport gshape.face;\nint main(){ std::println("{}", face()) # Ask mcpp for the fingerprint instead of inferring it from the build tree. # -# The obvious `find target -name build.ninja | head -1` is WRONG here and was +# The obvious `find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1` is WRONG here and was # flaky in exactly the way this test is meant to catch: after part 2 there are # TWO output dirs, and `head -1` picks whichever `find` happened to walk first. # It passed standalone and failed inside the suite. Assert on the value, not on @@ -71,7 +71,7 @@ FP_BEFORE="$(fingerprint)" # its time when the text is unchanged), and the fast path replays it untouched. # A `Compiling` line no longer tells the two apart: it states that a package's # steps ran, on either path (build progress design 2026-09-29). -planned_since() { find target -name build.ninja -newer "$1" 2>/dev/null | grep -q .; } +planned_since() { find target -name build.ninja -not -path '*/.build-mcpp/*' -newer "$1" 2>/dev/null | grep -q .; } touch .before-b1; sleep 1 "$MCPP" build > b1.log 2>&1 || { cat b1.log; echo "FAIL: no-change build"; exit 1; } planned_since .before-b1 && { diff --git a/tests/e2e/231_jobs_option.sh b/tests/e2e/231_jobs_option.sh index 0fb8b5be..46983216 100755 --- a/tests/e2e/231_jobs_option.sh +++ b/tests/e2e/231_jobs_option.sh @@ -119,7 +119,7 @@ echo "toolchain override OK" # on the GRAPH's own declaration and on the edges it emits — not on timing, # which on CI measures the runner's mood. "$MCPP" build --release > /dev/null 2>&1 -ninja_file=$(find target -name build.ninja | head -1) +ninja_file=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) grep -q 'schedule=none' "$ninja_file" \ || { echo "default should not use the split schedule:"; head -2 "$ninja_file"; exit 1; } @@ -135,7 +135,7 @@ cp mcpp.toml "$TMP/mcpp.toml.bak" printf '\n[build]\nbmi_schedule = "on"\n' >> mcpp.toml "$MCPP" build --release > "$TMP/sched-manifest.txt" 2>&1 \ || { echo "bmi_schedule = \"on\" failed to build:"; cat "$TMP/sched-manifest.txt"; exit 1; } -ninja_file=$(find target -name build.ninja | head -1) +ninja_file=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) grep -qE 'schedule=(detach-codegen|two-phase)' "$ninja_file" \ || { echo "[build] bmi_schedule = \"on\" did not select a split schedule:" head -2 "$ninja_file"; exit 1; } @@ -145,7 +145,7 @@ echo "manifest key bmi_schedule OK" rm -rf target MCPP_BMI_SCHEDULE=on "$MCPP" build --release > "$TMP/sched.txt" 2>&1 \ || { echo "schedule=on failed to build:"; cat "$TMP/sched.txt"; exit 1; } -ninja_file=$(find target -name build.ninja | head -1) +ninja_file=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) grep -q 'schedule=detach-codegen\|schedule=two-phase\|schedule=none' "$ninja_file" \ || { echo "graph does not declare its schedule:"; head -2 "$ninja_file"; exit 1; } diff --git a/tests/e2e/239_msvc_managed_toolset.sh b/tests/e2e/239_msvc_managed_toolset.sh index 3beb769e..a922b095 100755 --- a/tests/e2e/239_msvc_managed_toolset.sh +++ b/tests/e2e/239_msvc_managed_toolset.sh @@ -164,7 +164,7 @@ cat >> mcpp.toml <&1) || { echo "FAIL: clang build against the package: $out"; exit 1; } -res=$(find target -name resolution.json | head -1) +res=$(find target -name resolution.json -not -path '*/.build-mcpp/*' | head -1) python3 - "$res" "$TOOLSET" <<'PYEOF' import json, sys d = json.load(open(sys.argv[1])) diff --git a/tests/e2e/241_windows_ucrt_runtime_identity.sh b/tests/e2e/241_windows_ucrt_runtime_identity.sh index c7e106c5..9bc9f29f 100755 --- a/tests/e2e/241_windows_ucrt_runtime_identity.sh +++ b/tests/e2e/241_windows_ucrt_runtime_identity.sh @@ -36,7 +36,7 @@ EOF "$MCPP" build > build.log 2>&1 || { cat build.log; exit 1; } -RES="$(find target -name resolution.json | head -1)" +RES="$(find target -name resolution.json -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$RES" ]] || { echo "FAIL: no resolution.json"; exit 1; } python3 - "$RES" <<'PY' diff --git a/tests/e2e/247_bare_triple_conditional_native.sh b/tests/e2e/247_bare_triple_conditional_native.sh index 11b141fb..938f3d09 100755 --- a/tests/e2e/247_bare_triple_conditional_native.sh +++ b/tests/e2e/247_bare_triple_conditional_native.sh @@ -67,7 +67,7 @@ EOF count() { # $1 = macro name # The NAME, not `-D`: the prefix is dialect-specific (`/D` on MSVC), # and what this test is about is whether the section applied at all. - local nj; nj="$(find target -name build.ninja | head -1)" + local nj; nj="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -c -- "$1" "$nj" || true } diff --git a/tests/e2e/257_shared_library_pe.sh b/tests/e2e/257_shared_library_pe.sh index a138e561..71ec74e0 100755 --- a/tests/e2e/257_shared_library_pe.sh +++ b/tests/e2e/257_shared_library_pe.sh @@ -74,7 +74,7 @@ imp="$(find target -name 'libmathkit.dll.a' | head -1)" # targeting MSVC. Asserted here rather than only on Windows because this runs on # every Linux CI pass through mingw-cross, and the flag is equally meaningless # for a PE target whichever compiler emits it. -nj_pic="$(find target -name build.ninja | head -1)" +nj_pic="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -q '\-fPIC' "$nj_pic" && { grep -n 'fPIC' "$nj_pic" | head -3 echo "FAIL: -fPIC on a PE target. It means nothing here, and clang targeting" @@ -84,7 +84,7 @@ grep -q '\-fPIC' "$nj_pic" && { # It is a declared output of the link edge, not a side effect ninja knows nothing # about — otherwise the consumer that links it has no producer and ninja stops # with 'no known rule to make it'. -nj="$(find target -name build.ninja | head -1)" +nj="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE "^build bin/libmathkit\.dll \| bin/libmathkit\.dll\.a : " "$nj" || { grep -n 'libmathkit' "$nj" echo "FAIL: the import library is not an implicit output of the link edge" diff --git a/tests/e2e/258_shared_library_msvc_auto_def.sh b/tests/e2e/258_shared_library_msvc_auto_def.sh index d4892151..f00e651c 100755 --- a/tests/e2e/258_shared_library_msvc_auto_def.sh +++ b/tests/e2e/258_shared_library_msvc_auto_def.sh @@ -59,7 +59,7 @@ cd mathkit exit 1; } # ── the .def was generated, and it is a build-graph node ──────────────── -nj="$(find target -name build.ninja | head -1)" +nj="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE "^build .*mathkit\.def : coff_def " "$nj" || { grep -n 'def' "$nj" | head echo "FAIL: no coff_def edge. The .def has to be produced from the same" diff --git a/tests/e2e/260_module_extension_is_configuration.sh b/tests/e2e/260_module_extension_is_configuration.sh index 3f4556f5..cc2a1683 100755 --- a/tests/e2e/260_module_extension_is_configuration.sh +++ b/tests/e2e/260_module_extension_is_configuration.sh @@ -66,7 +66,7 @@ manifest '' rm -rf target if "$MCPP" build > undeclared.log 2>&1; then echo "FAIL: an undeclared '.ixx' was accepted." - nj="$(find target -name build.ninja | head -1)" + nj="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -n 'mathkit' "$nj" | head echo " Check whether its object is in the link edge — if it is not," echo " this 'success' produces an undefined reference at link time." diff --git a/tests/e2e/26_c_language_support.sh b/tests/e2e/26_c_language_support.sh index 8668496a..5cc23cfd 100755 --- a/tests/e2e/26_c_language_support.sh +++ b/tests/e2e/26_c_language_support.sh @@ -59,7 +59,7 @@ EOF "$MCPP" build > build.log 2>&1 || { cat build.log; echo "build failed"; exit 1; } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "no build.ninja generated"; exit 1; } # c_object rule must be present alongside cxx_object. diff --git a/tests/e2e/295_naming_the_host_target_changes_nothing.sh b/tests/e2e/295_naming_the_host_target_changes_nothing.sh index a9f858d3..ea3667ef 100755 --- a/tests/e2e/295_naming_the_host_target_changes_nothing.sh +++ b/tests/e2e/295_naming_the_host_target_changes_nothing.sh @@ -81,7 +81,7 @@ line_of() { # channel, extra args… → that line of build.ninja, or nothing local channel="$1"; shift rm -rf target "$MCPP" build "$@" >/dev/null 2>&1 || true - local f; f="$(find target -name build.ninja 2>/dev/null | head -1)" + local f; f="$(find target -name build.ninja -not -path '*/.build-mcpp/*' 2>/dev/null | head -1)" # AN EXPLICIT `return 0`, BECAUSE THIS FUNCTION IS ALLOWED TO FIND # NOTHING. The first draft ended on `[ -n "$f" ] && grep …`, whose exit # status under `set -e` is the test's — so the one case this test exists to diff --git a/tests/e2e/296_what_the_report_names_is_what_the_link_line_uses.sh b/tests/e2e/296_what_the_report_names_is_what_the_link_line_uses.sh index 007c0395..bc007ade 100755 --- a/tests/e2e/296_what_the_report_names_is_what_the_link_line_uses.sh +++ b/tests/e2e/296_what_the_report_names_is_what_the_link_line_uses.sh @@ -37,7 +37,7 @@ report=""; ldflags="" build_and_read() { # extra args… → sets $report and $ldflags rm -rf target report="$("$MCPP" build "$@" 2>&1 || true)" - local f; f="$(find target -name build.ninja 2>/dev/null | head -1)" + local f; f="$(find target -name build.ninja -not -path '*/.build-mcpp/*' 2>/dev/null | head -1)" ldflags="" if [ -n "$f" ]; then ldflags="$(grep -m1 '^ldflags' "$f" || true)" diff --git a/tests/e2e/307_symbol_provision.sh b/tests/e2e/307_symbol_provision.sh index 11ffbc20..4b2ae90a 100755 --- a/tests/e2e/307_symbol_provision.sh +++ b/tests/e2e/307_symbol_provision.sh @@ -98,7 +98,7 @@ cd app # ── half 1: the legitimate arrangement must stay silent ──────────────────── "$MCPP" build > silent.log 2>&1 || { cat silent.log; exit 1; } -res="$(find target -name resolution.json | head -1)" +res="$(find target -name resolution.json -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$res" ]] || { echo "FAIL: no resolution.json"; exit 1; } status="$(verdict "$res" status)" @@ -168,7 +168,7 @@ EOF # keyed off the directory that contains the conflicting verdict. found=0 conflict_dir="" -for r in $(find target -name resolution.json); do +for r in $(find target -name resolution.json -not -path '*/.build-mcpp/*'); do if [[ "$(verdict "$r" status)" == "conflict" ]]; then conflict_dir="$(dirname "$r")" "$PYTHON" - "$r" <<'PY' || exit 1 diff --git a/tests/e2e/315_blocking_check_gates_compilation.sh b/tests/e2e/315_blocking_check_gates_compilation.sh index 1e9b5ac9..722b54ce 100755 --- a/tests/e2e/315_blocking_check_gates_compilation.sh +++ b/tests/e2e/315_blocking_check_gates_compilation.sh @@ -73,7 +73,7 @@ if find target -name 'main*.o' | grep -q .; then fi # The edge that makes it a property rather than a race lost by the compiler. -nj=$(find target -name build.ninja | head -1) +nj=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [[ -n "$nj" ]] || { echo "FAIL: no build.ninja"; exit 1; } grep -qE '^build mcpp-actions-app : phony.*gate\.stamp' "$nj" || { echo "FAIL: the blocking check's stamp is not in the package's action phony" @@ -97,7 +97,7 @@ if ! find target -name 'main*.o' | grep -q .; then echo " not distinguishing anything, it is just always on" exit 1 fi -nj2=$(find target -name build.ninja | head -1) +nj2=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) if grep -qE '^build mcpp-actions-app : phony.*gate\.stamp' "$nj2"; then echo "FAIL: a NON-blocking check joined the ordering phony"; exit 1 fi diff --git a/tests/e2e/323_post_link_record_durability.sh b/tests/e2e/323_post_link_record_durability.sh index 9a1062d7..189b5e9f 100755 --- a/tests/e2e/323_post_link_record_durability.sh +++ b/tests/e2e/323_post_link_record_durability.sh @@ -31,7 +31,7 @@ printf 'int main(){ return 0; }\n' > src/main.cpp for t in one two; do printf 'int main(){ return 0; }\n' > "tests/t_$t.cpp"; done "$MCPP" build > b1.log 2>&1 || { cat b1.log; exit 1; } -OUT=$(dirname "$(find target -name build.ninja | head -1)") +OUT=$(dirname "$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)") [ -n "$OUT" ] || { echo "FAIL: no build.ninja"; exit 1; } REC="$OUT/.mcpp-runtime-verdicts.json" diff --git a/tests/e2e/324_fast_path_sees_path_deps.sh b/tests/e2e/324_fast_path_sees_path_deps.sh index 9a6d8d74..f65ef41c 100755 --- a/tests/e2e/324_fast_path_sees_path_deps.sh +++ b/tests/e2e/324_fast_path_sees_path_deps.sh @@ -39,7 +39,7 @@ cd app # its time when the text is unchanged), and the fast path replays it untouched. # A `Compiling` line no longer tells the two apart: it states that a package's # steps ran, on either path (build progress design 2026-09-29). -planned_since() { find target -name build.ninja -newer "$1" 2>/dev/null | grep -q .; } +planned_since() { find target -name build.ninja -not -path '*/.build-mcpp/*' -newer "$1" 2>/dev/null | grep -q .; } # on a path it never took. touch .before-warm; sleep 1 "$MCPP" build > warm.log 2>&1 || { cat warm.log; exit 1; } diff --git a/tests/e2e/40_llvm_bmi_cache.sh b/tests/e2e/40_llvm_bmi_cache.sh index 8176dfa4..94361a06 100755 --- a/tests/e2e/40_llvm_bmi_cache.sh +++ b/tests/e2e/40_llvm_bmi_cache.sh @@ -80,7 +80,7 @@ echo "$out2" | grep -qE "Cached mcpplibs\.cmdline v[0-9.]+ \([0-9]+ units?\)" || # ...and reuse must mean "not recompiled". The status line alone used to be # printed while ninja rebuilt every unit behind it, so assert on the graph. -NINJA="$(find target -name build.ninja | head -1)" +NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$NINJA" ]] || { echo "FAIL: no build.ninja"; exit 1; } if grep -qE ': (cxx_module|cxx_object|cxx_scan) .*mcpplibs' "$NINJA"; then echo "FAIL: cached dependency still has compile edges" diff --git a/tests/e2e/48_build_error_output.sh b/tests/e2e/48_build_error_output.sh index cbe4b36d..8be2a9a3 100644 --- a/tests/e2e/48_build_error_output.sh +++ b/tests/e2e/48_build_error_output.sh @@ -81,7 +81,7 @@ do fi done -build_ninja=$(find target -name build.ninja | head -1) +build_ninja=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [[ -n "$build_ninja" ]] || { echo "missing generated build.ninja" echo "$out" diff --git a/tests/e2e/49_bmi_cache_nested_custom_index.sh b/tests/e2e/49_bmi_cache_nested_custom_index.sh index d4164cf1..1757fbf8 100644 --- a/tests/e2e/49_bmi_cache_nested_custom_index.sh +++ b/tests/e2e/49_bmi_cache_nested_custom_index.sh @@ -131,7 +131,7 @@ grep -qE "Cached local-dev\.collision-lib v1\.0\.0 \((index [^,)]+, )?[0-9]+ uni # A reused dependency must not be recompiled — that was the whole defect. Its # outputs are declared by stage_file edges now, so there is no compile edge for # its sources at all. -NINJA="$(find target -name build.ninja | head -1)" +NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$NINJA" ]] || { echo "FAIL: no build.ninja"; exit 1; } if grep -qE ': (cxx_module|cxx_object) .*collision-lib' "$NINJA"; then echo "FAIL: cached dependency still has compile edges" diff --git a/tests/e2e/50_package_owned_build_flags.sh b/tests/e2e/50_package_owned_build_flags.sh index e3791aa0..c21d3035 100755 --- a/tests/e2e/50_package_owned_build_flags.sh +++ b/tests/e2e/50_package_owned_build_flags.sh @@ -81,7 +81,7 @@ cd app exit 1 } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "no build.ninja generated" exit 1 diff --git a/tests/e2e/54_package_owned_ldflags.sh b/tests/e2e/54_package_owned_ldflags.sh index 333b9054..14201996 100644 --- a/tests/e2e/54_package_owned_ldflags.sh +++ b/tests/e2e/54_package_owned_ldflags.sh @@ -66,7 +66,7 @@ cd app exit 1 } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "no build.ninja generated" exit 1 diff --git a/tests/e2e/59_cpp_standard_config.sh b/tests/e2e/59_cpp_standard_config.sh index 8345af61..1ac2834e 100644 --- a/tests/e2e/59_cpp_standard_config.sh +++ b/tests/e2e/59_cpp_standard_config.sh @@ -46,7 +46,7 @@ out=$("$binary") exit 1 } -build_ninja="$(find target -name build.ninja | head -1)" +build_ninja="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$build_ninja" ]] || { echo "FAIL: build.ninja missing"; exit 1; } grep -qE '^cxxflags = -std=c\+\+26' "$build_ninja" || { @@ -93,7 +93,7 @@ MCPP_SCANNER=p1689 "$MCPP" build --no-cache > "$TMP/build-p1689.log" 2>&1 || { echo "FAIL: C++26 P1689 scanner build failed" exit 1 } -p1689_ninja="$(find target -name build.ninja | head -1)" +p1689_ninja="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -qE '^cxxflags = -std=c\+\+26' "$p1689_ninja" || { echo "FAIL: P1689 build.ninja missing C++26 standard flag" cat "$p1689_ninja" diff --git a/tests/e2e/617_build_program_knows_the_cxx_stdlib.sh b/tests/e2e/617_build_program_knows_the_cxx_stdlib.sh index 82a968a9..6ca78880 100755 --- a/tests/e2e/617_build_program_knows_the_cxx_stdlib.sh +++ b/tests/e2e/617_build_program_knows_the_cxx_stdlib.sh @@ -72,7 +72,7 @@ echo "build program read: compiler='$compiler' stdlib='$stdlib'" [ -n "$stdlib" ] || { echo "FAIL: mcpp::cxx_stdlib() is empty"; exit 1; } # 1. against resolution.json, which the engine writes from its own resolution. -res=$(find target -name resolution.json | head -1) +res=$(find target -name resolution.json -not -path '*/.build-mcpp/*' | head -1) [ -n "$res" ] || { echo "FAIL: no resolution.json under target/"; exit 1; } engine=$(sed -n 's/.*"stdlib"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$res" | head -1) echo "resolution.json ($res) says: '$engine'" diff --git a/tests/e2e/62_runtime_library_dirs.sh b/tests/e2e/62_runtime_library_dirs.sh index 444fffc6..1ba0725b 100644 --- a/tests/e2e/62_runtime_library_dirs.sh +++ b/tests/e2e/62_runtime_library_dirs.sh @@ -87,7 +87,7 @@ cd app # A launch-time search directory belongs in RUNPATH, not in the linker's -L # namespace. The plugin is dlopen-only and has no link-time import; leaking # this directory into -L can silently select an unrelated same-name library. -NINJA=$(find target -name build.ninja | head -1) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) RUNTIME_DIR=$(realpath runtime) grep -F -- "-Wl,-rpath,$RUNTIME_DIR" "$NINJA" >/dev/null || { echo "FAIL: runtime directory missing from RUNPATH intent" diff --git a/tests/e2e/634_a_private_module_fragment_is_not_a_partition.sh b/tests/e2e/634_a_private_module_fragment_is_not_a_partition.sh index 07d92d1a..b032a2f3 100755 --- a/tests/e2e/634_a_private_module_fragment_is_not_a_partition.sh +++ b/tests/e2e/634_a_private_module_fragment_is_not_a_partition.sh @@ -76,7 +76,7 @@ EOF # THE POSITIVE LEG. An absence alone would also be satisfied by an mcpp that # failed earlier for an unrelated reason, so assert the artifact a correct # scan produces: a graph, with a module edge for this file. - graph=$(find target -name build.ninja | head -1) + graph=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [ -n "$graph" ] || { echo "FAIL [$spelling]: no build graph was written" cat build.log; exit 1; } grep -q "cxx_module .*pf\.cppm" "$graph" || { diff --git a/tests/e2e/636_a_module_extension_file_need_not_provide_a_module.sh b/tests/e2e/636_a_module_extension_file_need_not_provide_a_module.sh index abe0b489..ec600778 100755 --- a/tests/e2e/636_a_module_extension_file_need_not_provide_a_module.sh +++ b/tests/e2e/636_a_module_extension_file_need_not_provide_a_module.sh @@ -55,7 +55,7 @@ bin=$(find target -type f -name implunit -perm -u+x | head -1) [ -n "$bin" ] || { echo "FAIL: no binary"; exit 1; } "$bin" || { echo "FAIL: the program did not return 5"; exit 1; } -graph=$(find target -name build.ninja | head -1) +graph=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [ -n "$graph" ] || { echo "FAIL: no build graph"; exit 1; } # EVERY module edge states its language, AND THE DENOMINATOR IS ASSERTED. diff --git a/tests/e2e/638_pack_format_dispatch.sh b/tests/e2e/638_pack_format_dispatch.sh index aa0168ef..9dc346bb 100755 --- a/tests/e2e/638_pack_format_dispatch.sh +++ b/tests/e2e/638_pack_format_dispatch.sh @@ -112,7 +112,7 @@ EOF MCPP="${MCPP:-mcpp}" graph_dist() { sed -n '2p' "$1" | sed 's/.*;dist=//;s/;.*//'; } -find_graph() { find target -name build.ninja | head -1; } +find_graph() { find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1; } # ── 1. a plain build declares, and submits nothing ───────────────────────── "$MCPP" build --release > b1.log 2>&1 || { cat b1.log; echo "FAIL: plain build failed"; exit 1; } diff --git a/tests/e2e/646_runtime_deploy_places_files_in_a_directory.sh b/tests/e2e/646_runtime_deploy_places_files_in_a_directory.sh index 547bc0aa..e7e38c08 100755 --- a/tests/e2e/646_runtime_deploy_places_files_in_a_directory.sh +++ b/tests/e2e/646_runtime_deploy_places_files_in_a_directory.sh @@ -124,7 +124,7 @@ write_manifest '{ from = "assets/icd2/lvp_icd.json", to = "vulkan/icd.d" }' "$MCPP" build > merge.log 2>&1 || fail "identical bytes for one destination were refused" merge.log grep -q 'library_path' "$bin/vulkan/icd.d/lvp_icd.json" 2>/dev/null \ || fail "the merged destination does not carry the shared bytes" merge.log -G=$(find target -name build.ninja | head -1) +G=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) # One placement is its own `stage_file` edge; two or more (a Windows program # also places its C++ runtime DLLs) are one `stage_list` edge reading # placements.list (#734 E4). Either way the destination has both sources. diff --git a/tests/e2e/647_abi_threads_is_one_switch_for_the_graph.sh b/tests/e2e/647_abi_threads_is_one_switch_for_the_graph.sh index fa2e59cc..87f4de70 100755 --- a/tests/e2e/647_abi_threads_is_one_switch_for_the_graph.sh +++ b/tests/e2e/647_abi_threads_is_one_switch_for_the_graph.sh @@ -67,7 +67,7 @@ $2 TOML } -newest_ninja() { ls -t $(find target -name build.ninja) | head -1; } +newest_ninja() { ls -t $(find target -name build.ninja -not -path '*/.build-mcpp/*') | head -1; } pthread_in_dep_command() { # the compile command of mtdep.cppm carries -pthread local cdb cdb=$(dirname "$(newest_ninja)")/compile_commands.json diff --git a/tests/e2e/650_wasm_row_names_its_launcher.sh b/tests/e2e/650_wasm_row_names_its_launcher.sh index f4d4b6e6..4b22add5 100755 --- a/tests/e2e/650_wasm_row_names_its_launcher.sh +++ b/tests/e2e/650_wasm_row_names_its_launcher.sh @@ -145,7 +145,7 @@ TOML printf 'extern "C" int mcpp_factory() { return 42; }\n' > src/lib.cpp "$MCPP" build --target wasm32-emscripten > build.log 2>&1 \ || fail "a main()-less bin with --no-entry did not link" build.log -ninjafile=$(find target/wasm32-emscripten -name 'build.ninja' | head -1) +ninjafile=$(find target/wasm32-emscripten -name 'build.ninja' -not -path '*/.build-mcpp/*' | head -1) grep -q -- '--no-entry' "$ninjafile" || fail "--no-entry did not reach build.ninja" "$ninjafile" echo " ok: --no-entry reaches the link line as an ordinary ldflag" diff --git a/tests/e2e/651_a_build_program_deploys_what_it_generated.sh b/tests/e2e/651_a_build_program_deploys_what_it_generated.sh index 42d0fce8..9fecef01 100755 --- a/tests/e2e/651_a_build_program_deploys_what_it_generated.sh +++ b/tests/e2e/651_a_build_program_deploys_what_it_generated.sh @@ -99,7 +99,7 @@ int main() { EOF MCPP="${MCPP:-mcpp}" -find_graph() { find target -name build.ninja | head -1; } +find_graph() { find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1; } # ── 1. a plain build deploys the generated file beside the executable ────── # `target///bin/...`, never a bare `bin/` at the project diff --git a/tests/e2e/653_exceptions_is_the_second_abi_member.sh b/tests/e2e/653_exceptions_is_the_second_abi_member.sh index 9bc6b4b4..1f0601bb 100755 --- a/tests/e2e/653_exceptions_is_the_second_abi_member.sh +++ b/tests/e2e/653_exceptions_is_the_second_abi_member.sh @@ -58,7 +58,7 @@ main = "src/main.cpp" TOML printf 'import std;\nint main() { std::println("ok"); return 0; }\n' > src/main.cpp "$MCPP" build > plain-build.log 2>&1 || fail "the plain Linux build failed" plain-build.log -plain_ninja=$(find target -name build.ninja | head -1) +plain_ninja=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) cp "$plain_ninja" "$TMP/plain.ninja" rm -rf target cat >> mcpp.toml <<'TOML' @@ -67,7 +67,7 @@ cat >> mcpp.toml <<'TOML' exceptions = true TOML "$MCPP" build > with-table.log 2>&1 || fail "the Linux build with the table failed" with-table.log -with_ninja=$(find target -name build.ninja | head -1) +with_ninja=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) diff -u "$TMP/plain.ninja" "$with_ninja" > "$TMP/ninja.diff" \ || fail "a Linux build.ninja differs with the emscripten-only abi table present" "$TMP/ninja.diff" echo " ok: build.ninja is byte-identical" @@ -123,7 +123,7 @@ cd "$TMP/app" write_app '' "$MCPP" build --target wasm32-emscripten > build-noexc.log 2>&1 \ || fail "the build without the abi table failed (expected: it succeeds; only the run fails)" build-noexc.log -plain_dir=$(dirname "$(ls -t $(find target/wasm32-emscripten -name build.ninja) | head -1)") +plain_dir=$(dirname "$(ls -t $(find target/wasm32-emscripten -name build.ninja -not -path '*/.build-mcpp/*') | head -1)") if "$MCPP" run --target wasm32-emscripten > run-noexc.log 2>&1; then fail "mcpp run succeeded without -fexceptions -- the throw should have aborted" run-noexc.log fi @@ -138,7 +138,7 @@ echo "== 653/3: with exceptions, the marker prints, the flags reach both compile write_app $'[target.\'cfg(os = "emscripten")\'.abi]\nexceptions = true\n' "$MCPP" build --target wasm32-emscripten > build-exc.log 2>&1 \ || fail "the build with the abi table failed" build-exc.log -exc_dir=$(dirname "$(ls -t $(find target/wasm32-emscripten -name build.ninja) | head -1)") +exc_dir=$(dirname "$(ls -t $(find target/wasm32-emscripten -name build.ninja -not -path '*/.build-mcpp/*') | head -1)") [ "$exc_dir" != "$plain_dir" ] \ || fail "the build with exceptions reused the fingerprint directory of the build without" build-exc.log diff --git a/tests/e2e/654_a_requirement_on_the_target_axis.sh b/tests/e2e/654_a_requirement_on_the_target_axis.sh index fc938a7c..7420a6f2 100755 --- a/tests/e2e/654_a_requirement_on_the_target_axis.sh +++ b/tests/e2e/654_a_requirement_on_the_target_axis.sh @@ -63,7 +63,7 @@ $2 TOML } -newest_ninja() { ls -t $(find target -name build.ninja) | head -1; } +newest_ninja() { ls -t $(find target -name build.ninja -not -path '*/.build-mcpp/*') | head -1; } pthread_in_dep_command() { # the compile command of tgtdep.cppm carries -pthread local cdb cdb=$(dirname "$(newest_ninja)")/compile_commands.json diff --git a/tests/e2e/659_an_action_declaration_has_no_size_limit.sh b/tests/e2e/659_an_action_declaration_has_no_size_limit.sh index 3953155d..5770d742 100755 --- a/tests/e2e/659_an_action_declaration_has_no_size_limit.sh +++ b/tests/e2e/659_an_action_declaration_has_no_size_limit.sh @@ -130,7 +130,7 @@ EOF out="$("$MCPP" run 2>&1 | tail -1)" [[ "$out" == *LARGE_DECLARATION_OK* ]] || { echo "FAIL: the program did not run: '$out'"; exit 1; } -NINJA=$(find target -name build.ninja -print -quit) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' -print -quit) [ -n "$NINJA" ] || { echo "FAIL: no build.ninja"; exit 1; } # THE criterion: the edge in the graph names every declared output and every @@ -167,7 +167,7 @@ cat > src/extra.cpp <<'EOF' int extra() { return 1; } EOF "$MCPP" build > b2.log 2>&1 || { cat b2.log; echo "FAIL: rebuild after an unrelated source failed"; exit 1; } -NINJA=$(find target -name build.ninja -print -quit) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' -print -quit) outs=$(count_in_edge "output-[0-9]*\.txt") [ "$outs" = "$N" ] || { echo "FAIL: after a replay build.ninja names $outs of $N declared outputs"; exit 1; } echo "ok: the replayed record carries all $N outputs" diff --git a/tests/e2e/660_an_action_names_the_engine_through_mcpp_self.sh b/tests/e2e/660_an_action_names_the_engine_through_mcpp_self.sh index efeaf3f8..85d1d169 100755 --- a/tests/e2e/660_an_action_names_the_engine_through_mcpp_self.sh +++ b/tests/e2e/660_an_action_names_the_engine_through_mcpp_self.sh @@ -62,7 +62,7 @@ EOF out="$("$MCPP" run 2>&1 | tail -1)" [[ "$out" == *SELF_COPY_OK* ]] || { echo "FAIL: the program did not run: '$out'"; exit 1; } -NINJA=$(find target -name build.ninja -print -quit) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' -print -quit) [ -n "$NINJA" ] || { echo "FAIL: no build.ninja"; exit 1; } # The token is substituted, never emitted literally. diff --git a/tests/e2e/66_runtime_provides.sh b/tests/e2e/66_runtime_provides.sh index c9000977..6d84da18 100755 --- a/tests/e2e/66_runtime_provides.sh +++ b/tests/e2e/66_runtime_provides.sh @@ -52,7 +52,7 @@ EOF cd app "$MCPP" build > build.log 2>&1 || { cat build.log; echo "build failed"; exit 1; } -RES=$(find target -name resolution.json | head -1) +RES=$(find target -name resolution.json -not -path '*/.build-mcpp/*' | head -1) [[ -n "$RES" ]] || { echo "no resolution.json produced"; exit 1; } # The only provider entry must be the provides-declarer, with canonical ID. diff --git a/tests/e2e/682_why_deps_prints_the_resolved_graph.sh b/tests/e2e/682_why_deps_prints_the_resolved_graph.sh index 0ab895a5..f6f1c147 100755 --- a/tests/e2e/682_why_deps_prints_the_resolved_graph.sh +++ b/tests/e2e/682_why_deps_prints_the_resolved_graph.sh @@ -34,7 +34,7 @@ grep -qF "requested by mcpplibs.app@0.1.0 as 'demo.fw' in [dependencies]" why.lo grep -q "linked static (default)" why.log || fail "A: the link form is not listed" why.log # ── B ────────────────────────────────────────────────────────────────────── -json=$(find target -name resolution.json | head -1) +json=$(find target -name resolution.json -not -path '*/.build-mcpp/*' | head -1) python3 - "$json" <<'PY' || fail "B: record shape" "$json" import json, sys doc = json.load(open(sys.argv[1])) diff --git a/tests/e2e/703_the_llvm_row_on_the_msvc_abi_records_its_static_crt.sh b/tests/e2e/703_the_llvm_row_on_the_msvc_abi_records_its_static_crt.sh index d1235ced..ab9ff4e7 100755 --- a/tests/e2e/703_the_llvm_row_on_the_msvc_abi_records_its_static_crt.sh +++ b/tests/e2e/703_the_llvm_row_on_the_msvc_abi_records_its_static_crt.sh @@ -47,7 +47,7 @@ TOML } record_of() { # prints the distributable contract from the one resolution.json - tr -d ' \n\r' < "$(find target -name resolution.json | head -1)" \ + tr -d ' \n\r' < "$(find target -name resolution.json -not -path '*/.build-mcpp/*' | head -1)" \ | grep -o '"distributable":"[a-z-]*"' | head -1 | cut -d'"' -f4 } diff --git a/tests/e2e/70_per_target_flags.sh b/tests/e2e/70_per_target_flags.sh index 35e9b8df..bfe08f8e 100755 --- a/tests/e2e/70_per_target_flags.sh +++ b/tests/e2e/70_per_target_flags.sh @@ -61,7 +61,7 @@ EOF "$MCPP" build > build.log 2>&1 || { cat build.log; echo "build failed"; exit 1; } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "no build.ninja"; exit 1; } grep -q -- "-DROLE_SERVER=1" "$ninja_file" || { echo "server define missing from ninja"; exit 1; } grep -q -- "-DROLE_CLIENT=2" "$ninja_file" || { echo "client define missing from ninja"; exit 1; } diff --git a/tests/e2e/712_a_tool_package_contributes_nothing_to_its_consumer.sh b/tests/e2e/712_a_tool_package_contributes_nothing_to_its_consumer.sh index 8a59f1aa..8095de20 100755 --- a/tests/e2e/712_a_tool_package_contributes_nothing_to_its_consumer.sh +++ b/tests/e2e/712_a_tool_package_contributes_nothing_to_its_consumer.sh @@ -137,7 +137,7 @@ int main() { std::printf("tool %d\n", z_only_in_the_tool()); return 0; } EOF cd "$TMP/s2/app" "$MCPP" build --features windows-installer > b2.log 2>&1 || fail "the tool-with-a-library build failed" b2.log -ninja_file=$(find target -name build.ninja | head -1) +ninja_file=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) grep '^build bin/app' "$ninja_file" | grep -q 'spike_z' \ && fail "the tool's own library was linked into the application" "$ninja_file" grep -q 'spike_fw-installer' "$ninja_file" \ diff --git a/tests/e2e/714_features_names_a_dependency_feature.sh b/tests/e2e/714_features_names_a_dependency_feature.sh index 4a9df96e..b9bf1e75 100755 --- a/tests/e2e/714_features_names_a_dependency_feature.sh +++ b/tests/e2e/714_features_names_a_dependency_feature.sh @@ -88,7 +88,7 @@ cd "$TMP/app" # ── 1. no [features] table: the dependency's feature opens, no macro ───── "$MCPP" build --features spike.fw/installer > b1.log 2>&1 || fail "the dependency feature build failed" b1.log grep -q 'DEPBIN tool=\[[^]]' b1.log || fail "--features spike.fw/installer did not open the feature" b1.log -ninja_file=$(find target -name build.ninja | head -1) +ninja_file=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) grep -q 'MCPP_FEATURE_SPIKE_FW_INSTALLER' "$ninja_file" \ && fail "the dependency token became a macro of the root" "$ninja_file" diff --git a/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh b/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh index 30d8bb16..8e3b0bb3 100644 --- a/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh +++ b/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh @@ -91,7 +91,7 @@ cat > words.json <<'EOF' ["-DV_ESC=\"esc.h\"", "-DV_MID=mid", "-DV_SQ=sq", "-DV_SP=a b", "-DV_DOL=a$b", "-O0", "-DV_A=1", "-DV_B=long long", "-DV_DEF=\"def\"", "-DV_DSP=a b"] EOF -cdb=$(find target -name compile_commands.json | head -1) +cdb=$(find target -name compile_commands.json -not -path '*/.build-mcpp/*' | head -1) [ -n "$cdb" ] || cdb=compile_commands.json [ -f "$cdb" ] || fail "B: no compile_commands.json" "$MCPP" emit build-database --format json > db.json 2> db.err || fail "C: emit failed" db.err diff --git a/tests/e2e/738_a_graph_supplied_target_closes_the_hosts_own_search.sh b/tests/e2e/738_a_graph_supplied_target_closes_the_hosts_own_search.sh index 7ec4cf7b..fa9111e5 100755 --- a/tests/e2e/738_a_graph_supplied_target_closes_the_hosts_own_search.sh +++ b/tests/e2e/738_a_graph_supplied_target_closes_the_hosts_own_search.sh @@ -117,7 +117,7 @@ fi printf '%s\n' "$out" > build.log cdb="compile_commands.json" -[ -f "$cdb" ] || cdb="$(find target -name compile_commands.json | head -1)" +[ -f "$cdb" ] || cdb="$(find target -name compile_commands.json -not -path '*/.build-mcpp/*' | head -1)" [ -n "$cdb" ] && [ -f "$cdb" ] || fail "no compile_commands.json produced" home="${MCPP_HOME:-$HOME/.mcpp}" diff --git a/tests/e2e/73_issue131_per_target_cxxflag.sh b/tests/e2e/73_issue131_per_target_cxxflag.sh index 3a07cae0..2dca15c7 100755 --- a/tests/e2e/73_issue131_per_target_cxxflag.sh +++ b/tests/e2e/73_issue131_per_target_cxxflag.sh @@ -57,7 +57,7 @@ EOF # the shared unit (shared check) — both are #error guards above. "$MCPP" build > build.log 2>&1 || { cat build.log; echo "build failed"; exit 1; } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "no build.ninja"; exit 1; } grep -q -- "-fno-exceptions" "$ninja_file" || { echo "per-target cxxflag missing from build.ninja"; exit 1; } diff --git a/tests/e2e/760_msvc_toolset_is_chosen_once.sh b/tests/e2e/760_msvc_toolset_is_chosen_once.sh index ead68f77..695d1332 100755 --- a/tests/e2e/760_msvc_toolset_is_chosen_once.sh +++ b/tests/e2e/760_msvc_toolset_is_chosen_once.sh @@ -38,7 +38,7 @@ echo "runner: $VSROOT, default toolset $EXPECTED" # No developer prompt: the default selection must not depend on one. unset VCToolsInstallDir VSINSTALLDIR VCINSTALLDIR -resolution() { find target -name resolution.json | head -1; } +resolution() { find target -name resolution.json -not -path '*/.build-mcpp/*' | head -1; } # ── 1. The clang row, no sysroot declared: the machine's default ────────── "$MCPP" new chosen > /dev/null @@ -64,7 +64,7 @@ print("OK:", t["version"], t["origin"], sdk["version"]) PY # The driver is TOLD the toolset, with the same directory the record names. -NINJA="$(find target -name build.ninja | head -1)" +NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" grep -q -- "-Xmicrosoft-visualc-tools-root" "$NINJA" \ || { echo "FAIL: build.ninja does not pass the toolset to clang"; exit 1; } grep -q -- "-Xmicrosoft-windows-sdk-version" "$NINJA" \ @@ -99,7 +99,7 @@ grep -q "sysroot msvc@$EXPECTED → MSVC $EXPECTED (system" pinned.log \ || { echo "FAIL: pinned toolset not taken from the machine:"; cat pinned.log; exit 1; } # The declaration the environment makes is ignored for the version too: the # pinned build says the same version as the default one. -PINNED_NINJA="$(find target -name build.ninja | xargs ls -t | head -1)" +PINNED_NINJA="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | xargs ls -t | head -1)" [[ "$(grep -o -- '-fms-compatibility-version=[0-9.]*' "$PINNED_NINJA" | sort -u)" == "$MSVER" ]] \ || { echo "FAIL: the pinned build says another MSVC compiler version than $MSVER"; exit 1; } diff --git a/tests/e2e/779_a_build_program_declares_a_runtime_search_dir.sh b/tests/e2e/779_a_build_program_declares_a_runtime_search_dir.sh index e29a93fd..24630473 100755 --- a/tests/e2e/779_a_build_program_declares_a_runtime_search_dir.sh +++ b/tests/e2e/779_a_build_program_declares_a_runtime_search_dir.sh @@ -97,7 +97,7 @@ EOF MCPP="${MCPP:-mcpp}" cd app -find_graph() { find target -name build.ninja | head -1; } +find_graph() { find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1; } # ── 1. a plain build gets the RUNPATH treatment, never -L ────────────────── "$MCPP" build > b1.log 2>&1 || fail "build failed" b1.log diff --git a/tests/e2e/781_deleted_root_cdb_is_restored.sh b/tests/e2e/781_deleted_root_cdb_is_restored.sh index f5cc9536..a653d2b2 100755 --- a/tests/e2e/781_deleted_root_cdb_is_restored.sh +++ b/tests/e2e/781_deleted_root_cdb_is_restored.sh @@ -19,7 +19,7 @@ cd app "$MCPP" build > build1.log 2>&1 || { cat build1.log; echo "FAIL: the first build failed"; exit 1; } [[ -s compile_commands.json ]] || { echo "FAIL: no compile_commands.json after the first build"; exit 1; } -config_cdb=$(find target -name compile_commands.json | head -1) +config_cdb=$(find target -name compile_commands.json -not -path '*/.build-mcpp/*' | head -1) [[ -n "$config_cdb" ]] || { echo "FAIL: no configuration database under target/"; exit 1; } before=$(cat "$config_cdb") @@ -27,7 +27,7 @@ before=$(cat "$config_cdb") # its time when the text is unchanged), and the fast path replays it untouched. # A `Compiling` line no longer tells the two apart: it states that a package's # steps ran, on either path (build progress design 2026-09-29). -planned_since() { find target -name build.ninja -newer "$1" 2>/dev/null | grep -q .; } +planned_since() { find target -name build.ninja -not -path '*/.build-mcpp/*' -newer "$1" 2>/dev/null | grep -q .; } rm compile_commands.json touch .before-build2; sleep 1 diff --git a/tests/e2e/78_test_main_combinations.sh b/tests/e2e/78_test_main_combinations.sh index 74115fed..5e111250 100755 --- a/tests/e2e/78_test_main_combinations.sh +++ b/tests/e2e/78_test_main_combinations.sh @@ -66,7 +66,7 @@ echo "$out" | grep -q '3 passed; 0 failed' || { echo "FAIL: summary mismatch"; e # object with either .o / .obj — the gtest_main substring covers both. # Match by target name + `cxx_link` so path-separator (/, on Windows \) and the # .exe suffix don't matter. The link line is `build bin[.exe] : cxx_link ...`. -nj=$(find target -name build.ninja | head -1) +nj=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) own_link="$(grep -E 't_own_main_gtest(\.exe)? : cxx_link' "$nj" || true)" fw_link="$(grep -E 't_framework_main(\.exe)? : cxx_link' "$nj" || true)" [[ -n "$own_link" ]] || { echo "FAIL: no link line for t_own_main_gtest"; cat "$nj"; exit 1; } diff --git a/tests/e2e/790_a_prepare_action_populates_an_unknown_directory.sh b/tests/e2e/790_a_prepare_action_populates_an_unknown_directory.sh index e0289384..4a71ab3c 100644 --- a/tests/e2e/790_a_prepare_action_populates_an_unknown_directory.sh +++ b/tests/e2e/790_a_prepare_action_populates_an_unknown_directory.sh @@ -108,7 +108,7 @@ ran_prepare() { grep -qE 'PREPARE prep:install|__action(-stamp)? .*install\.sh' # ── 1. first build: header compiles, program runs through runtime_search_dir "$MCPP" build -v > b1.log 2>&1 || { cat b1.log; echo "FAIL: first build failed"; exit 1; } ran_prepare b1.log || { cat b1.log; echo "FAIL: the prepare action did not run on the first build"; exit 1; } -nj=$(find target -name build.ninja | head -1) +nj=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [[ -n "$nj" ]] || { echo "FAIL: no build.ninja"; exit 1; } grep -qE '^build [^:]*(prep\.stamp|prepgreet)' "$nj" >/dev/null || true # sanity, not load-bearing out1="$("$MCPP" run 2>&1)" || { echo "FAIL: run failed after the first build: $out1"; exit 1; } diff --git a/tests/e2e/79_gtest_regular_dep_feature_main.sh b/tests/e2e/79_gtest_regular_dep_feature_main.sh index 5a78fea8..c55cedd7 100755 --- a/tests/e2e/79_gtest_regular_dep_feature_main.sh +++ b/tests/e2e/79_gtest_regular_dep_feature_main.sh @@ -25,7 +25,7 @@ cd app "$MCPP" add compat.gtest@1.15.2 > /dev/null grep -q '^\[dependencies\.compat\]' mcpp.toml || { echo "FAIL: add did not write [dependencies.compat]"; cat mcpp.toml; exit 1; } "$MCPP" build > /dev/null || { echo "FAIL: #168 — build with regular-dep gtest failed"; exit 1; } -nj=$(find target -name build.ninja | xargs ls -t 2>/dev/null | head -1) +nj=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | xargs ls -t 2>/dev/null | head -1) if grep -q 'gtest_main' "$nj"; then echo "FAIL: gtest_main linked into app by default (would collide with main)"; exit 1 fi @@ -46,7 +46,7 @@ cat > src/main.cpp <<'EOF' TEST(App, ok) { EXPECT_EQ(1 + 1, 2); } EOF "$MCPP" build > /dev/null || { echo "FAIL: features=[main] build failed"; exit 1; } -nj=$(find target -name build.ninja | xargs ls -t 2>/dev/null | head -1) +nj=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | xargs ls -t 2>/dev/null | head -1) grep -q 'gtest_main' "$nj" || { echo "FAIL: features=[main] did not link gtest_main"; exit 1; } # (3) `mcpp add --dev` routes to [dev-dependencies]. diff --git a/tests/e2e/808_device_source_is_not_a_compile_unit.sh b/tests/e2e/808_device_source_is_not_a_compile_unit.sh index b64fe582..5f4435fe 100755 --- a/tests/e2e/808_device_source_is_not_a_compile_unit.sh +++ b/tests/e2e/808_device_source_is_not_a_compile_unit.sh @@ -35,7 +35,7 @@ cd "$TMP/ex/app" "$MCPP" build --configure-only > configure.log 2>&1 \ || { cat configure.log; echo "FAIL: configure-only did not succeed"; exit 1; } -ninja_file="$(find target -name build.ninja | head -1)" +ninja_file="$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" [[ -n "$ninja_file" ]] || { echo "FAIL: no build.ninja generated"; exit 1; } if grep -qE '^build [^:]*: cxx_object [^|]*answer\.toy' "$ninja_file"; then diff --git a/tests/e2e/810_two_packages_deploy_the_same_bytes_to_one_name.sh b/tests/e2e/810_two_packages_deploy_the_same_bytes_to_one_name.sh index da4c1c69..d61a097e 100755 --- a/tests/e2e/810_two_packages_deploy_the_same_bytes_to_one_name.sh +++ b/tests/e2e/810_two_packages_deploy_the_same_bytes_to_one_name.sh @@ -117,7 +117,7 @@ int main() { CPP } -find_graph() { find target -name build.ninja | head -1; } +find_graph() { find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1; } # ── 1, 2, 3: identical bytes ──────────────────────────────────────────────── write_dep 'shared payload' diff --git a/tests/e2e/813_conditional_dialect_cxxflags_root_only.sh b/tests/e2e/813_conditional_dialect_cxxflags_root_only.sh index a8ee12dd..660e5883 100755 --- a/tests/e2e/813_conditional_dialect_cxxflags_root_only.sh +++ b/tests/e2e/813_conditional_dialect_cxxflags_root_only.sh @@ -57,7 +57,7 @@ TOML printf 'import std;\nint main() { std::println("ok"); return 0; }\n' > src/main.cpp "$MCPP" build --toolchain gcc@16.1.0 > build1.log 2>&1 || fail "the build with a matching [target.linux.build] row failed" build1.log -ninja1=$(find target -name build.ninja | head -1) +ninja1=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [ -n "$ninja1" ] || fail "no build.ninja produced" build1.log cxxflags_line=$(grep -E '^cxxflags' "$ninja1" | head -1) @@ -105,7 +105,7 @@ TOML printf 'import std;\nint main() { std::println("ok"); return 0; }\n' > src/main.cpp "$MCPP" build --toolchain gcc@16.1.0 > build2.log 2>&1 || fail "the build with a non-matching [target.windows.build] row failed" build2.log -ninja2=$(find target -name build.ninja | head -1) +ninja2=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) grep -qF -- '-DX717A' "$ninja2" compile_commands.json \ && fail "-DX717A from a non-matching [target.windows.build] row reached a Linux build" "$ninja2" grep -E '^cxxflags' "$ninja2" | grep -qF -- '-DBASE717' \ @@ -136,7 +136,7 @@ printf 'import std;\nint main() { std::println("ok"); return 0; }\n' > src/main. write_cyc linux rm -rf target compile_commands.json "$MCPP" build --toolchain gcc@16.1.0 > cyc-a1.log 2>&1 || fail "build A (matching) failed" cyc-a1.log -grep -qF -- '-DCYC717' "$(find target -name build.ninja | head -1)" \ +grep -qF -- '-DCYC717' "$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" \ || fail "build A did not carry -DCYC717" cyc-a1.log stdA1=$(std_module_json_for '-DCYC717') [ -n "$stdA1" ] || fail "no std-module.json for build A" cyc-a1.log @@ -151,14 +151,14 @@ echo " ok: build A (linux, matching) carries -DCYC717 ($stdA1)" write_cyc windows rm -rf target compile_commands.json "$MCPP" build --toolchain gcc@16.1.0 > cyc-b.log 2>&1 || fail "build B (non-matching) failed" cyc-b.log -grep -qF -- '-DCYC717' "$(find target -name build.ninja | head -1)" compile_commands.json \ +grep -qF -- '-DCYC717' "$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" compile_commands.json \ && fail "build B (windows, non-matching on a Linux host) still carries -DCYC717" cyc-b.log echo " ok: build B (windows, non-matching) carries no -DCYC717 -- a different std BMI" write_cyc linux rm -rf target compile_commands.json "$MCPP" build --toolchain gcc@16.1.0 > cyc-a2.log 2>&1 || fail "build A2 (matching again) failed" cyc-a2.log -grep -qF -- '-DCYC717' "$(find target -name build.ninja | head -1)" \ +grep -qF -- '-DCYC717' "$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1)" \ || fail "build A2 did not carry -DCYC717" cyc-a2.log stdA2=$(std_module_json_for '-DCYC717') [ -n "$stdA2" ] || fail "no std-module.json for build A2" cyc-a2.log @@ -200,7 +200,7 @@ TOML printf 'import dep717;\nint main() { return value() == 42 ? 0 : 1; }\n' > src/main.cpp "$MCPP" build --toolchain gcc@16.1.0 > build4a.log 2>&1 || fail "the first build (dependency carries -DDEPFLAG_V1) failed" build4a.log -ninjas4a=$(find target -name build.ninja) +ninjas4a=$(find target -name build.ninja -not -path '*/.build-mcpp/*') [ "$(echo "$ninjas4a" | wc -l)" -eq 1 ] || fail "more than one build.ninja after the first build" build4a.log grep -qF -- '-DDEPFLAG_V1' "$ninjas4a" compile_commands.json \ && fail "the dependency's own dialect_cxxflags reached a command" "$ninjas4a" @@ -208,7 +208,7 @@ echo " ok: the dependency's -DDEPFLAG_V1 reaches neither build.ninja nor compil sed -i 's/-DDEPFLAG_V1/-DDEPFLAG_V2/' "$TMP/dep717/mcpp.toml" "$MCPP" build --toolchain gcc@16.1.0 > build4b.log 2>&1 || fail "the second build (dependency's flag changed) failed" build4b.log -ninjas4b=$(find target -name build.ninja) +ninjas4b=$(find target -name build.ninja -not -path '*/.build-mcpp/*') if [ "$(echo "$ninjas4b" | wc -l)" -ne 1 ]; then echo "build.ninja files after the second build:"; echo "$ninjas4b" fail "changing the dependency's own dialect_cxxflags left a second output directory behind (the fingerprint moved)" build4b.log diff --git a/tests/e2e/814_the_llvm_row_defaults_to_the_dynamic_crt.sh b/tests/e2e/814_the_llvm_row_defaults_to_the_dynamic_crt.sh index b455382f..15ec7883 100755 --- a/tests/e2e/814_the_llvm_row_defaults_to_the_dynamic_crt.sh +++ b/tests/e2e/814_the_llvm_row_defaults_to_the_dynamic_crt.sh @@ -57,7 +57,7 @@ TOML } record_of() { # prints the distributable contract from the one resolution.json - tr -d ' \n\r' < "$(find target -name resolution.json | head -1)" \ + tr -d ' \n\r' < "$(find target -name resolution.json -not -path '*/.build-mcpp/*' | head -1)" \ | grep -o '"distributable":"[a-z-]*"' | head -1 | cut -d'"' -f4 } diff --git a/tests/e2e/821_an_edge_states_its_advice_on_both_paths.sh b/tests/e2e/821_an_edge_states_its_advice_on_both_paths.sh index e9250f65..3d6c6164 100755 --- a/tests/e2e/821_an_edge_states_its_advice_on_both_paths.sh +++ b/tests/e2e/821_an_edge_states_its_advice_on_both_paths.sh @@ -64,7 +64,7 @@ echo "ok: A1 the full path reports the advice once" # its time when the text is unchanged), and the fast path replays it untouched. # A `Compiling` line no longer tells the two apart: it states that a package's # steps ran, on either path (build progress design 2026-09-29). -planned_since() { find target -name build.ninja -newer "$1" 2>/dev/null | grep -q .; } +planned_since() { find target -name build.ninja -not -path '*/.build-mcpp/*' -newer "$1" 2>/dev/null | grep -q .; } touch .before-b2 sleep 1 # a coarse file system clock must see the input as newer printf 'second\n' > data/probe.in diff --git a/tests/e2e/827_many_placements_run_as_one_process.sh b/tests/e2e/827_many_placements_run_as_one_process.sh index 5a6160ec..d9fccbd4 100755 --- a/tests/e2e/827_many_placements_run_as_one_process.sh +++ b/tests/e2e/827_many_placements_run_as_one_process.sh @@ -44,7 +44,7 @@ EOF # S1 program 50 "$MCPP" build > s1.log 2>&1 || fail "S1: the build failed" s1.log -NINJA=$(find target -name build.ninja | head -1) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [ "$(grep -c ': stage_list ' "$NINJA")" = 1 ] || fail "S1: not exactly one stage_list edge" "$NINJA" grep -q ': stage_file .*data' "$NINJA" && fail "S1: a per-file placement edge remains" "$NINJA" OUT=$(dirname "$NINJA") @@ -73,7 +73,7 @@ grep -qx changed "$DEST1" || fail "S3: the changed source did not arrive" "$DEST # S4 program 1 "$MCPP" build > s4.log 2>&1 || fail "S4: the build failed" s4.log -NINJA=$(find target -name build.ninja | head -1) +NINJA=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) grep -q ': stage_list ' "$NINJA" && fail "S4: one entry produced a list edge" "$NINJA" grep -q ': stage_file .*f1.txt' "$NINJA" || fail "S4: the single entry lost its per-file edge" "$NINJA" diff --git a/tests/e2e/849_a_staged_bmi_waits_for_a_module_compiled_here.sh b/tests/e2e/849_a_staged_bmi_waits_for_a_module_compiled_here.sh index 2d0a8236..92b05770 100755 --- a/tests/e2e/849_a_staged_bmi_waits_for_a_module_compiled_here.sh +++ b/tests/e2e/849_a_staged_bmi_waits_for_a_module_compiled_here.sh @@ -120,7 +120,7 @@ grep -qE 'Cached local-dev\.gen-dep v1\.0\.0' hit.log \ echo "ok: A, the package is staged from the cache" # ── B ────────────────────────────────────────────────────────────────────── -N=$(find target -name build.ninja | head -1) +N=$(find target -name build.ninja -not -path '*/.build-mcpp/*' | head -1) [ -n "$N" ] || fail "B: no build.ninja" D=$(dirname "$N") # The BMI directory and extension are the toolchain's (gcm.cache/*.gcm for diff --git a/tests/e2e/851_a_shared_member_is_compiled_once_per_configuration.sh b/tests/e2e/851_a_shared_member_is_compiled_once_per_configuration.sh index 076c7173..81f08225 100755 --- a/tests/e2e/851_a_shared_member_is_compiled_once_per_configuration.sh +++ b/tests/e2e/851_a_shared_member_is_compiled_once_per_configuration.sh @@ -111,7 +111,7 @@ lib_objects() { awk -F'\t' '$4 ~ /(^|\/)lib\/src\/lib\.m\.o$/' "$1/.ninja_log" | # ── A ────────────────────────────────────────────────────────────────────── "$MCPP" build --workspace > a.log 2>&1 || fail "A: the workspace did not build" a.log -dirs=$(find target -name build.ninja -exec dirname {} \; | sort) +dirs=$(find target -name build.ninja -not -path '*/.build-mcpp/*' -exec dirname {} \; | sort) [ "$(echo "$dirs" | wc -l | tr -d ' ')" = 2 ] || fail "A: expected two build directories, got: $dirs" a.log for d in $dirs; do [ "$(lib_objects "$d")" = 1 ] || fail "A: lib's object was compiled $(lib_objects "$d") times in $d" "$d/.ninja_log" @@ -137,7 +137,7 @@ echo "ok: B, one configuration compiles lib once, with its own flags and the uni "$MCPP" build -p app23b > c.log 2>&1 || fail "C: -p app23b did not build" c.log read -r _ xc _ <<< "$(run app23b)" [ "$xc" = 0 ] || fail "C: lib kept a feature app23b's closure does not ask for" c.log -[ "$(find target -name build.ninja | wc -l | tr -d ' ')" = 2 ] || fail "C: -p made a third build directory" c.log +[ "$(find target -name build.ninja -not -path '*/.build-mcpp/*' | wc -l | tr -d ' ')" = 2 ] || fail "C: -p made a third build directory" c.log echo "ok: C, -p plans its own closure's features in the same build directory" echo "PASS: 851_a_shared_member_is_compiled_once_per_configuration" diff --git a/tests/e2e/852_a_repeated_dash_p_selects_every_member_it_names.sh b/tests/e2e/852_a_repeated_dash_p_selects_every_member_it_names.sh index 9474cea7..38131bbb 100755 --- a/tests/e2e/852_a_repeated_dash_p_selects_every_member_it_names.sh +++ b/tests/e2e/852_a_repeated_dash_p_selects_every_member_it_names.sh @@ -38,7 +38,7 @@ done # The build directory of a command: the one directory under target/ holding a # build.ninja. -build_dir() { find target -name build.ninja -exec dirname {} \; | head -1; } +build_dir() { find target -name build.ninja -not -path '*/.build-mcpp/*' -exec dirname {} \; | head -1; } # How many times ninja built the object of member $2 in build directory $1. object_edges() { awk -F'\t' -v m="$2" '$4 ~ ("(^|/)obj/" m "/src/" m "\\.m\\.o$")' "$1/.ninja_log" | wc -l | tr -d ' '; } @@ -73,7 +73,7 @@ echo "ok: A, -p a -p b builds a and b, and c is untouched" "$MCPP" build -p a -p ./b -p b -p a > b2.log 2>&1 || fail "B: a member named twice did not build" b2.log [ "$(object_edges "$dir" a)" = 1 ] && [ "$(object_edges "$dir" b)" = 1 ] \ || fail "B: naming a member twice compiled an edge again" "$dir/.ninja_log" -[ "$(find target -name build.ninja | wc -l | tr -d ' ')" = 1 ] || fail "B: the order made a second build directory" b.log +[ "$(find target -name build.ninja -not -path '*/.build-mcpp/*' | wc -l | tr -d ' ')" = 1 ] || fail "B: the order made a second build directory" b.log echo "ok: B, the order of -p, and a member named twice, do not change the selection" echo "PASS: 852_a_repeated_dash_p_selects_every_member_it_names" diff --git a/tests/e2e/853_run_keeps_one_member_and_exclude_removes_members.sh b/tests/e2e/853_run_keeps_one_member_and_exclude_removes_members.sh index 8afa2d24..84e01ca0 100755 --- a/tests/e2e/853_run_keeps_one_member_and_exclude_removes_members.sh +++ b/tests/e2e/853_run_keeps_one_member_and_exclude_removes_members.sh @@ -38,7 +38,7 @@ for m in a b c; do printf 'int main() { return 0; }\n' > $m/tests/smoke.cpp done -build_dir() { find target -name build.ninja -exec dirname {} \; | head -1; } +build_dir() { find target -name build.ninja -not -path '*/.build-mcpp/*' -exec dirname {} \; | head -1; } refused() { # refused