refactor(toolchain): one producer for host-compile flags; build.mcpp gains MSVC modules (2026.8.2.2) - #334
Merged
Merged
Conversation
Three places assembled the same host-compile flags independently: flags.cppm for the main build, stdmod.cppm for the std module, and build_program.cppm for build.mcpp. The resolvers (linkmodel, clang driver model) were already shared; the ASSEMBLY was not, because the seam only produced strings and the one consumer that needs argv — build.mcpp, which execs directly with no shell — could not use it and hand-wrote the whole thing a second time. That split is where PR#332's batch came from. Every bug in it was the same sentence: flags.cppm knew about quoting / the macOS deployment target / the MSVC dialect, and the other two did not. A 0.0.9x fix had already corrected the same file once for the same reason (musl->static re-derived), and stdmod.cppm carried a comment instructing the reader to keep it in sync with flags.cppm by hand — the failure mode this repo wrote check_version_pins.sh to stop. Tokens are now the source of truth and strings are rendered from them, not the reverse. mcpp.toolchain.hostflags composes the resolvers once; the three consumers render with ninja escaping, shell quoting, or not at all. The knobs on HostFlagOptions are documented divergences, not switches to preserve accidents: the host helper keeps trusting clang's cfg on macOS/Windows (the macOS link needs libc++abi/unwind handling the main build's needs_explicit_libcxx path owns), the std module states -stdlib=libc++ explicitly (flags.cppm cannot — the same string feeds C compiles), and binutils -B / runtime dirs belong only to the one-shot host link. Pure refactor, verified as such: the global cxxflags/cflags/ldflags lines in build.ninja AND stdmod's std_build_commands are byte-identical before and after, under both gcc@16.1.0 and llvm@22.1.8, each from a clean target and dep cache. Byte-identity is not cosmetic here — std_build_commands feeds the std cache directory name, so a reordered flag would invalidate every user's std BMIs for no gain. hostflags lives in its own module rather than inside build_program.cppm's anonymous namespace: PR#332 established that adding a function there can miscompile a neighbouring one under clang 22.1.8 + C++20 modules + -O2.
`import mcpp;` / `import std;` in a build.mcpp used to fail under native MSVC with "not yet supported", and the stated reason was that mcpp had not wired up cl.exe's .ifc pipeline. That reason was wrong: the main build has compiled named modules with cl.exe for a while — e2e 99 produces real .ifc artifacts on every Windows CI run, and ninja_backend has emitted /interface /TP /ifcOutput and /scanDependencies all along. What was missing was only that build.mcpp hand-rolled its own compile path and never read those tables. build_mcpp_module now dispatches on BmiTraits + CommandDialect — the same rows the main build uses — so cl.exe needed no new pipeline, just the row that was already there. The gate is gone. bmi_reference_tokens covers the shape difference the traits do not have to care about: `-fmodule-file=std=<p>` is one argv word, `/reference std=<p>` is two, and only an argv consumer notices. The traits keep storing the string form for the ninja channel. The capability-parity guard in test_hostflags.cpp is the part meant to outlast this change: it walks every CompilerId and fails if a family is missing a dialect row, a module row, or host flags. Stating "build.mcpp has no capability list of its own" in prose is exactly how the MSVC gate outlived its own justification.
…ut it Placing it after the libc++ -isystem flags instead of right after -nostdinc++ changed the std module's command string, and that string is part of the std cache identity (std_build_commands feeds the cache directory name) — every user's std BMIs would have been invalidated for a reordering that buys nothing. Caught only because the first byte-equality check scanned the whole shared cache directory and compared stale entries alongside fresh ones; the check now wipes the std cache first so it compares like with like. Also moves the bundled mcpp module's compile out of build_program.cppm's anonymous namespace into mcpp.build.hostprogram. Growing that namespace reproduced PR#332's clang miscompilation exactly — Segmentation fault: 11 on every macOS build.mcpp e2e, in contract_env, which this change never touched.
Byte-equality with what earlier releases emitted is a compatibility surface here, not an implementation detail: stdmod folds its compile command into std_build_commands and the std cache directory name is derived from the metadata containing it, so a reordered flag invalidates every user's std BMIs. Verifying that by inspecting the cache after a build turned out to be unreliable — the cache is shared and accumulates entries from every toolchain used since, so the first check compared stale entries alongside fresh ones and passed while the string had in fact moved. These tests assert the spelling directly from a synthetic model, deterministically, and keep asserting it.
…'s cfg; drop -x for cl
Two failures the local runs could not see, both the same shape — a branch
that was dead while the MSVC gate existed, or a platform this machine is not.
host_compile_tokens returned early on the trust-cfg path, which on macOS IS
the build.mcpp path: no clang flags and no deployment target, so the std BMI
(built for 14.0) was rejected by a TU compiled without it. Trusting the cfg
means contributing no include paths or stdlib selection — not contributing
nothing. The deployment target is emitted regardless now, which is what the
hand-written host_base_flags did deliberately ("FIRST and unconditionally")
before this refactor absorbed it.
The `-x none` before the module objects was unconditional and harmless only
while cl.exe could not reach it. Removing the module gate made the dead
branch live, and cl answered `D9002: ignoring unknown option '-x'` — after
compiling the .ifc and reaching the link, which is further than build.mcpp
has ever gone under MSVC.
cl.exe rejected the std and mcpp module objects with "C2018: character 'U+10' is not permitted here" — it was compiling them as C++ source. `/TP` is not the counterpart of `-x c++`: GNU's is positional and lasts until `-x none`, cl's applies to EVERY input on the line, so the objects that follow are fed to the C++ frontend. The dialect now carries the per-FILE form (`/Tp<file>`) alongside the positional one, and the build.mcpp compile uses whichever the driver has. That is the same structural difference the row already records for libFlag (prefix vs suffix): the two drivers are not spelling the same concept differently, they have different concepts.
cl answered "C2230: could not find module 'std'" because the std reference was still built by string concatenation, producing one argv element with a space inside it. bmi_reference_tokens existed for exactly this and was only applied to the bundled mcpp module. Same shape for the third time — a table entry written for the ninja STRING channel concatenated into an argv element — so it now has a test that covers every family and both spellings, plus one for the language-force tokens. The compiler's own diagnostics name neither the flag nor the reason, which is why this kept costing a CI round each time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design:
.agents/docs/2026-08-02-host-compile-single-producer-design.mdPlan:
.agents/docs/2026-08-02-host-compile-implementation-plan.mdWhy
Three places assembled the same host-compile flags independently —
flags.cppmfor the main build,
stdmod.cppmfor the std module,build_program.cppmforbuild.mcpp. The resolvers were already shared; the assembly was not,because the seam only produced strings and the one consumer that needs argv
(
build.mcpp, which execs directly with no shell) could not use it.Every bug in PR #332's batch was that same sentence:
flags.cppmknew aboutquoting / the macOS deployment target / the MSVC dialect, and the other two did
not. A 0.0.9x fix had already corrected the same file once for the same reason,
and
stdmod.cppmcarried a comment telling the reader to keep it in sync withflags.cppmby hand — the failure mode this repo wrotecheck_version_pins.shto stop.
What changed
Tokens are the source of truth; strings are rendered from them.
mcpp.toolchain.hostflagscomposes the resolvers once; the three consumersrender with ninja escaping, shell quoting, or not at all.
The
HostFlagOptionsknobs are documented divergences, not switches preservingaccidents — the host helper keeps trusting clang's cfg on macOS/Windows (the
macOS link needs the libc++abi/unwind handling the main build's
needs_explicit_libcxxpath owns), the std module states-stdlib=libc++explicitly (
flags.cppmcannot — the same string feeds C compiles), andbinutils
-B/ runtime dirs belong only to the one-shot host link.build.mcppgains named modules under MSVC as a consequence, not as afeature. The gate said cl.exe's
.ifcpipeline was unimplemented; it wasnot — e2e 99 has been producing
.ifcartifacts on every Windows CI run.build_mcpp_modulenow dispatches onBmiTraits+CommandDialect, so therow that already existed is all it took.
Testing
The refactor is verified as a no-op, which is the only way to review a
change to
compute_flags(it serves every build): the globalcxxflags/cflags/ldflagslines inbuild.ninjaand stdmod'sstd_build_commandsare byte-identical before and after, under bothgcc@16.1.0andllvm@22.1.8, each from a cleantarget/and dep cache.Byte-identity is not cosmetic —
std_build_commandsfeeds the std cachedirectory name, so a reordered flag would invalidate every user's std BMIs for
no gain.
test_hostflags.cpp, including a capability-parity guardthat fails when a
CompilerIdis missing a dialect / module / flags row)covering glibc, musl (168) and mingw-cross (112)
work", and asserts a real
.ifccame outcheck_version_pins.shcleanhostflagsis its own module rather than a helper insidebuild_program.cppm's anonymous namespace: PR #332 established that adding afunction there can miscompile a neighbouring one under clang 22.1.8 + C++20
modules +
-O2.