0.0.85: descriptor grammar v2, xpkg parse (strict), scan_overrides + ddi audit, index version floor#200
Merged
Merged
Conversation
…criptor reader
Descriptor grammar v2 (design: .agents/docs/2026-07-08-index-version-semantics-
and-descriptor-grammar-design.md D1):
- LuaCursor gains long_bracket_level_at / read_long_bracket / at_string_start;
read_string accepts [[...]] / [=[...]=] with Lua first-newline stripping.
- skip_ws_and_comments / skip_table / read_table_body understand --[[ ]]
block comments and long strings (content may contain braces).
- strip_lua_comments_and_strings blanks long-string content and block
comments so the mcpp-segment locator's depth count stays correct.
- Unknown top-level mcpp-segment keys are now RECORDED (Manifest.
xpkgUnknownKeys) instead of silently dropped — surfaced by the upcoming
`mcpp xpkg parse`; `schema` becomes a known (informational) key.
Verified: 4 new unit tests (embedded ]], braces in content, decoy
`mcpp = {` inside block comment, CRLF/first-newline, unknown-key capture);
full `mcpp test` green; integration: mcpp-index fmtlib.fmt descriptor
rewritten with [==[ ]==] parses and its member test passes.
…tion `mcpp xpkg parse <file.lua> [--json] [--allow-unknown]` parses an xpkg descriptor with EXACTLY the resolver's grammar (synthesize_from_xpkg_lua + canonical identity), so index lint and user builds can never disagree (design D2). Strict by default: unknown mcpp-segment keys exit 1 — they would be silently ignored at build time; --allow-unknown downgrades to warnings. --json emits identity, versions, and verbatim generated_files contents (the lua5.4 parity oracle for descriptor migrations). Verified: e2e 93_xpkg_parse (long brackets + block-comment decoy, JSON round-trip byte-identical, unknown-key strictness both ways, malformed segment fails); parity vs lua5.4 extraction on fmtlib.fmt = identical.
manifest.cppm mixed three concerns in 2560 lines: the shared data model,
mcpp.toml parsing, and the xpkg .lua mcpp-segment reader. Now a directory
module with one data model and two independent surface grammars:
src/manifest/manifest.cppm primary — export import :types/:toml/:xpkg
src/manifest/types.cppm Manifest/Target/BuildConfig/errors + shared
helpers (module linkage, not exported)
src/manifest/toml.cppm parse_string/load/default_template
src/manifest/xpkg.cppm LuaCursor, identity, synthesize_from_xpkg_lua
Module name and exported API unchanged — zero importer churn. Full test
suite + e2e 93 green.
…he text scan
For sha256-pinned third-party module units whose imports sit behind
preprocessor guards (fmt's official src/fmt.cc: #ifdef FMT_IMPORT_STD
import std;), the import set is a CONSTANT — recomputing it per build with
compiler processes is the wrong cost placement, and the M1 text scan must
reject it. Instead the descriptor asserts the scan result as data:
scan_overrides = {
["*/src/fmt.cc"] = { provides = { "fmt" }, imports = { "std" } },
}
Matched files skip scan_file(); the declared unit enters the graph with
scanOverridden=true (the flag plan-vs-ddi verification keys on). Both
surfaces parse it: xpkg mcpp segment and mcpp.toml [scan_overrides."glob"].
Every glob must match >=1 collected source (unmatched = loud error).
A unit may declare at most one provided module.
Verified: 3 unit tests (xpkg parse, empty-declaration reject, toml parse);
full suite green; end-to-end: mcpp-index fmtlib.fmt rewritten WITHOUT its
3.4KB generated_files copy — upstream fmt.cc verbatim + -DFMT_IMPORT_STD +
override — builds and its member test passes (import fmt; works).
Design: .agents/docs/2026-07-08-scanner-backend-abstraction-design.md §3-pre.
…anner
The .ddi (compiler's own P1689 scan under real flags) is the ground truth
of what phase 4 saw. ninja_backend now embeds the planner's per-TU
assumption on the dyndep edge (--expect-provides / --expect-imports /
--expect-none); `mcpp dyndep --single` compares and fails the edge on
divergence with both sides named. Mandatory for scan_overrides units (an
assertion needs its auditor); MCPP_VERIFY_MODGRAPH=1 at generation time
extends it to every module unit. Zero extra compiler invocations.
Verified: 3 unit tests; e2e negative — an fmt override deliberately
omitting imports={std} fails at DYNDEP with
planned : provides [fmt] imports [<none>]
compiler: provides [fmt] imports [std]
while the correct declaration builds green.
Design: .agents/docs/2026-07-08-scanner-backend-abstraction-design.md §3d.
… choke point An index tree may carry index.toml ([index] spec / min_mcpp / latest_mcpp). The contract travels WITH the tree, so ONE check in read_identity_verified_xpkg_lua — the choke point every transport converges on (artifact snapshot, git clone, [indices] path, CI-restored cache) — covers all of them, offline included. own < min_mcpp → loud E0006 with the upgrade one-liner, once per index per process, then the resolve fails with the cause already printed. Missing index.toml → no constraint (back-compat, third-party indices). Malformed versions never brick a client. Escape hatch: MCPP_INDEX_FLOOR=ignore. Deviation from design D3 noted: staged-unpack + atomic swap for the artifact refresh lives in vendored xlings (separate codebase) — the open-time check is the mcpp-side enforcement; staging is a follow-up there. Verified: 3 unit tests (ordering incl. 0.0.90>0.0.85, malformed/absent tolerance, TOML round-trip); e2e on mcpp-index: min_mcpp=9.9.9 → E0006 + resolve stop; =0.0.84 → builds; MCPP_INDEX_FLOOR=ignore bypasses. Design: .agents/docs/2026-07-08-index-version-semantics-and-descriptor- grammar-design.md D3.
…ddi audit, index floor Version bump (mcpp.toml + MCPP_VERSION) and docs: [scan_overrides] section in 05-mcpp-toml.md. One-time compatibility note for release notes: mcpp-index will declare min_mcpp = 0.0.85 (index.toml floor) and start using long-bracket descriptors + scan_overrides; pre-0.0.85 binaries reading those descriptors fail to parse them — upgrade via install.sh. From 0.0.85 on, the floor mechanism turns every future evolution into a graceful keep-old-snapshot + upgrade hint instead of a parse error.
…se strict Dry-running the new lint over every existing mcpp-index descriptor found three gaps, all fixed: - list_xpkg_versions only accepted double-quoted version keys; tensorvia's ['0.1.1'] spelling made its versions invisible. Accept both quotes. - linux/macosx/windows per-platform sub-tables are KNOWN schema keys (the current platform's body is merged before the key loop); they no longer read as unknown on other platforms. - Form A descriptors (no mcpp segment; build info from the source's own mcpp.toml) are a valid form: `xpkg parse` reports form A and passes instead of failing on the missing segment. Corpus check: all 42 pkgs/*.lua in mcpp-index now pass `xpkg parse` (strict) with zero warnings.
…s link) GCC 15 (the aarch64-linux-musl cross toolchain) drops implicit template instantiations of module-attached types from partition object files — undefined refs to std::map<..., ScanOverride>::map() etc. at final link. Same split, same importer API, but as three plain modules re-exported by an umbrella mcpp.manifest, the pattern the rest of the codebase already links with everywhere. Shared parser helpers become exported (separate- module visibility requires it).
…e types module GCC 15 (aarch64-linux-musl cross) does not emit implicit std::map/... member instantiations of module-attached structs in importer objects; the old single-file mcpp.manifest provided them by accident (its parser code constructed every struct). A non-inline exported force_template_instantiations() in mcpp.manifest.types recreates that guarantee deliberately. Remove when the cross toolchain floor is GCC 16. Verified locally: mcpp build --target aarch64-linux-musl with the same xim gcc 15.1.0 toolchain links (static aarch64 ELF produced); native build + full unit suite green.
Sunrisepeak
force-pushed
the
feat/descriptor-grammar-v2
branch
from
b6d25f8 to
b2a1ca2
Compare
xlings bumped its default llvm to 22.1.8; the hardcoded sandbox link path and toolchain pin made the job fail at 'Configure dev mcpp sandbox' deterministically. Discover the installed version from ~/.xlings/data/xpkgs/xim-x-llvm/*/ and thread it via MCPP_LLVM_VER. (Install step keeps 'xlings install llvm -y || llvm@20.1.7' as fallback.)
xlings' default llvm moved to 22.1.8, whose libc++ headers reference std::__1::__hash_memory — undefined at link on macOS (ld64.lld), failing any gtest-using e2e (78_test_main_combinations). Pin the known-good 20.1.7 first, keep latest as fallback. Real fix (link against the toolchain's own libc++) belongs to the hermetic-toolchain-link track (.agents/docs/2026-07-07-hermetic-toolchain-link-model-design.md). The version-discovery from the previous commit stays (picks whatever is installed).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release train 0.0.85 — descriptor & index evolution
Design docs (in
.agents/docs/, dated 2026-07-08): index-version-semantics-and-descriptor-grammar (D1–D4), scanner-backend-abstraction (§3-pre, §3d), descriptor-index-evolution-roadmap (live progress table).D1 — descriptor grammar v2 (
b6c5793)LuaCursorlearns Lua long-bracket strings ([[…]]/[=[…]=], first-newline strip) and--[[ ]]block comments, including the sanitizer (long-string content may contain braces).generated_filesno longer needs 3.4 KB single-line escaped strings. Unknown mcpp-segment keys are now recorded instead of silently dropped;schemabecomes a known key.D2 —
mcpp xpkg parse(0b2b9b9)Parses a descriptor with EXACTLY the resolver's grammar — index lint and user builds can never disagree again. Strict by default: unknown keys exit 1 (they'd be silently ignored at build time);
--allow-unknowndowngrades.--jsonemits verbatim generated contents — the parity oracle for descriptor migrations (verified byte-identical vs lua5.4 on fmtlib.fmt).Refactor (
bc24f8e)manifest.cppm(2560 lines, three concerns) →src/manifest/{manifest,types,toml,xpkg}.cppmpartitions. Module name and API unchanged.scan_overrides (
ea7ea43)For pinned third-party module units whose imports sit behind preprocessor guards (fmt's official
src/fmt.cc), the descriptor asserts the scan result as data; matched files bypass the text scan. End-to-end: mcpp-index's fmt package rewritten WITHOUT its generated_files copy builds and passes its member test.Plan-vs-ddi reconciliation (
9211fd4)ninja dyndep edges carry the planner's per-TU assumption (
--expect-*);mcpp dyndep --singlecompares against the compiler's own P1689 scan and fails the edge on divergence, both sides printed. Mandatory for override units;MCPP_VERIFY_MODGRAPH=1extends to all.Index version floor (
94e4c6b)An index tree may carry
index.toml([index] min_mcpp). Checked once at the index-open choke point every transport converges on. Violation →E0006+ upgrade one-liner (mcpp explain E0006;MCPP_INDEX_FLOOR=ignorehatch). Missing file → no constraint.Compatibility
One-time break, deliberately taken while the user base is small: once mcpp-index declares
min_mcpp = "0.0.85"and migrates descriptors, pre-0.0.85 binaries fail to parse them. From 0.0.85 on the floor turns every future evolution into a graceful error with guidance.Verification
mcpp test), incl. new grammar/override/contract/reconciliation tests