From ca98c681a9394f54a8f0c5e4ac442bc39c28a5c3 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 13:15:08 -0400 Subject: [PATCH 01/62] docs(plans): FR-023 phase 1a implementation plan Twenty-three TDD tasks for TypeScript, Python, the dependency-conformance corpus and the docs/skills, executed from the committed design. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../plans/2026-09-11-fr-023-phase-1a.md | 1292 +++++++++++++++++ 1 file changed, 1292 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md new file mode 100644 index 000000000..edcaf5ec3 --- /dev/null +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -0,0 +1,1292 @@ +# FR-023 Phase 1a — Metadata dependencies (TypeScript + Python + corpus) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A consuming project declares a dependency on a library's generated shared-model artifact, syncs it into a committed snapshot + lock, references / extends / overlays its nodes, never generates or migrates them in `reference` mode, and fails loudly — at load and at `verify --deps` — when upstream changes what it built on. + +**Architecture:** The publisher runs a `sharedModelFile()` generator that emits one flattened canonical-JSON artifact plus a manifest; the consumer's Node `meta deps sync` copies that artifact into `.metaobjects/deps//` and pins its hash in `.metaobjects/deps.lock.json`; every port's collection resolver loads the snapshot files (with `dep:`-prefixed source ids) beside the project's own sources and exposes one predicate, `governs(fqn)`, that codegen and migrate thread through their existing scope seams. A usage-aware classifier diffs the foreign nodes the consumer uses when upstream moves. + +**Tech Stack:** TypeScript (Bun test runner, zod, `node:crypto`, `node:util` `parseArgs`), Python 3.11+ (`uv`, pytest, PyYAML), a file-shaped cross-port corpus under `fixtures/dependency-conformance/`. + +**Spec:** `docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md` (referred to below as DESIGN; every task names the § it implements). + +## Global Constraints + +Every task's requirements implicitly include this section. Reviewers: read a task's diff against these contracts. + +**Scope of this plan.** Phase 1a only: TypeScript (`metadata`, `sdk`, `codegen-ts`, `cli`), Python, `fixtures/dependency-conformance/`, docs, skills, CHANGELOG. Phase 1b (`packageBindings`, the `docs` badge) and the estate cutover are OUT. Java/Kotlin/C# are OUT (Phase 2). + +**No metamodel vocabulary change.** Nothing in this plan registers a type, subtype or attribute. `fixtures/registry-conformance/expected-registry.json` and `METAMODEL_VERSION` (`server/typescript/packages/metadata/src/registry-manifest.ts`, `"1.0"`) do not move; `node scripts/check-metamodel-version.mjs` must report no diff at the end. Config keys, the manifest, the lock and the override file are tool files. + +**Config — `.metaobjects/config.json` (DESIGN §3.1), strict at every level:** +```jsonc +{ "schema_version": 1, "sources": [], + "dependencies": [ { "name": "acme-common", "path": "../lib/metaobjects", "mode": "reference" } ] } +``` +A dependency spec carries `name` (`/^[a-z0-9][a-z0-9._-]*$/`, unique across the array), exactly ONE transport key — `path` (string) | `npm` (string, optional `dir`) | `python` (string, optional `dir`) — and `mode` ∈ `"reference" | "own"`, default `"reference"`. Any other key is a schema error. `dependencies` is read by every port at EVERY rung of the source ladder (DESIGN §2.3). + +**Manifest — `metaobjects.pkg.json` (DESIGN §3.2), generated, at the artifact's directory:** +```jsonc +{ "schema_version": 1, "name": "acme-common", "version": "1.0.0", "metamodelVersion": "1.0", + "artifact": "acme-common.metaobjects.json", "integrity": "sha256-…", + "packages": ["acme::common"], "nodes": ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"] } +``` +`packages` and `nodes` are sorted; `nodes` are resolution keys of every top-level node in the artifact. + +**Lock — `.metaobjects/deps.lock.json` (DESIGN §3.3), written only by `meta deps sync`:** +```jsonc +{ "schema_version": 1, "dependencies": { "acme-common": { + "version": "1.0.0", "metamodelVersion": "1.0", "mode": "reference", + "resolvedFrom": { "path": "../lib/metaobjects" }, "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-…", "packages": ["acme::common"], "nodes": ["acme::common::Address", "…"] } } } +``` +Keys sorted by name; no timestamps, no absolute paths, no interpreter paths. + +**Snapshot (DESIGN §3.4).** `.metaobjects/deps//` — bytes verbatim; never walked; loaded by the lock's `artifact` path with source id `dep:/`. + +**Artifact (DESIGN §3.5).** A canonical-JSON `metadata.root` document with NO root `package`; each top-level node carries an explicit `package`; raw (own-layer) form, `extends` preserved; top-level nodes sorted by resolution key; attribute keys alphabetized; body key order `name, package, extends, abstract, isArray, @attrs, children`; 2-space indent, LF, one trailing newline; Python emits with `ensure_ascii=False`. + +**Hash format.** `integrity` = `"sha256-" + lowercase hex sha256 of the artifact bytes`. Pinned corpus value: the committed `fixtures/dependency-conformance/artifacts/acme-common-v1.json` hashes to `sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d`. + +**Named constants (TS `sdk/src/dependencies.ts`, Python `metaobjects/config/dependencies.py`):** `DEPS_DIR = "deps"`, `LOCK_FILE = "deps.lock.json"`, `LOCAL_OVERRIDE_FILE = "deps.local.json"`, `MANIFEST_FILE = "metaobjects.pkg.json"`, `ARTIFACT_SUFFIX = ".metaobjects.json"`, `DEPENDENCY_SOURCE_ID_PREFIX = "dep:"`, `DEPENDENCY_MODES = ["reference", "own"]`, `DEFAULT_DEPENDENCY_MODE = "reference"`, `INTEGRITY_PREFIX = "sha256-"`. No metamodel string is ever inlined; metamodel names come from `@metaobjectsdev/metadata/constants` / `metaobjects.shared.*`. + +**Error codes (exactly these nine, registered in `fixtures/conformance/ERROR-CODES.json`, TS `ERROR_CODES`, Python `ErrorCode`):** `ERR_DEPENDENCY_UNRESOLVED`, `ERR_DEPENDENCY_MANIFEST_INVALID`, `ERR_DEPENDENCY_SNAPSHOT_STALE`, `ERR_DEPENDENCY_NODE_COLLISION`, `ERR_DEPENDENCY_OVERLAY_IMPLICIT`, `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`, `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`, `ERR_DEPENDENCY_UPSTREAM_DRIFT`, `ERR_DEPENDENCY_BREAKING_CHANGE`. + +**The foreign predicate (DESIGN §2.7).** `foreignOwner(fqn)` = the resolved dependency whose lock `nodes` contains `fqn`, else `undefined`. `governs(fqn)` = `foreignOwner(fqn) === undefined || foreignOwner(fqn).mode === "own"`. Foreignness is a set lookup on the lock's `nodes` — never package ownership, never file provenance. Codegen selects `inScope(fqn) && governs(fqn)`; migrate governs `governs(fqn) && (inMigrateScope?.(fqn) ?? true)`. + +**Overlay rules (DESIGN §2.4).** A local top-level node whose resolution key is in a dependency's `nodes` MUST carry `overlay: true` → else `ERR_DEPENDENCY_OVERLAY_IMPLICIT`. In `reference` mode a LOCAL contribution to a foreign node that is a `field.*` / `identity.*` / `index.*` / `relationship.*` / `source.*` child, a locally-set `@table` / `@schema` / `@column` / `@kind`, or a local TPH subtype (declares `@discriminatorValue`) whose discriminator base is foreign → `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`. `own` mode allows all of it. Extending a foreign abstract, referencing a foreign node by FQN, and declaring new nodes in the dependency's package are always allowed. + +**Classifier (DESIGN §2.6).** Footprint scopes: `whole` (targets of `extends` and of `field.object`/`field.map` `@objectRef`, `@payloadRef`, `@responseRef`, `@parameterRef`; closed over the target's `extends` chain), `key` (targets of `@references`, relationship `@objectRef`/`@through`, origin `@from`/`@of`/`@via`: the target's `identity.*` and `source.*` children plus the specifically named members), `existence` (`overlay: true` targets: the node's presence and `type.subType`, the attrs the overlay also sets, the parents of locally-added children; `@implementedBy`). Widest scope wins. Classes: node/descendant removed, `type.subType` changed, `extends`/`abstract`/`isArray` changed, attribute removed, attribute changed outside the widening table, attribute added outside the additive table → **breaking**; child node added → **compatible**; documentation attrs `description`, `title`, `notes`, `seeAlso`, `aliases`, `replacedBy` → **ignored**; `deprecated` added → **info**. Widening table (compatible in the stated direction only): `@maxLength` up, `@minLength` down, `@precision` up, `@max` up, `@min` down, `@required` true→false, `@values` superset, `@intValueMap` superset with existing entries unchanged. Additive table (attr appearing where none was, compatible): the documentation attrs, `@filterable`, `@sortable`. Anything not in a table is breaking. + +**D10 — local override (DESIGN §10 D10).** `.metaobjects/deps.local.json` = `{ "": { "path": "" } }`, never committed (`meta init` and `meta deps sync` add `deps.local.json` to `.metaobjects/.gitignore`). While present: every load in every port reads `/` in place, skips that dependency's snapshot hash check, and prints ONE warning per command: `loading from a local override (), not the committed snapshot`. `meta deps sync ` copies from the override path. `meta verify --deps` FAILS while any override is active (exit 1, message names the dependencies). An override naming an undeclared dependency → `ERR_DEPENDENCY_UNRESOLVED`; an override path without a manifest → `ERR_DEPENDENCY_UNRESOLVED`. + +**Source ids.** A dependency artifact loads as a `FileSource` whose `id` is `dep:/` (override or snapshot alike); local files keep `basename(path)`. + +**Discipline.** TDD: every task writes the failing test first and shows it failing. ADR-0039: read metadata through resolving accessors (`attr()`, `children()`, `attrs().get()`); an `own*()` call needs a comment naming its sanctioned case (own-mode serialization, root-level scans, overlay/merge machinery). Cross-package `instanceof` ban: identify nodes with the exported guards (`isMetaObject`, `isMetaField`, …) — never `x instanceof MetaSource` outside `metadata`. Constants discipline (above). Never mutate loaded metadata. Never edit a generated golden or `.hashes.json` by hand — regenerate through the tool and review the diff. `no-hardcoded-metadata-dir.test.ts` must pass with NO allowlist change. Public-repo hygiene: no private project names, no client names, no `/home/` or `~/` paths in any file, test, fixture, commit message or comment — the consumer is "a consuming app", the library is "a library project". ADR-0034: a new ejectable generator is listed by `meta eject --list`. + +**Commands.** TS tests are scoped per package: `cd server/typescript/packages/ && bun test ` — NEVER a bare `bun test` at the repo root. Typecheck from the repo root: `bun run --filter '@metaobjectsdev/' typecheck`. Python: `cd server/python && uv run --extra integration pytest -q`. Commit after each task with the subject given; never `git add -A` — stage the task's files by name. Commit-message and branch hygiene follow the public-repo rules above. + +--- + +### Task 1: Corpus skeleton, pinned artifacts, and the nine error codes + +**Files:** +- Create: `fixtures/dependency-conformance/README.md` +- Create: `fixtures/dependency-conformance/cases.json` +- Create: `fixtures/dependency-conformance/artifacts/acme-common-v1.json` +- Create: `fixtures/dependency-conformance/artifacts/acme-common-v1-widened.json` +- Create: `fixtures/dependency-conformance/artifacts/acme-common-v2-email-removed.json` +- Modify: `fixtures/conformance/ERROR-CODES.json` +- Modify: `server/typescript/packages/metadata/src/errors.ts` (the `ERROR_CODES` array, after `"ERR_COLLECTION_NOT_FOUND"`) +- Modify: `server/python/src/metaobjects/errors.py` (the `ErrorCode` enum, after `ERR_COLLECTION_NOT_FOUND`) +- Create: `server/typescript/packages/sdk/test/dependency-conformance.test.ts` + +**Read first:** DESIGN §6 (the corpus shape and case list), §3 (schemas), §2.11 (discipline). Model the runner on `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts` and the README on `fixtures/source-resolution-conformance/README.md`. + +**Interfaces:** +- Produces: `fixtures/dependency-conformance/cases.json` with the case schema below; the three artifact files (byte-exact); the nine codes in three registries; a TS runner that reads `cases.json` and fails every case until later tasks implement the resolver. + +**Case schema** (document it in the README; the runner implements exactly this): +```jsonc +{ "cases": [ { + "name": "…", // kebab-case, the contract + "tree": { "": "" }, // materialized under a fresh temp root + "treeFiles": { "": "artifacts/" }, // OPTIONAL: copied byte-for-byte from the corpus dir + "config": { … } | null, // written to /.metaobjects/config.json + "lock": { … }, // OPTIONAL: written to /.metaobjects/deps.lock.json + "localOverrides": { … }, // OPTIONAL: written to /.metaobjects/deps.local.json + "resolveFrom": ".", // OPTIONAL + "expectFiles": ["…"], // unordered set, project-root-relative (resolution arm) + "expectForeign": [""], // OPTIONAL: FQNs with a foreign owner + "expectGoverned": [""], // OPTIONAL: FQNs governs() admits, over every loaded top-level object + "expectOverrides": [""], // OPTIONAL: dependencies read from a local override + "expectLoadError": "ERR_*", // OPTIONAL: the collection resolves, then LOADING it fails with this code + "expectErrorFiles": ["dep:…"], // OPTIONAL with expectLoadError: source.files[0] of the first error + "expectError": "ERR_*", // resolution itself fails with this code + "classify": { // classifier arm (TS + Python only) + "old": { … }, "new": { … }, // two artifact documents, inline + "footprint": { "": "whole" | "key" | "existence" }, + "expectChanges": [ { "fqn": "…", "path": "…", "kind": "breaking" | "compatible" | "info" } ] } +} ] } +``` +Exactly one of `expectFiles`, `expectError`, `classify` is present per case; `expectLoadError` rides with `expectFiles`. + +- [ ] **Step 1: Write the three artifact files byte-exactly.** `acme-common-v1.json` is this content (2-space indent, LF line endings, ONE trailing newline, no BOM): + +```json +{ + "metadata.root": { + "children": [ + { + "object.value": { + "name": "Address", + "package": "acme::common", + "children": [ + { + "field.string": { + "name": "city" + } + }, + { + "field.string": { + "name": "street" + } + } + ] + } + }, + { + "object.entity": { + "name": "Audited", + "package": "acme::common", + "abstract": true, + "children": [ + { + "field.timestamp": { + "name": "createdAt" + } + } + ] + } + }, + { + "object.entity": { + "name": "Customer", + "package": "acme::common", + "children": [ + { + "source.rdb": { + "@table": "customers" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "email", + "@maxLength": 120 + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ] + } +} +``` +`acme-common-v1-widened.json` is identical except `"@maxLength": 200`. `acme-common-v2-email-removed.json` is identical except the whole `{ "field.string": { "name": "email", … } }` child is absent. + +- [ ] **Step 2: Verify the bytes.** Run from the repo root: +```bash +python3 -c 'import hashlib,sys; [print("sha256-"+hashlib.sha256(open(f,"rb").read()).hexdigest(), f) for f in sys.argv[1:]]' fixtures/dependency-conformance/artifacts/*.json +``` +Expected, exactly: +``` +sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d fixtures/dependency-conformance/artifacts/acme-common-v1.json +sha256-fa00b9f3c54a2c302cf269af051589afc0a3217999ac82c54c77e33494638c36 fixtures/dependency-conformance/artifacts/acme-common-v1-widened.json +sha256-c4ba364bff0071b164b127326c095d6d32915b57781271e0be64a7003c27ce7a fixtures/dependency-conformance/artifacts/acme-common-v2-email-removed.json +``` +If a hash differs, fix the bytes (whitespace, trailing newline, CRLF) — never change the pinned values. + +- [ ] **Step 3: Write `cases.json` with case 1** (the first of the DESIGN §6 list; later tasks append their own): +```json +{ + "cases": [ + { + "name": "no-dependencies-resolves-exactly-as-before", + "tree": { "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" }, + "config": { "schema_version": 1, "sources": [] }, + "expectFiles": ["metaobjects/meta.app.json"], + "expectForeign": [], + "expectGoverned": ["app::Order"] + } + ] +} +``` + +- [ ] **Step 4: Write the README** — the case schema above verbatim, the artifact hashes, the one-liner from Step 2, the rule "a `treeFiles` entry copies a corpus file byte-for-byte so a pinned hash can never drift", the sentence "Every port's runner reads THIS file; there is no per-port fixture", and the list of which arms each port runs (TS: all; Python: all; Java/C#/Kotlin: Phase 2). + +- [ ] **Step 5: Register the nine codes.** `ERROR-CODES.json` — add to `codes`, one-line descriptions: +``` +"ERR_DEPENDENCY_UNRESOLVED": "FR-023: a declared dependency's transport or local override could not locate a directory holding metaobjects.pkg.json, or an override names an undeclared dependency.", +"ERR_DEPENDENCY_MANIFEST_INVALID": "FR-023: a dependency's metaobjects.pkg.json fails its schema, names a different dependency, points at a missing or hash-mismatched artifact, or its artifact does not load standalone / does not declare exactly the listed packages and nodes.", +"ERR_DEPENDENCY_SNAPSHOT_STALE": "FR-023: the committed snapshot does not match .metaobjects/deps.lock.json (lock missing, entry missing or extra, artifact missing, or hash mismatch) — run `meta deps sync`.", +"ERR_DEPENDENCY_NODE_COLLISION": "FR-023: two dependencies export the same fully-qualified node.", +"ERR_DEPENDENCY_OVERLAY_IMPLICIT": "FR-023: a local top-level node redeclares a dependency's node without `overlay: true`.", +"ERR_DEPENDENCY_SCHEMA_NOT_OWNED": "FR-023: a local contribution changes the physical shape of a reference-mode dependency's node (a field/identity/index/relationship/source child, a physical attribute, or a TPH subtype of a foreign base).", +"ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE": "FR-023: a dependency's metamodelVersion major differs from this toolchain's.", +"ERR_DEPENDENCY_UPSTREAM_DRIFT": "FR-023: `meta deps check` / `verify --deps` found the installed dependency differs from the committed snapshot.", +"ERR_DEPENDENCY_BREAKING_CHANGE": "FR-023: `meta deps sync` refused an upstream change classified BREAKING for this consumer's footprint (override with --accept-breaking)." +``` +TS `errors.ts`: append the nine strings to `ERROR_CODES` immediately before `"ERR_UNKNOWN"`, each with a one-line `// FR-023 — …` comment. Python `errors.py`: append nine enum members after `ERR_COLLECTION_NOT_FOUND`. + +- [ ] **Step 6: Run the registry parity tests** (they must be green — TS asserts full agreement, Python asserts coverage): +```bash +cd server/typescript/packages/metadata && bun test test/errors.test.ts +cd server/python && uv run --extra integration pytest tests/unit/test_errors.py -q +``` +Expected: PASS. + +- [ ] **Step 7: Write the failing runner** `sdk/test/dependency-conformance.test.ts`. Copy the `materialize` shape from `source-resolution-conformance.test.ts` and extend it: copy `treeFiles` from `fixtures/dependency-conformance/`, write `lock` to `.metaobjects/deps.lock.json` and `localOverrides` to `.metaobjects/deps.local.json`. For each case: `classify` arm → `test.skip` for now with the name (Task 14 fills it); `expectError` → `await expect(resolveCollection(resolveDir)).rejects.toMatchObject({ code })`; `expectFiles` → assert the unordered set of project-root-relative files equals `expectFiles`, then, when present, `expectForeign` equals `collection.dependencies` lookups (`collection.foreignOwner(fqn) !== undefined`), `expectGoverned` equals the set of loaded top-level object FQNs for which `collection.governs(fqn)`, `expectOverrides` equals `collection.overrides`, and `expectLoadError` → `loadMemory(resolveDir, { files: collection.files, fileIds: collection.fileIds })` rejects with the code and (when `expectErrorFiles` given) `err.source.files` equals it. Include the `corpus is non-empty` guard test. + +- [ ] **Step 8: Run it and watch it fail** +```bash +cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts +``` +Expected: the non-empty guard passes; case 1 FAILS because `collection.foreignOwner` / `collection.governs` do not exist yet (a TypeError, not a resolution error). Do not implement anything in sdk here. + +- [ ] **Step 9: Typecheck is allowed to be red for the test file only** — confirm the production code typechecks: `bun run --filter '@metaobjectsdev/metadata' typecheck` → PASS. + +- [ ] **Step 10: Commit** +``` +feat(conformance): FR-023 dependency corpus skeleton, pinned artifacts, and nine error codes +``` + +--- + +### Task 2: A shared document serializer with explicit node-level packages (TS + Python) and its 4-port fixture + +**Files:** +- Modify: `server/typescript/packages/metadata/src/serializer-json.ts` (append; export `serializeSharedDocument`) +- Modify: `server/typescript/packages/metadata/src/naming.ts` (export `packageOfResolutionKey`) +- Modify: `server/typescript/packages/metadata/src/index.ts` (export both) +- Modify: `server/python/src/metaobjects/serializer_json.py` (add `serialize_shared_document`) +- Modify: `server/python/src/metaobjects/naming.py` (add `package_of_resolution_key`) +- Create: `fixtures/conformance/xpkg-node-level-package-no-root-package/input/meta.shared.json`, `expected.json` +- Test: `server/typescript/packages/metadata/test/serializer-shared-document.test.ts`, `server/python/tests/unit/test_serializer_shared_document.py` + +**Read first:** DESIGN §3.5 (the artifact form) and §2.9 (why `extends` is preserved, not flattened). `serializer-json.ts` `canonicalSerialize` (2-space indent + trailing `"\n"`, attr keys sorted, FR-016 physical-name rewrite) and `serializeNodeInner`'s body-key order; Python `serializer_json._body`. The parser already honours a node-level `package` on a top-level child (`parser-core.ts` `rootChildResolutionKey`; Python `parser.py` around the `file_default_package` capture). + +**Interfaces:** +- Produces TS: `packageOfResolutionKey(fqn: string): string` (`"a::b::C"` → `"a::b"`, `"C"` → `""`); `serializeSharedDocument(nodes: readonly MetaData[]): string`. +- Produces Python: `package_of_resolution_key(fqn: str) -> str`; `serialize_shared_document(nodes: list[MetaData]) -> str`. Both outputs are byte-identical for the same input. + +- [ ] **Step 1: Write the failing TS test** `serializer-shared-document.test.ts`: +```ts +import { describe, expect, test } from "bun:test"; +import { MetaDataLoader, serializeSharedDocument, packageOfResolutionKey } from "../src/index.js"; + +const LIB = JSON.stringify({ "metadata.root": { package: "acme::common", children: [ + { "object.entity": { name: "Customer", children: [ + { "source.rdb": { "@table": "customers" } }, { "field.long": { name: "id" } }, + { "field.string": { name: "email", "@maxLength": 120 } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } } ] } }, + { "object.entity": { name: "Audited", abstract: true, children: [ { "field.timestamp": { name: "createdAt" } } ] } }, + { "object.value": { name: "Address", children: [ { "field.string": { name: "street" } }, { "field.string": { name: "city" } } ] } }, +]}}); + +describe("serializeSharedDocument", () => { + test("packageOfResolutionKey", () => { + expect(packageOfResolutionKey("acme::common::Customer")).toBe("acme::common"); + expect(packageOfResolutionKey("Customer")).toBe(""); + }); + test("emits nodes sorted by resolution key with an explicit package and no root package", async () => { + const { root, errors } = await MetaDataLoader.fromString(LIB, "json"); + expect(errors).toEqual([]); + const out = serializeSharedDocument(root.objects()); + const doc = JSON.parse(out) as { "metadata.root": { package?: string; children: Record[] } }; + expect(doc["metadata.root"].package).toBeUndefined(); + const names = doc["metadata.root"].children.map((c) => Object.values(c)[0]!.name); + expect(names).toEqual(["Address", "Audited", "Customer"]); + for (const c of doc["metadata.root"].children) expect(Object.values(c)[0]!.package).toBe("acme::common"); + // body key order: name, package, then the rest + expect(Object.keys(Object.values(doc["metadata.root"].children[1]!)[0]!)).toEqual(["name", "package", "abstract", "children"]); + expect(out.endsWith("\n")).toBe(true); + expect(out).toBe(await Bun.file(`${import.meta.dir}/../../../../../fixtures/dependency-conformance/artifacts/acme-common-v1.json`).text()); + }); + test("a re-load of the document yields the same resolution keys", async () => { + const { root } = await MetaDataLoader.fromString(LIB, "json"); + const again = await MetaDataLoader.fromString(serializeSharedDocument(root.objects()), "json"); + expect(again.errors).toEqual([]); + expect(again.root.objects().map((o) => o.resolutionKey()).sort()).toEqual(["acme::common::Address", "acme::common::Audited", "acme::common::Customer"]); + }); + test("a root-level node (empty package) is refused", async () => { + const { root } = await MetaDataLoader.fromString(JSON.stringify({ "metadata.root": { children: [ { "object.value": { name: "Bare" } } ] } }), "json"); + expect(() => serializeSharedDocument(root.objects())).toThrow(/package/); + }); +}); +``` +Note the byte-equality assertion against the pinned corpus artifact — that is what makes Task 12's golden and the corpus agree. + +- [ ] **Step 2: Run it** — `cd server/typescript/packages/metadata && bun test test/serializer-shared-document.test.ts` → FAIL (`serializeSharedDocument` not exported). + +- [ ] **Step 3: Implement (TS).** In `naming.ts`: `packageOfResolutionKey(fqn)` = the substring before the last `PACKAGE_SEPARATOR`, or `""` (reuse the helper at the existing `lastIndexOf(PACKAGE_SEPARATOR)` site if it already computes exactly this — export it under this name either way). In `serializer-json.ts`: +```ts +export function serializeSharedDocument(nodes: readonly MetaData[]): string { + const sorted = [...nodes].sort((a, b) => (a.resolutionKey() < b.resolutionKey() ? -1 : a.resolutionKey() > b.resolutionKey() ? 1 : 0)); + const children = sorted.map((node) => { + const pkg = packageOfResolutionKey(node.resolutionKey()); + if (pkg === "") throw new Error(`serializeSharedDocument: ${node.resolutionKey()} has no package; a shared document carries only packaged nodes`); + const parsed = JSON.parse(canonicalSerialize(node)) as Record>; + const [fused, body] = Object.entries(parsed)[0]!; + const { [RESERVED_KEY_NAME]: name, [RESERVED_KEY_PACKAGE]: _drop, ...rest } = body; + return { [fused]: { [RESERVED_KEY_NAME]: name, [RESERVED_KEY_PACKAGE]: pkg, ...rest } }; + }); + return JSON.stringify({ [`${TYPE_METADATA}.${SUBTYPE_ROOT}`]: { [RESERVED_KEY_CHILDREN]: children } }, null, 2) + "\n"; +} +``` +`canonicalSerialize(node)` on a top-level object gives `{ "object.entity": { … } }`; `RESERVED_KEY_*` come from `shared/structural.ts`; `TYPE_METADATA`/`SUBTYPE_ROOT` from `shared/base-types.ts`. Export from `index.ts`. + +- [ ] **Step 4: Run** → PASS. `bun run --filter '@metaobjectsdev/metadata' typecheck` → PASS. + +- [ ] **Step 5: Write the failing Python test** `tests/unit/test_serializer_shared_document.py`: +```python +import json +from pathlib import Path +from metaobjects import MetaDataLoader +from metaobjects.naming import package_of_resolution_key +from metaobjects.serializer_json import serialize_shared_document + +LIB = json.dumps({"metadata.root": {"package": "acme::common", "children": [ + {"object.entity": {"name": "Customer", "children": [ + {"source.rdb": {"@table": "customers"}}, {"field.long": {"name": "id"}}, + {"field.string": {"name": "email", "@maxLength": 120}}, + {"identity.primary": {"name": "pk", "@fields": ["id"]}}]}}, + {"object.entity": {"name": "Audited", "abstract": True, "children": [{"field.timestamp": {"name": "createdAt"}}]}}, + {"object.value": {"name": "Address", "children": [{"field.string": {"name": "street"}}, {"field.string": {"name": "city"}}]}}, +]}}) +ARTIFACT = Path(__file__).resolve().parents[3] / "fixtures" / "dependency-conformance" / "artifacts" / "acme-common-v1.json" + +def _objects(root): + return [c for c in root.own_children() if c.type == "object"] # ADR-0039 sanctioned own: root-level scan + +def test_package_of_resolution_key(): + assert package_of_resolution_key("acme::common::Customer") == "acme::common" + assert package_of_resolution_key("Customer") == "" + +def test_shared_document_is_byte_identical_to_the_pinned_artifact(): + res = MetaDataLoader.from_string(LIB, "json"); assert not res.errors + assert serialize_shared_document(_objects(res.root)) == ARTIFACT.read_text(encoding="utf-8") + +def test_shared_document_reloads_to_the_same_resolution_keys(): + res = MetaDataLoader.from_string(LIB, "json") + again = MetaDataLoader.from_string(serialize_shared_document(_objects(res.root)), "json") + assert not again.errors + assert sorted(o.resolution_key() for o in _objects(again.root)) == ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"] +``` +(Replace the literal `"object"` with `TYPE_OBJECT` from `metaobjects.shared.base_types` — constants discipline.) Run: `cd server/python && uv run --extra integration pytest tests/unit/test_serializer_shared_document.py -q` → FAIL (ImportError). + +- [ ] **Step 6: Implement (Python)** mirroring Step 3: `package_of_resolution_key` in `naming.py`; `serialize_shared_document(nodes)` in `serializer_json.py` building each body from `_to_canonical(node, False)` after `_rewrite_source_rdb_physical_names`, re-ordering to `name, package, …rest`, sorting nodes by `resolution_key()`, and returning `json.dumps({"metadata.root": {"children": children}}, indent=2, ensure_ascii=False) + "\n"` (use `KEY_NAME`/`KEY_PACKAGE`/`KEY_CHILDREN` and the `TYPE_METADATA`/`SUBTYPE_ROOT` constants). Raise `ValueError` for an empty package. + +- [ ] **Step 7: Run** → PASS. + +- [ ] **Step 8: Add the 4-port loader fixture.** `fixtures/conformance/xpkg-node-level-package-no-root-package/input/meta.shared.json` = the exact content of `acme-common-v1.json`; `expected.json` = the same bytes (the canonical serialization of a root with no package whose children carry their own package is the input itself). Run the TS and Python loader corpora and the two JVM/C# lanes for this fixture: +```bash +cd server/typescript/packages/metadata && bun test test/conformance.test.ts +cd server/python && uv run --extra integration pytest tests/conformance -q -k xpkg-node-level +scripts/ci-local.sh --only java-fast +scripts/ci-local.sh --only csharp +``` +Expected: all PASS. If Java or C# fails on this fixture, STOP and report the diff — do not alter the fixture to fit a port. + +- [ ] **Step 9: Commit** +``` +feat(metadata): serializeSharedDocument — node-level packages, no root package (FR-023) +``` + +--- + +### Task 3: `FileSource` takes an explicit id (TS + Python) + +**Files:** +- Modify: `server/typescript/packages/metadata/src/loader/sources/file-source.ts` +- Modify: `server/python/src/metaobjects/loader/sources/file_source.py` +- Test: `server/typescript/packages/metadata/test/file-source-id.test.ts`, `server/python/tests/unit/test_file_source_id.py` + +**Read first:** DESIGN §2.5 ("Source ids carry provenance") and §2.11. Today `FileSource.id` is `basename(path)` (TS) / `self._path.name` (Python); the parser stamps `source.files: []` on every node parsed from that source (ADR-0009). + +**Interfaces:** +- Produces TS: `new FileSource(path, { id?: string })` — `id` defaults to `basename(path)`. +- Produces Python: `FileSource(path, format=None, id=None)` — `id` defaults to the basename. + +- [ ] **Step 1: Failing TS test** +```ts +import { describe, expect, test } from "bun:test"; +import { mkdtemp, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { FileSource } from "../src/loader/sources/file-source.js"; +import { MetaDataLoader } from "../src/index.js"; + +describe("FileSource id", () => { + test("defaults to the basename and accepts an explicit id that reaches node provenance", async () => { + const dir = await mkdtemp(join(tmpdir(), "fs-id-")); + const p = join(dir, "meta.a.json"); + await writeFile(p, JSON.stringify({ "metadata.root": { package: "p", children: [ { "object.value": { name: "V" } } ] } })); + expect(new FileSource(p).id).toBe("meta.a.json"); + const src = new FileSource(p, { id: "dep:acme-common/acme-common.metaobjects.json" }); + expect(src.id).toBe("dep:acme-common/acme-common.metaobjects.json"); + const { root, errors } = await new MetaDataLoader().load([src]); + expect(errors).toEqual([]); + const v = root.objects()[0]!; + expect("files" in v.source ? v.source.files : []).toEqual(["dep:acme-common/acme-common.metaobjects.json"]); + }); +}); +``` +Run `cd server/typescript/packages/metadata && bun test test/file-source-id.test.ts` → FAIL (constructor takes one argument; `id` is the basename). + +- [ ] **Step 2: Implement** — `constructor(path: string, opts?: { id?: string })`; `this.id = opts?.id ?? basename(path)`. Run → PASS. `bun run --filter '@metaobjectsdev/metadata' typecheck` → PASS. + +- [ ] **Step 3: Failing Python test** `tests/unit/test_file_source_id.py`: +```python +import json +from metaobjects import MetaDataLoader +from metaobjects.loader.sources.file_source import FileSource + +def test_file_source_id_defaults_to_basename_and_accepts_override(tmp_path): + p = tmp_path / "meta.a.json" + p.write_text(json.dumps({"metadata.root": {"package": "p", "children": [{"object.value": {"name": "V"}}]}})) + assert FileSource(p).id == "meta.a.json" + src = FileSource(p, id="dep:acme-common/acme-common.metaobjects.json") + assert src.id == "dep:acme-common/acme-common.metaobjects.json" + res = MetaDataLoader().load([src]); assert not res.errors + node = next(c for c in res.root.own_children()) # ADR-0039 sanctioned own: root-level scan + assert node.source.files == ["dep:acme-common/acme-common.metaobjects.json"] +``` +Run → FAIL. Implement: `def __init__(self, path, format=None, id: str | None = None)`; `self._id = id`; property `id` returns `self._id or self._path.name`. Run → PASS. + +- [ ] **Step 4: Run the existing loader suites to prove nothing moved** +```bash +cd server/typescript/packages/metadata && bun test test/parser-source-population.test.ts test/conformance.test.ts +cd server/python && uv run --extra integration pytest tests/unit/test_loader.py tests/conformance -q +``` +Expected: PASS. + +- [ ] **Step 5: Commit** +``` +feat(loader): FileSource accepts an explicit source id (FR-023 provenance) +``` + +--- + +### Task 4: The scope-pattern grammar moves into `metadata`; Python gains `scope.py` + +**Files:** +- Create: `server/typescript/packages/metadata/src/scope.ts` (moved from `sdk/src/scope.ts`, unchanged logic) +- Modify: `server/typescript/packages/sdk/src/scope.ts` → becomes `export { compileScope, matchesScope } from "@metaobjectsdev/metadata"; export type { Scope, CompiledScope } from "@metaobjectsdev/metadata";` +- Modify: `server/typescript/packages/metadata/src/index.ts` (export the four names) +- Create: `server/python/src/metaobjects/scope.py` +- Test: `server/typescript/packages/metadata/test/scope-conformance.test.ts` (new copy of the sdk runner pointing at `../src/scope.js`), `server/python/tests/conformance/test_scope_conformance.py` + +**Read first:** DESIGN §4.3 (why codegen-ts needs the grammar without depending on sdk) and `docs/features/metadata-sources.md` "Pattern grammar". The corpus is `fixtures/scope-conformance/cases.json`; the existing TS runner is `sdk/test/scope-conformance.test.ts` (it must keep passing unchanged, through the re-export). + +**Interfaces:** +- Produces TS (from `@metaobjectsdev/metadata`): `compileScope(scope: Scope): CompiledScope`, `matchesScope(fqn: string, compiled: CompiledScope): boolean`, types `Scope { include?: string[]; exclude?: string[] }`, `CompiledScope`. +- Produces Python: `metaobjects.scope.compile_scope(include: list[str] | None, exclude: list[str] | None) -> CompiledScope`, `matches_scope(fqn: str, compiled: CompiledScope) -> bool`; an unparseable pattern raises `ParseError(code=ErrorCode.ERR_SCOPE_PATTERN_INVALID)`. + +- [ ] **Step 1: Failing TS test** — copy `sdk/test/scope-conformance.test.ts` to `metadata/test/scope-conformance.test.ts`, change the import to `../src/scope.js` and the corpus path to `../../../../../fixtures/scope-conformance/cases.json`. Run `cd server/typescript/packages/metadata && bun test test/scope-conformance.test.ts` → FAIL (module missing). + +- [ ] **Step 2: Move the module.** `git mv server/typescript/packages/sdk/src/scope.ts server/typescript/packages/metadata/src/scope.ts`; fix its imports to relative (`./errors.js`, `./source.js`, `./shared/structural.js`); export from `metadata/src/index.ts`; recreate `sdk/src/scope.ts` as the two-line re-export. Run both runners: +```bash +cd server/typescript/packages/metadata && bun test test/scope-conformance.test.ts +cd server/typescript/packages/sdk && bun test test/scope-conformance.test.ts test/scope.test.ts test/collection.test.ts +``` +Expected: PASS. Typecheck: `bun run --filter '@metaobjectsdev/metadata' typecheck && bun run --filter '@metaobjectsdev/sdk' typecheck` → PASS. The browser-safety test in `metadata` must stay green (`bun test test/browser-safe*.test.ts` if present — the module uses no `node:` imports). + +- [ ] **Step 3: Failing Python runner** `tests/conformance/test_scope_conformance.py` — parametrize over `fixtures/scope-conformance/cases.json` exactly as the TS runner does (read its assertions: `include`/`exclude`/`fqn`/`expect` or `expectError`), asserting `matches_scope(fqn, compile_scope(include, exclude)) == expect`, and `ParseError` with `ErrorCode.ERR_SCOPE_PATTERN_INVALID` for error cases. Run `cd server/python && uv run --extra integration pytest tests/conformance/test_scope_conformance.py -q` → FAIL (ImportError). + +- [ ] **Step 4: Implement `scope.py`** as a line-for-line port of `metadata/src/scope.ts` (segment split on `::`, `*` within a segment, `**` = one or more whole segments, literal everything else, include-union then exclude, empty include = everything, case-sensitive; compile to `re` patterns). Run → PASS. + +- [ ] **Step 5: Commit** +``` +refactor(metadata): scope-pattern grammar lives in metadata; Python port (FR-023) +``` + +--- + +### Task 5: `dependencies` in the config schema (TS sdk + Python neutral config) + +**Files:** +- Modify: `server/typescript/packages/sdk/src/config.ts` +- Create: `server/typescript/packages/sdk/src/dependencies.ts` (constants + `DependencySpec` types + `DependencySpecSchema` only — the rest lands in Task 6) +- Modify: `server/typescript/packages/sdk/src/index.ts` (export the new module) +- Modify: `server/python/src/metaobjects/config/neutral_config.py` +- Create: `server/python/src/metaobjects/config/dependencies.py` (constants only) +- Test: `server/typescript/packages/sdk/test/config.test.ts` (extend), `server/python/tests/config/test_neutral_config.py` (extend) + +**Read first:** DESIGN §3.1, §2.3. `sdk/src/config.ts` — `ConfigSchema` is `.strict()` at every level and `SourceSpecSchema` carries a two-direction parity guard with the hand-written `SourceSpec`; replicate both for `DependencySpec`. Python `neutral_config.py` reads only the neutral subset and IGNORES unknown top-level keys by design. + +**Interfaces:** +- Produces TS (`sdk/src/dependencies.ts`): `DEPS_DIR`, `LOCK_FILE`, `LOCAL_OVERRIDE_FILE`, `MANIFEST_FILE`, `ARTIFACT_SUFFIX`, `DEPENDENCY_SOURCE_ID_PREFIX`, `INTEGRITY_PREFIX`, `DEPENDENCY_MODES = ["reference", "own"] as const`, `DEFAULT_DEPENDENCY_MODE = "reference"`, `type DependencyMode`, `type DependencySpec = { name: string; mode: DependencyMode } & ({ path: string } | { npm: string; dir?: string } | { python: string; dir?: string })`, `DependencySpecSchema` (zod), `dependencyName(spec)`. `Config.dependencies: DependencySpec[]` (default `[]`). +- Produces Python (`config/dependencies.py`): the same constants; `neutral_config.NeutralConfig.dependencies: list[dict[str, str]]` (each dict already validated: `name`, exactly one transport key, `mode` defaulted to `"reference"`, optional `dir` only beside `npm`/`python`). + +- [ ] **Step 1: Failing TS tests** (append to `sdk/test/config.test.ts`): +```ts +test("dependencies: a valid spec parses with mode defaulted", () => { + const cfg = ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "acme-common", path: "../lib/metaobjects" }] }); + expect(cfg.dependencies).toEqual([{ name: "acme-common", path: "../lib/metaobjects", mode: "reference" }]); +}); +test("dependencies: npm and python accept dir; path does not", () => { + expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "a", npm: "@acme/model", dir: "metaobjects" }] })).not.toThrow(); + expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "a", python: "acme_model", dir: "metaobjects", mode: "own" }] })).not.toThrow(); + expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "a", path: "x", dir: "y" }] })).toThrow(); +}); +test("dependencies: two transports, no transport, a bad name, a bad mode, a duplicate name, an unknown key are all refused", () => { + for (const bad of [ + [{ name: "a", path: "x", npm: "y" }], [{ name: "a" }], [{ name: "Bad Name", path: "x" }], + [{ name: "a", path: "x", mode: "shared" }], [{ name: "a", path: "x" }, { name: "a", npm: "y" }], [{ name: "a", path: "x", pathh: "typo" }], + ]) expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: bad })).toThrow(); +}); +test("dependencies: absent means []", () => { expect(ConfigSchema.parse({ schema_version: 1 }).dependencies).toEqual([]); }); +``` +Run `cd server/typescript/packages/sdk && bun test test/config.test.ts` → FAIL. + +- [ ] **Step 2: Implement.** In `dependencies.ts` define the constants and: +```ts +const DependencyName = z.string().regex(/^[a-z0-9][a-z0-9._-]*$/); +const Mode = z.enum(DEPENDENCY_MODES).default(DEFAULT_DEPENDENCY_MODE); +export const DependencySpecSchema = z.union([ + z.object({ name: DependencyName, path: z.string().min(1), mode: Mode }).strict(), + z.object({ name: DependencyName, npm: z.string().min(1), dir: z.string().min(1).optional(), mode: Mode }).strict(), + z.object({ name: DependencyName, python: z.string().min(1), dir: z.string().min(1).optional(), mode: Mode }).strict(), +]); +``` +In `config.ts`: `dependencies: z.array(DependencySpecSchema).default([]).refine((a) => new Set(a.map((d) => d.name)).size === a.length, { message: "dependencies: names must be unique" })`, plus the two-direction parity assignments between `z.infer` and the hand-written `DependencySpec` (copy the `_sourceSpecParity*` pattern). Export everything from `index.ts`. Run → PASS; `bun run --filter '@metaobjectsdev/sdk' typecheck` → PASS; `bun test test/no-hardcoded-metadata-dir.test.ts` → PASS. + +- [ ] **Step 3: Failing Python tests** (append to `tests/config/test_neutral_config.py`): +```python +def test_dependencies_parse_with_mode_default(tmp_path): + _write(tmp_path, {"schema_version": 1, "sources": [], "dependencies": [{"name": "acme-common", "path": "../lib"}]}) + cfg = read_neutral_config(tmp_path) + assert cfg.dependencies == [{"name": "acme-common", "path": "../lib", "mode": "reference"}] + +def test_dependencies_absent_is_empty(tmp_path): + _write(tmp_path, {"schema_version": 1, "sources": []}) + assert read_neutral_config(tmp_path).dependencies == [] + +@pytest.mark.parametrize("bad", [ + [{"name": "a", "path": "x", "npm": "y"}], [{"name": "a"}], [{"name": "Bad Name", "path": "x"}], + [{"name": "a", "path": "x", "mode": "shared"}], [{"name": "a", "path": "x"}, {"name": "a", "npm": "y"}], + [{"name": "a", "path": "x", "dir": "y"}], [{"name": "a", "path": "x", "pathh": "t"}], +]) +def test_dependencies_shape_errors(tmp_path, bad): + _write(tmp_path, {"schema_version": 1, "sources": [], "dependencies": bad}) + with pytest.raises(ParseError) as e: read_neutral_config(tmp_path) + assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND +``` +(`_write` writes `.metaobjects/config.json` under `tmp_path`; define it in the test module if the file has no helper.) The malformed-config code is `ERR_COLLECTION_NOT_FOUND`, the code this reader already uses for every shape error. Run `cd server/python && uv run --extra integration pytest tests/config/test_neutral_config.py -q` → FAIL. + +- [ ] **Step 4: Implement** — add `dependencies` to `NeutralConfig`; in `read_neutral_config` validate each entry (dict; `name` matching `^[a-z0-9][a-z0-9._-]*$`; exactly one of `path`/`npm`/`python`, a non-empty string; `dir` only with `npm`/`python`; `mode` in `DEPENDENCY_MODES`, default `"reference"`; no other keys; names unique). Run → PASS. + +- [ ] **Step 5: Commit** +``` +feat(config): `dependencies` in .metaobjects/config.json — sdk schema and Python neutral reader (FR-023) +``` + +--- + +### Task 6: Lock and manifest schemas, integrity hashing, and the resolved-dependency type (TS + Python) + +**Files:** +- Modify: `server/typescript/packages/sdk/src/dependencies.ts` +- Modify: `server/python/src/metaobjects/config/dependencies.py` +- Test: `server/typescript/packages/sdk/test/dependencies.test.ts`, `server/python/tests/config/test_dependencies.py` + +**Read first:** DESIGN §3.2, §3.3, §3.4, §4.2 steps 2–3. + +**Interfaces:** +- Produces TS: `LockSchema` / `type Lock`, `ManifestSchema` / `type Manifest`, `LocalOverrideSchema` / `type LocalOverrides` (`Record`), `sha256Integrity(bytes: Uint8Array): string`, `dependencySourceId(name: string, artifact: string): string` (`dep:/`), `readLock(configDir: string): Promise`, `writeLock(configDir: string, lock: Lock): Promise` (keys sorted, 2-space, trailing newline), `readLocalOverrides(configDir: string): Promise` (`{}` when absent), `interface ResolvedDependency { name; version; metamodelVersion; mode; packages: readonly string[]; nodes: readonly string[]; artifactPath: string; sourceId: string; override?: string }`. +- Produces Python: `sha256_integrity(b: bytes) -> str`, `dependency_source_id(name, artifact)`, `read_lock(config_dir) -> dict | None`, `read_local_overrides(config_dir) -> dict`, `ResolvedDependency` dataclass with the same fields, `validate_lock(raw) -> dict` and `validate_manifest(raw) -> dict` raising `ParseError(ERR_DEPENDENCY_SNAPSHOT_STALE)` / `ParseError(ERR_DEPENDENCY_MANIFEST_INVALID)` on shape errors. + +- [ ] **Step 1: Failing TS tests** `sdk/test/dependencies.test.ts`: +```ts +import { describe, expect, test } from "bun:test"; +import { readFile } from "node:fs/promises"; +import { LockSchema, ManifestSchema, sha256Integrity, dependencySourceId, INTEGRITY_PREFIX } from "../src/dependencies.js"; + +const ARTIFACT = `${import.meta.dir}/../../../../../fixtures/dependency-conformance/artifacts/acme-common-v1.json`; + +describe("dependency lock + manifest", () => { + test("integrity hashes the pinned artifact to the README value", async () => { + expect(sha256Integrity(new Uint8Array(await readFile(ARTIFACT)))).toBe("sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d"); + expect(INTEGRITY_PREFIX).toBe("sha256-"); + }); + test("source id", () => { expect(dependencySourceId("acme-common", "acme-common.metaobjects.json")).toBe("dep:acme-common/acme-common.metaobjects.json"); }); + const manifest = { schema_version: 1, name: "acme-common", version: "1.0.0", metamodelVersion: "1.0", artifact: "acme-common.metaobjects.json", + integrity: "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", packages: ["acme::common"], nodes: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"] }; + test("manifest parses; an unknown key, an unsorted nodes list, a bad integrity prefix are refused", () => { + expect(() => ManifestSchema.parse(manifest)).not.toThrow(); + expect(() => ManifestSchema.parse({ ...manifest, extra: 1 })).toThrow(); + expect(() => ManifestSchema.parse({ ...manifest, nodes: ["acme::common::Customer", "acme::common::Address"] })).toThrow(); + expect(() => ManifestSchema.parse({ ...manifest, integrity: "md5-abc" })).toThrow(); + }); + test("lock parses; keys must be sorted; resolvedFrom carries exactly one transport", () => { + const entry = { version: "1.0.0", metamodelVersion: "1.0", mode: "reference", resolvedFrom: { path: "../lib" }, artifact: "acme-common.metaobjects.json", + integrity: manifest.integrity, packages: ["acme::common"], nodes: manifest.nodes }; + expect(() => LockSchema.parse({ schema_version: 1, dependencies: { "acme-common": entry } })).not.toThrow(); + expect(() => LockSchema.parse({ schema_version: 1, dependencies: { "b": entry, "a": entry } })).toThrow(/sorted/); + expect(() => LockSchema.parse({ schema_version: 1, dependencies: { "a": { ...entry, resolvedFrom: { path: "x", npm: "y" } } } })).toThrow(); + }); +}); +``` +Run `cd server/typescript/packages/sdk && bun test test/dependencies.test.ts` → FAIL. + +- [ ] **Step 2: Implement** with zod (`.strict()` everywhere): `IntegritySchema = z.string().regex(/^sha256-[0-9a-f]{64}$/)`; `ManifestSchema` with `schema_version: z.literal(1)`, `name: DependencyName`, `version: z.string().min(1)`, `metamodelVersion: z.string().regex(/^\d+\.\d+$/)`, `artifact: z.string().endsWith(ARTIFACT_SUFFIX)`, `integrity`, `packages`/`nodes` as `z.array(z.string().min(1)).refine(sorted)`; `LockEntrySchema` = the manifest fields minus `schema_version`/`name` plus `mode` and `resolvedFrom` (a strict union `{path}|{npm,dir?}|{python,dir?}`); `LockSchema = { schema_version: 1, dependencies: z.record(DependencyName, LockEntrySchema) }.refine(keys sorted, { message: "deps.lock.json: dependency keys must be sorted" })`; `LocalOverrideSchema = z.record(DependencyName, z.object({ path: z.string().min(1) }).strict())`; `sha256Integrity` via `createHash("sha256")`; `readLock`/`writeLock`/`readLocalOverrides` over `join(configDir, DEFAULT_METAOBJECTS_DIR, LOCK_FILE | LOCAL_OVERRIDE_FILE)` (`DEFAULT_METAOBJECTS_DIR` from `./metadata-files.js`). Run → PASS; typecheck → PASS. + +- [ ] **Step 3: Failing Python tests** `tests/config/test_dependencies.py`: +```python +from pathlib import Path +import pytest +from metaobjects.config.dependencies import (sha256_integrity, dependency_source_id, validate_manifest, validate_lock, INTEGRITY_PREFIX) +from metaobjects.errors import ErrorCode, ParseError +ARTIFACT = Path(__file__).resolve().parents[3] / "fixtures" / "dependency-conformance" / "artifacts" / "acme-common-v1.json" +MANIFEST = {"schema_version": 1, "name": "acme-common", "version": "1.0.0", "metamodelVersion": "1.0", "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", "packages": ["acme::common"], + "nodes": ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"]} + +def test_integrity_matches_pinned_value(): + assert sha256_integrity(ARTIFACT.read_bytes()) == MANIFEST["integrity"]; assert INTEGRITY_PREFIX == "sha256-" + +def test_source_id(): + assert dependency_source_id("acme-common", "acme-common.metaobjects.json") == "dep:acme-common/acme-common.metaobjects.json" + +def test_manifest_shape_errors(): + validate_manifest(MANIFEST) + for bad in [{**MANIFEST, "extra": 1}, {**MANIFEST, "nodes": list(reversed(MANIFEST["nodes"]))}, {**MANIFEST, "integrity": "md5-x"}]: + with pytest.raises(ParseError) as e: validate_manifest(bad) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_MANIFEST_INVALID + +def test_lock_shape_errors(): + entry = {k: v for k, v in MANIFEST.items() if k not in ("schema_version", "name")} | {"mode": "reference", "resolvedFrom": {"path": "../lib"}} + validate_lock({"schema_version": 1, "dependencies": {"acme-common": entry}}) + with pytest.raises(ParseError) as e: validate_lock({"schema_version": 1, "dependencies": {"b": entry, "a": entry}}) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE +``` +Run → FAIL. Implement the module (hand-validated dicts, same rules as the zod schemas; `hashlib.sha256`). Run → PASS. + +- [ ] **Step 4: Commit** +``` +feat(deps): lock, manifest and override schemas with sha256 integrity (FR-023) +``` + +--- + +### Task 7: `resolveCollection` reads the lock and snapshot; `loadMemory` takes `fileIds`; every CLI load site passes both + +**Files:** +- Modify: `server/typescript/packages/sdk/src/dependencies.ts` (`verifySnapshot`, `foreignOwnerFactory`) +- Modify: `server/typescript/packages/sdk/src/collection.ts` +- Modify: `server/typescript/packages/sdk/src/sources.ts` (`ResolvedSource` gains optional `dependency?: string`, `id?: string`) +- Modify: `server/typescript/packages/sdk/src/memory.ts` (`LoadMemoryOptions.fileIds`) +- Create: `server/typescript/packages/cli/src/lib/collection-load-options.ts` +- Modify: every `loadMemory(` call in `server/typescript/packages/cli/src/commands/{gen,verify,migrate,docs,export,prompt-snapshot,upgrade,types}.ts` and `cli/src/lib/*.ts` to spread `...collectionLoadOptions(collection)` instead of `files: collection.files` +- Modify: `fixtures/dependency-conformance/cases.json` (append cases 2, 3, 20–27 below) +- Test: `sdk/test/dependency-conformance.test.ts` (from Task 1), `sdk/test/collection.test.ts`, `sdk/test/order-independence.test.ts`, `sdk/test/memory.test.ts` + +**Read first:** DESIGN §4.2, §2.3 ("load-time checks"), §3.3, §3.4. `collection.ts` — `resolveCollection` is THE authority; extend it after the own-source resolution. Do NOT implement overrides yet (Task 17) — `readLocalOverrides` is called but an active override is treated as an error `ERR_DEPENDENCY_UNRESOLVED` with message `local overrides are not supported yet` until Task 17 replaces that branch (write that branch so Task 17's diff is a replacement, not a rewrite). + +**Interfaces:** +- Produces: `Collection` gains `dependencies: readonly ResolvedDependency[]`, `ownFiles: readonly string[]`, `fileIds: ReadonlyMap` (entries for dependency artifacts only), `overrides: readonly string[]` (empty until Task 17), `foreignOwner(fqn: string): ResolvedDependency | undefined`, `governs(fqn: string): boolean`. `files` = artifacts (name order) then own files. `verifySnapshot(configDir, specs, lock, overrides): Promise` throws `ParseError` codes per the table below. `loadMemory(root, { files, fileIds })` builds `new FileSource(p, { id: fileIds.get(p) })`. `collectionLoadOptions(collection)` returns `{ files: collection.files, fileIds: collection.fileIds }`. + +**Load-time check table (`verifySnapshot`):** config declares dependencies but no lock → `ERR_DEPENDENCY_SNAPSHOT_STALE`; a lock entry with no config spec, or a spec with no lock entry → `ERR_DEPENDENCY_SNAPSHOT_STALE`; lock `mode` ≠ spec `mode` → `ERR_DEPENDENCY_SNAPSHOT_STALE`; artifact file missing → `ERR_DEPENDENCY_SNAPSHOT_STALE`; `sha256Integrity(bytes) !== entry.integrity` → `ERR_DEPENDENCY_SNAPSHOT_STALE`; lock `metamodelVersion` major ≠ `METAMODEL_VERSION` major → `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`; two entries whose `nodes` intersect → `ERR_DEPENDENCY_NODE_COLLISION`. Every message ends with `run \`meta deps sync\`` for the stale family. A config with `dependencies: []` and no lock is fine; a lock with zero entries and no dependencies is fine. + +- [ ] **Step 1: Append the corpus cases.** The lock entry used below is (`LOCK_V1`): +```json +{ "schema_version": 1, "dependencies": { "acme-common": { "version": "1.0.0", "metamodelVersion": "1.0", "mode": "reference", "resolvedFrom": { "path": "../acme-common/metaobjects" }, "artifact": "acme-common.metaobjects.json", "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", "packages": ["acme::common"], "nodes": ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"] } } } +``` +the config `CONFIG_REF` = `{ "schema_version": 1, "sources": [], "dependencies": [{ "name": "acme-common", "path": "../acme-common/metaobjects", "mode": "reference" }] }`, the snapshot entry `SNAP` = `"treeFiles": { ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" }`, and the app file `APP` = `"metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}"`. (The `path` transport need not exist on disk — only `meta deps sync` resolves transports.) Add: + - `a-dependency-adds-its-artifact-to-the-resolved-set`: tree `APP` + `SNAP`, `CONFIG_REF`, `LOCK_V1`; `expectFiles: [".metaobjects/deps/acme-common/acme-common.metaobjects.json", "metaobjects/meta.app.json"]`, `expectForeign: ["acme::common::Address","acme::common::Audited","acme::common::Customer"]`, `expectGoverned: ["app::Order"]`. + - `dependency-order-in-config-does-not-change-the-resolved-set`: two dependencies `acme-common` and `acme-extra` (a second artifact: put in `tree` the string of `acme-common-v1.json` with every `acme::common` replaced by `acme::extra`, at `.metaobjects/deps/acme-extra/acme-extra.metaobjects.json`; compute its hash with the README one-liner and pin it in this case's lock — record the value in the case as `"integrity"`), declared `acme-extra` first in the config; `expectFiles` = both artifacts + `APP`; `expectForeign` = all six FQNs. + - `a-missing-lock-is-stale`: `APP` + `SNAP`, `CONFIG_REF`, no lock → `expectError: "ERR_DEPENDENCY_SNAPSHOT_STALE"`. + - `a-lock-entry-without-a-config-entry-is-stale`: `APP` + `SNAP`, config `{ "schema_version": 1, "sources": [] }`, `LOCK_V1` → `ERR_DEPENDENCY_SNAPSHOT_STALE`. + - `a-config-entry-without-a-lock-entry-is-stale`: `APP`, `CONFIG_REF`, lock `{ "schema_version": 1, "dependencies": {} }` → `ERR_DEPENDENCY_SNAPSHOT_STALE`. + - `a-missing-artifact-is-stale`: `APP` (no `SNAP`), `CONFIG_REF`, `LOCK_V1` → `ERR_DEPENDENCY_SNAPSHOT_STALE`. + - `an-artifact-whose-hash-differs-is-stale`: `APP` + `treeFiles` pointing the snapshot path at `artifacts/acme-common-v1-widened.json`, `CONFIG_REF`, `LOCK_V1` → `ERR_DEPENDENCY_SNAPSHOT_STALE`. + - `two-dependencies-exporting-one-fqn-collide`: two dependencies whose artifacts are byte-identical copies of `acme-common-v1.json` (`acme-common` and `acme-dup`, both `treeFiles` → `artifacts/acme-common-v1.json`, both lock entries with the v1 hash and the same `nodes`) → `ERR_DEPENDENCY_NODE_COLLISION`. + - `an-incompatible-metamodel-major-is-refused`: `LOCK_V1` with `"metamodelVersion": "2.0"` → `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`. + - `dependencies-are-read-under-a-native-source-surface`: `resolveFrom: "app"`, tree `app/metaobjects/meta.app.json` (APP content) + `app/.metaobjects/deps/acme-common/…` via `treeFiles`, config `CONFIG_REF` under `app/`, `LOCK_V1` → `expectFiles: ["app/.metaobjects/deps/acme-common/acme-common.metaobjects.json", "app/metaobjects/meta.app.json"]`. (The Python runner in Task 18 additionally writes a `metaobjects.config.yaml` with `metadata: metaobjects` beside it to prove the rung-2 path still reads dependencies.) + +- [ ] **Step 2: Run the corpus** → the new cases FAIL (`collection.dependencies` undefined / no stale detection). + +- [ ] **Step 3: Failing unit tests.** Append to `sdk/test/memory.test.ts`: `loadMemory` with `fileIds` yields a node whose `source.files[0]` is the mapped id (materialize a temp project whose only file is mapped to `"dep:x/x.metaobjects.json"`). Append to `order-independence.test.ts`: permuting `dependencies` in the config leaves `collection.files` and `collection.fileIds` identical (two dependencies as in the corpus case). Run → FAIL. + +- [ ] **Step 4: Implement.** `verifySnapshot` per the table (read `.metaobjects/deps//` bytes; compare; build `ResolvedDependency` with `artifactPath` absolute and `sourceId = dependencySourceId(name, artifact)`); `foreignOwnerFactory(deps)` builds a `Map` once. In `resolveCollection`: after `sources` are resolved, `const lock = await readLock(configDir); const overrides = await readLocalOverrides(configDir); const deps = specs.length === 0 && lock === undefined ? [] : await verifySnapshot(configDir, cfg.dependencies, lock, overrides)`; `files = [...deps.map(d => d.artifactPath), ...own]`; `fileIds = new Map(deps.map(d => [d.artifactPath, d.sourceId]))`; `foreignOwner`, `governs`, `ownFiles`, `overrides: []`. A project with no config carries `dependencies: []`. `loadMemory`: `paths.map((p) => new FileSource(p, fileIds?.has(p) ? { id: fileIds.get(p) } : undefined))` — note `FileSource` is imported from `@metaobjectsdev/metadata/core`. Create `collectionLoadOptions` and sweep every `loadMemory(` call in `cli/src` (grep `loadMemory(`): each becomes `loadMemory(dir, { ...collectionLoadOptions(collection), ...loadMemoryOptionsFrom(cfg) })`. The requirements line in `verify.ts` (`counted over N metadata file(s)`) gains `, M from dependencies` where `M = collection.dependencies.length`. + +- [ ] **Step 5: Run** +```bash +cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts test/collection.test.ts test/order-independence.test.ts test/memory.test.ts test/no-hardcoded-metadata-dir.test.ts test/source-resolution-conformance.test.ts +cd server/typescript/packages/cli && bun test test/collection-routing.test.ts test/verify-requirements-e2e.test.ts test/gen-list.test.ts +bun run --filter '@metaobjectsdev/sdk' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck +``` +Expected: PASS (cases 1, 2, 3, 20–27 green; the rest still red/skipped). + +- [ ] **Step 6: Commit** +``` +feat(sdk): resolveCollection loads the dependency snapshot behind the lock; loadMemory takes fileIds (FR-023) +``` + +--- + +### Task 8: The boundary validator and `declaredOverlayKeys` + +**Files:** +- Modify: `server/typescript/packages/metadata/src/loader/meta-data-loader.ts` (export `declaredOverlayKeys`) +- Modify: `server/typescript/packages/sdk/src/dependencies.ts` (`validateDependencyBoundary`) +- Modify: `server/typescript/packages/cli/src/lib/collection-load-options.ts` (add `loadCollection(dir, collection, extra)` = `loadMemory` + boundary validation, used by every command) +- Modify: the eight CLI load sites to call `loadCollection` (from Task 7's sweep) +- Modify: `fixtures/dependency-conformance/cases.json` (append cases 8, 10–14) +- Test: `sdk/test/dependency-conformance.test.ts`, `sdk/test/dependencies.test.ts`, `metadata/test/declared-overlay-keys.test.ts` + +**Read first:** DESIGN §2.4 (both rules), §4.4. In `meta-data-loader.ts`, `_partitionOverlayLast` / `_isOverlayOnlySource` / `_rootIsOverlayOnly` already parse a source's raw content and inspect top-level `overlay: true` — reuse that walker. TPH: a subtype declares `@discriminatorValue`; its discriminator base is the nearest `extends` ancestor carrying `@discriminator` (see `migrate-ts/src/expected-schema.ts` `discriminatorBaseOf`; re-implement the two-line walk in sdk with resolving accessors — do not import migrate-ts). + +**Interfaces:** +- Produces TS: `declaredOverlayKeys(content: string, format: MetaDataFormat, fileDefaultPackage?: string): Promise>` — resolution keys (`::`, using the node's own `package` else the document root's) of top-level nodes carrying `overlay: true`. `validateDependencyBoundary(root: MetaRoot, collection: Collection, overlayKeys: ReadonlySet): void` — throws `ParseError` with the codes below. `loadCollection(startDir, collection, options)` in cli: loads, computes `overlayKeys` over `collection.ownFiles`, validates, returns the root. + +**Rules (DESIGN §4.4):** for each top-level node `n` (ADR-0039 sanctioned own: `root.ownChildren()` — the root is never extended) with `owner = collection.foreignOwner(n.resolutionKey())`: +1. `owner` defined, and `n` has a local contribution — `n.source.format === "merged"` with a contributor file not starting with `dep:`, or `n.source.files` contains a non-`dep:` entry — and `n.resolutionKey() ∉ overlayKeys` → `ERR_DEPENDENCY_OVERLAY_IMPLICIT`, message ` redeclares a node of dependency '' without 'overlay: true' — an amendment of a dependency's node must fail loudly when that node disappears; add 'overlay: true'`. +2. `owner.mode === "reference"` and any of: a child `c` of `n` (ADR-0039 sanctioned own: `n.ownChildren()` — the local contribution layer) whose `c.source.files` has a non-`dep:` entry and `c.type ∈ {field, identity, index, relationship, source}`; OR a locally-contributed attr among `@table`/`@schema`/`@column`/`@kind` (detect via `n.source.format === "merged"` and the attr being set — a merged node cannot say which contributor set an attr, so refuse when the node is merged with a local contributor AND carries any of those four attrs whose value differs from the artifact's: read the artifact's own copy by loading `collection.dependencies` artifact standalone once per validation, cached); OR `n` is local (no owner), declares `@discriminatorValue`, and its discriminator base's owner is `reference` → `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`, message `: a reference-mode dependency's node cannot gain here — declare your own object in your own package and extend it, or take mode "own" if you instantiate this model`. +Use `isMetaObject` from `@metaobjectsdev/metadata` and node `type` comparisons against `TYPE_FIELD`/`TYPE_IDENTITY`/`TYPE_INDEX`/`TYPE_RELATIONSHIP`/`TYPE_SOURCE` constants — never `instanceof`. + +- [ ] **Step 1: Failing metadata test** `declared-overlay-keys.test.ts`: a JSON document with root package `acme::common` and children `{ "object.entity": { name: "Customer", overlay: true } }` and `{ "object.value": { name: "Local" } }` → `declaredOverlayKeys(content, "json")` equals `new Set(["acme::common::Customer"])`; a YAML document (sigil-free `metadata:` root, `overlay: true` bare) gives the same; a node with its own `package: other` yields `other::Customer`. Run `cd server/typescript/packages/metadata && bun test test/declared-overlay-keys.test.ts` → FAIL. Implement by extracting the walk from `_rootIsOverlayOnly` into an exported function that returns keys; keep `_rootIsOverlayOnly` delegating to it. Run → PASS. + +- [ ] **Step 2: Append corpus cases** (all with `CONFIG_REF`, `LOCK_V1`, `SNAP` from Task 7 unless stated; `expectFiles` = the artifact + the app file; put the local file at `metaobjects/meta.app.json`): + - `a-local-node-may-live-in-the-dependency-package`: `{"metadata.root":{"package":"acme::common","children":[{"object.value":{"name":"LocalNote","children":[{"field.string":{"name":"text"}}]}}]}}` → `expectForeign` the three, `expectGoverned: ["acme::common::LocalNote"]`. + - `a-flagged-overlay-of-a-foreign-node-adds-presentation`: `{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"email","children":[{"view.text":{}}]}}]}}]}}` → loads clean; `expectGoverned: []`; `expectForeign` the three. (If `view.text` is not a registered subtype in this build, use `{"validator.length":{"@max":120}}` under the field instead — check `meta types view` first and use a registered presentation/validation child.) + - `an-unflagged-redeclaration-of-a-foreign-node-is-implicit`: same as above without `"overlay": true` → `expectFiles` + `expectLoadError: "ERR_DEPENDENCY_OVERLAY_IMPLICIT"`. + - `a-structural-overlay-of-a-reference-node-is-not-owned`: `{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"nickname"}}]}}]}}` → `expectLoadError: "ERR_DEPENDENCY_SCHEMA_NOT_OWNED"`. + - `a-structural-overlay-of-an-own-node-is-allowed`: same local file, config with `"mode": "own"` and `LOCK_V1` with `"mode": "own"` → loads clean; `expectGoverned: ["acme::common::Address","acme::common::Audited","acme::common::Customer"]`. + - `a-tph-subtype-of-a-foreign-reference-base-is-not-owned`: artifact-side base needs `@discriminator` — this case uses its OWN artifact in `tree` (not `treeFiles`): the v1 artifact text with `Customer`'s body gaining `"@discriminator": "kind"` and a child `{"field.enum":{"name":"kind","@values":["retail","corporate"]}}` (insert the attr after `"package"`, the child after `source.rdb`; compute and pin the hash in this case's lock); local file `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"CorporateCustomer","extends":"acme::common::Customer","@discriminatorValue":"corporate","children":[{"field.string":{"name":"vatId"}}]}}]}}` → `expectLoadError: "ERR_DEPENDENCY_SCHEMA_NOT_OWNED"`. + +- [ ] **Step 3: Run the corpus** → the six cases FAIL. Implement `validateDependencyBoundary` + `loadCollection` and the sweep. Run: +```bash +cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts test/dependencies.test.ts +cd server/typescript/packages/cli && bun test test/collection-routing.test.ts test/verify-requirements-e2e.test.ts +bun run --filter '@metaobjectsdev/sdk' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck +``` +Expected: PASS (cases 8, 10–14 green). + +- [ ] **Step 4: Commit** +``` +feat(sdk): dependency boundary validator — explicit overlays, reference-mode schema ownership (FR-023) +``` + +--- + +### Task 9: Codegen selection excludes foreign nodes (`gen`, `verify --codegen`, shared enums) + +**Files:** +- Modify: `server/typescript/packages/cli/src/commands/gen.ts` (the `runGen({ scope })` call; the explicit-entity refusal) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (`computeCodegenDrift(..., genCollection.inScope)` → the composed predicate) +- Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (`RunGenOptions.governs?: (fqn: string) => boolean`; thread into `renderSharedEnumsFile`) +- Modify: `server/typescript/packages/codegen-ts/src/templates/enums-file.ts` (`renderSharedEnumsFile(root, opts?: { exclude?: (fqn: string) => boolean })`) +- Modify: `server/typescript/packages/codegen-ts/src/generators/entity-file.ts` (pass the exclusion) +- Modify: `fixtures/dependency-conformance/cases.json` (append cases 4–7, 9) +- Test: `cli/test/gen-foreign-nodes.test.ts` (new), `codegen-ts/test/shared-enums-foreign.test.ts` (new), `sdk/test/dependency-conformance.test.ts` + +**Read first:** DESIGN §2.7 (the table), §4.5. `runner.ts` "single choke point for entity selection" (`opts.scope`); `gen.ts` passes `scope: genCollection.inScope`; `renderSharedEnumsFile(ctx.loadedRoot)` reads the whole loaded root. + +**Interfaces:** +- Consumes: `Collection.governs`, `Collection.foreignOwner` (Task 7), `loadCollection` (Task 8). +- Produces: `runGen({ scope, governs })` — the runner composes `scope(fqn) && governs(fqn)` at the choke point and passes `exclude: (fqn) => !governs(fqn)` to the shared-enums renderer; `gen` prints `meta gen — N node(s) from dependencies not generated here (reference mode): ` when N > 0 (stderr, once); `meta gen ` in reference mode exits 2 with ` is declared by dependency '' (reference mode) and is not generated here`. + +- [ ] **Step 1: Append corpus cases** (`CONFIG_REF`, `LOCK_V1`, `SNAP`): + - `a-foreign-node-is-referenceable-by-fqn-from-a-local-entity`: local `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"Order","children":[{"source.rdb":{"@table":"orders"}},{"field.long":{"name":"id"}},{"field.object":{"name":"shipTo","@objectRef":"acme::common::Address","@storage":"jsonb"}},{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}` → `expectGoverned: ["app::Order"]`, `expectForeign` the three. + - `a-foreign-node-is-not-selected-in-reference-mode`: `APP` → `expectGoverned: ["app::Order"]`. + - `a-foreign-node-is-selected-in-own-mode`: `APP`, config + lock with `mode: "own"` → `expectGoverned: ["acme::common::Address","acme::common::Audited","acme::common::Customer","app::Order"]`. + - `a-local-object-may-extend-a-foreign-abstract`: local `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"Ticket","extends":"acme::common::Audited","children":[{"source.rdb":{"@table":"tickets"}},{"field.long":{"name":"id"}},{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}` → `expectGoverned: ["app::Ticket"]`. + - `a-bare-reference-does-not-reach-a-foreign-node`: local `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"Order","children":[{"source.rdb":{"@table":"orders"}},{"field.long":{"name":"id"}},{"field.object":{"name":"shipTo","@objectRef":"Address","@storage":"jsonb"}},{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}` → `expectFiles` + `expectLoadError: "ERR_UNRESOLVED_OBJECT_REF"`. + Run the corpus → these pass already for the `expectGoverned` arm IF Task 7's `governs` is right (they exercise the predicate, not the CLI). Confirm they are green; if any is red, fix `governs` here. + +- [ ] **Step 2: Failing codegen-ts test** `shared-enums-foreign.test.ts`: load a root (via `MetaDataLoader.fromString`) with two package-level abstract `field.enum`s, `acme::common::Kind` (`@values: ["a","b"]`) consumed by a local entity's field via `extends`, and `acme::common::Unused`; assert `renderSharedEnumsFile(root)` names both, and `renderSharedEnumsFile(root, { exclude: (fqn) => fqn.startsWith("acme::common::") })` returns `null` (nothing materialized). Run `cd server/typescript/packages/codegen-ts && bun test test/shared-enums-foreign.test.ts` → FAIL. Implement the option in `enums-file.ts` (filter `materializedSharedEnums(root)` by `!exclude(e.resolutionKey())`) and thread `governs` from `RunGenOptions` through the runner into the `entityFile` generator's call. Run → PASS. + +- [ ] **Step 3: Failing cli test** `gen-foreign-nodes.test.ts` — scaffold a temp consumer project exactly like corpus case `a-foreign-node-is-referenceable-by-fqn-from-a-local-entity` (copy the pinned artifact from `fixtures/dependency-conformance/artifacts/acme-common-v1.json`, write `CONFIG_REF` and `LOCK_V1`), plus a minimal `metaobjects.config.ts` with `entityFile()` from an owned generator (mirror how `cli/test/gen-list.test.ts` or `gen-split-tree-single-import.test.ts` scaffolds a config) and run `genCommand([], cwd)`. Assert: exit 0; the output directory contains `Order.ts` and NOT `Address.ts`/`Customer.ts`/`Audited.ts`; stderr contains `3 node(s) from dependencies not generated here`. Second test: `genCommand(["Customer"], cwd)` exits 2 and stderr contains `declared by dependency 'acme-common' (reference mode)`. Third test: flip config + lock to `mode: "own"` → `Customer.ts` IS emitted. Run → FAIL. Implement in `gen.ts` (compute `foreign = root.objects().filter(o => !genCollection.governs(o.resolutionKey()))`; refuse an explicit entity whose owner is reference-mode; pass `governs: genCollection.governs`) and mirror in `verify.ts`'s `computeCodegenDrift` call. Run → PASS. + +- [ ] **Step 4: Run** +```bash +cd server/typescript/packages/cli && bun test test/gen-foreign-nodes.test.ts test/gen-list.test.ts test/gen-split-tree-single-import.test.ts test/verify-requirements-e2e.test.ts +cd server/typescript/packages/codegen-ts && bun test test/shared-enums-foreign.test.ts test/ai-llm-call-codegen.test.ts +bun run --filter '@metaobjectsdev/codegen-ts' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck +``` +Expected: PASS. + +- [ ] **Step 5: Commit** +``` +feat(cli,codegen-ts): reference-mode dependency nodes are never selected for codegen (FR-023) +``` + +--- + +### Task 10: Migrate and `verify --db` never govern reference-mode nodes + +**Files:** +- Modify: `server/typescript/packages/cli/src/lib/migrate-scope.ts` (add `governedPredicate`, `foreignNote`) +- Modify: `server/typescript/packages/cli/src/commands/migrate.ts` (the three `scopeExpectedSchema(built, collection.inMigrateScope)` / `offlineScope` sites) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (`const schemaScope = collection.inMigrateScope;`) +- Modify: `server/typescript/packages/migrate-ts/src/scope.ts` (header comment: one sentence — the predicate may also exclude dependency-owned objects) +- Test: `cli/test/migrate-foreign-nodes.test.ts` (new) + +**Read first:** DESIGN §2.7 (migrate row), §4.5. `cli/test/migrate-scope.test.ts` — copy its scaffolding (`runBaseline`, `runOfflineGenerate`, the sqlite dialect). `migrate-ts/src/scope.ts` — `scopeExpectedSchema(built, predicate)` returns `{ snapshot, outOfScope, declaredSchemas }`; nothing in migrate-ts changes. + +**Interfaces:** +- Produces: `governedPredicate(collection: Collection): { predicate: ((fqn: string) => boolean) | undefined; foreignNames: string[] }` — `predicate` is `undefined` when the collection has no reference-mode dependencies AND no `inMigrateScope` (byte-identical behavior for every existing project), else `(fqn) => collection.governs(fqn) && (collection.inMigrateScope?.(fqn) ?? true)`; `foreignNote(command: string, names: readonly string[]): string` = `` `meta ${command} — ${names.length} object(s) declared by dependencies (reference mode, governed by their publisher): ${names.join(", ")}` ``. + +- [ ] **Step 1: Failing test** `migrate-foreign-nodes.test.ts`: scaffold a consumer whose artifact is `acme-common-v1.json` (Customer has `@table: customers`) and whose local model is `APP` (`orders`), `CONFIG_REF` + `LOCK_V1`, `migrate.dialect: "sqlite"`. (a) `runOfflineGenerate` proposes `CREATE TABLE orders` and NOT `customers`, and the captured stderr contains `1 object(s) declared by dependencies (reference mode, governed by their publisher): acme::common::Customer`. (b) Adopt a baseline snapshot that contains a `customers` table (write the sqlite DB with both tables through the baseline path used in `migrate-scope.test.ts`), re-run: no `DROP TABLE customers` is proposed. (c) `mode: "own"` in config + lock: `CREATE TABLE customers` IS proposed. Run `cd server/typescript/packages/cli && bun test test/migrate-foreign-nodes.test.ts` → FAIL. + +- [ ] **Step 2: Implement** `governedPredicate` + `foreignNote`; replace the three migrate sites with `const { predicate, foreignNames } = governedPredicate(collection); const scoped = scopeExpectedSchema(built, predicate); if (foreignNames.length) log.warn(foreignNote("migrate", foreignNames));` (for the offline pipeline, `offlineScope = predicate`); in `verify.ts` set `schemaScope = governedPredicate(collection).predicate` and print `foreignNote("verify --db", …)`. `migrateScopeMismatch` keeps reading `collection.inMigrateScope` alone. + +- [ ] **Step 3: Run** +```bash +cd server/typescript/packages/cli && bun test test/migrate-foreign-nodes.test.ts test/migrate-scope.test.ts test/migrate-offline-gen.test.ts test/migrate-baseline.test.ts test/verify-replay.test.ts +bun run --filter '@metaobjectsdev/cli' typecheck +``` +Expected: PASS. + +- [ ] **Step 4: Commit** +``` +feat(cli): migrate and verify --db leave reference-mode dependency tables to their publisher (FR-023) +``` + +--- + +### Task 11: The load-time failure corpus cases (no new machinery) + +**Files:** +- Modify: `fixtures/dependency-conformance/cases.json` (append cases 15–19 and 28) +- Modify: `fixtures/dependency-conformance/README.md` (a table: upstream change → consumer construct → existing error code) +- Test: `sdk/test/dependency-conformance.test.ts` + +**Read first:** DESIGN §2.5 (the table). These cases pin that after a snapshot is replaced by a newer upstream, the LOADER's existing errors fail the consumer. They use `acme-common-v2-email-removed.json` as the snapshot (hash `sha256-c4ba364bff0071b164b127326c095d6d32915b57781271e0be64a7003c27ce7a` in the lock) and a local model written against v1. + +- [ ] **Step 1: Append** (each: `CONFIG_REF`; `LOCK_V2` = `LOCK_V1` with the v2 integrity; `treeFiles` → `artifacts/acme-common-v2-email-removed.json`; `expectFiles` = artifact + app file; plus `expectLoadError`): + - `an-overlay-whose-target-was-removed-fails`: lock/snapshot as `LOCK_V1`/v1 but the local file overlays a node the artifact never had: `{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Invoice","overlay":true,"children":[{"validator.length":{"@max":5}}]}}]}}` → `expectLoadError: "ERR_OVERLAY_NO_TARGET"`. + - `an-extends-whose-target-was-removed-fails`: local `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"T","extends":"acme::common::Gone","children":[{"source.rdb":{"@table":"t"}},{"field.long":{"name":"id"}},{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}` → `ERR_UNRESOLVED_SUPER`. + - `a-reference-whose-target-was-removed-fails`: local with `"@objectRef":"acme::common::Gone"` on a `field.object` → `ERR_UNRESOLVED_OBJECT_REF`. + - `a-dotted-extends-whose-member-changed-subtype-fails`: snapshot = a case-local artifact (in `tree`) = v1 with `Customer.email` retyped to `field.long`; local `{"metadata.root":{"package":"app","children":[{"field.string":{"name":"contact","extends":"acme::common::Customer.email"}}]}}` → `ERR_EXTENDS_TARGET_MISMATCH`. Pin this artifact's hash in the case's lock (README one-liner). + - `an-overlay-attr-the-base-now-sets-differently-conflicts`: snapshot v1 (`@maxLength: 120`); local `{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"email","@maxLength":80}}]}}]}}` → `ERR_MERGE_CONFLICT`. + - `a-foreign-file-error-names-the-dependency`: snapshot = a case-local artifact in `tree` = v1 with `Address.city` retyped to `{"field.bogus":{"name":"city"}}` (pin its hash in the case's lock via the README one-liner); local `APP` → `expectLoadError: "ERR_UNKNOWN_SUBTYPE"`, `expectErrorFiles: ["dep:acme-common/acme-common.metaobjects.json"]` — the one case that pins the `dep:` source-id shape. + Note case 19 is NOT `ERR_DEPENDENCY_SCHEMA_NOT_OWNED` — the merge conflict fires in the parser before the boundary validator runs; the runner asserts the FIRST error's code. + +- [ ] **Step 2: Run** `cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts` → all six PASS with no source change. If one does not, the design table is wrong for that construct: STOP and report the actual code — do not adjust the case. + +- [ ] **Step 3: README table** — add the five rows (upstream change / consumer construct / code) under a heading "What fails at load, with no new machinery". + +- [ ] **Step 4: Commit** +``` +test(conformance): FR-023 load-time failures after an upstream change are the loader's own errors +``` + +--- + +### Task 12: `sharedModelFile()` — the publisher generator (TS) + +**Files:** +- Create: `server/typescript/packages/codegen-ts/src/generators/shared-model-file.ts` +- Modify: `server/typescript/packages/codegen-ts/src/generator.ts` (`GenContext.registry?: TypeRegistry`, `GenContext.sourceFiles?: readonly string[]`) +- Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (fill both fields; `RunGenOptions.sourceFiles?`) +- Modify: `server/typescript/packages/codegen-ts/src/generator-registry.ts` (entry `shared-model`) +- Modify: `server/typescript/packages/codegen-ts/src/generators/index.ts` (export) +- Modify: `fixtures/generator-registry-conformance/registry.json` (entry `shared-model`, `"ports": ["typescript"]` — Task 19 adds `python`) +- Modify: `server/typescript/packages/cli/src/commands/gen.ts` (pass `sourceFiles: genCollection.ownFiles`) +- Modify: the ejectable list the `meta eject --list` command reads (grep `ejectable` under `cli/src`; add `shared-model` with the same shape as its neighbours) +- Test: `codegen-ts/test/shared-model-file.test.ts`, `codegen-ts/test/generator-registry.test.ts` (existing — must stay green), `cli/test/eject.test.ts` (existing) + +**Read first:** DESIGN §2.2, §4.3, §3.2, §3.5. `oncePerRun` in `generator.ts`; `MetaDataLoader` + `FileSource` (`@metaobjectsdev/metadata/core`); `serializeSharedDocument` (Task 2); `compileScope`/`matchesScope` from `@metaobjectsdev/metadata` (Task 4); `sha256Integrity` lives in sdk and codegen-ts must NOT depend on sdk, so this task MOVES `INTEGRITY_PREFIX` to `@metaobjectsdev/metadata/constants` (a new `dependency-constants.ts` module barreled there; sdk's `dependencies.ts` re-exports it from metadata instead of defining it) and adds a three-line `integrityOf(bytes: Uint8Array): string` (`createHash("sha256")`) in `codegen-ts/src/generators/shared-model-file.ts`. + +**Interfaces:** +- Produces: `INTEGRITY_PREFIX` now exported from `@metaobjectsdev/metadata/constants` (sdk re-exports it; Task 6's tests stay green). `sharedModelFile(opts: { name: string; version?: string; files?: readonly string[]; include: readonly string[]; exclude?: readonly string[]; target?: string }): Generator` (name `shared-model`). Emits two files into its target: `.metaobjects.json` and `metaobjects.pkg.json`. `version` defaults to the nearest `package.json`'s `version` walking up from `ctx.projectRoot` (error if none). `files` defaults to `ctx.sourceFiles`. Excludes `requirement.*` and `template.*` nodes always. + +**Algorithm (DESIGN §4.3):** (1) standalone load of `files` with `ctx.registry` (strict) → errors are thrown as `Error("shared-model: ")`; (2) select top-level objects/fields (any top-level node whose type is not `requirement`/`template`) by `matchesScope(resolutionKey, compileScope({ include, exclude }))`; empty → throw `shared-model: include/exclude selected no nodes`; (3) closure: for every selected node and each descendant, every `extends` target (object part of a dotted ref) and every attr named in `REF_BEARING_ATTR_NAMES` (object part) must be a selected resolution key, else throw `shared-model: references , which is not selected — include it or exclude the referrer` listing every pair; (4) `artifact = serializeSharedDocument(selected)`; (5) re-load `artifact` with `composeRegistry(coreProviders)` strict → errors throw `shared-model: the export needs vocabulary the core registry does not register: `; (6) manifest per §3.2 with `nodes` = sorted resolution keys, `packages` = sorted distinct `packageOfResolutionKey`s, `metamodelVersion = METAMODEL_VERSION`, `integrity = integrityOf(artifact)`; serialized `JSON.stringify(manifest, null, 2) + "\n"`. + +- [ ] **Step 1: Failing test** `shared-model-file.test.ts` — scaffold a temp publisher with `metaobjects/meta.lib.json` (the `LIB` document from Task 2, root package `acme::common`) plus `metaobjects/admin-ui/meta.lib.ui.json` (`{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"email","children":[{"validator.length":{"@max":100}}]}}]}}]}}`) and a `package.json` with `"version": "1.0.0"`. Run `runGen` programmatically (mirror `codegen-ts/test/dry-run-writes-nothing.test.ts` for the harness) with `generators: [sharedModelFile({ name: "acme-common", files: ["metaobjects/meta.lib.json"], include: ["acme::common::**"] })]`. Assert: `acme-common.metaobjects.json` is byte-identical to `fixtures/dependency-conformance/artifacts/acme-common-v1.json` (the overlay file was not listed, so the validator child is absent); `metaobjects.pkg.json` parses to the manifest in Task 6's test (`integrity` = the pinned v1 hash). Second test: `include: ["acme::common::Customer"]` alone succeeds (Customer references nothing outside itself); `include: ["acme::common::Address"]` with a local file where `Address` `extends: acme::common::Audited` throws `/references acme::common::Audited, which is not selected/`. Third: a `files` list including the overlay file yields an artifact containing `validator.length`. Fourth: two runs produce identical bytes. Fifth: a document whose selected node carries an attr registered only by a consumer provider (`ctx.registry` composed with a test provider registering `@acmeLocal` on `object.entity`) throws `/core registry does not register/`. Run `cd server/typescript/packages/codegen-ts && bun test test/shared-model-file.test.ts` → FAIL. + +- [ ] **Step 2: Implement** the generator, the `GenContext` fields, the runner plumbing, the registry entry (`description: "One flattened canonical-JSON shared-model artifact + manifest for consumers (FR-023)."`, `tier: "native"`, `options: "name, include, exclude?, files?, version?, target?"`), the `registry.json` entry (`"concept": "The publisher's shared-model artifact (a flattened canonical document + manifest) consumed through `dependencies`.", "tier": "native", "ports": ["typescript"]`), the eject listing, and `gen.ts` passing `sourceFiles: genCollection.ownFiles`. + +- [ ] **Step 3: Run** +```bash +cd server/typescript/packages/codegen-ts && bun test test/shared-model-file.test.ts test/generator-registry.test.ts +cd server/typescript/packages/cli && bun test test/eject.test.ts test/gen-list.test.ts +cd server/python && uv run --extra integration pytest tests/codegen/test_cli_registry.py -q +bun run --filter '@metaobjectsdev/codegen-ts' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck && bun run --filter '@metaobjectsdev/metadata' typecheck && bun run --filter '@metaobjectsdev/sdk' typecheck +``` +Expected: PASS (the Python registry test stays green because `ports` lists only `typescript`). + +- [ ] **Step 4: Commit** +``` +feat(codegen-ts): sharedModelFile() emits the flattened shared-model artifact and manifest (FR-023) +``` + +--- + +### Task 13: `meta deps sync` with the `path` transport, and `meta deps list` + +**Files:** +- Create: `server/typescript/packages/cli/src/commands/deps.ts` +- Create: `server/typescript/packages/cli/src/lib/dependency-transports.ts` (`resolveTransport` — `path` only here; Task 16 adds `npm`/`python`) +- Create: `server/typescript/packages/cli/src/lib/dependency-sync.ts` +- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`DEPS_OPTIONS`, `parseDepsArgs`) +- Modify: `server/typescript/packages/cli/src/index.ts` (register `deps`; help text; `FORMAT_AWARE_COMMANDS` gains `"deps"`) +- Test: `cli/test/deps-sync.test.ts`, `cli/test/unit/args-deps.test.ts`, `cli/test/help-lists-every-flag.test.ts` (existing — must stay green) + +**Read first:** DESIGN §4.1 (steps 1–5, 7–8; classification is Task 15), §2.1 (transports), §3.2–3.4. `cli/src/index.ts` dispatch (`case "migrate"` shape); `args.ts` `parseArgs` style with an exported options table; `sdk` exports from Tasks 5–7. + +**Interfaces:** +- Produces: `depsCommand(args: string[], cwd: string, fmt: OutputFormat): Promise` with verbs `sync […] [--dry-run]`, `list`; `resolveTransport(spec: DependencySpec, configDir: string): Promise` (a directory); `syncOne(configDir, spec, dir, opts): Promise<{ entry: LockEntry; artifactBytes: Uint8Array; changed: boolean }>`; `validateResolvedPackage(dir, spec): Promise<{ manifest: Manifest; artifactBytes: Uint8Array }>` performing DESIGN §4.1 steps 2–4 with the codes `ERR_DEPENDENCY_MANIFEST_INVALID` / `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`; `checkCollisions(entries): void` → `ERR_DEPENDENCY_NODE_COLLISION`. + +**Behavior:** `sync` resolves each declared dependency (all, or the named subset — an unknown name exits 2), validates, then (unless `--dry-run`) empties `.metaobjects/deps//`, writes the artifact, updates the lock entry (`resolvedFrom` = the spec's transport keys verbatim, `mode` = the spec's mode), prunes lock entries and snapshot dirs for names no longer declared, writes the lock with sorted keys; then loads the whole collection through `loadCollection` (Task 8) so a sync producing an unloadable model exits 1 in the same command. Report lines (stderr): `synced ( node(s), package(s)) from path ` / `unchanged ` / `would sync …` under `--dry-run`. `list` prints one line per lock entry: `\t\t\t\t`; `--format json` emits the lock entries. `sync` also ensures `.metaobjects/.gitignore` contains a `deps.local.json` line (append if the file exists and lacks it; create with that one line if absent) — the D10 hygiene half that belongs to sync. + +- [ ] **Step 1: Failing args test** `cli/test/unit/args-deps.test.ts`: `parseDepsArgs(["sync", "acme-common", "--dry-run"])` → `{ verb: "sync", names: ["acme-common"], dryRun: true }`; `["list"]` → `{ verb: "list", … }`; `["frobnicate"]` throws; `["sync", "--bogus"]` throws. Run → FAIL. Implement `DEPS_OPTIONS = { "dry-run": { type: "boolean" } } as const` and `parseDepsArgs(argv): { verb: "sync" | "list"; names: string[]; dryRun: boolean }` — `accept-breaking` is NOT parsed here (a flag that does nothing must not ship; Task 15 adds it with its behavior). Run → PASS. + +- [ ] **Step 2: Failing command test** `deps-sync.test.ts`. Scaffold `/lib/metaobjects/` holding `metaobjects.pkg.json` + `acme-common.metaobjects.json` (copy the pinned v1 artifact and the manifest from Task 6's test data) and `/app/` with `.metaobjects/config.json` = `{ "schema_version": 1, "sources": [], "dependencies": [{ "name": "acme-common", "path": "../lib/metaobjects" }] }` and `metaobjects/meta.app.json` = `APP`. Assert, in order: (a) `depsCommand(["sync"], app)` exits 0; `.metaobjects/deps/acme-common/acme-common.metaobjects.json` equals the pinned bytes; `.metaobjects/deps.lock.json` equals exactly the `LOCK_V1` text from Task 7 with `"resolvedFrom": { "path": "../lib/metaobjects" }` (2-space, trailing newline, sorted keys); `.metaobjects/.gitignore` contains `deps.local.json`. (b) a second `sync` exits 0, reports `unchanged`, and leaves the lock byte-identical. (c) `--dry-run` after editing the upstream artifact to the widened version writes nothing and prints `would sync`. (d) removing the dependency from config and syncing prunes the lock entry and deletes `.metaobjects/deps/acme-common/`. (e) `sync` against a lib dir whose manifest `name` is `other` exits 1 with `ERR_DEPENDENCY_MANIFEST_INVALID` in stderr; a lib dir with no manifest → `ERR_DEPENDENCY_UNRESOLVED`; a manifest whose `integrity` does not match its artifact → `ERR_DEPENDENCY_MANIFEST_INVALID`; a manifest with `metamodelVersion: "2.0"` → `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`; an artifact whose top-level nodes differ from `manifest.nodes` → `ERR_DEPENDENCY_MANIFEST_INVALID`. (f) `depsCommand(["list"], app)` prints a line starting with `acme-common\t1.0.0\treference\tsha256-10fbf886`. Run `cd server/typescript/packages/cli && bun test test/deps-sync.test.ts` → FAIL. + +- [ ] **Step 3: Implement** `dependency-transports.ts` (`path`: `isAbsolute ? spec.path : resolve(configDir, spec.path)`; must be a directory holding `MANIFEST_FILE`, else `ERR_DEPENDENCY_UNRESOLVED`), `dependency-sync.ts` (validation per §4.1: parse manifest with `ManifestSchema` → `ERR_DEPENDENCY_MANIFEST_INVALID`; `manifest.name !== spec.name` → same; read artifact, hash, compare; `metamodelVersion` major vs `METAMODEL_VERSION`; standalone load of the artifact with `composeRegistry(coreProviders)` strict via `MetaDataLoader.fromString` → any error → `ERR_DEPENDENCY_MANIFEST_INVALID`; top-level resolution keys (sorted) must equal `manifest.nodes`, packages equal `manifest.packages`), `deps.ts`, the dispatch, and the help text (`meta deps sync|list`). Errors are `ParseError`s with the codes; the command prints `meta deps: ` and exits 1 (2 for a usage error). + +- [ ] **Step 4: Run** +```bash +cd server/typescript/packages/cli && bun test test/deps-sync.test.ts test/unit/args-deps.test.ts test/help-lists-every-flag.test.ts test/help-and-exit.test.ts test/cli.test.ts +bun run --filter '@metaobjectsdev/cli' typecheck +``` +Expected: PASS. + +- [ ] **Step 5: Commit** +``` +feat(cli): meta deps sync (path transport) and meta deps list (FR-023) +``` + +--- + +### Task 14: The usage-aware classifier and the footprint walker + +**Files:** +- Create: `server/typescript/packages/metadata/src/dependency-diff.ts` +- Modify: `server/typescript/packages/metadata/src/index.ts` (export) +- Modify: `server/typescript/packages/sdk/src/dependencies.ts` (`dependencyFootprint`) +- Modify: `fixtures/dependency-conformance/cases.json` (append classify cases 29–46) +- Modify: `sdk/test/dependency-conformance.test.ts` (implement the `classify` arm — remove the `test.skip`) +- Test: `metadata/test/dependency-diff.test.ts`, `sdk/test/dependencies.test.ts` (footprint) + +**Read first:** DESIGN §2.6 (footprint scopes, the class table, the widening and additive tables — copy them into named constants), §4.6. `REF_BEARING_ATTR_NAMES` (`naming-refs.ts`) is the reference-attr set; `RELATIONSHIP_ATTR_OBJECT_REF`/`RELATIONSHIP_ATTR_THROUGH` are in it; `IDENTITY_REFERENCE_ATTR_REFERENCES` too. `@implementedBy` is `REQUIREMENT_ATTR_IMPLEMENTED_BY` (check `requirement` constants). Documentation attrs are the common attrs `description`, `title`, `notes`, `seeAlso`, `aliases`, `replacedBy`, `deprecated` — import their constants from the documentation provider's constants module, never inline. + +**Interfaces:** +- Produces (metadata): `type FootprintScope = "whole" | "key" | "existence"`; `type Change = { fqn: string; path: string; kind: "breaking" | "compatible" | "info"; before?: unknown; after?: unknown; rule: string }`; `classifyDependencyChanges(oldDoc: string, newDoc: string, footprint: ReadonlyMap): Change[]` — both docs are artifact strings (canonical JSON); `WIDENING_ATTRS`, `ADDITIVE_ATTRS`, `DOC_ATTRS` exported constants. +- Produces (sdk): `dependencyFootprint(root: MetaRoot, collection: Collection): Map`. + +**Diff algorithm:** parse both documents; index top-level nodes by `::`; for each footprint fqn: absent in `new` → one `breaking` change at path `""` with rule `node-removed`; else restrict both bodies by scope (`whole`: everything; `key`: children whose fused key starts with `identity.` or `source.`, plus `extends`/`abstract`; `existence`: the fused key only) and walk: children matched by `(fused key, name)` → removed child → `breaking` (`child-removed`), added child → `compatible` (`child-added`); a body key `extends`/`abstract`/`isArray` differing → `breaking`; `@`-attr removed → `breaking` (`attr-removed`) unless in `DOC_ATTRS` (ignored); `@`-attr added → `compatible` if in `ADDITIVE_ATTRS` or `DOC_ATTRS` (`deprecated` → `info`), else `breaking` (`attr-added`); `@`-attr changed → ignored if `DOC_ATTRS`, `compatible` if the `WIDENING_ATTRS` rule for that attr holds in the stated direction (`maxLength`/`precision`/`max` numeric up; `minLength`/`min` numeric down; `required` `true→false`; `values` old ⊂ new; `intValueMap` every old entry present and equal), else `breaking` (`attr-changed`). `path` is the JSON-pointer-like `children[]..` chain by names, e.g. `field.string:email/@maxLength`. + +- [ ] **Step 1: Append classify cases** — each `{ "name", "classify": { "old": , "new": , "footprint": {…}, "expectChanges": [...] } }` where `old` is the v1 artifact document (as a JSON object, not a string) unless stated, footprint `{ "acme::common::Customer": "whole" }` unless stated: + 29 `removed-node-is-breaking`: `new` = v1 without `Customer` → `[{ "fqn": "acme::common::Customer", "path": "", "kind": "breaking" }]`. + 30 `removed-member-is-breaking`: `new` = v2 (email removed) → `[{ "fqn": "acme::common::Customer", "path": "field.string:email", "kind": "breaking" }]`. + 31 `subtype-change-is-breaking`: `new` = v1 with `email` retyped `{"field.long":{"name":"email"}}` (attrs dropped) → exactly ONE change: `[{ "fqn": "acme::common::Customer", "path": "email", "kind": "breaking" }]` with rule `subtype-changed`. Children are matched by NAME first and fused key second, so a retyped member is one breaking change, never a removal plus an addition. + 32 `isarray-change-is-breaking`: email gains `"isArray": true` → breaking at `field.string:email/isArray`. + 33 `required-true-to-false-is-compatible`: old email has `"@required": true`, new `false` → compatible. + 34 `required-false-to-true-is-breaking`: reverse → breaking. + 35 `maxlength-widened-is-compatible`: `new` = v1-widened → compatible at `field.string:email/@maxLength`. + 36 `maxlength-narrowed-is-breaking`: old widened (200) → new v1 (120) → breaking. + 37 `enum-member-added-is-compatible`: both docs carry a `field.enum` `kind` on Customer with `@values ["a","b"]` → `["a","b","c"]` → compatible. + 38 `enum-member-removed-is-breaking`: reverse → breaking. + 39 `added-child-is-compatible`: new Customer gains `{"field.string":{"name":"phone"}}` → compatible at `field.string:phone`. + 40 `description-change-is-ignored`: Customer `"description"` differs → `expectChanges: []`. + 41 `unknown-attr-change-is-breaking`: email `"@column": "email_addr"` → `"@column": "mail"` → breaking. + 42 `deprecated-added-is-info`: Customer gains `"deprecated": true` → info. + 43 `a-change-outside-the-footprint-is-not-reported`: new = v2 (email removed) but footprint `{ "acme::common::Address": "whole" }` → `[]`. + 44 `a-non-key-member-change-on-an-fk-target-is-not-reported`: footprint `{ "acme::common::Customer": "key" }`, new = v1-widened → `[]`. + 45 `a-key-field-change-on-an-fk-target-is-breaking`: footprint `key`, new = v1 with `identity.primary` `@fields` = `["email"]` → breaking at `identity.primary:pk/@fields`. + 46 `an-inherited-member-change-on-an-extends-target-is-breaking`: footprint `{ "acme::common::Audited": "whole" }`, new = v1 with `Audited.createdAt` retyped `field.string` → breaking at `createdAt`. + (Documentation attrs: check the exact key spelling the documentation provider registers — `description` is a bare common attr in canonical JSON? It is an `@`-prefixed attr in canonical form: `"@description"`. Use the canonical spelling in the cases; the runner passes documents through unchanged.) + +- [ ] **Step 2: Implement the runner's classify arm** (parse docs to strings via `JSON.stringify(old, null, 2) + "\n"`, call `classifyDependencyChanges`, compare `{fqn, path, kind}` triples as unordered sets). Run → the 18 cases FAIL (module missing). + +- [ ] **Step 3: Failing unit tests** `metadata/test/dependency-diff.test.ts` — the widening table row by row (one `test` per entry of `WIDENING_ATTRS`, both directions) and `DOC_ATTRS` ignored; `sdk/test/dependencies.test.ts` — `dependencyFootprint` over a loaded consumer (artifact v1 + a local model that `extends` `Audited`, `@objectRef`s `Address`, `@references` `Customer` via `identity.reference`, and overlays `Customer` with `overlay: true`) yields exactly `{ "acme::common::Audited": "whole", "acme::common::Address": "whole", "acme::common::Customer": "key" }` — `Customer` is both overlaid (`existence`) and FK-referenced (`key`); the widest scope wins. Run → FAIL. + +- [ ] **Step 4: Implement** `dependency-diff.ts` and `dependencyFootprint` (walk every LOCAL top-level node — `collection.foreignOwner(fqn) === undefined` — and its descendants with resolving accessors; collect `extends` targets (object part of a dotted ref), `REF_BEARING_ATTR_NAMES` values split by attribute into `whole` vs `key` per the Global Constraints, `@implementedBy` → existence; a local contribution to a foreign node → existence; close `whole` entries over the target's `extends` chain). + +- [ ] **Step 5: Run** +```bash +cd server/typescript/packages/metadata && bun test test/dependency-diff.test.ts +cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts test/dependencies.test.ts +bun run --filter '@metaobjectsdev/metadata' typecheck && bun run --filter '@metaobjectsdev/sdk' typecheck +``` +Expected: PASS (cases 29–46 green). + +- [ ] **Step 6: Commit** +``` +feat(metadata,sdk): usage-aware dependency change classifier and footprint (FR-023) +``` + +--- + +### Task 15: `sync` refuses BREAKING upstream changes; `meta deps check` and `meta verify --deps` + +**Files:** +- Modify: `server/typescript/packages/cli/src/lib/dependency-sync.ts` (`classifyAgainstSnapshot`, `checkDependencies`) +- Modify: `server/typescript/packages/cli/src/commands/deps.ts` (`--accept-breaking []`, `check`) +- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`DEPS_OPTIONS["accept-breaking"]`, `VERIFY_OPTIONS.deps`) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (`--deps` subverb; its own line in the report; exit aggregation) +- Modify: `server/typescript/packages/cli/src/index.ts` (help text) +- Test: `cli/test/deps-check.test.ts`, `cli/test/unit/args-deps.test.ts`, `cli/test/unit/args-verify.test.ts`, `cli/test/help-lists-every-flag.test.ts` + +**Read first:** DESIGN §4.1 step 6, §4.2 (`meta deps check`), §2.6 "Where it fires". `verify.ts` subverb handling (`templates || codegen || docs || db …` decides "any explicit"); the design says `--deps` is one more subverb and is NOT part of the bare-`verify` default. + +**Interfaces:** +- Produces: `classifyAgainstSnapshot(configDir, collection, name, newArtifact): Change[]` (loads the consumer model via `loadCollection`, computes the footprint for `name`, reads the current snapshot artifact, calls `classifyDependencyChanges`); `checkDependencies(cwd): Promise<{ exitCode: number; report: string[] }>`; `meta deps sync` exits 1 with `ERR_DEPENDENCY_BREAKING_CHANGE` and the per-change report (` : ()`) when any `breaking` change exists and the dependency is not named by `--accept-breaking` (bare `--accept-breaking` = all); `meta deps check` = resolve + validate + hash every dependency, compare each `integrity` to the lock, classify differences, exit 1 with `ERR_DEPENDENCY_UPSTREAM_DRIFT` naming the dependency, locked vs upstream version, and each change line; unreachable upstream → exit 1 `ERR_DEPENDENCY_UNRESOLVED`; also runs the snapshot integrity check. `meta verify --deps` runs `checkDependencies` and prints `meta verify — deps: N dependency(ies) in sync` or the drift report. + +- [ ] **Step 1: Failing tests.** Extend `args-deps.test.ts`: `["sync", "--accept-breaking"]` → `acceptBreaking: []` (all); `["sync", "--accept-breaking", "acme-common"]` → `["acme-common"]`; `["check"]` → verb check. Extend `args-verify.test.ts`: `--deps` sets `deps: true` and counts as an explicit subverb. `deps-check.test.ts` (scaffold as Task 13 (a) — lib dir with the v1 artifact + manifest, app synced once — then, as separate tests): + (a) `depsCommand(["check"], app)` exits 0 and stderr contains `in sync`. + (b) Replace the lib's artifact with `acme-common-v1-widened.json` and rewrite its manifest with the widened integrity (`sha256-fa00b9f3c54a2c302cf269af051589afc0a3217999ac82c54c77e33494638c36`). With the local model `APP` (which references nothing foreign): `check` exits 1, stderr contains `ERR_DEPENDENCY_UPSTREAM_DRIFT`, `acme-common 1.0.0 -> 1.0.0`, and `0 change(s) in your footprint`. + (c) Change the local model so `Order` carries `{"identity.reference":{"name":"fkCustomer","@fields":["customerId"],"@references":"acme::common::Customer"}}` plus `{"field.long":{"name":"customerId"}}`: `check` still reports drift with `0 change(s) in your footprint` — `key` scope excludes `@maxLength`. + (d) Replace the lib's artifact with `acme-common-v2-email-removed.json` (+ manifest with `sha256-c4ba364bff0071b164b127326c095d6d32915b57781271e0be64a7003c27ce7a`) and change the local model to `{"metadata.root":{"package":"app","children":[{"field.string":{"name":"contact","extends":"acme::common::Customer.email"}}]}}`: `check` exits 1 and prints `acme::common::Customer field.string:email: breaking (child-removed)`; `depsCommand(["sync"], app)` exits 1 with `ERR_DEPENDENCY_BREAKING_CHANGE` and the snapshot still holds the v1 bytes; `depsCommand(["sync", "--accept-breaking", "acme-common"], app)` writes the v2 bytes and the v2 hash into the lock, then exits 1 because the post-sync load fails with `ERR_UNRESOLVED_SUPER` (assert both the files and the exit code). + (e) `verifyCommand(["--deps"], app)` returns the same exit code as `check` in states (a) and (d) and prints a line starting `meta verify — deps:`. +Run `cd server/typescript/packages/cli && bun test test/deps-check.test.ts test/unit/args-deps.test.ts test/unit/args-verify.test.ts` → FAIL. + +- [ ] **Step 2: Implement.** In `sync`: before writing, if a snapshot exists for the name, `classifyAgainstSnapshot`; print every change; refuse on breaking unless accepted. Write, then run the post-sync load; a load failure exits 1 but the write stands (the report says `synced … ; the model no longer loads: — fix your metadata`). `check`: never writes. `verify --deps`: calls `checkDependencies(cwd)`. + +- [ ] **Step 3: Run** +```bash +cd server/typescript/packages/cli && bun test test/deps-check.test.ts test/deps-sync.test.ts test/unit/args-deps.test.ts test/unit/args-verify.test.ts test/help-lists-every-flag.test.ts test/verify-replay.test.ts +bun run --filter '@metaobjectsdev/cli' typecheck +``` +Expected: PASS. + +- [ ] **Step 4: Commit** +``` +feat(cli): meta deps check, verify --deps, and sync's breaking-change refusal (FR-023) +``` + +--- + +### Task 16: The `npm` and `python` transports + +**Files:** +- Modify: `server/typescript/packages/cli/src/lib/dependency-transports.ts` +- Test: `cli/test/dependency-transports.test.ts` + +**Read first:** DESIGN §2.1 transports table (the `npm` and `python` rows, the interpreter ladder), §9 (the `find_spec` risk). + +**Interfaces:** +- Produces: `resolveTransport(spec, configDir)` handles `npm` (`createRequire(join(configDir, "package.json")).resolve("/package.json")` → its `dirname`, then `spec.dir` if set) and `python` (spawn the interpreter chosen by the ladder `process.env.METAOBJECTS_PYTHON` → `${process.env.VIRTUAL_ENV}/bin/python` → `/.venv/bin/python` → `python3`, with `-c "import importlib.util,sys; s=importlib.util.find_spec(sys.argv[1]); print(list(s.submodule_search_locations)[0])" `; the printed directory, then `spec.dir`); both must contain `MANIFEST_FILE` else `ERR_DEPENDENCY_UNRESOLVED` whose message names what was tried and, for `python`, which interpreter answered. `resolveTransport` returns `{ dir, detail }` where `detail` is the report suffix (`from npm @acme/model` / `from python acme_model via `). + +- [ ] **Step 1: Failing test** `dependency-transports.test.ts`: (a) `npm`: a temp app with `package.json` and `node_modules/@acme/model/package.json` (`{"name":"@acme/model","version":"1.0.0"}`) + `node_modules/@acme/model/metaobjects/metaobjects.pkg.json` → `resolveTransport({ name: "m", npm: "@acme/model", dir: "metaobjects", mode: "reference" }, app)` returns that directory; without `dir` and with the manifest at the package root, the root. A missing package → `ERR_DEPENDENCY_UNRESOLVED`. (b) `python`: write an executable stub script `/fake-python` (`#!/bin/sh`, prints `/site/acme_model` and exits 0), set `process.env.METAOBJECTS_PYTHON` to it for the test, place the manifest at `/site/acme_model/metaobjects/metaobjects.pkg.json` → resolves with `dir: "metaobjects"`; a stub that exits 1 → `ERR_DEPENDENCY_UNRESOLVED` with the interpreter path in the message. (c) opt-in real-interpreter test guarded by `Bun.which("python3") !== null`: `python: "json"` (stdlib package) resolves to a directory (no manifest → `ERR_DEPENDENCY_UNRESOLVED`, which proves the spawn path). Run → FAIL. Implement. Run → PASS; typecheck → PASS. + +- [ ] **Step 2: Commit** +``` +feat(cli): npm and python dependency transports (FR-023) +``` + +--- + +### Task 17: The local co-development override (D10) — TS + +**Files:** +- Modify: `server/typescript/packages/sdk/src/dependencies.ts` (`verifySnapshot` honours overrides) +- Modify: `server/typescript/packages/sdk/src/collection.ts` (`overrides`) +- Modify: `server/typescript/packages/cli/src/lib/collection-load-options.ts` (`warnLocalOverrides(collection)` — one `log.warn` per overridden dependency, called from `loadCollection`) +- Modify: `server/typescript/packages/cli/src/commands/deps.ts` (`sync` copies from the override; `check` and `verify --deps` refuse) +- Modify: `server/typescript/packages/cli/src/commands/init.ts` (`METAOBJECTS_GITIGNORE_BODY` gains `deps.local.json` under a comment "the local co-development override — never committed") +- Modify: `fixtures/dependency-conformance/cases.json` (append cases 47–49) +- Test: `sdk/test/dependency-conformance.test.ts`, `cli/test/deps-override.test.ts`, `cli/test/init.test.ts` + +**Read first:** DESIGN §10 D10 (every bullet is a requirement). Task 7 left the branch `local overrides are not supported yet` in `verifySnapshot` — replace it. + +**Interfaces:** +- Produces: with `.metaobjects/deps.local.json` = `{ "": { "path": "" } }`: `verifySnapshot` reads `/metaobjects.pkg.json` (must exist and parse → else `ERR_DEPENDENCY_UNRESOLVED`; its `name` must equal the dependency's; an override for a name not in `dependencies` → `ERR_DEPENDENCY_UNRESOLVED`), sets `artifactPath = /`, `nodes`/`packages`/`version`/`metamodelVersion` from THAT manifest (not the lock), `override = `, and skips the lock entry/hash check for that name (a missing lock entry for an overridden name is fine); `Collection.overrides` lists the names; the warning text is exactly `loading from a local override (), not the committed snapshot`. `deps sync` with an override active resolves that dependency from the override path instead of its transport (report suffix `from local override `) and writes the snapshot + lock as usual. `deps check` / `verify --deps` exit 1 with `local override(s) active: — remove .metaobjects/deps.local.json to verify the committed snapshot` before checking anything. `meta init` writes the gitignore line. + +- [ ] **Step 1: Append corpus cases**: + - `an-active-override-replaces-the-snapshot-and-skips-its-hash-check`: tree `APP` + `treeFiles` snapshot → `artifacts/acme-common-v1.json` AND `lib/metaobjects/acme-common.metaobjects.json` → `artifacts/acme-common-v1-widened.json` + `lib/metaobjects/metaobjects.pkg.json` (manifest text with the widened hash); `CONFIG_REF`; `LOCK_V1` with a WRONG integrity (`sha256-0000…0000`, 64 zeros); `localOverrides: { "acme-common": { "path": "lib/metaobjects" } }` → `expectFiles: ["lib/metaobjects/acme-common.metaobjects.json", "metaobjects/meta.app.json"]`, `expectOverrides: ["acme-common"]`, `expectForeign` the three. + - `an-override-whose-path-lacks-a-manifest-is-unresolved`: same but no manifest file at the override path → `expectError: "ERR_DEPENDENCY_UNRESOLVED"`. + - `an-override-for-an-undeclared-dependency-is-unresolved`: `localOverrides: { "ghost": { "path": "lib" } }` with `CONFIG_REF` + `LOCK_V1` + `SNAP` → `expectError: "ERR_DEPENDENCY_UNRESOLVED"`. + Run → FAIL (the Task 7 stub branch throws for case 47). + +- [ ] **Step 2: Failing cli tests** `deps-override.test.ts`: scaffold as Task 13 with the snapshot synced; write `deps.local.json` pointing at `../lib/metaobjects` after replacing the lib artifact with the widened one (+ manifest); (a) `genCommand([], app)` stderr contains the exact warning line and the emitted model reflects the override (assert via `verify --codegen`? simpler: `exportCommand` output contains `"@maxLength": 200`); (b) `depsCommand(["check"], app)` exits 1 with `local override(s) active: acme-common`; `verifyCommand(["--deps"], app)` likewise; (c) `depsCommand(["sync"], app)` exits 0, report contains `from local override`, the snapshot now holds the widened bytes and the lock the widened hash; (d) delete `deps.local.json` → `check` exits 0. `init.test.ts`: the scaffolded `.metaobjects/.gitignore` contains a line `deps.local.json`. Run → FAIL. + +- [ ] **Step 3: Implement** all of the above. + +- [ ] **Step 4: Run** +```bash +cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts test/dependencies.test.ts +cd server/typescript/packages/cli && bun test test/deps-override.test.ts test/deps-sync.test.ts test/deps-check.test.ts test/init.test.ts test/ignored-scaffold-check.test.ts +bun run --filter '@metaobjectsdev/sdk' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck +``` +Expected: PASS (cases 47–49 green). + +- [ ] **Step 5: Commit** +``` +feat(sdk,cli): local co-development override via .metaobjects/deps.local.json (FR-023 D10) +``` + +--- + +### Task 18: Python — the collection resolver, CLI, runner, boundary validator, and D10 + +**Files:** +- Modify: `server/python/src/metaobjects/config/dependencies.py` (`verify_snapshot`, `foreign_owner_factory`, `validate_dependency_boundary`, `declared_overlay_keys`, `read_local_overrides` use) +- Modify: `server/python/src/metaobjects/config/source_resolver.py` (`resolve_collection` returns `Collection`; keep `resolve_collection_files(root) -> list[Path]` as a thin wrapper for existing callers) +- Modify: `server/python/src/metaobjects/cli.py` (`resolve_metadata_location` returns the `Collection` at every rung; `_load_root_from_paths` → `_load_root_from_collection` building `FileSource(path, id=…)` and calling `loader.load(sources)`; boundary check after every load; `run_gen(..., governs=…)`; `entities:` naming a reference-mode foreign node → exit 2; the override warning printed once per command) +- Modify: `server/python/src/metaobjects/codegen/runner.py` (`run_gen(..., governs: Callable[[str], bool] | None = None)`) +- Create: `server/python/tests/conformance/test_dependency_conformance.py` +- Test: `tests/config/test_dependencies.py` (extend), `tests/codegen/test_cli_dependencies.py` (new) + +**Read first:** DESIGN §4.2, §4.4, §4.5 (Python rows), §10 D10. The TS reference: `sdk/src/dependencies.ts` + `collection.ts` after Task 17 — port the behavior, including every error code and the table in Task 7. `cli.py` `resolve_metadata_location` (rungs 2–4) and `_load_root_from_paths`; `runner.py` `_objects`. + +**Interfaces:** +- Produces: `source_resolver.Collection` dataclass: `config_dir: Path`, `files: list[Path]`, `own_files: list[Path]`, `file_ids: dict[Path, str]`, `dependencies: list[ResolvedDependency]`, `overrides: list[str]`, `foreign_owner(fqn) -> ResolvedDependency | None`, `governs(fqn) -> bool`. `run_gen(..., governs=None)` filters `_objects` by `governs(o.resolution_key())`. `validate_dependency_boundary(root, collection, overlay_keys)` raises `ParseError` with the same codes and rules as Task 8. `declared_overlay_keys(path: Path) -> set[str]` parses the raw file (`json.loads` for `.json`; `yaml.safe_load` for `.yaml`/`.yml`, root key `metadata` or `metadata.root`) and returns resolution keys of top-level nodes with `overlay: True`. + +- [ ] **Step 1: Write the Python corpus runner** `tests/conformance/test_dependency_conformance.py` mirroring the TS runner (materialize `tree`, `treeFiles`, `config`, `lock`, `localOverrides`; for `expectFiles` compare the set of files relative to the temp root; `expectForeign`/`expectGoverned`/`expectOverrides`; `expectLoadError` loads via the same code path `cli._load_root_from_collection` and asserts the first error code; `expectError` asserts `ParseError.code`; the `classify` arm is `pytest.skip("Task 19")` for now). For `dependencies-are-read-under-a-native-source-surface` ALSO write `app/metaobjects.config.yaml` containing `metadata: metaobjects\ntargets:\n t:\n outDir: out\n generators: [names]\n` and resolve through `cli.resolve_metadata_location(load_project_config(...), root)` to prove rung 2 still adds the artifact. Run `cd server/python && uv run --extra integration pytest tests/conformance/test_dependency_conformance.py -q` → FAIL. + +- [ ] **Step 2: Failing CLI tests** `tests/codegen/test_cli_dependencies.py`: scaffold a consumer (pinned v1 artifact under `.metaobjects/deps/acme-common/`, `CONFIG_REF`, `LOCK_V1`, `APP`, `metaobjects.config.yaml` with a `names` target); (a) `metaobjects gen` (call `cli.main([...])` with cwd set) writes `order_names.py` and NOT `customer_names.py`; (b) `entities: [Customer]` in the target → exit 2, stderr contains `declared by dependency 'acme-common' (reference mode)`; (c) corrupt the lock hash → `gen` exits non-zero BEFORE writing anything, stderr contains `ERR_DEPENDENCY_SNAPSHOT_STALE`; (d) with `deps.local.json` pointing at a widened lib dir (+ manifest), `gen` succeeds and stderr contains the exact warning line from D10; (e) `verify --codegen` shares the selection (no drift after (a)). Run → FAIL. + +- [ ] **Step 3: Implement** everything listed under Files. Keep `resolve_collection_files` so `cli.py` callers outside the ladder are untouched. + +- [ ] **Step 4: Run** +```bash +cd server/python && uv run --extra integration pytest tests/conformance/test_dependency_conformance.py tests/config tests/codegen/test_cli_dependencies.py tests/codegen/test_cli_config_gen.py tests/codegen/test_cli_config_verify.py tests/conformance/test_source_resolution_conformance.py -q +``` +Expected: PASS (cases 1–28 and 47–49 green in Python; 29–46 skipped). + +- [ ] **Step 5: Commit** +``` +feat(python): dependency snapshot loading, foreign-node selection, boundary validator, and the local override (FR-023) +``` + +--- + +### Task 19: Python — the `shared-model` generator, `targets..options`, and the classifier twin + +**Files:** +- Create: `server/python/src/metaobjects/codegen/generators/shared_model.py` +- Modify: `server/python/src/metaobjects/codegen/generator_registry.py` (entry `shared-model`) +- Modify: `fixtures/generator-registry-conformance/registry.json` (`shared-model.ports` gains `"python"`) +- Modify: `server/python/src/metaobjects/codegen/project_config.py` (`TARGET_KEYS` gains `"options"`; `TargetConfig.options: dict[str, dict[str, object]]`), `server/python/src/metaobjects/codegen/metaobjects-config.schema.json` (`options` object keyed by generator name) +- Modify: `server/python/src/metaobjects/cli.py` (config-mode gen passes `options[]` to the factory when the registry entry accepts options; `shared-model` needs `name`/`include`/`exclude?`/`files?`/`version?`; `GenContext` gains `registry` and `source_files`) +- Modify: `server/python/src/metaobjects/codegen/generator.py` (`GenContext.registry`, `GenContext.source_files`) +- Create: `server/python/src/metaobjects/dependency_diff.py` (`classify_dependency_changes`, `WIDENING_ATTRS`, `ADDITIVE_ATTRS`, `DOC_ATTRS`) +- Modify: `server/python/src/metaobjects/config/dependencies.py` (`dependency_footprint`) +- Test: `tests/codegen/test_shared_model.py`, `tests/unit/test_dependency_diff.py`, `tests/conformance/test_dependency_conformance.py` (enable the classify arm), `tests/codegen/test_cli_registry.py` (existing — stays green) + +**Read first:** DESIGN §4.3, §4.6, §2.6; the TS implementations from Tasks 12 and 14 are the reference — port behavior and messages, not structure. `serialize_shared_document` (Task 2), `scope.py` (Task 4), `sha256_integrity` (Task 6). + +**Interfaces:** +- Produces: registry name `shared-model`; `shared_model(name, include, exclude=None, files=None, version=None) -> Generator`; `classify_dependency_changes(old_doc: str, new_doc: str, footprint: dict[str, str]) -> list[Change]` (`Change` dataclass: `fqn, path, kind, before, after, rule`); `dependency_footprint(root, collection) -> dict[str, tuple[ResolvedDependency, str]]`. `version` defaults to `pyproject.toml` `[project].version` found walking up from the config dir (use `tomllib`). + +- [ ] **Step 1: Failing generator test** `tests/codegen/test_shared_model.py`: a temp publisher with `metaobjects/meta.lib.json` (Task 2's `LIB`), an overlay file under `metaobjects/admin-ui/`, a `pyproject.toml` with `version = "1.0.0"`, and `metaobjects.config.yaml`: +```yaml +metadata: metaobjects +targets: + shared: + outDir: out/shared + generators: [shared-model] + options: + shared-model: + name: acme-common + files: [metaobjects/meta.lib.json] + include: ["acme::common::**"] +``` +`metaobjects gen` → `out/shared/acme-common.metaobjects.json` is byte-identical to the pinned v1 artifact and `metaobjects.pkg.json` equals the manifest from Task 6's test data; a closure failure (`include: ["acme::common::Address"]` with `Address` extending `Audited`) exits 1 with `references acme::common::Audited, which is not selected`; a consumer-provider attr in the selection fails with `core registry does not register`. Run → FAIL. Implement the generator, config `options`, `GenContext` fields, registry entries. Run → PASS, and: +```bash +cd server/python && uv run --extra integration pytest tests/codegen/test_shared_model.py tests/codegen/test_cli_registry.py tests/codegen/test_constants_config.py -q +cd server/typescript/packages/codegen-ts && bun test test/generator-registry.test.ts +``` +Expected: PASS (both ports now list `shared-model`). + +- [ ] **Step 2: Failing classifier tests** `tests/unit/test_dependency_diff.py` — the widening rows both directions, `DOC_ATTRS` ignored, and the footprint test mirroring Task 14 Step 3 (`{Audited: whole, Address: whole, Customer: key}`). Enable the `classify` arm in the Python corpus runner. Run → FAIL. Implement `dependency_diff.py` and `dependency_footprint`. Run: +```bash +cd server/python && uv run --extra integration pytest tests/unit/test_dependency_diff.py tests/conformance/test_dependency_conformance.py -q +``` +Expected: PASS (cases 29–46 green in Python). + +- [ ] **Step 3: Commit** +``` +feat(python): shared-model generator, per-target generator options, and the change classifier (FR-023) +``` + +--- + +### Task 20: The requirements ledger and `verify --templates` skip foreign nodes + +**Files:** +- Modify: `server/typescript/packages/cli/src/lib/requirement-check.ts` (`scanRequirements(root, opts?: { governs?: (fqn: string) => boolean })`) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (pass `governs`; the templates gate filters `template.*` nodes by `governs`) +- Test: `cli/test/verify-requirements-foreign.test.ts` (new) + +**Read first:** DESIGN §2.7 (ledger and templates rows). `verify.ts` lines around `scanRequirements(root)` / `summariseRequirements` and the `--templates` loop over `template.*` nodes. + +- [ ] **Step 1: Failing test** — scaffold a consumer whose artifact (case-local, in the test) carries `requirement.functional` node `acme::common::CustomerCanBeCreated` (`@implementedBy: acme::common::Customer`, `@status: live`) beside the v1 nodes (compute its hash for the lock), and a local model with one entity `app::Order` and no requirements. Run `verifyCommand([], app)` and assert the requirements line reads `… 0 entries … 0/1 entities claimed, counted over 2 metadata file(s), 1 from dependencies` — the foreign requirement and the foreign entity are not counted. Then add a local `requirement.functional` with `@implementedBy: acme::common::Customer` → loads clean, `0/1 entities claimed` still (a foreign entity is never in the denominator). Run → FAIL. Implement. Run → PASS; `bun test test/verify-requirements-e2e.test.ts test/docs-requirements-surface.test.ts` → PASS; typecheck → PASS. + +- [ ] **Step 2: Commit** +``` +feat(cli): verify's ledger and template gates skip dependency-owned nodes (FR-023) +``` + +--- + +### Task 21: `meta init` stops scaffolding `package.meta.json` and negates the dependency files in `.gitignore` + +**Files:** +- Modify: `server/typescript/packages/cli/src/commands/init.ts` (remove the `package.meta.json` scaffold block; remove `!package.meta.json` from `METAOBJECTS_GITIGNORE_BODY`; add `!deps/` and `!deps.lock.json` under the "These ARE meant to be tracked" comment — `deps.local.json` was added in Task 17) +- Modify: `server/typescript/packages/cli/test/init.test.ts` (flip the manifest test: asserts the file is NOT created; a pre-existing one is left untouched) +- Modify: `server/typescript/packages/cli/README.md` (the `meta init` scaffold list no longer names `package.meta.json`) + +**Read first:** DESIGN §0a (deprecation row), §10 ruling; the sdk `@deprecated` JSDoc on `package.ts`/`workspace.ts` is ALREADY DONE (`ef51fd1e1`) — do not touch sdk here. + +- [ ] **Step 1: Flip the test** — `test("does not scaffold package.meta.json (deprecated; removed in 2.0)")`: `existsSync(join(cwd, ".metaobjects", "package.meta.json"))` is `false` after `init({ cwd })`; a second test writes a stale one first and asserts `init` preserves it byte-for-byte. Also assert `.metaobjects/.gitignore` contains `!deps/`, `!deps.lock.json` and `deps.local.json`. Run `cd server/typescript/packages/cli && bun test test/init.test.ts` → FAIL. Implement. Run → PASS; `bun test test/init-layout.test.ts test/unit/init-scaffold-config.test.ts test/ignored-scaffold-check.test.ts` → PASS. + +- [ ] **Step 2: Commit** +``` +chore(cli): meta init no longer scaffolds package.meta.json; tracks deps/ and deps.lock.json (FR-023) +``` + +--- + +### Task 22: Docs, skills, CHANGELOG, CONFORMANCE matrix, roadmap, agent-context bundle + +**Files:** +- Create: `docs/features/metadata-dependencies.md` +- Modify: `docs/features/metadata-sources.md` (the ladder paragraph; "Vendoring — airgapped and hermetic builds"; "The workspace `extends:` walk is retired" gains the deprecation schedule) +- Modify: `docs/features/abstracts-and-inheritance.md` ("Overlays vs. extends — different concepts" gains the cross-repository paragraph and the `overlay: true` rule) +- Modify: `docs/features/cli.md` (matrix row + the `verify --deps` subverb row) +- Modify: `docs/CONFORMANCE.md` (corpus row + per-corpus section) +- Modify: `docs/README.md` (layout + "when to read which" rows) +- Modify: `spec/roadmap.md` (FR-023 status cell → Phase 1a shipped; the "doc-first quick wins" bullet closes) +- Modify: `agent-context/skills/metaobjects-authoring/SKILL.md`, `agent-context/skills/metaobjects-codegen/SKILL.md`, `agent-context/skills/metaobjects-verify/SKILL.md` +- Modify: `CHANGELOG.md` (`[Unreleased]`) +- Regenerate: the bundled agent context (`bun run --filter '@metaobjectsdev/sdk' bundle-agent-context`) and whatever `fixtures/agent-context-conformance/` needs (`cd server/typescript/packages/sdk && bun test test/agent-context-conformance.test.ts` tells you) + +**Read first:** DESIGN §0, §2.1–§2.7, §2.9, §10 (rulings 2, 3, 5 and D10 — the skills content is a ruling), §8 task 21's sentence: the skills teach, as ordinary work, (1) extending and overlaying a foreign node with `overlay: true`, (2) widening a `sharedModelFile` selection or splitting it into several artifacts, (3) co-developing with a `deps.local.json` override. Public-repo hygiene applies to every line. + +- [ ] **Step 1: Write `docs/features/metadata-dependencies.md`** with these sections, each with a runnable example: Declaring a dependency (config shape; transports `path`/`npm`/`python`; `mode`); Syncing (`meta deps sync`, what is committed: `deps/` + `deps.lock.json`; `meta deps list`); Referencing, extending and overlaying (FQN references; `extends` a foreign abstract; `overlay: true` is required — the exact `ERR_DEPENDENCY_OVERLAY_IMPLICIT` message; reference mode refuses structural overlays — `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`; own mode); What foreign means for each command (the §2.7 table); What breaks when upstream changes (the §2.5 table; the classifier: footprint scopes, the breaking/compatible/ignored/info rules, the widening and additive tables verbatim; `meta deps check` / `verify --deps`; `sync --accept-breaking`); Publishing (`sharedModelFile()` for TS and the `shared-model` target for Python, `files`/`include`/`exclude`, closure, core-vocabulary rule, where the artifact ships per ecosystem, "start with one artifact; widen `include` or add a second `sharedModelFile` when a consumer needs a different slice"); Co-developing (D10 — the file, the warning, sync-from-override, verify refusal, gitignore); Errors (the nine codes, one row each); The legacy pattern (three sentences for readers who knew the resources-only JAR). + +- [ ] **Step 2: Edit the existing docs** as listed (metadata-sources: "`dependencies` is read at every rung, including under a port's native surface"; cli.md row: `| **Dependency sync** (`deps`) | **Node `meta`** | `meta deps sync` / `check` / `list` | **any backend** — sync/check are Node-only (ADR-0015 pattern); every port loads the committed snapshot |` and the subverb row `| `verify --deps` | **Dependency drift** — the installed dependency versus the committed snapshot, classified against this consumer's footprint | no |`; CONFORMANCE row `| [`fixtures/dependency-conformance/`](../fixtures/dependency-conformance/) | 49 cases (31 resolution/load + 18 classify) | ✓ (reference implementation) | — (Phase 2) | — (Phase 2) | — (Phase 2) | ✓ |`). + +- [ ] **Step 3: Skills.** In `metaobjects-authoring/SKILL.md` under the `overlay` paragraph add "Building on another repository's model": a consumer declares a dependency, references by FQN, `extends` foreign abstracts, and overlays with `overlay: true` (required); an overlay in reference mode adds presentation, validators and docs, never structure; when upstream removes or renames the target the build fails — that failure is the feature. In `metaobjects-codegen/SKILL.md` add "Publishing a shared model": the `sharedModelFile()` / `shared-model` example, and the two ordinary operations "widen `include`" and "add a second `sharedModelFile` with its own `name` for a different audience" — with the rule that a node not in the selection is not a contract. In `metaobjects-verify/SKILL.md` add "Dependency drift": `verify --deps`, reading a BREAKING report (` : breaking ()`), `meta deps sync --accept-breaking ` and then fixing the model, and the D10 loop (write `deps.local.json`, expect the warning, `sync` to land, delete the file before `verify --deps`). + +- [ ] **Step 4: CHANGELOG `[Unreleased]`** — Added: `dependencies` in `.metaobjects/config.json`; `meta deps sync|check|list`; `meta verify --deps`; `sharedModelFile()` / `shared-model`; `.metaobjects/deps.local.json`; the nine codes; `fixtures/dependency-conformance/`. Changed: `FileSource` takes an `id` option; `Collection` gains `dependencies`, `ownFiles`, `fileIds`, `overrides`, `foreignOwner`, `governs`; `loadMemory` takes `fileIds`; the scope grammar is exported from `@metaobjectsdev/metadata` (sdk re-exports); `renderSharedEnumsFile` takes an `exclude` option; `GenContext` gains `registry` and `sourceFiles`; Python `targets..options`. Deprecated: `meta init` no longer writes `package.meta.json` (the sdk exports were deprecated in `ef51fd1e1`; removal at 2.0). State: `metamodelVersion` stays `1.0` — no registered vocabulary changed. + +- [ ] **Step 5: Regenerate and verify** +```bash +bun run --filter '@metaobjectsdev/sdk' bundle-agent-context +cd server/typescript/packages/sdk && bun test test/agent-context-conformance.test.ts test/agent-context-capability-grounding.test.ts test/dogfood-examples.test.ts +cd server/typescript/packages/cli && bun test test/help-lists-every-flag.test.ts test/docs-command.test.ts +``` +Expected: PASS; if the agent-context corpus needs regeneration, follow the instruction its failing test prints and commit the regenerated files in this same commit. + +- [ ] **Step 6: Commit** +``` +docs(fr-023): metadata dependencies — feature page, skills, CLI matrix, conformance matrix, changelog +``` + +--- + +### Task 23: The gate + +**Files:** none (a verification task). If anything is red, fix it in a follow-up commit on this branch and re-run. + +- [ ] **Step 1: Cleaned tree.** `git status --short` must be empty (everything committed). Run `git clean -fdX -n` to see what a clean CI tree would drop; if generated artifacts you depend on appear, they are untracked build state and must not be needed by any test. + +- [ ] **Step 2: Run the three lanes** +```bash +scripts/ci-local.sh --only ts +scripts/ci-local.sh --only python +scripts/ci-local.sh --only gates +``` +Expected: each exits 0. The `gates` lane includes `node scripts/check-metamodel-version.mjs` (must report no vocabulary diff), `publish-set parity`, `ci lane selection`, and the fixture lints (`ERROR-CODES.json` codes referenced by the new corpus exist in the registry). + +- [ ] **Step 3: Cross-check the design's discipline list** (DESIGN §2.11): grep the diff of this branch against `main` for `instanceof Meta` outside `server/typescript/packages/metadata/src` (expected: none), for `own(Children|Attrs|Fields)\(` without an ADR-0039 comment on the same or previous line (expected: none new), for `/home/` and `~/` (expected: none), and for the private names the public-repo hook denies (`git config hooks.denyListPath` names the list; run the hook: `.githooks/pre-commit` against a no-op commit). + +- [ ] **Step 4: Record the result** in the final message to the coordinator: the three lane exit codes, the corpus counts (TS: 49/49; Python: 49/49), and any deviation from this plan with the task number it affected. From b282a00a95dc264d4cc45acef5bc5a894864b221 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 13:32:15 -0400 Subject: [PATCH 02/62] =?UTF-8?q?docs(plans):=20FR-023=20phase=201a=20?= =?UTF-8?q?=E2=80=94=20pre-flight=20rulings=20written=20into=20the=20task?= =?UTF-8?q?=20sections?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../plans/2026-09-11-fr-023-phase-1a.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index edcaf5ec3..b1445b3d7 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -771,6 +771,8 @@ feat(sdk): dependency boundary validator — explicit overlays, reference-mode s --- +**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** `validateDependencyBoundary` stays SYNCHRONOUS: the resolved-dependency record produced by the snapshot check (Task 6/7) carries the artifact text it already read to hash, and the validator parses the foreign subtree from that — no file I/O in the validator, no async ripple through the CLI load sites. Add the missing test: a `reference`-mode overlay whose local contribution sets a physical attribute (`@table`, `@schema`, `@column`, `@kind`) differing from the artifact → `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`. + ### Task 9: Codegen selection excludes foreign nodes (`gen`, `verify --codegen`, shared enums) **Files:** @@ -815,6 +817,8 @@ feat(cli,codegen-ts): reference-mode dependency nodes are never selected for cod --- +**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** There is NO existing refusal path to extend: today `gen`/`runner` only warn on an unmatched entity name. Add a NEW pre-check in `gen.ts` (and `verify --codegen`'s selection) that runs before generation: an explicitly named entity (`meta gen ` / `entities:`) whose FQN is foreign in `reference` mode → refuse with exit code 2 and the message naming the dependency. Leave the existing warn-on-unmatched behaviour for non-foreign names unchanged. + ### Task 10: Migrate and `verify --db` never govern reference-mode nodes **Files:** @@ -1013,6 +1017,8 @@ feat(metadata,sdk): usage-aware dependency change classifier and footprint (FR-0 --- +**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** Add the registered `summary` documentation attribute to the ignored documentation-attr set, and build that set from the named constants in `doc-constants.ts` (including `DOC_ATTR_SUMMARY`), never string literals. Add a classifier test: a summary-only upstream edit is ignored. + ### Task 15: `sync` refuses BREAKING upstream changes; `meta deps check` and `meta verify --deps` **Files:** @@ -1113,6 +1119,8 @@ feat(sdk,cli): local co-development override via .metaobjects/deps.local.json (F --- +**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** The collection resolver's fast path (no declared dependencies and no lock → skip snapshot verification) must ALSO require that no override is present. A `deps.local.json` entry naming a dependency that is not declared → `ERR_DEPENDENCY_UNRESOLVED`, never silently ignored. Add that test. + ### Task 18: Python — the collection resolver, CLI, runner, boundary validator, and D10 **Files:** @@ -1147,6 +1155,8 @@ feat(python): dependency snapshot loading, foreign-node selection, boundary vali --- +**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** `resolve_collection_files` does not exist. Add a new resolver that returns the full result (files + resolved dependencies + foreign predicate inputs) and keep `resolve_collection(root) -> list[Path]` as a thin projection of it, so its existing shape and callers are unchanged. ALSO (cross-port parity with Task 9): Python's materialized shared-enum pass in `entity_model.py` must exclude foreign `reference`-mode enums, exactly as the TS shared-enums artifact does; add the Python test mirroring Task 9's. + ### Task 19: Python — the `shared-model` generator, `targets..options`, and the classifier twin **Files:** @@ -1216,6 +1226,8 @@ feat(cli): verify's ledger and template gates skip dependency-owned nodes (FR-02 --- +**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** DROP the `verify --templates` `governs` filter from this task: `sharedModelFile` / `shared-model` never exports `template.*` nodes, so the filter could never fire. This task keeps only the requirements-ledger exclusion (foreign nodes are not counted in `entitiesTotal`, a foreign `requirement.*` is not the consumer's claim). + ### Task 21: `meta init` stops scaffolding `package.meta.json` and negates the dependency files in `.gitignore` **Files:** @@ -1234,6 +1246,8 @@ chore(cli): meta init no longer scaffolds package.meta.json; tracks deps/ and de --- +**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** `cli/test/init-layout.test.ts` does not exist. Put the tests in the existing `init` test file under `server/typescript/packages/cli/test/` (find it with `git ls-files | grep init`). + ### Task 22: Docs, skills, CHANGELOG, CONFORMANCE matrix, roadmap, agent-context bundle **Files:** @@ -1273,6 +1287,8 @@ docs(fr-023): metadata dependencies — feature page, skills, CLI matrix, confor --- +**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** Add `docs/features/entities.md` to this task's file list (the design's §5.3 names it; the file exists). + ### Task 23: The gate **Files:** none (a verification task). If anything is red, fix it in a follow-up commit on this branch and re-run. From 1fee2e03ec62c7306c8d5b3a38f6dc74f6519057 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 13:39:24 -0400 Subject: [PATCH 03/62] feat(conformance): FR-023 dependency corpus skeleton, pinned artifacts, and nine error codes Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- fixtures/conformance/ERROR-CODES.json | 11 +- fixtures/dependency-conformance/README.md | 120 +++++++++++++ .../artifacts/acme-common-v1-widened.json | 70 ++++++++ .../artifacts/acme-common-v1.json | 70 ++++++++ .../acme-common-v2-email-removed.json | 64 +++++++ fixtures/dependency-conformance/cases.json | 14 ++ server/python/src/metaobjects/errors.py | 26 +++ .../packages/metadata/src/errors.ts | 26 +++ .../sdk/test/dependency-conformance.test.ts | 162 ++++++++++++++++++ 9 files changed, 562 insertions(+), 1 deletion(-) create mode 100644 fixtures/dependency-conformance/README.md create mode 100644 fixtures/dependency-conformance/artifacts/acme-common-v1-widened.json create mode 100644 fixtures/dependency-conformance/artifacts/acme-common-v1.json create mode 100644 fixtures/dependency-conformance/artifacts/acme-common-v2-email-removed.json create mode 100644 fixtures/dependency-conformance/cases.json create mode 100644 server/typescript/packages/sdk/test/dependency-conformance.test.ts diff --git a/fixtures/conformance/ERROR-CODES.json b/fixtures/conformance/ERROR-CODES.json index f3b77dc60..27bc964c4 100644 --- a/fixtures/conformance/ERROR-CODES.json +++ b/fixtures/conformance/ERROR-CODES.json @@ -88,6 +88,15 @@ "ERR_ENUM_EXTENDS_VALUES_CONFLICT": "A field.enum both extends a shared package-level abstract enum and declares its own @values. One shared enum type has one member set — the own @values would be silently dropped in codegen. Remove the own @values to inherit the shared set, or extend a concrete (non-shared) enum instead.", "ERR_ENUM_INT_VALUE_MAP_ARRAY": "A field.enum carries @intValueMap together with isArray=true. Int-backing is a persistence-layer codec and no port implements it element-wise over an array column, so the combination would silently persist member SYMBOLS into an integer array. An array-of-enum stays string-backed: drop @intValueMap, or make the field scalar.", "ERR_REQUIREMENT_RETIRED_HAS_IMPLEMENTORS": "A requirement.* with @status: retired declares @implementedBy. Refused rather than exempted (FR-039): a retired capability has no implementation by definition, so forbidding the attribute makes the dangling-reference class unreachable instead of silently tolerated.", - "ERR_REQUIREMENT_SUPERSEDED_BY_NOT_RETIRED": "@supersededBy on a requirement whose @status is not `retired`. The attribute names what REPLACED a withdrawn capability; on a live one there is nothing to have replaced it." + "ERR_REQUIREMENT_SUPERSEDED_BY_NOT_RETIRED": "@supersededBy on a requirement whose @status is not `retired`. The attribute names what REPLACED a withdrawn capability; on a live one there is nothing to have replaced it.", + "ERR_DEPENDENCY_UNRESOLVED": "FR-023: a declared dependency's transport or local override could not locate a directory holding metaobjects.pkg.json, or an override names an undeclared dependency.", + "ERR_DEPENDENCY_MANIFEST_INVALID": "FR-023: a dependency's metaobjects.pkg.json fails its schema, names a different dependency, points at a missing or hash-mismatched artifact, or its artifact does not load standalone / does not declare exactly the listed packages and nodes.", + "ERR_DEPENDENCY_SNAPSHOT_STALE": "FR-023: the committed snapshot does not match .metaobjects/deps.lock.json (lock missing, entry missing or extra, artifact missing, or hash mismatch) — run `meta deps sync`.", + "ERR_DEPENDENCY_NODE_COLLISION": "FR-023: two dependencies export the same fully-qualified node.", + "ERR_DEPENDENCY_OVERLAY_IMPLICIT": "FR-023: a local top-level node redeclares a dependency's node without `overlay: true`.", + "ERR_DEPENDENCY_SCHEMA_NOT_OWNED": "FR-023: a local contribution changes the physical shape of a reference-mode dependency's node (a field/identity/index/relationship/source child, a physical attribute, or a TPH subtype of a foreign base).", + "ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE": "FR-023: a dependency's metamodelVersion major differs from this toolchain's.", + "ERR_DEPENDENCY_UPSTREAM_DRIFT": "FR-023: `meta deps check` / `verify --deps` found the installed dependency differs from the committed snapshot.", + "ERR_DEPENDENCY_BREAKING_CHANGE": "FR-023: `meta deps sync` refused an upstream change classified BREAKING for this consumer's footprint (override with --accept-breaking)." } } diff --git a/fixtures/dependency-conformance/README.md b/fixtures/dependency-conformance/README.md new file mode 100644 index 000000000..a12a257d6 --- /dev/null +++ b/fixtures/dependency-conformance/README.md @@ -0,0 +1,120 @@ +# dependency-conformance + +Pins how FR-023 metadata dependencies (declared in `.metaobjects/config.json`, +resolved via a manifest + lock + committed snapshot) resolve, govern codegen/ +migrate, overlay, fail, and classify upstream changes — across every port. +Every port's runner reads THIS file; there is no per-port fixture. + +## Shape + +``` +cases.json # { cases: [ { name, tree, treeFiles?, config, lock?, localOverrides?, + # resolveFrom?, expectFiles?, expectForeign?, expectGoverned?, + # expectOverrides?, expectLoadError?, expectErrorFiles?, + # expectError?, classify? } ] } +README.md +artifacts/ # pinned dependency artifacts, referenced by cases via `treeFiles` + acme-common-v1.json + acme-common-v1-widened.json + acme-common-v2-email-removed.json +``` + +## Case schema + +```jsonc +{ "cases": [ { + "name": "…", // kebab-case, the contract + "tree": { "": "" }, // materialized under a fresh temp root + "treeFiles": { "": "artifacts/" }, // OPTIONAL: copied byte-for-byte from the corpus dir + "config": { … } | null, // written to /.metaobjects/config.json + "lock": { … }, // OPTIONAL: written to /.metaobjects/deps.lock.json + "localOverrides": { … }, // OPTIONAL: written to /.metaobjects/deps.local.json + "resolveFrom": ".", // OPTIONAL + "expectFiles": ["…"], // unordered set, project-root-relative (resolution arm) + "expectForeign": [""], // OPTIONAL: FQNs with a foreign owner + "expectGoverned": [""], // OPTIONAL: FQNs governs() admits, over every loaded top-level object + "expectOverrides": [""], // OPTIONAL: dependencies read from a local override + "expectLoadError": "ERR_*", // OPTIONAL: the collection resolves, then LOADING it fails with this code + "expectErrorFiles": ["dep:…"], // OPTIONAL with expectLoadError: source.files[0] of the first error + "expectError": "ERR_*", // resolution itself fails with this code + "classify": { // classifier arm (TS + Python only) + "old": { … }, "new": { … }, // two artifact documents, inline + "footprint": { "": "whole" | "key" | "existence" }, + "expectChanges": [ { "fqn": "…", "path": "…", "kind": "breaking" | "compatible" | "info" } ] } +} ] } +``` + +Exactly one of `expectFiles`, `expectError`, `classify` is present per case; `expectLoadError` +rides with `expectFiles`. + +- **`tree`** — a map of project-root-relative path → file content, materialized in a fresh + temporary directory (same shape as `source-resolution-conformance`). +- **`treeFiles`** — OPTIONAL, a map of project-root-relative path → a path under this + corpus directory (`artifacts/`). A `treeFiles` entry copies a corpus file + byte-for-byte so a pinned hash can never drift — it is never re-serialized or + re-encoded on the way into the case's temp directory, unlike `tree`, which writes a + JSON string value as text. +- **`config`** — written verbatim to `/.metaobjects/config.json`. `null` means + no config file is created. +- **`lock`** — OPTIONAL, written verbatim to `/.metaobjects/deps.lock.json`. +- **`localOverrides`** — OPTIONAL, written verbatim to `/.metaobjects/deps.local.json` + (D10 — local co-development override). +- **`resolveFrom`** — OPTIONAL, project-root-relative directory the resolver is invoked + against; default `"."`. +- **`expectFiles`** — the resolution arm. Project-root-relative paths, compared as an + unordered set (same contract as `source-resolution-conformance`). +- **`expectForeign`** — OPTIONAL, alongside `expectFiles`: FQNs for which + `collection.foreignOwner(fqn) !== undefined`. +- **`expectGoverned`** — OPTIONAL, alongside `expectFiles`: the FQN set, over every loaded + top-level object, for which `collection.governs(fqn)` is true. +- **`expectOverrides`** — OPTIONAL, alongside `expectFiles`: the dependency names + `collection.overrides` reads from an active `deps.local.json`. +- **`expectLoadError`** — OPTIONAL, alongside `expectFiles`: the collection resolves + cleanly, but LOADING it (parsing the resolved files into a metadata tree) fails with + this code. +- **`expectErrorFiles`** — OPTIONAL, alongside `expectLoadError`: `source.files[0]` of the + first load error — asserts the failure names the right file, e.g. a dependency artifact + (`dep:/`) rather than a local one. +- **`expectError`** — the resolution-failure arm: `resolveCollection` itself must reject + with this exact code. +- **`classify`** — the classifier arm (TS + Python only; see below). Two inline artifact + documents (`old`/`new`), a footprint map, and the expected change list. + +## Pinned artifacts + +`artifacts/acme-common-v1.json` is the base publisher artifact (`acme::common::Address`, +`acme::common::Audited` [abstract], `acme::common::Customer` with an `email` +`field.string @maxLength: 120`). `acme-common-v1-widened.json` is identical except +`@maxLength: 200` (a compatible widening). `acme-common-v2-email-removed.json` is +identical except the whole `email` field child is absent (a breaking removal). + +Verify the bytes from the repo root: + +```bash +python3 -c 'import hashlib,sys; [print("sha256-"+hashlib.sha256(open(f,"rb").read()).hexdigest(), f) for f in sys.argv[1:]]' fixtures/dependency-conformance/artifacts/*.json +``` + +Expected, exactly: + +``` +sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d fixtures/dependency-conformance/artifacts/acme-common-v1.json +sha256-fa00b9f3c54a2c302cf269af051589afc0a3217999ac82c54c77e33494638c36 fixtures/dependency-conformance/artifacts/acme-common-v1-widened.json +sha256-c4ba364bff0071b164b127326c095d6d32915b57781271e0be64a7003c27ce7a fixtures/dependency-conformance/artifacts/acme-common-v2-email-removed.json +``` + +If a hash differs, fix the bytes (whitespace, trailing newline, CRLF) — never change the +pinned values; a hash change here is a change to what every port's `sync`/manifest test +asserts against. + +## Which arms each port runs + +- **TypeScript** — all arms (resolution/foreignness, overlays, load-time failure, lock/ + snapshot integrity, classification). +- **Python** — all arms. +- **Java / C# / Kotlin** — Phase 2 (out of scope for this plan; these ports do not read + `dependencies` yet). + +## Reference implementation + +`server/typescript/packages/sdk/src/collection.ts` (`resolveCollection`) and +`server/typescript/packages/sdk/src/memory.ts` (`loadMemory`). diff --git a/fixtures/dependency-conformance/artifacts/acme-common-v1-widened.json b/fixtures/dependency-conformance/artifacts/acme-common-v1-widened.json new file mode 100644 index 000000000..624afa092 --- /dev/null +++ b/fixtures/dependency-conformance/artifacts/acme-common-v1-widened.json @@ -0,0 +1,70 @@ +{ + "metadata.root": { + "children": [ + { + "object.value": { + "name": "Address", + "package": "acme::common", + "children": [ + { + "field.string": { + "name": "city" + } + }, + { + "field.string": { + "name": "street" + } + } + ] + } + }, + { + "object.entity": { + "name": "Audited", + "package": "acme::common", + "abstract": true, + "children": [ + { + "field.timestamp": { + "name": "createdAt" + } + } + ] + } + }, + { + "object.entity": { + "name": "Customer", + "package": "acme::common", + "children": [ + { + "source.rdb": { + "@table": "customers" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "email", + "@maxLength": 200 + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ] + } +} diff --git a/fixtures/dependency-conformance/artifacts/acme-common-v1.json b/fixtures/dependency-conformance/artifacts/acme-common-v1.json new file mode 100644 index 000000000..72f2dd9ae --- /dev/null +++ b/fixtures/dependency-conformance/artifacts/acme-common-v1.json @@ -0,0 +1,70 @@ +{ + "metadata.root": { + "children": [ + { + "object.value": { + "name": "Address", + "package": "acme::common", + "children": [ + { + "field.string": { + "name": "city" + } + }, + { + "field.string": { + "name": "street" + } + } + ] + } + }, + { + "object.entity": { + "name": "Audited", + "package": "acme::common", + "abstract": true, + "children": [ + { + "field.timestamp": { + "name": "createdAt" + } + } + ] + } + }, + { + "object.entity": { + "name": "Customer", + "package": "acme::common", + "children": [ + { + "source.rdb": { + "@table": "customers" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "email", + "@maxLength": 120 + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ] + } +} diff --git a/fixtures/dependency-conformance/artifacts/acme-common-v2-email-removed.json b/fixtures/dependency-conformance/artifacts/acme-common-v2-email-removed.json new file mode 100644 index 000000000..2c70682d3 --- /dev/null +++ b/fixtures/dependency-conformance/artifacts/acme-common-v2-email-removed.json @@ -0,0 +1,64 @@ +{ + "metadata.root": { + "children": [ + { + "object.value": { + "name": "Address", + "package": "acme::common", + "children": [ + { + "field.string": { + "name": "city" + } + }, + { + "field.string": { + "name": "street" + } + } + ] + } + }, + { + "object.entity": { + "name": "Audited", + "package": "acme::common", + "abstract": true, + "children": [ + { + "field.timestamp": { + "name": "createdAt" + } + } + ] + } + }, + { + "object.entity": { + "name": "Customer", + "package": "acme::common", + "children": [ + { + "source.rdb": { + "@table": "customers" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ] + } +} diff --git a/fixtures/dependency-conformance/cases.json b/fixtures/dependency-conformance/cases.json new file mode 100644 index 000000000..097513bf2 --- /dev/null +++ b/fixtures/dependency-conformance/cases.json @@ -0,0 +1,14 @@ +{ + "cases": [ + { + "name": "no-dependencies-resolves-exactly-as-before", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "config": { "schema_version": 1, "sources": [] }, + "expectFiles": ["metaobjects/meta.app.json"], + "expectForeign": [], + "expectGoverned": ["app::Order"] + } + ] +} diff --git a/server/python/src/metaobjects/errors.py b/server/python/src/metaobjects/errors.py index 904a584a8..4df536735 100644 --- a/server/python/src/metaobjects/errors.py +++ b/server/python/src/metaobjects/errors.py @@ -125,6 +125,32 @@ class ErrorCode(str, Enum): ERR_SCOPE_PATTERN_INVALID = "ERR_SCOPE_PATTERN_INVALID" # Phase-1 metadata-source-resolution — no metadata collection was discovered: no config declaring sources, and no default metaobjects/ directory. ERR_COLLECTION_NOT_FOUND = "ERR_COLLECTION_NOT_FOUND" + # FR-023 — a declared dependency's transport or local override could not locate a + # directory holding metaobjects.pkg.json, or an override names an undeclared dependency. + ERR_DEPENDENCY_UNRESOLVED = "ERR_DEPENDENCY_UNRESOLVED" + # FR-023 — a dependency's metaobjects.pkg.json fails its schema, names a different + # dependency, points at a missing/hash-mismatched artifact, or the artifact does not + # load standalone / does not declare exactly the listed packages and nodes. + ERR_DEPENDENCY_MANIFEST_INVALID = "ERR_DEPENDENCY_MANIFEST_INVALID" + # FR-023 — the committed snapshot does not match .metaobjects/deps.lock.json (lock + # missing, entry missing or extra, artifact missing, or hash mismatch). + ERR_DEPENDENCY_SNAPSHOT_STALE = "ERR_DEPENDENCY_SNAPSHOT_STALE" + # FR-023 — two dependencies export the same fully-qualified node. + ERR_DEPENDENCY_NODE_COLLISION = "ERR_DEPENDENCY_NODE_COLLISION" + # FR-023 — a local top-level node redeclares a dependency's node without `overlay: true`. + ERR_DEPENDENCY_OVERLAY_IMPLICIT = "ERR_DEPENDENCY_OVERLAY_IMPLICIT" + # FR-023 — a local contribution changes the physical shape of a reference-mode + # dependency's node (a field/identity/index/relationship/source child, a physical + # attribute, or a TPH subtype of a foreign base). + ERR_DEPENDENCY_SCHEMA_NOT_OWNED = "ERR_DEPENDENCY_SCHEMA_NOT_OWNED" + # FR-023 — a dependency's metamodelVersion major differs from this toolchain's. + ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE = "ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE" + # FR-023 — `meta deps check` / `verify --deps` found the installed dependency differs + # from the committed snapshot. + ERR_DEPENDENCY_UPSTREAM_DRIFT = "ERR_DEPENDENCY_UPSTREAM_DRIFT" + # FR-023 — `meta deps sync` refused an upstream change classified BREAKING for this + # consumer's footprint (override with --accept-breaking). + ERR_DEPENDENCY_BREAKING_CHANGE = "ERR_DEPENDENCY_BREAKING_CHANGE" # FR-016 / ADR-0018 — per-kind physical-name aliases on source.rdb. ERR_PHYSICAL_NAME_KIND_MISMATCH = "ERR_PHYSICAL_NAME_KIND_MISMATCH" ERR_PHYSICAL_NAME_MULTIPLE = "ERR_PHYSICAL_NAME_MULTIPLE" diff --git a/server/typescript/packages/metadata/src/errors.ts b/server/typescript/packages/metadata/src/errors.ts index 0e8f383d2..91e967045 100644 --- a/server/typescript/packages/metadata/src/errors.ts +++ b/server/typescript/packages/metadata/src/errors.ts @@ -260,6 +260,32 @@ export const ERROR_CODES = [ // Phase-1 metadata-source-resolution — no metadata collection was discovered: // no config declaring sources, and no default metaobjects/ directory. "ERR_COLLECTION_NOT_FOUND", + // FR-023 — a declared dependency's transport or local override could not locate a + // directory holding metaobjects.pkg.json, or an override names an undeclared dependency. + "ERR_DEPENDENCY_UNRESOLVED", + // FR-023 — a dependency's metaobjects.pkg.json fails its schema, names a different + // dependency, points at a missing/hash-mismatched artifact, or the artifact does not + // load standalone / does not declare exactly the listed packages and nodes. + "ERR_DEPENDENCY_MANIFEST_INVALID", + // FR-023 — the committed snapshot does not match .metaobjects/deps.lock.json (lock + // missing, entry missing or extra, artifact missing, or hash mismatch). + "ERR_DEPENDENCY_SNAPSHOT_STALE", + // FR-023 — two dependencies export the same fully-qualified node. + "ERR_DEPENDENCY_NODE_COLLISION", + // FR-023 — a local top-level node redeclares a dependency's node without `overlay: true`. + "ERR_DEPENDENCY_OVERLAY_IMPLICIT", + // FR-023 — a local contribution changes the physical shape of a reference-mode + // dependency's node (a field/identity/index/relationship/source child, a physical + // attribute, or a TPH subtype of a foreign base). + "ERR_DEPENDENCY_SCHEMA_NOT_OWNED", + // FR-023 — a dependency's metamodelVersion major differs from this toolchain's. + "ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE", + // FR-023 — `meta deps check` / `verify --deps` found the installed dependency differs + // from the committed snapshot. + "ERR_DEPENDENCY_UPSTREAM_DRIFT", + // FR-023 — `meta deps sync` refused an upstream change classified BREAKING for this + // consumer's footprint (override with --accept-breaking). + "ERR_DEPENDENCY_BREAKING_CHANGE", "ERR_UNKNOWN", ] as const; diff --git a/server/typescript/packages/sdk/test/dependency-conformance.test.ts b/server/typescript/packages/sdk/test/dependency-conformance.test.ts new file mode 100644 index 000000000..3a02bbb3f --- /dev/null +++ b/server/typescript/packages/sdk/test/dependency-conformance.test.ts @@ -0,0 +1,162 @@ +// Runs the shared dependency corpus (FR-023) against the TypeScript reference +// implementation. Every port ships an equivalent runner reading this same file +// (fixtures/dependency-conformance/, see its README for the case schema). +// +// This runner is written AHEAD of the implementation (TDD): the `classify` arm +// is skipped until the classifier lands (Task 14), and the `expectFiles` arm +// already calls `collection.foreignOwner` / `collection.governs` / +// `collection.overrides`, none of which exist on `Collection` yet — those +// calls are expected to fail with a TypeError until later tasks add them. +import { describe, expect, test } from "bun:test"; +import { copyFile, mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join, relative, resolve, sep } from "node:path"; +import { TYPE_OBJECT } from "@metaobjectsdev/metadata"; +import { resolveCollection } from "../src/collection.js"; +import { loadMemory } from "../src/memory.js"; + +interface ClassifyCase { + readonly old: unknown; + readonly new: unknown; + readonly footprint: Record; + readonly expectChanges: ReadonlyArray<{ + readonly fqn: string; + readonly path: string; + readonly kind: "breaking" | "compatible" | "info"; + }>; +} + +interface Case { + readonly name: string; + readonly tree: Record; + /** OPTIONAL: project-root-relative path -> a path under this corpus dir + * (`artifacts/`), copied byte-for-byte. See the corpus README. */ + readonly treeFiles?: Record; + readonly config: unknown | null; + /** OPTIONAL: written to `/.metaobjects/deps.lock.json`. */ + readonly lock?: unknown; + /** OPTIONAL: written to `/.metaobjects/deps.local.json` (D10). */ + readonly localOverrides?: unknown; + readonly resolveFrom?: string; + readonly expectFiles?: readonly string[]; + readonly expectForeign?: readonly string[]; + readonly expectGoverned?: readonly string[]; + readonly expectOverrides?: readonly string[]; + readonly expectLoadError?: string; + readonly expectErrorFiles?: readonly string[]; + readonly expectError?: string; + readonly classify?: ClassifyCase; +} + +const CORPUS_DIR = resolve(import.meta.dir, "../../../../../fixtures/dependency-conformance"); +const CORPUS = join(CORPUS_DIR, "cases.json"); + +/** Materializes `c.tree` (and `c.treeFiles`, copied byte-for-byte) under a + * fresh temp root, then writes `config` / `lock` / `localOverrides` (each + * when non-null/present) under `/.metaobjects/`. Mirrors + * `source-resolution-conformance.test.ts`'s `materialize`, extended for the + * dependency-corpus-only fields. */ +async function materialize(c: Case): Promise<{ root: string; resolveDir: string }> { + const root = await mkdtemp(join(tmpdir(), "mo-dep-conf-")); + for (const [rel, content] of Object.entries(c.tree)) { + const abs = join(root, rel); + await mkdir(dirname(abs), { recursive: true }); + await writeFile(abs, content); + } + for (const [rel, corpusRel] of Object.entries(c.treeFiles ?? {})) { + const abs = join(root, rel); + await mkdir(dirname(abs), { recursive: true }); + await copyFile(join(CORPUS_DIR, corpusRel), abs); + } + const resolveDir = resolve(root, c.resolveFrom ?? "."); + const metaobjectsDir = join(resolveDir, ".metaobjects"); + if (c.config !== null) { + await mkdir(metaobjectsDir, { recursive: true }); + await writeFile(join(metaobjectsDir, "config.json"), JSON.stringify(c.config, null, 2)); + } + if (c.lock !== undefined) { + await mkdir(metaobjectsDir, { recursive: true }); + await writeFile(join(metaobjectsDir, "deps.lock.json"), JSON.stringify(c.lock, null, 2)); + } + if (c.localOverrides !== undefined) { + await mkdir(metaobjectsDir, { recursive: true }); + await writeFile(join(metaobjectsDir, "deps.local.json"), JSON.stringify(c.localOverrides, null, 2)); + } + return { root, resolveDir }; +} + +const cases: Case[] = JSON.parse(await readFile(CORPUS, "utf8")).cases; + +describe("dependency conformance", () => { + test("corpus is non-empty (a silent zero-case run is a failed gate)", () => { + expect(cases.length).toBeGreaterThan(0); + }); + + for (const c of cases) { + // The classifier arm has no runner machinery yet (Task 14 fills it in) — + // skipped rather than failed, so the corpus can carry classify cases + // ahead of the classifier existing. + if (c.classify !== undefined) { + test.skip(c.name, () => {}); + continue; + } + + test(c.name, async () => { + const { root, resolveDir } = await materialize(c); + + if (c.expectError !== undefined) { + await expect(resolveCollection(resolveDir, { explicitDir: resolveDir })).rejects.toMatchObject({ + code: c.expectError, + }); + return; + } + + // A case with neither expectFiles, expectError, nor classify is a malformed + // corpus entry, not "expect zero files" — fail loudly rather than silently + // passing it (same discipline as source-resolution-conformance). + if (c.expectFiles === undefined) { + throw new Error(`corpus case "${c.name}" has neither expectFiles, expectError, nor classify`); + } + + const collection = await resolveCollection(resolveDir, { explicitDir: resolveDir }); + const got = collection.files.map((f) => relative(root, f).split(sep).join("/")).sort(); + expect(got).toEqual([...c.expectFiles].sort()); + + if (c.expectForeign !== undefined) { + for (const fqn of c.expectForeign) { + expect(collection.foreignOwner(fqn)).not.toBeUndefined(); + } + } + + if (c.expectGoverned !== undefined) { + const loaded = await loadMemory(resolveDir, { files: collection.files }); + const governed = loaded + .childrenOfType(TYPE_OBJECT) + .map((n) => n.resolutionKey()) + .filter((fqn) => collection.governs(fqn)) + .sort(); + expect(governed).toEqual([...c.expectGoverned].sort()); + } + + if (c.expectOverrides !== undefined) { + expect([...collection.overrides].sort()).toEqual([...c.expectOverrides].sort()); + } + + if (c.expectLoadError !== undefined) { + const attempt = loadMemory(resolveDir, { files: collection.files, fileIds: collection.fileIds }); + await expect(attempt).rejects.toMatchObject({ code: c.expectLoadError }); + if (c.expectErrorFiles !== undefined) { + let thrown: unknown; + try { + await attempt; + } catch (e) { + thrown = e; + } + expect((thrown as { source?: { files?: readonly string[] } }).source?.files).toEqual( + c.expectErrorFiles, + ); + } + } + }); + } +}); From dc5d0720fb591d95904d1492f532e8e0a8ba4de0 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 13:51:52 -0400 Subject: [PATCH 04/62] =?UTF-8?q?feat(metadata):=20serializeSharedDocument?= =?UTF-8?q?=20=E2=80=94=20node-level=20packages,=20no=20root=20package=20(?= =?UTF-8?q?FR-023)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A shared document is one canonical-JSON metadata.root with NO root package, whose top-level nodes each carry their own explicit package, so it re-loads to the same resolution keys in every port. Each node is its own-layer canonical form (extends preserved, attrs alphabetized, FR-016 rewrite applied); top-level nodes sort by resolution key, children keep their authored order, and the body key order is name, package, then the canonical rest. A node with no package is refused. - TS: serializeSharedDocument + packageOfResolutionKey, exported from the index. - Python: serialize_shared_document + package_of_resolution_key, byte-identical (both reproduce the pinned dependency-conformance artifact acme-common-v1.json). - fixtures/conformance/xpkg-node-level-package-no-root-package: the artifact as loader input and as expected canonical output; green in TS, Python, Java, Kotlin and C#. - The metamodel fixture count in docs/CONFORMANCE.md, AGENTS.md and README.md moves 313 -> 314. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- AGENTS.md | 2 +- README.md | 2 +- docs/CONFORMANCE.md | 6 +- .../expected.json | 70 +++++++++++++++++++ .../input/meta.shared.json | 70 +++++++++++++++++++ server/python/src/metaobjects/naming.py | 8 +++ .../python/src/metaobjects/serializer_json.py | 36 +++++++++- .../unit/test_serializer_shared_document.py | 51 ++++++++++++++ .../typescript/packages/metadata/src/index.ts | 4 +- .../packages/metadata/src/naming.ts | 11 +++ .../packages/metadata/src/serializer-json.ts | 47 ++++++++++++- .../test/serializer-shared-document.test.ts | 45 ++++++++++++ 12 files changed, 343 insertions(+), 9 deletions(-) create mode 100644 fixtures/conformance/xpkg-node-level-package-no-root-package/expected.json create mode 100644 fixtures/conformance/xpkg-node-level-package-no-root-package/input/meta.shared.json create mode 100644 server/python/tests/unit/test_serializer_shared_document.py create mode 100644 server/typescript/packages/metadata/test/serializer-shared-document.test.ts diff --git a/AGENTS.md b/AGENTS.md index c67e8237e..bd0dc5ec0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,7 +66,7 @@ PyPI has had no product change since `0.25.0` — nothing is broken. - **Kotlin** — `codegen-kotlin` (KotlinPoet on JVM): entity + Exposed table + Spring controller + payload + relations + filter allowlist + validator + stored-proc + output-parser generators. `integration-tests-kotlin` runs the persistence-conformance corpus through Exposed against Testcontainers Postgres. **Cross-port conformance corpora** (every port runs the shared corpus): -- Metamodel: `fixtures/conformance/` (313 fixtures; 21 shared corpora in total — per-corpus counts + the corpus x port matrix live in `docs/CONFORMANCE.md`). TS / C# / Java / Python all green. +- Metamodel: `fixtures/conformance/` (314 fixtures; 21 shared corpora in total — per-corpus counts + the corpus x port matrix live in `docs/CONFORMANCE.md`). TS / C# / Java / Python all green. - Render: `fixtures/render-conformance/`. TS / C# / Java / Kotlin / Python byte-identical. - Persistence: `fixtures/persistence-conformance/`. **Query** scenarios run on every port (TS / C# / Java / Kotlin / Python), each provisioning its test DB by executing the committed, TS-produced `canonical/schema.postgres.sql` (Postgres only — Derby dropped for the cross-port query corpus, ADR-0015). The **migration** scenarios are exercised by **TS only** (TS owns schema migrations). **The corpus now gates WRITES, not just reads (SP-H):** an `op: roundtrip` scenario type INSERTs through each port's runtime/ORM write codec (NOT raw SQL), reads the row back, and asserts the wire-normalized value. The `AllTypes` entity (`roundtrip-all-types.yaml`) carries one field of **every** persistable `field.*` subtype — string/int/long/double/float/decimal/boolean/date/time/timestamp(+tz)/currency/enum/uuid/object — plus an **array-of-VO** `field.object @isArray @storage:jsonb` column (`labels`, written as 2-element / empty-`[]` / single-element arrays across the three rows) — so every subtype write+read (incl. the array-of-value-object jsonb codec) round-trips through every port against Testcontainers PG. (`field.byte`/`field.short`/`field.class` were cut as non-functional registration-only stubs — the matrix tracks only genuinely-supported subtypes; see `fixtures/registry-conformance/README.md` → "Per-subtype write-round-trip matrix".) - API-contract: `fixtures/api-contract-conformance/`. TS / C# / Java / Kotlin / Python all green — each port runs **two lanes**: a hand-rolled reference server AND its **generated** API artifact booted over HTTP (the deployed controller/routes; TS+C# full-stack vs Testcontainers PG, Java/Kotlin/Python generated controller + in-memory repo behind the consumer seam). The generated fan-out found 10 real deployment bugs golden snapshots missed. diff --git a/README.md b/README.md index 2bca767ac..a53df79fa 100644 --- a/README.md +++ b/README.md @@ -177,7 +177,7 @@ metaobjects/ ├── CLAUDE.md # project instructions for Claude ├── spec/ # canonical metamodel docs, ADRs, roadmap ├── fixtures/ # 21 cross-language conformance corpora — the oracle -│ ├── conformance/ # metamodel (loader + serializer + navigation), 313 fixtures +│ ├── conformance/ # metamodel (loader + serializer + navigation), 314 fixtures │ ├── yaml-conformance/ # YAML authoring desugar │ ├── render-conformance/ # FR-004 byte-identical render oracle │ ├── verify-conformance/ # FR-004 template-drift gate diff --git a/docs/CONFORMANCE.md b/docs/CONFORMANCE.md index 28ecb21fd..4771e8b3f 100644 --- a/docs/CONFORMANCE.md +++ b/docs/CONFORMANCE.md @@ -25,7 +25,7 @@ regenerate with `ls -d fixtures//*/ | wc -l`. | Corpus | Fixtures | TS | Java | Kotlin | C# | Python | |---|---|---|---|---|---|---| -| [`fixtures/conformance/`](../fixtures/conformance/) (metamodel) | 313 | ✓ | ✓ | inherits via `metadata-ktx` | ✓ | ✓ | +| [`fixtures/conformance/`](../fixtures/conformance/) (metamodel) | 314 | ✓ | ✓ | inherits via `metadata-ktx` | ✓ | ✓ | | [`fixtures/yaml-conformance/`](../fixtures/yaml-conformance/) | 15 | 15 / 15 | 14 / 15 (1 ledgered: `yaml-quoted-leading-zero` — Java pipeline strips quotes off `"007"`) | inherits via Java | 14 / 15 (1 ledgered: `error-yaml-coerced-hex-in-string` — YamlDotNet doesn't coerce `0xFF`) | 15 / 15 | | [`fixtures/verify-conformance/`](../fixtures/verify-conformance/) | 31 | ✓ | ✓ | inherits via Java | ✓ | ✓ | | [`fixtures/verify-strict-conformance/`](../fixtures/verify-strict-conformance/) | 1 | ✓ | — | — | — | ✓ | @@ -118,7 +118,7 @@ unit-test runners (`bun test`, `dotnet test`, `pytest`, `mvn test`) pull Docker. ## Fixture-to-doc mapping -### `fixtures/conformance/` — metamodel loader + canonical serializer (313) +### `fixtures/conformance/` — metamodel loader + canonical serializer (314) | Fixture prefix | Feature doc | |---|---| @@ -238,7 +238,7 @@ grammar rather than four. ## Orphaned fixtures (tested but not yet documented) -The fixtures in the eight corpora mapped above (metamodel 313 + yaml 15 + verify 31 +The fixtures in the eight corpora mapped above (metamodel 314 + yaml 15 + verify 31 + render 15 + persistence 33 + api-contract 41 + source-resolution 25 + scope 10) each map to a feature doc. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider composition, agent context, docs emit) diff --git a/fixtures/conformance/xpkg-node-level-package-no-root-package/expected.json b/fixtures/conformance/xpkg-node-level-package-no-root-package/expected.json new file mode 100644 index 000000000..72f2dd9ae --- /dev/null +++ b/fixtures/conformance/xpkg-node-level-package-no-root-package/expected.json @@ -0,0 +1,70 @@ +{ + "metadata.root": { + "children": [ + { + "object.value": { + "name": "Address", + "package": "acme::common", + "children": [ + { + "field.string": { + "name": "city" + } + }, + { + "field.string": { + "name": "street" + } + } + ] + } + }, + { + "object.entity": { + "name": "Audited", + "package": "acme::common", + "abstract": true, + "children": [ + { + "field.timestamp": { + "name": "createdAt" + } + } + ] + } + }, + { + "object.entity": { + "name": "Customer", + "package": "acme::common", + "children": [ + { + "source.rdb": { + "@table": "customers" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "email", + "@maxLength": 120 + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/xpkg-node-level-package-no-root-package/input/meta.shared.json b/fixtures/conformance/xpkg-node-level-package-no-root-package/input/meta.shared.json new file mode 100644 index 000000000..72f2dd9ae --- /dev/null +++ b/fixtures/conformance/xpkg-node-level-package-no-root-package/input/meta.shared.json @@ -0,0 +1,70 @@ +{ + "metadata.root": { + "children": [ + { + "object.value": { + "name": "Address", + "package": "acme::common", + "children": [ + { + "field.string": { + "name": "city" + } + }, + { + "field.string": { + "name": "street" + } + } + ] + } + }, + { + "object.entity": { + "name": "Audited", + "package": "acme::common", + "abstract": true, + "children": [ + { + "field.timestamp": { + "name": "createdAt" + } + } + ] + } + }, + { + "object.entity": { + "name": "Customer", + "package": "acme::common", + "children": [ + { + "source.rdb": { + "@table": "customers" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "email", + "@maxLength": 120 + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ] + } +} diff --git a/server/python/src/metaobjects/naming.py b/server/python/src/metaobjects/naming.py index e11890bc8..a3028c3cd 100644 --- a/server/python/src/metaobjects/naming.py +++ b/server/python/src/metaobjects/naming.py @@ -108,6 +108,14 @@ def strip_package(name: str) -> str: return name.rsplit(PACKAGE_SEP, 1)[-1] +def package_of_resolution_key(fqn: str) -> str: + """The package half of a resolution key — the complement of :func:`strip_package` + (``a::b::C`` → ``a::b``; a root-level ``C`` → ``""``). Mirrors TS + ``packageOfResolutionKey``. + """ + return fqn.rpartition(PACKAGE_SEP)[0] + + def resolve_index_name(node: MetaData) -> str: """THE database name of an ``identity.secondary`` / ``index.lookup``. diff --git a/server/python/src/metaobjects/serializer_json.py b/server/python/src/metaobjects/serializer_json.py index e5847ec5b..93301fb0b 100644 --- a/server/python/src/metaobjects/serializer_json.py +++ b/server/python/src/metaobjects/serializer_json.py @@ -11,7 +11,8 @@ SOURCE_ATTR_TABLE, SOURCE_SUBTYPE_RDB, ) -from .shared.base_types import TYPE_SOURCE +from .naming import package_of_resolution_key +from .shared.base_types import SUBTYPE_ROOT, TYPE_METADATA, TYPE_SOURCE from .shared.separators import ATTR_PREFIX, FUSED_KEY_SEP from .shared.structural import ( KEY_ABSTRACT, @@ -42,6 +43,39 @@ def canonical_serialize_effective(node: MetaData) -> str: return _serialize(node, effective=True) +def serialize_shared_document(nodes: list[MetaData]) -> str: + """The FR-023 shared-model artifact form: one canonical-JSON ``metadata.root`` + document with NO root ``package``, whose top-level nodes each carry their own + explicit ``package`` (so the document re-loads to the same resolution keys in + every port). + + Each node is its :func:`canonical_serialize` form — raw own-layer, ``extends`` + preserved, attribute keys alphabetized, the FR-016 physical-name rewrite applied. + Top-level nodes are sorted by resolution key; each node's children keep their + authored order. Body key order: name, package, then the canonical rest. + Byte-identical to TS ``serializeSharedDocument``. + """ + children: list[dict[str, object]] = [] + for node in sorted(nodes, key=lambda n: n.resolution_key()): + pkg = package_of_resolution_key(node.resolution_key()) + if not pkg: + raise ValueError( + f"serialize_shared_document: {node.resolution_key()} has no package; " + f"a shared document carries only packaged nodes" + ) + fused = f"{node.type}{FUSED_KEY_SEP}{node.sub_type}" + body = _body(node, False) + # The rewrite edits source.rdb bodies IN PLACE, so ``body`` stays current. + _rewrite_source_rdb_physical_names({fused: body}) + # The node's own ``package`` (if it declared one) is replaced by the + # RESOLVED one: a node inheriting its file's root package declares none. + ordered: dict[str, object] = {KEY_NAME: body[KEY_NAME], KEY_PACKAGE: pkg} + ordered.update((k, v) for k, v in body.items() if k not in (KEY_NAME, KEY_PACKAGE)) + children.append({fused: ordered}) + doc = {f"{TYPE_METADATA}{FUSED_KEY_SEP}{SUBTYPE_ROOT}": {KEY_CHILDREN: children}} + return json.dumps(doc, indent=2, ensure_ascii=False) + "\n" + + def _serialize(node: MetaData, effective: bool) -> str: parsed = _to_canonical(node, effective) # FR-016 / ADR-0018 — rewrite legacy @table → kind-matching alias on diff --git a/server/python/tests/unit/test_serializer_shared_document.py b/server/python/tests/unit/test_serializer_shared_document.py new file mode 100644 index 000000000..2b9b561a7 --- /dev/null +++ b/server/python/tests/unit/test_serializer_shared_document.py @@ -0,0 +1,51 @@ +import json +from pathlib import Path + +import pytest + +from metaobjects import MetaDataLoader +from metaobjects.naming import package_of_resolution_key +from metaobjects.serializer_json import serialize_shared_document +from metaobjects.shared.base_types import TYPE_OBJECT + +# Address declares `city` before `street`: a shared document keeps each node's +# AUTHORED child order (own-layer form — only top-level nodes are sorted), and the +# pinned corpus artifact lists them in that order. +LIB = json.dumps({"metadata.root": {"package": "acme::common", "children": [ + {"object.entity": {"name": "Customer", "children": [ + {"source.rdb": {"@table": "customers"}}, {"field.long": {"name": "id"}}, + {"field.string": {"name": "email", "@maxLength": 120}}, + {"identity.primary": {"name": "pk", "@fields": ["id"]}}]}}, + {"object.entity": {"name": "Audited", "abstract": True, "children": [{"field.timestamp": {"name": "createdAt"}}]}}, + {"object.value": {"name": "Address", "children": [{"field.string": {"name": "city"}}, {"field.string": {"name": "street"}}]}}, +]}}) +# tests/unit/ → parents[4] is the repo root (unit, tests, python, server, root). +ARTIFACT = Path(__file__).resolve().parents[4] / "fixtures" / "dependency-conformance" / "artifacts" / "acme-common-v1.json" + + +def _objects(root): + return [c for c in root.own_children() if c.type == TYPE_OBJECT] # ADR-0039 sanctioned own: root-level scan + + +def test_package_of_resolution_key(): + assert package_of_resolution_key("acme::common::Customer") == "acme::common" + assert package_of_resolution_key("Customer") == "" + + +def test_shared_document_is_byte_identical_to_the_pinned_artifact(): + res = MetaDataLoader.from_string(LIB, "json") + assert not res.errors + assert serialize_shared_document(_objects(res.root)) == ARTIFACT.read_text(encoding="utf-8") + + +def test_shared_document_reloads_to_the_same_resolution_keys(): + res = MetaDataLoader.from_string(LIB, "json") + again = MetaDataLoader.from_string(serialize_shared_document(_objects(res.root)), "json") + assert not again.errors + assert sorted(o.resolution_key() for o in _objects(again.root)) == ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"] + + +def test_a_root_level_node_without_a_package_is_refused(): + res = MetaDataLoader.from_string(json.dumps({"metadata.root": {"children": [{"object.value": {"name": "Bare"}}]}}), "json") + with pytest.raises(ValueError, match="package"): + serialize_shared_document(_objects(res.root)) diff --git a/server/typescript/packages/metadata/src/index.ts b/server/typescript/packages/metadata/src/index.ts index e87bc0564..7a31350db 100644 --- a/server/typescript/packages/metadata/src/index.ts +++ b/server/typescript/packages/metadata/src/index.ts @@ -235,7 +235,7 @@ export type { ParseOptions, ParseResult } from "./parser-core.js"; export { parseJson } from "./parser-json.js"; // Serializer -export { serializeJson, canonicalSerialize, inferAttrSubType } from "./serializer-json.js"; +export { serializeJson, canonicalSerialize, serializeSharedDocument, inferAttrSubType } from "./serializer-json.js"; export type { SerializeOptions } from "./serializer-json.js"; // Super resolution helper (most resolution moved into parser; this is the lookup utility) @@ -290,7 +290,7 @@ export { resolveTableName, resolveColumnName, resolveTableSchema, resolveIndexName, primaryRdbSource, sourceAddressKey, buildNameMap, - stripPackage, + stripPackage, packageOfResolutionKey, } from "./naming.js"; export type { EntityNameMap, ColumnNamingStrategy } from "./naming.js"; diff --git a/server/typescript/packages/metadata/src/naming.ts b/server/typescript/packages/metadata/src/naming.ts index 82799e862..9a1672cdf 100644 --- a/server/typescript/packages/metadata/src/naming.ts +++ b/server/typescript/packages/metadata/src/naming.ts @@ -30,6 +30,17 @@ export function stripPackage(name: string | undefined): string { return idx === -1 ? name : name.slice(idx + PACKAGE_SEPARATOR.length); } +/** + * The package half of a resolution key — the complement of `stripPackage` + * (`"a::b::C"` → `"a::b"`; a root-level `"C"` → `""`). A resolution key is + * `package::name` for a packaged node and the bare name otherwise, so the text + * before the LAST separator is exactly the node's package. + */ +export function packageOfResolutionKey(fqn: string): string { + const idx = fqn.lastIndexOf(PACKAGE_SEPARATOR); + return idx === -1 ? "" : fqn.slice(0, idx); +} + export function toSnakeCase(s: string): string { return s .replace(/([A-Z]+)([A-Z][a-z])/g, "$1_$2") diff --git a/server/typescript/packages/metadata/src/serializer-json.ts b/server/typescript/packages/metadata/src/serializer-json.ts index c1ea866f7..a8ebae582 100644 --- a/server/typescript/packages/metadata/src/serializer-json.ts +++ b/server/typescript/packages/metadata/src/serializer-json.ts @@ -32,7 +32,8 @@ import { DEFAULT_SOURCE_KIND, PHYSICAL_NAME_ATTR_BY_KIND, } from "./persistence/source/source-constants.js"; -import { TYPE_SOURCE } from "./shared/base-types.js"; +import { TYPE_METADATA, TYPE_SOURCE, SUBTYPE_ROOT } from "./shared/base-types.js"; +import { packageOfResolutionKey } from "./naming.js"; const SOURCE_RDB_FUSED_KEY = `${TYPE_SOURCE}${TYPE_SUBTYPE_SEPARATOR}${SOURCE_SUBTYPE_RDB}`; @@ -301,3 +302,47 @@ function sortAttrValue(value: unknown): unknown { } return value; } + +// --------------------------------------------------------------------------- +// serializeSharedDocument — the FR-023 shared-model artifact form +// +// One canonical-JSON `metadata.root` document holding top-level nodes from any +// number of packages: NO root `package`, and every top-level node carries its own +// explicit `package` (a root-level child may name its package — ADR-0029's +// addressing model — so the document re-loads to the same resolution keys in +// every port). Each node is its canonicalSerialize form: raw own-layer, `extends` +// preserved (not flattened), attribute keys alphabetized, the FR-016 physical-name +// rewrite applied. Top-level nodes are sorted by resolution key; each node's +// children keep their authored order. Body key order: name, package, then the +// canonical rest. Byte-identical to Python's `serialize_shared_document`. +// --------------------------------------------------------------------------- + +export function serializeSharedDocument(nodes: readonly MetaData[]): string { + const sorted = [...nodes].sort((a, b) => { + const ka = a.resolutionKey(); + const kb = b.resolutionKey(); + return ka < kb ? -1 : ka > kb ? 1 : 0; + }); + const children = sorted.map((node) => { + const pkg = packageOfResolutionKey(node.resolutionKey()); + if (pkg === "") { + throw new Error( + `serializeSharedDocument: ${node.resolutionKey()} has no package; a shared document carries only packaged nodes`, + ); + } + const parsed = JSON.parse(canonicalSerialize(node)) as Record>; + const [fused, body] = Object.entries(parsed)[0]!; + // The node's own `package` (if it declared one) is replaced by the RESOLVED + // one: a node inheriting its file's root package declares none of its own. + const ordered: Record = { + [RESERVED_KEY_NAME]: body[RESERVED_KEY_NAME], + [RESERVED_KEY_PACKAGE]: pkg, + }; + for (const [key, value] of Object.entries(body)) { + if (key !== RESERVED_KEY_NAME && key !== RESERVED_KEY_PACKAGE) ordered[key] = value; + } + return { [fused]: ordered }; + }); + const doc = { [fusedKey(TYPE_METADATA, SUBTYPE_ROOT)]: { [RESERVED_KEY_CHILDREN]: children } }; + return JSON.stringify(doc, null, 2) + "\n"; +} diff --git a/server/typescript/packages/metadata/test/serializer-shared-document.test.ts b/server/typescript/packages/metadata/test/serializer-shared-document.test.ts new file mode 100644 index 000000000..6f78b1fbd --- /dev/null +++ b/server/typescript/packages/metadata/test/serializer-shared-document.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, test } from "bun:test"; +import { MetaDataLoader, serializeSharedDocument, packageOfResolutionKey } from "../src/index.js"; + +// Address declares `city` before `street`: a shared document keeps each node's +// AUTHORED child order (own-layer form — only top-level nodes are sorted), and the +// pinned corpus artifact below lists them in that order. +const LIB = JSON.stringify({ "metadata.root": { package: "acme::common", children: [ + { "object.entity": { name: "Customer", children: [ + { "source.rdb": { "@table": "customers" } }, { "field.long": { name: "id" } }, + { "field.string": { name: "email", "@maxLength": 120 } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } } ] } }, + { "object.entity": { name: "Audited", abstract: true, children: [ { "field.timestamp": { name: "createdAt" } } ] } }, + { "object.value": { name: "Address", children: [ { "field.string": { name: "city" } }, { "field.string": { name: "street" } } ] } }, +]}}); + +describe("serializeSharedDocument", () => { + test("packageOfResolutionKey", () => { + expect(packageOfResolutionKey("acme::common::Customer")).toBe("acme::common"); + expect(packageOfResolutionKey("Customer")).toBe(""); + }); + test("emits nodes sorted by resolution key with an explicit package and no root package", async () => { + const { root, errors } = await MetaDataLoader.fromString(LIB, "json"); + expect(errors).toEqual([]); + const out = serializeSharedDocument(root.objects()); + const doc = JSON.parse(out) as { "metadata.root": { package?: string; children: Record[] } }; + expect(doc["metadata.root"].package).toBeUndefined(); + const names = doc["metadata.root"].children.map((c) => Object.values(c)[0]!.name); + expect(names).toEqual(["Address", "Audited", "Customer"]); + for (const c of doc["metadata.root"].children) expect(Object.values(c)[0]!.package).toBe("acme::common"); + // body key order: name, package, then the rest + expect(Object.keys(Object.values(doc["metadata.root"].children[1]!)[0]!)).toEqual(["name", "package", "abstract", "children"]); + expect(out.endsWith("\n")).toBe(true); + expect(out).toBe(await Bun.file(`${import.meta.dir}/../../../../../fixtures/dependency-conformance/artifacts/acme-common-v1.json`).text()); + }); + test("a re-load of the document yields the same resolution keys", async () => { + const { root } = await MetaDataLoader.fromString(LIB, "json"); + const again = await MetaDataLoader.fromString(serializeSharedDocument(root.objects()), "json"); + expect(again.errors).toEqual([]); + expect(again.root.objects().map((o) => o.resolutionKey()).sort()).toEqual(["acme::common::Address", "acme::common::Audited", "acme::common::Customer"]); + }); + test("a root-level node (empty package) is refused", async () => { + const { root } = await MetaDataLoader.fromString(JSON.stringify({ "metadata.root": { children: [ { "object.value": { name: "Bare" } } ] } }), "json"); + expect(() => serializeSharedDocument(root.objects())).toThrow(/package/); + }); +}); From bb84e31e7d51e1c1a9753dd554eb9d69cb96608d Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 13:53:32 -0400 Subject: [PATCH 05/62] =?UTF-8?q?docs(plans):=20FR-023=20phase=201a=20?= =?UTF-8?q?=E2=80=94=20LIB=20Address=20member=20order=20matches=20the=20pi?= =?UTF-8?q?nned=20artifact?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index b1445b3d7..ea45ffeb2 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -283,7 +283,7 @@ const LIB = JSON.stringify({ "metadata.root": { package: "acme::common", childre { "field.string": { name: "email", "@maxLength": 120 } }, { "identity.primary": { name: "pk", "@fields": ["id"] } } ] } }, { "object.entity": { name: "Audited", abstract: true, children: [ { "field.timestamp": { name: "createdAt" } } ] } }, - { "object.value": { name: "Address", children: [ { "field.string": { name: "street" } }, { "field.string": { name: "city" } } ] } }, + { "object.value": { name: "Address", children: [ { "field.string": { name: "city" } }, { "field.string": { name: "street" } } ] } }, ]}}); describe("serializeSharedDocument", () => { @@ -354,7 +354,7 @@ LIB = json.dumps({"metadata.root": {"package": "acme::common", "children": [ {"field.string": {"name": "email", "@maxLength": 120}}, {"identity.primary": {"name": "pk", "@fields": ["id"]}}]}}, {"object.entity": {"name": "Audited", "abstract": True, "children": [{"field.timestamp": {"name": "createdAt"}}]}}, - {"object.value": {"name": "Address", "children": [{"field.string": {"name": "street"}}, {"field.string": {"name": "city"}}]}}, + {"object.value": {"name": "Address", "children": [{"field.string": {"name": "city"}}, {"field.string": {"name": "street"}}]}}, ]}}) ARTIFACT = Path(__file__).resolve().parents[3] / "fixtures" / "dependency-conformance" / "artifacts" / "acme-common-v1.json" From bfff6d4ed422ca1a30228754aa098b2751315873 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 14:00:46 -0400 Subject: [PATCH 06/62] docs: count the dependency-conformance corpus (22 shared corpora) fixtures/dependency-conformance/ is a shared corpus, so the stated corpora count moves 21 -> 22 wherever it is typed (docs/CONFORMANCE.md, AGENTS.md, README.md), and CONFORMANCE.md gains its row. The row states the corpus's actual state: one case so far, a TypeScript runner that is red by design until the resolver lands, no Python runner yet, and Java/Kotlin/C# deferred to Phase 2. The llms mirrors carry the metamodel fixture count 314, and the site payload is regenerated through `bun run site:payload` (counts only: fixtures 314, corpora 22). scripts/site/counts.test.ts and the payload --check are green again. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- AGENTS.md | 2 +- README.md | 2 +- docs/CONFORMANCE.md | 3 ++- docs/llms/llms-full.txt | 2 +- docs/llms/llms.txt | 2 +- examples/showcase/site-payload.json | 4 ++-- 6 files changed, 8 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index bd0dc5ec0..8a1c72f30 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,7 +66,7 @@ PyPI has had no product change since `0.25.0` — nothing is broken. - **Kotlin** — `codegen-kotlin` (KotlinPoet on JVM): entity + Exposed table + Spring controller + payload + relations + filter allowlist + validator + stored-proc + output-parser generators. `integration-tests-kotlin` runs the persistence-conformance corpus through Exposed against Testcontainers Postgres. **Cross-port conformance corpora** (every port runs the shared corpus): -- Metamodel: `fixtures/conformance/` (314 fixtures; 21 shared corpora in total — per-corpus counts + the corpus x port matrix live in `docs/CONFORMANCE.md`). TS / C# / Java / Python all green. +- Metamodel: `fixtures/conformance/` (314 fixtures; 22 shared corpora in total — per-corpus counts + the corpus x port matrix live in `docs/CONFORMANCE.md`). TS / C# / Java / Python all green. - Render: `fixtures/render-conformance/`. TS / C# / Java / Kotlin / Python byte-identical. - Persistence: `fixtures/persistence-conformance/`. **Query** scenarios run on every port (TS / C# / Java / Kotlin / Python), each provisioning its test DB by executing the committed, TS-produced `canonical/schema.postgres.sql` (Postgres only — Derby dropped for the cross-port query corpus, ADR-0015). The **migration** scenarios are exercised by **TS only** (TS owns schema migrations). **The corpus now gates WRITES, not just reads (SP-H):** an `op: roundtrip` scenario type INSERTs through each port's runtime/ORM write codec (NOT raw SQL), reads the row back, and asserts the wire-normalized value. The `AllTypes` entity (`roundtrip-all-types.yaml`) carries one field of **every** persistable `field.*` subtype — string/int/long/double/float/decimal/boolean/date/time/timestamp(+tz)/currency/enum/uuid/object — plus an **array-of-VO** `field.object @isArray @storage:jsonb` column (`labels`, written as 2-element / empty-`[]` / single-element arrays across the three rows) — so every subtype write+read (incl. the array-of-value-object jsonb codec) round-trips through every port against Testcontainers PG. (`field.byte`/`field.short`/`field.class` were cut as non-functional registration-only stubs — the matrix tracks only genuinely-supported subtypes; see `fixtures/registry-conformance/README.md` → "Per-subtype write-round-trip matrix".) - API-contract: `fixtures/api-contract-conformance/`. TS / C# / Java / Kotlin / Python all green — each port runs **two lanes**: a hand-rolled reference server AND its **generated** API artifact booted over HTTP (the deployed controller/routes; TS+C# full-stack vs Testcontainers PG, Java/Kotlin/Python generated controller + in-memory repo behind the consumer seam). The generated fan-out found 10 real deployment bugs golden snapshots missed. diff --git a/README.md b/README.md index a53df79fa..b6b593e99 100644 --- a/README.md +++ b/README.md @@ -176,7 +176,7 @@ metaobjects/ ├── README.md # you are here ├── CLAUDE.md # project instructions for Claude ├── spec/ # canonical metamodel docs, ADRs, roadmap -├── fixtures/ # 21 cross-language conformance corpora — the oracle +├── fixtures/ # 22 cross-language conformance corpora — the oracle │ ├── conformance/ # metamodel (loader + serializer + navigation), 314 fixtures │ ├── yaml-conformance/ # YAML authoring desugar │ ├── render-conformance/ # FR-004 byte-identical render oracle diff --git a/docs/CONFORMANCE.md b/docs/CONFORMANCE.md index 4771e8b3f..26c471e7a 100644 --- a/docs/CONFORMANCE.md +++ b/docs/CONFORMANCE.md @@ -1,6 +1,6 @@ # Conformance coverage -The MetaObjects standard ships **21 shared conformance corpora** under +The MetaObjects standard ships **22 shared conformance corpora** under [`fixtures/`](../fixtures/). Every port runs every corpus that is *applicable to it* and asserts the same expected behaviour against the same fixtures. **This page is the inverse index**: fixture → feature doc + per-port pass status, and it is the @@ -44,6 +44,7 @@ regenerate with `ls -d fixtures//*/ | wc -l`. | [`fixtures/provider-composition-conformance/`](../fixtures/provider-composition-conformance/) | 9 (5 error-shape + 4 compose-load) | ✓ | ✓ | — (JVM registry via Java) | ✓ | ✓ | | [`fixtures/source-resolution-conformance/`](../fixtures/source-resolution-conformance/) | 25 cases | ✓ (reference implementation) | ✓ | inherits via Java | ✓ | ✓ | | [`fixtures/scope-conformance/`](../fixtures/scope-conformance/) | 10 cases | ✓ (reference implementation) | — | — | — | — | +| [`fixtures/dependency-conformance/`](../fixtures/dependency-conformance/) | 1 case (FR-023 Phase 1a, in progress — cases land with the implementation) | runner in place (`sdk/test/dependency-conformance.test.ts`), red by design until the resolver lands | — (Phase 2) | — (Phase 2) | — (Phase 2) | — (no runner yet) | | [`fixtures/agent-context-conformance/`](../fixtures/agent-context-conformance/) | 4 | ✓ (the emitter is TS-owned) | — | — | — | — | | [`fixtures/metamodel-docs/`](../fixtures/metamodel-docs/) | 1 | ✓ (docs emit is TS-owned) | — | — | — | — | diff --git a/docs/llms/llms-full.txt b/docs/llms/llms-full.txt index fa0e19118..3697ddf78 100644 --- a/docs/llms/llms-full.txt +++ b/docs/llms/llms-full.txt @@ -112,7 +112,7 @@ What MetaObjects does not do is write the assertion. A generated stub is a place | C# | NuGet `1.0.1` (.NET tool) | Loader + canonical serializer + EF Core + ASP.NET codegen + render/verify. `dotnet meta` tool. | | Python | PyPI `1.0.1` | Loader + serializer + render + verify + codegen + `ObjectManager` runtime. Fully green across all corpora. | -Conformance fixtures live at [`fixtures/`](https://github.com/metaobjectsdev/metaobjects/tree/main/fixtures). Every port runs the shared corpus byte-identically: metamodel (`conformance/`, 313 fixtures), render, persistence (Testcontainers Postgres, with an `op: roundtrip` gate so every `field.*` subtype write+read round-trips on every port), api-contract (41 scenarios — 26 core plus TPH / M:N / jsonb / write-through — two lanes: a reference server AND each port's generated API booted over HTTP), registry (byte-matched metamodel-vocabulary manifest, live + green in all five ports), and yaml/verify. +Conformance fixtures live at [`fixtures/`](https://github.com/metaobjectsdev/metaobjects/tree/main/fixtures). Every port runs the shared corpus byte-identically: metamodel (`conformance/`, 314 fixtures), render, persistence (Testcontainers Postgres, with an `op: roundtrip` gate so every `field.*` subtype write+read round-trips on every port), api-contract (41 scenarios — 26 core plus TPH / M:N / jsonb / write-through — two lanes: a reference server AND each port's generated API booted over HTTP), registry (byte-matched metamodel-vocabulary manifest, live + green in all five ports), and yaml/verify. --- diff --git a/docs/llms/llms.txt b/docs/llms/llms.txt index 728a4837e..460dee96c 100644 --- a/docs/llms/llms.txt +++ b/docs/llms/llms.txt @@ -31,7 +31,7 @@ This `llms.txt` is the short index; the deep, version-matched how-to is the scaf ## Spec and standard - [Specification (canonical, target-agnostic)](https://github.com/metaobjectsdev/metaobjects/tree/main/spec): the normative metadata schema and semantics that every implementation must conform to. -- [Conformance fixtures](https://github.com/metaobjectsdev/metaobjects/tree/main/fixtures): cross-port corpora — `conformance/` (metamodel, 313 fixtures), `render-conformance/`, `persistence-conformance/` (against Testcontainers Postgres, including an `op: roundtrip` gate so every `field.*` subtype write+read round-trips on every port), `api-contract-conformance/` (41 scenarios — 26 core REST contract + filter operators, plus TPH / M:N / jsonb / write-through — run in two lanes: a reference server AND each port's generated API booted over HTTP), `registry-conformance/` (byte-matched metamodel-vocabulary manifest, live + green in all five ports), and `yaml-conformance/` / `verify-conformance/`. Every port runs the shared corpus byte-identically. +- [Conformance fixtures](https://github.com/metaobjectsdev/metaobjects/tree/main/fixtures): cross-port corpora — `conformance/` (metamodel, 314 fixtures), `render-conformance/`, `persistence-conformance/` (against Testcontainers Postgres, including an `op: roundtrip` gate so every `field.*` subtype write+read round-trips on every port), `api-contract-conformance/` (41 scenarios — 26 core REST contract + filter operators, plus TPH / M:N / jsonb / write-through — run in two lanes: a reference server AND each port's generated API booted over HTTP), `registry-conformance/` (byte-matched metamodel-vocabulary manifest, live + green in all five ports), and `yaml-conformance/` / `verify-conformance/`. Every port runs the shared corpus byte-identically. - [Roadmap](https://github.com/metaobjectsdev/metaobjects/blob/main/spec/roadmap.md): current + planned work across all implementations. ## The five pillars diff --git a/examples/showcase/site-payload.json b/examples/showcase/site-payload.json index eacf9815b..6623f91c1 100644 --- a/examples/showcase/site-payload.json +++ b/examples/showcase/site-payload.json @@ -7,8 +7,8 @@ "metamodel": "1.0" }, "counts": { - "fixtures": 313, - "corpora": 21, + "fixtures": 314, + "corpora": 22, "baseTypes": 14 }, "snippets": { From 12df4ab272062b94dfa78be5a8df0e4a308f2f23 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 14:06:31 -0400 Subject: [PATCH 07/62] feat(loader): FileSource accepts an explicit source id (FR-023 provenance) new FileSource(path, { id? }) (TS) and FileSource(path, format=None, id=None) (Python) accept an explicit source id, defaulting to basename(path) as before. The parser stamps source.files with whatever id a node's FileSource carries (ADR-0009), so a dependency artifact loaded later with id: "dep:/" will record where it really came from. Default behavior for every existing caller (single positional path argument) is unchanged. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../metaobjects/loader/sources/file_source.py | 12 ++++++++++-- .../python/tests/unit/test_file_source_id.py | 17 +++++++++++++++++ .../src/loader/sources/file-source.ts | 12 ++++++++++-- .../metadata/test/file-source-id.test.ts | 19 +++++++++++++++++++ 4 files changed, 56 insertions(+), 4 deletions(-) create mode 100644 server/python/tests/unit/test_file_source_id.py create mode 100644 server/typescript/packages/metadata/test/file-source-id.test.ts diff --git a/server/python/src/metaobjects/loader/sources/file_source.py b/server/python/src/metaobjects/loader/sources/file_source.py index f3cebaa78..0b8b2e973 100644 --- a/server/python/src/metaobjects/loader/sources/file_source.py +++ b/server/python/src/metaobjects/loader/sources/file_source.py @@ -21,9 +21,17 @@ def _infer_format(path: Path) -> MetaDataFormat: class FileSource(MetaDataSource): """A single on-disk file, decoded eagerly via ``utf-8-sig``.""" - def __init__(self, path: Path | str, format: MetaDataFormat | None = None) -> None: + def __init__( + self, + path: Path | str, + format: MetaDataFormat | None = None, + id: str | None = None, + ) -> None: self._path = Path(path) self._format = format if format is not None else _infer_format(self._path) + # Explicit id (FR-023: dependency snapshots load with `dep:/`) + # overrides the default basename. + self._id = id @property def path(self) -> Path: @@ -31,7 +39,7 @@ def path(self) -> Path: @property def id(self) -> str: - return self._path.name + return self._id or self._path.name @property def format(self) -> MetaDataFormat: diff --git a/server/python/tests/unit/test_file_source_id.py b/server/python/tests/unit/test_file_source_id.py new file mode 100644 index 000000000..b4c23a912 --- /dev/null +++ b/server/python/tests/unit/test_file_source_id.py @@ -0,0 +1,17 @@ +import json + +from metaobjects import MetaDataLoader +from metaobjects.loader.sources.file_source import FileSource + + +def test_file_source_id_defaults_to_basename_and_accepts_override(tmp_path): + p = tmp_path / "meta.a.json" + p.write_text(json.dumps({"metadata.root": {"package": "p", "children": [{"object.value": {"name": "V"}}]}})) + assert FileSource(p).id == "meta.a.json" + src = FileSource(p, id="dep:acme-common/acme-common.metaobjects.json") + assert src.id == "dep:acme-common/acme-common.metaobjects.json" + res = MetaDataLoader().load([src]) + assert not res.errors + node = next(c for c in res.root.own_children()) # ADR-0039 sanctioned own: root-level scan + # source.files is a tuple (existing Python convention — see test_source_on_node.py). + assert node.source.files == ("dep:acme-common/acme-common.metaobjects.json",) diff --git a/server/typescript/packages/metadata/src/loader/sources/file-source.ts b/server/typescript/packages/metadata/src/loader/sources/file-source.ts index 93977fadc..db9b6e71b 100644 --- a/server/typescript/packages/metadata/src/loader/sources/file-source.ts +++ b/server/typescript/packages/metadata/src/loader/sources/file-source.ts @@ -27,16 +27,24 @@ async function getReadText(): Promise<(path: string) => Promise> { return _readText; } +/** Options for {@link FileSource}. */ +export interface FileSourceOptions { + /** Explicit source id (e.g. `dep:/` for a dependency snapshot). + * Defaults to `basename(path)` when omitted. */ + id?: string; +} + /** A metadata source backed by a file on disk. */ export class FileSource implements MetaDataSource { readonly id: string; readonly format: MetaDataFormat; private readonly _path: string; - constructor(path: string) { + constructor(path: string, opts?: FileSourceOptions) { this._path = path; // basename() for readable error messages; cross-platform (handles both / and \). Full path retained for read(). - this.id = basename(path); + // An explicit id (FR-023: dependency snapshots load with `dep:/`) overrides the default. + this.id = opts?.id ?? basename(path); this.format = inferFormat(path); } diff --git a/server/typescript/packages/metadata/test/file-source-id.test.ts b/server/typescript/packages/metadata/test/file-source-id.test.ts new file mode 100644 index 000000000..698fda92a --- /dev/null +++ b/server/typescript/packages/metadata/test/file-source-id.test.ts @@ -0,0 +1,19 @@ +import { describe, expect, test } from "bun:test"; +import { mkdtemp, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { FileSource } from "../src/loader/sources/file-source.js"; +import { MetaDataLoader } from "../src/index.js"; + +describe("FileSource id", () => { + test("defaults to the basename and accepts an explicit id that reaches node provenance", async () => { + const dir = await mkdtemp(join(tmpdir(), "fs-id-")); + const p = join(dir, "meta.a.json"); + await writeFile(p, JSON.stringify({ "metadata.root": { package: "p", children: [ { "object.value": { name: "V" } } ] } })); + expect(new FileSource(p).id).toBe("meta.a.json"); + const src = new FileSource(p, { id: "dep:acme-common/acme-common.metaobjects.json" }); + expect(src.id).toBe("dep:acme-common/acme-common.metaobjects.json"); + const { root, errors } = await new MetaDataLoader().load([src]); + expect(errors).toEqual([]); + const v = root.objects()[0]!; + expect("files" in v.source ? v.source.files : []).toEqual(["dep:acme-common/acme-common.metaobjects.json"]); + }); +}); From 72ea87661ce00fb356a4b072fc1fd9aaddaa03bf Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 14:10:47 -0400 Subject: [PATCH 08/62] fix(loader): an explicit empty FileSource id is honoured in Python as in TS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Python's id property used `self._id or self._path.name`, so an explicit `id=""` fell through to the basename default — diverging from TS, where `??` (nullish coalescing) already honours `id: ""`. Switched to an explicit None-check (`self._id if self._id is not None else self._path.name`) and pinned the empty-string case in both ports' tests. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../python/src/metaobjects/loader/sources/file_source.py | 2 +- server/python/tests/unit/test_file_source_id.py | 6 ++++++ .../packages/metadata/test/file-source-id.test.ts | 7 +++++++ 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/server/python/src/metaobjects/loader/sources/file_source.py b/server/python/src/metaobjects/loader/sources/file_source.py index 0b8b2e973..7f4bc83c1 100644 --- a/server/python/src/metaobjects/loader/sources/file_source.py +++ b/server/python/src/metaobjects/loader/sources/file_source.py @@ -39,7 +39,7 @@ def path(self) -> Path: @property def id(self) -> str: - return self._id or self._path.name + return self._id if self._id is not None else self._path.name @property def format(self) -> MetaDataFormat: diff --git a/server/python/tests/unit/test_file_source_id.py b/server/python/tests/unit/test_file_source_id.py index b4c23a912..4ba805be5 100644 --- a/server/python/tests/unit/test_file_source_id.py +++ b/server/python/tests/unit/test_file_source_id.py @@ -15,3 +15,9 @@ def test_file_source_id_defaults_to_basename_and_accepts_override(tmp_path): node = next(c for c in res.root.own_children()) # ADR-0039 sanctioned own: root-level scan # source.files is a tuple (existing Python convention — see test_source_on_node.py). assert node.source.files == ("dep:acme-common/acme-common.metaobjects.json",) + + +def test_file_source_empty_string_id_is_honoured_not_treated_as_absent(tmp_path): + p = tmp_path / "meta.a.json" + p.write_text(json.dumps({"metadata.root": {"package": "p", "children": []}})) + assert FileSource(p, id="").id == "" diff --git a/server/typescript/packages/metadata/test/file-source-id.test.ts b/server/typescript/packages/metadata/test/file-source-id.test.ts index 698fda92a..ef7bb8e36 100644 --- a/server/typescript/packages/metadata/test/file-source-id.test.ts +++ b/server/typescript/packages/metadata/test/file-source-id.test.ts @@ -16,4 +16,11 @@ describe("FileSource id", () => { const v = root.objects()[0]!; expect("files" in v.source ? v.source.files : []).toEqual(["dep:acme-common/acme-common.metaobjects.json"]); }); + + test("an explicit empty-string id is honoured, not treated as absent", async () => { + const dir = await mkdtemp(join(tmpdir(), "fs-id-empty-")); + const p = join(dir, "meta.a.json"); + await writeFile(p, JSON.stringify({ "metadata.root": { package: "p", children: [] } })); + expect(new FileSource(p, { id: "" }).id).toBe(""); + }); }); From d20d8a32745cdb6ab89df653c1aae8b5b92c0934 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 14:16:58 -0400 Subject: [PATCH 09/62] refactor(metadata): scope-pattern grammar lives in metadata; Python port (FR-023) Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- server/python/src/metaobjects/scope.py | 95 +++++++++++++++++ .../conformance/test_scope_conformance.py | 59 +++++++++++ .../typescript/packages/metadata/src/index.ts | 6 ++ .../typescript/packages/metadata/src/scope.ts | 97 +++++++++++++++++ .../metadata/test/scope-conformance.test.ts | 29 +++++ server/typescript/packages/sdk/src/scope.ts | 100 ++---------------- 6 files changed, 293 insertions(+), 93 deletions(-) create mode 100644 server/python/src/metaobjects/scope.py create mode 100644 server/python/tests/conformance/test_scope_conformance.py create mode 100644 server/typescript/packages/metadata/src/scope.ts create mode 100644 server/typescript/packages/metadata/test/scope-conformance.test.ts diff --git a/server/python/src/metaobjects/scope.py b/server/python/src/metaobjects/scope.py new file mode 100644 index 000000000..5eb0bbec9 --- /dev/null +++ b/server/python/src/metaobjects/scope.py @@ -0,0 +1,95 @@ +"""FR-023 §4.3 — the scope-pattern grammar. + +A line-for-line port of the TypeScript scope engine +(``server/typescript/packages/metadata/src/scope.ts``): a pure, no-I/O module +deciding whether a fully-qualified node name falls inside a consumer's +declared ``include``/``exclude`` scope. A cross-language conformance corpus +(``fixtures/scope-conformance/cases.json``) pins its semantics exactly, so +pattern behavior must match TS (and the other ports) byte-for-byte. +""" +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Optional, Sequence + +from .errors import ErrorCode, ParseError +from .shared.separators import PACKAGE_SEP + +# One package segment: any run of characters containing no separator char. +_SEGMENT = "[^:]+" +# One or more segments, separator-joined — the "**" expansion. +_SEGMENTS = f"{_SEGMENT}(?:{PACKAGE_SEP}{_SEGMENT})*" + +# Regex metacharacters to escape for literal matching. Mirrors TS +# `escapeLiteral`'s `/[.*+?^${}()|[\]\\]/g` character class exactly. +_METACHAR_RE = re.compile(r"[.*+?^${}()|\[\]\\]") + + +def _escape_literal(text: str) -> str: + return _METACHAR_RE.sub(lambda m: "\\" + m.group(0), text) + + +def _compile_segment(segment: str, pattern: str) -> str: + """Compile one segment. ``**`` spans segments; ``*`` never crosses a separator.""" + if len(segment) == 0: + raise ParseError( + f'empty segment in scope pattern "{pattern}"', + ErrorCode.ERR_SCOPE_PATTERN_INVALID, + ) + # A segment surviving the split on the two-character PACKAGE_SEP ("::") + # can still contain a lone ":" when the pattern has an odd colon run — + # e.g. "acme:::Order".split("::") => ["acme", ":Order"]. _SEGMENT + # ([^:]+) already excludes ":" from a well-formed segment, so a leftover + # ":" here means the separator was malformed, not that ":" is meant + # literally. Left unchecked, _escape_literal treats it as a literal + # character and compiles a regex requiring three colons in a row — + # which no legal "::"-joined fully-qualified name can ever contain, so + # the pattern would silently match nothing instead of failing loud. + if ":" in segment: + raise ParseError( + f'scope pattern "{pattern}" has a malformed separator (an odd run ' + 'of ":") — segments are joined by "::", never a single ":"', + ErrorCode.ERR_SCOPE_PATTERN_INVALID, + ) + if segment == "**": + return f"(?:{_SEGMENTS})" + # `*` inside a segment matches any characters except the separator char. + return "[^:]*".join(_escape_literal(part) for part in segment.split("*")) + + +def compile_pattern(pattern: str) -> "re.Pattern[str]": + if len(pattern) == 0: + raise ParseError( + "scope pattern must not be empty", + ErrorCode.ERR_SCOPE_PATTERN_INVALID, + ) + body = PACKAGE_SEP.join( + _compile_segment(segment, pattern) for segment in pattern.split(PACKAGE_SEP) + ) + return re.compile(f"^{body}$") + + +@dataclass(frozen=True) +class CompiledScope: + include: tuple["re.Pattern[str]", ...] + exclude: tuple["re.Pattern[str]", ...] + + +def compile_scope( + include: Optional[Sequence[str]] = None, + exclude: Optional[Sequence[str]] = None, +) -> CompiledScope: + """Absent or empty ``include`` means "everything"; ``exclude`` is applied after.""" + return CompiledScope( + include=tuple(compile_pattern(p) for p in (include or ())), + exclude=tuple(compile_pattern(p) for p in (exclude or ())), + ) + + +def matches_scope(fqn: str, compiled: CompiledScope) -> bool: + """True when ``fqn`` is inside the scope. An empty ``include`` means everything.""" + included = len(compiled.include) == 0 or any(p.match(fqn) for p in compiled.include) + if not included: + return False + return not any(p.match(fqn) for p in compiled.exclude) diff --git a/server/python/tests/conformance/test_scope_conformance.py b/server/python/tests/conformance/test_scope_conformance.py new file mode 100644 index 000000000..0ee468791 --- /dev/null +++ b/server/python/tests/conformance/test_scope_conformance.py @@ -0,0 +1,59 @@ +"""Runs the shared scope-pattern corpus against this port. + +Reads `fixtures/scope-conformance/cases.json` — the single committed source of +truth, shared with the TS runner (`metadata/test/scope-conformance.test.ts`, +re-run unchanged through sdk's re-export at `sdk/test/scope-conformance.test.ts`). +There is no per-port fixture. +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from metaobjects.errors import ErrorCode, ParseError +from metaobjects.scope import compile_scope, matches_scope + +_CORPUS = ( + Path(__file__).resolve().parents[4] + / "fixtures" + / "scope-conformance" + / "cases.json" +) + +_CASES = json.loads(_CORPUS.read_text())["cases"] + + +def test_corpus_is_non_empty() -> None: + """A silent zero-case run is a failed gate, not a pass. + + `@pytest.mark.parametrize` over an empty list simply collects zero tests — + pytest reports that as a SKIP, not a failure, so a corpus that quietly lost + its cases (e.g. a bad path, a JSON-parsing bug) would report green here + with nothing actually checked. Mirrors the TS runner's identically-named + guard (`scope-conformance.test.ts`). + """ + assert len(_CASES) > 0 + + +@pytest.mark.parametrize("case", _CASES, ids=[c["name"] for c in _CASES]) +def test_scope_conformance_case(case: dict) -> None: + scope = case["scope"] + include = scope.get("include") + exclude = scope.get("exclude") + + expect_error = case.get("expectError") + if expect_error is not None: + with pytest.raises(ParseError) as excinfo: + compile_scope(include, exclude) + assert excinfo.value.code == ErrorCode.ERR_SCOPE_PATTERN_INVALID + return + + compiled = compile_scope(include, exclude) + for entry in case["expect"]: + fqn = entry["fqn"] + assert matches_scope(fqn, compiled) == entry["matches"], ( + f"case {case['name']!r}: matches_scope({fqn!r}, ...) expected " + f"{entry['matches']!r}" + ) diff --git a/server/typescript/packages/metadata/src/index.ts b/server/typescript/packages/metadata/src/index.ts index 7a31350db..00c1ed70a 100644 --- a/server/typescript/packages/metadata/src/index.ts +++ b/server/typescript/packages/metadata/src/index.ts @@ -278,6 +278,12 @@ export type { } from "./source.js"; export { codeSource } from "./source.js"; +// FR-023 §4.3 — scope-pattern grammar (moved from sdk; sdk re-exports these +// for compatibility, and codegen-ts's publisher generator depends on this +// package directly rather than on sdk). +export { compileScope, matchesScope } from "./scope.js"; +export type { Scope, CompiledScope } from "./scope.js"; + // Attribute-schema validation pass (Phase A3) export { validateAttrSchema } from "./attr-schema-validate.js"; export type { AttrSchemaValidationResult } from "./attr-schema-validate.js"; diff --git a/server/typescript/packages/metadata/src/scope.ts b/server/typescript/packages/metadata/src/scope.ts new file mode 100644 index 000000000..3a494c57b --- /dev/null +++ b/server/typescript/packages/metadata/src/scope.ts @@ -0,0 +1,97 @@ +// server/typescript/packages/metadata/src/scope.ts +// +// FR-023 §4.3 — the scope-pattern grammar. Moved here from +// `@metaobjectsdev/sdk` so `codegen-ts`'s publisher generator +// (`sharedModelFile()`) can select its exports with the same patterns +// without taking a dependency on sdk. `@metaobjectsdev/sdk` re-exports +// `compileScope` / `matchesScope` / `Scope` / `CompiledScope` from here +// unchanged, so existing importers of the scope API from sdk keep working. +// +// A pure, no-I/O module deciding whether a fully-qualified node name falls +// inside a consumer's declared `include`/`exclude` scope. Source resolution +// and discovery (later phase-1 tasks) build on this; a cross-language +// conformance corpus pins its semantics, so exact pattern behavior matters. +// +// Uses no `node:` imports — stays browser-safe like the rest of the root +// entry (see `test/browser-safety.test.ts`). +import { PACKAGE_SEPARATOR } from "./shared/structural.js"; +import { ParseError } from "./errors.js"; +import { codeSource } from "./source.js"; + +/** A consumer-side output filter over fully-qualified node names. */ +export interface Scope { + /** Absent or empty means "everything". */ + readonly include?: readonly string[]; + /** Applied after `include`. */ + readonly exclude?: readonly string[]; +} + +export interface CompiledScope { + readonly include: readonly RegExp[]; + readonly exclude: readonly RegExp[]; +} + +/** One package segment: any run of characters containing no separator char. */ +const SEGMENT = "[^:]+"; +/** One or more segments, separator-joined — the `**` expansion. */ +const SEGMENTS = `${SEGMENT}(?:${PACKAGE_SEPARATOR}${SEGMENT})*`; + +function escapeLiteral(text: string): string { + return text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} + +/** Compile one segment. `**` spans segments; `*` never crosses a separator. */ +function compileSegment(segment: string, pattern: string): string { + if (segment.length === 0) { + throw new ParseError(`empty segment in scope pattern "${pattern}"`, { + code: "ERR_SCOPE_PATTERN_INVALID", + source: codeSource("compileSegment"), + }); + } + // A segment surviving the split on the two-character PACKAGE_SEPARATOR + // ("::") can still contain a lone ":" when the pattern has an odd colon + // run — e.g. "acme:::Order".split("::") => ["acme", ":Order"]. SEGMENT + // ([^:]+) already excludes ":" from a well-formed segment, so a leftover + // ":" here means the separator was malformed, not that ":" is meant + // literally. Left unchecked, escapeLiteral treats it as a literal + // character and compiles a regex requiring three colons in a row — which + // no legal "::"-joined fully-qualified name can ever contain, so the + // pattern silently matches nothing instead of failing loud. + if (segment.includes(":")) { + throw new ParseError( + `scope pattern "${pattern}" has a malformed separator (an odd run of ":") — segments are joined by "::", never a single ":"`, + { code: "ERR_SCOPE_PATTERN_INVALID", source: codeSource("compileSegment") }, + ); + } + if (segment === "**") return `(?:${SEGMENTS})`; + // `*` inside a segment matches any characters except the separator char. + return segment.split("*").map(escapeLiteral).join("[^:]*"); +} + +export function compilePattern(pattern: string): RegExp { + if (pattern.length === 0) { + throw new ParseError(`scope pattern must not be empty`, { + code: "ERR_SCOPE_PATTERN_INVALID", + source: codeSource("compilePattern"), + }); + } + const body = pattern + .split(PACKAGE_SEPARATOR) + .map((segment) => compileSegment(segment, pattern)) + .join(PACKAGE_SEPARATOR); + return new RegExp(`^${body}$`); +} + +export function compileScope(scope: Scope): CompiledScope { + return { + include: (scope.include ?? []).map(compilePattern), + exclude: (scope.exclude ?? []).map(compilePattern), + }; +} + +/** True when `fqn` is inside the scope. An empty `include` means everything. */ +export function matchesScope(fqn: string, compiled: CompiledScope): boolean { + const included = compiled.include.length === 0 || compiled.include.some((re) => re.test(fqn)); + if (!included) return false; + return !compiled.exclude.some((re) => re.test(fqn)); +} diff --git a/server/typescript/packages/metadata/test/scope-conformance.test.ts b/server/typescript/packages/metadata/test/scope-conformance.test.ts new file mode 100644 index 000000000..656da8f8c --- /dev/null +++ b/server/typescript/packages/metadata/test/scope-conformance.test.ts @@ -0,0 +1,29 @@ +// server/typescript/packages/metadata/test/scope-conformance.test.ts +import { describe, test, expect } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { compileScope, matchesScope, type Scope } from "../src/scope.js"; + +interface Case { + name: string; + scope: Scope; + expect: Array<{ fqn: string; matches: boolean }>; +} + +const CORPUS = join(import.meta.dir, "../../../../../fixtures/scope-conformance/cases.json"); +const cases = (JSON.parse(readFileSync(CORPUS, "utf8")) as { cases: Case[] }).cases; + +describe("scope-conformance corpus", () => { + test("corpus is non-empty (a silent zero-case run is a failed gate)", () => { + expect(cases.length).toBeGreaterThan(0); + }); + for (const c of cases) { + test(c.name, () => { + const compiled = compileScope(c.scope); + for (const e of c.expect) { + expect({ fqn: e.fqn, matches: matchesScope(e.fqn, compiled) }) + .toEqual({ fqn: e.fqn, matches: e.matches }); + } + }); + } +}); diff --git a/server/typescript/packages/sdk/src/scope.ts b/server/typescript/packages/sdk/src/scope.ts index 26d7da398..819dc0ea2 100644 --- a/server/typescript/packages/sdk/src/scope.ts +++ b/server/typescript/packages/sdk/src/scope.ts @@ -1,95 +1,9 @@ // server/typescript/packages/sdk/src/scope.ts // -// Phase-1 metadata-source-resolution — the scope pattern engine. -// -// A pure, no-I/O module deciding whether a fully-qualified node name falls -// inside a consumer's declared `include`/`exclude` scope. Source resolution -// and discovery (later phase-1 tasks) build on this; a cross-language -// conformance corpus pins its semantics, so exact pattern behavior matters. - -// NOTE: PACKAGE_SEPARATOR is NOT re-exported from the browser-safe -// `@metaobjectsdev/metadata/constants` barrel (that barrel only re-exports -// the per-concern `*-constants.ts` modules; `PACKAGE_SEPARATOR` lives in -// `shared/structural.ts`, exported from the package root). This package -// (`@metaobjectsdev/sdk`) is server-side, not a `client/web/**` browser -// package, so importing metamodel values from the root — the same thing -// `memory.ts` and `forge-types.ts` in this package already do — is correct. -import { PACKAGE_SEPARATOR, ParseError, codeSource } from "@metaobjectsdev/metadata"; - -/** A consumer-side output filter over fully-qualified node names. */ -export interface Scope { - /** Absent or empty means "everything". */ - readonly include?: readonly string[]; - /** Applied after `include`. */ - readonly exclude?: readonly string[]; -} - -export interface CompiledScope { - readonly include: readonly RegExp[]; - readonly exclude: readonly RegExp[]; -} - -/** One package segment: any run of characters containing no separator char. */ -const SEGMENT = "[^:]+"; -/** One or more segments, separator-joined — the `**` expansion. */ -const SEGMENTS = `${SEGMENT}(?:${PACKAGE_SEPARATOR}${SEGMENT})*`; - -function escapeLiteral(text: string): string { - return text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); -} - -/** Compile one segment. `**` spans segments; `*` never crosses a separator. */ -function compileSegment(segment: string, pattern: string): string { - if (segment.length === 0) { - throw new ParseError(`empty segment in scope pattern "${pattern}"`, { - code: "ERR_SCOPE_PATTERN_INVALID", - source: codeSource("compileSegment"), - }); - } - // A segment surviving the split on the two-character PACKAGE_SEPARATOR - // ("::") can still contain a lone ":" when the pattern has an odd colon - // run — e.g. "acme:::Order".split("::") => ["acme", ":Order"]. SEGMENT - // ([^:]+) already excludes ":" from a well-formed segment, so a leftover - // ":" here means the separator was malformed, not that ":" is meant - // literally. Left unchecked, escapeLiteral treats it as a literal - // character and compiles a regex requiring three colons in a row — which - // no legal "::"-joined fully-qualified name can ever contain, so the - // pattern silently matches nothing instead of failing loud. - if (segment.includes(":")) { - throw new ParseError( - `scope pattern "${pattern}" has a malformed separator (an odd run of ":") — segments are joined by "::", never a single ":"`, - { code: "ERR_SCOPE_PATTERN_INVALID", source: codeSource("compileSegment") }, - ); - } - if (segment === "**") return `(?:${SEGMENTS})`; - // `*` inside a segment matches any characters except the separator char. - return segment.split("*").map(escapeLiteral).join("[^:]*"); -} - -export function compilePattern(pattern: string): RegExp { - if (pattern.length === 0) { - throw new ParseError(`scope pattern must not be empty`, { - code: "ERR_SCOPE_PATTERN_INVALID", - source: codeSource("compilePattern"), - }); - } - const body = pattern - .split(PACKAGE_SEPARATOR) - .map((segment) => compileSegment(segment, pattern)) - .join(PACKAGE_SEPARATOR); - return new RegExp(`^${body}$`); -} - -export function compileScope(scope: Scope): CompiledScope { - return { - include: (scope.include ?? []).map(compilePattern), - exclude: (scope.exclude ?? []).map(compilePattern), - }; -} - -/** True when `fqn` is inside the scope. An empty `include` means everything. */ -export function matchesScope(fqn: string, compiled: CompiledScope): boolean { - const included = compiled.include.length === 0 || compiled.include.some((re) => re.test(fqn)); - if (!included) return false; - return !compiled.exclude.some((re) => re.test(fqn)); -} +// FR-023 §4.3 — the scope-pattern grammar moved to `@metaobjectsdev/metadata` +// (pure string code, browser-safe) so `codegen-ts` can use it without +// depending on sdk. Re-exported here unchanged so existing importers of the +// scope API from `@metaobjectsdev/sdk` — including this package's own +// `collection.ts` and `index.ts` — keep working without modification. +export { compileScope, matchesScope } from "@metaobjectsdev/metadata"; +export type { Scope, CompiledScope } from "@metaobjectsdev/metadata"; From 1918a02a53572a904fa7f10b10cd6c954423506d Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 14:26:29 -0400 Subject: [PATCH 10/62] =?UTF-8?q?feat(config):=20`dependencies`=20in=20.me?= =?UTF-8?q?taobjects/config.json=20=E2=80=94=20sdk=20schema=20and=20Python?= =?UTF-8?q?=20neutral=20reader=20(FR-023)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 5 of FR-023 phase 1a. Adds the `dependencies` key (DESIGN §3.1): a declared name, exactly one transport (`path` | `npm` | `python`, the latter two with an optional `dir`), and a `mode` (`reference` | `own`, default `reference`). Strict at every level, mirroring the existing `sources`/`SourceSpec` pattern: an unknown key, a second transport, no transport, a malformed name, a bad mode, or a duplicate name all refuse to parse. - `sdk/src/dependencies.ts`: the DESIGN-pinned constants (`DEPS_DIR`, `LOCK_FILE`, `LOCAL_OVERRIDE_FILE`, `MANIFEST_FILE`, `ARTIFACT_SUFFIX`, `DEPENDENCY_SOURCE_ID_PREFIX`, `INTEGRITY_PREFIX`, `DEPENDENCY_MODES`, `DEFAULT_DEPENDENCY_MODE`), `DependencySpec`/`DependencyMode` types, and `DependencySpecSchema` (zod), exported from the package barrel. - `sdk/src/config.ts`: `ConfigSchema.dependencies` (default `[]`, unique-names refine) plus the two-direction compile-time parity guard between the schema and the hand-written `DependencySpec`, matching the existing `SourceSpec` guard. - `server/python/.../config/dependencies.py`: the same constants. - `server/python/.../config/neutral_config.py`: `NeutralConfig.dependencies` and validation matching the TS schema's refusals exactly, coded `ERR_COLLECTION_NOT_FOUND` (this reader's existing shape-error code). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../python/src/metaobjects/config/__init__.py | 20 ++++ .../src/metaobjects/config/dependencies.py | 39 ++++++++ .../src/metaobjects/config/neutral_config.py | 92 ++++++++++++++++++- .../tests/config/test_neutral_config.py | 37 ++++++++ server/typescript/packages/sdk/src/config.ts | 19 ++++ .../packages/sdk/src/dependencies.ts | 91 ++++++++++++++++++ server/typescript/packages/sdk/src/index.ts | 18 ++++ .../packages/sdk/test/config.test.ts | 19 ++++ 8 files changed, 334 insertions(+), 1 deletion(-) create mode 100644 server/python/src/metaobjects/config/dependencies.py create mode 100644 server/typescript/packages/sdk/src/dependencies.ts diff --git a/server/python/src/metaobjects/config/__init__.py b/server/python/src/metaobjects/config/__init__.py index c0881621e..09760a0a5 100644 --- a/server/python/src/metaobjects/config/__init__.py +++ b/server/python/src/metaobjects/config/__init__.py @@ -5,11 +5,31 @@ TS-only key never becomes a four-port change. See `docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §4. """ +from .dependencies import ( + ARTIFACT_SUFFIX, + DEFAULT_DEPENDENCY_MODE, + DEPENDENCY_MODES, + DEPENDENCY_SOURCE_ID_PREFIX, + DEPS_DIR, + INTEGRITY_PREFIX, + LOCAL_OVERRIDE_FILE, + LOCK_FILE, + MANIFEST_FILE, +) from .neutral_config import DEFAULT_METADATA_DIR, NeutralConfig, read_neutral_config from .source_resolver import resolve_collection, resolve_sources __all__ = [ + "ARTIFACT_SUFFIX", + "DEFAULT_DEPENDENCY_MODE", "DEFAULT_METADATA_DIR", + "DEPENDENCY_MODES", + "DEPENDENCY_SOURCE_ID_PREFIX", + "DEPS_DIR", + "INTEGRITY_PREFIX", + "LOCAL_OVERRIDE_FILE", + "LOCK_FILE", + "MANIFEST_FILE", "NeutralConfig", "read_neutral_config", "resolve_collection", diff --git a/server/python/src/metaobjects/config/dependencies.py b/server/python/src/metaobjects/config/dependencies.py new file mode 100644 index 000000000..3da7db1b4 --- /dev/null +++ b/server/python/src/metaobjects/config/dependencies.py @@ -0,0 +1,39 @@ +"""FR-023 — metadata dependencies: constants shared by the `dependencies` key +of `.metaobjects/config.json` (DESIGN §3.1) and by later tasks (manifest, +lock, snapshot, sync). Mirrors +`server/typescript/packages/sdk/src/dependencies.ts` — constants only here; +resolving a declared dependency to bytes on disk lands in a later task. +""" +from __future__ import annotations + +#: Directory (under `.metaobjects/`) holding the synced snapshot artifacts, +#: one subdirectory per dependency name: `.metaobjects/deps//`. +DEPS_DIR = "deps" + +#: `meta deps sync`'s output — the only writer (DESIGN §3.3). +LOCK_FILE = "deps.lock.json" + +#: D10 co-development override — never committed (DESIGN §10 D10). +LOCAL_OVERRIDE_FILE = "deps.local.json" + +#: The publisher-generated manifest sitting beside a dependency's artifact +#: (DESIGN §3.2). +MANIFEST_FILE = "metaobjects.pkg.json" + +#: Suffix of a dependency's canonical-JSON artifact file, e.g. +#: `acme-common.metaobjects.json` (DESIGN §3.5). +ARTIFACT_SUFFIX = ".metaobjects.json" + +#: Prefix of a dependency artifact's source id: `dep:/` +#: (DESIGN §2.3, "Source ids"). +DEPENDENCY_SOURCE_ID_PREFIX = "dep:" + +#: Prefix of the `integrity` field's value: `"sha256-" + lowercase hex sha256 +#: of the artifact bytes` (DESIGN §3, "Hash format"). +INTEGRITY_PREFIX = "sha256-" + +#: The two modes a declared dependency may run in (DESIGN §2.4, §2.7). +DEPENDENCY_MODES = ("reference", "own") + +#: `mode`'s default when a dependency spec omits it. +DEFAULT_DEPENDENCY_MODE = "reference" diff --git a/server/python/src/metaobjects/config/neutral_config.py b/server/python/src/metaobjects/config/neutral_config.py index feedadea4..6ce28ce75 100644 --- a/server/python/src/metaobjects/config/neutral_config.py +++ b/server/python/src/metaobjects/config/neutral_config.py @@ -1,11 +1,14 @@ from __future__ import annotations import json +import re from dataclasses import dataclass from pathlib import Path from metaobjects.errors import ErrorCode, ParseError +from .dependencies import DEFAULT_DEPENDENCY_MODE, DEPENDENCY_MODES + #: The DEFAULT value of `sources` when the key is absent or empty — never a #: requirement, and never assumed to exist by any other code path. DEFAULT_METADATA_DIR = "metaobjects" @@ -13,6 +16,13 @@ _METAOBJECTS_DIR = ".metaobjects" _CONFIG_FILE = "config.json" +#: `/^[a-z0-9][a-z0-9._-]*$/` — mirrors the TS `DependencyName` regex in +#: `sdk/src/dependencies.ts` exactly (DESIGN §3.1). +_DEPENDENCY_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9._-]*$") + +#: The one transport key a dependency spec may carry. +_DEPENDENCY_TRANSPORT_KEYS = ("path", "npm", "python") + @dataclass(frozen=True) class NeutralConfig: @@ -21,6 +31,11 @@ class NeutralConfig: #: Raw source specs, each a single-key mapping (`path` / `resource` / `package`). sources: list[dict[str, str]] + #: Declared metadata dependencies (FR-023) — each dict already validated: + #: `name`, exactly one transport key (`path` / `npm` / `python`), `mode` + #: defaulted to `"reference"`, optional `dir` only beside `npm`/`python`. + dependencies: list[dict[str, str]] + def read_neutral_config(config_dir: Path) -> NeutralConfig | None: """Read the neutral subset from ``config_dir/.metaobjects/config.json``. @@ -83,5 +98,80 @@ def read_neutral_config(config_dir: Path) -> NeutralConfig | None: code=ErrorCode.ERR_COLLECTION_NOT_FOUND, ) + dependencies_raw = raw.get("dependencies", []) + if not isinstance(dependencies_raw, list): + raise ParseError( + f"{path}: 'dependencies' must be an array", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + dependencies = [_validate_dependency_spec(d, path) for d in dependencies_raw] + names = [d["name"] for d in dependencies] + if len(set(names)) != len(names): + raise ParseError( + f"{path}: 'dependencies' names must be unique", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + # Unknown top-level keys are IGNORED by design — see the module docstring. - return NeutralConfig(sources=[dict(s) for s in sources]) + return NeutralConfig(sources=[dict(s) for s in sources], dependencies=dependencies) + + +def _validate_dependency_spec(dep: object, path: Path) -> dict[str, str]: + """Validate one entry of `dependencies` (DESIGN §3.1) and return it in + normalized form (`mode` always present). Mirrors the TS + `DependencySpecSchema` union in `sdk/src/dependencies.ts` exactly: a + `name`, exactly one transport key (`path` | `npm` | `python`), an + optional `dir` that is legal only beside `npm`/`python`, an optional + `mode` defaulting to `"reference"`, and no other keys. + """ + + def fail(reason: str) -> ParseError: + return ParseError( + f"{path}: invalid 'dependencies' entry ({reason})", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + if not isinstance(dep, dict): + raise fail("must be an object") + + name = dep.get("name") + if not isinstance(name, str) or not _DEPENDENCY_NAME_RE.match(name): + raise fail("'name' must match ^[a-z0-9][a-z0-9._-]*$") + + transports = [k for k in _DEPENDENCY_TRANSPORT_KEYS if k in dep] + if len(transports) != 1: + raise fail("exactly one of 'path' / 'npm' / 'python' is required") + transport = transports[0] + + transport_value = dep[transport] + if not isinstance(transport_value, str) or not transport_value.strip(): + raise fail(f"'{transport}' must be a non-empty string") + + allowed_keys = {"name", transport, "mode"} + if transport in ("npm", "python"): + allowed_keys.add("dir") + extra_keys = set(dep.keys()) - allowed_keys + if extra_keys: + raise fail(f"unknown key(s): {sorted(extra_keys)}") + + result: dict[str, str] = {"name": name, transport: transport_value} + + if "dir" in dep: + dir_value = dep["dir"] + if not isinstance(dir_value, str) or not dir_value.strip(): + raise fail("'dir' must be a non-empty string") + result["dir"] = dir_value + + mode = dep.get("mode", DEFAULT_DEPENDENCY_MODE) + # `isinstance(mode, bool)` must be checked separately (same reason as + # `schema_version` above): `True`/`False` are `int` subclasses in + # Python, and `mode not in DEPENDENCY_MODES` alone would let a boolean + # through if either mode string ever coincided with `1`/`0` — it never + # does today, but the explicit check keeps this in lockstep with the + # `schema_version` guard's reasoning rather than relying on that + # coincidence. + if isinstance(mode, bool) or mode not in DEPENDENCY_MODES: + raise fail(f"'mode' must be one of {DEPENDENCY_MODES}") + result["mode"] = mode + + return result diff --git a/server/python/tests/config/test_neutral_config.py b/server/python/tests/config/test_neutral_config.py index b24cc178e..349a1b073 100644 --- a/server/python/tests/config/test_neutral_config.py +++ b/server/python/tests/config/test_neutral_config.py @@ -101,3 +101,40 @@ def test_non_string_source_value_raises(tmp_path: Path) -> None: _write_config(tmp_path, {"schema_version": 1, "sources": [{"path": 123}]}) with pytest.raises(ParseError): read_neutral_config(tmp_path) + + +# FR-023 — `dependencies` (DESIGN §3.1). Read as part of the neutral subset, +# same as `sources`: every port reads it at every rung of the source ladder. + + +def test_dependencies_parse_with_mode_default(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 1, "sources": [], "dependencies": [{"name": "acme-common", "path": "../lib"}]}) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.dependencies == [{"name": "acme-common", "path": "../lib", "mode": "reference"}] + + +def test_dependencies_absent_is_empty(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 1, "sources": []}) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.dependencies == [] + + +@pytest.mark.parametrize( + "bad", + [ + [{"name": "a", "path": "x", "npm": "y"}], + [{"name": "a"}], + [{"name": "Bad Name", "path": "x"}], + [{"name": "a", "path": "x", "mode": "shared"}], + [{"name": "a", "path": "x"}, {"name": "a", "npm": "y"}], + [{"name": "a", "path": "x", "dir": "y"}], + [{"name": "a", "path": "x", "pathh": "t"}], + ], +) +def test_dependencies_shape_errors(tmp_path: Path, bad: object) -> None: + _write_config(tmp_path, {"schema_version": 1, "sources": [], "dependencies": bad}) + with pytest.raises(ParseError) as e: + read_neutral_config(tmp_path) + assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND diff --git a/server/typescript/packages/sdk/src/config.ts b/server/typescript/packages/sdk/src/config.ts index 0e70467d7..4c70a24d9 100644 --- a/server/typescript/packages/sdk/src/config.ts +++ b/server/typescript/packages/sdk/src/config.ts @@ -2,6 +2,7 @@ import { z } from "zod"; import { readFile, writeFile } from "node:fs/promises"; import { join } from "node:path"; import type { SourceSpec } from "./sources.js"; +import { DependencySpecSchema, type DependencySpec } from "./dependencies.js"; const DialectEnum = z.enum(["sqlite", "postgres", "d1"]); @@ -106,6 +107,15 @@ const _sourceSpecParitySpecToInfer: z.infer = {} as Sou void _sourceSpecParityInferToSpec; void _sourceSpecParitySpecToInfer; +// Compile-time parity, BOTH directions, for `DependencySpecSchema` / +// `DependencySpec` — same rationale as the `SourceSpec` pair above: a +// one-directional assignment would let either side add an arm the other +// never gained and still compile clean. +const _dependencySpecParityInferToSpec: DependencySpec = {} as z.infer; +const _dependencySpecParitySpecToInfer: z.infer = {} as DependencySpec; +void _dependencySpecParityInferToSpec; +void _dependencySpecParitySpecToInfer; + /** Mirrors the hand-written `Scope` interface in `./scope.ts`. An absent or * empty `include` means "everything" — see `matchesScope`. */ const ScopeSchema = z @@ -130,6 +140,15 @@ export const ConfigSchema = z.object({ }) .default({}), sources: z.array(SourceSpecSchema).default([]), + /** Declared metadata dependencies (FR-023) — read from this file at every + * rung of the source ladder, by every port, not just TypeScript (DESIGN + * §2.3). Names must be unique across the array. */ + dependencies: z + .array(DependencySpecSchema) + .default([]) + .refine((a) => new Set(a.map((d) => d.name)).size === a.length, { + message: "dependencies: names must be unique", + }), /** Output filter applied across every command — see `./scope.ts`. Absent * means "everything" (no filtering), matching `Scope`'s own contract. */ scope: ScopeSchema.optional(), diff --git a/server/typescript/packages/sdk/src/dependencies.ts b/server/typescript/packages/sdk/src/dependencies.ts new file mode 100644 index 000000000..856a52c1a --- /dev/null +++ b/server/typescript/packages/sdk/src/dependencies.ts @@ -0,0 +1,91 @@ +// server/typescript/packages/sdk/src/dependencies.ts +// +// FR-023 — metadata dependencies: the `dependencies` key of +// `.metaobjects/config.json` (DESIGN §3.1) plus the constants every later +// task (manifest, lock, snapshot, sync) shares. This module carries the +// schema and the constants only — resolving a declared dependency to bytes +// on disk, the manifest, the lock and the sync command land in later tasks. +import { z } from "zod"; + +/** Directory (under `.metaobjects/`) holding the synced snapshot artifacts, + * one subdirectory per dependency name: `.metaobjects/deps//`. */ +export const DEPS_DIR = "deps"; + +/** `meta deps sync`'s output — the only writer (DESIGN §3.3). */ +export const LOCK_FILE = "deps.lock.json"; + +/** D10 co-development override — never committed (DESIGN §10 D10). */ +export const LOCAL_OVERRIDE_FILE = "deps.local.json"; + +/** The publisher-generated manifest sitting beside a dependency's artifact + * (DESIGN §3.2). */ +export const MANIFEST_FILE = "metaobjects.pkg.json"; + +/** Suffix of a dependency's canonical-JSON artifact file, e.g. + * `acme-common.metaobjects.json` (DESIGN §3.5). */ +export const ARTIFACT_SUFFIX = ".metaobjects.json"; + +/** Prefix of a dependency artifact's `FileSource` id: `dep:/` + * (DESIGN §2.3, "Source ids"). */ +export const DEPENDENCY_SOURCE_ID_PREFIX = "dep:"; + +/** Prefix of the `integrity` field's value: `"sha256-" + lowercase hex sha256 + * of the artifact bytes` (DESIGN §3, "Hash format"). */ +export const INTEGRITY_PREFIX = "sha256-"; + +/** The two modes a declared dependency may run in (DESIGN §2.4, §2.7). */ +export const DEPENDENCY_MODES = ["reference", "own"] as const; + +/** `mode`'s default when a dependency spec omits it. */ +export const DEFAULT_DEPENDENCY_MODE: (typeof DEPENDENCY_MODES)[number] = "reference"; + +export type DependencyMode = (typeof DEPENDENCY_MODES)[number]; + +/** + * A declared dependency: a `name`, exactly one transport (`path` | `npm` | + * `python`), and a `mode`. `npm`/`python` optionally carry `dir` — the + * subdirectory under the resolved package holding `metaobjects.pkg.json`; + * `path` never does, since the path itself already names that directory. + * + * Mirrors the hand-written union below in `DependencySpecSchema` — the same + * two-direction parity guard `SourceSpec`/`SourceSpecSchema` carries (see + * `config.ts`), so the schema and this type cannot silently drift. + */ +export type DependencySpec = { readonly name: string; readonly mode: DependencyMode } & ( + | { readonly path: string } + | { readonly npm: string; readonly dir?: string | undefined } + | { readonly python: string; readonly dir?: string | undefined } +); + +/** `/^[a-z0-9][a-z0-9._-]*$/` — lowercase, digits, `.`/`_`/`-`, starting with + * an alphanumeric (DESIGN §3.1). Deliberately narrower than an npm package + * name (no leading `@acme/`) — this is the LOCAL alias the consumer's own + * config, lock and `.metaobjects/deps//` directory key on, not the + * transport's own package name. */ +const DependencyName = z.string().regex(/^[a-z0-9][a-z0-9._-]*$/); + +const Mode = z.enum(DEPENDENCY_MODES).default(DEFAULT_DEPENDENCY_MODE); + +/** + * `.strict()` on every arm, same rationale as `SourceSpecSchema` in + * `config.ts`: a config schema that silently strips an unknown key would let + * `{ name: "a", path: "x", pathh: "typo" }` parse clean instead of erroring + * on the typo. A `z.union` (rather than a discriminated union) because the + * three arms share no single literal discriminator key — the transport key + * itself (`path`/`npm`/`python`) is what distinguishes them, and that is + * exactly what the "two transports in one spec" / "no transport" refusals + * below exercise: neither shape matches any arm, so the union fails closed. + */ +export const DependencySpecSchema = z.union([ + z.object({ name: DependencyName, path: z.string().min(1), mode: Mode }).strict(), + z.object({ name: DependencyName, npm: z.string().min(1), dir: z.string().min(1).optional(), mode: Mode }).strict(), + z.object({ name: DependencyName, python: z.string().min(1), dir: z.string().min(1).optional(), mode: Mode }).strict(), +]); + +/** The declared name of a dependency spec — `name` is common to all three + * transport arms, so this needs no narrowing. Exists so callers never + * inline `.name` and so later tasks (manifest/lock keying) have one place + * this projection is defined. */ +export function dependencyName(spec: DependencySpec): string { + return spec.name; +} diff --git a/server/typescript/packages/sdk/src/index.ts b/server/typescript/packages/sdk/src/index.ts index 6d51f737f..10cc892da 100644 --- a/server/typescript/packages/sdk/src/index.ts +++ b/server/typescript/packages/sdk/src/index.ts @@ -19,6 +19,24 @@ export { ConfigSchema, DEFAULT_CONFIG, loadConfig, saveConfig, AllowTokenEnum } from "./config.js"; export type { Config } from "./config.js"; +// Metadata dependencies (FR-023) — the `dependencies` config key's schema +// and constants. Resolution, the manifest, the lock and sync land in later +// tasks. +export { + DEPS_DIR, + LOCK_FILE, + LOCAL_OVERRIDE_FILE, + MANIFEST_FILE, + ARTIFACT_SUFFIX, + DEPENDENCY_SOURCE_ID_PREFIX, + INTEGRITY_PREFIX, + DEPENDENCY_MODES, + DEFAULT_DEPENDENCY_MODE, + DependencySpecSchema, + dependencyName, +} from "./dependencies.js"; +export type { DependencyMode, DependencySpec } from "./dependencies.js"; + // Meta Forge metadata types + attribute name constants (registered into a // TypeRegistry to let Loader parse decision/principle/etc. children + the // @forge* attribute namespace). diff --git a/server/typescript/packages/sdk/test/config.test.ts b/server/typescript/packages/sdk/test/config.test.ts index 98972872c..72d9d6512 100644 --- a/server/typescript/packages/sdk/test/config.test.ts +++ b/server/typescript/packages/sdk/test/config.test.ts @@ -184,3 +184,22 @@ describe("ConfigSchema — phase-1 source resolution", () => { expect(p.scope).toBeUndefined(); }); }); + +describe("ConfigSchema — dependencies (FR-023)", () => { + test("dependencies: a valid spec parses with mode defaulted", () => { + const cfg = ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "acme-common", path: "../lib/metaobjects" }] }); + expect(cfg.dependencies).toEqual([{ name: "acme-common", path: "../lib/metaobjects", mode: "reference" }]); + }); + test("dependencies: npm and python accept dir; path does not", () => { + expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "a", npm: "@acme/model", dir: "metaobjects" }] })).not.toThrow(); + expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "a", python: "acme_model", dir: "metaobjects", mode: "own" }] })).not.toThrow(); + expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "a", path: "x", dir: "y" }] })).toThrow(); + }); + test("dependencies: two transports, no transport, a bad name, a bad mode, a duplicate name, an unknown key are all refused", () => { + for (const bad of [ + [{ name: "a", path: "x", npm: "y" }], [{ name: "a" }], [{ name: "Bad Name", path: "x" }], + [{ name: "a", path: "x", mode: "shared" }], [{ name: "a", path: "x" }, { name: "a", npm: "y" }], [{ name: "a", path: "x", pathh: "typo" }], + ]) expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: bad })).toThrow(); + }); + test("dependencies: absent means []", () => { expect(ConfigSchema.parse({ schema_version: 1 }).dependencies).toEqual([]); }); +}); From 25297ae254de8796e7ee498cade1f0a125704a5e Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 19:39:33 -0400 Subject: [PATCH 11/62] docs(fr-023): exclusion keys on `packages` with a loud refusal; runtime option cut MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The re-cut left six rulings open. All are now settled, and the design and plan are corrected to match. - Exclusion is keyed on the lock's `packages`, not its `nodes`. The design argued `nodes` on a single empirical claim: that legacy consumers declare new top-level nodes into a dependency's package. That was checked against the estate and no instance distinguishes the two keys — the one legacy consumer declaring into a shared framework package adds no class the framework does not already declare, because it amends rather than adds, and an amendment is an overlay. The claim is false as written; §2.4 and §2.7 are corrected and §11.5 records the evidence. - The one failure mode a package-keyed rule introduces — a consumer node in a dependency package silently never generating — becomes a loud `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, raised after load. An overlay merges into a node whose FQN is in the lock's `nodes`, so it passes untouched; only a genuinely new object is refused. `nodes` is retained in the manifest and lock for artifact integrity, collision detection and that refusal. - A wildcard `include` does not opt a dependency in; only a literal package prefix does. - Task 16, the ObjectManager scope predicate, is cut: "runtime" meant what metadata gets loaded, which the collection resolver already does. The task number is kept so the progress ledger's references stay resolvable. - `verify --deps` fails when the publisher is unreachable, and stays out of the default verify run. - Python keeps applying only the import rule, not the user's full `scope`, to its own objects — confirmed against the tree, where no caller of the scope grammar exists. Error codes after Task 6 become seven. `metamodelVersion` is unaffected: the gate reads the registry manifest, and an error code is not registered vocabulary. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../plans/2026-09-11-fr-023-phase-1a.md | 861 +++++++----------- ...-11-fr-023-metadata-dependencies-design.md | 209 ++++- 2 files changed, 541 insertions(+), 529 deletions(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index ea45ffeb2..ae84463e3 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -2,28 +2,32 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** A consuming project declares a dependency on a library's generated shared-model artifact, syncs it into a committed snapshot + lock, references / extends / overlays its nodes, never generates or migrates them in `reference` mode, and fails loudly — at load and at `verify --deps` — when upstream changes what it built on. +> **Re-cut 2026-09-11 (DESIGN §11, "B+").** Tasks 1–5 are DONE and kept verbatim. Tasks 6 onward are the B+ sequence: thin dependencies (`{ name, path }`, snapshot + sha256 lock, `deps sync` / `check`, `sharedModelFile()`), imported metadata LOAD-ONLY by default (excluded from codegen, CLI, schema and the ledger unless a scope's `include` names the package), the overlay authoring lint, and the pruning of `mode` and three error codes that Tasks 1 and 5 registered under the superseded design. No `foreignOwner`, no modes, no classifier, no `npm`/`python` transport, no local override, no runtime predicate (Task 16 CUT 2026-09-11). -**Architecture:** The publisher runs a `sharedModelFile()` generator that emits one flattened canonical-JSON artifact plus a manifest; the consumer's Node `meta deps sync` copies that artifact into `.metaobjects/deps//` and pins its hash in `.metaobjects/deps.lock.json`; every port's collection resolver loads the snapshot files (with `dep:`-prefixed source ids) beside the project's own sources and exposes one predicate, `governs(fqn)`, that codegen and migrate thread through their existing scope seams. A usage-aware classifier diffs the foreign nodes the consumer uses when upstream moves. +**Goal:** A consuming project declares a dependency on a library's generated shared-model artifact, syncs it into a committed snapshot + lock, references / extends / overlays its nodes, and — unless it names the imported package in its own `scope.include` / `migrate.scope` — never generates from, migrates or counts them; it fails loudly at load when upstream removed what it built on, and at `verify --deps` when upstream moved at all. -**Tech Stack:** TypeScript (Bun test runner, zod, `node:crypto`, `node:util` `parseArgs`), Python 3.11+ (`uv`, pytest, PyYAML), a file-shaped cross-port corpus under `fixtures/dependency-conformance/`. +**Architecture:** The publisher runs a `sharedModelFile()` generator that emits one flattened canonical-JSON artifact plus a manifest; the consumer's Node `meta deps sync` copies that artifact into `.metaobjects/deps//` and pins its hash (plus the exported `nodes` and `packages`) in `.metaobjects/deps.lock.json`; every port's collection resolver loads the snapshot files (with `dep:`-prefixed source ids) ahead of the project's own sources and composes ONE predicate per surface from the existing `scope` / `migrate.scope` declarations and the lock's `packages`. Codegen, schema tooling and the requirements ledger all thread that predicate through seams that already exist. -**Spec:** `docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md` (referred to below as DESIGN; every task names the § it implements). +**Tech Stack:** TypeScript (Bun test runner, zod, `node:crypto`, `node:util` `parseArgs`), Python 3.11+ (`uv`, pytest, PyYAML), a file-shaped cross-port corpus under `fixtures/dependency-conformance/`, Testcontainers Postgres (the `integration-tests` package) for the one schema test that needs real schemas. + +**Spec:** `docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md` (referred to below as DESIGN; §11 is the ruling that governs Tasks 6+; every task names the § it implements). ## Global Constraints -Every task's requirements implicitly include this section. Reviewers: read a task's diff against these contracts. +Every task's requirements implicitly include this section. Reviewers: read a task's diff against these contracts. Where this section and DESIGN §§1–10 disagree, DESIGN §11 and this section win (the §11 table names what is superseded). -**Scope of this plan.** Phase 1a only: TypeScript (`metadata`, `sdk`, `codegen-ts`, `cli`), Python, `fixtures/dependency-conformance/`, docs, skills, CHANGELOG. Phase 1b (`packageBindings`, the `docs` badge) and the estate cutover are OUT. Java/Kotlin/C# are OUT (Phase 2). +**Scope of this plan.** Phase 1a only: TypeScript (`metadata`, `sdk`, `codegen-ts`, `migrate-ts`, `cli`, one Postgres test in `integration-tests`), Python, `fixtures/dependency-conformance/`, docs, skills, CHANGELOG. OUT (DESIGN §11.3–§11.4): `mode`, `foreignOwner` / provenance / the boundary validator, the usage-aware classifier and `--accept-breaking`, the `npm` / `python` transports, the D10 local override (`deps.local.json`), the Python `shared-model` generator and `targets..options`, `packageBindings`, the docs badge, **the `ObjectManager` runtime scope predicate (Task 16 — CUT 2026-09-11)**, Java/Kotlin/C# (Phase 2). -**No metamodel vocabulary change.** Nothing in this plan registers a type, subtype or attribute. `fixtures/registry-conformance/expected-registry.json` and `METAMODEL_VERSION` (`server/typescript/packages/metadata/src/registry-manifest.ts`, `"1.0"`) do not move; `node scripts/check-metamodel-version.mjs` must report no diff at the end. Config keys, the manifest, the lock and the override file are tool files. +**No metamodel vocabulary change.** Nothing in this plan registers a type, subtype or attribute. `fixtures/registry-conformance/expected-registry.json` and `METAMODEL_VERSION` (`server/typescript/packages/metadata/src/registry-manifest.ts`, `"1.0"`) do not move; `node scripts/check-metamodel-version.mjs` must report no diff at the end. Config keys, the manifest and the lock are tool files. -**Config — `.metaobjects/config.json` (DESIGN §3.1), strict at every level:** +**Config — `.metaobjects/config.json` (DESIGN §3.1 as amended by §11), strict at every level:** ```jsonc { "schema_version": 1, "sources": [], - "dependencies": [ { "name": "acme-common", "path": "../lib/metaobjects", "mode": "reference" } ] } + "scope": { "include": ["app::**"] }, // existing — codegen + CLI + ledger + runtime + "migrate": { "scope": ["app::**"] }, // existing — schema tooling + "dependencies": [ { "name": "acme-common", "path": "../lib/metaobjects" } ] } ``` -A dependency spec carries `name` (`/^[a-z0-9][a-z0-9._-]*$/`, unique across the array), exactly ONE transport key — `path` (string) | `npm` (string, optional `dir`) | `python` (string, optional `dir`) — and `mode` ∈ `"reference" | "own"`, default `"reference"`. Any other key is a schema error. `dependencies` is read by every port at EVERY rung of the source ladder (DESIGN §2.3). +A dependency spec carries `name` (`/^[a-z0-9][a-z0-9._-]*$/`, unique across the array) and exactly ONE transport key. **Only `path` resolves in Phase 1a.** The `npm` / `python` arms (optional `dir`) STAY in the schema as reserved (the `sources` `resource` / `package` precedent) and `meta deps sync` refuses them with `ERR_DEPENDENCY_UNRESOLVED` ("transport `npm` is not supported by this toolchain yet; use `path`"). **There is no `mode` key** — Task 6 removes it. Any other key is a schema error. `dependencies` is read by every port at EVERY rung of the source ladder (DESIGN §2.3). **Manifest — `metaobjects.pkg.json` (DESIGN §3.2), generated, at the artifact's directory:** ```jsonc @@ -33,41 +37,66 @@ A dependency spec carries `name` (`/^[a-z0-9][a-z0-9._-]*$/`, unique across the ``` `packages` and `nodes` are sorted; `nodes` are resolution keys of every top-level node in the artifact. -**Lock — `.metaobjects/deps.lock.json` (DESIGN §3.3), written only by `meta deps sync`:** +**Lock — `.metaobjects/deps.lock.json` (DESIGN §3.3 minus `mode`), written only by `meta deps sync`:** ```jsonc { "schema_version": 1, "dependencies": { "acme-common": { - "version": "1.0.0", "metamodelVersion": "1.0", "mode": "reference", + "version": "1.0.0", "metamodelVersion": "1.0", "resolvedFrom": { "path": "../lib/metaobjects" }, "artifact": "acme-common.metaobjects.json", "integrity": "sha256-…", "packages": ["acme::common"], "nodes": ["acme::common::Address", "…"] } } } ``` -Keys sorted by name; no timestamps, no absolute paths, no interpreter paths. +Keys sorted by name; no timestamps, no absolute paths. The reference lock used by corpus cases and tests is `LOCK_V1`: +```json +{ "schema_version": 1, "dependencies": { "acme-common": { "version": "1.0.0", "metamodelVersion": "1.0", "resolvedFrom": { "path": "../acme-common/metaobjects" }, "artifact": "acme-common.metaobjects.json", "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", "packages": ["acme::common"], "nodes": ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"] } } } +``` +with `CONFIG_REF` = `{ "schema_version": 1, "sources": [], "dependencies": [{ "name": "acme-common", "path": "../acme-common/metaobjects" }] }`, `SNAP` = `"treeFiles": { ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" }`, and `APP` = `"metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}"`. (The `path` need not exist on disk for a resolution case — only `meta deps sync` / `deps check` resolve it.) **Snapshot (DESIGN §3.4).** `.metaobjects/deps//` — bytes verbatim; never walked; loaded by the lock's `artifact` path with source id `dep:/`. -**Artifact (DESIGN §3.5).** A canonical-JSON `metadata.root` document with NO root `package`; each top-level node carries an explicit `package`; raw (own-layer) form, `extends` preserved; top-level nodes sorted by resolution key; attribute keys alphabetized; body key order `name, package, extends, abstract, isArray, @attrs, children`; 2-space indent, LF, one trailing newline; Python emits with `ensure_ascii=False`. +**Artifact (DESIGN §3.5).** A canonical-JSON `metadata.root` document with NO root `package`; each top-level node carries an explicit `package`; raw (own-layer) form, `extends` preserved; top-level nodes sorted by resolution key; attribute keys alphabetized; body key order `name, package, extends, abstract, isArray, @attrs, children`; 2-space indent, LF, one trailing newline; Python emits with `ensure_ascii=False`. Produced by `serializeSharedDocument` (Task 2). -**Hash format.** `integrity` = `"sha256-" + lowercase hex sha256 of the artifact bytes`. Pinned corpus value: the committed `fixtures/dependency-conformance/artifacts/acme-common-v1.json` hashes to `sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d`. +**Hash format.** `integrity` = `"sha256-" + lowercase hex sha256 of the artifact bytes`. Pinned corpus values (`fixtures/dependency-conformance/README.md`): `acme-common-v1.json` → `sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d`; `acme-common-v1-widened.json` → `sha256-fa00b9f3c54a2c302cf269af051589afc0a3217999ac82c54c77e33494638c36`; `acme-common-v2-email-removed.json` → `sha256-c4ba364bff0071b164b127326c095d6d32915b57781271e0be64a7003c27ce7a`. -**Named constants (TS `sdk/src/dependencies.ts`, Python `metaobjects/config/dependencies.py`):** `DEPS_DIR = "deps"`, `LOCK_FILE = "deps.lock.json"`, `LOCAL_OVERRIDE_FILE = "deps.local.json"`, `MANIFEST_FILE = "metaobjects.pkg.json"`, `ARTIFACT_SUFFIX = ".metaobjects.json"`, `DEPENDENCY_SOURCE_ID_PREFIX = "dep:"`, `DEPENDENCY_MODES = ["reference", "own"]`, `DEFAULT_DEPENDENCY_MODE = "reference"`, `INTEGRITY_PREFIX = "sha256-"`. No metamodel string is ever inlined; metamodel names come from `@metaobjectsdev/metadata/constants` / `metaobjects.shared.*`. +**Named constants (TS `sdk/src/dependencies.ts`, Python `metaobjects/config/dependencies.py`):** `DEPS_DIR = "deps"`, `LOCK_FILE = "deps.lock.json"`, `MANIFEST_FILE = "metaobjects.pkg.json"`, `ARTIFACT_SUFFIX = ".metaobjects.json"`, `DEPENDENCY_SOURCE_ID_PREFIX = "dep:"`, `INTEGRITY_PREFIX = "sha256-"`. `LOCAL_OVERRIDE_FILE`, `DEPENDENCY_MODES` and `DEFAULT_DEPENDENCY_MODE` are REMOVED by Task 6. No metamodel string is ever inlined; metamodel names come from `@metaobjectsdev/metadata/constants` / `metaobjects.shared.*`. -**Error codes (exactly these nine, registered in `fixtures/conformance/ERROR-CODES.json`, TS `ERROR_CODES`, Python `ErrorCode`):** `ERR_DEPENDENCY_UNRESOLVED`, `ERR_DEPENDENCY_MANIFEST_INVALID`, `ERR_DEPENDENCY_SNAPSHOT_STALE`, `ERR_DEPENDENCY_NODE_COLLISION`, `ERR_DEPENDENCY_OVERLAY_IMPLICIT`, `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`, `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`, `ERR_DEPENDENCY_UPSTREAM_DRIFT`, `ERR_DEPENDENCY_BREAKING_CHANGE`. +**Error codes (exactly these SEVEN after Task 6, registered in `fixtures/conformance/ERROR-CODES.json`, TS `ERROR_CODES`, Python `ErrorCode`):** `ERR_DEPENDENCY_UNRESOLVED`, `ERR_DEPENDENCY_MANIFEST_INVALID`, `ERR_DEPENDENCY_SNAPSHOT_STALE`, `ERR_DEPENDENCY_NODE_COLLISION`, `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`, `ERR_DEPENDENCY_UPSTREAM_DRIFT`, `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`. Task 1 registered nine; Task 6 removes `ERR_DEPENDENCY_OVERLAY_IMPLICIT`, `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`, `ERR_DEPENDENCY_BREAKING_CHANGE` and ADDS `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, in all three registries together. Adding a code does NOT move `metamodelVersion` — `scripts/check-metamodel-version.mjs` reads `expected-registry.json` only, and an error code is not registered vocabulary. -**The foreign predicate (DESIGN §2.7).** `foreignOwner(fqn)` = the resolved dependency whose lock `nodes` contains `fqn`, else `undefined`. `governs(fqn)` = `foreignOwner(fqn) === undefined || foreignOwner(fqn).mode === "own"`. Foreignness is a set lookup on the lock's `nodes` — never package ownership, never file provenance. Codegen selects `inScope(fqn) && governs(fqn)`; migrate governs `governs(fqn) && (inMigrateScope?.(fqn) ?? true)`. +**The default-exclusion rule (DESIGN §11.1 item 2) — one definition, every surface.** +- `imported(fqn)` — `packageOf(fqn)` is in the union of every lock entry's `packages`. A set lookup on PACKAGES; never per-object, never file provenance. (Ruled 2026-09-11 — DESIGN §11.5. The estate has no case where a per-object key differs from a per-package one, and `packages` is one concept rather than two.) +- **`ERR_DEPENDENCY_PACKAGE_NOT_OWNED`** — the refusal that keeps the package rule from failing silently. After load, a top-level object whose `packageOf(fqn)` is a dependency package and whose FQN is NOT in that dependency's lock `nodes` was declared locally into a package the consumer does not own → raise, naming the object, its package and the dependency, with the fix: `'' is declared here but package '' is owned by dependency '' — declare it in your own package and extend '::', or amend an existing node with overlay: true`. An OVERLAY merges into the existing node, whose FQN IS in `nodes`, so it passes untouched; only a genuinely new top-level object is refused. This is the ONE thing the lock's `nodes` is still read for at load time (besides collision detection at sync). +- `explicitlyIncluded(pkg, patterns)` — some pattern in `patterns` names `pkg` literally: split on `::`, drop the final segment, the rest contain no `*` and join to exactly `pkg`. `acme::common::**` and `acme::common::Address` name `acme::common`; `acme::**`, `**`, an empty or absent list do not. +- `Collection.inScope(fqn)` (codegen, `verify --codegen`, shared enums, `meta gen `, the ledger) = `matchesScope(fqn, scope) && (!imported(fqn) || explicitlyIncluded(packageOf(fqn), scope.include))`. +- `Collection.inMigrateScope(fqn)` (`migrate`, `verify --db`, offline generate, replay) = `(declaredMigrateScope?.(fqn) ?? true) && (!imported(fqn) || explicitlyIncluded(packageOf(fqn), migrate.scope))`; **`undefined` iff the project declares no `migrate.scope` AND resolves no dependencies** (the byte-identical path). `Collection.declaredMigrateScope` is the user's `migrate.scope` alone (in lockstep with `migrateScopePatterns`) and is what the "scope matched nothing" refusal reads. +- `packageOf(fqn)` is `packageOfResolutionKey` (`metadata/src/naming.ts`, Task 2) / `package_of_resolution_key`. +- A project with no dependencies resolves and behaves byte-identically to today on every surface. -**Overlay rules (DESIGN §2.4).** A local top-level node whose resolution key is in a dependency's `nodes` MUST carry `overlay: true` → else `ERR_DEPENDENCY_OVERLAY_IMPLICIT`. In `reference` mode a LOCAL contribution to a foreign node that is a `field.*` / `identity.*` / `index.*` / `relationship.*` / `source.*` child, a locally-set `@table` / `@schema` / `@column` / `@kind`, or a local TPH subtype (declares `@discriminatorValue`) whose discriminator base is foreign → `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`. `own` mode allows all of it. Extending a foreign abstract, referencing a foreign node by FQN, and declaring new nodes in the dependency's package are always allowed. +**Two-sided schema exclusion (DESIGN §11.1 item 2).** `scopeExpectedSchema(built, inScope, { imported })` removes an object with `imported(fqn) && !inScope(fqn)` from the expected side BEFORE `declaredSchemas` is computed, adds its qualified name to `outOfScope` (so `scopedDiffInputs` suppresses it on the actual side), and then runs the existing user-scope filter over the remainder unchanged. Importing a table-backed entity must NEVER make `migrate` or `verify --db` propose a `DROP` of the publisher's other tables, and must never propose creating the imported table — unless `migrate.scope` names the package, in which case the consumer governs it (the legacy own-the-shared-model case). -**Classifier (DESIGN §2.6).** Footprint scopes: `whole` (targets of `extends` and of `field.object`/`field.map` `@objectRef`, `@payloadRef`, `@responseRef`, `@parameterRef`; closed over the target's `extends` chain), `key` (targets of `@references`, relationship `@objectRef`/`@through`, origin `@from`/`@of`/`@via`: the target's `identity.*` and `source.*` children plus the specifically named members), `existence` (`overlay: true` targets: the node's presence and `type.subType`, the attrs the overlay also sets, the parents of locally-added children; `@implementedBy`). Widest scope wins. Classes: node/descendant removed, `type.subType` changed, `extends`/`abstract`/`isArray` changed, attribute removed, attribute changed outside the widening table, attribute added outside the additive table → **breaking**; child node added → **compatible**; documentation attrs `description`, `title`, `notes`, `seeAlso`, `aliases`, `replacedBy` → **ignored**; `deprecated` added → **info**. Widening table (compatible in the stated direction only): `@maxLength` up, `@minLength` down, `@precision` up, `@max` up, `@min` down, `@required` true→false, `@values` superset, `@intValueMap` superset with existing entries unchanged. Additive table (attr appearing where none was, compatible): the documentation attrs, `@filterable`, `@sortable`. Anything not in a table is breaking. +**Runtime option — CUT (2026-09-11).** Task 16 is not built. The maintainer's "runtime" meant *what metadata gets loaded*, which the collection resolver (Task 8) already does; the `ObjectManager` predicate was scope nobody asked for. DESIGN §11.4 carries the trigger to revisit. -**D10 — local override (DESIGN §10 D10).** `.metaobjects/deps.local.json` = `{ "": { "path": "" } }`, never committed (`meta init` and `meta deps sync` add `deps.local.json` to `.metaobjects/.gitignore`). While present: every load in every port reads `/` in place, skips that dependency's snapshot hash check, and prints ONE warning per command: `loading from a local override (), not the committed snapshot`. `meta deps sync ` copies from the override path. `meta verify --deps` FAILS while any override is active (exit 1, message names the dependencies). An override naming an undeclared dependency → `ERR_DEPENDENCY_UNRESOLVED`; an override path without a manifest → `ERR_DEPENDENCY_UNRESOLVED`. +**Overlay lint (DESIGN §11.1 item 4).** `meta verify` (TS) reports, as an ADVISORY authoring finding (never fails the build; its own capped section like the requirement lint; muted by `--no-overlay-lint` / `META_NO_OVERLAY_LINT=1`), every top-level `(type, resolutionKey)` declared in two or more of the collection's files (dependency artifacts included, since they are the bases) where MORE THAN ONE declaration lacks `overlay: true`. Exactly one unflagged declaration is the base and is fine; every additional unflagged one is a finding naming its file: ` is redeclared in without overlay: true — add the flag so a removed or renamed target fails loudly (ERR_OVERLAY_NO_TARGET) instead of silently becoming a new object`. Diagnostic code `WARN_OVERLAY_IMPLICIT`, a CLI-local constant like `ERR_REQUIREMENT_LINK_ABOVE_FLOOR` in `requirement-check.ts` — NOT an `ERROR-CODES.json` entry (it is a TypeScript-only verify finding, not a loader code). No loader behaviour changes. -**Source ids.** A dependency artifact loads as a `FileSource` whose `id` is `dep:/` (override or snapshot alike); local files keep `basename(path)`. +**Source ids.** A dependency artifact loads as a `FileSource` whose `id` is `dep:/`; local files keep `basename(path)`. Diagnostic only — nothing decides on them. + +**Corpus (`fixtures/dependency-conformance/cases.json`) — the B+ case schema.** Exactly one of `expectFiles` / `expectError` per case; the optional arms ride with `expectFiles`: `expectImported` (the FQN set, over EVERY loaded top-level object, for which `collection.imported(fqn)` is true), `expectSelected` (the set for which `collection.inScope(fqn)` is true), `expectMigrateGoverned` (the set for which `collection.inMigrateScope?.(fqn) ?? true` is true), `expectLoadError`, `expectErrorFiles`. **Every set arm is EXHAUSTIVE — asserted as set equality over all loaded top-level objects, never as a per-listed existence check** (the Task 1 ruling). `treeFiles`, `tree`, `config`, `lock`, `resolveFrom` as in the README. `localOverrides`, `expectOverrides`, `expectForeign`, `expectGoverned` and `classify` are REMOVED by Task 6. + +**Pre-flight rulings carried into B+** (each is also restated in its task): +- Task 1 ⚠ → every corpus set arm is exhaustive (above). +- Task 1 minor → `sdk/test/dependency-conformance.test.ts` inlines `"deps.lock.json"`; backfill with `LOCK_FILE` (Task 6). +- Task 4 minor (cross-port semantics, fix before merge) → Python `scope.py` `matches_scope` must use `re.fullmatch`, not `re.match` (Task 6). +- Task 4 minor → `docs/CONFORMANCE.md` and `fixtures/scope-conformance/README.md` still name `sdk/src/scope.ts` as the reference and TS as the only runner (Task 20). +- T9 ruling → `meta gen ` naming an excluded import is a new pre-check that refuses (exit 2); the existing warn-on-unmatched path is unchanged (Task 10). +- T14 ruling (`summary` in the ignored doc attrs) → MOOT: the classifier is dropped. +- T17 ruling (the no-override fast path) → MOOT: D10 is dropped. +- T18 ruling → a new full-result Python resolver (`resolve_collection_full`); `resolve_collection(root) -> list[Path]` stays a thin projection (public shape unchanged); Python shared-enum selection follows the using entity (parity with Task 10) (Task 18). +- T20 ruling → no `governs` filter on `verify --templates` (dead: exports never carry `template.*`); the ledger exclusion stays (Task 12). +- T21 ruling → `init` tests go in the existing `cli/test/init.test.ts` (Task 19). +- T22 ruling → `docs/features/entities.md` is in the docs task (Task 20). **Discipline.** TDD: every task writes the failing test first and shows it failing. ADR-0039: read metadata through resolving accessors (`attr()`, `children()`, `attrs().get()`); an `own*()` call needs a comment naming its sanctioned case (own-mode serialization, root-level scans, overlay/merge machinery). Cross-package `instanceof` ban: identify nodes with the exported guards (`isMetaObject`, `isMetaField`, …) — never `x instanceof MetaSource` outside `metadata`. Constants discipline (above). Never mutate loaded metadata. Never edit a generated golden or `.hashes.json` by hand — regenerate through the tool and review the diff. `no-hardcoded-metadata-dir.test.ts` must pass with NO allowlist change. Public-repo hygiene: no private project names, no client names, no `/home/` or `~/` paths in any file, test, fixture, commit message or comment — the consumer is "a consuming app", the library is "a library project". ADR-0034: a new ejectable generator is listed by `meta eject --list`. -**Commands.** TS tests are scoped per package: `cd server/typescript/packages/ && bun test ` — NEVER a bare `bun test` at the repo root. Typecheck from the repo root: `bun run --filter '@metaobjectsdev/' typecheck`. Python: `cd server/python && uv run --extra integration pytest -q`. Commit after each task with the subject given; never `git add -A` — stage the task's files by name. Commit-message and branch hygiene follow the public-repo rules above. +**Commands.** TS tests are scoped per package: `cd server/typescript/packages/ && bun test ` — NEVER a bare `bun test` at the repo root. Typecheck from the repo root: `bun run --filter '@metaobjectsdev/' typecheck`. Python: `cd server/python && uv run --extra integration pytest -q`. Postgres: `cd server/typescript/packages/integration-tests && bun test ` (uses `METAOBJECTS_TEST_PG_URL` when set, else Testcontainers). Commit after each task with the subject given; never `git add -A` — stage the task's files by name. Commit-message and branch hygiene follow the public-repo rules above. --- - ### Task 1: Corpus skeleton, pinned artifacts, and the nine error codes **Files:** @@ -584,725 +613,515 @@ feat(config): `dependencies` in .metaobjects/config.json — sdk schema and Pyth --- -### Task 6: Lock and manifest schemas, integrity hashing, and the resolved-dependency type (TS + Python) +### Task 6: Prune `mode`, the local-override constant and the three superseded error codes; fix the two carried minors + +**Files:** +- Modify: `server/typescript/packages/sdk/src/dependencies.ts` (remove `DEPENDENCY_MODES`, `DEFAULT_DEPENDENCY_MODE`, `DependencyMode`, `LOCAL_OVERRIDE_FILE`, the `mode` member of `DependencySpec` and the `Mode` zod field on every arm) +- Modify: `server/typescript/packages/sdk/src/index.ts` (drop the removed exports) +- Modify: `server/typescript/packages/sdk/test/config.test.ts` (the "FR-023" describe: no `mode` in the parsed shape; `mode: "own"` / `mode: "shared"` become schema ERRORS as unknown keys) +- Modify: `server/python/src/metaobjects/config/dependencies.py` (remove `DEPENDENCY_MODES`, `DEFAULT_DEPENDENCY_MODE`, `LOCAL_OVERRIDE_FILE`) +- Modify: `server/python/src/metaobjects/config/neutral_config.py` (`_validate_dependency_spec`: `allowed_keys = {"name", transport}` (+ `dir` beside `npm`/`python`); no `mode` handling; docstrings) +- Modify: `server/python/tests/config/test_neutral_config.py` (`test_dependencies_parse_with_mode_default` → `test_dependencies_parse` asserting `[{"name": "acme-common", "path": "../lib"}]`; the `"mode": "shared"` parametrize row stays as an unknown-key error) +- Modify: `fixtures/conformance/ERROR-CODES.json`, `server/typescript/packages/metadata/src/errors.ts`, `server/python/src/metaobjects/errors.py` (remove `ERR_DEPENDENCY_OVERLAY_IMPLICIT`, `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`, `ERR_DEPENDENCY_BREAKING_CHANGE`; ADD `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` — `"FR-023: a local top-level node is declared into a package owned by a dependency."`; reword `ERR_DEPENDENCY_UNRESOLVED` to drop "or local override … or an override names an undeclared dependency"). All three registries move together — `errors.test.ts` asserts `ERROR_CODES` == the JSON's keys. +- Modify: `fixtures/dependency-conformance/README.md` (case schema: remove `localOverrides`, `expectOverrides`, `expectForeign`, `expectGoverned`, `classify`; add `expectImported`, `expectSelected`, `expectMigrateGoverned` with the EXHAUSTIVE wording; remove the "classification (TS + Python only)" paragraph; the "which arms each port runs" list becomes resolution / integrity / load-time failure for TS and Python) +- Modify: `server/typescript/packages/sdk/test/dependency-conformance.test.ts` (the `Case` interface and runner: drop the classify skip, `localOverrides`, `expectOverrides`, `expectForeign`, `expectGoverned`; add the three exhaustive set arms calling `collection.imported` / `collection.inScope` / `collection.inMigrateScope`; write the lock under `LOCK_FILE` imported from `../src/dependencies.js` instead of the inlined string) +- Modify: `server/python/src/metaobjects/scope.py` (`matches_scope`: `re.fullmatch`) +- Test: `server/python/tests/conformance/test_scope_conformance.py` (extend: a pattern must NOT match an fqn with a trailing newline), `server/typescript/packages/metadata/test/errors.test.ts`, `server/python/tests/unit/test_errors.py` (existing parity tests — must stay green) + +**Read first:** DESIGN §11.3 (what is superseded and why). Task 5's diff (`git show 1918a02a5`) — this task is its inverse for `mode` only. `metadata/test/errors.test.ts` asserts `ERROR_CODES` == the JSON's keys; `tests/unit/test_errors.py` asserts `ErrorCode` ⊇ the JSON's keys — so all three registries move together. Carried minors: Task 1 (`LOCK_FILE` backfill), Task 4 (`re.fullmatch` — `$` matches before a trailing `\n` in Python `re.match`, unlike JS). + +**Interfaces:** +- Produces TS: `DependencySpec = { readonly name: string } & ({ path: string } | { npm: string; dir?: string } | { python: string; dir?: string })`; `DependencySpecSchema` with no `mode`; `Config.dependencies: DependencySpec[]`. +- Produces Python: `NeutralConfig.dependencies` entries carry `name` + one transport key (+ `dir`), nothing else. +- Removes: the three codes; `LOCAL_OVERRIDE_FILE`; `DEPENDENCY_MODES`; `DEFAULT_DEPENDENCY_MODE`; `DependencyMode`. + +- [ ] **Step 1: Failing tests.** TS `sdk/test/config.test.ts`: change the first FR-023 test to expect `[{ name: "acme-common", path: "../lib/metaobjects" }]` (no `mode`) and add `{ name: "a", path: "x", mode: "reference" }` to the refused list. Python: rename/rewrite `test_dependencies_parse_with_mode_default` as above; add `[{"name": "a", "path": "x", "mode": "reference"}]` to `bad`. Python scope: `assert not matches_scope("acme::Order\n", compile_scope({"include": ["acme::*"]}))`. Run: +```bash +cd server/typescript/packages/sdk && bun test test/config.test.ts +cd server/python && uv run --extra integration pytest tests/config/test_neutral_config.py tests/conformance/test_scope_conformance.py -q +``` +Expected: FAIL (`mode` still parses; `re.match` still matches). + +- [ ] **Step 2: Implement** every removal listed under Files. In `errors.ts` delete the three entries and their comments; in `errors.py` the three enum members; in `ERROR-CODES.json` the three keys. Rewrite the README case-schema block and the runner. `re.fullmatch` in `scope.py`. + +- [ ] **Step 3: Run** +```bash +cd server/typescript/packages/sdk && bun test test/config.test.ts test/dependency-conformance.test.ts +cd server/typescript/packages/metadata && bun test test/errors.test.ts test/scope-conformance.test.ts +cd server/python && uv run --extra integration pytest tests/config tests/unit/test_errors.py tests/conformance/test_scope_conformance.py -q +bun run --filter '@metaobjectsdev/sdk' typecheck && bun run --filter '@metaobjectsdev/metadata' typecheck +grep -rn 'OVERLAY_IMPLICIT\|SCHEMA_NOT_OWNED\|BREAKING_CHANGE\|DEPENDENCY_MODES\|LOCAL_OVERRIDE_FILE\|deps.local.json' server fixtures --include=*.ts --include=*.py --include=*.json --include=*.md | grep -v superpowers +``` +Expected: tests PASS; typecheck PASS; the grep prints NOTHING (the corpus README and runner included). The one-case corpus (`no-dependencies-resolves-exactly-as-before`) fails on `collection.imported` not existing — expected until Task 8. + +- [ ] **Step 4: Commit** +``` +refactor(deps): retire `mode`, the local override and three superseded error codes (FR-023 §11) +``` + +--- + +### Task 7: Lock and manifest schemas, integrity hashing, and the resolved-dependency type (TS + Python) **Files:** - Modify: `server/typescript/packages/sdk/src/dependencies.ts` - Modify: `server/python/src/metaobjects/config/dependencies.py` -- Test: `server/typescript/packages/sdk/test/dependencies.test.ts`, `server/python/tests/config/test_dependencies.py` +- Test: `server/typescript/packages/sdk/test/dependencies.test.ts` (new), `server/python/tests/config/test_dependencies.py` (new) -**Read first:** DESIGN §3.2, §3.3, §3.4, §4.2 steps 2–3. +**Read first:** DESIGN §3.2, §3.3 (minus `mode`), §11.2 row "`.metaobjects/config.json`". The pinned hashes in the corpus README. `sdk/src/config.ts` for the `.strict()`-everywhere convention. **Interfaces:** -- Produces TS: `LockSchema` / `type Lock`, `ManifestSchema` / `type Manifest`, `LocalOverrideSchema` / `type LocalOverrides` (`Record`), `sha256Integrity(bytes: Uint8Array): string`, `dependencySourceId(name: string, artifact: string): string` (`dep:/`), `readLock(configDir: string): Promise`, `writeLock(configDir: string, lock: Lock): Promise` (keys sorted, 2-space, trailing newline), `readLocalOverrides(configDir: string): Promise` (`{}` when absent), `interface ResolvedDependency { name; version; metamodelVersion; mode; packages: readonly string[]; nodes: readonly string[]; artifactPath: string; sourceId: string; override?: string }`. -- Produces Python: `sha256_integrity(b: bytes) -> str`, `dependency_source_id(name, artifact)`, `read_lock(config_dir) -> dict | None`, `read_local_overrides(config_dir) -> dict`, `ResolvedDependency` dataclass with the same fields, `validate_lock(raw) -> dict` and `validate_manifest(raw) -> dict` raising `ParseError(ERR_DEPENDENCY_SNAPSHOT_STALE)` / `ParseError(ERR_DEPENDENCY_MANIFEST_INVALID)` on shape errors. +- Produces TS: `IntegritySchema` (`/^sha256-[0-9a-f]{64}$/`); `ManifestSchema` (`schema_version: z.literal(1)`, `name: DependencyName`, `version: z.string().min(1)`, `metamodelVersion: z.string().regex(/^\d+\.\d+$/)`, `artifact: z.string().endsWith(ARTIFACT_SUFFIX)`, `integrity`, `packages` / `nodes` as `z.array(z.string().min(1)).refine(sorted)`); `LockEntrySchema` = the manifest fields minus `schema_version`/`name` plus `resolvedFrom` (a strict union `{ path } | { npm, dir? } | { python, dir? }`); `LockSchema = { schema_version: 1, dependencies: z.record(DependencyName, LockEntrySchema) }.refine(keys sorted, { message: "deps.lock.json: dependency keys must be sorted" })`; `type Manifest`, `type Lock`, `type LockEntry`; `type ResolvedDependency = { name; version; packages: readonly string[]; nodes: readonly string[]; artifactPath: string; sourceId: string }`; `sha256Integrity(bytes: Uint8Array | string): string`; `readLock(configDir): Promise`; `writeLock(configDir, lock): Promise` (sorted keys, 2-space indent, trailing newline); `dependencySourceId(name, artifact)` = `` `${DEPENDENCY_SOURCE_ID_PREFIX}${name}/${artifact}` ``. +- Produces Python: `validate_manifest(obj) -> dict`, `validate_lock(obj) -> dict`, `sha256_integrity(data: bytes) -> str`, `read_lock(config_dir: Path) -> dict | None`, `dependency_source_id(name, artifact)`; `ResolvedDependency` dataclass (same fields). Hand-validated dicts with the same rules; a violation raises `ParseError(code=ErrorCode.ERR_DEPENDENCY_MANIFEST_INVALID)` for manifest shape and `ERR_DEPENDENCY_SNAPSHOT_STALE` for lock shape. -- [ ] **Step 1: Failing TS tests** `sdk/test/dependencies.test.ts`: +- [ ] **Step 1: Failing TS tests** (`sdk/test/dependencies.test.ts`): ```ts -import { describe, expect, test } from "bun:test"; import { readFile } from "node:fs/promises"; -import { LockSchema, ManifestSchema, sha256Integrity, dependencySourceId, INTEGRITY_PREFIX } from "../src/dependencies.js"; - -const ARTIFACT = `${import.meta.dir}/../../../../../fixtures/dependency-conformance/artifacts/acme-common-v1.json`; - -describe("dependency lock + manifest", () => { - test("integrity hashes the pinned artifact to the README value", async () => { - expect(sha256Integrity(new Uint8Array(await readFile(ARTIFACT)))).toBe("sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d"); - expect(INTEGRITY_PREFIX).toBe("sha256-"); - }); - test("source id", () => { expect(dependencySourceId("acme-common", "acme-common.metaobjects.json")).toBe("dep:acme-common/acme-common.metaobjects.json"); }); +import { resolve } from "node:path"; +import { LockSchema, ManifestSchema, sha256Integrity } from "../src/dependencies.js"; +const CORPUS = resolve(import.meta.dir, "../../../../../fixtures/dependency-conformance/artifacts"); +test("sha256Integrity of the pinned v1 artifact equals the README value", async () => { + expect(sha256Integrity(await readFile(`${CORPUS}/acme-common-v1.json`))).toBe("sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d"); +}); +test("manifest and lock schemas: sorted arrays, sorted keys, one transport", () => { const manifest = { schema_version: 1, name: "acme-common", version: "1.0.0", metamodelVersion: "1.0", artifact: "acme-common.metaobjects.json", integrity: "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", packages: ["acme::common"], nodes: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"] }; - test("manifest parses; an unknown key, an unsorted nodes list, a bad integrity prefix are refused", () => { - expect(() => ManifestSchema.parse(manifest)).not.toThrow(); - expect(() => ManifestSchema.parse({ ...manifest, extra: 1 })).toThrow(); - expect(() => ManifestSchema.parse({ ...manifest, nodes: ["acme::common::Customer", "acme::common::Address"] })).toThrow(); - expect(() => ManifestSchema.parse({ ...manifest, integrity: "md5-abc" })).toThrow(); - }); - test("lock parses; keys must be sorted; resolvedFrom carries exactly one transport", () => { - const entry = { version: "1.0.0", metamodelVersion: "1.0", mode: "reference", resolvedFrom: { path: "../lib" }, artifact: "acme-common.metaobjects.json", - integrity: manifest.integrity, packages: ["acme::common"], nodes: manifest.nodes }; - expect(() => LockSchema.parse({ schema_version: 1, dependencies: { "acme-common": entry } })).not.toThrow(); - expect(() => LockSchema.parse({ schema_version: 1, dependencies: { "b": entry, "a": entry } })).toThrow(/sorted/); - expect(() => LockSchema.parse({ schema_version: 1, dependencies: { "a": { ...entry, resolvedFrom: { path: "x", npm: "y" } } } })).toThrow(); - }); + expect(() => ManifestSchema.parse(manifest)).not.toThrow(); + expect(() => ManifestSchema.parse({ ...manifest, nodes: ["b", "a"] })).toThrow(/sorted/); + expect(() => ManifestSchema.parse({ ...manifest, mode: "reference" })).toThrow(); + const { schema_version: _s, name: _n, ...rest } = manifest; + const entry = { ...rest, resolvedFrom: { path: "x" } }; + expect(() => LockSchema.parse({ schema_version: 1, dependencies: { "acme-common": entry } })).not.toThrow(); + expect(() => LockSchema.parse({ schema_version: 1, dependencies: { b: entry, a: entry } })).toThrow(/sorted/); + expect(() => LockSchema.parse({ schema_version: 1, dependencies: { a: { ...entry, resolvedFrom: { path: "x", npm: "y" } } } })).toThrow(); + expect(() => LockSchema.parse({ schema_version: 1, dependencies: { a: { ...entry, mode: "own" } } })).toThrow(); }); ``` -Run `cd server/typescript/packages/sdk && bun test test/dependencies.test.ts` → FAIL. +Run `cd server/typescript/packages/sdk && bun test test/dependencies.test.ts` → FAIL (module has no such exports). -- [ ] **Step 2: Implement** with zod (`.strict()` everywhere): `IntegritySchema = z.string().regex(/^sha256-[0-9a-f]{64}$/)`; `ManifestSchema` with `schema_version: z.literal(1)`, `name: DependencyName`, `version: z.string().min(1)`, `metamodelVersion: z.string().regex(/^\d+\.\d+$/)`, `artifact: z.string().endsWith(ARTIFACT_SUFFIX)`, `integrity`, `packages`/`nodes` as `z.array(z.string().min(1)).refine(sorted)`; `LockEntrySchema` = the manifest fields minus `schema_version`/`name` plus `mode` and `resolvedFrom` (a strict union `{path}|{npm,dir?}|{python,dir?}`); `LockSchema = { schema_version: 1, dependencies: z.record(DependencyName, LockEntrySchema) }.refine(keys sorted, { message: "deps.lock.json: dependency keys must be sorted" })`; `LocalOverrideSchema = z.record(DependencyName, z.object({ path: z.string().min(1) }).strict())`; `sha256Integrity` via `createHash("sha256")`; `readLock`/`writeLock`/`readLocalOverrides` over `join(configDir, DEFAULT_METAOBJECTS_DIR, LOCK_FILE | LOCAL_OVERRIDE_FILE)` (`DEFAULT_METAOBJECTS_DIR` from `./metadata-files.js`). Run → PASS; typecheck → PASS. +- [ ] **Step 2: Implement** with zod (`.strict()` everywhere) and `createHash("sha256")` from `node:crypto`; `readLock`/`writeLock` over `join(configDir, DEFAULT_METAOBJECTS_DIR, LOCK_FILE)` (`DEFAULT_METAOBJECTS_DIR` from `./metadata-files.js`). Run → PASS; `bun run --filter '@metaobjectsdev/sdk' typecheck` → PASS. -- [ ] **Step 3: Failing Python tests** `tests/config/test_dependencies.py`: -```python -from pathlib import Path -import pytest -from metaobjects.config.dependencies import (sha256_integrity, dependency_source_id, validate_manifest, validate_lock, INTEGRITY_PREFIX) -from metaobjects.errors import ErrorCode, ParseError -ARTIFACT = Path(__file__).resolve().parents[3] / "fixtures" / "dependency-conformance" / "artifacts" / "acme-common-v1.json" -MANIFEST = {"schema_version": 1, "name": "acme-common", "version": "1.0.0", "metamodelVersion": "1.0", "artifact": "acme-common.metaobjects.json", - "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", "packages": ["acme::common"], - "nodes": ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"]} +- [ ] **Step 3: Failing Python tests** (`tests/config/test_dependencies.py`): the same three assertions (`sha256_integrity` of the pinned file; `validate_manifest` accepting the manifest and rejecting unsorted `nodes` and a `mode` key; `validate_lock` rejecting unsorted keys, two transports, and `mode`). Run → FAIL. Implement (`hashlib.sha256`). Run → PASS. -def test_integrity_matches_pinned_value(): - assert sha256_integrity(ARTIFACT.read_bytes()) == MANIFEST["integrity"]; assert INTEGRITY_PREFIX == "sha256-" - -def test_source_id(): - assert dependency_source_id("acme-common", "acme-common.metaobjects.json") == "dep:acme-common/acme-common.metaobjects.json" - -def test_manifest_shape_errors(): - validate_manifest(MANIFEST) - for bad in [{**MANIFEST, "extra": 1}, {**MANIFEST, "nodes": list(reversed(MANIFEST["nodes"]))}, {**MANIFEST, "integrity": "md5-x"}]: - with pytest.raises(ParseError) as e: validate_manifest(bad) - assert e.value.code == ErrorCode.ERR_DEPENDENCY_MANIFEST_INVALID - -def test_lock_shape_errors(): - entry = {k: v for k, v in MANIFEST.items() if k not in ("schema_version", "name")} | {"mode": "reference", "resolvedFrom": {"path": "../lib"}} - validate_lock({"schema_version": 1, "dependencies": {"acme-common": entry}}) - with pytest.raises(ParseError) as e: validate_lock({"schema_version": 1, "dependencies": {"b": entry, "a": entry}}) - assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE +- [ ] **Step 4: Run** +```bash +cd server/typescript/packages/sdk && bun test test/dependencies.test.ts +cd server/python && uv run --extra integration pytest tests/config/test_dependencies.py -q ``` -Run → FAIL. Implement the module (hand-validated dicts, same rules as the zod schemas; `hashlib.sha256`). Run → PASS. +Expected: PASS. -- [ ] **Step 4: Commit** +- [ ] **Step 5: Commit** ``` -feat(deps): lock, manifest and override schemas with sha256 integrity (FR-023) +feat(deps): lock and manifest schemas with sha256 integrity (FR-023) ``` --- -### Task 7: `resolveCollection` reads the lock and snapshot; `loadMemory` takes `fileIds`; every CLI load site passes both +### Task 8: `resolveCollection` reads the lock and snapshot, composes the scope predicates, and every CLI load site passes `fileIds` **Files:** -- Modify: `server/typescript/packages/sdk/src/dependencies.ts` (`verifySnapshot`, `foreignOwnerFactory`) +- Modify: `server/typescript/packages/sdk/src/dependencies.ts` (`verifySnapshot`, `importedPackagesOf(deps)`, `importedNodesOf(deps)`, `explicitlyIncludes(patterns, pkg)`) - Modify: `server/typescript/packages/sdk/src/collection.ts` - Modify: `server/typescript/packages/sdk/src/sources.ts` (`ResolvedSource` gains optional `dependency?: string`, `id?: string`) - Modify: `server/typescript/packages/sdk/src/memory.ts` (`LoadMemoryOptions.fileIds`) - Create: `server/typescript/packages/cli/src/lib/collection-load-options.ts` -- Modify: every `loadMemory(` call in `server/typescript/packages/cli/src/commands/{gen,verify,migrate,docs,export,prompt-snapshot,upgrade,types}.ts` and `cli/src/lib/*.ts` to spread `...collectionLoadOptions(collection)` instead of `files: collection.files` -- Modify: `fixtures/dependency-conformance/cases.json` (append cases 2, 3, 20–27 below) -- Test: `sdk/test/dependency-conformance.test.ts` (from Task 1), `sdk/test/collection.test.ts`, `sdk/test/order-independence.test.ts`, `sdk/test/memory.test.ts` +- Modify: every `loadMemory(` call in `server/typescript/packages/cli/src/commands/{gen,verify,migrate,docs,prompt-snapshot}.ts` (and any under `cli/src/lib/`) to spread `...collectionLoadOptions(collection)` instead of `files: collection.files` +- Modify: `fixtures/dependency-conformance/cases.json` (append the cases below) +- Test: `sdk/test/dependency-conformance.test.ts` (Task 6's runner), `sdk/test/collection.test.ts`, `sdk/test/order-independence.test.ts`, `sdk/test/memory.test.ts`, `sdk/test/dependencies.test.ts` (extend: `explicitlyIncludes`) -**Read first:** DESIGN §4.2, §2.3 ("load-time checks"), §3.3, §3.4. `collection.ts` — `resolveCollection` is THE authority; extend it after the own-source resolution. Do NOT implement overrides yet (Task 17) — `readLocalOverrides` is called but an active override is treated as an error `ERR_DEPENDENCY_UNRESOLVED` with message `local overrides are not supported yet` until Task 17 replaces that branch (write that branch so Task 17's diff is a replacement, not a rewrite). +**Read first:** DESIGN §4.2, §2.3 ("load-time checks"), §11.1 item 2 (the rule), Global Constraints "The default-exclusion rule". `collection.ts` — `resolveCollection` is THE authority; extend it after the own-source resolution. `metadata/src/naming.ts` `packageOfResolutionKey` (Task 2). **Interfaces:** -- Produces: `Collection` gains `dependencies: readonly ResolvedDependency[]`, `ownFiles: readonly string[]`, `fileIds: ReadonlyMap` (entries for dependency artifacts only), `overrides: readonly string[]` (empty until Task 17), `foreignOwner(fqn: string): ResolvedDependency | undefined`, `governs(fqn: string): boolean`. `files` = artifacts (name order) then own files. `verifySnapshot(configDir, specs, lock, overrides): Promise` throws `ParseError` codes per the table below. `loadMemory(root, { files, fileIds })` builds `new FileSource(p, { id: fileIds.get(p) })`. `collectionLoadOptions(collection)` returns `{ files: collection.files, fileIds: collection.fileIds }`. - -**Load-time check table (`verifySnapshot`):** config declares dependencies but no lock → `ERR_DEPENDENCY_SNAPSHOT_STALE`; a lock entry with no config spec, or a spec with no lock entry → `ERR_DEPENDENCY_SNAPSHOT_STALE`; lock `mode` ≠ spec `mode` → `ERR_DEPENDENCY_SNAPSHOT_STALE`; artifact file missing → `ERR_DEPENDENCY_SNAPSHOT_STALE`; `sha256Integrity(bytes) !== entry.integrity` → `ERR_DEPENDENCY_SNAPSHOT_STALE`; lock `metamodelVersion` major ≠ `METAMODEL_VERSION` major → `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`; two entries whose `nodes` intersect → `ERR_DEPENDENCY_NODE_COLLISION`. Every message ends with `run \`meta deps sync\`` for the stale family. A config with `dependencies: []` and no lock is fine; a lock with zero entries and no dependencies is fine. - -- [ ] **Step 1: Append the corpus cases.** The lock entry used below is (`LOCK_V1`): -```json -{ "schema_version": 1, "dependencies": { "acme-common": { "version": "1.0.0", "metamodelVersion": "1.0", "mode": "reference", "resolvedFrom": { "path": "../acme-common/metaobjects" }, "artifact": "acme-common.metaobjects.json", "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", "packages": ["acme::common"], "nodes": ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"] } } } -``` -the config `CONFIG_REF` = `{ "schema_version": 1, "sources": [], "dependencies": [{ "name": "acme-common", "path": "../acme-common/metaobjects", "mode": "reference" }] }`, the snapshot entry `SNAP` = `"treeFiles": { ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" }`, and the app file `APP` = `"metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}"`. (The `path` transport need not exist on disk — only `meta deps sync` resolves transports.) Add: - - `a-dependency-adds-its-artifact-to-the-resolved-set`: tree `APP` + `SNAP`, `CONFIG_REF`, `LOCK_V1`; `expectFiles: [".metaobjects/deps/acme-common/acme-common.metaobjects.json", "metaobjects/meta.app.json"]`, `expectForeign: ["acme::common::Address","acme::common::Audited","acme::common::Customer"]`, `expectGoverned: ["app::Order"]`. - - `dependency-order-in-config-does-not-change-the-resolved-set`: two dependencies `acme-common` and `acme-extra` (a second artifact: put in `tree` the string of `acme-common-v1.json` with every `acme::common` replaced by `acme::extra`, at `.metaobjects/deps/acme-extra/acme-extra.metaobjects.json`; compute its hash with the README one-liner and pin it in this case's lock — record the value in the case as `"integrity"`), declared `acme-extra` first in the config; `expectFiles` = both artifacts + `APP`; `expectForeign` = all six FQNs. +- Produces sdk: `Collection` gains `dependencies: readonly ResolvedDependency[]`, `ownFiles: readonly string[]`, `fileIds: ReadonlyMap` (entries for artifacts only), `importedPackages: readonly string[]` (sorted union of lock `packages`), `importedNodes: ReadonlySet` (union of lock `nodes` — for the ownership refusal only), `imported(fqn: string): boolean` (TRUE iff `packageOf(fqn)` is in `importedPackages`), `declaredMigrateScope: ((fqn) => boolean) | undefined` (the user's `migrate.scope` alone; lockstep with `migrateScopePatterns`); `inScope` and `inMigrateScope` become the COMPOSED predicates per the Global Constraints (`inMigrateScope` is `undefined` iff no `migrate.scope` AND `dependencies.length === 0`). `files` = artifacts (name order) then own files. `verifySnapshot(configDir, specs, lock): Promise` throws per the table below. `explicitlyIncludes(patterns: readonly string[] | undefined, pkg: string): boolean` — pure, exported, unit-tested. `loadMemory(root, { files, fileIds })` builds `new FileSource(p, { id: fileIds.get(p) })` (`FileSource` from `@metaobjectsdev/metadata/core`). `collectionLoadOptions(collection)` returns `{ files: collection.files, fileIds: collection.fileIds }`. +- **`verifySnapshot` table:** config declares dependencies but no lock → `ERR_DEPENDENCY_SNAPSHOT_STALE`; a lock entry with no spec, or a spec with no entry → `ERR_DEPENDENCY_SNAPSHOT_STALE`; artifact missing → `ERR_DEPENDENCY_SNAPSHOT_STALE`; `sha256Integrity(bytes) !== entry.integrity` → `ERR_DEPENDENCY_SNAPSHOT_STALE`; lock `metamodelVersion` major ≠ `METAMODEL_VERSION` major → `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`; two entries whose `nodes` intersect → `ERR_DEPENDENCY_NODE_COLLISION`. Every stale message ends with `` run `meta deps sync` ``. `dependencies: []` with no lock is fine; a lock with zero entries and no dependencies is fine. + +- [ ] **Step 1: Append the corpus cases** (constants from the Global Constraints): + - `a-dependency-adds-its-artifact-to-the-resolved-set`: `APP` + `SNAP`, `CONFIG_REF`, `LOCK_V1`; `expectFiles: [".metaobjects/deps/acme-common/acme-common.metaobjects.json", "metaobjects/meta.app.json"]`; `expectImported: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"]`; `expectSelected: ["app::Order"]`; `expectMigrateGoverned: ["app::Order"]`. + - `dependency-order-in-config-does-not-change-the-resolved-set`: two dependencies `acme-common` and `acme-extra` (a second artifact: the string of `acme-common-v1.json` with every `acme::common` replaced by `acme::extra`, placed in `tree` at `.metaobjects/deps/acme-extra/acme-extra.metaobjects.json`; compute its hash with the README one-liner and pin it in this case's lock as `integrity`), `acme-extra` declared FIRST in the config; `expectFiles` = both artifacts + `APP`; `expectImported` = all six FQNs; `expectSelected: ["app::Order"]`. + - `a-scope-include-naming-the-package-selects-its-nodes`: `APP` + `SNAP`, config `CONFIG_REF` + `"scope": { "include": ["app::**", "acme::common::**"] }`, `LOCK_V1`; `expectSelected: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer", "app::Order"]`; `expectMigrateGoverned: ["app::Order"]` (no `migrate.scope` → schema still excludes the import). + - `a-wildcard-include-does-not-name-a-package`: same with `"scope": { "include": ["**"] }`; `expectSelected: ["app::Order"]`. + - `a-migrate-scope-naming-the-package-governs-its-tables`: `APP` + `SNAP`, `CONFIG_REF` + `"migrate": { "scope": ["app::**", "acme::common::**"] }`, `LOCK_V1`; `expectMigrateGoverned: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer", "app::Order"]`; `expectSelected: ["app::Order"]`. + - `a-local-node-in-a-dependency-package-is-refused`: `SNAP`, `CONFIG_REF`, `LOCK_V1`, plus a local file `metaobjects/meta.ext.json` declaring `{"metadata.root":{"package":"acme::common","children":[{"object.value":{"name":"Note","children":[{"field.string":{"name":"text"}}]}}]}}` — package `acme::common` is the dependency's and `acme::common::Note` is not in its `nodes` → `expectLoadError: "ERR_DEPENDENCY_PACKAGE_NOT_OWNED"` (rides with `expectFiles`, which is the artifact + the local file). (Ruled 2026-09-11: under the package-keyed rule this node would otherwise be silently excluded from its own project's codegen.) + - `an-overlay-of-a-dependency-node-is-not-refused`: same, but the local file declares `{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"email","overlay":true,"children":[{"view.text":{"name":"emailView"}}]}}]}}` in package `acme::common` — the FQN IS in `nodes`, so it merges: NO error, `expectImported` = the three, `expectSelected: []`. - `a-missing-lock-is-stale`: `APP` + `SNAP`, `CONFIG_REF`, no lock → `expectError: "ERR_DEPENDENCY_SNAPSHOT_STALE"`. - `a-lock-entry-without-a-config-entry-is-stale`: `APP` + `SNAP`, config `{ "schema_version": 1, "sources": [] }`, `LOCK_V1` → `ERR_DEPENDENCY_SNAPSHOT_STALE`. - `a-config-entry-without-a-lock-entry-is-stale`: `APP`, `CONFIG_REF`, lock `{ "schema_version": 1, "dependencies": {} }` → `ERR_DEPENDENCY_SNAPSHOT_STALE`. - `a-missing-artifact-is-stale`: `APP` (no `SNAP`), `CONFIG_REF`, `LOCK_V1` → `ERR_DEPENDENCY_SNAPSHOT_STALE`. - `an-artifact-whose-hash-differs-is-stale`: `APP` + `treeFiles` pointing the snapshot path at `artifacts/acme-common-v1-widened.json`, `CONFIG_REF`, `LOCK_V1` → `ERR_DEPENDENCY_SNAPSHOT_STALE`. - - `two-dependencies-exporting-one-fqn-collide`: two dependencies whose artifacts are byte-identical copies of `acme-common-v1.json` (`acme-common` and `acme-dup`, both `treeFiles` → `artifacts/acme-common-v1.json`, both lock entries with the v1 hash and the same `nodes`) → `ERR_DEPENDENCY_NODE_COLLISION`. + - `two-dependencies-exporting-one-fqn-collide`: `acme-common` and `acme-dup`, both `treeFiles` → `artifacts/acme-common-v1.json`, both lock entries with the v1 hash and the same `nodes` → `ERR_DEPENDENCY_NODE_COLLISION`. - `an-incompatible-metamodel-major-is-refused`: `LOCK_V1` with `"metamodelVersion": "2.0"` → `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`. - - `dependencies-are-read-under-a-native-source-surface`: `resolveFrom: "app"`, tree `app/metaobjects/meta.app.json` (APP content) + `app/.metaobjects/deps/acme-common/…` via `treeFiles`, config `CONFIG_REF` under `app/`, `LOCK_V1` → `expectFiles: ["app/.metaobjects/deps/acme-common/acme-common.metaobjects.json", "app/metaobjects/meta.app.json"]`. (The Python runner in Task 18 additionally writes a `metaobjects.config.yaml` with `metadata: metaobjects` beside it to prove the rung-2 path still reads dependencies.) + - `dependencies-are-read-under-a-native-source-surface`: `resolveFrom: "app"`, tree `app/metaobjects/meta.app.json` (APP content) + `app/.metaobjects/deps/acme-common/…` via `treeFiles`, config `CONFIG_REF` under `app/`, `LOCK_V1` → `expectFiles: ["app/.metaobjects/deps/acme-common/acme-common.metaobjects.json", "app/metaobjects/meta.app.json"]`. (Task 18's Python runner additionally writes `metaobjects.config.yaml` with `metadata: metaobjects` beside it.) -- [ ] **Step 2: Run the corpus** → the new cases FAIL (`collection.dependencies` undefined / no stale detection). +- [ ] **Step 2: Run the corpus** → the new cases FAIL (`collection.imported` undefined / no stale detection). -- [ ] **Step 3: Failing unit tests.** Append to `sdk/test/memory.test.ts`: `loadMemory` with `fileIds` yields a node whose `source.files[0]` is the mapped id (materialize a temp project whose only file is mapped to `"dep:x/x.metaobjects.json"`). Append to `order-independence.test.ts`: permuting `dependencies` in the config leaves `collection.files` and `collection.fileIds` identical (two dependencies as in the corpus case). Run → FAIL. +- [ ] **Step 3: Failing unit tests.** `sdk/test/dependencies.test.ts`: `explicitlyIncludes(["acme::common::**"], "acme::common")` true; `(["acme::common::Address"], "acme::common")` true; `(["acme::**"], "acme::common")` false; `(["**"], …)` false; `(undefined, …)` false; `([], …)` false; `(["acme::common::sub::**"], "acme::common")` false. `sdk/test/memory.test.ts`: `loadMemory` with `fileIds` yields a node whose `source.files[0]` is the mapped id. `order-independence.test.ts`: permuting `dependencies` in the config leaves `collection.files` and `collection.fileIds` identical. `collection.test.ts`: a project with no dependencies has `inMigrateScope === undefined` when it declares no `migrate.scope` (byte-identical path), and `inScope` identical to today's for three FQNs. Run → FAIL. -- [ ] **Step 4: Implement.** `verifySnapshot` per the table (read `.metaobjects/deps//` bytes; compare; build `ResolvedDependency` with `artifactPath` absolute and `sourceId = dependencySourceId(name, artifact)`); `foreignOwnerFactory(deps)` builds a `Map` once. In `resolveCollection`: after `sources` are resolved, `const lock = await readLock(configDir); const overrides = await readLocalOverrides(configDir); const deps = specs.length === 0 && lock === undefined ? [] : await verifySnapshot(configDir, cfg.dependencies, lock, overrides)`; `files = [...deps.map(d => d.artifactPath), ...own]`; `fileIds = new Map(deps.map(d => [d.artifactPath, d.sourceId]))`; `foreignOwner`, `governs`, `ownFiles`, `overrides: []`. A project with no config carries `dependencies: []`. `loadMemory`: `paths.map((p) => new FileSource(p, fileIds?.has(p) ? { id: fileIds.get(p) } : undefined))` — note `FileSource` is imported from `@metaobjectsdev/metadata/core`. Create `collectionLoadOptions` and sweep every `loadMemory(` call in `cli/src` (grep `loadMemory(`): each becomes `loadMemory(dir, { ...collectionLoadOptions(collection), ...loadMemoryOptionsFrom(cfg) })`. The requirements line in `verify.ts` (`counted over N metadata file(s)`) gains `, M from dependencies` where `M = collection.dependencies.length`. +- [ ] **Step 4: Implement.** `verifySnapshot` per the table (reads `.metaobjects/deps//`; builds `ResolvedDependency` with `artifactPath` absolute and `sourceId = dependencySourceId(name, artifact)`); `importedPackagesOf(deps)` builds a `Set` of every lock entry's `packages` (the exclusion key); `importedNodesOf(deps)` a `Set` of every `nodes` entry, read ONLY by the ownership refusal. In `resolveCollection`, after `sources`: `const lock = await readLock(configDir); const deps = cfg.dependencies.length === 0 && lock === undefined ? [] : await verifySnapshot(configDir, cfg.dependencies, lock)`; `files = [...deps.map(d => d.artifactPath), ...own]`; `fileIds`; `imported = (fqn) => importedPackages.has(packageOfResolutionKey(fqn))`; the composed `inScope` = `matchesScope(fqn, scope) && (!imported(fqn) || explicitlyIncludes(scopeSpec?.include, packageOfResolutionKey(fqn)))`; `declaredMigrateScope` = today's `inMigrateScope`; `inMigrateScope` = `undefined` when `migrateSpec === undefined && deps.length === 0`, else `(fqn) => (declaredMigrateScope?.(fqn) ?? true) && (!imported(fqn) || explicitlyIncludes(migrateSpec, packageOfResolutionKey(fqn)))`. `loadMemory`: `paths.map((p) => new FileSource(p, fileIds?.has(p) ? { id: fileIds.get(p) } : undefined))`; `LoadMemoryOptions` also takes `importedPackages?` / `importedNodes?`, and AFTER the load `loadMemory` walks the root's top-level objects (ADR-0039 sanctioned own: root-level scan) and throws `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` for the first whose package is in `importedPackages` and whose resolution key is not in `importedNodes` — so every CLI load site swept below gets the refusal for free, and the corpus's `expectLoadError` arm exercises it in both ports. Create `collectionLoadOptions` and sweep every `loadMemory(` in `cli/src` (grep `loadMemory(`). The requirements line in `verify.ts` (`counted over N metadata file(s)`) gains `, M from dependencies` where `M = collection.dependencies.length`. - [ ] **Step 5: Run** ```bash -cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts test/collection.test.ts test/order-independence.test.ts test/memory.test.ts test/no-hardcoded-metadata-dir.test.ts test/source-resolution-conformance.test.ts -cd server/typescript/packages/cli && bun test test/collection-routing.test.ts test/verify-requirements-e2e.test.ts test/gen-list.test.ts +cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts test/dependencies.test.ts test/collection.test.ts test/order-independence.test.ts test/memory.test.ts test/no-hardcoded-metadata-dir.test.ts test/source-resolution-conformance.test.ts +cd server/typescript/packages/cli && bun test test/collection-routing.test.ts test/verify-requirements-e2e.test.ts test/gen-list.test.ts test/migrate-scope.test.ts bun run --filter '@metaobjectsdev/sdk' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck ``` -Expected: PASS (cases 1, 2, 3, 20–27 green; the rest still red/skipped). +Expected: PASS — every corpus case appended so far green; `migrate-scope.test.ts` unchanged (no dependencies ⇒ `inMigrateScope` is exactly what it was). - [ ] **Step 6: Commit** ``` -feat(sdk): resolveCollection loads the dependency snapshot behind the lock; loadMemory takes fileIds (FR-023) +feat(sdk): resolveCollection loads dependency snapshots first and composes the default-exclusion scope (FR-023 §11) ``` --- -### Task 8: The boundary validator and `declaredOverlayKeys` +### Task 9: The load-time failure corpus cases (no new machinery) **Files:** -- Modify: `server/typescript/packages/metadata/src/loader/meta-data-loader.ts` (export `declaredOverlayKeys`) -- Modify: `server/typescript/packages/sdk/src/dependencies.ts` (`validateDependencyBoundary`) -- Modify: `server/typescript/packages/cli/src/lib/collection-load-options.ts` (add `loadCollection(dir, collection, extra)` = `loadMemory` + boundary validation, used by every command) -- Modify: the eight CLI load sites to call `loadCollection` (from Task 7's sweep) -- Modify: `fixtures/dependency-conformance/cases.json` (append cases 8, 10–14) -- Test: `sdk/test/dependency-conformance.test.ts`, `sdk/test/dependencies.test.ts`, `metadata/test/declared-overlay-keys.test.ts` - -**Read first:** DESIGN §2.4 (both rules), §4.4. In `meta-data-loader.ts`, `_partitionOverlayLast` / `_isOverlayOnlySource` / `_rootIsOverlayOnly` already parse a source's raw content and inspect top-level `overlay: true` — reuse that walker. TPH: a subtype declares `@discriminatorValue`; its discriminator base is the nearest `extends` ancestor carrying `@discriminator` (see `migrate-ts/src/expected-schema.ts` `discriminatorBaseOf`; re-implement the two-line walk in sdk with resolving accessors — do not import migrate-ts). - -**Interfaces:** -- Produces TS: `declaredOverlayKeys(content: string, format: MetaDataFormat, fileDefaultPackage?: string): Promise>` — resolution keys (`::`, using the node's own `package` else the document root's) of top-level nodes carrying `overlay: true`. `validateDependencyBoundary(root: MetaRoot, collection: Collection, overlayKeys: ReadonlySet): void` — throws `ParseError` with the codes below. `loadCollection(startDir, collection, options)` in cli: loads, computes `overlayKeys` over `collection.ownFiles`, validates, returns the root. - -**Rules (DESIGN §4.4):** for each top-level node `n` (ADR-0039 sanctioned own: `root.ownChildren()` — the root is never extended) with `owner = collection.foreignOwner(n.resolutionKey())`: -1. `owner` defined, and `n` has a local contribution — `n.source.format === "merged"` with a contributor file not starting with `dep:`, or `n.source.files` contains a non-`dep:` entry — and `n.resolutionKey() ∉ overlayKeys` → `ERR_DEPENDENCY_OVERLAY_IMPLICIT`, message ` redeclares a node of dependency '' without 'overlay: true' — an amendment of a dependency's node must fail loudly when that node disappears; add 'overlay: true'`. -2. `owner.mode === "reference"` and any of: a child `c` of `n` (ADR-0039 sanctioned own: `n.ownChildren()` — the local contribution layer) whose `c.source.files` has a non-`dep:` entry and `c.type ∈ {field, identity, index, relationship, source}`; OR a locally-contributed attr among `@table`/`@schema`/`@column`/`@kind` (detect via `n.source.format === "merged"` and the attr being set — a merged node cannot say which contributor set an attr, so refuse when the node is merged with a local contributor AND carries any of those four attrs whose value differs from the artifact's: read the artifact's own copy by loading `collection.dependencies` artifact standalone once per validation, cached); OR `n` is local (no owner), declares `@discriminatorValue`, and its discriminator base's owner is `reference` → `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`, message `: a reference-mode dependency's node cannot gain here — declare your own object in your own package and extend it, or take mode "own" if you instantiate this model`. -Use `isMetaObject` from `@metaobjectsdev/metadata` and node `type` comparisons against `TYPE_FIELD`/`TYPE_IDENTITY`/`TYPE_INDEX`/`TYPE_RELATIONSHIP`/`TYPE_SOURCE` constants — never `instanceof`. +- Modify: `fixtures/dependency-conformance/cases.json` (append the six cases) +- Modify: `fixtures/dependency-conformance/README.md` (a table: upstream change → consumer construct → existing error code) +- Test: `sdk/test/dependency-conformance.test.ts` -- [ ] **Step 1: Failing metadata test** `declared-overlay-keys.test.ts`: a JSON document with root package `acme::common` and children `{ "object.entity": { name: "Customer", overlay: true } }` and `{ "object.value": { name: "Local" } }` → `declaredOverlayKeys(content, "json")` equals `new Set(["acme::common::Customer"])`; a YAML document (sigil-free `metadata:` root, `overlay: true` bare) gives the same; a node with its own `package: other` yields `other::Customer`. Run `cd server/typescript/packages/metadata && bun test test/declared-overlay-keys.test.ts` → FAIL. Implement by extracting the walk from `_rootIsOverlayOnly` into an exported function that returns keys; keep `_rootIsOverlayOnly` delegating to it. Run → PASS. +**Read first:** DESIGN §2.5 (the table) and §11.1 item 4. The v2 artifact `acme-common-v2-email-removed.json` (Customer without `email`) is "upstream after a change"; a case that uses it pins its hash (`sha256-c4ba…ce7a`) in its lock. These cases prove the loader's EXISTING errors are the first drift gate — the task adds no code. -- [ ] **Step 2: Append corpus cases** (all with `CONFIG_REF`, `LOCK_V1`, `SNAP` from Task 7 unless stated; `expectFiles` = the artifact + the app file; put the local file at `metaobjects/meta.app.json`): - - `a-local-node-may-live-in-the-dependency-package`: `{"metadata.root":{"package":"acme::common","children":[{"object.value":{"name":"LocalNote","children":[{"field.string":{"name":"text"}}]}}]}}` → `expectForeign` the three, `expectGoverned: ["acme::common::LocalNote"]`. - - `a-flagged-overlay-of-a-foreign-node-adds-presentation`: `{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"email","children":[{"view.text":{}}]}}]}}]}}` → loads clean; `expectGoverned: []`; `expectForeign` the three. (If `view.text` is not a registered subtype in this build, use `{"validator.length":{"@max":120}}` under the field instead — check `meta types view` first and use a registered presentation/validation child.) - - `an-unflagged-redeclaration-of-a-foreign-node-is-implicit`: same as above without `"overlay": true` → `expectFiles` + `expectLoadError: "ERR_DEPENDENCY_OVERLAY_IMPLICIT"`. - - `a-structural-overlay-of-a-reference-node-is-not-owned`: `{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"nickname"}}]}}]}}` → `expectLoadError: "ERR_DEPENDENCY_SCHEMA_NOT_OWNED"`. - - `a-structural-overlay-of-an-own-node-is-allowed`: same local file, config with `"mode": "own"` and `LOCK_V1` with `"mode": "own"` → loads clean; `expectGoverned: ["acme::common::Address","acme::common::Audited","acme::common::Customer"]`. - - `a-tph-subtype-of-a-foreign-reference-base-is-not-owned`: artifact-side base needs `@discriminator` — this case uses its OWN artifact in `tree` (not `treeFiles`): the v1 artifact text with `Customer`'s body gaining `"@discriminator": "kind"` and a child `{"field.enum":{"name":"kind","@values":["retail","corporate"]}}` (insert the attr after `"package"`, the child after `source.rdb`; compute and pin the hash in this case's lock); local file `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"CorporateCustomer","extends":"acme::common::Customer","@discriminatorValue":"corporate","children":[{"field.string":{"name":"vatId"}}]}}]}}` → `expectLoadError: "ERR_DEPENDENCY_SCHEMA_NOT_OWNED"`. +- [ ] **Step 1: Append** (each: `expectFiles` = the artifact + the local file, plus `expectLoadError`): + - `an-overlay-whose-target-was-removed-fails`: snapshot = v2 (email removed), lock with the v2 hash; local `metaobjects/meta.ov.json` = `{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"email","overlay":true,"children":[{"view.text":{"name":"emailView"}}]}}]}}]}}` → the OBJECT still exists; the FIELD overlay has no target → `expectLoadError: "ERR_OVERLAY_NO_TARGET"`. Add a sibling case where the local file overlays a whole object absent from the artifact (`"name": "Invoice", "overlay": true`) → `ERR_OVERLAY_NO_TARGET`. + - `an-extends-whose-target-was-removed-fails`: v1 snapshot; local `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"Vip","extends":"acme::common::Gone","children":[…a field…]}}]}}` → `ERR_UNRESOLVED_SUPER`. + - `a-reference-whose-target-was-removed-fails`: v1 snapshot; local entity with `{"field.object":{"name":"addr","@objectRef":"acme::common::Missing","@storage":"jsonb"}}` → `ERR_UNRESOLVED_OBJECT_REF`. + - `a-dotted-extends-whose-member-changed-subtype-fails`: v1 snapshot; local `{"field.int":{"name":"email","extends":"acme::common::Customer.email"}}` (v1's `email` is `field.string`) → `ERR_EXTENDS_TARGET_MISMATCH`. + - `an-overlay-attr-the-base-now-sets-differently-conflicts`: v1 snapshot; local overlay `{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"email","overlay":true,"@maxLength":80}}]}}` (base says 120) → `ERR_MERGE_CONFLICT`. + - `a-dependency-file-error-names-the-dependency`: a lock + snapshot whose artifact is malformed JSON (`tree` content `{"metadata.root":`; pin ITS hash in the lock so the snapshot check passes) → `expectLoadError` = the parse error code the loader raises for malformed JSON (read `parser.ts`; use that code), `expectErrorFiles: ["dep:acme-common/acme-common.metaobjects.json"]`. -- [ ] **Step 3: Run the corpus** → the six cases FAIL. Implement `validateDependencyBoundary` + `loadCollection` and the sweep. Run: -```bash -cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts test/dependencies.test.ts -cd server/typescript/packages/cli && bun test test/collection-routing.test.ts test/verify-requirements-e2e.test.ts -bun run --filter '@metaobjectsdev/sdk' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck -``` -Expected: PASS (cases 8, 10–14 green). +- [ ] **Step 2: Run** `cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts` → the six cases PASS with no source change. If one does not, the DESIGN §2.5 table is wrong for that row — fix the CASE to the loader's real code and record the correction in the README table (the loader is the authority; this task documents it, it does not change it). -- [ ] **Step 4: Commit** +- [ ] **Step 3: Commit** ``` -feat(sdk): dependency boundary validator — explicit overlays, reference-mode schema ownership (FR-023) +test(deps): the loader's existing errors are the first cross-repo drift gate (FR-023) ``` --- -**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** `validateDependencyBoundary` stays SYNCHRONOUS: the resolved-dependency record produced by the snapshot check (Task 6/7) carries the artifact text it already read to hash, and the validator parses the foreign subtree from that — no file I/O in the validator, no async ripple through the CLI load sites. Add the missing test: a `reference`-mode overlay whose local contribution sets a physical attribute (`@table`, `@schema`, `@column`, `@kind`) differing from the artifact → `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`. - -### Task 9: Codegen selection excludes foreign nodes (`gen`, `verify --codegen`, shared enums) +### Task 10: Codegen selection — `gen`, `verify --codegen`, shared enums, and refusal by name (TS) **Files:** -- Modify: `server/typescript/packages/cli/src/commands/gen.ts` (the `runGen({ scope })` call; the explicit-entity refusal) -- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (`computeCodegenDrift(..., genCollection.inScope)` → the composed predicate) -- Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (`RunGenOptions.governs?: (fqn: string) => boolean`; thread into `renderSharedEnumsFile`) -- Modify: `server/typescript/packages/codegen-ts/src/templates/enums-file.ts` (`renderSharedEnumsFile(root, opts?: { exclude?: (fqn: string) => boolean })`) -- Modify: `server/typescript/packages/codegen-ts/src/generators/entity-file.ts` (pass the exclusion) -- Modify: `fixtures/dependency-conformance/cases.json` (append cases 4–7, 9) -- Test: `cli/test/gen-foreign-nodes.test.ts` (new), `codegen-ts/test/shared-enums-foreign.test.ts` (new), `sdk/test/dependency-conformance.test.ts` +- Modify: `server/typescript/packages/cli/src/commands/gen.ts` (the pre-check refusing a positional that names an excluded import) +- Modify: `server/typescript/packages/codegen-ts/src/templates/enums-file.ts` (`renderSharedEnumsFile(root, opts?: { select?: (fqn: string) => boolean })`) +- Modify: `server/typescript/packages/codegen-ts/src/generators/entity-file.ts` (pass `select` from `ctx`) +- Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (`RunGenOptions.scope` is threaded into `GenContext` as `select` for the templates that render whole-model artifacts — read how `entityFile` reaches `ctx.loadedRoot` and add `ctx.select`) +- Modify: `server/typescript/packages/codegen-ts/src/generator.ts` (`GenContext.select?: (fqn: string) => boolean`) +- Test: `cli/test/gen-imported-nodes.test.ts` (new), `codegen-ts/test/shared-enums-imported.test.ts` (new) -**Read first:** DESIGN §2.7 (the table), §4.5. `runner.ts` "single choke point for entity selection" (`opts.scope`); `gen.ts` passes `scope: genCollection.inScope`; `renderSharedEnumsFile(ctx.loadedRoot)` reads the whole loaded root. +**Read first:** DESIGN §11.1 item 2 (codegen bullet; the shared-enum rule). `gen.ts` already passes `scope: genCollection.inScope` and `verify.ts` already passes `genCollection.inScope` to `computeCodegenDrift` — after Task 8 both are the COMPOSED predicate, so entity selection needs NO change here; this task covers the two things selection does not: the shared-enums artifact and the positional. `runner.ts:286-330` (`entityFilter ∩ scope`). `codegen-ts/src/templates/enums-file.ts` — `materializedSharedEnums(root)` walks the whole root. Pre-flight ruling T9: refusal is a NEW pre-check (exit 2); the existing warn-on-unmatched is unchanged. **Interfaces:** -- Consumes: `Collection.governs`, `Collection.foreignOwner` (Task 7), `loadCollection` (Task 8). -- Produces: `runGen({ scope, governs })` — the runner composes `scope(fqn) && governs(fqn)` at the choke point and passes `exclude: (fqn) => !governs(fqn)` to the shared-enums renderer; `gen` prints `meta gen — N node(s) from dependencies not generated here (reference mode): ` when N > 0 (stderr, once); `meta gen ` in reference mode exits 2 with ` is declared by dependency '' (reference mode) and is not generated here`. - -- [ ] **Step 1: Append corpus cases** (`CONFIG_REF`, `LOCK_V1`, `SNAP`): - - `a-foreign-node-is-referenceable-by-fqn-from-a-local-entity`: local `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"Order","children":[{"source.rdb":{"@table":"orders"}},{"field.long":{"name":"id"}},{"field.object":{"name":"shipTo","@objectRef":"acme::common::Address","@storage":"jsonb"}},{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}` → `expectGoverned: ["app::Order"]`, `expectForeign` the three. - - `a-foreign-node-is-not-selected-in-reference-mode`: `APP` → `expectGoverned: ["app::Order"]`. - - `a-foreign-node-is-selected-in-own-mode`: `APP`, config + lock with `mode: "own"` → `expectGoverned: ["acme::common::Address","acme::common::Audited","acme::common::Customer","app::Order"]`. - - `a-local-object-may-extend-a-foreign-abstract`: local `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"Ticket","extends":"acme::common::Audited","children":[{"source.rdb":{"@table":"tickets"}},{"field.long":{"name":"id"}},{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}` → `expectGoverned: ["app::Ticket"]`. - - `a-bare-reference-does-not-reach-a-foreign-node`: local `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"Order","children":[{"source.rdb":{"@table":"orders"}},{"field.long":{"name":"id"}},{"field.object":{"name":"shipTo","@objectRef":"Address","@storage":"jsonb"}},{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}` → `expectFiles` + `expectLoadError: "ERR_UNRESOLVED_OBJECT_REF"`. - Run the corpus → these pass already for the `expectGoverned` arm IF Task 7's `governs` is right (they exercise the predicate, not the CLI). Confirm they are green; if any is red, fix `governs` here. - -- [ ] **Step 2: Failing codegen-ts test** `shared-enums-foreign.test.ts`: load a root (via `MetaDataLoader.fromString`) with two package-level abstract `field.enum`s, `acme::common::Kind` (`@values: ["a","b"]`) consumed by a local entity's field via `extends`, and `acme::common::Unused`; assert `renderSharedEnumsFile(root)` names both, and `renderSharedEnumsFile(root, { exclude: (fqn) => fqn.startsWith("acme::common::") })` returns `null` (nothing materialized). Run `cd server/typescript/packages/codegen-ts && bun test test/shared-enums-foreign.test.ts` → FAIL. Implement the option in `enums-file.ts` (filter `materializedSharedEnums(root)` by `!exclude(e.resolutionKey())`) and thread `governs` from `RunGenOptions` through the runner into the `entityFile` generator's call. Run → PASS. +- Produces: `renderSharedEnumsFile(root, { select })` emits an abstract enum iff at least one entity with `select(entity.resolutionKey())` true has a field that resolves to it (`resolveSharedEnumDecl`-style, via the field's `extends` chain — read `materializedSharedEnums`); with no `select`, byte-identical to today. `gen.ts`: before `runGen`, for each `cliConfig.entities` name, find the loaded objects with that bare name; if EVERY match is `imported && !inScope` → `log.error("meta gen: '' () is imported from dependency '' and is not generated here — add its package to scope.include in .metaobjects/config.json to generate it")`, exit 2. -- [ ] **Step 3: Failing cli test** `gen-foreign-nodes.test.ts` — scaffold a temp consumer project exactly like corpus case `a-foreign-node-is-referenceable-by-fqn-from-a-local-entity` (copy the pinned artifact from `fixtures/dependency-conformance/artifacts/acme-common-v1.json`, write `CONFIG_REF` and `LOCK_V1`), plus a minimal `metaobjects.config.ts` with `entityFile()` from an owned generator (mirror how `cli/test/gen-list.test.ts` or `gen-split-tree-single-import.test.ts` scaffolds a config) and run `genCommand([], cwd)`. Assert: exit 0; the output directory contains `Order.ts` and NOT `Address.ts`/`Customer.ts`/`Audited.ts`; stderr contains `3 node(s) from dependencies not generated here`. Second test: `genCommand(["Customer"], cwd)` exits 2 and stderr contains `declared by dependency 'acme-common' (reference mode)`. Third test: flip config + lock to `mode: "own"` → `Customer.ts` IS emitted. Run → FAIL. Implement in `gen.ts` (compute `foreign = root.objects().filter(o => !genCollection.governs(o.resolutionKey()))`; refuse an explicit entity whose owner is reference-mode; pass `governs: genCollection.governs`) and mirror in `verify.ts`'s `computeCodegenDrift` call. Run → PASS. +- [ ] **Step 1: Failing tests.** `cli/test/gen-imported-nodes.test.ts`: scaffold a consumer (`APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1`, a `metaobjects.config.ts` with the owned `entityFile()` + `barrel()` as `gen.test.ts` does). (a) `meta gen` writes `Order.ts` and NOT `Customer.ts` / `Address.ts`. (b) With `"scope": { "include": ["app::**", "acme::common::**"] }`: `Customer.ts` IS written. (c) `meta gen Customer` (no include) → exit 2, stderr contains `imported from dependency 'acme-common'` and `scope.include`. (d) `meta verify --codegen` after (a) is green and reports no drift for the absent `Customer.ts`. `codegen-ts/test/shared-enums-imported.test.ts`: a root with an abstract `field.enum` `lib::Status` (`@values: ["a","b"]`) and two entities — `app::Order` with a field `extends: "lib::Status"` and `lib::Thing` with the same — `renderSharedEnumsFile(root, { select: (f) => f === "app::Order" })` contains `Status`; `select: () => false` returns `null`; no `select` equals today's output. Run → FAIL. -- [ ] **Step 4: Run** +- [ ] **Step 2: Implement.** Run: ```bash -cd server/typescript/packages/cli && bun test test/gen-foreign-nodes.test.ts test/gen-list.test.ts test/gen-split-tree-single-import.test.ts test/verify-requirements-e2e.test.ts -cd server/typescript/packages/codegen-ts && bun test test/shared-enums-foreign.test.ts test/ai-llm-call-codegen.test.ts +cd server/typescript/packages/codegen-ts && bun test test/shared-enums-imported.test.ts test/enums-file.test.ts +cd server/typescript/packages/cli && bun test test/gen-imported-nodes.test.ts test/gen.test.ts test/gen-list.test.ts test/verify-codegen.test.ts bun run --filter '@metaobjectsdev/codegen-ts' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck ``` -Expected: PASS. +Expected: PASS. (Golden gate: `cd server/typescript && bun test packages/codegen-ts/test` must show no golden change — `select` is absent for every golden project.) -- [ ] **Step 5: Commit** +- [ ] **Step 3: Commit** ``` -feat(cli,codegen-ts): reference-mode dependency nodes are never selected for codegen (FR-023) +feat(codegen-ts): imported nodes are not generated unless scope.include names their package (FR-023 §11) ``` --- -**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** There is NO existing refusal path to extend: today `gen`/`runner` only warn on an unmatched entity name. Add a NEW pre-check in `gen.ts` (and `verify --codegen`'s selection) that runs before generation: an explicitly named entity (`meta gen ` / `entities:`) whose FQN is foreign in `reference` mode → refuse with exit code 2 and the message naming the dependency. Leave the existing warn-on-unmatched behaviour for non-foreign names unchanged. - -### Task 10: Migrate and `verify --db` never govern reference-mode nodes +### Task 11: Schema tooling — two-sided exclusion of excluded imports, with the `declaredSchemas` correction, proven on Postgres **Files:** -- Modify: `server/typescript/packages/cli/src/lib/migrate-scope.ts` (add `governedPredicate`, `foreignNote`) -- Modify: `server/typescript/packages/cli/src/commands/migrate.ts` (the three `scopeExpectedSchema(built, collection.inMigrateScope)` / `offlineScope` sites) -- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (`const schemaScope = collection.inMigrateScope;`) -- Modify: `server/typescript/packages/migrate-ts/src/scope.ts` (header comment: one sentence — the predicate may also exclude dependency-owned objects) -- Test: `cli/test/migrate-foreign-nodes.test.ts` (new) +- Modify: `server/typescript/packages/migrate-ts/src/scope.ts` (`scopeExpectedSchema(built, inScope, opts?: { imported?: ObjectScopePredicate })`) +- Modify: `server/typescript/packages/cli/src/lib/migrate-scope.ts` (`dependencyNote(command, names)`; `migrateScopeMismatch` reads `collection.declaredMigrateScope`) +- Modify: `server/typescript/packages/cli/src/commands/migrate.ts` (the three sites: `scopeExpectedSchema(built, collection.inMigrateScope)` at the online + apply pipelines and `offlineScope` for the offline pipeline — each passes `{ imported: collection.imported }`; print `dependencyNote` when any excluded import declared a table or view) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (`schemaScope` site: same `{ imported }`; the replay path's `inScope`; the note) +- Test: `cli/test/migrate-imported-nodes.test.ts` (new, sqlite), `server/typescript/packages/integration-tests/test/dependency-imported-table-pg.test.ts` (new, Postgres), `migrate-ts/test/scope.test.ts` (extend) -**Read first:** DESIGN §2.7 (migrate row), §4.5. `cli/test/migrate-scope.test.ts` — copy its scaffolding (`runBaseline`, `runOfflineGenerate`, the sqlite dialect). `migrate-ts/src/scope.ts` — `scopeExpectedSchema(built, predicate)` returns `{ snapshot, outOfScope, declaredSchemas }`; nothing in migrate-ts changes. +**Read first:** DESIGN §11.1 item 2 (schema bullet — the hazard and the correction), Global Constraints "Two-sided schema exclusion". `migrate-ts/src/scope.ts` header (the three obligations; "a scope narrows objects, never schemas" — which is right for `migrate.scope` and wrong for an import) and `scopeExpectedSchema` (computes `declared` from `built.snapshot` BEFORE filtering). `cli/test/migrate-scope.test.ts` — copy its scaffolding (`runBaseline`, `runOfflineGenerate`, sqlite). `integration-tests/test/jsonb-to-array-type-change-pg.test.ts` — the Postgres precedent (`startPostgres`, `introspectPostgres`, `buildExpectedSchema`, `diff`, `emit`). `migrate-ts/src/diff/index.ts:149-171` — `scopeSchemas`. **Interfaces:** -- Produces: `governedPredicate(collection: Collection): { predicate: ((fqn: string) => boolean) | undefined; foreignNames: string[] }` — `predicate` is `undefined` when the collection has no reference-mode dependencies AND no `inMigrateScope` (byte-identical behavior for every existing project), else `(fqn) => collection.governs(fqn) && (collection.inMigrateScope?.(fqn) ?? true)`; `foreignNote(command: string, names: readonly string[]): string` = `` `meta ${command} — ${names.length} object(s) declared by dependencies (reference mode, governed by their publisher): ${names.join(", ")}` ``. +- Produces migrate-ts: `scopeExpectedSchema(built, inScope, opts?)` — when `opts.imported` is given, first partition `built.snapshot`'s tables and views by `imported(fqn) && !inScope(fqn)` (via `built.provenance`); the excluded half goes to `outOfScope`; `declared = declaredSchemasOf()`; then the existing filter over the remainder. Without `opts`, byte-identical. An `inScope` of `undefined` with `opts.imported` given still performs the import partition (Task 8 guarantees `inMigrateScope` is defined whenever dependencies exist, so this branch is defensive). +- Produces cli: `dependencyNote(command, names)` = `` `meta ${command} — ${names.length} object(s) from dependencies not governed here (name the package in migrate.scope to own them): ${names.join(", ")}` ``. -- [ ] **Step 1: Failing test** `migrate-foreign-nodes.test.ts`: scaffold a consumer whose artifact is `acme-common-v1.json` (Customer has `@table: customers`) and whose local model is `APP` (`orders`), `CONFIG_REF` + `LOCK_V1`, `migrate.dialect: "sqlite"`. (a) `runOfflineGenerate` proposes `CREATE TABLE orders` and NOT `customers`, and the captured stderr contains `1 object(s) declared by dependencies (reference mode, governed by their publisher): acme::common::Customer`. (b) Adopt a baseline snapshot that contains a `customers` table (write the sqlite DB with both tables through the baseline path used in `migrate-scope.test.ts`), re-run: no `DROP TABLE customers` is proposed. (c) `mode: "own"` in config + lock: `CREATE TABLE customers` IS proposed. Run `cd server/typescript/packages/cli && bun test test/migrate-foreign-nodes.test.ts` → FAIL. +- [ ] **Step 1: Failing migrate-ts unit test** (`migrate-ts/test/scope.test.ts`, extend): build an `ExpectedSchemaWithProvenance` by hand with `app.orders` (fqn `app::Order`) and `public.customers` (fqn `acme::common::Customer`); `scopeExpectedSchema(built, () => true, { imported: (f) => f === "acme::common::Customer" })` — hmm, `inScope` returns true for the import here, so nothing is excluded; use `inScope: (f) => f === "app::Order"`: `snapshot.tables` = `[app.orders]`, `outOfScope` = `["public.customers"]`, `declaredSchemas` = `["app"]` (NOT `["app","public"]`). And with `inScope: () => true`: `declaredSchemas` = `["app","public"]`, `outOfScope` = `[]`. Run → FAIL. -- [ ] **Step 2: Implement** `governedPredicate` + `foreignNote`; replace the three migrate sites with `const { predicate, foreignNames } = governedPredicate(collection); const scoped = scopeExpectedSchema(built, predicate); if (foreignNames.length) log.warn(foreignNote("migrate", foreignNames));` (for the offline pipeline, `offlineScope = predicate`); in `verify.ts` set `schemaScope = governedPredicate(collection).predicate` and print `foreignNote("verify --db", …)`. `migrateScopeMismatch` keeps reading `collection.inMigrateScope` alone. +- [ ] **Step 2: Failing sqlite CLI test** (`cli/test/migrate-imported-nodes.test.ts`): consumer = `APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1`, `migrate.dialect: "sqlite"`. (a) `runOfflineGenerate` proposes `CREATE TABLE orders` and NOT `customers`; stderr contains `1 object(s) from dependencies not governed here`. (b) Baseline with both tables present; re-run: no `DROP TABLE customers`. (c) `"migrate": { "scope": ["app::**", "acme::common::**"] }`: `CREATE TABLE customers` IS proposed and the note is absent. Run → FAIL. -- [ ] **Step 3: Run** +- [ ] **Step 3: Failing Postgres test** (`integration-tests/test/dependency-imported-table-pg.test.ts`) — the one that can see schemas: load a consumer model in-memory (`InMemoryStringSource`) consisting of the v1 artifact text PLUS a local root `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"Order","children":[{"source.rdb":{"@table":"orders","@schema":"app"}},{"field.long":{"name":"id"}},{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}`. On a fresh Postgres run raw SQL: `CREATE TABLE public.customers (id bigint primary key, email varchar(120))` (the imported table, as the publisher created it) and `CREATE TABLE public.invoices (id bigint primary key)` (a publisher table the artifact does NOT export). `built = buildExpectedSchemaWithProvenance(root, "postgres")`; `scoped = scopeExpectedSchema(built, (f) => f === "app::Order", { imported: (f) => f.startsWith("acme::common::") })`; `actual = introspectPostgres(...)`; `ops = diff({ ...scopedDiffInputs(scoped, collectUnmanagedNames(root)), actual, dialect: "postgres" })`. Assert: `ops` contains a create for `app.orders`; contains NO drop of `public.invoices`; contains NO create/drop/alter touching `public.customers`. Counter-assert (the legacy own case): `scopeExpectedSchema(built, () => true, { imported })` → `ops` contains a DROP of `public.invoices` (the consumer took `public` by including the package — correct and visible) and nothing for `public.customers` (it matches). Run `cd server/typescript/packages/integration-tests && bun test test/dependency-imported-table-pg.test.ts` → FAIL (the first form proposes `DROP TABLE public.invoices` today). + +- [ ] **Step 4: Implement** the three files. `migrateScopeMismatch`: replace its read of `inMigrateScope` with `declaredMigrateScope` (the user's patterns — the composed predicate would refuse a consumer whose only table-declaring objects are imported and who declared no scope at all, with a message naming a scope that does not exist). + +- [ ] **Step 5: Run** ```bash -cd server/typescript/packages/cli && bun test test/migrate-foreign-nodes.test.ts test/migrate-scope.test.ts test/migrate-offline-gen.test.ts test/migrate-baseline.test.ts test/verify-replay.test.ts -bun run --filter '@metaobjectsdev/cli' typecheck +cd server/typescript/packages/migrate-ts && bun test test/scope.test.ts +cd server/typescript/packages/cli && bun test test/migrate-imported-nodes.test.ts test/migrate-scope.test.ts test/migrate-offline-gen.test.ts test/migrate-baseline.test.ts test/verify-replay.test.ts test/verify-db.test.ts +cd server/typescript/packages/integration-tests && bun test test/dependency-imported-table-pg.test.ts +bun run --filter '@metaobjectsdev/migrate-ts' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck ``` -Expected: PASS. +Expected: PASS. (`integration-tests` uses `METAOBJECTS_TEST_PG_URL` when exported; otherwise it starts a container.) -- [ ] **Step 4: Commit** +- [ ] **Step 6: Commit** ``` -feat(cli): migrate and verify --db leave reference-mode dependency tables to their publisher (FR-023) +feat(migrate): imported tables leave both sides of the schema diff, and never widen its schema scope (FR-023 §11) ``` --- -### Task 11: The load-time failure corpus cases (no new machinery) +### Task 12: The requirements ledger skips excluded imports **Files:** -- Modify: `fixtures/dependency-conformance/cases.json` (append cases 15–19 and 28) -- Modify: `fixtures/dependency-conformance/README.md` (a table: upstream change → consumer construct → existing error code) -- Test: `sdk/test/dependency-conformance.test.ts` - -**Read first:** DESIGN §2.5 (the table). These cases pin that after a snapshot is replaced by a newer upstream, the LOADER's existing errors fail the consumer. They use `acme-common-v2-email-removed.json` as the snapshot (hash `sha256-c4ba364bff0071b164b127326c095d6d32915b57781271e0be64a7003c27ce7a` in the lock) and a local model written against v1. +- Modify: `server/typescript/packages/cli/src/lib/requirement-check.ts` (`scanRequirements(root, opts?: { coverable?: (fqn: string) => boolean })` — `coverableEntities` applies it) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (pass `{ coverable: collection.inScope }`) +- Test: `cli/test/verify-requirements-imported.test.ts` (new) -- [ ] **Step 1: Append** (each: `CONFIG_REF`; `LOCK_V2` = `LOCK_V1` with the v2 integrity; `treeFiles` → `artifacts/acme-common-v2-email-removed.json`; `expectFiles` = artifact + app file; plus `expectLoadError`): - - `an-overlay-whose-target-was-removed-fails`: lock/snapshot as `LOCK_V1`/v1 but the local file overlays a node the artifact never had: `{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Invoice","overlay":true,"children":[{"validator.length":{"@max":5}}]}}]}}` → `expectLoadError: "ERR_OVERLAY_NO_TARGET"`. - - `an-extends-whose-target-was-removed-fails`: local `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"T","extends":"acme::common::Gone","children":[{"source.rdb":{"@table":"t"}},{"field.long":{"name":"id"}},{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}` → `ERR_UNRESOLVED_SUPER`. - - `a-reference-whose-target-was-removed-fails`: local with `"@objectRef":"acme::common::Gone"` on a `field.object` → `ERR_UNRESOLVED_OBJECT_REF`. - - `a-dotted-extends-whose-member-changed-subtype-fails`: snapshot = a case-local artifact (in `tree`) = v1 with `Customer.email` retyped to `field.long`; local `{"metadata.root":{"package":"app","children":[{"field.string":{"name":"contact","extends":"acme::common::Customer.email"}}]}}` → `ERR_EXTENDS_TARGET_MISMATCH`. Pin this artifact's hash in the case's lock (README one-liner). - - `an-overlay-attr-the-base-now-sets-differently-conflicts`: snapshot v1 (`@maxLength: 120`); local `{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"email","@maxLength":80}}]}}]}}` → `ERR_MERGE_CONFLICT`. - - `a-foreign-file-error-names-the-dependency`: snapshot = a case-local artifact in `tree` = v1 with `Address.city` retyped to `{"field.bogus":{"name":"city"}}` (pin its hash in the case's lock via the README one-liner); local `APP` → `expectLoadError: "ERR_UNKNOWN_SUBTYPE"`, `expectErrorFiles: ["dep:acme-common/acme-common.metaobjects.json"]` — the one case that pins the `dep:` source-id shape. - Note case 19 is NOT `ERR_DEPENDENCY_SCHEMA_NOT_OWNED` — the merge conflict fires in the parser before the boundary validator runs; the runner asserts the FIRST error's code. +**Read first:** DESIGN §2.7 row "requirements ledger" (as amended: the denominator excludes excluded imports; a local `@implementedBy` may still name an imported member — resolution runs over the whole loaded model). Pre-flight ruling T20: NO `governs` filter on `verify --templates` (dead — exports never carry `template.*`); only the ledger changes. `requirement-check.ts:77 coverableEntities`. -- [ ] **Step 2: Run** `cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts` → all six PASS with no source change. If one does not, the design table is wrong for that construct: STOP and report the actual code — do not adjust the case. +- [ ] **Step 1: Failing test.** Consumer = `APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1` plus a `metaobjects/meta.req.json` with one `requirement.functional` `@implementedBy: "app::Order"`. `meta verify` (default run): the requirements summary counts 1 coverable entity, 0 uncovered; NO `WARN_REQUIREMENT_OBJECT_UNCLAIMED` for `acme::common::Customer`; the summary line says `, 1 from dependencies`. With `"scope": { "include": ["app::**", "acme::common::**"] }`: `Customer` IS reported unclaimed. Run → FAIL. -- [ ] **Step 3: README table** — add the five rows (upstream change / consumer construct / code) under a heading "What fails at load, with no new machinery". +- [ ] **Step 2: Implement.** Run `cd server/typescript/packages/cli && bun test test/verify-requirements-imported.test.ts test/verify-requirements-e2e.test.ts test/requirement-check.test.ts` and `bun run --filter '@metaobjectsdev/cli' typecheck` → PASS. -- [ ] **Step 4: Commit** +- [ ] **Step 3: Commit** ``` -test(conformance): FR-023 load-time failures after an upstream change are the loader's own errors +feat(cli): the requirements ledger does not count imported entities the project does not include (FR-023 §11) ``` --- -### Task 12: `sharedModelFile()` — the publisher generator (TS) +### Task 13: `sharedModelFile()` — the publisher generator (TS) **Files:** - Create: `server/typescript/packages/codegen-ts/src/generators/shared-model-file.ts` - Modify: `server/typescript/packages/codegen-ts/src/generator.ts` (`GenContext.registry?: TypeRegistry`, `GenContext.sourceFiles?: readonly string[]`) -- Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (fill both fields; `RunGenOptions.sourceFiles?`) -- Modify: `server/typescript/packages/codegen-ts/src/generator-registry.ts` (entry `shared-model`) +- Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (fill both; `RunGenOptions.sourceFiles?`) +- Modify: `server/typescript/packages/codegen-ts/src/generator-registry.ts` (entry `shared-model`, `options: "name, include, exclude?, files?, version?, target?"`) - Modify: `server/typescript/packages/codegen-ts/src/generators/index.ts` (export) -- Modify: `fixtures/generator-registry-conformance/registry.json` (entry `shared-model`, `"ports": ["typescript"]` — Task 19 adds `python`) +- Modify: `fixtures/generator-registry-conformance/registry.json` (entry `shared-model`, `"ports": ["typescript"]`, `tier: "native"`) - Modify: `server/typescript/packages/cli/src/commands/gen.ts` (pass `sourceFiles: genCollection.ownFiles`) -- Modify: the ejectable list the `meta eject --list` command reads (grep `ejectable` under `cli/src`; add `shared-model` with the same shape as its neighbours) -- Test: `codegen-ts/test/shared-model-file.test.ts`, `codegen-ts/test/generator-registry.test.ts` (existing — must stay green), `cli/test/eject.test.ts` (existing) +- Modify: the ejectable list (`cli/src/commands/eject.ts` `ejectableNames()` reads the registry — confirm `shared-model` appears; if the list is curated, add it with the same shape as its neighbours) +- Test: `codegen-ts/test/shared-model-file.test.ts` (new), `codegen-ts/test/generator-registry.test.ts` (existing — stays green), `cli/test/eject.test.ts` (existing) -**Read first:** DESIGN §2.2, §4.3, §3.2, §3.5. `oncePerRun` in `generator.ts`; `MetaDataLoader` + `FileSource` (`@metaobjectsdev/metadata/core`); `serializeSharedDocument` (Task 2); `compileScope`/`matchesScope` from `@metaobjectsdev/metadata` (Task 4); `sha256Integrity` lives in sdk and codegen-ts must NOT depend on sdk, so this task MOVES `INTEGRITY_PREFIX` to `@metaobjectsdev/metadata/constants` (a new `dependency-constants.ts` module barreled there; sdk's `dependencies.ts` re-exports it from metadata instead of defining it) and adds a three-line `integrityOf(bytes: Uint8Array): string` (`createHash("sha256")`) in `codegen-ts/src/generators/shared-model-file.ts`. +**Read first:** DESIGN §2.2, §4.3, §3.2, §3.5. `serializeSharedDocument` (Task 2, `metadata/src/serializer-json.ts`); `compileScope` / `matchesScope` from `@metaobjectsdev/metadata` (Task 4); `REF_BEARING_ATTR_NAMES` (`naming-refs.ts`); `MetaDataLoader` + `coreProviders` for the two standalone loads; `oncePerRun`. The generator excludes `requirement.*` and `template.*` always. **Interfaces:** -- Produces: `INTEGRITY_PREFIX` now exported from `@metaobjectsdev/metadata/constants` (sdk re-exports it; Task 6's tests stay green). `sharedModelFile(opts: { name: string; version?: string; files?: readonly string[]; include: readonly string[]; exclude?: readonly string[]; target?: string }): Generator` (name `shared-model`). Emits two files into its target: `.metaobjects.json` and `metaobjects.pkg.json`. `version` defaults to the nearest `package.json`'s `version` walking up from `ctx.projectRoot` (error if none). `files` defaults to `ctx.sourceFiles`. Excludes `requirement.*` and `template.*` nodes always. - -**Algorithm (DESIGN §4.3):** (1) standalone load of `files` with `ctx.registry` (strict) → errors are thrown as `Error("shared-model: ")`; (2) select top-level objects/fields (any top-level node whose type is not `requirement`/`template`) by `matchesScope(resolutionKey, compileScope({ include, exclude }))`; empty → throw `shared-model: include/exclude selected no nodes`; (3) closure: for every selected node and each descendant, every `extends` target (object part of a dotted ref) and every attr named in `REF_BEARING_ATTR_NAMES` (object part) must be a selected resolution key, else throw `shared-model: references , which is not selected — include it or exclude the referrer` listing every pair; (4) `artifact = serializeSharedDocument(selected)`; (5) re-load `artifact` with `composeRegistry(coreProviders)` strict → errors throw `shared-model: the export needs vocabulary the core registry does not register: `; (6) manifest per §3.2 with `nodes` = sorted resolution keys, `packages` = sorted distinct `packageOfResolutionKey`s, `metamodelVersion = METAMODEL_VERSION`, `integrity = integrityOf(artifact)`; serialized `JSON.stringify(manifest, null, 2) + "\n"`. +- Produces: `sharedModelFile(opts: { name: string; include: readonly string[]; exclude?: readonly string[]; files?: readonly string[]; version?: string; target?: string }): Generator`. Emits `.metaobjects.json` (via `serializeSharedDocument` over the selected nodes, sorted by resolution key) and `metaobjects.pkg.json` (`integrity` over the artifact bytes; `version` = `opts.version` ?? nearest `package.json` `version` walking up from `ctx.projectRoot`; `metamodelVersion` = `METAMODEL_VERSION`). Errors (thrown as generator errors, so `runGen` reports them by generator name): empty selection; closure failure listing `(referrer → target)` pairs with "include it or exclude the referrer"; the core-provider re-load failing ("the export needs a provider this toolchain does not ship; Phase 1 exports must load with core vocabulary: "). -- [ ] **Step 1: Failing test** `shared-model-file.test.ts` — scaffold a temp publisher with `metaobjects/meta.lib.json` (the `LIB` document from Task 2, root package `acme::common`) plus `metaobjects/admin-ui/meta.lib.ui.json` (`{"metadata.root":{"package":"acme::common","children":[{"object.entity":{"name":"Customer","overlay":true,"children":[{"field.string":{"name":"email","children":[{"validator.length":{"@max":100}}]}}]}}]}}`) and a `package.json` with `"version": "1.0.0"`. Run `runGen` programmatically (mirror `codegen-ts/test/dry-run-writes-nothing.test.ts` for the harness) with `generators: [sharedModelFile({ name: "acme-common", files: ["metaobjects/meta.lib.json"], include: ["acme::common::**"] })]`. Assert: `acme-common.metaobjects.json` is byte-identical to `fixtures/dependency-conformance/artifacts/acme-common-v1.json` (the overlay file was not listed, so the validator child is absent); `metaobjects.pkg.json` parses to the manifest in Task 6's test (`integrity` = the pinned v1 hash). Second test: `include: ["acme::common::Customer"]` alone succeeds (Customer references nothing outside itself); `include: ["acme::common::Address"]` with a local file where `Address` `extends: acme::common::Audited` throws `/references acme::common::Audited, which is not selected/`. Third: a `files` list including the overlay file yields an artifact containing `validator.length`. Fourth: two runs produce identical bytes. Fifth: a document whose selected node carries an attr registered only by a consumer provider (`ctx.registry` composed with a test provider registering `@acmeLocal` on `object.entity`) throws `/core registry does not register/`. Run `cd server/typescript/packages/codegen-ts && bun test test/shared-model-file.test.ts` → FAIL. +- [ ] **Step 1: Failing tests** (`codegen-ts/test/shared-model-file.test.ts`): a publisher project whose two source files declare `acme::common::{Address, Audited, Customer}` exactly as `acme-common-v1.json` does plus an internal `acme::common::Secret` entity and a `requirement.functional`; `sharedModelFile({ name: "acme-common", include: ["acme::common::**"], exclude: ["acme::common::Secret"], version: "1.0.0" })` through `runGen` emits (a) `acme-common.metaobjects.json` BYTE-EQUAL to `fixtures/dependency-conformance/artifacts/acme-common-v1.json` (fix the publisher fixture until it is — the pinned artifact is the contract), (b) a manifest equal to the Global Constraints example with the pinned v1 hash; (c) determinism: two runs, identical bytes; (d) closure: `include: ["acme::common::Customer"]` alone fails naming `acme::common::Customer → acme::common::Address` (Customer's `field.object` → Address in the fixture; read the artifact to confirm the real edge); (e) a `files` subset that omits the file declaring `Customer` exports two nodes; (f) a node carrying a consumer-provider attr (register a throwaway provider in the test) fails the core re-load with the message above. Run → FAIL. -- [ ] **Step 2: Implement** the generator, the `GenContext` fields, the runner plumbing, the registry entry (`description: "One flattened canonical-JSON shared-model artifact + manifest for consumers (FR-023)."`, `tier: "native"`, `options: "name, include, exclude?, files?, version?, target?"`), the `registry.json` entry (`"concept": "The publisher's shared-model artifact (a flattened canonical document + manifest) consumed through `dependencies`.", "tier": "native", "ports": ["typescript"]`), the eject listing, and `gen.ts` passing `sourceFiles: genCollection.ownFiles`. +- [ ] **Step 2: Implement.** Run `cd server/typescript/packages/codegen-ts && bun test test/shared-model-file.test.ts test/generator-registry.test.ts`, `cd server/typescript/packages/cli && bun test test/eject.test.ts test/gen.test.ts`, `bun run --filter '@metaobjectsdev/codegen-ts' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck` → PASS. Also `cd server/python && uv run --extra integration pytest tests/conformance/test_generator_registry_conformance.py -q` → PASS (the registry entry says `typescript` only). -- [ ] **Step 3: Run** -```bash -cd server/typescript/packages/codegen-ts && bun test test/shared-model-file.test.ts test/generator-registry.test.ts -cd server/typescript/packages/cli && bun test test/eject.test.ts test/gen-list.test.ts -cd server/python && uv run --extra integration pytest tests/codegen/test_cli_registry.py -q -bun run --filter '@metaobjectsdev/codegen-ts' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck && bun run --filter '@metaobjectsdev/metadata' typecheck && bun run --filter '@metaobjectsdev/sdk' typecheck -``` -Expected: PASS (the Python registry test stays green because `ports` lists only `typescript`). - -- [ ] **Step 4: Commit** +- [ ] **Step 3: Commit** ``` -feat(codegen-ts): sharedModelFile() emits the flattened shared-model artifact and manifest (FR-023) +feat(codegen-ts): sharedModelFile() emits a publisher's flattened shared-model artifact and manifest (FR-023) ``` --- -### Task 13: `meta deps sync` with the `path` transport, and `meta deps list` +### Task 14: `meta deps sync` (`path` transport), `meta deps list`, `--dry-run` **Files:** - Create: `server/typescript/packages/cli/src/commands/deps.ts` -- Create: `server/typescript/packages/cli/src/lib/dependency-transports.ts` (`resolveTransport` — `path` only here; Task 16 adds `npm`/`python`) -- Create: `server/typescript/packages/cli/src/lib/dependency-sync.ts` -- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`DEPS_OPTIONS`, `parseDepsArgs`) -- Modify: `server/typescript/packages/cli/src/index.ts` (register `deps`; help text; `FORMAT_AWARE_COMMANDS` gains `"deps"`) +- Create: `server/typescript/packages/cli/src/lib/dependency-sync.ts` (pure functions over a resolved directory: `readManifestDir`, `validateAgainstSpec`, `standaloneLoadCheck`, `planSync`, `applySync`) +- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`DEPS_OPTIONS = { "dry-run": boolean, format: string }`, `parseDepsArgs` with positionals `sync | check | list […]`) +- Modify: `server/typescript/packages/cli/src/index.ts` (register `deps` beside `eject`; help text; `FORMAT_AWARE_COMMANDS` gains `"deps"`) - Test: `cli/test/deps-sync.test.ts`, `cli/test/unit/args-deps.test.ts`, `cli/test/help-lists-every-flag.test.ts` (existing — must stay green) -**Read first:** DESIGN §4.1 (steps 1–5, 7–8; classification is Task 15), §2.1 (transports), §3.2–3.4. `cli/src/index.ts` dispatch (`case "migrate"` shape); `args.ts` `parseArgs` style with an exported options table; `sdk` exports from Tasks 5–7. +**Read first:** DESIGN §4.1 steps 1–5, 7–8 (step 6, classification, is superseded — §11.3), §2.1 "Transports" (`path` only). Global Constraints "Config" (the `npm` / `python` refusal). `eject.ts` for a command's shape and `index.ts` wiring. **Interfaces:** -- Produces: `depsCommand(args: string[], cwd: string, fmt: OutputFormat): Promise` with verbs `sync […] [--dry-run]`, `list`; `resolveTransport(spec: DependencySpec, configDir: string): Promise` (a directory); `syncOne(configDir, spec, dir, opts): Promise<{ entry: LockEntry; artifactBytes: Uint8Array; changed: boolean }>`; `validateResolvedPackage(dir, spec): Promise<{ manifest: Manifest; artifactBytes: Uint8Array }>` performing DESIGN §4.1 steps 2–4 with the codes `ERR_DEPENDENCY_MANIFEST_INVALID` / `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`; `checkCollisions(entries): void` → `ERR_DEPENDENCY_NODE_COLLISION`. - -**Behavior:** `sync` resolves each declared dependency (all, or the named subset — an unknown name exits 2), validates, then (unless `--dry-run`) empties `.metaobjects/deps//`, writes the artifact, updates the lock entry (`resolvedFrom` = the spec's transport keys verbatim, `mode` = the spec's mode), prunes lock entries and snapshot dirs for names no longer declared, writes the lock with sorted keys; then loads the whole collection through `loadCollection` (Task 8) so a sync producing an unloadable model exits 1 in the same command. Report lines (stderr): `synced ( node(s), package(s)) from path ` / `unchanged ` / `would sync …` under `--dry-run`. `list` prints one line per lock entry: `\t\t\t\t`; `--format json` emits the lock entries. `sync` also ensures `.metaobjects/.gitignore` contains a `deps.local.json` line (append if the file exists and lacks it; create with that one line if absent) — the D10 hygiene half that belongs to sync. +- `meta deps sync […] [--dry-run] [--format text|json|toon]`, for each declared dependency in name order: (1) `path` → `resolve(configDir, spec.path)`; `npm` / `python` → `ERR_DEPENDENCY_UNRESOLVED` "transport `` is not supported by this toolchain yet; use `path`"; missing dir → `ERR_DEPENDENCY_UNRESOLVED` naming what was tried. (2) Read `/metaobjects.pkg.json` strict; absent / invalid / `name ≠ spec.name` / artifact missing / `sha256(artifact) ≠ integrity` → `ERR_DEPENDENCY_MANIFEST_INVALID`. (3) `metamodelVersion` major ≠ `METAMODEL_VERSION` major → `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`; minor greater → warning. (4) Standalone strict load with `coreProviders`; any error, or top-level resolution keys ≠ `manifest.nodes`, or packages ≠ `manifest.packages` → `ERR_DEPENDENCY_MANIFEST_INVALID`. (5) `nodes` intersecting another entry's → `ERR_DEPENDENCY_NODE_COLLISION`. (7) Unless `--dry-run`: empty `.metaobjects/deps//`, copy the artifact, update the lock entry (`resolvedFrom` = the spec's transport verbatim), prune entries no longer declared, `writeLock`. (8) Report per dependency: `synced → (→)` / `unchanged ` / `would sync …`. Then load the full collection once (`loadMemory` with `collectionLoadOptions`) so a sync that produces an unloadable model fails in the same command with the loader's error. Exit 1 on any error; 0 otherwise. +- `meta deps list`: one line per lock entry — ` node(s) `; `--format json` emits the lock's entries. -- [ ] **Step 1: Failing args test** `cli/test/unit/args-deps.test.ts`: `parseDepsArgs(["sync", "acme-common", "--dry-run"])` → `{ verb: "sync", names: ["acme-common"], dryRun: true }`; `["list"]` → `{ verb: "list", … }`; `["frobnicate"]` throws; `["sync", "--bogus"]` throws. Run → FAIL. Implement `DEPS_OPTIONS = { "dry-run": { type: "boolean" } } as const` and `parseDepsArgs(argv): { verb: "sync" | "list"; names: string[]; dryRun: boolean }` — `accept-breaking` is NOT parsed here (a flag that does nothing must not ship; Task 15 adds it with its behavior). Run → PASS. +- [ ] **Step 1: Failing tests.** `deps-sync.test.ts`: a temp publisher dir holding `acme-common-v1.json` + a hand-written manifest (the Global Constraints example) and a consumer with `CONFIG_REF` pointing at it (the path is relative to the consumer's config dir — lay the two out as siblings). (a) `sync` creates the snapshot byte-equal to the artifact and a lock equal to `LOCK_V1` with `resolvedFrom.path` = the spec's string; (b) a second `sync` prints `unchanged` and leaves the lock bytes identical; (c) `--dry-run` writes nothing; (d) replace the publisher's files with the widened artifact + its manifest: `sync` prints `synced … (10fbf886→fa00b9f3)` and the lock's `integrity` is the widened hash; (e) removing the dependency from config and syncing prunes the lock entry and the snapshot dir; (f) a manifest whose `nodes` omits `Customer` → `ERR_DEPENDENCY_MANIFEST_INVALID`; (g) `{ name: "x", npm: "@acme/model" }` → `ERR_DEPENDENCY_UNRESOLVED` with "not supported by this toolchain yet"; (h) `list` after (a) prints `acme-common 1.0.0 10fbf886 3 node(s) acme::common`. `args-deps.test.ts`: subverb parsing; an unknown subverb is a usage error (exit 2). Run → FAIL. -- [ ] **Step 2: Failing command test** `deps-sync.test.ts`. Scaffold `/lib/metaobjects/` holding `metaobjects.pkg.json` + `acme-common.metaobjects.json` (copy the pinned v1 artifact and the manifest from Task 6's test data) and `/app/` with `.metaobjects/config.json` = `{ "schema_version": 1, "sources": [], "dependencies": [{ "name": "acme-common", "path": "../lib/metaobjects" }] }` and `metaobjects/meta.app.json` = `APP`. Assert, in order: (a) `depsCommand(["sync"], app)` exits 0; `.metaobjects/deps/acme-common/acme-common.metaobjects.json` equals the pinned bytes; `.metaobjects/deps.lock.json` equals exactly the `LOCK_V1` text from Task 7 with `"resolvedFrom": { "path": "../lib/metaobjects" }` (2-space, trailing newline, sorted keys); `.metaobjects/.gitignore` contains `deps.local.json`. (b) a second `sync` exits 0, reports `unchanged`, and leaves the lock byte-identical. (c) `--dry-run` after editing the upstream artifact to the widened version writes nothing and prints `would sync`. (d) removing the dependency from config and syncing prunes the lock entry and deletes `.metaobjects/deps/acme-common/`. (e) `sync` against a lib dir whose manifest `name` is `other` exits 1 with `ERR_DEPENDENCY_MANIFEST_INVALID` in stderr; a lib dir with no manifest → `ERR_DEPENDENCY_UNRESOLVED`; a manifest whose `integrity` does not match its artifact → `ERR_DEPENDENCY_MANIFEST_INVALID`; a manifest with `metamodelVersion: "2.0"` → `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`; an artifact whose top-level nodes differ from `manifest.nodes` → `ERR_DEPENDENCY_MANIFEST_INVALID`. (f) `depsCommand(["list"], app)` prints a line starting with `acme-common\t1.0.0\treference\tsha256-10fbf886`. Run `cd server/typescript/packages/cli && bun test test/deps-sync.test.ts` → FAIL. - -- [ ] **Step 3: Implement** `dependency-transports.ts` (`path`: `isAbsolute ? spec.path : resolve(configDir, spec.path)`; must be a directory holding `MANIFEST_FILE`, else `ERR_DEPENDENCY_UNRESOLVED`), `dependency-sync.ts` (validation per §4.1: parse manifest with `ManifestSchema` → `ERR_DEPENDENCY_MANIFEST_INVALID`; `manifest.name !== spec.name` → same; read artifact, hash, compare; `metamodelVersion` major vs `METAMODEL_VERSION`; standalone load of the artifact with `composeRegistry(coreProviders)` strict via `MetaDataLoader.fromString` → any error → `ERR_DEPENDENCY_MANIFEST_INVALID`; top-level resolution keys (sorted) must equal `manifest.nodes`, packages equal `manifest.packages`), `deps.ts`, the dispatch, and the help text (`meta deps sync|list`). Errors are `ParseError`s with the codes; the command prints `meta deps: ` and exits 1 (2 for a usage error). - -- [ ] **Step 4: Run** +- [ ] **Step 2: Implement.** Run ```bash -cd server/typescript/packages/cli && bun test test/deps-sync.test.ts test/unit/args-deps.test.ts test/help-lists-every-flag.test.ts test/help-and-exit.test.ts test/cli.test.ts +cd server/typescript/packages/cli && bun test test/deps-sync.test.ts test/unit/args-deps.test.ts test/help-lists-every-flag.test.ts bun run --filter '@metaobjectsdev/cli' typecheck ``` Expected: PASS. -- [ ] **Step 5: Commit** -``` -feat(cli): meta deps sync (path transport) and meta deps list (FR-023) -``` - ---- - -### Task 14: The usage-aware classifier and the footprint walker - -**Files:** -- Create: `server/typescript/packages/metadata/src/dependency-diff.ts` -- Modify: `server/typescript/packages/metadata/src/index.ts` (export) -- Modify: `server/typescript/packages/sdk/src/dependencies.ts` (`dependencyFootprint`) -- Modify: `fixtures/dependency-conformance/cases.json` (append classify cases 29–46) -- Modify: `sdk/test/dependency-conformance.test.ts` (implement the `classify` arm — remove the `test.skip`) -- Test: `metadata/test/dependency-diff.test.ts`, `sdk/test/dependencies.test.ts` (footprint) - -**Read first:** DESIGN §2.6 (footprint scopes, the class table, the widening and additive tables — copy them into named constants), §4.6. `REF_BEARING_ATTR_NAMES` (`naming-refs.ts`) is the reference-attr set; `RELATIONSHIP_ATTR_OBJECT_REF`/`RELATIONSHIP_ATTR_THROUGH` are in it; `IDENTITY_REFERENCE_ATTR_REFERENCES` too. `@implementedBy` is `REQUIREMENT_ATTR_IMPLEMENTED_BY` (check `requirement` constants). Documentation attrs are the common attrs `description`, `title`, `notes`, `seeAlso`, `aliases`, `replacedBy`, `deprecated` — import their constants from the documentation provider's constants module, never inline. - -**Interfaces:** -- Produces (metadata): `type FootprintScope = "whole" | "key" | "existence"`; `type Change = { fqn: string; path: string; kind: "breaking" | "compatible" | "info"; before?: unknown; after?: unknown; rule: string }`; `classifyDependencyChanges(oldDoc: string, newDoc: string, footprint: ReadonlyMap): Change[]` — both docs are artifact strings (canonical JSON); `WIDENING_ATTRS`, `ADDITIVE_ATTRS`, `DOC_ATTRS` exported constants. -- Produces (sdk): `dependencyFootprint(root: MetaRoot, collection: Collection): Map`. - -**Diff algorithm:** parse both documents; index top-level nodes by `::`; for each footprint fqn: absent in `new` → one `breaking` change at path `""` with rule `node-removed`; else restrict both bodies by scope (`whole`: everything; `key`: children whose fused key starts with `identity.` or `source.`, plus `extends`/`abstract`; `existence`: the fused key only) and walk: children matched by `(fused key, name)` → removed child → `breaking` (`child-removed`), added child → `compatible` (`child-added`); a body key `extends`/`abstract`/`isArray` differing → `breaking`; `@`-attr removed → `breaking` (`attr-removed`) unless in `DOC_ATTRS` (ignored); `@`-attr added → `compatible` if in `ADDITIVE_ATTRS` or `DOC_ATTRS` (`deprecated` → `info`), else `breaking` (`attr-added`); `@`-attr changed → ignored if `DOC_ATTRS`, `compatible` if the `WIDENING_ATTRS` rule for that attr holds in the stated direction (`maxLength`/`precision`/`max` numeric up; `minLength`/`min` numeric down; `required` `true→false`; `values` old ⊂ new; `intValueMap` every old entry present and equal), else `breaking` (`attr-changed`). `path` is the JSON-pointer-like `children[]..` chain by names, e.g. `field.string:email/@maxLength`. - -- [ ] **Step 1: Append classify cases** — each `{ "name", "classify": { "old": , "new": , "footprint": {…}, "expectChanges": [...] } }` where `old` is the v1 artifact document (as a JSON object, not a string) unless stated, footprint `{ "acme::common::Customer": "whole" }` unless stated: - 29 `removed-node-is-breaking`: `new` = v1 without `Customer` → `[{ "fqn": "acme::common::Customer", "path": "", "kind": "breaking" }]`. - 30 `removed-member-is-breaking`: `new` = v2 (email removed) → `[{ "fqn": "acme::common::Customer", "path": "field.string:email", "kind": "breaking" }]`. - 31 `subtype-change-is-breaking`: `new` = v1 with `email` retyped `{"field.long":{"name":"email"}}` (attrs dropped) → exactly ONE change: `[{ "fqn": "acme::common::Customer", "path": "email", "kind": "breaking" }]` with rule `subtype-changed`. Children are matched by NAME first and fused key second, so a retyped member is one breaking change, never a removal plus an addition. - 32 `isarray-change-is-breaking`: email gains `"isArray": true` → breaking at `field.string:email/isArray`. - 33 `required-true-to-false-is-compatible`: old email has `"@required": true`, new `false` → compatible. - 34 `required-false-to-true-is-breaking`: reverse → breaking. - 35 `maxlength-widened-is-compatible`: `new` = v1-widened → compatible at `field.string:email/@maxLength`. - 36 `maxlength-narrowed-is-breaking`: old widened (200) → new v1 (120) → breaking. - 37 `enum-member-added-is-compatible`: both docs carry a `field.enum` `kind` on Customer with `@values ["a","b"]` → `["a","b","c"]` → compatible. - 38 `enum-member-removed-is-breaking`: reverse → breaking. - 39 `added-child-is-compatible`: new Customer gains `{"field.string":{"name":"phone"}}` → compatible at `field.string:phone`. - 40 `description-change-is-ignored`: Customer `"description"` differs → `expectChanges: []`. - 41 `unknown-attr-change-is-breaking`: email `"@column": "email_addr"` → `"@column": "mail"` → breaking. - 42 `deprecated-added-is-info`: Customer gains `"deprecated": true` → info. - 43 `a-change-outside-the-footprint-is-not-reported`: new = v2 (email removed) but footprint `{ "acme::common::Address": "whole" }` → `[]`. - 44 `a-non-key-member-change-on-an-fk-target-is-not-reported`: footprint `{ "acme::common::Customer": "key" }`, new = v1-widened → `[]`. - 45 `a-key-field-change-on-an-fk-target-is-breaking`: footprint `key`, new = v1 with `identity.primary` `@fields` = `["email"]` → breaking at `identity.primary:pk/@fields`. - 46 `an-inherited-member-change-on-an-extends-target-is-breaking`: footprint `{ "acme::common::Audited": "whole" }`, new = v1 with `Audited.createdAt` retyped `field.string` → breaking at `createdAt`. - (Documentation attrs: check the exact key spelling the documentation provider registers — `description` is a bare common attr in canonical JSON? It is an `@`-prefixed attr in canonical form: `"@description"`. Use the canonical spelling in the cases; the runner passes documents through unchanged.) - -- [ ] **Step 2: Implement the runner's classify arm** (parse docs to strings via `JSON.stringify(old, null, 2) + "\n"`, call `classifyDependencyChanges`, compare `{fqn, path, kind}` triples as unordered sets). Run → the 18 cases FAIL (module missing). - -- [ ] **Step 3: Failing unit tests** `metadata/test/dependency-diff.test.ts` — the widening table row by row (one `test` per entry of `WIDENING_ATTRS`, both directions) and `DOC_ATTRS` ignored; `sdk/test/dependencies.test.ts` — `dependencyFootprint` over a loaded consumer (artifact v1 + a local model that `extends` `Audited`, `@objectRef`s `Address`, `@references` `Customer` via `identity.reference`, and overlays `Customer` with `overlay: true`) yields exactly `{ "acme::common::Audited": "whole", "acme::common::Address": "whole", "acme::common::Customer": "key" }` — `Customer` is both overlaid (`existence`) and FK-referenced (`key`); the widest scope wins. Run → FAIL. - -- [ ] **Step 4: Implement** `dependency-diff.ts` and `dependencyFootprint` (walk every LOCAL top-level node — `collection.foreignOwner(fqn) === undefined` — and its descendants with resolving accessors; collect `extends` targets (object part of a dotted ref), `REF_BEARING_ATTR_NAMES` values split by attribute into `whole` vs `key` per the Global Constraints, `@implementedBy` → existence; a local contribution to a foreign node → existence; close `whole` entries over the target's `extends` chain). - -- [ ] **Step 5: Run** -```bash -cd server/typescript/packages/metadata && bun test test/dependency-diff.test.ts -cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts test/dependencies.test.ts -bun run --filter '@metaobjectsdev/metadata' typecheck && bun run --filter '@metaobjectsdev/sdk' typecheck -``` -Expected: PASS (cases 29–46 green). - -- [ ] **Step 6: Commit** +- [ ] **Step 3: Commit** ``` -feat(metadata,sdk): usage-aware dependency change classifier and footprint (FR-023) +feat(cli): meta deps sync and list — a committed snapshot pinned by a sha256 lock (FR-023) ``` --- -**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** Add the registered `summary` documentation attribute to the ignored documentation-attr set, and build that set from the named constants in `doc-constants.ts` (including `DOC_ATTR_SUMMARY`), never string literals. Add a classifier test: a summary-only upstream edit is ignored. - -### Task 15: `sync` refuses BREAKING upstream changes; `meta deps check` and `meta verify --deps` +### Task 15: `meta deps check` and `meta verify --deps` — the hash-compare drift gate **Files:** -- Modify: `server/typescript/packages/cli/src/lib/dependency-sync.ts` (`classifyAgainstSnapshot`, `checkDependencies`) -- Modify: `server/typescript/packages/cli/src/commands/deps.ts` (`--accept-breaking []`, `check`) -- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`DEPS_OPTIONS["accept-breaking"]`, `VERIFY_OPTIONS.deps`) -- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (`--deps` subverb; its own line in the report; exit aggregation) +- Modify: `server/typescript/packages/cli/src/lib/dependency-sync.ts` (`checkDependencies(configDir, specs, lock) → { name, status: "current" | "drifted" | "unresolved", lockIntegrity, installedIntegrity, lockVersion, installedVersion }[]`) +- Modify: `server/typescript/packages/cli/src/commands/deps.ts` (`check`) +- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`VERIFY_OPTIONS.deps: boolean`) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (`--deps` subverb: its own section and structured row; exit aggregation; `anyExplicit` includes it) - Modify: `server/typescript/packages/cli/src/index.ts` (help text) -- Test: `cli/test/deps-check.test.ts`, `cli/test/unit/args-deps.test.ts`, `cli/test/unit/args-verify.test.ts`, `cli/test/help-lists-every-flag.test.ts` +- Test: `cli/test/deps-check.test.ts`, `cli/test/unit/args-verify.test.ts`, `cli/test/help-lists-every-flag.test.ts` -**Read first:** DESIGN §4.1 step 6, §4.2 (`meta deps check`), §2.6 "Where it fires". `verify.ts` subverb handling (`templates || codegen || docs || db …` decides "any explicit"); the design says `--deps` is one more subverb and is NOT part of the bare-`verify` default. +**Read first:** DESIGN §2.3 ("the lock pins bytes"), §4.1 (the `check` verb as amended by §11: no classification), `verify.ts:161-165` (how subverbs are selected) and the templates section for a subverb's reporting shape. -**Interfaces:** -- Produces: `classifyAgainstSnapshot(configDir, collection, name, newArtifact): Change[]` (loads the consumer model via `loadCollection`, computes the footprint for `name`, reads the current snapshot artifact, calls `classifyDependencyChanges`); `checkDependencies(cwd): Promise<{ exitCode: number; report: string[] }>`; `meta deps sync` exits 1 with `ERR_DEPENDENCY_BREAKING_CHANGE` and the per-change report (` : ()`) when any `breaking` change exists and the dependency is not named by `--accept-breaking` (bare `--accept-breaking` = all); `meta deps check` = resolve + validate + hash every dependency, compare each `integrity` to the lock, classify differences, exit 1 with `ERR_DEPENDENCY_UPSTREAM_DRIFT` naming the dependency, locked vs upstream version, and each change line; unreachable upstream → exit 1 `ERR_DEPENDENCY_UNRESOLVED`; also runs the snapshot integrity check. `meta verify --deps` runs `checkDependencies` and prints `meta verify — deps: N dependency(ies) in sync` or the drift report. - -- [ ] **Step 1: Failing tests.** Extend `args-deps.test.ts`: `["sync", "--accept-breaking"]` → `acceptBreaking: []` (all); `["sync", "--accept-breaking", "acme-common"]` → `["acme-common"]`; `["check"]` → verb check. Extend `args-verify.test.ts`: `--deps` sets `deps: true` and counts as an explicit subverb. `deps-check.test.ts` (scaffold as Task 13 (a) — lib dir with the v1 artifact + manifest, app synced once — then, as separate tests): - (a) `depsCommand(["check"], app)` exits 0 and stderr contains `in sync`. - (b) Replace the lib's artifact with `acme-common-v1-widened.json` and rewrite its manifest with the widened integrity (`sha256-fa00b9f3c54a2c302cf269af051589afc0a3217999ac82c54c77e33494638c36`). With the local model `APP` (which references nothing foreign): `check` exits 1, stderr contains `ERR_DEPENDENCY_UPSTREAM_DRIFT`, `acme-common 1.0.0 -> 1.0.0`, and `0 change(s) in your footprint`. - (c) Change the local model so `Order` carries `{"identity.reference":{"name":"fkCustomer","@fields":["customerId"],"@references":"acme::common::Customer"}}` plus `{"field.long":{"name":"customerId"}}`: `check` still reports drift with `0 change(s) in your footprint` — `key` scope excludes `@maxLength`. - (d) Replace the lib's artifact with `acme-common-v2-email-removed.json` (+ manifest with `sha256-c4ba364bff0071b164b127326c095d6d32915b57781271e0be64a7003c27ce7a`) and change the local model to `{"metadata.root":{"package":"app","children":[{"field.string":{"name":"contact","extends":"acme::common::Customer.email"}}]}}`: `check` exits 1 and prints `acme::common::Customer field.string:email: breaking (child-removed)`; `depsCommand(["sync"], app)` exits 1 with `ERR_DEPENDENCY_BREAKING_CHANGE` and the snapshot still holds the v1 bytes; `depsCommand(["sync", "--accept-breaking", "acme-common"], app)` writes the v2 bytes and the v2 hash into the lock, then exits 1 because the post-sync load fails with `ERR_UNRESOLVED_SUPER` (assert both the files and the exit code). - (e) `verifyCommand(["--deps"], app)` returns the same exit code as `check` in states (a) and (d) and prints a line starting `meta verify — deps:`. -Run `cd server/typescript/packages/cli && bun test test/deps-check.test.ts test/unit/args-deps.test.ts test/unit/args-verify.test.ts` → FAIL. +**Interfaces:** `check` resolves each declared dependency exactly as `sync` steps 1–2 do (a resolution failure is `unresolved`), compares the installed manifest's `integrity` — re-hashing the installed artifact, never trusting the manifest alone — with the lock's. Any `drifted` or `unresolved` → `ERR_DEPENDENCY_UPSTREAM_DRIFT`, exit 1, one line per dependency: `acme-common: drifted — lock 1.0.0 (10fbf886), installed 1.1.0 (fa00b9f3); run meta deps sync and review the artifact diff`. `verify --deps` runs the same function and adds a `deps` row to the report; an `unresolved` dependency FAILS the check (a check that cannot check must not pass). `--deps` is never part of the default run (it needs the publisher reachable, which CI may not have). -- [ ] **Step 2: Implement.** In `sync`: before writing, if a snapshot exists for the name, `classifyAgainstSnapshot`; print every change; refuse on breaking unless accepted. Write, then run the post-sync load; a load failure exits 1 but the write stands (the report says `synced … ; the model no longer loads: — fix your metadata`). `check`: never writes. `verify --deps`: calls `checkDependencies(cwd)`. +- [ ] **Step 1: Failing tests.** `deps-check.test.ts` on Task 14's scaffold: after `sync`, `check` exits 0 and prints `current`; replace the publisher with the widened artifact + manifest: `check` exits 1 with the line above and `verify --deps` exits 1 with a `deps` row; delete the publisher dir: `unresolved`, exit 1; `verify` (no `--deps`) exits 0 regardless. `args-verify.test.ts`: `--deps` parses. Run → FAIL. -- [ ] **Step 3: Run** +- [ ] **Step 2: Implement.** Run ```bash -cd server/typescript/packages/cli && bun test test/deps-check.test.ts test/deps-sync.test.ts test/unit/args-deps.test.ts test/unit/args-verify.test.ts test/help-lists-every-flag.test.ts test/verify-replay.test.ts +cd server/typescript/packages/cli && bun test test/deps-check.test.ts test/deps-sync.test.ts test/unit/args-verify.test.ts test/help-lists-every-flag.test.ts test/verify-templates.test.ts bun run --filter '@metaobjectsdev/cli' typecheck ``` Expected: PASS. -- [ ] **Step 4: Commit** +- [ ] **Step 3: Commit** ``` -feat(cli): meta deps check, verify --deps, and sync's breaking-change refusal (FR-023) +feat(cli): meta deps check and verify --deps fail when the installed dependency differs from the lock (FR-023) ``` --- -### Task 16: The `npm` and `python` transports +### Task 16: ~~Runtime scoping — `ObjectManager` takes a scope predicate~~ — **CUT (2026-09-11)** -**Files:** -- Modify: `server/typescript/packages/cli/src/lib/dependency-transports.ts` -- Test: `cli/test/dependency-transports.test.ts` - -**Read first:** DESIGN §2.1 transports table (the `npm` and `python` rows, the interpreter ladder), §9 (the `find_spec` risk). - -**Interfaces:** -- Produces: `resolveTransport(spec, configDir)` handles `npm` (`createRequire(join(configDir, "package.json")).resolve("/package.json")` → its `dirname`, then `spec.dir` if set) and `python` (spawn the interpreter chosen by the ladder `process.env.METAOBJECTS_PYTHON` → `${process.env.VIRTUAL_ENV}/bin/python` → `/.venv/bin/python` → `python3`, with `-c "import importlib.util,sys; s=importlib.util.find_spec(sys.argv[1]); print(list(s.submodule_search_locations)[0])" `; the printed directory, then `spec.dir`); both must contain `MANIFEST_FILE` else `ERR_DEPENDENCY_UNRESOLVED` whose message names what was tried and, for `python`, which interpreter answered. `resolveTransport` returns `{ dir, detail }` where `detail` is the report suffix (`from npm @acme/model` / `from python acme_model via `). - -- [ ] **Step 1: Failing test** `dependency-transports.test.ts`: (a) `npm`: a temp app with `package.json` and `node_modules/@acme/model/package.json` (`{"name":"@acme/model","version":"1.0.0"}`) + `node_modules/@acme/model/metaobjects/metaobjects.pkg.json` → `resolveTransport({ name: "m", npm: "@acme/model", dir: "metaobjects", mode: "reference" }, app)` returns that directory; without `dir` and with the manifest at the package root, the root. A missing package → `ERR_DEPENDENCY_UNRESOLVED`. (b) `python`: write an executable stub script `/fake-python` (`#!/bin/sh`, prints `/site/acme_model` and exits 0), set `process.env.METAOBJECTS_PYTHON` to it for the test, place the manifest at `/site/acme_model/metaobjects/metaobjects.pkg.json` → resolves with `dir: "metaobjects"`; a stub that exits 1 → `ERR_DEPENDENCY_UNRESOLVED` with the interpreter path in the message. (c) opt-in real-interpreter test guarded by `Bun.which("python3") !== null`: `python: "json"` (stdlib package) resolves to a directory (no manifest → `ERR_DEPENDENCY_UNRESOLVED`, which proves the spawn path). Run → FAIL. Implement. Run → PASS; typecheck → PASS. - -- [ ] **Step 2: Commit** -``` -feat(cli): npm and python dependency transports (FR-023) -``` +Not built. The maintainer's "runtime" meant *what metadata gets loaded* — the collection +resolver of Task 8 — not an `ObjectManager` predicate, and he confirmed that path "already +works as is". Nothing in `runtime-ts` or `server/python/.../runtime/` changes in Phase 1a. +The task number is kept so the progress ledger's references stay resolvable. Trigger to +revisit: DESIGN §11.4 (an app that loads one collection but must serve only part of it). --- -### Task 17: The local co-development override (D10) — TS +### Task 17: The overlay authoring lint (`meta verify`, TS) and `declaredTopLevelKeys` **Files:** -- Modify: `server/typescript/packages/sdk/src/dependencies.ts` (`verifySnapshot` honours overrides) -- Modify: `server/typescript/packages/sdk/src/collection.ts` (`overrides`) -- Modify: `server/typescript/packages/cli/src/lib/collection-load-options.ts` (`warnLocalOverrides(collection)` — one `log.warn` per overridden dependency, called from `loadCollection`) -- Modify: `server/typescript/packages/cli/src/commands/deps.ts` (`sync` copies from the override; `check` and `verify --deps` refuse) -- Modify: `server/typescript/packages/cli/src/commands/init.ts` (`METAOBJECTS_GITIGNORE_BODY` gains `deps.local.json` under a comment "the local co-development override — never committed") -- Modify: `fixtures/dependency-conformance/cases.json` (append cases 47–49) -- Test: `sdk/test/dependency-conformance.test.ts`, `cli/test/deps-override.test.ts`, `cli/test/init.test.ts` - -**Read first:** DESIGN §10 D10 (every bullet is a requirement). Task 7 left the branch `local overrides are not supported yet` in `verifySnapshot` — replace it. - -**Interfaces:** -- Produces: with `.metaobjects/deps.local.json` = `{ "": { "path": "" } }`: `verifySnapshot` reads `/metaobjects.pkg.json` (must exist and parse → else `ERR_DEPENDENCY_UNRESOLVED`; its `name` must equal the dependency's; an override for a name not in `dependencies` → `ERR_DEPENDENCY_UNRESOLVED`), sets `artifactPath = /`, `nodes`/`packages`/`version`/`metamodelVersion` from THAT manifest (not the lock), `override = `, and skips the lock entry/hash check for that name (a missing lock entry for an overridden name is fine); `Collection.overrides` lists the names; the warning text is exactly `loading from a local override (), not the committed snapshot`. `deps sync` with an override active resolves that dependency from the override path instead of its transport (report suffix `from local override `) and writes the snapshot + lock as usual. `deps check` / `verify --deps` exit 1 with `local override(s) active: — remove .metaobjects/deps.local.json to verify the committed snapshot` before checking anything. `meta init` writes the gitignore line. +- Modify: `server/typescript/packages/metadata/src/loader/meta-data-loader.ts` (export `declaredTopLevelKeys(content: string, format: MetaDataFormat): Promise>` — the structural walk `_rootIsOverlayOnly` performs, generalized to return every top-level declaration's `(type, resolutionKey, overlay)`; `_isOverlayOnlySource` is re-expressed over it) +- Modify: `server/typescript/packages/metadata/src/index.ts` (export) +- Create: `server/typescript/packages/cli/src/lib/overlay-lint.ts` (`lintOverlays(collection, readSource) → Diagnostic[]`; `WARN_OVERLAY_IMPLICIT`) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (its own section after the requirement lint; `--no-overlay-lint`; `META_NO_OVERLAY_LINT`; structured rows with source `"lint"`) +- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`VERIFY_OPTIONS["no-overlay-lint"]`) +- Test: `metadata/test/declared-top-level-keys.test.ts`, `cli/test/verify-overlay-lint.test.ts`, `cli/test/help-lists-every-flag.test.ts` -- [ ] **Step 1: Append corpus cases**: - - `an-active-override-replaces-the-snapshot-and-skips-its-hash-check`: tree `APP` + `treeFiles` snapshot → `artifacts/acme-common-v1.json` AND `lib/metaobjects/acme-common.metaobjects.json` → `artifacts/acme-common-v1-widened.json` + `lib/metaobjects/metaobjects.pkg.json` (manifest text with the widened hash); `CONFIG_REF`; `LOCK_V1` with a WRONG integrity (`sha256-0000…0000`, 64 zeros); `localOverrides: { "acme-common": { "path": "lib/metaobjects" } }` → `expectFiles: ["lib/metaobjects/acme-common.metaobjects.json", "metaobjects/meta.app.json"]`, `expectOverrides: ["acme-common"]`, `expectForeign` the three. - - `an-override-whose-path-lacks-a-manifest-is-unresolved`: same but no manifest file at the override path → `expectError: "ERR_DEPENDENCY_UNRESOLVED"`. - - `an-override-for-an-undeclared-dependency-is-unresolved`: `localOverrides: { "ghost": { "path": "lib" } }` with `CONFIG_REF` + `LOCK_V1` + `SNAP` → `expectError: "ERR_DEPENDENCY_UNRESOLVED"`. - Run → FAIL (the Task 7 stub branch throws for case 47). +**Read first:** DESIGN §11.1 item 4; Global Constraints "Overlay lint". `meta-data-loader.ts:334-420` (the walker; the resolution key of a root child is its own `package` if present else the root's, `::`, its `name` — mirror `rootChildResolutionKey` in `parser-core.ts`). `requirement-lint.ts` + `verify.ts:661-690` (the section, cap and mute pair to copy). `parser-core.ts:1094-1100`: the parser silently merges an unflagged redeclaration — the lint exists because of that line; it does not change it. -- [ ] **Step 2: Failing cli tests** `deps-override.test.ts`: scaffold as Task 13 with the snapshot synced; write `deps.local.json` pointing at `../lib/metaobjects` after replacing the lib artifact with the widened one (+ manifest); (a) `genCommand([], app)` stderr contains the exact warning line and the emitted model reflects the override (assert via `verify --codegen`? simpler: `exportCommand` output contains `"@maxLength": 200`); (b) `depsCommand(["check"], app)` exits 1 with `local override(s) active: acme-common`; `verifyCommand(["--deps"], app)` likewise; (c) `depsCommand(["sync"], app)` exits 0, report contains `from local override`, the snapshot now holds the widened bytes and the lock the widened hash; (d) delete `deps.local.json` → `check` exits 0. `init.test.ts`: the scaffolded `.metaobjects/.gitignore` contains a line `deps.local.json`. Run → FAIL. +**Interfaces:** `lintOverlays` reads every file in `collection.files` (artifacts first — they are the bases), collects `(type, key) → [{ file, overlay }]` in load order, and for each key with two or more declarations reports every unflagged declaration after the FIRST unflagged one: `Diagnostic { severity: "warn", code: WARN_OVERLAY_IMPLICIT, message }` with the message in the Global Constraints (`file` is the collection-relative path, or the `dep:` id for an artifact). Unreadable or unparsable files are skipped (the loader reports those). -- [ ] **Step 3: Implement** all of the above. +- [ ] **Step 1: Failing tests.** `declared-top-level-keys.test.ts`: a JSON root with `package: "app"` and children `object.entity Order` (no flag) and `object.entity Order` (`overlay: true`, own `package: "acme::common"`) returns `[{ type: "object", key: "app::Order", overlay: false }, { type: "object", key: "acme::common::Order", overlay: true }]`; a YAML source (sigil-free) with `overlay: true` on one child returns `overlay: true` for it. `verify-overlay-lint.test.ts`: (a) consumer = `APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1` plus `metaobjects/meta.ov.json` redeclaring `acme::common::Customer` WITHOUT the flag (adds a `view.text` child) → `meta verify` exits 0 (advisory) and stderr contains `acme::common::Customer is redeclared in metaobjects/meta.ov.json without overlay: true`; (b) with `overlay: true` → no such line; (c) the three-way case: `meta.a.json`, `meta.b.json`, `meta.c.json` all declaring `app::Subscriber` unflagged → TWO findings naming `meta.b.json` and `meta.c.json`; (d) `--no-overlay-lint` silences (a). Run → FAIL. -- [ ] **Step 4: Run** +- [ ] **Step 2: Implement.** Run ```bash -cd server/typescript/packages/sdk && bun test test/dependency-conformance.test.ts test/dependencies.test.ts -cd server/typescript/packages/cli && bun test test/deps-override.test.ts test/deps-sync.test.ts test/deps-check.test.ts test/init.test.ts test/ignored-scaffold-check.test.ts -bun run --filter '@metaobjectsdev/sdk' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck +cd server/typescript/packages/metadata && bun test test/declared-top-level-keys.test.ts test/loader-overlay-partition.test.ts +cd server/typescript/packages/cli && bun test test/verify-overlay-lint.test.ts test/help-lists-every-flag.test.ts test/verify-requirements-e2e.test.ts +bun run --filter '@metaobjectsdev/metadata' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck ``` -Expected: PASS (cases 47–49 green). +Expected: PASS (`loader-overlay-partition` or whichever existing test covers `_partitionOverlayLast` must stay green — find it with `grep -rl partitionOverlay metadata/test`). -- [ ] **Step 5: Commit** +- [ ] **Step 3: Commit** ``` -feat(sdk,cli): local co-development override via .metaobjects/deps.local.json (FR-023 D10) +feat(cli): meta verify reports an unflagged cross-file redeclaration as an overlay authoring finding (FR-023 §11) ``` --- -**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** The collection resolver's fast path (no declared dependencies and no lock → skip snapshot verification) must ALSO require that no override is present. A `deps.local.json` entry naming a dependency that is not declared → `ERR_DEPENDENCY_UNRESOLVED`, never silently ignored. Add that test. - -### Task 18: Python — the collection resolver, CLI, runner, boundary validator, and D10 +### Task 18: Python — the collection resolver, CLI load path, codegen selection, and the corpus runner **Files:** -- Modify: `server/python/src/metaobjects/config/dependencies.py` (`verify_snapshot`, `foreign_owner_factory`, `validate_dependency_boundary`, `declared_overlay_keys`, `read_local_overrides` use) -- Modify: `server/python/src/metaobjects/config/source_resolver.py` (`resolve_collection` returns `Collection`; keep `resolve_collection_files(root) -> list[Path]` as a thin wrapper for existing callers) -- Modify: `server/python/src/metaobjects/cli.py` (`resolve_metadata_location` returns the `Collection` at every rung; `_load_root_from_paths` → `_load_root_from_collection` building `FileSource(path, id=…)` and calling `loader.load(sources)`; boundary check after every load; `run_gen(..., governs=…)`; `entities:` naming a reference-mode foreign node → exit 2; the override warning printed once per command) -- Modify: `server/python/src/metaobjects/codegen/runner.py` (`run_gen(..., governs: Callable[[str], bool] | None = None)`) -- Create: `server/python/tests/conformance/test_dependency_conformance.py` +- Modify: `server/python/src/metaobjects/config/dependencies.py` (`verify_snapshot`, `imported_from`, `explicitly_includes`, `Collection` dataclass: `files`, `own_files`, `file_ids`, `dependencies`, `imported_packages`, `imported(fqn)`, `in_scope(fqn)`, `in_migrate_scope(fqn) | None`) +- Modify: `server/python/src/metaobjects/config/neutral_config.py` (read `scope.include` / `scope.exclude` and `migrate.scope` into the neutral subset — READ ONLY for the explicit-include rule; the Python CLI still does not apply the user's `scope` to its own objects — CONFIRMED 2026-09-11 against the tree: `server/python/src` has no caller of `matches_scope` / `compile_scope` and `neutral_config.py` parses no `scope` key today, so applying it would be NEW behaviour, not parity) +- Modify: `server/python/src/metaobjects/config/source_resolver.py` (NEW `resolve_collection_full(root) -> Collection`; `resolve_collection(root) -> list[Path]` becomes a thin projection of it — public shape unchanged, per the T18 ruling) +- Modify: `server/python/src/metaobjects/cli.py` (`resolve_metadata_location` returns the `Collection` at every rung — rung 2 (`metadata:` in `metaobjects.config.yaml`) builds it from that path AND the neutral config's dependencies; `_load_root_from_paths` → `_load_root_from_collection`, building `FileSource(path, id=…)` for each file and calling `MetaDataLoader(...).load(sources)`; `run_gen(..., select=collection.in_scope)`; a target's `entities:` (or `--entities`) naming an object with `imported and not in_scope` → `error: '' () is imported from dependency '' and is not generated here — add its package to scope.include in .metaobjects/config.json`, exit 2) +- Modify: `server/python/src/metaobjects/codegen/runner.py` (`run_gen(..., select: Callable[[str], bool] | None = None)` — applied after `entity_filter`, over `o.resolution_key()`; the shared-enum pass already takes the selected entities, so it follows the using entity — parity with Task 10) +- Create: `server/python/tests/conformance/test_dependency_conformance.py` (the corpus runner: all arms; exhaustive set assertions) - Test: `tests/config/test_dependencies.py` (extend), `tests/codegen/test_cli_dependencies.py` (new) -**Read first:** DESIGN §4.2, §4.4, §4.5 (Python rows), §10 D10. The TS reference: `sdk/src/dependencies.ts` + `collection.ts` after Task 17 — port the behavior, including every error code and the table in Task 7. `cli.py` `resolve_metadata_location` (rungs 2–4) and `_load_root_from_paths`; `runner.py` `_objects`. +**Read first:** DESIGN §4.2, §11.1 item 2, §11.2 row "Python". Task 8's TS implementation (mirror it; the corpus is the contract). `cli.py:243-330` (`_load_root` / `_load_root_from_paths`), `cli.py:557-600` (`resolve_metadata_location`), `runner.py:57-80`, `file_source.py` (`id=`, Task 3). Pre-flight ruling T18 (thin projection; shared-enum parity). -**Interfaces:** -- Produces: `source_resolver.Collection` dataclass: `config_dir: Path`, `files: list[Path]`, `own_files: list[Path]`, `file_ids: dict[Path, str]`, `dependencies: list[ResolvedDependency]`, `overrides: list[str]`, `foreign_owner(fqn) -> ResolvedDependency | None`, `governs(fqn) -> bool`. `run_gen(..., governs=None)` filters `_objects` by `governs(o.resolution_key())`. `validate_dependency_boundary(root, collection, overlay_keys)` raises `ParseError` with the same codes and rules as Task 8. `declared_overlay_keys(path: Path) -> set[str]` parses the raw file (`json.loads` for `.json`; `yaml.safe_load` for `.yaml`/`.yml`, root key `metadata` or `metadata.root`) and returns resolution keys of top-level nodes with `overlay: True`. +**Interfaces:** `imported(fqn) = package_of_resolution_key(fqn) in imported_packages`; `in_scope(fqn) = (not imported(fqn)) or explicitly_includes(scope_include, package_of_resolution_key(fqn))` — NOTE: unlike TS this does NOT apply the user's `scope` patterns to the project's own objects (the Python CLI has never applied `scope`; RULED 2026-09-11 to stay as is). Python's load path mirrors TS's `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` refusal after building the root. `in_migrate_scope` mirrors TS for completeness (nothing in the Python CLI consumes it — schema is TS-owned, ADR-0015). -- [ ] **Step 1: Write the Python corpus runner** `tests/conformance/test_dependency_conformance.py` mirroring the TS runner (materialize `tree`, `treeFiles`, `config`, `lock`, `localOverrides`; for `expectFiles` compare the set of files relative to the temp root; `expectForeign`/`expectGoverned`/`expectOverrides`; `expectLoadError` loads via the same code path `cli._load_root_from_collection` and asserts the first error code; `expectError` asserts `ParseError.code`; the `classify` arm is `pytest.skip("Task 19")` for now). For `dependencies-are-read-under-a-native-source-surface` ALSO write `app/metaobjects.config.yaml` containing `metadata: metaobjects\ntargets:\n t:\n outDir: out\n generators: [names]\n` and resolve through `cli.resolve_metadata_location(load_project_config(...), root)` to prove rung 2 still adds the artifact. Run `cd server/python && uv run --extra integration pytest tests/conformance/test_dependency_conformance.py -q` → FAIL. +- [ ] **Step 1: The runner.** `test_dependency_conformance.py` reads `fixtures/dependency-conformance/cases.json`; materializes `tree` / `treeFiles` / `config` / `lock`; `expectError` → `resolve_collection_full` raises with that code; `expectFiles` → set equality on relative paths; `expectImported` / `expectSelected` / `expectMigrateGoverned` → EXHAUSTIVE set equality over every loaded top-level object (`root.own_children()` of type object — ADR-0039 sanctioned own: root-level scan); `expectLoadError` / `expectErrorFiles` → load via the CLI's load path. Run → FAIL on every dependency case. -- [ ] **Step 2: Failing CLI tests** `tests/codegen/test_cli_dependencies.py`: scaffold a consumer (pinned v1 artifact under `.metaobjects/deps/acme-common/`, `CONFIG_REF`, `LOCK_V1`, `APP`, `metaobjects.config.yaml` with a `names` target); (a) `metaobjects gen` (call `cli.main([...])` with cwd set) writes `order_names.py` and NOT `customer_names.py`; (b) `entities: [Customer]` in the target → exit 2, stderr contains `declared by dependency 'acme-common' (reference mode)`; (c) corrupt the lock hash → `gen` exits non-zero BEFORE writing anything, stderr contains `ERR_DEPENDENCY_SNAPSHOT_STALE`; (d) with `deps.local.json` pointing at a widened lib dir (+ manifest), `gen` succeeds and stderr contains the exact warning line from D10; (e) `verify --codegen` shares the selection (no drift after (a)). Run → FAIL. +- [ ] **Step 2: Failing CLI tests** (`tests/codegen/test_cli_dependencies.py`): consumer = `APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1` with a `metaobjects.config.yaml` (`metadata: metaobjects`, one target `generators: [names]`): `metaobjects gen` writes `order_names.py` and NOT `customer_names.py`; with `scope.include` naming `acme::common` in config.json, `customer_names.py` IS written; a target `entities: [Customer]` without the include → exit 2 with the message; a stale snapshot (edit one byte) → the stale error before generation; `metaobjects verify --codegen` shares the selection. Run → FAIL. -- [ ] **Step 3: Implement** everything listed under Files. Keep `resolve_collection_files` so `cli.py` callers outside the ladder are untouched. - -- [ ] **Step 4: Run** +- [ ] **Step 3: Implement.** Run ```bash -cd server/python && uv run --extra integration pytest tests/conformance/test_dependency_conformance.py tests/config tests/codegen/test_cli_dependencies.py tests/codegen/test_cli_config_gen.py tests/codegen/test_cli_config_verify.py tests/conformance/test_source_resolution_conformance.py -q +cd server/python && uv run --extra integration pytest tests/conformance/test_dependency_conformance.py tests/config tests/codegen/test_cli_dependencies.py tests/codegen -q ``` -Expected: PASS (cases 1–28 and 47–49 green in Python; 29–46 skipped). +Expected: PASS — every corpus case green in Python. -- [ ] **Step 5: Commit** -``` -feat(python): dependency snapshot loading, foreign-node selection, boundary validator, and the local override (FR-023) -``` - ---- - -**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** `resolve_collection_files` does not exist. Add a new resolver that returns the full result (files + resolved dependencies + foreign predicate inputs) and keep `resolve_collection(root) -> list[Path]` as a thin projection of it, so its existing shape and callers are unchanged. ALSO (cross-port parity with Task 9): Python's materialized shared-enum pass in `entity_model.py` must exclude foreign `reference`-mode enums, exactly as the TS shared-enums artifact does; add the Python test mirroring Task 9's. - -### Task 19: Python — the `shared-model` generator, `targets..options`, and the classifier twin - -**Files:** -- Create: `server/python/src/metaobjects/codegen/generators/shared_model.py` -- Modify: `server/python/src/metaobjects/codegen/generator_registry.py` (entry `shared-model`) -- Modify: `fixtures/generator-registry-conformance/registry.json` (`shared-model.ports` gains `"python"`) -- Modify: `server/python/src/metaobjects/codegen/project_config.py` (`TARGET_KEYS` gains `"options"`; `TargetConfig.options: dict[str, dict[str, object]]`), `server/python/src/metaobjects/codegen/metaobjects-config.schema.json` (`options` object keyed by generator name) -- Modify: `server/python/src/metaobjects/cli.py` (config-mode gen passes `options[]` to the factory when the registry entry accepts options; `shared-model` needs `name`/`include`/`exclude?`/`files?`/`version?`; `GenContext` gains `registry` and `source_files`) -- Modify: `server/python/src/metaobjects/codegen/generator.py` (`GenContext.registry`, `GenContext.source_files`) -- Create: `server/python/src/metaobjects/dependency_diff.py` (`classify_dependency_changes`, `WIDENING_ATTRS`, `ADDITIVE_ATTRS`, `DOC_ATTRS`) -- Modify: `server/python/src/metaobjects/config/dependencies.py` (`dependency_footprint`) -- Test: `tests/codegen/test_shared_model.py`, `tests/unit/test_dependency_diff.py`, `tests/conformance/test_dependency_conformance.py` (enable the classify arm), `tests/codegen/test_cli_registry.py` (existing — stays green) - -**Read first:** DESIGN §4.3, §4.6, §2.6; the TS implementations from Tasks 12 and 14 are the reference — port behavior and messages, not structure. `serialize_shared_document` (Task 2), `scope.py` (Task 4), `sha256_integrity` (Task 6). - -**Interfaces:** -- Produces: registry name `shared-model`; `shared_model(name, include, exclude=None, files=None, version=None) -> Generator`; `classify_dependency_changes(old_doc: str, new_doc: str, footprint: dict[str, str]) -> list[Change]` (`Change` dataclass: `fqn, path, kind, before, after, rule`); `dependency_footprint(root, collection) -> dict[str, tuple[ResolvedDependency, str]]`. `version` defaults to `pyproject.toml` `[project].version` found walking up from the config dir (use `tomllib`). - -- [ ] **Step 1: Failing generator test** `tests/codegen/test_shared_model.py`: a temp publisher with `metaobjects/meta.lib.json` (Task 2's `LIB`), an overlay file under `metaobjects/admin-ui/`, a `pyproject.toml` with `version = "1.0.0"`, and `metaobjects.config.yaml`: -```yaml -metadata: metaobjects -targets: - shared: - outDir: out/shared - generators: [shared-model] - options: - shared-model: - name: acme-common - files: [metaobjects/meta.lib.json] - include: ["acme::common::**"] -``` -`metaobjects gen` → `out/shared/acme-common.metaobjects.json` is byte-identical to the pinned v1 artifact and `metaobjects.pkg.json` equals the manifest from Task 6's test data; a closure failure (`include: ["acme::common::Address"]` with `Address` extending `Audited`) exits 1 with `references acme::common::Audited, which is not selected`; a consumer-provider attr in the selection fails with `core registry does not register`. Run → FAIL. Implement the generator, config `options`, `GenContext` fields, registry entries. Run → PASS, and: -```bash -cd server/python && uv run --extra integration pytest tests/codegen/test_shared_model.py tests/codegen/test_cli_registry.py tests/codegen/test_constants_config.py -q -cd server/typescript/packages/codegen-ts && bun test test/generator-registry.test.ts -``` -Expected: PASS (both ports now list `shared-model`). - -- [ ] **Step 2: Failing classifier tests** `tests/unit/test_dependency_diff.py` — the widening rows both directions, `DOC_ATTRS` ignored, and the footprint test mirroring Task 14 Step 3 (`{Audited: whole, Address: whole, Customer: key}`). Enable the `classify` arm in the Python corpus runner. Run → FAIL. Implement `dependency_diff.py` and `dependency_footprint`. Run: -```bash -cd server/python && uv run --extra integration pytest tests/unit/test_dependency_diff.py tests/conformance/test_dependency_conformance.py -q -``` -Expected: PASS (cases 29–46 green in Python). - -- [ ] **Step 3: Commit** -``` -feat(python): shared-model generator, per-target generator options, and the change classifier (FR-023) -``` - ---- - -### Task 20: The requirements ledger and `verify --templates` skip foreign nodes - -**Files:** -- Modify: `server/typescript/packages/cli/src/lib/requirement-check.ts` (`scanRequirements(root, opts?: { governs?: (fqn: string) => boolean })`) -- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (pass `governs`; the templates gate filters `template.*` nodes by `governs`) -- Test: `cli/test/verify-requirements-foreign.test.ts` (new) - -**Read first:** DESIGN §2.7 (ledger and templates rows). `verify.ts` lines around `scanRequirements(root)` / `summariseRequirements` and the `--templates` loop over `template.*` nodes. - -- [ ] **Step 1: Failing test** — scaffold a consumer whose artifact (case-local, in the test) carries `requirement.functional` node `acme::common::CustomerCanBeCreated` (`@implementedBy: acme::common::Customer`, `@status: live`) beside the v1 nodes (compute its hash for the lock), and a local model with one entity `app::Order` and no requirements. Run `verifyCommand([], app)` and assert the requirements line reads `… 0 entries … 0/1 entities claimed, counted over 2 metadata file(s), 1 from dependencies` — the foreign requirement and the foreign entity are not counted. Then add a local `requirement.functional` with `@implementedBy: acme::common::Customer` → loads clean, `0/1 entities claimed` still (a foreign entity is never in the denominator). Run → FAIL. Implement. Run → PASS; `bun test test/verify-requirements-e2e.test.ts test/docs-requirements-surface.test.ts` → PASS; typecheck → PASS. - -- [ ] **Step 2: Commit** +- [ ] **Step 4: Commit** ``` -feat(cli): verify's ledger and template gates skip dependency-owned nodes (FR-023) +feat(python): dependencies resolve, load first and are excluded from gen unless scope.include names their package (FR-023 §11) ``` --- -**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** DROP the `verify --templates` `governs` filter from this task: `sharedModelFile` / `shared-model` never exports `template.*` nodes, so the filter could never fire. This task keeps only the requirements-ledger exclusion (foreign nodes are not counted in `entitiesTotal`, a foreign `requirement.*` is not the consumer's claim). - -### Task 21: `meta init` stops scaffolding `package.meta.json` and negates the dependency files in `.gitignore` +### Task 19: `meta init` stops scaffolding `package.meta.json` and tracks the dependency files **Files:** -- Modify: `server/typescript/packages/cli/src/commands/init.ts` (remove the `package.meta.json` scaffold block; remove `!package.meta.json` from `METAOBJECTS_GITIGNORE_BODY`; add `!deps/` and `!deps.lock.json` under the "These ARE meant to be tracked" comment — `deps.local.json` was added in Task 17) -- Modify: `server/typescript/packages/cli/test/init.test.ts` (flip the manifest test: asserts the file is NOT created; a pre-existing one is left untouched) +- Modify: `server/typescript/packages/cli/src/commands/init.ts` (remove the `package.meta.json` scaffold block at ~706-716; remove `!package.meta.json` from `METAOBJECTS_GITIGNORE_BODY`; add `!deps/` and `!deps.lock.json` under the "These ARE meant to be tracked" comment; print one line when a pre-existing `package.meta.json` is found: `note: .metaobjects/package.meta.json is deprecated (nothing reads it; removed in 2.0) — see docs/features/metadata-dependencies.md`) +- Modify: `server/typescript/packages/cli/test/init.test.ts` (flip the manifest test: the file is NOT created; a pre-existing one is left untouched; the `.gitignore` body contains the two negations) - Modify: `server/typescript/packages/cli/README.md` (the `meta init` scaffold list no longer names `package.meta.json`) -**Read first:** DESIGN §0a (deprecation row), §10 ruling; the sdk `@deprecated` JSDoc on `package.ts`/`workspace.ts` is ALREADY DONE (`ef51fd1e1`) — do not touch sdk here. - -- [ ] **Step 1: Flip the test** — `test("does not scaffold package.meta.json (deprecated; removed in 2.0)")`: `existsSync(join(cwd, ".metaobjects", "package.meta.json"))` is `false` after `init({ cwd })`; a second test writes a stale one first and asserts `init` preserves it byte-for-byte. Also assert `.metaobjects/.gitignore` contains `!deps/`, `!deps.lock.json` and `deps.local.json`. Run `cd server/typescript/packages/cli && bun test test/init.test.ts` → FAIL. Implement. Run → PASS; `bun test test/init-layout.test.ts test/unit/init-scaffold-config.test.ts test/ignored-scaffold-check.test.ts` → PASS. +**Read first:** DESIGN D9, §5.1 "cli" (init), pre-flight ruling T21 (tests go in the existing `init.test.ts`). `sdk/src/package.ts` is already `@deprecated` — no sdk change. -- [ ] **Step 2: Commit** +- [ ] **Step 1: Failing test** — flip `init.test.ts` as above. Run `cd server/typescript/packages/cli && bun test test/init.test.ts` → FAIL. +- [ ] **Step 2: Implement.** Run the same + `bun run --filter '@metaobjectsdev/cli' typecheck` → PASS. +- [ ] **Step 3: Commit** ``` -chore(cli): meta init no longer scaffolds package.meta.json; tracks deps/ and deps.lock.json (FR-023) +feat(cli): meta init tracks .metaobjects/deps and deps.lock.json and no longer scaffolds package.meta.json (FR-023) ``` --- -**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** `cli/test/init-layout.test.ts` does not exist. Put the tests in the existing `init` test file under `server/typescript/packages/cli/test/` (find it with `git ls-files | grep init`). - -### Task 22: Docs, skills, CHANGELOG, CONFORMANCE matrix, roadmap, agent-context bundle +### Task 20: Docs, skills, CHANGELOG, CONFORMANCE matrix, roadmap, agent-context bundle **Files:** -- Create: `docs/features/metadata-dependencies.md` -- Modify: `docs/features/metadata-sources.md` (the ladder paragraph; "Vendoring — airgapped and hermetic builds"; "The workspace `extends:` walk is retired" gains the deprecation schedule) -- Modify: `docs/features/abstracts-and-inheritance.md` ("Overlays vs. extends — different concepts" gains the cross-repository paragraph and the `overlay: true` rule) -- Modify: `docs/features/cli.md` (matrix row + the `verify --deps` subverb row) -- Modify: `docs/CONFORMANCE.md` (corpus row + per-corpus section) +- Create: `docs/features/metadata-dependencies.md` — declare → `deps sync` → reference / extend / overlay (with `overlay: true`, always, in an overlay-only file by convention) → what is excluded by default and how `scope.include` / `migrate.scope` re-admit a package (the legacy "own the shared model" case) → what fails at load (the Task 9 table, and `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` when you declare a new node into a dependency's package) and at `verify --deps` → publishing with `sharedModelFile()` → vendoring vs depending → the legacy resources-JAR pattern, for readers who knew it → deferred: transports, override, classifier, bindings +- Modify: `docs/features/metadata-sources.md` (the ladder paragraph gains "dependencies are read at every rung"; "Vendoring" points at dependencies; "The workspace `extends:` walk is retired" gains the 2.0 removal note) +- Modify: `docs/features/abstracts-and-inheritance.md` ("Overlays vs. extends" gains the cross-repository paragraph and the explicit-flag rule) and `docs/features/entities.md` (the same rule where entities are introduced — pre-flight ruling T22) +- Modify: `docs/features/cli.md` (matrix rows: **Dependency sync** (`deps`) | Node `meta` | any backend; `verify --deps`; `verify`'s overlay lint) +- Modify: `docs/CONFORMANCE.md` (the dependency-conformance row: case count, TS + Python runners; and the Task 4 carried minor — the scope-conformance section names `metadata/src/scope.ts` as the reference and lists TS + Python runners) and `fixtures/scope-conformance/README.md` (same minor) - Modify: `docs/README.md` (layout + "when to read which" rows) -- Modify: `spec/roadmap.md` (FR-023 status cell → Phase 1a shipped; the "doc-first quick wins" bullet closes) -- Modify: `agent-context/skills/metaobjects-authoring/SKILL.md`, `agent-context/skills/metaobjects-codegen/SKILL.md`, `agent-context/skills/metaobjects-verify/SKILL.md` -- Modify: `CHANGELOG.md` (`[Unreleased]`) -- Regenerate: the bundled agent context (`bun run --filter '@metaobjectsdev/sdk' bundle-agent-context`) and whatever `fixtures/agent-context-conformance/` needs (`cd server/typescript/packages/sdk && bun test test/agent-context-conformance.test.ts` tells you) - -**Read first:** DESIGN §0, §2.1–§2.7, §2.9, §10 (rulings 2, 3, 5 and D10 — the skills content is a ruling), §8 task 21's sentence: the skills teach, as ordinary work, (1) extending and overlaying a foreign node with `overlay: true`, (2) widening a `sharedModelFile` selection or splitting it into several artifacts, (3) co-developing with a `deps.local.json` override. Public-repo hygiene applies to every line. - -- [ ] **Step 1: Write `docs/features/metadata-dependencies.md`** with these sections, each with a runnable example: Declaring a dependency (config shape; transports `path`/`npm`/`python`; `mode`); Syncing (`meta deps sync`, what is committed: `deps/` + `deps.lock.json`; `meta deps list`); Referencing, extending and overlaying (FQN references; `extends` a foreign abstract; `overlay: true` is required — the exact `ERR_DEPENDENCY_OVERLAY_IMPLICIT` message; reference mode refuses structural overlays — `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`; own mode); What foreign means for each command (the §2.7 table); What breaks when upstream changes (the §2.5 table; the classifier: footprint scopes, the breaking/compatible/ignored/info rules, the widening and additive tables verbatim; `meta deps check` / `verify --deps`; `sync --accept-breaking`); Publishing (`sharedModelFile()` for TS and the `shared-model` target for Python, `files`/`include`/`exclude`, closure, core-vocabulary rule, where the artifact ships per ecosystem, "start with one artifact; widen `include` or add a second `sharedModelFile` when a consumer needs a different slice"); Co-developing (D10 — the file, the warning, sync-from-override, verify refusal, gitignore); Errors (the nine codes, one row each); The legacy pattern (three sentences for readers who knew the resources-only JAR). - -- [ ] **Step 2: Edit the existing docs** as listed (metadata-sources: "`dependencies` is read at every rung, including under a port's native surface"; cli.md row: `| **Dependency sync** (`deps`) | **Node `meta`** | `meta deps sync` / `check` / `list` | **any backend** — sync/check are Node-only (ADR-0015 pattern); every port loads the committed snapshot |` and the subverb row `| `verify --deps` | **Dependency drift** — the installed dependency versus the committed snapshot, classified against this consumer's footprint | no |`; CONFORMANCE row `| [`fixtures/dependency-conformance/`](../fixtures/dependency-conformance/) | 49 cases (31 resolution/load + 18 classify) | ✓ (reference implementation) | — (Phase 2) | — (Phase 2) | — (Phase 2) | ✓ |`). - -- [ ] **Step 3: Skills.** In `metaobjects-authoring/SKILL.md` under the `overlay` paragraph add "Building on another repository's model": a consumer declares a dependency, references by FQN, `extends` foreign abstracts, and overlays with `overlay: true` (required); an overlay in reference mode adds presentation, validators and docs, never structure; when upstream removes or renames the target the build fails — that failure is the feature. In `metaobjects-codegen/SKILL.md` add "Publishing a shared model": the `sharedModelFile()` / `shared-model` example, and the two ordinary operations "widen `include`" and "add a second `sharedModelFile` with its own `name` for a different audience" — with the rule that a node not in the selection is not a contract. In `metaobjects-verify/SKILL.md` add "Dependency drift": `verify --deps`, reading a BREAKING report (` : breaking ()`), `meta deps sync --accept-breaking ` and then fixing the model, and the D10 loop (write `deps.local.json`, expect the warning, `sync` to land, delete the file before `verify --deps`). +- Modify: `spec/roadmap.md` (FR-023 status cell → Phase 1a shipped per DESIGN §11; the "doc-first quick wins" bullet closes; the §299 description row is updated to the B+ shape) +- Modify: `agent-context/skills/metaobjects-authoring/SKILL.md` (the `overlay` section: extending and overlaying a dependency's nodes is the expected way to build on a shared model; `overlay: true` on every such amendment; what fails when upstream changes; a local node in the dependency's package is the consumer's own), `metaobjects-codegen/SKILL.md` (`sharedModelFile()`; the default-exclusion rule and `scope.include`), `metaobjects-verify/SKILL.md` (`verify --deps`, the stale-snapshot error, the overlay lint) +- Modify: `CHANGELOG.md` `[Unreleased]` — Added (`dependencies`, `meta deps sync|check|list`, `verify --deps`, `sharedModelFile()`, `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, the overlay lint, `declaredTopLevelKeys`, `serializeSharedDocument`, `FileSource` `id`); Changed (`Collection` members; `inScope` / `inMigrateScope` compose the default-exclusion rule — byte-identical for a project without dependencies; scope grammar moved to `metadata`; `scopeExpectedSchema` third argument; `scanRequirements` option); Deprecated (`package.meta.json` scaffold; sdk `package.ts` / `workspace.ts` exports — removal at 2.0). `metamodelVersion` stays `1.0` — say so. +- Regenerate the agent-context corpus in the SAME commit (`node scripts/bundle-agent-context.mjs` or the documented command — the sdk `agent-context/bundle.test.ts` asserts it). -- [ ] **Step 4: CHANGELOG `[Unreleased]`** — Added: `dependencies` in `.metaobjects/config.json`; `meta deps sync|check|list`; `meta verify --deps`; `sharedModelFile()` / `shared-model`; `.metaobjects/deps.local.json`; the nine codes; `fixtures/dependency-conformance/`. Changed: `FileSource` takes an `id` option; `Collection` gains `dependencies`, `ownFiles`, `fileIds`, `overrides`, `foreignOwner`, `governs`; `loadMemory` takes `fileIds`; the scope grammar is exported from `@metaobjectsdev/metadata` (sdk re-exports); `renderSharedEnumsFile` takes an `exclude` option; `GenContext` gains `registry` and `sourceFiles`; Python `targets..options`. Deprecated: `meta init` no longer writes `package.meta.json` (the sdk exports were deprecated in `ef51fd1e1`; removal at 2.0). State: `metamodelVersion` stays `1.0` — no registered vocabulary changed. +**Read first:** DESIGN §5.3, §11. `docs/features/metadata-sources.md` §"Vendoring" and §"retired". The skills' existing `overlay` sections (grep `overlay`). Public-repo hygiene: no private names, no home paths. -- [ ] **Step 5: Regenerate and verify** +- [ ] **Step 1: Write** every file above. +- [ ] **Step 2: Run** ```bash -bun run --filter '@metaobjectsdev/sdk' bundle-agent-context -cd server/typescript/packages/sdk && bun test test/agent-context-conformance.test.ts test/agent-context-capability-grounding.test.ts test/dogfood-examples.test.ts -cd server/typescript/packages/cli && bun test test/help-lists-every-flag.test.ts test/docs-command.test.ts +cd server/typescript/packages/sdk && bun test test/agent-context +cd scripts && bun test site/counts.test.ts +node scripts/check-metamodel-version.mjs ``` -Expected: PASS; if the agent-context corpus needs regeneration, follow the instruction its failing test prints and commit the regenerated files in this same commit. - -- [ ] **Step 6: Commit** +Expected: PASS; the metamodel-version gate reports no diff. +- [ ] **Step 3: Commit** ``` -docs(fr-023): metadata dependencies — feature page, skills, CLI matrix, conformance matrix, changelog +docs: metadata dependencies — declare, sync, build on, and what fails when upstream moves (FR-023 §11) ``` --- -**Pre-flight ruling (controller, 2026-09-11 — binding over the text above):** Add `docs/features/entities.md` to this task's file list (the design's §5.3 names it; the file exists). - -### Task 23: The gate - -**Files:** none (a verification task). If anything is red, fix it in a follow-up commit on this branch and re-run. - -- [ ] **Step 1: Cleaned tree.** `git status --short` must be empty (everything committed). Run `git clean -fdX -n` to see what a clean CI tree would drop; if generated artifacts you depend on appear, they are untracked build state and must not be needed by any test. +### Task 21: The gate -- [ ] **Step 2: Run the three lanes** +- [ ] **Step 1:** From the repo root, on the tree that will land (rebased on `origin/main` first — gate the tree that lands, not the branch as it was): ```bash scripts/ci-local.sh --only ts scripts/ci-local.sh --only python scripts/ci-local.sh --only gates ``` -Expected: each exits 0. The `gates` lane includes `node scripts/check-metamodel-version.mjs` (must report no vocabulary diff), `publish-set parity`, `ci lane selection`, and the fixture lints (`ERROR-CODES.json` codes referenced by the new corpus exist in the registry). +Expected: all three green. `gates` includes `check-metamodel-version.mjs` (no diff), `no-hardcoded-metadata-dir` (no allowlist change), `publish-set parity` (unchanged), `ci lane selection`. -- [ ] **Step 3: Cross-check the design's discipline list** (DESIGN §2.11): grep the diff of this branch against `main` for `instanceof Meta` outside `server/typescript/packages/metadata/src` (expected: none), for `own(Children|Attrs|Fields)\(` without an ADR-0039 comment on the same or previous line (expected: none new), for `/home/` and `~/` (expected: none), and for the private names the public-repo hook denies (`git config hooks.denyListPath` names the list; run the hook: `.githooks/pre-commit` against a no-op commit). +- [ ] **Step 2:** Confirm the deletions held: +```bash +grep -rn 'OVERLAY_IMPLICIT\|SCHEMA_NOT_OWNED\|BREAKING_CHANGE\|DEPENDENCY_MODES\|deps.local.json\|foreignOwner\|governs(' server fixtures docs/features agent-context --include=*.ts --include=*.py --include=*.json --include=*.md +``` +Expected: no output. + +Then the positive check — the ruling's new refusal must be registered in all three registries: +```bash +grep -rn 'ERR_DEPENDENCY_PACKAGE_NOT_OWNED' fixtures/conformance/ERROR-CODES.json server/typescript/packages/metadata/src/errors.ts server/python/src/metaobjects/errors.py +``` +Expected: one hit in each of the three files. -- [ ] **Step 4: Record the result** in the final message to the coordinator: the three lane exit codes, the corpus counts (TS: 49/49; Python: 49/49), and any deviation from this plan with the task number it affected. +- [ ] **Step 3:** Run review + simplify per unit (`superpowers:requesting-code-review`, then `code-simplifier`) before merge; then hand off per `superpowers:finishing-a-development-branch`. No commit for this task. diff --git a/docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md b/docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md index d578c212c..7bb0ef7b1 100644 --- a/docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md +++ b/docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md @@ -302,8 +302,13 @@ immaterial). Then, exactly as within one project: — the local, unshared additions the maintainer describes; in `own` mode, anything at all, including persistence (`source.rdb`, `@column`, new fields) — the legacy `database-overlay.json`. -- **Declare new nodes in the dependency's package.** Allowed; the legacy consumers did. - Ownership is per node, not per package (D7). +- **Declare new nodes in the dependency's package.** **REFUSED** — + `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` (§11.1 item 2, ruled 2026-09-11). This bullet + previously read *"Allowed; the legacy consumers did."* That claim was checked against the + legacy estate and is **false**: the one legacy consumer that declares into a shared + package (declared across both a framework module and the app consuming it) + adds **no** class the framework does not already declare — it amends, via the + data/ui/baseui overlay split, and an amendment is an overlay, not a new node. See §11.5. - **Reference** foreign nodes by FQN from any ref-bearing attribute. Bare references stay package-local (ADR-0042), so a consumer node *in the dependency's package* may use a bare name for a foreign sibling — the same rule as today, no exception. @@ -444,12 +449,19 @@ model — you generate its code and own its tables* (a metadata-only library). F pair it is wrong: the library owns its tables. A per-node override (`own: ["…"]`) is deferred (§9). -**Why the lock's `nodes` and not provenance or packages.** A consumer overlay merges into the -foreign node; the merged envelope's `contributors` are sorted alphabetically, not by -declaration, so provenance cannot say which file was the base. And consumers may declare -into the dependency's package, so package ownership does not hold. The generated manifest -already knows exactly which top-level nodes it exported; the lock carries that list; the -predicate is a set lookup, cross-port trivial, and the corpus can assert it. +**Why not provenance.** A consumer overlay merges into the foreign node; the merged +envelope's `contributors` are sorted alphabetically, not by declaration, so provenance +cannot say which file was the base. That rules provenance out as the ownership test and is +unaffected by the 2026-09-11 ruling. + +**Why `packages` and not `nodes` is the exclusion key (superseded this paragraph).** This +section previously argued for `nodes` on the grounds that "consumers may declare into the +dependency's package, so package ownership does not hold." The estate does not bear that +out (§11.5), and a per-object key costs a second mental model — "ownership is per node, not +per package" — for a case with no instance. The exclusion key is `packages`; `nodes` +remains in the manifest and lock for artifact integrity, collision detection, and the +`ERR_DEPENDENCY_PACKAGE_NOT_OWNED` refusal that makes the package rule fail loudly instead +of silently. ### 2.8 D7(b) — Codegen imports across the boundary (the FR-025 slice, Phase 1b) @@ -1144,3 +1156,184 @@ working tree instead of its committed snapshot. The legacy loop was `mvn install Prior art: Go `replace`, Cargo `[patch]`, npm/pnpm `link`, Maven local-repository snapshots; the stored survey's #301 (override without editing the lock). + +--- + +## 11. Phase 1a scope (ruled 2026-09-11) + +_A scope review after Task 5 asked whether v2 was over-built for the driving case. It was, in +the places this section names. The ruling — "B+", thin dependencies plus default exclusion of +imported metadata — supersedes the decisions marked below. Everything else in §§0–10 stands. +The plan (`docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md`) is re-cut to match from Task 6._ + +### 11.1 What Phase 1a builds + +1. **Thin dependencies.** `dependencies: [{ name, path }]` in `.metaobjects/config.json` + (§3.1 minus `mode`); `meta deps sync` copies the publisher's generated artifact into the + committed snapshot `.metaobjects/deps//` and writes `.metaobjects/deps.lock.json` + (§3.3 minus `mode`: `version`, `metamodelVersion`, `resolvedFrom`, `artifact`, + `integrity`, `packages`, `nodes`). Every port's collection resolver loads the snapshot + files **first**, then the project's own files, and refuses to load when the snapshot does + not match the lock (`ERR_DEPENDENCY_SNAPSHOT_STALE`), when two dependencies export one + node (`ERR_DEPENDENCY_NODE_COLLISION`), or when a lock's `metamodelVersion` major differs + from the toolchain's (`ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`). `meta deps check` and + `meta verify --deps` compare the installed artifact's hash with the lock's and fail with + `ERR_DEPENDENCY_UPSTREAM_DRIFT`. The publisher's artifact is generated by + `sharedModelFile()` (§2.2, §4.3, TypeScript) — closure-checked, re-loaded with core + providers, drift-gated by `verify --codegen`. + +2. **Imported metadata is load-only by default.** A dependency's nodes are loaded so the + consumer's model can *resolve* against them — reference by FQN, `extends`, `overlay: true` + — and are **excluded from every action surface** unless the consumer's own scope names + their package explicitly. In the maintainer's words: *no metadata is imported in runtime, + code-generated from, or used on the `meta` CLI unless explicitly imported for that + purpose.* + + The mechanism composes the two scope declarations that already exist with one rule + built from the lock: + + - `imported(fqn)` — `packageOf(fqn)` is in the union of every lock entry's `packages`. + A package the consumer's dependencies own; not a per-object list. (Ruled 2026-09-11 — + see §11.5. The lock's `nodes` stays, for integrity, collision detection and the + ownership refusal below, but it is NOT the exclusion key.) + - **A consumer may not declare a new top-level node into a dependency's package.** Post + load, a top-level object whose `packageOf(fqn)` is a dependency package and whose FQN + is NOT in that dependency's `nodes` was declared locally into a package the consumer + does not own: `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, naming the object, the package and + the dependency, with the fix ("declare it in your own package and `extends` the + dependency's node, or overlay the dependency's node with `overlay: true`"). An + OVERLAY merges into the existing node, whose FQN *is* in `nodes`, so it passes + untouched — only a genuinely new object is refused. This is what keeps a + package-keyed rule from failing silently: without it, such an object would simply + never generate, with no error and no output. + - `explicitlyIncluded(pkg, patterns)` — some pattern in the list names `pkg` literally: + split the pattern on `::`, drop the final (name) segment; the remaining segments contain + no `*` and join to exactly `pkg`. `acme::common::**` and `acme::common::Address` name + `acme::common`; `acme::**`, `**` and an absent list do not. + - **Codegen and CLI** (`gen`, `verify --codegen`, the shared-enums artifact, the + requirements ledger's denominator, and — through the option in §11.2 — the runtime): + `inScope(fqn) = matchesScope(fqn, scope) && (!imported(fqn) || + explicitlyIncluded(packageOf(fqn), scope.include))`. + - **Schema** (`migrate`, `verify --db`, offline generate, replay): `inMigrateScope(fqn) = + (declaredMigrateScope?.(fqn) ?? true) && (!imported(fqn) || + explicitlyIncluded(packageOf(fqn), migrate.scope))`. The predicate is `undefined` — the + byte-identical, untouched path — only when the project declares no `migrate.scope` **and** + no dependencies. The suppression is **two-sided** through the seam `migrate.scope` already + uses (`scopeExpectedSchema` → `outOfScope` → `unmanagedNames`), with one correction: + an imported object the scope excludes is removed **before** `declaredSchemas` is + computed. `scopeExpectedSchema` pins `diff`'s schema scope to the unscoped model on + purpose ("a scope narrows objects, never schemas"), which is right for `migrate.scope` — + the same model declared into that schema — and wrong for an import: the consumer never + declared into the publisher's schema, so leaving it in scope would turn every table the + publisher did *not* export into a `DROP` candidate. The refusal for a `migrate.scope` + that matches nothing keeps reading the *declared* scope alone. + - A shared enum is emitted for the entities that are selected: an abstract enum reaches + `enums.ts` iff a selected entity's field resolves to it, decided by the using entity and + never by the enum's own package (bindings are deferred, and a selected entity's generated + file must compile). + - `meta gen ` / a Python target's `entities:` naming an excluded import is refused by + name (exit 2) with the fix stated (`scope.include`), rather than silently generating + nothing. + + **Explicit include replaces `mode`.** The legacy "consumer instantiates the shared model" + case — the library's `own` mode — is: name the package in `scope.include` (generate it) + and in `migrate.scope` (own its tables). Both are per-project declarations that already + exist; nothing new is registered. + +3. **Runtime scoping — DEFERRED (ruled 2026-09-11).** The maintainer's "runtime" meant + *what metadata gets loaded*, which is item 2's collection resolver, not an + `ObjectManager` predicate. Nobody asked for the `ObjectManager` option, so it is not + built; the trigger to revisit is in §11.4. The original reasoning is kept below for + whoever picks it up. + + ~~The runtime does not read `.metaobjects/config.json`, so it cannot + know what is imported; the collection does. Each port's `ObjectManager` gains an optional + predicate over the entity's fully-qualified name; an entity the predicate rejects is + refused by name with a message that says it is loaded for resolution only. The documented + idiom passes the collection's composed predicate: + + ```ts + new ObjectManager({ metadata: root, driver, scope: collection.inScope }) + ``` + ```python + ObjectManager(root, driver, column_naming="snake_case", scope=collection.in_scope) + ``` + Absent, the runtime behaves exactly as today (every loaded entity is addressable) — a + default-open choice, because a runtime that silently hid entities from an app that never + opted in would be a behaviour change for every existing consumer.~~ + +4. **Overlays.** No new loader rule. A consumer's amendment of a node it does not own must + say `overlay: true` — that is what makes an upstream removal `ERR_OVERLAY_NO_TARGET` + (raised by all four ports today) instead of a silent new object; the docs and the skills + teach it. `meta verify` gains a TypeScript **authoring finding** (advisory, never fails a + build, own cap, `--no-overlay-lint` / `META_NO_OVERLAY_LINT=1`): a top-level node declared + in two or more files where more than one declaration lacks `overlay: true`. It fires while + the base still exists, which is the only moment the missing flag can be caught. Verified: + without the flag the TypeScript parser "silently reuse[s] existing or create[s] new" and the + Python merge appends on a miss; with it, both fail loudly. + +### 11.2 The surfaces, in one table + +| Surface | Change | +|---|---| +| `.metaobjects/config.json` | `dependencies: [{ name, path }]`; `npm` / `python` arms stay in the schema and are refused by `deps sync` (`ERR_DEPENDENCY_UNRESOLVED`, "not supported by this toolchain yet") — the `sources` `resource`/`package` precedent | +| `sdk` `Collection` | `dependencies`, `ownFiles`, `fileIds`, `importedPackages`, `importedNodes`, `imported(fqn)` (package-keyed); `inScope` and `inMigrateScope` become the composed predicates; `declaredMigrateScope` is the user's alone | +| `metadata` | `LockSchema`/`ManifestSchema` live in sdk; `declaredTopLevelKeys(source)` exported from the loader for the overlay lint | +| `codegen-ts` | `sharedModelFile()`; `GenContext.registry` / `sourceFiles`; `renderSharedEnumsFile(root, { select })` | +| `migrate-ts` | `scopeExpectedSchema(built, inScope, { imported })` — the `declaredSchemas` correction | +| `cli` | `meta deps sync \| check \| list`; `verify --deps`; the overlay lint; `gen` refusal by name; `init` no longer scaffolds `package.meta.json` | +| `runtime-ts` | *(none — the runtime option is deferred; see §11.4)* | +| Python | `config/dependencies.py` (lock, manifest, integrity, `Collection` with `in_scope`), `resolve_collection_full`, `run_gen(select=)`, `entities:` refusal, `ObjectManager(scope=)` | +| Corpus | `fixtures/dependency-conformance/` arms: resolution, `expectImported` / `expectSelected` / `expectMigrateGoverned` (all **exhaustive** over the loaded top-level objects), load-time failure, lock/snapshot integrity. TS + Python | + +### 11.3 Superseded by this ruling + +| v2 decision | Status | +|---|---| +| D4 `mode: reference \| own`; §2.7's per-mode table; ruling 3 | **Superseded.** Explicit include replaces it. `DEPENDENCY_MODES` / `DEFAULT_DEPENDENCY_MODE` and the `mode` key are removed from the config schema and the lock. | +| D7 `Collection.foreignOwner` / `governs`; source-id provenance as a load-bearing input; the boundary validator (§4.4) with `ERR_DEPENDENCY_OVERLAY_IMPLICIT` and `ERR_DEPENDENCY_SCHEMA_NOT_OWNED` | **Superseded.** `imported(fqn)` is a set lookup on the lock's `packages` (§11.5); nothing tracks which file contributed what. The two codes are removed from `ERROR-CODES.json`, `errors.ts` and `errors.py`. `dep:` source ids stay — diagnostic only. | +| D6(b) the usage-aware classifier, `--accept-breaking`, `ERR_DEPENDENCY_BREAKING_CHANGE`, corpus cases 29–46 | **Superseded.** `deps check` / `verify --deps` compare hashes; the committed artifact's diff in the consumer's history is the review. The code is removed from the three registries. | +| §2.1 transports `npm` / `python`; D10 local override (§10); `deps.local.json` | **Deferred** (§9). `path` reaches a sibling checkout and an installed location alike; an override file only exists because the other transports point at installed packages. | +| §5.2 Python `shared-model` generator and `targets..options`; §2.7 docs badge (1b) | **Deferred** (§9). | +| D7 the lock's `nodes` as the EXCLUSION key; §2.4's "declare new nodes in the dependency's package — allowed, the legacy consumers did"; §2.7's per-node ownership argument | **Superseded** (2026-09-11, §11.5). `packages` is the exclusion key; a local top-level node in a dependency package is `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`. `nodes` stays in the manifest/lock for integrity, collision and that refusal. | +| §11.1 item 3's `ObjectManager` scope predicate (`runtime-ts`, Python) | **Deferred** (§11.4). "Runtime" meant metadata loading, which item 2 already does. | +| §2.5–§2.6's claim that a table-backed export needs a guard | **Withdrawn.** With two-sided default exclusion and the `declaredSchemas` correction, exporting a table-backed entity is safe; the Postgres-backed test in the plan is the proof. | + +### 11.4 Still deferred (§9), with the trigger that brings each back + +the `ObjectManager` scope predicate (an app that loads one collection but must serve only +part of it at runtime — distinct from what it loads); +`npm` / `python` / `git` / `maven` / `nuget` transports (a consumer that cannot reach the +publisher by `path`); the local override (with the transports); the usage-aware classifier +(consumers a publisher cannot see); per-node provenance and the boundary validator (a shared +database where a consumer must be *prevented* from amending a publisher's table, rather than +told); the Python publisher generator (a Python-only publisher); bindings (a consumer whose +generated code names an imported type). + +### 11.5 The exclusion key — evidence and ruling (2026-09-11) + +The re-cut left one question open: is imported-ness keyed on the lock's `nodes` (per +object) or its `packages` (per package)? The design argued `nodes`, on one empirical claim +(§2.4): *"Declare new nodes in the dependency's package. Allowed; the legacy consumers +did."* The two keys differ in exactly one case — a consumer declaring a **brand-new** +top-level node into a dependency's package — so that claim was the whole argument. It was +checked. + +| Evidence | Finding | +|---|---| +| The driving consumer | Declares exactly one package (its own). Zero overlays. Nothing in the publisher's package. `nodes` and `packages` are indistinguishable here. | +| The legacy Java estate, a shared framework package | Declared across **two** modules — the framework (publisher) and the app consuming it. So a consumer really does declare *into* a dependency's package. | +| …but which classes? | Publisher: `Base, Group, GroupType, Role, User, UserGroup, UserRole`. Consumer: `Group, GroupType, Role, User, UserGroup, UserRole`. **New in consumer: none.** The consumer amends via the data/ui/baseui overlay split. An amendment is an overlay — its FQN is in `nodes`, so both keys agree. | +| New nodes in a shared package elsewhere | Found — two classes declared in a `*.ui.xml` that its sibling `*.data.xml` does not declare — but **within one module**, the same owner's ui file adding to its own data file. No dependency boundary is crossed. | + +**No instance in the estate distinguishes the two keys.** §2.4's claim is false as written. + +**Ruled:** `packages` is the exclusion key — it is the maintainer's own model ("all metadata +loads into one big tree, then you include and exclude packages"), it is one concept rather +than two, and it needs no "ownership is per node, not per package" caveat. The one failure +mode a package-keyed rule introduces — a consumer node in a dependency package silently +never generating — is closed by making it a loud error +(`ERR_DEPENDENCY_PACKAGE_NOT_OWNED`) rather than a silent exclusion. `nodes` is retained in +the manifest and lock, where it still earns its place: `sync` verifies the artifact contains +exactly the nodes the manifest claims, two dependencies exporting one FQN collide, and the +refusal above needs it to tell an overlay from a new declaration. From 24b3fcb7c33247f72bcd71e6cad1c3b19d23f5ec Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 20:02:51 -0400 Subject: [PATCH 12/62] =?UTF-8?q?refactor(deps):=20retire=20`mode`,=20the?= =?UTF-8?q?=20local=20override=20and=20three=20superseded=20error=20codes?= =?UTF-8?q?=20(FR-023=20=C2=A711)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 6 of FR-023 phase 1a. The B+ re-cut (DESIGN §11) supersedes v2's mode: reference | own and the D7/D6(b) machinery it depended on; this task is the pruning pass, plus one addition, plus two carried minors. No new feature. - `sdk/src/dependencies.ts` + `config/dependencies.py`: remove `DEPENDENCY_MODES`, `DEFAULT_DEPENDENCY_MODE`, `DependencyMode`, `LOCAL_OVERRIDE_FILE`, and the `mode` member of `DependencySpec` / the Mode zod field on every schema arm. `DependencySpec` is now `{ name } & (path | npm+dir? | python+dir?)` with no mode — a `mode` key of ANY value (including "reference", the old default) is now an unrecognized-key schema error, since explicit `scope.include` replaces it (DESIGN §11.1 item 2). - `neutral_config.py` `_validate_dependency_spec`: drops `mode` handling and its allowed-key entry to match. - `fixtures/conformance/ERROR-CODES.json` + `errors.ts` + `errors.py`: remove `ERR_DEPENDENCY_OVERLAY_IMPLICIT`, `ERR_DEPENDENCY_SCHEMA_NOT_OWNED`, `ERR_DEPENDENCY_BREAKING_CHANGE` (superseded — §11.3); add `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` (raised by a later task — the package-keyed exclusion rule of §11.5 needs a loud refusal when a local top-level node lands in a dependency's package); reword `ERR_DEPENDENCY_UNRESOLVED` to drop the retired local-override clause. All three registries move together (`errors.test.ts` / `test_errors.py` assert parity) — verified with both green. - `fixtures/dependency-conformance/`: the corpus's case schema drops `localOverrides` / `expectOverrides` / `expectForeign` / `expectGoverned` / `classify`; adds `expectImported` / `expectSelected` / `expectMigrateGoverned`, each the EXHAUSTIVE set over every loaded top-level object admitted by `collection.imported` / `collection.inScope` / `collection.inMigrateScope` respectively (DESIGN §11.2). The TS runner is rewritten to match; `collection.imported` doesn't exist on `Collection` until a later task, so the corpus's one case now fails on that TypeError — expected, left failing per the plan. - Two carried minors from earlier reviews: Python `matches_scope` now uses `re.fullmatch` (bare `re.match` lets `$` match just before a trailing newline, a genuine JS/Python semantic divergence — a literal, non-wildcarded pattern isolates it, since a wildcard segment's `[^:]*` legitimately consumes a literal newline as content either way); the TS corpus runner writes the lock file under `LOCK_FILE` imported from `dependencies.ts` rather than the inlined `"deps.lock.json"` string. TDD: config.test.ts / test_neutral_config.py / test_scope_conformance.py edited to fail first (mode still parsed; re.match still matched), then made to pass. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- fixtures/conformance/ERROR-CODES.json | 8 +- fixtures/dependency-conformance/README.md | 62 ++++----- fixtures/dependency-conformance/cases.json | 5 +- .../python/src/metaobjects/config/__init__.py | 6 - .../src/metaobjects/config/dependencies.py | 9 -- .../src/metaobjects/config/neutral_config.py | 33 ++--- server/python/src/metaobjects/errors.py | 17 +-- server/python/src/metaobjects/scope.py | 15 +- .../tests/config/test_neutral_config.py | 5 +- .../conformance/test_scope_conformance.py | 16 +++ .../packages/metadata/src/errors.ts | 17 +-- .../packages/sdk/src/dependencies.ts | 30 ++-- server/typescript/packages/sdk/src/index.ts | 5 +- .../packages/sdk/test/config.test.ts | 11 +- .../sdk/test/dependency-conformance.test.ts | 131 ++++++++++-------- 15 files changed, 180 insertions(+), 190 deletions(-) diff --git a/fixtures/conformance/ERROR-CODES.json b/fixtures/conformance/ERROR-CODES.json index 27bc964c4..18b0f4263 100644 --- a/fixtures/conformance/ERROR-CODES.json +++ b/fixtures/conformance/ERROR-CODES.json @@ -89,14 +89,12 @@ "ERR_ENUM_INT_VALUE_MAP_ARRAY": "A field.enum carries @intValueMap together with isArray=true. Int-backing is a persistence-layer codec and no port implements it element-wise over an array column, so the combination would silently persist member SYMBOLS into an integer array. An array-of-enum stays string-backed: drop @intValueMap, or make the field scalar.", "ERR_REQUIREMENT_RETIRED_HAS_IMPLEMENTORS": "A requirement.* with @status: retired declares @implementedBy. Refused rather than exempted (FR-039): a retired capability has no implementation by definition, so forbidding the attribute makes the dangling-reference class unreachable instead of silently tolerated.", "ERR_REQUIREMENT_SUPERSEDED_BY_NOT_RETIRED": "@supersededBy on a requirement whose @status is not `retired`. The attribute names what REPLACED a withdrawn capability; on a live one there is nothing to have replaced it.", - "ERR_DEPENDENCY_UNRESOLVED": "FR-023: a declared dependency's transport or local override could not locate a directory holding metaobjects.pkg.json, or an override names an undeclared dependency.", + "ERR_DEPENDENCY_UNRESOLVED": "FR-023: a declared dependency's transport could not locate a directory holding metaobjects.pkg.json.", "ERR_DEPENDENCY_MANIFEST_INVALID": "FR-023: a dependency's metaobjects.pkg.json fails its schema, names a different dependency, points at a missing or hash-mismatched artifact, or its artifact does not load standalone / does not declare exactly the listed packages and nodes.", "ERR_DEPENDENCY_SNAPSHOT_STALE": "FR-023: the committed snapshot does not match .metaobjects/deps.lock.json (lock missing, entry missing or extra, artifact missing, or hash mismatch) — run `meta deps sync`.", "ERR_DEPENDENCY_NODE_COLLISION": "FR-023: two dependencies export the same fully-qualified node.", - "ERR_DEPENDENCY_OVERLAY_IMPLICIT": "FR-023: a local top-level node redeclares a dependency's node without `overlay: true`.", - "ERR_DEPENDENCY_SCHEMA_NOT_OWNED": "FR-023: a local contribution changes the physical shape of a reference-mode dependency's node (a field/identity/index/relationship/source child, a physical attribute, or a TPH subtype of a foreign base).", + "ERR_DEPENDENCY_PACKAGE_NOT_OWNED": "FR-023: a local top-level node is declared into a package owned by a dependency.", "ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE": "FR-023: a dependency's metamodelVersion major differs from this toolchain's.", - "ERR_DEPENDENCY_UPSTREAM_DRIFT": "FR-023: `meta deps check` / `verify --deps` found the installed dependency differs from the committed snapshot.", - "ERR_DEPENDENCY_BREAKING_CHANGE": "FR-023: `meta deps sync` refused an upstream change classified BREAKING for this consumer's footprint (override with --accept-breaking)." + "ERR_DEPENDENCY_UPSTREAM_DRIFT": "FR-023: `meta deps check` / `verify --deps` found the installed dependency differs from the committed snapshot." } } diff --git a/fixtures/dependency-conformance/README.md b/fixtures/dependency-conformance/README.md index a12a257d6..38f34178b 100644 --- a/fixtures/dependency-conformance/README.md +++ b/fixtures/dependency-conformance/README.md @@ -1,17 +1,18 @@ # dependency-conformance Pins how FR-023 metadata dependencies (declared in `.metaobjects/config.json`, -resolved via a manifest + lock + committed snapshot) resolve, govern codegen/ -migrate, overlay, fail, and classify upstream changes — across every port. -Every port's runner reads THIS file; there is no per-port fixture. +resolved via a manifest + lock + committed snapshot) resolve, are excluded by +default from codegen/migrate selection unless explicitly included, overlay, +and fail — across every port. Every port's runner reads THIS file; there is +no per-port fixture. ## Shape ``` -cases.json # { cases: [ { name, tree, treeFiles?, config, lock?, localOverrides?, - # resolveFrom?, expectFiles?, expectForeign?, expectGoverned?, - # expectOverrides?, expectLoadError?, expectErrorFiles?, - # expectError?, classify? } ] } +cases.json # { cases: [ { name, tree, treeFiles?, config, lock?, + # resolveFrom?, expectFiles?, expectImported?, + # expectSelected?, expectMigrateGoverned?, + # expectLoadError?, expectErrorFiles?, expectError? } ] } README.md artifacts/ # pinned dependency artifacts, referenced by cases via `treeFiles` acme-common-v1.json @@ -28,24 +29,23 @@ artifacts/ # pinned dependency artifacts, referenced by cases via `tre "treeFiles": { "": "artifacts/" }, // OPTIONAL: copied byte-for-byte from the corpus dir "config": { … } | null, // written to /.metaobjects/config.json "lock": { … }, // OPTIONAL: written to /.metaobjects/deps.lock.json - "localOverrides": { … }, // OPTIONAL: written to /.metaobjects/deps.local.json "resolveFrom": ".", // OPTIONAL "expectFiles": ["…"], // unordered set, project-root-relative (resolution arm) - "expectForeign": [""], // OPTIONAL: FQNs with a foreign owner - "expectGoverned": [""], // OPTIONAL: FQNs governs() admits, over every loaded top-level object - "expectOverrides": [""], // OPTIONAL: dependencies read from a local override + "expectImported": [""], // OPTIONAL: the EXHAUSTIVE set, over every loaded top-level + // object, for which collection.imported(fqn) is true + "expectSelected": [""], // OPTIONAL: the EXHAUSTIVE set, over every loaded top-level + // object, for which collection.inScope(fqn) is true + "expectMigrateGoverned": [""], // OPTIONAL: the EXHAUSTIVE set, over every loaded top-level + // object, for which collection.inMigrateScope admits it + // (undefined inMigrateScope admits everything) "expectLoadError": "ERR_*", // OPTIONAL: the collection resolves, then LOADING it fails with this code "expectErrorFiles": ["dep:…"], // OPTIONAL with expectLoadError: source.files[0] of the first error - "expectError": "ERR_*", // resolution itself fails with this code - "classify": { // classifier arm (TS + Python only) - "old": { … }, "new": { … }, // two artifact documents, inline - "footprint": { "": "whole" | "key" | "existence" }, - "expectChanges": [ { "fqn": "…", "path": "…", "kind": "breaking" | "compatible" | "info" } ] } + "expectError": "ERR_*" // resolution itself fails with this code } ] } ``` -Exactly one of `expectFiles`, `expectError`, `classify` is present per case; `expectLoadError` -rides with `expectFiles`. +Exactly one of `expectFiles`, `expectError` is present per case; `expectLoadError` rides +with `expectFiles`. - **`tree`** — a map of project-root-relative path → file content, materialized in a fresh temporary directory (same shape as `source-resolution-conformance`). @@ -57,18 +57,21 @@ rides with `expectFiles`. - **`config`** — written verbatim to `/.metaobjects/config.json`. `null` means no config file is created. - **`lock`** — OPTIONAL, written verbatim to `/.metaobjects/deps.lock.json`. -- **`localOverrides`** — OPTIONAL, written verbatim to `/.metaobjects/deps.local.json` - (D10 — local co-development override). - **`resolveFrom`** — OPTIONAL, project-root-relative directory the resolver is invoked against; default `"."`. - **`expectFiles`** — the resolution arm. Project-root-relative paths, compared as an unordered set (same contract as `source-resolution-conformance`). -- **`expectForeign`** — OPTIONAL, alongside `expectFiles`: FQNs for which - `collection.foreignOwner(fqn) !== undefined`. -- **`expectGoverned`** — OPTIONAL, alongside `expectFiles`: the FQN set, over every loaded - top-level object, for which `collection.governs(fqn)` is true. -- **`expectOverrides`** — OPTIONAL, alongside `expectFiles`: the dependency names - `collection.overrides` reads from an active `deps.local.json`. +- **`expectImported`** — OPTIONAL, alongside `expectFiles`: the EXHAUSTIVE set, over every + loaded top-level object, for which `collection.imported(fqn)` is true — a package a + dependency owns (DESIGN §11.5: the exclusion key is package-keyed). +- **`expectSelected`** — OPTIONAL, alongside `expectFiles`: the EXHAUSTIVE set, over every + loaded top-level object, for which `collection.inScope(fqn)` is true — the codegen/CLI + selection predicate, composed from the declared scope and the default exclusion of + imported metadata (DESIGN §11.1 item 2). +- **`expectMigrateGoverned`** — OPTIONAL, alongside `expectFiles`: the EXHAUSTIVE set, over + every loaded top-level object, for which `collection.inMigrateScope` admits it — an + `undefined` predicate (no `migrate.scope` declared and no dependencies) admits every + object. - **`expectLoadError`** — OPTIONAL, alongside `expectFiles`: the collection resolves cleanly, but LOADING it (parsing the resolved files into a metadata tree) fails with this code. @@ -77,8 +80,6 @@ rides with `expectFiles`. (`dep:/`) rather than a local one. - **`expectError`** — the resolution-failure arm: `resolveCollection` itself must reject with this exact code. -- **`classify`** — the classifier arm (TS + Python only; see below). Two inline artifact - documents (`old`/`new`), a footprint map, and the expected change list. ## Pinned artifacts @@ -108,9 +109,8 @@ asserts against. ## Which arms each port runs -- **TypeScript** — all arms (resolution/foreignness, overlays, load-time failure, lock/ - snapshot integrity, classification). -- **Python** — all arms. +- **TypeScript** — resolution, load-time failure, lock/snapshot integrity. +- **Python** — resolution, load-time failure, lock/snapshot integrity. - **Java / C# / Kotlin** — Phase 2 (out of scope for this plan; these ports do not read `dependencies` yet). diff --git a/fixtures/dependency-conformance/cases.json b/fixtures/dependency-conformance/cases.json index 097513bf2..80ad41a20 100644 --- a/fixtures/dependency-conformance/cases.json +++ b/fixtures/dependency-conformance/cases.json @@ -7,8 +7,9 @@ }, "config": { "schema_version": 1, "sources": [] }, "expectFiles": ["metaobjects/meta.app.json"], - "expectForeign": [], - "expectGoverned": ["app::Order"] + "expectImported": [], + "expectSelected": ["app::Order"], + "expectMigrateGoverned": ["app::Order"] } ] } diff --git a/server/python/src/metaobjects/config/__init__.py b/server/python/src/metaobjects/config/__init__.py index 09760a0a5..aaf4178b3 100644 --- a/server/python/src/metaobjects/config/__init__.py +++ b/server/python/src/metaobjects/config/__init__.py @@ -7,12 +7,9 @@ """ from .dependencies import ( ARTIFACT_SUFFIX, - DEFAULT_DEPENDENCY_MODE, - DEPENDENCY_MODES, DEPENDENCY_SOURCE_ID_PREFIX, DEPS_DIR, INTEGRITY_PREFIX, - LOCAL_OVERRIDE_FILE, LOCK_FILE, MANIFEST_FILE, ) @@ -21,13 +18,10 @@ __all__ = [ "ARTIFACT_SUFFIX", - "DEFAULT_DEPENDENCY_MODE", "DEFAULT_METADATA_DIR", - "DEPENDENCY_MODES", "DEPENDENCY_SOURCE_ID_PREFIX", "DEPS_DIR", "INTEGRITY_PREFIX", - "LOCAL_OVERRIDE_FILE", "LOCK_FILE", "MANIFEST_FILE", "NeutralConfig", diff --git a/server/python/src/metaobjects/config/dependencies.py b/server/python/src/metaobjects/config/dependencies.py index 3da7db1b4..d1f2cbc12 100644 --- a/server/python/src/metaobjects/config/dependencies.py +++ b/server/python/src/metaobjects/config/dependencies.py @@ -13,9 +13,6 @@ #: `meta deps sync`'s output — the only writer (DESIGN §3.3). LOCK_FILE = "deps.lock.json" -#: D10 co-development override — never committed (DESIGN §10 D10). -LOCAL_OVERRIDE_FILE = "deps.local.json" - #: The publisher-generated manifest sitting beside a dependency's artifact #: (DESIGN §3.2). MANIFEST_FILE = "metaobjects.pkg.json" @@ -31,9 +28,3 @@ #: Prefix of the `integrity` field's value: `"sha256-" + lowercase hex sha256 #: of the artifact bytes` (DESIGN §3, "Hash format"). INTEGRITY_PREFIX = "sha256-" - -#: The two modes a declared dependency may run in (DESIGN §2.4, §2.7). -DEPENDENCY_MODES = ("reference", "own") - -#: `mode`'s default when a dependency spec omits it. -DEFAULT_DEPENDENCY_MODE = "reference" diff --git a/server/python/src/metaobjects/config/neutral_config.py b/server/python/src/metaobjects/config/neutral_config.py index 6ce28ce75..73d4deb54 100644 --- a/server/python/src/metaobjects/config/neutral_config.py +++ b/server/python/src/metaobjects/config/neutral_config.py @@ -7,8 +7,6 @@ from metaobjects.errors import ErrorCode, ParseError -from .dependencies import DEFAULT_DEPENDENCY_MODE, DEPENDENCY_MODES - #: The DEFAULT value of `sources` when the key is absent or empty — never a #: requirement, and never assumed to exist by any other code path. DEFAULT_METADATA_DIR = "metaobjects" @@ -32,8 +30,8 @@ class NeutralConfig: sources: list[dict[str, str]] #: Declared metadata dependencies (FR-023) — each dict already validated: - #: `name`, exactly one transport key (`path` / `npm` / `python`), `mode` - #: defaulted to `"reference"`, optional `dir` only beside `npm`/`python`. + #: `name`, exactly one transport key (`path` / `npm` / `python`), optional + #: `dir` only beside `npm`/`python`. dependencies: list[dict[str, str]] @@ -117,12 +115,13 @@ def read_neutral_config(config_dir: Path) -> NeutralConfig | None: def _validate_dependency_spec(dep: object, path: Path) -> dict[str, str]: - """Validate one entry of `dependencies` (DESIGN §3.1) and return it in - normalized form (`mode` always present). Mirrors the TS - `DependencySpecSchema` union in `sdk/src/dependencies.ts` exactly: a - `name`, exactly one transport key (`path` | `npm` | `python`), an - optional `dir` that is legal only beside `npm`/`python`, an optional - `mode` defaulting to `"reference"`, and no other keys. + """Validate one entry of `dependencies` (DESIGN §3.1) and return it + normalized. Mirrors the TS `DependencySpecSchema` union in + `sdk/src/dependencies.ts` exactly: a `name`, exactly one transport key + (`path` | `npm` | `python`), an optional `dir` that is legal only beside + `npm`/`python`, and no other keys. `mode` is REMOVED (FR-023 §11.3 — + superseded by explicit `scope.include`); any `mode` key is now an + unknown-key error like any other unrecognized key. """ def fail(reason: str) -> ParseError: @@ -147,7 +146,7 @@ def fail(reason: str) -> ParseError: if not isinstance(transport_value, str) or not transport_value.strip(): raise fail(f"'{transport}' must be a non-empty string") - allowed_keys = {"name", transport, "mode"} + allowed_keys = {"name", transport} if transport in ("npm", "python"): allowed_keys.add("dir") extra_keys = set(dep.keys()) - allowed_keys @@ -162,16 +161,4 @@ def fail(reason: str) -> ParseError: raise fail("'dir' must be a non-empty string") result["dir"] = dir_value - mode = dep.get("mode", DEFAULT_DEPENDENCY_MODE) - # `isinstance(mode, bool)` must be checked separately (same reason as - # `schema_version` above): `True`/`False` are `int` subclasses in - # Python, and `mode not in DEPENDENCY_MODES` alone would let a boolean - # through if either mode string ever coincided with `1`/`0` — it never - # does today, but the explicit check keeps this in lockstep with the - # `schema_version` guard's reasoning rather than relying on that - # coincidence. - if isinstance(mode, bool) or mode not in DEPENDENCY_MODES: - raise fail(f"'mode' must be one of {DEPENDENCY_MODES}") - result["mode"] = mode - return result diff --git a/server/python/src/metaobjects/errors.py b/server/python/src/metaobjects/errors.py index 4df536735..6c64b4952 100644 --- a/server/python/src/metaobjects/errors.py +++ b/server/python/src/metaobjects/errors.py @@ -125,8 +125,8 @@ class ErrorCode(str, Enum): ERR_SCOPE_PATTERN_INVALID = "ERR_SCOPE_PATTERN_INVALID" # Phase-1 metadata-source-resolution — no metadata collection was discovered: no config declaring sources, and no default metaobjects/ directory. ERR_COLLECTION_NOT_FOUND = "ERR_COLLECTION_NOT_FOUND" - # FR-023 — a declared dependency's transport or local override could not locate a - # directory holding metaobjects.pkg.json, or an override names an undeclared dependency. + # FR-023 — a declared dependency's transport could not locate a directory holding + # metaobjects.pkg.json. ERR_DEPENDENCY_UNRESOLVED = "ERR_DEPENDENCY_UNRESOLVED" # FR-023 — a dependency's metaobjects.pkg.json fails its schema, names a different # dependency, points at a missing/hash-mismatched artifact, or the artifact does not @@ -137,20 +137,15 @@ class ErrorCode(str, Enum): ERR_DEPENDENCY_SNAPSHOT_STALE = "ERR_DEPENDENCY_SNAPSHOT_STALE" # FR-023 — two dependencies export the same fully-qualified node. ERR_DEPENDENCY_NODE_COLLISION = "ERR_DEPENDENCY_NODE_COLLISION" - # FR-023 — a local top-level node redeclares a dependency's node without `overlay: true`. - ERR_DEPENDENCY_OVERLAY_IMPLICIT = "ERR_DEPENDENCY_OVERLAY_IMPLICIT" - # FR-023 — a local contribution changes the physical shape of a reference-mode - # dependency's node (a field/identity/index/relationship/source child, a physical - # attribute, or a TPH subtype of a foreign base). - ERR_DEPENDENCY_SCHEMA_NOT_OWNED = "ERR_DEPENDENCY_SCHEMA_NOT_OWNED" + # FR-023 — a local top-level node is declared into a package owned by a dependency + # (§11.5 — imported-ness is package-keyed; a genuinely new object in a dependency's + # package would otherwise silently never generate). + ERR_DEPENDENCY_PACKAGE_NOT_OWNED = "ERR_DEPENDENCY_PACKAGE_NOT_OWNED" # FR-023 — a dependency's metamodelVersion major differs from this toolchain's. ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE = "ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE" # FR-023 — `meta deps check` / `verify --deps` found the installed dependency differs # from the committed snapshot. ERR_DEPENDENCY_UPSTREAM_DRIFT = "ERR_DEPENDENCY_UPSTREAM_DRIFT" - # FR-023 — `meta deps sync` refused an upstream change classified BREAKING for this - # consumer's footprint (override with --accept-breaking). - ERR_DEPENDENCY_BREAKING_CHANGE = "ERR_DEPENDENCY_BREAKING_CHANGE" # FR-016 / ADR-0018 — per-kind physical-name aliases on source.rdb. ERR_PHYSICAL_NAME_KIND_MISMATCH = "ERR_PHYSICAL_NAME_KIND_MISMATCH" ERR_PHYSICAL_NAME_MULTIPLE = "ERR_PHYSICAL_NAME_MULTIPLE" diff --git a/server/python/src/metaobjects/scope.py b/server/python/src/metaobjects/scope.py index 5eb0bbec9..e31352554 100644 --- a/server/python/src/metaobjects/scope.py +++ b/server/python/src/metaobjects/scope.py @@ -88,8 +88,17 @@ def compile_scope( def matches_scope(fqn: str, compiled: CompiledScope) -> bool: - """True when ``fqn`` is inside the scope. An empty ``include`` means everything.""" - included = len(compiled.include) == 0 or any(p.match(fqn) for p in compiled.include) + """True when ``fqn`` is inside the scope. An empty ``include`` means everything. + + Uses ``fullmatch``, not ``match``: every compiled pattern already anchors + with ``^...$`` (`compile_pattern`), but Python's bare ``$`` matches just + before a trailing newline even so — so ``re.match`` would let + ``"acme::Order\\n"`` match a pattern meant only for ``"acme::Order"``. JS + `$` never does that, so `match` here would be a genuine cross-port + semantic divergence; `fullmatch` requires the newline to be consumed too, + closing it. + """ + included = len(compiled.include) == 0 or any(p.fullmatch(fqn) for p in compiled.include) if not included: return False - return not any(p.match(fqn) for p in compiled.exclude) + return not any(p.fullmatch(fqn) for p in compiled.exclude) diff --git a/server/python/tests/config/test_neutral_config.py b/server/python/tests/config/test_neutral_config.py index 349a1b073..7d744515b 100644 --- a/server/python/tests/config/test_neutral_config.py +++ b/server/python/tests/config/test_neutral_config.py @@ -107,11 +107,11 @@ def test_non_string_source_value_raises(tmp_path: Path) -> None: # same as `sources`: every port reads it at every rung of the source ladder. -def test_dependencies_parse_with_mode_default(tmp_path: Path) -> None: +def test_dependencies_parse(tmp_path: Path) -> None: _write_config(tmp_path, {"schema_version": 1, "sources": [], "dependencies": [{"name": "acme-common", "path": "../lib"}]}) cfg = read_neutral_config(tmp_path) assert cfg is not None - assert cfg.dependencies == [{"name": "acme-common", "path": "../lib", "mode": "reference"}] + assert cfg.dependencies == [{"name": "acme-common", "path": "../lib"}] def test_dependencies_absent_is_empty(tmp_path: Path) -> None: @@ -128,6 +128,7 @@ def test_dependencies_absent_is_empty(tmp_path: Path) -> None: [{"name": "a"}], [{"name": "Bad Name", "path": "x"}], [{"name": "a", "path": "x", "mode": "shared"}], + [{"name": "a", "path": "x", "mode": "reference"}], [{"name": "a", "path": "x"}, {"name": "a", "npm": "y"}], [{"name": "a", "path": "x", "dir": "y"}], [{"name": "a", "path": "x", "pathh": "t"}], diff --git a/server/python/tests/conformance/test_scope_conformance.py b/server/python/tests/conformance/test_scope_conformance.py index 0ee468791..477d1a4e2 100644 --- a/server/python/tests/conformance/test_scope_conformance.py +++ b/server/python/tests/conformance/test_scope_conformance.py @@ -57,3 +57,19 @@ def test_scope_conformance_case(case: dict) -> None: f"case {case['name']!r}: matches_scope({fqn!r}, ...) expected " f"{entry['matches']!r}" ) + + +def test_scope_pattern_does_not_match_a_trailing_newline() -> None: + """Carried minor (Task 4 review): Python's `re.match` lets a compiled + `^...$` pattern match a string with a trailing newline, because bare `$` + matches just before it — a divergence from JS, where `$` never does that. + `matches_scope` must use `re.fullmatch` so the two ports agree. + + A wildcarded pattern (`acme::*`) does not exercise this: its trailing + `[^:]*` legitimately consumes a literal `\\n` as content, so `match` and + `fullmatch` agree. Only a literal, non-wildcarded pattern isolates the + `$`-before-trailing-newline behavior this fixes. + """ + compiled = compile_scope(include=["acme::Order"]) + assert matches_scope("acme::Order", compiled) + assert not matches_scope("acme::Order\n", compiled) diff --git a/server/typescript/packages/metadata/src/errors.ts b/server/typescript/packages/metadata/src/errors.ts index 91e967045..913b010d4 100644 --- a/server/typescript/packages/metadata/src/errors.ts +++ b/server/typescript/packages/metadata/src/errors.ts @@ -260,8 +260,8 @@ export const ERROR_CODES = [ // Phase-1 metadata-source-resolution — no metadata collection was discovered: // no config declaring sources, and no default metaobjects/ directory. "ERR_COLLECTION_NOT_FOUND", - // FR-023 — a declared dependency's transport or local override could not locate a - // directory holding metaobjects.pkg.json, or an override names an undeclared dependency. + // FR-023 — a declared dependency's transport could not locate a directory holding + // metaobjects.pkg.json. "ERR_DEPENDENCY_UNRESOLVED", // FR-023 — a dependency's metaobjects.pkg.json fails its schema, names a different // dependency, points at a missing/hash-mismatched artifact, or the artifact does not @@ -272,20 +272,15 @@ export const ERROR_CODES = [ "ERR_DEPENDENCY_SNAPSHOT_STALE", // FR-023 — two dependencies export the same fully-qualified node. "ERR_DEPENDENCY_NODE_COLLISION", - // FR-023 — a local top-level node redeclares a dependency's node without `overlay: true`. - "ERR_DEPENDENCY_OVERLAY_IMPLICIT", - // FR-023 — a local contribution changes the physical shape of a reference-mode - // dependency's node (a field/identity/index/relationship/source child, a physical - // attribute, or a TPH subtype of a foreign base). - "ERR_DEPENDENCY_SCHEMA_NOT_OWNED", + // FR-023 — a local top-level node is declared into a package owned by a dependency + // (§11.5 — imported-ness is package-keyed; a genuinely new object in a dependency's + // package would otherwise silently never generate). + "ERR_DEPENDENCY_PACKAGE_NOT_OWNED", // FR-023 — a dependency's metamodelVersion major differs from this toolchain's. "ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE", // FR-023 — `meta deps check` / `verify --deps` found the installed dependency differs // from the committed snapshot. "ERR_DEPENDENCY_UPSTREAM_DRIFT", - // FR-023 — `meta deps sync` refused an upstream change classified BREAKING for this - // consumer's footprint (override with --accept-breaking). - "ERR_DEPENDENCY_BREAKING_CHANGE", "ERR_UNKNOWN", ] as const; diff --git a/server/typescript/packages/sdk/src/dependencies.ts b/server/typescript/packages/sdk/src/dependencies.ts index 856a52c1a..a1beeebf8 100644 --- a/server/typescript/packages/sdk/src/dependencies.ts +++ b/server/typescript/packages/sdk/src/dependencies.ts @@ -14,9 +14,6 @@ export const DEPS_DIR = "deps"; /** `meta deps sync`'s output — the only writer (DESIGN §3.3). */ export const LOCK_FILE = "deps.lock.json"; -/** D10 co-development override — never committed (DESIGN §10 D10). */ -export const LOCAL_OVERRIDE_FILE = "deps.local.json"; - /** The publisher-generated manifest sitting beside a dependency's artifact * (DESIGN §3.2). */ export const MANIFEST_FILE = "metaobjects.pkg.json"; @@ -33,25 +30,17 @@ export const DEPENDENCY_SOURCE_ID_PREFIX = "dep:"; * of the artifact bytes` (DESIGN §3, "Hash format"). */ export const INTEGRITY_PREFIX = "sha256-"; -/** The two modes a declared dependency may run in (DESIGN §2.4, §2.7). */ -export const DEPENDENCY_MODES = ["reference", "own"] as const; - -/** `mode`'s default when a dependency spec omits it. */ -export const DEFAULT_DEPENDENCY_MODE: (typeof DEPENDENCY_MODES)[number] = "reference"; - -export type DependencyMode = (typeof DEPENDENCY_MODES)[number]; - /** - * A declared dependency: a `name`, exactly one transport (`path` | `npm` | - * `python`), and a `mode`. `npm`/`python` optionally carry `dir` — the - * subdirectory under the resolved package holding `metaobjects.pkg.json`; - * `path` never does, since the path itself already names that directory. + * A declared dependency: a `name` and exactly one transport (`path` | `npm` | + * `python`). `npm`/`python` optionally carry `dir` — the subdirectory under + * the resolved package holding `metaobjects.pkg.json`; `path` never does, + * since the path itself already names that directory. * * Mirrors the hand-written union below in `DependencySpecSchema` — the same * two-direction parity guard `SourceSpec`/`SourceSpecSchema` carries (see * `config.ts`), so the schema and this type cannot silently drift. */ -export type DependencySpec = { readonly name: string; readonly mode: DependencyMode } & ( +export type DependencySpec = { readonly name: string } & ( | { readonly path: string } | { readonly npm: string; readonly dir?: string | undefined } | { readonly python: string; readonly dir?: string | undefined } @@ -64,8 +53,6 @@ export type DependencySpec = { readonly name: string; readonly mode: DependencyM * transport's own package name. */ const DependencyName = z.string().regex(/^[a-z0-9][a-z0-9._-]*$/); -const Mode = z.enum(DEPENDENCY_MODES).default(DEFAULT_DEPENDENCY_MODE); - /** * `.strict()` on every arm, same rationale as `SourceSpecSchema` in * `config.ts`: a config schema that silently strips an unknown key would let @@ -75,11 +62,12 @@ const Mode = z.enum(DEPENDENCY_MODES).default(DEFAULT_DEPENDENCY_MODE); * itself (`path`/`npm`/`python`) is what distinguishes them, and that is * exactly what the "two transports in one spec" / "no transport" refusals * below exercise: neither shape matches any arm, so the union fails closed. + * `mode` is REMOVED (FR-023 §11.3 — superseded by explicit `scope.include`). */ export const DependencySpecSchema = z.union([ - z.object({ name: DependencyName, path: z.string().min(1), mode: Mode }).strict(), - z.object({ name: DependencyName, npm: z.string().min(1), dir: z.string().min(1).optional(), mode: Mode }).strict(), - z.object({ name: DependencyName, python: z.string().min(1), dir: z.string().min(1).optional(), mode: Mode }).strict(), + z.object({ name: DependencyName, path: z.string().min(1) }).strict(), + z.object({ name: DependencyName, npm: z.string().min(1), dir: z.string().min(1).optional() }).strict(), + z.object({ name: DependencyName, python: z.string().min(1), dir: z.string().min(1).optional() }).strict(), ]); /** The declared name of a dependency spec — `name` is common to all three diff --git a/server/typescript/packages/sdk/src/index.ts b/server/typescript/packages/sdk/src/index.ts index 10cc892da..cf5bff1f7 100644 --- a/server/typescript/packages/sdk/src/index.ts +++ b/server/typescript/packages/sdk/src/index.ts @@ -25,17 +25,14 @@ export type { Config } from "./config.js"; export { DEPS_DIR, LOCK_FILE, - LOCAL_OVERRIDE_FILE, MANIFEST_FILE, ARTIFACT_SUFFIX, DEPENDENCY_SOURCE_ID_PREFIX, INTEGRITY_PREFIX, - DEPENDENCY_MODES, - DEFAULT_DEPENDENCY_MODE, DependencySpecSchema, dependencyName, } from "./dependencies.js"; -export type { DependencyMode, DependencySpec } from "./dependencies.js"; +export type { DependencySpec } from "./dependencies.js"; // Meta Forge metadata types + attribute name constants (registered into a // TypeRegistry to let Loader parse decision/principle/etc. children + the diff --git a/server/typescript/packages/sdk/test/config.test.ts b/server/typescript/packages/sdk/test/config.test.ts index 72d9d6512..0fea75abc 100644 --- a/server/typescript/packages/sdk/test/config.test.ts +++ b/server/typescript/packages/sdk/test/config.test.ts @@ -186,19 +186,20 @@ describe("ConfigSchema — phase-1 source resolution", () => { }); describe("ConfigSchema — dependencies (FR-023)", () => { - test("dependencies: a valid spec parses with mode defaulted", () => { + test("dependencies: a valid spec parses with no mode", () => { const cfg = ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "acme-common", path: "../lib/metaobjects" }] }); - expect(cfg.dependencies).toEqual([{ name: "acme-common", path: "../lib/metaobjects", mode: "reference" }]); + expect(cfg.dependencies).toEqual([{ name: "acme-common", path: "../lib/metaobjects" }]); }); test("dependencies: npm and python accept dir; path does not", () => { expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "a", npm: "@acme/model", dir: "metaobjects" }] })).not.toThrow(); - expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "a", python: "acme_model", dir: "metaobjects", mode: "own" }] })).not.toThrow(); + expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "a", python: "acme_model", dir: "metaobjects" }] })).not.toThrow(); expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: [{ name: "a", path: "x", dir: "y" }] })).toThrow(); }); - test("dependencies: two transports, no transport, a bad name, a bad mode, a duplicate name, an unknown key are all refused", () => { + test("dependencies: two transports, no transport, a bad name, a mode key, a duplicate name, an unknown key are all refused", () => { for (const bad of [ [{ name: "a", path: "x", npm: "y" }], [{ name: "a" }], [{ name: "Bad Name", path: "x" }], - [{ name: "a", path: "x", mode: "shared" }], [{ name: "a", path: "x" }, { name: "a", npm: "y" }], [{ name: "a", path: "x", pathh: "typo" }], + [{ name: "a", path: "x", mode: "shared" }], [{ name: "a", path: "x", mode: "reference" }], + [{ name: "a", path: "x" }, { name: "a", npm: "y" }], [{ name: "a", path: "x", pathh: "typo" }], ]) expect(() => ConfigSchema.parse({ schema_version: 1, dependencies: bad })).toThrow(); }); test("dependencies: absent means []", () => { expect(ConfigSchema.parse({ schema_version: 1 }).dependencies).toEqual([]); }); diff --git a/server/typescript/packages/sdk/test/dependency-conformance.test.ts b/server/typescript/packages/sdk/test/dependency-conformance.test.ts index 3a02bbb3f..f4e8ce83d 100644 --- a/server/typescript/packages/sdk/test/dependency-conformance.test.ts +++ b/server/typescript/packages/sdk/test/dependency-conformance.test.ts @@ -2,28 +2,45 @@ // implementation. Every port ships an equivalent runner reading this same file // (fixtures/dependency-conformance/, see its README for the case schema). // -// This runner is written AHEAD of the implementation (TDD): the `classify` arm -// is skipped until the classifier lands (Task 14), and the `expectFiles` arm -// already calls `collection.foreignOwner` / `collection.governs` / -// `collection.overrides`, none of which exist on `Collection` yet — those -// calls are expected to fail with a TypeError until later tasks add them. +// This runner is written AHEAD of the implementation (TDD): `collection.imported` +// does not exist on `Collection` yet (a later task adds the exclusion-key +// composition, DESIGN §11.1 item 2), so any case carrying `expectImported` / +// `expectSelected` / `expectMigrateGoverned` is expected to fail until then. import { describe, expect, test } from "bun:test"; import { copyFile, mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { dirname, join, relative, resolve, sep } from "node:path"; import { TYPE_OBJECT } from "@metaobjectsdev/metadata"; -import { resolveCollection } from "../src/collection.js"; -import { loadMemory } from "../src/memory.js"; +import { resolveCollection, type Collection } from "../src/collection.js"; +import { LOCK_FILE } from "../src/dependencies.js"; +import { loadMemory, type LoadMemoryOptions } from "../src/memory.js"; -interface ClassifyCase { - readonly old: unknown; - readonly new: unknown; - readonly footprint: Record; - readonly expectChanges: ReadonlyArray<{ - readonly fqn: string; - readonly path: string; - readonly kind: "breaking" | "compatible" | "info"; - }>; +/** + * `imported` lands on `Collection` in a later task (DESIGN §11.1 item 2's + * exclusion-key composition) — this narrow extension lets the corpus runner + * reference it ahead of the implementation without an `any` escape hatch. + * Until that task, the runtime object has no such member, so the cast below + * compiles clean but the call throws (`collection.imported is not a + * function`) — exactly the failure DESIGN decision #3 (Task 6) expects for + * the one case that exercises it. + */ +interface CollectionWithImported extends Collection { + readonly imported: (fqn: string) => boolean; +} + +/** + * `fileIds` (path -> `FileSource` id, carried on `Collection` and threaded + * through to `loadMemory`) lands ahead of Task 6 in the FR-023 sequence, same + * ahead-of-implementation situation as `imported` above. Optional here + * (unlike `imported`) because this corpus's `expectLoadError` arm only wants + * provenance to flow through WHEN it exists — it does not need the absence to + * throw. + */ +interface CollectionWithFileIds extends Collection { + readonly fileIds?: ReadonlyMap | undefined; +} +interface LoadMemoryOptionsWithFileIds extends LoadMemoryOptions { + readonly fileIds?: ReadonlyMap | undefined; } interface Case { @@ -35,25 +52,28 @@ interface Case { readonly config: unknown | null; /** OPTIONAL: written to `/.metaobjects/deps.lock.json`. */ readonly lock?: unknown; - /** OPTIONAL: written to `/.metaobjects/deps.local.json` (D10). */ - readonly localOverrides?: unknown; readonly resolveFrom?: string; readonly expectFiles?: readonly string[]; - readonly expectForeign?: readonly string[]; - readonly expectGoverned?: readonly string[]; - readonly expectOverrides?: readonly string[]; + /** OPTIONAL, exhaustive over every loaded top-level object: the FQNs for + * which `collection.imported(fqn)` is true. */ + readonly expectImported?: readonly string[]; + /** OPTIONAL, exhaustive over every loaded top-level object: the FQNs for + * which `collection.inScope(fqn)` is true. */ + readonly expectSelected?: readonly string[]; + /** OPTIONAL, exhaustive over every loaded top-level object: the FQNs + * `collection.inMigrateScope` admits (undefined admits everything). */ + readonly expectMigrateGoverned?: readonly string[]; readonly expectLoadError?: string; readonly expectErrorFiles?: readonly string[]; readonly expectError?: string; - readonly classify?: ClassifyCase; } const CORPUS_DIR = resolve(import.meta.dir, "../../../../../fixtures/dependency-conformance"); const CORPUS = join(CORPUS_DIR, "cases.json"); /** Materializes `c.tree` (and `c.treeFiles`, copied byte-for-byte) under a - * fresh temp root, then writes `config` / `lock` / `localOverrides` (each - * when non-null/present) under `/.metaobjects/`. Mirrors + * fresh temp root, then writes `config` / `lock` (each when non-null/present) + * under `/.metaobjects/`. Mirrors * `source-resolution-conformance.test.ts`'s `materialize`, extended for the * dependency-corpus-only fields. */ async function materialize(c: Case): Promise<{ root: string; resolveDir: string }> { @@ -76,11 +96,7 @@ async function materialize(c: Case): Promise<{ root: string; resolveDir: string } if (c.lock !== undefined) { await mkdir(metaobjectsDir, { recursive: true }); - await writeFile(join(metaobjectsDir, "deps.lock.json"), JSON.stringify(c.lock, null, 2)); - } - if (c.localOverrides !== undefined) { - await mkdir(metaobjectsDir, { recursive: true }); - await writeFile(join(metaobjectsDir, "deps.local.json"), JSON.stringify(c.localOverrides, null, 2)); + await writeFile(join(metaobjectsDir, LOCK_FILE), JSON.stringify(c.lock, null, 2)); } return { root, resolveDir }; } @@ -93,14 +109,6 @@ describe("dependency conformance", () => { }); for (const c of cases) { - // The classifier arm has no runner machinery yet (Task 14 fills it in) — - // skipped rather than failed, so the corpus can carry classify cases - // ahead of the classifier existing. - if (c.classify !== undefined) { - test.skip(c.name, () => {}); - continue; - } - test(c.name, async () => { const { root, resolveDir } = await materialize(c); @@ -111,39 +119,48 @@ describe("dependency conformance", () => { return; } - // A case with neither expectFiles, expectError, nor classify is a malformed - // corpus entry, not "expect zero files" — fail loudly rather than silently + // A case with neither expectFiles nor expectError is a malformed corpus + // entry, not "expect zero files" — fail loudly rather than silently // passing it (same discipline as source-resolution-conformance). if (c.expectFiles === undefined) { - throw new Error(`corpus case "${c.name}" has neither expectFiles, expectError, nor classify`); + throw new Error(`corpus case "${c.name}" has neither expectFiles nor expectError`); } const collection = await resolveCollection(resolveDir, { explicitDir: resolveDir }); const got = collection.files.map((f) => relative(root, f).split(sep).join("/")).sort(); expect(got).toEqual([...c.expectFiles].sort()); - if (c.expectForeign !== undefined) { - for (const fqn of c.expectForeign) { - expect(collection.foreignOwner(fqn)).not.toBeUndefined(); - } - } - - if (c.expectGoverned !== undefined) { + if ( + c.expectImported !== undefined || + c.expectSelected !== undefined || + c.expectMigrateGoverned !== undefined + ) { const loaded = await loadMemory(resolveDir, { files: collection.files }); - const governed = loaded - .childrenOfType(TYPE_OBJECT) - .map((n) => n.resolutionKey()) - .filter((fqn) => collection.governs(fqn)) - .sort(); - expect(governed).toEqual([...c.expectGoverned].sort()); - } + const topLevel = loaded.childrenOfType(TYPE_OBJECT).map((n) => n.resolutionKey()); - if (c.expectOverrides !== undefined) { - expect([...collection.overrides].sort()).toEqual([...c.expectOverrides].sort()); + if (c.expectImported !== undefined) { + const withImported = collection as CollectionWithImported; + const imported = topLevel.filter((fqn) => withImported.imported(fqn)).sort(); + expect(imported).toEqual([...c.expectImported].sort()); + } + if (c.expectSelected !== undefined) { + const selected = topLevel.filter((fqn) => collection.inScope(fqn)).sort(); + expect(selected).toEqual([...c.expectSelected].sort()); + } + if (c.expectMigrateGoverned !== undefined) { + const migrateGoverned = topLevel + .filter((fqn) => collection.inMigrateScope?.(fqn) ?? true) + .sort(); + expect(migrateGoverned).toEqual([...c.expectMigrateGoverned].sort()); + } } if (c.expectLoadError !== undefined) { - const attempt = loadMemory(resolveDir, { files: collection.files, fileIds: collection.fileIds }); + const options: LoadMemoryOptionsWithFileIds = { + files: collection.files, + fileIds: (collection as CollectionWithFileIds).fileIds, + }; + const attempt = loadMemory(resolveDir, options); await expect(attempt).rejects.toMatchObject({ code: c.expectLoadError }); if (c.expectErrorFiles !== undefined) { let thrown: unknown; From 4987236e6b160beabe61f25b9f69bca766d57b74 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 20:05:47 -0400 Subject: [PATCH 13/62] =?UTF-8?q?docs(fr-023):=20pre-flight=20rulings=20?= =?UTF-8?q?=E2=80=94=20the=20refusal=20covers=20both=20loadMemory=20arms?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections from the Tasks 6-21 conflict scan, made before Task 6 ran and committed after it so its diff stayed scoped to its own files. - The ownership refusal must fire on BOTH of loadMemory's arms. As first written it covered only the caller-supplied `files` arm, so every embedder calling loadMemory(repoRoot) with no files would have silently lost it. collectionLoadOptions now returns all four fields. - The codegen runner's options type is RunGenOpts, not RunGenOptions, verified against the source. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index ae84463e3..15d993ae0 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -756,7 +756,7 @@ feat(deps): lock and manifest schemas with sha256 integrity (FR-023) - [ ] **Step 3: Failing unit tests.** `sdk/test/dependencies.test.ts`: `explicitlyIncludes(["acme::common::**"], "acme::common")` true; `(["acme::common::Address"], "acme::common")` true; `(["acme::**"], "acme::common")` false; `(["**"], …)` false; `(undefined, …)` false; `([], …)` false; `(["acme::common::sub::**"], "acme::common")` false. `sdk/test/memory.test.ts`: `loadMemory` with `fileIds` yields a node whose `source.files[0]` is the mapped id. `order-independence.test.ts`: permuting `dependencies` in the config leaves `collection.files` and `collection.fileIds` identical. `collection.test.ts`: a project with no dependencies has `inMigrateScope === undefined` when it declares no `migrate.scope` (byte-identical path), and `inScope` identical to today's for three FQNs. Run → FAIL. -- [ ] **Step 4: Implement.** `verifySnapshot` per the table (reads `.metaobjects/deps//`; builds `ResolvedDependency` with `artifactPath` absolute and `sourceId = dependencySourceId(name, artifact)`); `importedPackagesOf(deps)` builds a `Set` of every lock entry's `packages` (the exclusion key); `importedNodesOf(deps)` a `Set` of every `nodes` entry, read ONLY by the ownership refusal. In `resolveCollection`, after `sources`: `const lock = await readLock(configDir); const deps = cfg.dependencies.length === 0 && lock === undefined ? [] : await verifySnapshot(configDir, cfg.dependencies, lock)`; `files = [...deps.map(d => d.artifactPath), ...own]`; `fileIds`; `imported = (fqn) => importedPackages.has(packageOfResolutionKey(fqn))`; the composed `inScope` = `matchesScope(fqn, scope) && (!imported(fqn) || explicitlyIncludes(scopeSpec?.include, packageOfResolutionKey(fqn)))`; `declaredMigrateScope` = today's `inMigrateScope`; `inMigrateScope` = `undefined` when `migrateSpec === undefined && deps.length === 0`, else `(fqn) => (declaredMigrateScope?.(fqn) ?? true) && (!imported(fqn) || explicitlyIncludes(migrateSpec, packageOfResolutionKey(fqn)))`. `loadMemory`: `paths.map((p) => new FileSource(p, fileIds?.has(p) ? { id: fileIds.get(p) } : undefined))`; `LoadMemoryOptions` also takes `importedPackages?` / `importedNodes?`, and AFTER the load `loadMemory` walks the root's top-level objects (ADR-0039 sanctioned own: root-level scan) and throws `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` for the first whose package is in `importedPackages` and whose resolution key is not in `importedNodes` — so every CLI load site swept below gets the refusal for free, and the corpus's `expectLoadError` arm exercises it in both ports. Create `collectionLoadOptions` and sweep every `loadMemory(` in `cli/src` (grep `loadMemory(`). The requirements line in `verify.ts` (`counted over N metadata file(s)`) gains `, M from dependencies` where `M = collection.dependencies.length`. +- [ ] **Step 4: Implement.** `verifySnapshot` per the table (reads `.metaobjects/deps//`; builds `ResolvedDependency` with `artifactPath` absolute and `sourceId = dependencySourceId(name, artifact)`); `importedPackagesOf(deps)` builds a `Set` of every lock entry's `packages` (the exclusion key); `importedNodesOf(deps)` a `Set` of every `nodes` entry, read ONLY by the ownership refusal. In `resolveCollection`, after `sources`: `const lock = await readLock(configDir); const deps = cfg.dependencies.length === 0 && lock === undefined ? [] : await verifySnapshot(configDir, cfg.dependencies, lock)`; `files = [...deps.map(d => d.artifactPath), ...own]`; `fileIds`; `imported = (fqn) => importedPackages.has(packageOfResolutionKey(fqn))`; the composed `inScope` = `matchesScope(fqn, scope) && (!imported(fqn) || explicitlyIncludes(scopeSpec?.include, packageOfResolutionKey(fqn)))`; `declaredMigrateScope` = today's `inMigrateScope`; `inMigrateScope` = `undefined` when `migrateSpec === undefined && deps.length === 0`, else `(fqn) => (declaredMigrateScope?.(fqn) ?? true) && (!imported(fqn) || explicitlyIncludes(migrateSpec, packageOfResolutionKey(fqn)))`. `loadMemory`: `paths.map((p) => new FileSource(p, fileIds?.has(p) ? { id: fileIds.get(p) } : undefined))`; `LoadMemoryOptions` also takes `importedPackages?` / `importedNodes?`, and AFTER the load `loadMemory` walks the root's top-level objects (ADR-0039 sanctioned own: root-level scan) and throws `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` for the first whose package is in `importedPackages` and whose resolution key is not in `importedNodes`. **`loadMemory` has TWO arms and the refusal must fire on BOTH** (pre-flight ruling 2026-09-11, verified at `memory.ts:137-139`): the caller-supplied arm (`options.files` given — the 9 CLI sites, which pass the two sets via `collectionLoadOptions`) AND the self-resolving arm (`options.files` absent, where `loadMemory` calls `resolveCollection(repoRoot)` itself — there it takes `importedPackages` / `importedNodes` from the collection it just resolved, never from the caller). A refusal wired only into the first arm is silently absent for every embedder that calls `loadMemory(repoRoot)` with no `files`. `collectionLoadOptions(collection)` therefore returns `{ files, fileIds, importedPackages, importedNodes }`. Create `collectionLoadOptions` and sweep every `loadMemory(` in `cli/src` (grep `loadMemory(`). The requirements line in `verify.ts` (`counted over N metadata file(s)`) gains `, M from dependencies` where `M = collection.dependencies.length`. - [ ] **Step 5: Run** ```bash @@ -805,7 +805,7 @@ test(deps): the loader's existing errors are the first cross-repo drift gate (FR - Modify: `server/typescript/packages/cli/src/commands/gen.ts` (the pre-check refusing a positional that names an excluded import) - Modify: `server/typescript/packages/codegen-ts/src/templates/enums-file.ts` (`renderSharedEnumsFile(root, opts?: { select?: (fqn: string) => boolean })`) - Modify: `server/typescript/packages/codegen-ts/src/generators/entity-file.ts` (pass `select` from `ctx`) -- Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (`RunGenOptions.scope` is threaded into `GenContext` as `select` for the templates that render whole-model artifacts — read how `entityFile` reaches `ctx.loadedRoot` and add `ctx.select`) +- Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (`RunGenOpts.scope` — the real exported name is `RunGenOpts`, NOT `RunGenOptions`, verified at `runner.ts:45`; pre-flight ruling 2026-09-11 — is threaded into `GenContext` as `select` for the templates that render whole-model artifacts — read how `entityFile` reaches `ctx.loadedRoot` and add `ctx.select`) - Modify: `server/typescript/packages/codegen-ts/src/generator.ts` (`GenContext.select?: (fqn: string) => boolean`) - Test: `cli/test/gen-imported-nodes.test.ts` (new), `codegen-ts/test/shared-enums-imported.test.ts` (new) From 822ed0e9ef0d2625773756fb69a7d194ecb7bc81 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 20:08:08 -0400 Subject: [PATCH 14/62] docs(fr-023): correct a plan test example that could not catch its own bug MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 6's RED-phase example was wrong twice: compile_scope takes positional sequences, not a dict (passing a dict compiles its keys, silently yielding a stub pattern), and the wildcard it used compiles to a character class that admits a newline — so match and fullmatch agreed and the assertion could never have failed against the defect it was written for. The literal pattern is the only form where the two diverge. Measured both ways before correcting. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 15d993ae0..5d187c952 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -635,7 +635,7 @@ feat(config): `dependencies` in .metaobjects/config.json — sdk schema and Pyth - Produces Python: `NeutralConfig.dependencies` entries carry `name` + one transport key (+ `dir`), nothing else. - Removes: the three codes; `LOCAL_OVERRIDE_FILE`; `DEPENDENCY_MODES`; `DEFAULT_DEPENDENCY_MODE`; `DependencyMode`. -- [ ] **Step 1: Failing tests.** TS `sdk/test/config.test.ts`: change the first FR-023 test to expect `[{ name: "acme-common", path: "../lib/metaobjects" }]` (no `mode`) and add `{ name: "a", path: "x", mode: "reference" }` to the refused list. Python: rename/rewrite `test_dependencies_parse_with_mode_default` as above; add `[{"name": "a", "path": "x", "mode": "reference"}]` to `bad`. Python scope: `assert not matches_scope("acme::Order\n", compile_scope({"include": ["acme::*"]}))`. Run: +- [ ] **Step 1: Failing tests.** TS `sdk/test/config.test.ts`: change the first FR-023 test to expect `[{ name: "acme-common", path: "../lib/metaobjects" }]` (no `mode`) and add `{ name: "a", path: "x", mode: "reference" }` to the refused list. Python: rename/rewrite `test_dependencies_parse_with_mode_default` as above; add `[{"name": "a", "path": "x", "mode": "reference"}]` to `bad`. Python scope: `assert not matches_scope("acme::Order\n", compile_scope(["acme::Order"]))` — NOTE (corrected 2026-09-11, this line was wrong twice): `compile_scope` takes POSITIONAL sequences `(include, exclude)`, NOT a dict; and the pattern must be LITERAL. A wildcard `"acme::*"` compiles to `^acme::[^:]*$`, and `[^:]` admits `\n`, so `match` and `fullmatch` BOTH return True — it cannot isolate the bug. `"acme::Order"` compiles to `^acme::Order$`, where `match` is True and `fullmatch` is False: the divergence, exactly. Run: ```bash cd server/typescript/packages/sdk && bun test test/config.test.ts cd server/python && uv run --extra integration pytest tests/config/test_neutral_config.py tests/conformance/test_scope_conformance.py -q From 98dbb9513c8ee54686173b04817e2aa7132a95ff Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 20:25:29 -0400 Subject: [PATCH 15/62] feat(deps): lock and manifest schemas with sha256 integrity (FR-023) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 7 (Phase 1a): the data contracts for `metaobjects.pkg.json` (ManifestSchema) and `.metaobjects/deps.lock.json` (LockSchema/LockEntrySchema), plus sha256 content hashing, pure and with no consumer yet — the collection resolver that folds a dependency's artifact into the loaded tree, and the package-based exclusion key, land next. - `packages`/`nodes` are both required and both validated sorted on every schema, per the 2026-09-11 ruling (§11.5): `packages` is the exclusion key, `nodes` stays for sync's artifact check, node-collision detection and the package-ownership refusal. - No `mode` field anywhere (Task 6 removed it; a resurrected `mode` key is a hard parse error via `.strict()` in TS and an unknown-key check in Python). - `writeLock` sorts dependency keys, 2-space indent, trailing newline, no timestamps or host paths — byte-identical across machines. - Python hand-validates (no schema library), raising `ParseError` with `ERR_DEPENDENCY_MANIFEST_INVALID` / `ERR_DEPENDENCY_SNAPSHOT_STALE`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../src/metaobjects/config/dependencies.py | 291 +++++++++++++++++- .../python/tests/config/test_dependencies.py | 89 ++++++ .../packages/sdk/src/dependencies.ts | 193 +++++++++++- server/typescript/packages/sdk/src/index.ts | 16 +- .../packages/sdk/test/dependencies.test.ts | 51 +++ 5 files changed, 628 insertions(+), 12 deletions(-) create mode 100644 server/python/tests/config/test_dependencies.py create mode 100644 server/typescript/packages/sdk/test/dependencies.test.ts diff --git a/server/python/src/metaobjects/config/dependencies.py b/server/python/src/metaobjects/config/dependencies.py index d1f2cbc12..1abac781a 100644 --- a/server/python/src/metaobjects/config/dependencies.py +++ b/server/python/src/metaobjects/config/dependencies.py @@ -1,11 +1,21 @@ """FR-023 — metadata dependencies: constants shared by the `dependencies` key -of `.metaobjects/config.json` (DESIGN §3.1) and by later tasks (manifest, -lock, snapshot, sync). Mirrors -`server/typescript/packages/sdk/src/dependencies.ts` — constants only here; -resolving a declared dependency to bytes on disk lands in a later task. +of `.metaobjects/config.json` (DESIGN §3.1), plus the manifest/lock +validation and integrity hashing every later task (the collection resolver) +builds on. Mirrors `server/typescript/packages/sdk/src/dependencies.ts` — +hand-validated dicts here rather than a schema library, same rules; resolving +a declared dependency to bytes on disk lands in a later task. """ from __future__ import annotations +import hashlib +import json +import re +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from metaobjects.errors import ErrorCode, ParseError + #: Directory (under `.metaobjects/`) holding the synced snapshot artifacts, #: one subdirectory per dependency name: `.metaobjects/deps//`. DEPS_DIR = "deps" @@ -28,3 +38,276 @@ #: Prefix of the `integrity` field's value: `"sha256-" + lowercase hex sha256 #: of the artifact bytes` (DESIGN §3, "Hash format"). INTEGRITY_PREFIX = "sha256-" + +#: Directory (relative to a project root) holding `.metaobjects/config.json`, +#: the synced snapshot and the lock — a private, module-local convention +#: mirroring `neutral_config._METAOBJECTS_DIR`. Not a named constant in the +#: cross-port list (DEPS_DIR/LOCK_FILE/etc. above): those are basenames +#: shared with TypeScript; this is Python's own path-join detail, same as +#: TypeScript's `DEFAULT_METAOBJECTS_DIR` (`sdk/src/metadata-files.ts`). +_METAOBJECTS_DIR = ".metaobjects" + +#: Mirrors the TS `DependencyName` regex in `sdk/src/dependencies.ts` exactly +#: (DESIGN §3.1) — also the shape of a manifest's `name` and a lock's +#: dependency keys. +_DEPENDENCY_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9._-]*$") + +#: `major.minor`, no patch — the toolchain's `METAMODEL_VERSION` at +#: generation time (DESIGN §3.2). +_METAMODEL_VERSION_RE = re.compile(r"^\d+\.\d+$") + +#: `sha256-` + 64 lowercase hex digits (DESIGN §3, "Hash format"). +_INTEGRITY_RE = re.compile(r"^sha256-[0-9a-f]{64}$") + +#: Allowed top-level keys of `metaobjects.pkg.json` (DESIGN §3.2). No `mode` +#: (FR-023 §11.3 — removed by Task 6, not reintroduced). +_MANIFEST_KEYS = { + "schema_version", + "name", + "version", + "metamodelVersion", + "artifact", + "integrity", + "packages", + "nodes", +} + +#: Allowed top-level keys of `.metaobjects/deps.lock.json` (DESIGN §3.3). +_LOCK_KEYS = {"schema_version", "dependencies"} + +#: Allowed keys of one `deps.lock.json` `dependencies` entry — the manifest +#: fields minus `schema_version`/`name` (both implicit) plus `resolvedFrom`. +#: No `mode`. +_LOCK_ENTRY_KEYS = { + "version", + "metamodelVersion", + "artifact", + "integrity", + "packages", + "nodes", + "resolvedFrom", +} + +#: The one transport key a lock entry's `resolvedFrom` carries. +_RESOLVED_FROM_TRANSPORT_KEYS = ("path", "npm", "python") + + +def _is_sorted(values: list[str]) -> bool: + """`True` iff `values` is already in strict ascending order.""" + return all(values[i] < values[i + 1] for i in range(len(values) - 1)) + + +def _validate_sorted_string_array( + obj: dict[str, Any], field: str, code: ErrorCode, context: str +) -> list[str]: + value = obj.get(field) + if not isinstance(value, list) or not all(isinstance(v, str) and v for v in value): + raise ParseError( + f"{context}: '{field}' must be an array of non-empty strings", code=code + ) + if not _is_sorted(value): + raise ParseError(f"{context}: '{field}' must be sorted", code=code) + return list(value) + + +def _validate_common_fields(obj: dict[str, Any], code: ErrorCode, context: str) -> dict[str, Any]: + """Validate the fields a manifest and a lock entry share: `version`, + `metamodelVersion`, `artifact`, `integrity`, `packages`, `nodes`. Same + shape in both callers — only the failure `code` and diagnostic `context` + differ (manifest vs. one lock entry).""" + version = obj.get("version") + if not isinstance(version, str) or not version: + raise ParseError(f"{context}: 'version' must be a non-empty string", code=code) + + metamodel_version = obj.get("metamodelVersion") + if not isinstance(metamodel_version, str) or not _METAMODEL_VERSION_RE.match( + metamodel_version + ): + raise ParseError(f"{context}: 'metamodelVersion' must match ^\\d+\\.\\d+$", code=code) + + artifact = obj.get("artifact") + if not isinstance(artifact, str) or not artifact.endswith(ARTIFACT_SUFFIX): + raise ParseError(f"{context}: 'artifact' must end with {ARTIFACT_SUFFIX!r}", code=code) + + integrity = obj.get("integrity") + if not isinstance(integrity, str) or not _INTEGRITY_RE.match(integrity): + raise ParseError( + f"{context}: 'integrity' must match ^sha256-[0-9a-f]{{64}}$", code=code + ) + + packages = _validate_sorted_string_array(obj, "packages", code, context) + nodes = _validate_sorted_string_array(obj, "nodes", code, context) + + return { + "version": version, + "metamodelVersion": metamodel_version, + "artifact": artifact, + "integrity": integrity, + "packages": packages, + "nodes": nodes, + } + + +def validate_manifest(obj: object) -> dict[str, Any]: + """Validate a parsed `metaobjects.pkg.json` object (DESIGN §3.2). + + Raises `ParseError(code=ERR_DEPENDENCY_MANIFEST_INVALID)` on any shape + violation — an unknown key (in particular a resurrected `mode`), an + unsorted `packages`/`nodes`, or a malformed field. Returns the manifest + normalized (the same fields, a plain dict). + """ + code = ErrorCode.ERR_DEPENDENCY_MANIFEST_INVALID + context = MANIFEST_FILE + if not isinstance(obj, dict): + raise ParseError(f"{context}: must be an object", code=code) + + extra_keys = set(obj.keys()) - _MANIFEST_KEYS + if extra_keys: + raise ParseError(f"{context}: unknown key(s): {sorted(extra_keys)}", code=code) + + schema_version = obj.get("schema_version") + if isinstance(schema_version, bool) or schema_version != 1: + raise ParseError(f"{context}: 'schema_version' must be 1", code=code) + + name = obj.get("name") + if not isinstance(name, str) or not _DEPENDENCY_NAME_RE.match(name): + raise ParseError(f"{context}: 'name' must match ^[a-z0-9][a-z0-9._-]*$", code=code) + + common = _validate_common_fields(obj, code, context) + return {"schema_version": 1, "name": name, **common} + + +def _validate_resolved_from(obj: object, code: ErrorCode, context: str) -> dict[str, str]: + """Validate a lock entry's `resolvedFrom` — exactly one of `path` / `npm` + / `python`, `dir` legal only beside `npm`/`python`. Mirrors + `neutral_config._validate_dependency_spec`'s transport check, minus + `name` (a lock entry is already keyed by name).""" + if not isinstance(obj, dict): + raise ParseError(f"{context}: 'resolvedFrom' must be an object", code=code) + + transports = [k for k in _RESOLVED_FROM_TRANSPORT_KEYS if k in obj] + if len(transports) != 1: + raise ParseError( + f"{context}: 'resolvedFrom' must have exactly one of 'path' / 'npm' / 'python'", + code=code, + ) + transport = transports[0] + + transport_value = obj[transport] + if not isinstance(transport_value, str) or not transport_value.strip(): + raise ParseError( + f"{context}: 'resolvedFrom.{transport}' must be a non-empty string", code=code + ) + + allowed_keys = {transport} + if transport in ("npm", "python"): + allowed_keys.add("dir") + extra_keys = set(obj.keys()) - allowed_keys + if extra_keys: + raise ParseError( + f"{context}: 'resolvedFrom' unknown key(s): {sorted(extra_keys)}", code=code + ) + + result: dict[str, str] = {transport: transport_value} + if "dir" in obj: + dir_value = obj["dir"] + if not isinstance(dir_value, str) or not dir_value.strip(): + raise ParseError( + f"{context}: 'resolvedFrom.dir' must be a non-empty string", code=code + ) + result["dir"] = dir_value + return result + + +def validate_lock(obj: object) -> dict[str, Any]: + """Validate a parsed `.metaobjects/deps.lock.json` object (DESIGN §3.3). + + Raises `ParseError(code=ERR_DEPENDENCY_SNAPSHOT_STALE)` on any shape + violation — unsorted dependency keys, an entry with an unknown key + (`mode` included), two `resolvedFrom` transports, or a malformed field. + Returns the lock normalized: `dependencies` a plain dict, each entry's + fields as `_validate_common_fields` + `resolvedFrom` returns them. + """ + code = ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + context = LOCK_FILE + if not isinstance(obj, dict): + raise ParseError(f"{context}: must be an object", code=code) + + extra_keys = set(obj.keys()) - _LOCK_KEYS + if extra_keys: + raise ParseError(f"{context}: unknown key(s): {sorted(extra_keys)}", code=code) + + schema_version = obj.get("schema_version") + if isinstance(schema_version, bool) or schema_version != 1: + raise ParseError(f"{context}: 'schema_version' must be 1", code=code) + + dependencies = obj.get("dependencies") + if not isinstance(dependencies, dict): + raise ParseError(f"{context}: 'dependencies' must be an object", code=code) + + names = list(dependencies.keys()) + if not all(isinstance(n, str) for n in names): + raise ParseError(f"{context}: 'dependencies' keys must be strings", code=code) + if not _is_sorted(names): + raise ParseError(f"{context}: dependency keys must be sorted", code=code) + + validated: dict[str, Any] = {} + for name, entry in dependencies.items(): + entry_context = f"{context} ({name})" + if not isinstance(entry, dict): + raise ParseError(f"{entry_context}: must be an object", code=code) + entry_extra = set(entry.keys()) - _LOCK_ENTRY_KEYS + if entry_extra: + raise ParseError( + f"{entry_context}: unknown key(s): {sorted(entry_extra)}", code=code + ) + common = _validate_common_fields(entry, code, entry_context) + resolved_from = _validate_resolved_from(entry.get("resolvedFrom"), code, entry_context) + validated[name] = {**common, "resolvedFrom": resolved_from} + + return {"schema_version": 1, "dependencies": validated} + + +def sha256_integrity(data: bytes) -> str: + """`"sha256-" + lowercase hex sha256 of `data`` (DESIGN §3, "Hash format").""" + return f"{INTEGRITY_PREFIX}{hashlib.sha256(data).hexdigest()}" + + +def dependency_source_id(name: str, artifact: str) -> str: + """`dep:/` — a dependency artifact's source id (DESIGN + §2.3, "Source ids").""" + return f"{DEPENDENCY_SOURCE_ID_PREFIX}{name}/{artifact}" + + +def read_lock(config_dir: Path) -> dict[str, Any] | None: + """Read `.metaobjects/deps.lock.json` under `config_dir`, if present. + + Returns `None` when the file does not exist — a project declaring no + dependencies has no lock file, and that is not an error. A present but + malformed file (bad JSON, or a shape `validate_lock` rejects) raises. + """ + path = config_dir / _METAOBJECTS_DIR / LOCK_FILE + if not path.is_file(): + return None + try: + raw = json.loads(path.read_text()) + except (OSError, ValueError) as e: + raise ParseError( + f"{path} exists but could not be read as JSON: {e}", + code=ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE, + ) from e + return validate_lock(raw) + + +@dataclass(frozen=True) +class ResolvedDependency: + """A resolved dependency, ready for the collection resolver (a later + task) to fold its artifact into the loaded tree. Same fields as the TS + `ResolvedDependency` type in `sdk/src/dependencies.ts`, Pythonic case.""" + + name: str + version: str + packages: tuple[str, ...] + nodes: tuple[str, ...] + artifact_path: str + source_id: str diff --git a/server/python/tests/config/test_dependencies.py b/server/python/tests/config/test_dependencies.py new file mode 100644 index 000000000..7494660d2 --- /dev/null +++ b/server/python/tests/config/test_dependencies.py @@ -0,0 +1,89 @@ +"""FR-023 Task 7 — lock and manifest schemas, integrity hashing. Pure +validation and helpers only: nothing here resolves a dependency to bytes on +disk or consumes these shapes (the collection resolver is a later task). See +docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md +§3.2 (manifest), §3.3 (lock, minus `mode` — DESIGN §11.1/§11.3). +""" + +import copy +from pathlib import Path + +import pytest + +from metaobjects.config.dependencies import sha256_integrity, validate_lock, validate_manifest +from metaobjects.errors import ErrorCode, ParseError + +CORPUS = Path(__file__).resolve().parents[4] / "fixtures" / "dependency-conformance" / "artifacts" + +MANIFEST = { + "schema_version": 1, + "name": "acme-common", + "version": "1.0.0", + "metamodelVersion": "1.0", + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": ["acme::common"], + "nodes": ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"], +} + + +def test_sha256_integrity_of_the_pinned_v1_artifact_equals_the_readme_value() -> None: + data = (CORPUS / "acme-common-v1.json").read_bytes() + assert sha256_integrity(data) == ( + "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d" + ) + + +def test_validate_manifest_accepts_the_manifest() -> None: + validate_manifest(copy.deepcopy(MANIFEST)) + + +def test_validate_manifest_rejects_unsorted_nodes() -> None: + bad = {**copy.deepcopy(MANIFEST), "nodes": ["b", "a"]} + with pytest.raises(ParseError) as e: + validate_manifest(bad) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_MANIFEST_INVALID + assert "sorted" in str(e.value) + + +def test_validate_manifest_rejects_a_mode_key() -> None: + bad = {**copy.deepcopy(MANIFEST), "mode": "reference"} + with pytest.raises(ParseError) as e: + validate_manifest(bad) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_MANIFEST_INVALID + + +def _entry() -> dict: + rest = copy.deepcopy(MANIFEST) + del rest["schema_version"] + del rest["name"] + rest["resolvedFrom"] = {"path": "x"} + return rest + + +def test_validate_lock_accepts_one_sorted_entry() -> None: + validate_lock({"schema_version": 1, "dependencies": {"acme-common": _entry()}}) + + +def test_validate_lock_rejects_unsorted_keys() -> None: + entry = _entry() + with pytest.raises(ParseError) as e: + validate_lock({"schema_version": 1, "dependencies": {"b": entry, "a": entry}}) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + assert "sorted" in str(e.value) + + +def test_validate_lock_rejects_two_transports() -> None: + entry = _entry() + entry["resolvedFrom"] = {"path": "x", "npm": "y"} + with pytest.raises(ParseError) as e: + validate_lock({"schema_version": 1, "dependencies": {"a": entry}}) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + + +def test_validate_lock_rejects_a_mode_key() -> None: + entry = _entry() + entry["mode"] = "own" + with pytest.raises(ParseError) as e: + validate_lock({"schema_version": 1, "dependencies": {"a": entry}}) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE diff --git a/server/typescript/packages/sdk/src/dependencies.ts b/server/typescript/packages/sdk/src/dependencies.ts index a1beeebf8..3cb78c246 100644 --- a/server/typescript/packages/sdk/src/dependencies.ts +++ b/server/typescript/packages/sdk/src/dependencies.ts @@ -1,11 +1,17 @@ // server/typescript/packages/sdk/src/dependencies.ts // // FR-023 — metadata dependencies: the `dependencies` key of -// `.metaobjects/config.json` (DESIGN §3.1) plus the constants every later -// task (manifest, lock, snapshot, sync) shares. This module carries the -// schema and the constants only — resolving a declared dependency to bytes -// on disk, the manifest, the lock and the sync command land in later tasks. +// `.metaobjects/config.json` (DESIGN §3.1), the manifest/lock schemas and +// integrity hashing (DESIGN §3.2, §3.3, minus `mode` — DESIGN §11.1/§11.3), +// and the constants every later task shares. This module carries the +// schemas and helpers only — the collection resolver that actually folds a +// dependency's artifact into the loaded tree (`verifySnapshot`, exclusion by +// `packages`) lands in a later task. +import { createHash } from "node:crypto"; +import { readFile, stat, writeFile } from "node:fs/promises"; +import { join } from "node:path"; import { z } from "zod"; +import { DEFAULT_METAOBJECTS_DIR } from "./metadata-files.js"; /** Directory (under `.metaobjects/`) holding the synced snapshot artifacts, * one subdirectory per dependency name: `.metaobjects/deps//`. */ @@ -77,3 +83,182 @@ export const DependencySpecSchema = z.union([ export function dependencyName(spec: DependencySpec): string { return spec.name; } + +/** `/^sha256-[0-9a-f]{64}$/` — the `integrity` field's shape (DESIGN §3, + * "Hash format"): {@link INTEGRITY_PREFIX} followed by 64 lowercase hex + * digits (a sha256 digest). */ +export const IntegritySchema = z.string().regex(/^sha256-[0-9a-f]{64}$/); + +/** `true` iff `values` is already in strict ascending order. Bounded by the + * loop (`i` ranges over `1..values.length-1`, so both indices below are + * always in range) — the non-null assertions are safe, not a widening of + * `noUncheckedIndexedAccess`. */ +function isSortedAscending(values: readonly string[]): boolean { + for (let i = 1; i < values.length; i++) { + if (values[i - 1]! >= values[i]!) return false; + } + return true; +} + +/** + * A non-empty string array that must already be in ascending order — the + * shared shape of `packages` and `nodes` on both {@link ManifestSchema} and + * {@link LockEntrySchema}. Sortedness is validated, never imposed here: the + * writer (the publisher's `sharedModelFile()` generator, and `meta deps + * sync` copying the manifest into the lock) is responsible for producing + * sorted output; this schema only refuses to load one that isn't. + */ +function sortedStringArray(fieldName: string) { + return z + .array(z.string().min(1)) + .refine(isSortedAscending, { message: `${fieldName} must be sorted` }); +} + +/** + * `metaobjects.pkg.json` (DESIGN §3.2) — generated by the publisher's + * `sharedModelFile()` and read by the consumer's `meta deps sync`. + * `.strict()`: an unknown key — in particular a resurrected `mode` (FR-023 + * §11.3, removed by Task 6 and not reintroduced) — is a hard parse error + * rather than one zod silently strips. + */ +export const ManifestSchema = z + .object({ + schema_version: z.literal(1), + name: DependencyName, + /** The host package's own version (semver-ish; not narrowed further here). */ + version: z.string().min(1), + /** The toolchain's `METAMODEL_VERSION` at generation time — `major.minor`, + * no patch. */ + metamodelVersion: z.string().regex(/^\d+\.\d+$/), + /** The sibling artifact file's basename, e.g. `acme-common.metaobjects.json`. */ + artifact: z.string().endsWith(ARTIFACT_SUFFIX), + /** `sha256Integrity` of the artifact's bytes. */ + integrity: IntegritySchema, + /** Every package a top-level node in the artifact declares, sorted — + * the exclusion key (DESIGN §11.5): a consumer's top-level node whose + * package is in here, and whose FQN is NOT in `nodes`, is declared into + * a package it does not own (`ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, a + * later task). */ + packages: sortedStringArray("packages"), + /** The resolution key of every top-level node in the artifact, sorted — + * the public surface. Retained for `sync`'s artifact-vs-manifest check, + * node-collision detection, and the package-ownership refusal above + * (DESIGN §11.5) — it is NOT the exclusion key itself. */ + nodes: sortedStringArray("nodes"), + }) + .strict(); + +export type Manifest = z.infer; + +/** + * The one transport a lock entry's `resolvedFrom` carries — the declared + * spec's transport, verbatim, minus `name` (a lock entry is already keyed by + * name under `LockSchema.dependencies`). Mirrors `DependencySpecSchema`'s + * three arms exactly, `name` aside, including the `.strict()` on every arm: + * `{ path: "x", npm: "y" }` must match no arm, never silently pick one. + */ +const ResolvedFromSchema = z.union([ + z.object({ path: z.string().min(1) }).strict(), + z.object({ npm: z.string().min(1), dir: z.string().min(1).optional() }).strict(), + z.object({ python: z.string().min(1), dir: z.string().min(1).optional() }).strict(), +]); + +/** + * One entry of `.metaobjects/deps.lock.json`'s `dependencies` map (DESIGN + * §3.3) — the manifest's fields minus `schema_version`/`name` (both + * implicit: every entry is schema_version 1, and it is already keyed by + * name) plus `resolvedFrom`. No `mode` (FR-023 §11.3 — removed, not + * reintroduced). + */ +export const LockEntrySchema = z + .object({ + version: z.string().min(1), + metamodelVersion: z.string().regex(/^\d+\.\d+$/), + artifact: z.string().endsWith(ARTIFACT_SUFFIX), + integrity: IntegritySchema, + packages: sortedStringArray("packages"), + nodes: sortedStringArray("nodes"), + resolvedFrom: ResolvedFromSchema, + }) + .strict(); + +export type LockEntry = z.infer; + +/** + * `.metaobjects/deps.lock.json` (DESIGN §3.3) — written by `meta deps sync` + * only. `dependencies` is keyed by dependency name; the keys must already be + * sorted — deterministic across machines, no timestamps, no host paths. + */ +export const LockSchema = z + .object({ + schema_version: z.literal(1), + dependencies: z.record(DependencyName, LockEntrySchema), + }) + .strict() + .refine((lock) => isSortedAscending(Object.keys(lock.dependencies)), { + message: "deps.lock.json: dependency keys must be sorted", + }); + +export type Lock = z.infer; + +/** + * A resolved dependency, ready for the collection resolver (a later task) to + * fold its artifact into the loaded tree. Nothing in this task produces one + * — declared here (a named, documented interface) rather than left for a + * later task to invent ad hoc or reach for `any`, so that task's signature + * is fixed now. + */ +export type ResolvedDependency = { + readonly name: string; + readonly version: string; + readonly packages: readonly string[]; + readonly nodes: readonly string[]; + readonly artifactPath: string; + readonly sourceId: string; +}; + +/** `"sha256-" + lowercase hex sha256 of `bytes`` (DESIGN §3, "Hash format"). */ +export function sha256Integrity(bytes: Uint8Array | string): string { + return `${INTEGRITY_PREFIX}${createHash("sha256").update(bytes).digest("hex")}`; +} + +/** `dep:/` — a dependency artifact's `FileSource` id + * (DESIGN §2.3, "Source ids"). */ +export function dependencySourceId(name: string, artifact: string): string { + return `${DEPENDENCY_SOURCE_ID_PREFIX}${name}/${artifact}`; +} + +/** + * Read `.metaobjects/deps.lock.json` under `configDir`, if present. Returns + * `undefined` when the file does not exist — a project declaring no + * dependencies has no lock file, and that is not an error. A present but + * malformed file (bad JSON, or a shape {@link LockSchema} rejects) throws. + */ +export async function readLock(configDir: string): Promise { + const path = join(configDir, DEFAULT_METAOBJECTS_DIR, LOCK_FILE); + try { + const s = await stat(path); + if (!s.isFile()) return undefined; + } catch { + return undefined; + } + const raw = await readFile(path, "utf8"); + return LockSchema.parse(JSON.parse(raw)); +} + +/** + * Write `.metaobjects/deps.lock.json` under `configDir`. Deterministic + * output regardless of the input map's own key order: dependency keys + * sorted, 2-space indent, trailing newline — no timestamps, no host paths — + * so two machines that synced the same upstream bytes produce a + * byte-identical lock file. + */ +export async function writeLock(configDir: string, lock: Lock): Promise { + const sortedDependencies = Object.fromEntries( + Object.entries(lock.dependencies).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)), + ); + const sorted: Lock = { ...lock, dependencies: sortedDependencies }; + LockSchema.parse(sorted); // validate before writing, mirrors saveConfig (config.ts) + const path = join(configDir, DEFAULT_METAOBJECTS_DIR, LOCK_FILE); + await writeFile(path, JSON.stringify(sorted, null, 2) + "\n", "utf8"); +} diff --git a/server/typescript/packages/sdk/src/index.ts b/server/typescript/packages/sdk/src/index.ts index cf5bff1f7..b3564bd0c 100644 --- a/server/typescript/packages/sdk/src/index.ts +++ b/server/typescript/packages/sdk/src/index.ts @@ -19,9 +19,9 @@ export { ConfigSchema, DEFAULT_CONFIG, loadConfig, saveConfig, AllowTokenEnum } from "./config.js"; export type { Config } from "./config.js"; -// Metadata dependencies (FR-023) — the `dependencies` config key's schema -// and constants. Resolution, the manifest, the lock and sync land in later -// tasks. +// Metadata dependencies (FR-023) — the `dependencies` config key's schema, +// constants, and the manifest/lock schemas + integrity hashing (Task 7). +// The collection resolver that actually consumes them lands in a later task. export { DEPS_DIR, LOCK_FILE, @@ -31,8 +31,16 @@ export { INTEGRITY_PREFIX, DependencySpecSchema, dependencyName, + IntegritySchema, + ManifestSchema, + LockEntrySchema, + LockSchema, + sha256Integrity, + dependencySourceId, + readLock, + writeLock, } from "./dependencies.js"; -export type { DependencySpec } from "./dependencies.js"; +export type { DependencySpec, Manifest, LockEntry, Lock, ResolvedDependency } from "./dependencies.js"; // Meta Forge metadata types + attribute name constants (registered into a // TypeRegistry to let Loader parse decision/principle/etc. children + the diff --git a/server/typescript/packages/sdk/test/dependencies.test.ts b/server/typescript/packages/sdk/test/dependencies.test.ts new file mode 100644 index 000000000..0cd5d16d7 --- /dev/null +++ b/server/typescript/packages/sdk/test/dependencies.test.ts @@ -0,0 +1,51 @@ +// FR-023 Task 7 — lock and manifest schemas, integrity hashing. Pure schemas +// and helpers only: nothing here resolves a dependency to bytes on disk or +// consumes these types (the collection resolver is a later task). See +// docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md +// §3.2 (manifest), §3.3 (lock, minus `mode` — DESIGN §11.1/§11.3). +import { readFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import { expect, test } from "bun:test"; +import { LockSchema, ManifestSchema, sha256Integrity } from "../src/dependencies.js"; + +const CORPUS = resolve(import.meta.dir, "../../../../../fixtures/dependency-conformance/artifacts"); + +test("sha256Integrity of the pinned v1 artifact equals the README value", async () => { + expect(sha256Integrity(await readFile(`${CORPUS}/acme-common-v1.json`))).toBe( + "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + ); +}); + +test("manifest and lock schemas: sorted arrays, sorted keys, one transport", () => { + const manifest = { + schema_version: 1, + name: "acme-common", + version: "1.0.0", + metamodelVersion: "1.0", + artifact: "acme-common.metaobjects.json", + integrity: "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + packages: ["acme::common"], + nodes: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"], + }; + expect(() => ManifestSchema.parse(manifest)).not.toThrow(); + expect(() => ManifestSchema.parse({ ...manifest, nodes: ["b", "a"] })).toThrow(/sorted/); + expect(() => ManifestSchema.parse({ ...manifest, mode: "reference" })).toThrow(); + + const { schema_version: _s, name: _n, ...rest } = manifest; + const entry = { ...rest, resolvedFrom: { path: "x" } }; + expect(() => + LockSchema.parse({ schema_version: 1, dependencies: { "acme-common": entry } }), + ).not.toThrow(); + expect(() => + LockSchema.parse({ schema_version: 1, dependencies: { b: entry, a: entry } }), + ).toThrow(/sorted/); + expect(() => + LockSchema.parse({ + schema_version: 1, + dependencies: { a: { ...entry, resolvedFrom: { path: "x", npm: "y" } } }, + }), + ).toThrow(); + expect(() => + LockSchema.parse({ schema_version: 1, dependencies: { a: { ...entry, mode: "own" } } }), + ).toThrow(); +}); From 516b9b51fb3a05d95d46dccb3e64d98a0ec2c87b Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 20:37:16 -0400 Subject: [PATCH 16/62] test(deps): cover writeLock/readLock byte determinism and dependencySourceId MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix round 1 on Task 7 review (Important finding): readLock/writeLock (TS) and read_lock/dependency_source_id (both ports) shipped with zero test coverage. Adds a writeLock→readLock round-trip asserting the exact on-disk bytes (sorted dependency keys, 2-space indent, trailing newline, no host paths or timestamps) against a hand-written expected literal captured from a real run, plus direct assertions for dependencySourceId/dependency_source_id and read_lock's absent-file, present-file, malformed-JSON and shape-violation cases. No production code changed — the functions were already correct; every added test passed on first run. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../python/tests/config/test_dependencies.py | 55 +++++++- .../packages/sdk/test/dependencies.test.ts | 127 +++++++++++++++++- 2 files changed, 178 insertions(+), 4 deletions(-) diff --git a/server/python/tests/config/test_dependencies.py b/server/python/tests/config/test_dependencies.py index 7494660d2..6850aa2a9 100644 --- a/server/python/tests/config/test_dependencies.py +++ b/server/python/tests/config/test_dependencies.py @@ -6,11 +6,19 @@ """ import copy +import json from pathlib import Path import pytest -from metaobjects.config.dependencies import sha256_integrity, validate_lock, validate_manifest +from metaobjects.config.dependencies import ( + LOCK_FILE, + dependency_source_id, + read_lock, + sha256_integrity, + validate_lock, + validate_manifest, +) from metaobjects.errors import ErrorCode, ParseError CORPUS = Path(__file__).resolve().parents[4] / "fixtures" / "dependency-conformance" / "artifacts" @@ -87,3 +95,48 @@ def test_validate_lock_rejects_a_mode_key() -> None: with pytest.raises(ParseError) as e: validate_lock({"schema_version": 1, "dependencies": {"a": entry}}) assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + + +def test_dependency_source_id_is_dep_name_slash_artifact() -> None: + assert ( + dependency_source_id("acme-common", "acme-common.metaobjects.json") + == "dep:acme-common/acme-common.metaobjects.json" + ) + + +def test_read_lock_returns_none_when_the_file_does_not_exist(tmp_path: Path) -> None: + # No `.metaobjects/` directory at all — a project declaring no + # dependencies has no lock file, and that is not an error. + assert read_lock(tmp_path) is None + + +def test_read_lock_reads_and_validates_a_present_lock(tmp_path: Path) -> None: + entry = _entry() + lock = {"schema_version": 1, "dependencies": {"acme-common": entry}} + d = tmp_path / ".metaobjects" + d.mkdir(parents=True) + (d / LOCK_FILE).write_text(json.dumps(lock)) + + assert read_lock(tmp_path) == validate_lock(lock) + + +def test_read_lock_raises_on_malformed_json(tmp_path: Path) -> None: + d = tmp_path / ".metaobjects" + d.mkdir(parents=True) + (d / LOCK_FILE).write_text("{ not json") + with pytest.raises(ParseError) as e: + read_lock(tmp_path) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + + +def test_read_lock_raises_on_a_shape_violation(tmp_path: Path) -> None: + entry = _entry() + entry["mode"] = "own" # unknown key — same violation validate_lock rejects + lock = {"schema_version": 1, "dependencies": {"acme-common": entry}} + d = tmp_path / ".metaobjects" + d.mkdir(parents=True) + (d / LOCK_FILE).write_text(json.dumps(lock)) + + with pytest.raises(ParseError) as e: + read_lock(tmp_path) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE diff --git a/server/typescript/packages/sdk/test/dependencies.test.ts b/server/typescript/packages/sdk/test/dependencies.test.ts index 0cd5d16d7..df9e83d63 100644 --- a/server/typescript/packages/sdk/test/dependencies.test.ts +++ b/server/typescript/packages/sdk/test/dependencies.test.ts @@ -3,10 +3,22 @@ // consumes these types (the collection resolver is a later task). See // docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md // §3.2 (manifest), §3.3 (lock, minus `mode` — DESIGN §11.1/§11.3). +import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; import { readFile } from "node:fs/promises"; -import { resolve } from "node:path"; -import { expect, test } from "bun:test"; -import { LockSchema, ManifestSchema, sha256Integrity } from "../src/dependencies.js"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { + dependencySourceId, + LOCK_FILE, + LockSchema, + ManifestSchema, + readLock, + sha256Integrity, + writeLock, + type Lock, +} from "../src/dependencies.js"; +import { DEFAULT_METAOBJECTS_DIR } from "../src/metadata-files.js"; const CORPUS = resolve(import.meta.dir, "../../../../../fixtures/dependency-conformance/artifacts"); @@ -49,3 +61,112 @@ test("manifest and lock schemas: sorted arrays, sorted keys, one transport", () LockSchema.parse({ schema_version: 1, dependencies: { a: { ...entry, mode: "own" } } }), ).toThrow(); }); + +test("dependencySourceId is dep:/", () => { + expect(dependencySourceId("acme-common", "acme-common.metaobjects.json")).toBe( + "dep:acme-common/acme-common.metaobjects.json", + ); +}); + +const PINNED_INTEGRITY = "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d"; + +let projectRoot: string; +beforeEach(() => { + projectRoot = mkdtempSync(join(tmpdir(), "metaobjects-deps-lock-")); + mkdirSync(join(projectRoot, DEFAULT_METAOBJECTS_DIR), { recursive: true }); +}); +afterEach(() => { + rmSync(projectRoot, { recursive: true, force: true }); +}); + +test("readLock returns undefined when no lock file exists", async () => { + expect(await readLock(projectRoot)).toBeUndefined(); +}); + +test("writeLock: exact on-disk bytes — sorted keys, 2-space indent, trailing newline, no host paths/timestamps", async () => { + // Deliberately out-of-order insertion ("b-dep" before "a-dep") — writeLock + // must sort regardless of the caller's own map order. + const entryA = { + version: "1.0.0", + metamodelVersion: "1.0", + artifact: "a.metaobjects.json", + integrity: PINNED_INTEGRITY, + packages: ["acme::a"], + nodes: ["acme::a::Foo"], + resolvedFrom: { path: "../a" }, + }; + const entryB = { + version: "2.0.0", + metamodelVersion: "1.0", + artifact: "b.metaobjects.json", + integrity: PINNED_INTEGRITY, + packages: ["acme::b"], + nodes: ["acme::b::Bar"], + resolvedFrom: { path: "../b" }, + }; + const lock: Lock = { + schema_version: 1, + dependencies: { "b-dep": entryB, "a-dep": entryA }, + }; + + await writeLock(projectRoot, lock); + + const path = join(projectRoot, DEFAULT_METAOBJECTS_DIR, LOCK_FILE); + const raw = await readFile(path, "utf8"); + + // The exact bytes `meta deps sync` would write — asserted against the + // written TEXT, not the re-parsed object, since the byte-for-byte + // guarantee (DESIGN §3.3) is about what lands on disk. + const expected = + "{\n" + + ' "schema_version": 1,\n' + + ' "dependencies": {\n' + + ' "a-dep": {\n' + + ' "version": "1.0.0",\n' + + ' "metamodelVersion": "1.0",\n' + + ' "artifact": "a.metaobjects.json",\n' + + ` "integrity": "${PINNED_INTEGRITY}",\n` + + ' "packages": [\n' + + ' "acme::a"\n' + + " ],\n" + + ' "nodes": [\n' + + ' "acme::a::Foo"\n' + + " ],\n" + + ' "resolvedFrom": {\n' + + ' "path": "../a"\n' + + " }\n" + + " },\n" + + ' "b-dep": {\n' + + ' "version": "2.0.0",\n' + + ' "metamodelVersion": "1.0",\n' + + ' "artifact": "b.metaobjects.json",\n' + + ` "integrity": "${PINNED_INTEGRITY}",\n` + + ' "packages": [\n' + + ' "acme::b"\n' + + " ],\n" + + ' "nodes": [\n' + + ' "acme::b::Bar"\n' + + " ],\n" + + ' "resolvedFrom": {\n' + + ' "path": "../b"\n' + + " }\n" + + " }\n" + + " }\n" + + "}\n"; + expect(raw).toBe(expected); + + // Restated as independent, narrower checks so a future format change + // fails with a pinpointed message rather than only a giant string diff. + expect(raw.endsWith("\n")).toBe(true); + expect(raw.endsWith("\n\n")).toBe(false); + expect(raw.indexOf('"a-dep"')).toBeLessThan(raw.indexOf('"b-dep"')); + expect(raw).toContain('\n "dependencies"'); // 2-space indent, level 1 + expect(raw).toContain('\n "a-dep"'); // level 2 + expect(raw).toContain('\n "version"'); // level 3 + expect(raw).not.toContain(projectRoot); // no absolute host path leaked in + expect(raw).not.toMatch(/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}/); // no ISO timestamp + + const reloaded = await readLock(projectRoot); + expect(reloaded).toEqual(lock); + expect(reloaded && Object.keys(reloaded.dependencies)).toEqual(["a-dep", "b-dep"]); +}); From 7e994d084c8cac425afe43f89b9b9540fde08afc Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 20:59:21 -0400 Subject: [PATCH 17/62] =?UTF-8?q?feat(sdk):=20resolveCollection=20loads=20?= =?UTF-8?q?dependency=20snapshots=20first=20and=20composes=20the=20default?= =?UTF-8?q?-exclusion=20scope=20(FR-023=20=C2=A711)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `resolveCollection` now verifies a project's committed snapshot against `.metaobjects/deps.lock.json`, leads its file list with the dependency artifacts (name order, then the project's own files), and composes the two predicates every downstream surface acts on: inScope(fqn) = matchesScope(fqn, scope) && (!imported(fqn) || explicitlyIncludes(scope.include, pkg)) inMigrateScope(fqn) = (declaredMigrateScope?.(fqn) ?? true) && (!imported(fqn) || explicitlyIncludes(migrate.scope, pkg)) `imported` keys on the lock's `packages` (DESIGN §11.5), not its `nodes`. `explicitlyIncludes` requires a pattern to NAME a package literally, so `acme::common::**` opts in and `**` does not: a project that writes a wildcard to mean "all of my model" must not thereby generate, and migrate, somebody else's. `inMigrateScope` stays `undefined` exactly when the project declares no `migrate.scope` AND resolves no dependencies — the untouched path that leaves migrate's expected schema alone. A project with no dependencies resolves as it always did; `nodes` is retained for one reader, the new post-load `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` refusal, which tells an overlay of an imported node from a new local declaration in its package. `loadMemory` gains `fileIds` (so an artifact's nodes carry `dep:/` provenance rather than a local-looking basename) plus the imported sets, and runs the refusal AFTER the loader's own errors on BOTH arms — the caller-supplied file list and the self-resolving one, so an embedder calling `loadMemory(repoRoot)` does not silently lose it. The nine CLI load sites now thread all four through one helper, `collectionLoadOptions`. Corpus: 15 cases appended to fixtures/dependency-conformance (resolution, scope/migrate-scope composition, the ownership refusal and its overlay counter-case, five staleness arms, node collision, metamodel-major incompatibility, and a native-source-surface layout). The corpus is fully green; the sanctioned `collection.imported is not a function` failure ends here. Also fixes one pre-existing break this task's cli typecheck was the first command to see: the sdk root barrel exports two types named `Manifest` (the agent-context one via `export *`, FR-023's explicitly), and an explicit export shadows a star one, so `agent-context-staleness.ts` was resolving the wrong type once `dist/` was rebuilt. It now imports from the `/agent-context` subpath, as `init.ts` already did. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- fixtures/dependency-conformance/cases.json | 732 +++++++++++++++++- .../packages/cli/src/commands/docs.ts | 3 +- .../packages/cli/src/commands/gen.ts | 3 +- .../packages/cli/src/commands/migrate.ts | 9 +- .../cli/src/commands/prompt-snapshot.ts | 3 +- .../packages/cli/src/commands/verify.ts | 8 +- .../cli/src/lib/agent-context-staleness.ts | 8 +- .../cli/src/lib/collection-load-options.ts | 36 + .../typescript/packages/sdk/src/collection.ts | 132 +++- .../packages/sdk/src/dependencies.ts | 205 ++++- server/typescript/packages/sdk/src/index.ts | 9 +- server/typescript/packages/sdk/src/memory.ts | 123 ++- server/typescript/packages/sdk/src/sources.ts | 12 +- .../packages/sdk/test/collection.test.ts | 38 + .../packages/sdk/test/dependencies.test.ts | 25 + .../sdk/test/dependency-conformance.test.ts | 59 +- .../packages/sdk/test/memory.test.ts | 137 +++- .../sdk/test/order-independence.test.ts | 85 +- 18 files changed, 1554 insertions(+), 73 deletions(-) create mode 100644 server/typescript/packages/cli/src/lib/collection-load-options.ts diff --git a/fixtures/dependency-conformance/cases.json b/fixtures/dependency-conformance/cases.json index 80ad41a20..31de23905 100644 --- a/fixtures/dependency-conformance/cases.json +++ b/fixtures/dependency-conformance/cases.json @@ -5,11 +5,735 @@ "tree": { "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" }, - "config": { "schema_version": 1, "sources": [] }, - "expectFiles": ["metaobjects/meta.app.json"], + "config": { + "schema_version": 1, + "sources": [] + }, + "expectFiles": [ + "metaobjects/meta.app.json" + ], "expectImported": [], - "expectSelected": ["app::Order"], - "expectMigrateGoverned": ["app::Order"] + "expectSelected": [ + "app::Order" + ], + "expectMigrateGoverned": [ + "app::Order" + ] + }, + { + "name": "a-dependency-adds-its-artifact-to-the-resolved-set", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.app.json" + ], + "expectImported": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ], + "expectSelected": [ + "app::Order" + ], + "expectMigrateGoverned": [ + "app::Order" + ] + }, + { + "name": "dependency-order-in-config-does-not-change-the-resolved-set", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}", + ".metaobjects/deps/acme-extra/acme-extra.metaobjects.json": "{\n \"metadata.root\": {\n \"children\": [\n {\n \"object.value\": {\n \"name\": \"Address\",\n \"package\": \"acme::extra\",\n \"children\": [\n {\n \"field.string\": {\n \"name\": \"city\"\n }\n },\n {\n \"field.string\": {\n \"name\": \"street\"\n }\n }\n ]\n }\n },\n {\n \"object.entity\": {\n \"name\": \"Audited\",\n \"package\": \"acme::extra\",\n \"abstract\": true,\n \"children\": [\n {\n \"field.timestamp\": {\n \"name\": \"createdAt\"\n }\n }\n ]\n }\n },\n {\n \"object.entity\": {\n \"name\": \"Customer\",\n \"package\": \"acme::extra\",\n \"children\": [\n {\n \"source.rdb\": {\n \"@table\": \"customers\"\n }\n },\n {\n \"field.long\": {\n \"name\": \"id\"\n }\n },\n {\n \"field.string\": {\n \"name\": \"email\",\n \"@maxLength\": 120\n }\n },\n {\n \"identity.primary\": {\n \"name\": \"pk\",\n \"@fields\": [\n \"id\"\n ]\n }\n }\n ]\n }\n }\n ]\n }\n}\n" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-extra", + "path": "../acme-extra/metaobjects" + }, + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + }, + "acme-extra": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-extra/metaobjects" + }, + "artifact": "acme-extra.metaobjects.json", + "integrity": "sha256-414c8f1c557169623108102945fd248e510c9be152ca6aa93a8ee8837cd0d837", + "packages": [ + "acme::extra" + ], + "nodes": [ + "acme::extra::Address", + "acme::extra::Audited", + "acme::extra::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + ".metaobjects/deps/acme-extra/acme-extra.metaobjects.json", + "metaobjects/meta.app.json" + ], + "expectImported": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer", + "acme::extra::Address", + "acme::extra::Audited", + "acme::extra::Customer" + ], + "expectSelected": [ + "app::Order" + ] + }, + { + "name": "a-scope-include-naming-the-package-selects-its-nodes", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ], + "scope": { + "include": [ + "app::**", + "acme::common::**" + ] + } + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.app.json" + ], + "expectSelected": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer", + "app::Order" + ], + "expectMigrateGoverned": [ + "app::Order" + ] + }, + { + "name": "a-wildcard-include-does-not-name-a-package", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ], + "scope": { + "include": [ + "**" + ] + } + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.app.json" + ], + "expectSelected": [ + "app::Order" + ] + }, + { + "name": "a-migrate-scope-naming-the-package-governs-its-tables", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ], + "migrate": { + "scope": [ + "app::**", + "acme::common::**" + ] + } + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.app.json" + ], + "expectSelected": [ + "app::Order" + ], + "expectMigrateGoverned": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer", + "app::Order" + ] + }, + { + "name": "a-local-node-in-a-dependency-package-is-refused", + "tree": { + "metaobjects/meta.ext.json": "{\"metadata.root\":{\"package\":\"acme::common\",\"children\":[{\"object.value\":{\"name\":\"Note\",\"children\":[{\"field.string\":{\"name\":\"text\"}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.ext.json" + ], + "expectLoadError": "ERR_DEPENDENCY_PACKAGE_NOT_OWNED" + }, + { + "name": "an-overlay-of-a-dependency-node-is-not-refused", + "tree": { + "metaobjects/meta.ext.json": "{\"metadata.root\":{\"package\":\"acme::common\",\"children\":[{\"object.entity\":{\"name\":\"Customer\",\"overlay\":true,\"children\":[{\"field.string\":{\"name\":\"email\",\"overlay\":true,\"children\":[{\"view.text\":{\"name\":\"emailView\"}}]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.ext.json" + ], + "expectImported": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ], + "expectSelected": [] + }, + { + "name": "a-missing-lock-is-stale", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "expectError": "ERR_DEPENDENCY_SNAPSHOT_STALE" + }, + { + "name": "a-lock-entry-without-a-config-entry-is-stale", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectError": "ERR_DEPENDENCY_SNAPSHOT_STALE" + }, + { + "name": "a-config-entry-without-a-lock-entry-is-stale", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": {} + }, + "expectError": "ERR_DEPENDENCY_SNAPSHOT_STALE" + }, + { + "name": "a-missing-artifact-is-stale", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectError": "ERR_DEPENDENCY_SNAPSHOT_STALE" + }, + { + "name": "an-artifact-whose-hash-differs-is-stale", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1-widened.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectError": "ERR_DEPENDENCY_SNAPSHOT_STALE" + }, + { + "name": "two-dependencies-exporting-one-fqn-collide", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json", + ".metaobjects/deps/acme-dup/acme-dup.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + }, + { + "name": "acme-dup", + "path": "../acme-dup/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + }, + "acme-dup": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-dup/metaobjects" + }, + "artifact": "acme-dup.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectError": "ERR_DEPENDENCY_NODE_COLLISION" + }, + { + "name": "an-incompatible-metamodel-major-is-refused", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "2.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectError": "ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE" + }, + { + "name": "dependencies-are-read-under-a-native-source-surface", + "tree": { + "app/metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + "app/.metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "resolveFrom": "app", + "expectFiles": [ + "app/.metaobjects/deps/acme-common/acme-common.metaobjects.json", + "app/metaobjects/meta.app.json" + ] } ] } diff --git a/server/typescript/packages/cli/src/commands/docs.ts b/server/typescript/packages/cli/src/commands/docs.ts index eca93f7bd..056a1534f 100644 --- a/server/typescript/packages/cli/src/commands/docs.ts +++ b/server/typescript/packages/cli/src/commands/docs.ts @@ -15,6 +15,7 @@ import { resolve as resolvePath, basename } from "node:path"; import { mkdir, writeFile } from "node:fs/promises"; import { log } from "../lib/log.js"; import { loadMemoryOptionsFrom, loadMetaobjectsConfig, resolveGenConfigDir } from "../lib/load-metaobjects-config.js"; +import { collectionLoadOptions } from "../lib/collection-load-options.js"; import { loadMemory, resolveCollection, resolveConfigDir, type Collection } from "@metaobjectsdev/sdk"; import { existsSync } from "node:fs"; import { join } from "node:path"; @@ -515,7 +516,7 @@ export async function docsCommand( let root; try { root = await loadMemory(collection.configDir, { - files: collection.files, + ...collectionLoadOptions(collection), ...configLoadOptions, }); } catch (err) { diff --git a/server/typescript/packages/cli/src/commands/gen.ts b/server/typescript/packages/cli/src/commands/gen.ts index abd7849ea..d522e70f4 100644 --- a/server/typescript/packages/cli/src/commands/gen.ts +++ b/server/typescript/packages/cli/src/commands/gen.ts @@ -2,6 +2,7 @@ import { relative } from "node:path"; import { parseGenArgs } from "../lib/args.js"; import { resolveGenConfig } from "../lib/config.js"; import { loadMemoryOptionsFrom, loadMetaobjectsConfig, resolveGenCollection, resolveGenConfigDir } from "../lib/load-metaobjects-config.js"; +import { collectionLoadOptions } from "../lib/collection-load-options.js"; import { formatGenResult, formatGenResultToon, type GenFileEntry, type GenFileStatus } from "../lib/output.js"; import { formatGenResultJson } from "../lib/output-json.js"; import type { OutputFormat } from "../lib/format.js"; @@ -122,7 +123,7 @@ export async function genCommand(args: string[], cwd: string, fmt: OutputFormat let metadata; try { metadata = await loadMemory(genCollection.configDir, { - files: genCollection.files, + ...collectionLoadOptions(genCollection), ...loadMemoryOptionsFrom(forgeConfig), }); } catch (err) { diff --git a/server/typescript/packages/cli/src/commands/migrate.ts b/server/typescript/packages/cli/src/commands/migrate.ts index 3651d3438..01eda14d1 100644 --- a/server/typescript/packages/cli/src/commands/migrate.ts +++ b/server/typescript/packages/cli/src/commands/migrate.ts @@ -13,6 +13,7 @@ import { buildKyselyFromUrl, redactUrl } from "../lib/kysely.js"; import { log } from "../lib/log.js"; import { loadMemory, resolveCollection, resolveConfigDir, type Collection } from "@metaobjectsdev/sdk"; import { loadMemoryOptionsFrom, loadMetaobjectsConfig, resolveGenConfigDir } from "../lib/load-metaobjects-config.js"; +import { collectionLoadOptions } from "../lib/collection-load-options.js"; import { migrateScopeMismatch, outOfScopeNote } from "../lib/migrate-scope.js"; import { allowOptionFor, @@ -629,7 +630,7 @@ export async function migrateCommand( let metadata; try { metadata = await loadMemory(collection.configDir, { - files: collection.files, + ...collectionLoadOptions(collection), ...postgresLoadOptions, }); } catch (err) { @@ -1057,7 +1058,7 @@ export async function runBaseline( try { const collection = await resolveCollection(metaRoot); metadata = await loadMemory(collection.configDir, { - files: collection.files, + ...collectionLoadOptions(collection), ...baselineLoadOptions, }); } catch (err) { @@ -1196,7 +1197,7 @@ export async function runOfflineGenerate( try { collection = await resolveCollection(metaRoot); metadata = await loadMemory(collection.configDir, { - files: collection.files, + ...collectionLoadOptions(collection), ...offlineLoadOptions, }); } catch (err) { @@ -1461,7 +1462,7 @@ async function runD1Migrate( let metadata; try { metadata = await loadMemory(collection.configDir, { - files: collection.files, + ...collectionLoadOptions(collection), ...d1LoadOptions, }); } catch (err) { diff --git a/server/typescript/packages/cli/src/commands/prompt-snapshot.ts b/server/typescript/packages/cli/src/commands/prompt-snapshot.ts index 89f2a639e..5d2ffa00c 100644 --- a/server/typescript/packages/cli/src/commands/prompt-snapshot.ts +++ b/server/typescript/packages/cli/src/commands/prompt-snapshot.ts @@ -16,6 +16,7 @@ import { FileProvider } from "../lib/file-provider.js"; import { snapshotPaths, unifiedDiff } from "../lib/snapshot.js"; import { reportLoadError } from "../lib/load-error.js"; import { loadMemoryOptionsFrom, loadMetaobjectsConfig, resolveGenConfigDir } from "../lib/load-metaobjects-config.js"; +import { collectionLoadOptions } from "../lib/collection-load-options.js"; import { loadMemory, resolveCollection } from "@metaobjectsdev/sdk"; import { TYPE_TEMPLATE, TEMPLATE_ATTR_TEXT_REF, TEMPLATE_ATTR_FORMAT } from "@metaobjectsdev/metadata"; import { render, ESCAPERS, type RenderFormat } from "@metaobjectsdev/render"; @@ -75,7 +76,7 @@ export async function promptSnapshotCommand(args: string[], cwd: string): Promis let root; try { root = await loadMemory(collection.configDir, { - files: collection.files, + ...collectionLoadOptions(collection), ...configLoadOptions, }); } catch (err) { diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index d3ce8fcc2..893447278 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -24,6 +24,7 @@ import { replayRemedy } from "../lib/replay-remedy.js"; import { FileProvider } from "../lib/file-provider.js"; import { derivePayloadFieldTree } from "../lib/payload-field-tree.js"; import { loadMemoryOptionsFrom, loadMetaobjectsConfig, resolveGenCollection, resolveGenConfigDir } from "../lib/load-metaobjects-config.js"; +import { collectionLoadOptions } from "../lib/collection-load-options.js"; import { computeCodegenDrift } from "../lib/codegen-drift.js"; import { computeDocsDrift } from "../lib/docs-drift.js"; import { @@ -248,7 +249,7 @@ export async function verifyCommand( let root: Awaited>; try { root = await loadMemory(collection.configDir, { - files: collection.files, + ...collectionLoadOptions(collection), ...configLoadOptions, strict: !flags.lax, }); @@ -617,7 +618,8 @@ export async function verifyCommand( `meta verify — requirements: ${s.total} entries (${s.functional} functional, ` + `${s.architectural} architectural) — ${parts.join(", ")}; ` + `${s.entitiesClaimed}/${s.entitiesTotal} entities claimed, ` + - `counted over ${collection.files.length} metadata file(s).`, + `counted over ${collection.files.length} metadata file(s), ` + + `${collection.dependencies.length} from dependencies.`, ); if (s.undecided > 0) { say( @@ -1249,7 +1251,7 @@ export async function verifyCommand( if (genCollection !== collection) { try { codegenRoot = await loadMemory(genCollection.configDir, { - files: genCollection.files, + ...collectionLoadOptions(genCollection), ...configLoadOptions, strict: !flags.lax, }); diff --git a/server/typescript/packages/cli/src/lib/agent-context-staleness.ts b/server/typescript/packages/cli/src/lib/agent-context-staleness.ts index 0745ef2dc..7703da83d 100644 --- a/server/typescript/packages/cli/src/lib/agent-context-staleness.ts +++ b/server/typescript/packages/cli/src/lib/agent-context-staleness.ts @@ -1,6 +1,12 @@ import { existsSync, readFileSync } from "node:fs"; import { join } from "node:path"; -import { AGENT_CONTEXT_MANIFEST_PATH, agentContextStaleness, type Manifest } from "@metaobjectsdev/sdk"; +// The agent-context subpath, not the root barrel: `@metaobjectsdev/sdk` exports TWO +// types named `Manifest` — this one (the scaffolded agent context) via `export *`, +// and FR-023's `metaobjects.pkg.json` explicitly. An explicit export shadows a +// star one, so the root barrel now hands out the dependency manifest under this +// name. `init.ts` already imports these from the subpath; this was the last site +// reading them from the root. +import { AGENT_CONTEXT_MANIFEST_PATH, agentContextStaleness, type Manifest } from "@metaobjectsdev/sdk/agent-context"; import { cliVersion } from "./version.js"; import { log } from "./log.js"; diff --git a/server/typescript/packages/cli/src/lib/collection-load-options.ts b/server/typescript/packages/cli/src/lib/collection-load-options.ts new file mode 100644 index 000000000..c9602de5f --- /dev/null +++ b/server/typescript/packages/cli/src/lib/collection-load-options.ts @@ -0,0 +1,36 @@ +// server/typescript/packages/cli/src/lib/collection-load-options.ts +// +// FR-023 — everything a resolved `Collection` contributes to a metadata LOAD, +// in one helper. +// +// One helper rather than four properties spread by hand at each of the nine +// `loadMemory` call sites, for the reason `loadMemoryOptionsFrom` exists next +// door (#333): threading one and forgetting another is how a capability reaches +// some commands and not others. Here the stakes are higher than a missing +// provider — a site that passes `files` alone loads a dependency's artifact +// under a local-looking source id AND loses the +// `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` refusal, so a node declared into a +// dependency's package is silently excluded from that command's output instead +// of being refused by name. +import type { Collection, LoadMemoryOptions } from "@metaobjectsdev/sdk"; + +/** + * The load half of a `Collection`: the resolved file list (dependency artifacts + * first, then the project's own files), the `dep:/` source ids + * those artifacts load under, and the two imported sets the ownership refusal + * reads. + * + * The return type is a `Pick` of {@link LoadMemoryOptions} rather than a + * hand-written shape, so renaming or retyping an option there is a compile + * error here rather than a silently-dropped key. + */ +export function collectionLoadOptions( + collection: Collection, +): Required> { + return { + files: collection.files, + fileIds: collection.fileIds, + importedPackages: collection.importedPackages, + importedNodes: collection.importedNodes, + }; +} diff --git a/server/typescript/packages/sdk/src/collection.ts b/server/typescript/packages/sdk/src/collection.ts index 67143b4c2..b4faa9d0a 100644 --- a/server/typescript/packages/sdk/src/collection.ts +++ b/server/typescript/packages/sdk/src/collection.ts @@ -12,11 +12,26 @@ // name — this is where that assumption is allowed to live, exactly once. import { dirname, extname, join, resolve } from "node:path"; import { readdir, readFile } from "node:fs/promises"; -import { ParseError, codeSource, SUBTYPE_ROOT, TYPE_METADATA } from "@metaobjectsdev/metadata"; +import { + ParseError, + codeSource, + packageOfResolutionKey, + SUBTYPE_ROOT, + TYPE_METADATA, +} from "@metaobjectsdev/metadata"; import { CONFIG_FILE, loadConfig, type Config } from "./config.js"; import { discoverCollectionRoot, exists, isDir } from "./discovery.js"; import { compileScope, matchesScope, type Scope } from "./scope.js"; import { DEFAULT_METADATA_DIR, DEFAULT_METAOBJECTS_DIR, isMetadataFile } from "./metadata-files.js"; +import { + explicitlyIncludes, + importedNodesOf, + importedPackagesOf, + readLock, + verifySnapshot, + type DependencySpec, + type ResolvedDependency, +} from "./dependencies.js"; import { DEFAULT_SOURCES, orderedPathSpecs, @@ -33,7 +48,13 @@ export interface Collection { /** Canonically-ordered absolute metadata file paths — see `resolveSources`. * Canonical, not sorted: within a directory source the walk order the * toolchain has always used is preserved, because it survives into - * generated output. */ + * generated output. + * + * FR-023: a project with dependencies leads with their snapshot ARTIFACTS, + * in dependency-name order, then its own files ({@link ownFiles}) — the + * bases a consumer's `extends` and `overlay: true` resolve against must be + * in the tree before the declarations that amend them. A project with no + * dependencies gets exactly the list it always got. */ readonly files: readonly string[]; /** Same set, carrying the contributing spec for provenance. */ readonly sources: readonly ResolvedSource[]; @@ -44,6 +65,32 @@ export interface Collection { * comes from" (`meta docs --site` groups its pages by source root) must not * silently lose a declared source because it happens to be empty today. */ readonly sourceRoots: readonly string[]; + /** FR-023 — the project's OWN metadata files: `files` without the dependency + * artifacts it leads with. Identical to `files` when nothing is imported. */ + readonly ownFiles: readonly string[]; + /** FR-023 — the resolved dependencies, in dependency-NAME order (never the + * config's declaration order), each verified against `.metaobjects/deps.lock.json` + * before this resolves. Empty for a project that declares none. */ + readonly dependencies: readonly ResolvedDependency[]; + /** FR-023 — the `FileSource` id each file loads under, for the dependency + * ARTIFACTS only (`dep:/`); own files are absent and keep the + * default `basename(path)`. Threaded to `loadMemory` so every node an artifact + * contributed carries provenance naming the dependency rather than a filename + * that reads like a local file. */ + readonly fileIds: ReadonlyMap; + /** FR-023 — the sorted union of every dependency's `packages`: THE exclusion + * key (DESIGN §11.5). Package-keyed, not node-keyed. */ + readonly importedPackages: readonly string[]; + /** FR-023 — the union of every dependency's `nodes`. Read by ONE consumer, the + * `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` refusal, which needs it to tell an + * overlay of an imported node from a new local declaration in its package. NOT + * the exclusion key. */ + readonly importedNodes: ReadonlySet; + /** FR-023 — is `fqn` in a package one of this project's dependencies owns? + * Imported metadata is loaded so the project's own model can RESOLVE against + * it, and excluded from every action surface unless the project's own scope + * names its package (DESIGN §11.1 item 2). */ + readonly imported: (fqn: string) => boolean; /** * Output filter for codegen: does this fully-qualified name survive the * collection's `scope`? Always defined — an unconfigured project compiles to @@ -67,6 +114,12 @@ export interface Collection { * matched nothing" refusal can name the patterns that missed. Always in * lockstep with `inMigrateScope`: both undefined, or both present. */ readonly migrateScopePatterns: readonly string[] | undefined; + /** FR-023 — the user's declared `migrate.scope` ALONE, before the imported + * suppression `inMigrateScope` composes onto it. In lockstep with + * `migrateScopePatterns` (both undefined, or both present), which is what + * keeps the "your migrate.scope matched nothing" refusal reading the DECLARED + * scope rather than the composed one. */ + readonly declaredMigrateScope: ((fqn: string) => boolean) | undefined; } /** The canonical-JSON document root key (`metadata.root`) and the sigil-free @@ -195,6 +248,7 @@ export async function resolveCollection( let specs: readonly SourceSpec[] = DEFAULT_SOURCES; let scopeSpec: Config["scope"]; let migrateSpec: string[] | undefined; + let dependencySpecs: readonly DependencySpec[] = []; if (hasConfig) { // No try/catch here: a config.json that EXISTS but fails to load @@ -207,6 +261,7 @@ export async function resolveCollection( if (cfg.sources.length > 0) specs = cfg.sources; scopeSpec = cfg.scope; migrateSpec = cfg.migrate?.scope; + dependencySpecs = cfg.dependencies; } // Only the DEFAULT is allowed to be absent — an explicitly declared source @@ -247,22 +302,85 @@ export async function resolveCollection( ); } - const sources = await resolveSources(configDir, specs); + const ownSources = await resolveSources(configDir, specs); + + // FR-023 — the dependency snapshot, verified against the lock (DESIGN §4.2). + // It resolves BEFORE the predicates below because every one of them closes + // over what it returns. A project that declares no dependencies AND has no + // lock file takes the short-circuit: nothing is read, nothing is verified, + // and everything below collapses to exactly its pre-FR-023 behaviour. + const lock = await readLock(configDir); + const dependencies = + dependencySpecs.length === 0 && lock === undefined + ? [] + : await verifySnapshot(configDir, dependencySpecs, lock); + + // The exclusion key is the lock's `packages`; `nodes` is carried for the + // ownership refusal alone (DESIGN §11.5). + const importedPackages = importedPackagesOf(dependencies); + const importedNodes = importedNodesOf(dependencies); + const imported = (fqn: string): boolean => importedPackages.has(packageOfResolutionKey(fqn)); + + // The artifacts LEAD the file list, in dependency-name order — see `files`. + // They carry `dependency`/`id` rather than a declared spec, which is what + // distinguishes them from a file some `sources` entry contributed. + const dependencySources: readonly ResolvedSource[] = dependencies.map((d) => ({ + file: d.artifactPath, + spec: { path: d.artifactPath }, + dependency: d.name, + id: d.sourceId, + })); + const ownFiles = ownSources.map((s) => s.file); + const scope = compileScope(toScope(scopeSpec)); const migrateScope = migrateSpec === undefined ? undefined : compileScope({ include: migrateSpec }); + const declaredMigrateScope = + migrateScope === undefined + ? undefined + : (fqn: string): boolean => matchesScope(fqn, migrateScope); + return { configDir, - files: sources.map((s) => s.file), - sources, + files: [...dependencySources.map((s) => s.file), ...ownFiles], + ownFiles, + sources: [...dependencySources, ...ownSources], + dependencies, + fileIds: new Map(dependencies.map((d) => [d.artifactPath, d.sourceId])), + importedPackages: [...importedPackages].sort(), + importedNodes, + imported, // Canonical (content) order, from `resolveSources`'s own ordering — so this // list is a pure function of the source SET, exactly like `files`. + // DECLARED sources only: a dependency's artifact is not a source root, and + // listing one would put another project's tree in "where this model comes + // from" (`meta docs --site` groups its pages by these). sourceRoots: [ ...new Set(orderedPathSpecs(specs).map((spec) => resolveSpecPath(configDir, spec))), ], - inScope: (fqn: string): boolean => matchesScope(fqn, scope), + // The declared scope, AND the default exclusion of imported metadata + // (DESIGN §11.1 item 2): a dependency's node survives only when the + // project's own `scope.include` NAMES its package. A wildcard reaches it + // and does not name it — see `explicitlyIncludes`. + inScope: (fqn: string): boolean => { + if (!matchesScope(fqn, scope)) return false; + const pkg = packageOfResolutionKey(fqn); + return !importedPackages.has(pkg) || explicitlyIncludes(scopeSpec?.include, pkg); + }, + // Undefined ONLY when the project declares no `migrate.scope` AND imports + // nothing — migrate-ts reads that undefined as "govern everything loaded", + // and it is what leaves the expected schema untouched. With a dependency + // present the predicate must exist even with no declared scope, because the + // publisher's tables are in the loaded tree and nobody here declared them. inMigrateScope: - migrateScope === undefined ? undefined : (fqn: string): boolean => matchesScope(fqn, migrateScope), + migrateSpec === undefined && dependencies.length === 0 + ? undefined + : (fqn: string): boolean => { + if (!(declaredMigrateScope?.(fqn) ?? true)) return false; + const pkg = packageOfResolutionKey(fqn); + return !importedPackages.has(pkg) || explicitlyIncludes(migrateSpec, pkg); + }, migrateScopePatterns: migrateSpec, + declaredMigrateScope, }; } diff --git a/server/typescript/packages/sdk/src/dependencies.ts b/server/typescript/packages/sdk/src/dependencies.ts index 3cb78c246..ccae3d920 100644 --- a/server/typescript/packages/sdk/src/dependencies.ts +++ b/server/typescript/packages/sdk/src/dependencies.ts @@ -3,14 +3,15 @@ // FR-023 — metadata dependencies: the `dependencies` key of // `.metaobjects/config.json` (DESIGN §3.1), the manifest/lock schemas and // integrity hashing (DESIGN §3.2, §3.3, minus `mode` — DESIGN §11.1/§11.3), -// and the constants every later task shares. This module carries the -// schemas and helpers only — the collection resolver that actually folds a -// dependency's artifact into the loaded tree (`verifySnapshot`, exclusion by -// `packages`) lands in a later task. +// the constants every port shares, and the snapshot verification + +// exclusion-key helpers `resolveCollection` composes into its predicates +// (`verifySnapshot`, `importedPackagesOf`, `importedNodesOf`, +// `explicitlyIncludes` — DESIGN §4.2, §11.1 item 2). import { createHash } from "node:crypto"; import { readFile, stat, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { z } from "zod"; +import { codeSource, METAMODEL_VERSION, packageOfResolutionKey, ParseError } from "@metaobjectsdev/metadata"; import { DEFAULT_METAOBJECTS_DIR } from "./metadata-files.js"; /** Directory (under `.metaobjects/`) holding the synced snapshot artifacts, @@ -262,3 +263,199 @@ export async function writeLock(configDir: string, lock: Lock): Promise { const path = join(configDir, DEFAULT_METAOBJECTS_DIR, LOCK_FILE); await writeFile(path, JSON.stringify(sorted, null, 2) + "\n", "utf8"); } + +// --------------------------------------------------------------------------- +// Snapshot verification + the exclusion key (DESIGN §4.2, §11.1 item 2) +// --------------------------------------------------------------------------- + +/** The MAJOR half of a `major.minor` metamodel version. The metadata contract + * is promised on the major alone (ADR-0035 Amendment 2): a dependency + * published against `1.3` loads fine here at `1.0`, one published against + * `2.0` does not. */ +function metamodelMajor(version: string): string { + return version.split(".")[0] ?? version; +} + +/** + * Every stale-snapshot refusal, in one place so every one of them ends with the + * command that fixes it. `never` (a function DECLARATION, so control-flow + * narrowing applies at the call sites) — a caller writes the check and the + * message, never the throw. + */ +function staleSnapshot(detail: string): never { + throw new ParseError(`${detail}; run \`meta deps sync\``, { + code: "ERR_DEPENDENCY_SNAPSHOT_STALE", + source: codeSource("verifySnapshot"), + }); +} + +/** + * Verify a project's committed snapshot against its lock, and resolve the + * dependencies the collection will load FIRST (DESIGN §4.2 step 2-3). + * + * The lock is the contract and the snapshot is the payload, so every way they + * can disagree is one refusal with one remedy — a missing lock, a lock entry + * with no spec, a spec with no lock entry, a missing artifact, an artifact whose + * bytes hash to something else. Two failures are NOT staleness and get their own + * codes, because `meta deps sync` would not fix either: a dependency published + * against a different metamodel MAJOR (`ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`) + * and two dependencies exporting the same fully-qualified node + * (`ERR_DEPENDENCY_NODE_COLLISION` — whichever loaded second would silently win). + * + * Result order is dependency NAME order, never the config's declaration order: + * the artifacts lead the file list, so declaration order would otherwise decide + * what the loader sees first (see `test/order-independence.test.ts`). + * + * A project with no dependencies and no lock resolves to `[]` without touching + * the filesystem — the byte-identical path. + * + * @param configDir absolute directory of the declaring config (the parent of + * `.metaobjects/`), which is where both the lock and the snapshot live. + */ +export async function verifySnapshot( + configDir: string, + specs: readonly DependencySpec[], + lock: Lock | undefined, +): Promise { + const declared = specs.map(dependencyName); + + if (lock === undefined) { + // No dependencies AND no lock is the untouched project, not a stale one. + if (declared.length === 0) return []; + staleSnapshot( + `${declared.length} dependenc${declared.length === 1 ? "y is" : "ies are"} declared ` + + `(${declared.join(", ")}) but there is no ${DEFAULT_METAOBJECTS_DIR}/${LOCK_FILE}`, + ); + } + + const entries = lock.dependencies; + const declaredNames = new Set(declared); + for (const name of Object.keys(entries)) { + if (!declaredNames.has(name)) { + staleSnapshot( + `${DEFAULT_METAOBJECTS_DIR}/${LOCK_FILE} locks dependency "${name}", which ` + + `${DEFAULT_METAOBJECTS_DIR}/config.json no longer declares`, + ); + } + } + + const resolved: ResolvedDependency[] = []; + for (const name of [...declaredNames].sort()) { + const entry = entries[name]; + if (entry === undefined) { + staleSnapshot( + `dependency "${name}" is declared but ${DEFAULT_METAOBJECTS_DIR}/${LOCK_FILE} has no entry for it`, + ); + } + + const artifactPath = join(configDir, DEFAULT_METAOBJECTS_DIR, DEPS_DIR, name, entry.artifact); + const bytes = await readFile(artifactPath).catch(() => undefined); + if (bytes === undefined) { + staleSnapshot(`the committed snapshot for "${name}" is missing (expected ${artifactPath})`); + } + const actual = sha256Integrity(bytes); + if (actual !== entry.integrity) { + staleSnapshot( + `the committed snapshot for "${name}" does not match the lock — ${artifactPath} hashes ` + + `to ${actual}, the lock records ${entry.integrity}`, + ); + } + + if (metamodelMajor(entry.metamodelVersion) !== metamodelMajor(METAMODEL_VERSION)) { + throw new ParseError( + `dependency "${name}" was published against metamodel ${entry.metamodelVersion}; ` + + `this toolchain speaks ${METAMODEL_VERSION}. A different metamodel MAJOR is a different ` + + `metadata contract — upgrade the toolchain, or use a release of "${name}" built against it.`, + { code: "ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE", source: codeSource("verifySnapshot") }, + ); + } + + resolved.push({ + name, + version: entry.version, + packages: entry.packages, + nodes: entry.nodes, + artifactPath, + sourceId: dependencySourceId(name, entry.artifact), + }); + } + + // Collision is checked across the WHOLE resolved set rather than pairwise as + // each is read, so the error names the two dependencies in name order however + // the config declared them. + const owner = new Map(); + for (const dep of resolved) { + for (const node of dep.nodes) { + const prior = owner.get(node); + if (prior !== undefined) { + throw new ParseError( + `dependencies "${prior}" and "${dep.name}" both export "${node}" — one fully-qualified ` + + `node cannot come from two places, and whichever loaded second would silently win`, + { code: "ERR_DEPENDENCY_NODE_COLLISION", source: codeSource("verifySnapshot") }, + ); + } + owner.set(node, dep.name); + } + } + + return resolved; +} + +/** + * THE exclusion key (DESIGN §11.5, ruled 2026-09-11): every PACKAGE the + * resolved dependencies own. A loaded node whose package is in here is + * imported — load-only unless the consumer's own scope names that package. + * + * Package-keyed, not node-keyed: it is one concept rather than two, and it + * matches the maintainer's own model ("everything loads into one tree, then you + * include and exclude packages"). The one hole that opens — a local node + * declared into a dependency's package silently never generating — is closed by + * the refusal in `memory.ts`, which is the only reader of `importedNodesOf`. + */ +export function importedPackagesOf(deps: readonly ResolvedDependency[]): Set { + return new Set(deps.flatMap((d) => [...d.packages])); +} + +/** + * Every fully-qualified node the resolved dependencies export. + * + * Read by ONE caller: the `ERR_DEPENDENCY_PACKAGE_NOT_OWNED` refusal, which + * needs it to tell a local OVERLAY of a dependency's node (its key is in here, + * so it merges and passes) from a genuinely new local declaration in the + * dependency's package (not in here, and refused). It is NOT the exclusion key + * — see {@link importedPackagesOf}. + */ +export function importedNodesOf(deps: readonly ResolvedDependency[]): Set { + return new Set(deps.flatMap((d) => [...d.nodes])); +} + +/** + * Does some pattern in `patterns` name `pkg` LITERALLY (DESIGN §11.1 item 2)? + * + * Drop the pattern's final segment — which names the node — and what remains + * must be wildcard-free and equal to `pkg`. So `acme::common::**` and + * `acme::common::Address` both name `acme::common`; `acme::**` and `**` reach + * its nodes but name nothing, and an absent or empty list names nothing. + * + * That asymmetry is the point. Imported metadata is load-only by default, and + * the opt-in has to be an act of naming: a project that writes `scope.include: + * ["**"]` to mean "all of MY model" must not thereby start generating, and + * migrating, someone else's. Matching is therefore NOT `matchesScope` — a + * pattern that MATCHES a package's nodes is a weaker statement than one that + * NAMES the package. + * + * `packageOfResolutionKey` does the segment drop, so this and the resolution-key + * grammar can never disagree about where a package ends. A `pkg` of `""` (a + * root-level node, which a dependency artifact cannot contain — every top-level + * node in one carries an explicit package) is never named: fail-closed. + */ +export function explicitlyIncludes( + patterns: readonly string[] | undefined, + pkg: string, +): boolean { + if (patterns === undefined || pkg === "") return false; + return patterns.some((pattern) => { + const named = packageOfResolutionKey(pattern); + return named !== "" && !named.includes("*") && named === pkg; + }); +} diff --git a/server/typescript/packages/sdk/src/index.ts b/server/typescript/packages/sdk/src/index.ts index b3564bd0c..43707165d 100644 --- a/server/typescript/packages/sdk/src/index.ts +++ b/server/typescript/packages/sdk/src/index.ts @@ -20,8 +20,9 @@ export { ConfigSchema, DEFAULT_CONFIG, loadConfig, saveConfig, AllowTokenEnum } export type { Config } from "./config.js"; // Metadata dependencies (FR-023) — the `dependencies` config key's schema, -// constants, and the manifest/lock schemas + integrity hashing (Task 7). -// The collection resolver that actually consumes them lands in a later task. +// constants, the manifest/lock schemas + integrity hashing, and the snapshot +// verification + exclusion-key helpers `resolveCollection` composes into +// `imported` / `inScope` / `inMigrateScope` (DESIGN §4.2, §11.1 item 2). export { DEPS_DIR, LOCK_FILE, @@ -39,6 +40,10 @@ export { dependencySourceId, readLock, writeLock, + verifySnapshot, + importedPackagesOf, + importedNodesOf, + explicitlyIncludes, } from "./dependencies.js"; export type { DependencySpec, Manifest, LockEntry, Lock, ResolvedDependency } from "./dependencies.js"; diff --git a/server/typescript/packages/sdk/src/memory.ts b/server/typescript/packages/sdk/src/memory.ts index 8bf161299..b38c7ede9 100644 --- a/server/typescript/packages/sdk/src/memory.ts +++ b/server/typescript/packages/sdk/src/memory.ts @@ -1,7 +1,11 @@ import { + codeSource, composeRegistry, coreProviders, MetaDataLoader, + packageOfResolutionKey, + ParseError, + TYPE_OBJECT, type MetaDataTypeProvider, type MetaRoot, } from "@metaobjectsdev/metadata"; @@ -47,6 +51,35 @@ export interface LoadMemoryOptions { * way; it can no longer diverge from what the config declares. */ files?: readonly string[]; + /** + * FR-023 — the `FileSource` id to load a given path under, normally + * `resolveCollection(...).fileIds`. A dependency's snapshot artifact is a + * file on the consumer's own disk, so without this its provenance would read + * like a local file; mapped, every node it contributes carries + * `dep:/` (ADR-0009). Paths absent from the map keep the + * default `basename(path)`. + * + * Read only on the caller-supplied-`files` arm — with no `files`, the + * collection this function resolves supplies its own. + */ + fileIds?: ReadonlyMap; + /** + * FR-023 — the packages this project's dependencies own + * (`resolveCollection(...).importedPackages`). With {@link importedNodes}, + * enables the post-load ownership refusal: a top-level object declared into + * one of these packages that the dependency does not export is + * `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`. Omitted (or empty), nothing is checked. + * + * Read only on the caller-supplied-`files` arm, same as {@link fileIds}. + */ + importedPackages?: readonly string[]; + /** + * FR-023 — every fully-qualified node those dependencies export + * (`resolveCollection(...).importedNodes`). It is what tells an OVERLAY of an + * imported node (its key is in here; it merged, and passes) from a new local + * declaration in the dependency's package (refused). + */ + importedNodes?: ReadonlySet; /** * MetaObjects-shipped library packages to load ALONGSIDE the project's own files * (e.g. `["ai"]` for `metaobjects::ai::LlmCallBase`). Prepended, so an @@ -134,9 +167,26 @@ export async function loadMemory( // scanned `/` directly, so a caller that copied the // routed shape but forgot `files` silently loaded from a directory the // project's config may never have mentioned. - const paths = options?.files !== undefined - ? [...options.files] - : [...(await resolveCollection(repoRoot)).files]; + let paths: string[]; + let fileIds: ReadonlyMap | undefined; + let importedPackages: readonly string[] | undefined; + let importedNodes: ReadonlySet | undefined; + if (options?.files !== undefined) { + paths = [...options.files]; + fileIds = options.fileIds; + importedPackages = options.importedPackages; + importedNodes = options.importedNodes; + } else { + // FR-023: on this arm the COLLECTION is the authority for all four, never + // the caller — an embedder calling `loadMemory(repoRoot)` with no `files` + // passes none of them, and must still get the dependency artifacts' source + // ids and the ownership refusal that the routed CLI commands get. + const collection = await resolveCollection(repoRoot); + paths = [...collection.files]; + fileIds = collection.fileIds; + importedPackages = collection.importedPackages; + importedNodes = collection.importedNodes; + } const loader = new MetaDataLoader({ registry, @@ -154,12 +204,77 @@ export async function loadMemory( options?.libraries !== undefined && options.libraries.length > 0 ? (await import("@metaobjectsdev/metadata/library")).librarySources([...options.libraries]) : []; - const result = await loader.load([...libSources, ...paths.map((p) => new FileSource(p))]); + const result = await loader.load([ + ...libSources, + ...paths.map((p) => { + const id = fileIds?.get(p); + return id === undefined ? new FileSource(p) : new FileSource(p, { id }); + }), + ]); if (result.errors.length > 0) { const first = result.errors[0]!; throw first; } + // AFTER the loader's own errors, never before — the ordering is the whole + // mechanism. A local `overlay: true` whose target the upstream removed fails + // during load with ERR_OVERLAY_NO_TARGET, and the merged tree drops the + // overlay flag, so this walk cannot tell an overlay from a new declaration. + // Only an UNFLAGGED new declaration survives to here. + refuseUnownedPackages(result.root, importedPackages, importedNodes); + return result.root; } + +/** + * FR-023 §11.5 — a consumer may not declare a NEW top-level node into a package + * one of its dependencies owns. + * + * This is what keeps the package-keyed exclusion rule from failing silently. + * Imported-ness is decided by PACKAGE, so such a node would be excluded from + * this project's own codegen, migrate and ledger — producing no output and no + * error. An overlay of the dependency's own node is untouched: its resolution + * key is in `importedNodes`, because the node it merged into came from the + * artifact. + * + * No-op when nothing is imported, which is every project that declares no + * dependencies. + */ +function refuseUnownedPackages( + root: MetaRoot, + importedPackages: readonly string[] | undefined, + importedNodes: ReadonlySet | undefined, +): void { + if (importedPackages === undefined || importedPackages.length === 0) return; + const packages = new Set(importedPackages); + const nodes = importedNodes ?? new Set(); + + // ADR-0039 SANCTIONED own-accessor case: a root-level scan. `MetaRoot` has no + // super, so own and effective children are the same set here — and the + // question asked is precisely "what did this tree declare at the top level", + // which is the own layer by definition. + for (const node of root.ownChildrenOfType(TYPE_OBJECT)) { + const key = node.resolutionKey(); + const pkg = packageOfResolutionKey(key); + if (!packages.has(pkg) || nodes.has(key)) continue; + throw new ParseError( + `"${key}" is declared here, but the package "${pkg}" belongs to a metadata dependency ` + + `this project imports — and "${key}" is not one of the nodes that dependency exports. ` + + `A node in an imported package would be excluded from this project's own codegen and ` + + `schema with no output and no error, so it is refused instead.`, + { + code: "ERR_DEPENDENCY_PACKAGE_NOT_OWNED", + // The offending node's own provenance — the file and json path the + // message itself cannot name. + source: node.source, + node: { type: node.type, subtype: node.subType, name: node.name, fqn: key }, + suggestions: [ + `Declare it in a package this project owns, and 'extends' the dependency's node if it needs its shape.`, + `If it was meant to AMEND the dependency's node, give it that node's name and 'overlay: true'.`, + `If this project really does own "${pkg}", name that package in 'scope.include' and stop importing it.`, + ], + }, + ); + } +} diff --git a/server/typescript/packages/sdk/src/sources.ts b/server/typescript/packages/sdk/src/sources.ts index 5f301d85a..1a2f9513b 100644 --- a/server/typescript/packages/sdk/src/sources.ts +++ b/server/typescript/packages/sdk/src/sources.ts @@ -36,8 +36,18 @@ export type SourceSpec = export interface ResolvedSource { /** Absolute path of one metadata file. */ readonly file: string; - /** The spec that contributed it — provenance for diagnostics. */ + /** The spec that contributed it — provenance for diagnostics. For a + * dependency's snapshot artifact (below) this is the artifact's own path: + * nobody DECLARED it as a source, so `dependency` is what identifies it. */ readonly spec: SourceSpec; + /** FR-023 — set only on a dependency's committed snapshot artifact: the + * dependency NAME it was resolved from. Absent on every file a declared + * `sources` spec contributed, which is what tells the two apart. */ + readonly dependency?: string; + /** FR-023 — the `FileSource` id this file loads under (`dep:/`), + * set alongside `dependency`. Absent for own files, which keep the default + * `basename(path)`. */ + readonly id?: string; } /** Used when `sources` is absent or empty in `.metaobjects/config.json`. A diff --git a/server/typescript/packages/sdk/test/collection.test.ts b/server/typescript/packages/sdk/test/collection.test.ts index 93427d0c4..8061786a8 100644 --- a/server/typescript/packages/sdk/test/collection.test.ts +++ b/server/typescript/packages/sdk/test/collection.test.ts @@ -210,3 +210,41 @@ describe("resolveCollection — pointed at a metadata directory (#344)", () => { expect(msg).toContain('{ "path"'); }); }); + +// FR-023 Task 8 — the guarantee the whole dependency feature is built around: a +// project that declares NO dependencies must resolve byte-identically to the way +// it did before dependencies existed. The composed predicates (DESIGN §11.1 item +// 2) collapse to their pre-FR-023 selves when there is nothing imported, and +// `inMigrateScope` in particular stays UNDEFINED — migrate-ts reads that undefined +// as "govern everything loaded", so producing a predicate here would quietly +// change what every existing project's `meta migrate` compares. +describe("resolveCollection — a project with no dependencies (FR-023)", () => { + test("inMigrateScope stays UNDEFINED when no migrate.scope is declared", async () => { + write("metaobjects/meta.a.json", "{}"); + config(".", { dependencies: [] }); + const c = await resolveCollection(root); + expect(c.inMigrateScope).toBeUndefined(); + expect(c.migrateScopePatterns).toBeUndefined(); + }); + + test("inScope is the declared scope alone, unchanged", async () => { + write("metaobjects/meta.a.json", "{}"); + config(".", { scope: { include: ["acme::**"], exclude: ["acme::internal::**"] } }); + const c = await resolveCollection(root); + expect(c.inScope("acme::Order")).toBe(true); + expect(c.inScope("acme::internal::Secret")).toBe(false); + expect(c.inScope("other::Order")).toBe(false); + }); + + test("the dependency-derived members are empty, and files are the own files", async () => { + write("metaobjects/meta.a.json", "{}"); + config(".", {}); + const c = await resolveCollection(root); + expect(c.dependencies).toEqual([]); + expect(c.importedPackages).toEqual([]); + expect([...c.importedNodes]).toEqual([]); + expect([...c.fileIds]).toEqual([]); + expect(c.imported("anything::at::All")).toBe(false); + expect(c.files).toEqual(c.ownFiles); + }); +}); diff --git a/server/typescript/packages/sdk/test/dependencies.test.ts b/server/typescript/packages/sdk/test/dependencies.test.ts index df9e83d63..1346e1fbb 100644 --- a/server/typescript/packages/sdk/test/dependencies.test.ts +++ b/server/typescript/packages/sdk/test/dependencies.test.ts @@ -10,6 +10,7 @@ import { join, resolve } from "node:path"; import { afterEach, beforeEach, expect, test } from "bun:test"; import { dependencySourceId, + explicitlyIncludes, LOCK_FILE, LockSchema, ManifestSchema, @@ -170,3 +171,27 @@ test("writeLock: exact on-disk bytes — sorted keys, 2-space indent, trailing n expect(reloaded).toEqual(lock); expect(reloaded && Object.keys(reloaded.dependencies)).toEqual(["a-dep", "b-dep"]); }); + +// FR-023 Task 8 — `explicitlyIncludes` is the "names a package LITERALLY" half of +// the default-exclusion rule (DESIGN §11.1 item 2): a dependency's package is +// admitted to a surface only when the consumer's own pattern list names that +// package outright. Pure and exported because two predicates compose it +// (`inScope` over `scope.include`, `inMigrateScope` over `migrate.scope`) and a +// wildcard must NOT be able to opt a consumer into generating someone else's +// model by accident. +test("explicitlyIncludes: a pattern names a package only when its literal segments ARE that package", () => { + // Drop the final (name) segment; what remains must be wildcard-free and join + // to exactly the package. + expect(explicitlyIncludes(["acme::common::**"], "acme::common")).toBe(true); + expect(explicitlyIncludes(["acme::common::Address"], "acme::common")).toBe(true); + // A wildcard standing where a package segment would be names nothing: `acme::**` + // reaches acme::common's nodes but never NAMES acme::common. + expect(explicitlyIncludes(["acme::**"], "acme::common")).toBe(false); + expect(explicitlyIncludes(["**"], "acme::common")).toBe(false); + // No list at all, and an empty list, admit nothing — an undeclared scope means + // "everything the project OWNS", never "everything it imported too". + expect(explicitlyIncludes(undefined, "acme::common")).toBe(false); + expect(explicitlyIncludes([], "acme::common")).toBe(false); + // A DEEPER package is a different package, not a match. + expect(explicitlyIncludes(["acme::common::sub::**"], "acme::common")).toBe(false); +}); diff --git a/server/typescript/packages/sdk/test/dependency-conformance.test.ts b/server/typescript/packages/sdk/test/dependency-conformance.test.ts index f4e8ce83d..774234a27 100644 --- a/server/typescript/packages/sdk/test/dependency-conformance.test.ts +++ b/server/typescript/packages/sdk/test/dependency-conformance.test.ts @@ -2,10 +2,10 @@ // implementation. Every port ships an equivalent runner reading this same file // (fixtures/dependency-conformance/, see its README for the case schema). // -// This runner is written AHEAD of the implementation (TDD): `collection.imported` -// does not exist on `Collection` yet (a later task adds the exclusion-key -// composition, DESIGN §11.1 item 2), so any case carrying `expectImported` / -// `expectSelected` / `expectMigrateGoverned` is expected to fail until then. +// The predicates under test are the COMPOSED ones (DESIGN §11.1 item 2): +// `collection.imported` keys on the lock's `packages`, and `inScope` / +// `inMigrateScope` exclude what a dependency owns unless the project's own scope +// NAMES that package. import { describe, expect, test } from "bun:test"; import { copyFile, mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; @@ -16,31 +16,23 @@ import { LOCK_FILE } from "../src/dependencies.js"; import { loadMemory, type LoadMemoryOptions } from "../src/memory.js"; /** - * `imported` lands on `Collection` in a later task (DESIGN §11.1 item 2's - * exclusion-key composition) — this narrow extension lets the corpus runner - * reference it ahead of the implementation without an `any` escape hatch. - * Until that task, the runtime object has no such member, so the cast below - * compiles clean but the call throws (`collection.imported is not a - * function`) — exactly the failure DESIGN decision #3 (Task 6) expects for - * the one case that exercises it. + * Everything a resolved collection contributes to a LOAD — the file list, the + * `dep:/` source ids, and the two imported sets the ownership + * refusal reads. + * + * All four, at every load this runner performs, because that is what the ported + * runners and the CLI's own `collectionLoadOptions` do: a runner that passed + * `files` alone would load a corpus case differently from the way production + * loads the same project, and the `expectLoadError` arm would see no refusal to + * assert. */ -interface CollectionWithImported extends Collection { - readonly imported: (fqn: string) => boolean; -} - -/** - * `fileIds` (path -> `FileSource` id, carried on `Collection` and threaded - * through to `loadMemory`) lands ahead of Task 6 in the FR-023 sequence, same - * ahead-of-implementation situation as `imported` above. Optional here - * (unlike `imported`) because this corpus's `expectLoadError` arm only wants - * provenance to flow through WHEN it exists — it does not need the absence to - * throw. - */ -interface CollectionWithFileIds extends Collection { - readonly fileIds?: ReadonlyMap | undefined; -} -interface LoadMemoryOptionsWithFileIds extends LoadMemoryOptions { - readonly fileIds?: ReadonlyMap | undefined; +function loadOptions(collection: Collection): LoadMemoryOptions { + return { + files: collection.files, + fileIds: collection.fileIds, + importedPackages: collection.importedPackages, + importedNodes: collection.importedNodes, + }; } interface Case { @@ -135,12 +127,11 @@ describe("dependency conformance", () => { c.expectSelected !== undefined || c.expectMigrateGoverned !== undefined ) { - const loaded = await loadMemory(resolveDir, { files: collection.files }); + const loaded = await loadMemory(resolveDir, loadOptions(collection)); const topLevel = loaded.childrenOfType(TYPE_OBJECT).map((n) => n.resolutionKey()); if (c.expectImported !== undefined) { - const withImported = collection as CollectionWithImported; - const imported = topLevel.filter((fqn) => withImported.imported(fqn)).sort(); + const imported = topLevel.filter((fqn) => collection.imported(fqn)).sort(); expect(imported).toEqual([...c.expectImported].sort()); } if (c.expectSelected !== undefined) { @@ -156,11 +147,7 @@ describe("dependency conformance", () => { } if (c.expectLoadError !== undefined) { - const options: LoadMemoryOptionsWithFileIds = { - files: collection.files, - fileIds: (collection as CollectionWithFileIds).fileIds, - }; - const attempt = loadMemory(resolveDir, options); + const attempt = loadMemory(resolveDir, loadOptions(collection)); await expect(attempt).rejects.toMatchObject({ code: c.expectLoadError }); if (c.expectErrorFiles !== undefined) { let thrown: unknown; diff --git a/server/typescript/packages/sdk/test/memory.test.ts b/server/typescript/packages/sdk/test/memory.test.ts index e1dae4c05..266aee314 100644 --- a/server/typescript/packages/sdk/test/memory.test.ts +++ b/server/typescript/packages/sdk/test/memory.test.ts @@ -1,8 +1,9 @@ import { describe, test, expect } from "bun:test"; -import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; +import { mkdtempSync, mkdirSync, readFileSync, writeFileSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; -import { join } from "node:path"; +import { join, resolve } from "node:path"; import { loadMemory } from "../src/memory.js"; +import { sha256Integrity } from "../src/dependencies.js"; import { forgeTypesProvider } from "../src/forge-types.js"; import { rejectedCode } from "./support/error-code.js"; @@ -438,3 +439,135 @@ describe("loadMemory with an explicit file set", () => { } }); }); + +// FR-023 Task 8 — `fileIds` is how a dependency's artifact keeps its identity +// through the load: the snapshot is a file on the consumer's disk, so its default +// source id would be the artifact's basename and every diagnostic about it would +// read like a local file. The collection maps its path to `dep:/` +// and `loadMemory` builds the FileSource with that id, so the provenance stamped +// on every node the artifact contributed (ADR-0009) names the DEPENDENCY. +describe("loadMemory with fileIds (FR-023)", () => { + const nodeSourceFile = (node: { source: unknown }): string | undefined => { + const src = node.source; + if (typeof src !== "object" || src === null || !("files" in src)) return undefined; + const files = (src as { files: readonly string[] }).files; + return files[0]; + }; + + test("a mapped path loads under that id; an unmapped one keeps the basename", async () => { + const dir = mkdtempSync(join(tmpdir(), "metaobjects-memory-fileids-")); + try { + mkdirSync(join(dir, "model"), { recursive: true }); + const file = join(dir, "model/acme-common.metaobjects.json"); + writeFileSync(file, JSON.stringify({ + "metadata.root": { children: [ + { "object.entity": { name: "Customer", package: "acme::common", children: [ + { "field.string": { name: "email" } }] } }] }, + }), "utf8"); + + const id = "dep:acme-common/acme-common.metaobjects.json"; + const withIds = await loadMemory(dir, { files: [file], fileIds: new Map([[file, id]]) }); + const customer = withIds.children().find((c) => c.name === "Customer"); + expect(customer).toBeDefined(); + expect(nodeSourceFile(customer!)).toBe(id); + + // Same file, no map — the default basename, exactly as before FR-023. + const plain = await loadMemory(dir, { files: [file] }); + expect(nodeSourceFile(plain.children().find((c) => c.name === "Customer")!)).toBe( + "acme-common.metaobjects.json", + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); + +// FR-023 — `loadMemory` has TWO arms, and the ownership refusal has to fire on +// BOTH. The routed CLI commands take the first (they hold a collection already +// and pass it through `collectionLoadOptions`); an embedder that calls +// `loadMemory(repoRoot)` with no `files` takes the second, where this function +// resolves the collection itself. Wired into the first arm only, the refusal +// would be silently absent for every such embedder — and the shared corpus +// cannot catch that, because its runner always passes a file list. +describe("loadMemory — the self-resolving arm reads dependencies too (FR-023)", () => { + const ARTIFACTS = resolve(import.meta.dir, "../../../../../fixtures/dependency-conformance/artifacts"); + const ARTIFACT = "acme-common.metaobjects.json"; + + /** A consumer of `acme-common`, optionally with a local file of its own. */ + function consumer(local?: { readonly name: string; readonly body: string }): string { + const dir = mkdtempSync(join(tmpdir(), "metaobjects-memory-deps-")); + const text = readFileSync(join(ARTIFACTS, "acme-common-v1.json"), "utf8"); + mkdirSync(join(dir, ".metaobjects/deps/acme-common"), { recursive: true }); + writeFileSync(join(dir, ".metaobjects/deps/acme-common", ARTIFACT), text, "utf8"); + mkdirSync(join(dir, "metaobjects"), { recursive: true }); + writeFileSync(join(dir, ".metaobjects/config.json"), JSON.stringify({ + schema_version: 1, + sources: [], + dependencies: [{ name: "acme-common", path: "../acme-common/metaobjects" }], + }), "utf8"); + writeFileSync(join(dir, ".metaobjects/deps.lock.json"), JSON.stringify({ + schema_version: 1, + dependencies: { "acme-common": { + version: "1.0.0", + metamodelVersion: "1.0", + resolvedFrom: { path: "../acme-common/metaobjects" }, + artifact: ARTIFACT, + integrity: sha256Integrity(text), + packages: ["acme::common"], + nodes: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"], + } }, + }), "utf8"); + if (local !== undefined) writeFileSync(join(dir, "metaobjects", local.name), local.body, "utf8"); + return dir; + } + + test("the artifact loads under its dep: id with no `files` option", async () => { + const dir = consumer({ name: "meta.app.json", body: JSON.stringify({ + "metadata.root": { package: "app", children: [ + { "object.entity": { name: "Order", children: [{ "field.string": { name: "id" } }] } }] }, + }) }); + try { + const root = await loadMemory(dir); + const customer = root.children().find((c) => c.name === "Customer"); + expect(customer).toBeDefined(); + const src = customer!.source; + expect("files" in src ? (src as { files: readonly string[] }).files[0] : undefined).toBe( + `dep:acme-common/${ARTIFACT}`, + ); + // The project's own file still loads, and keeps its own basename. + expect(root.children().map((c) => c.name)).toContain("Order"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test("a local node in the dependency's package is refused on this arm too", async () => { + const dir = consumer({ name: "meta.ext.json", body: JSON.stringify({ + "metadata.root": { package: "acme::common", children: [ + { "object.value": { name: "Note", children: [{ "field.string": { name: "text" } }] } }] }, + }) }); + try { + expect(await rejectedCode(loadMemory(dir))).toBe("ERR_DEPENDENCY_PACKAGE_NOT_OWNED"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test("an OVERLAY of the dependency's own node is not refused", async () => { + // The distinction the refusal turns on: this node's resolution key IS in the + // lock's `nodes`, so it merged into the imported Customer rather than + // declaring a new one. + const dir = consumer({ name: "meta.ov.json", body: JSON.stringify({ + "metadata.root": { package: "acme::common", children: [ + { "object.entity": { name: "Customer", overlay: true, children: [ + { "field.string": { name: "nickname" } }] } }] }, + }) }); + try { + const root = await loadMemory(dir); + const customer = root.children().find((c) => c.name === "Customer"); + expect(customer?.children().map((c) => c.name)).toContain("nickname"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/server/typescript/packages/sdk/test/order-independence.test.ts b/server/typescript/packages/sdk/test/order-independence.test.ts index 9d4b149e0..f563dec5f 100644 --- a/server/typescript/packages/sdk/test/order-independence.test.ts +++ b/server/typescript/packages/sdk/test/order-independence.test.ts @@ -45,10 +45,12 @@ // the amended test inventing a bar the design never set, not a real // defect. import { describe, test, expect, beforeEach, afterEach } from "bun:test"; -import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs"; +import { mkdtempSync, readFileSync, rmSync, mkdirSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; -import { join } from "node:path"; +import { join, resolve } from "node:path"; import { resolveSources, type SourceSpec } from "../src/sources.js"; +import { resolveCollection } from "../src/collection.js"; +import { sha256Integrity } from "../src/dependencies.js"; let root: string; const write = (rel: string, body: object) => { @@ -190,3 +192,82 @@ describe("order independence", () => { } }); }); + +// FR-023 — the same premise, one rung out: a project's DEPENDENCIES are a +// declared SET too, so permuting the `dependencies` array in +// `.metaobjects/config.json` must not change a single thing the resolver +// produces. It matters more here than for `sources`, because the dependency +// artifacts are loaded FIRST and their order is therefore the order the loader +// sees the bases in — if declaration order leaked through, two developers with +// the same lock would generate from differently-ordered trees. +describe("order independence — declared dependencies (FR-023)", () => { + const ARTIFACTS = resolve(import.meta.dir, "../../../../../fixtures/dependency-conformance/artifacts"); + const COMMON = readFileSync(join(ARTIFACTS, "acme-common-v1.json"), "utf8"); + // A second, distinct publisher: the same bytes in another package, so the two + // dependencies export no FQN in common (which would be a collision, not an + // ordering question). + const EXTRA = COMMON.replaceAll("acme::common", "acme::extra"); + + const writeText = (rel: string, body: string): void => { + mkdirSync(join(root, rel, ".."), { recursive: true }); + writeFileSync(join(root, rel), body, "utf8"); + }; + + const lockEntry = (name: string, text: string, pkg: string) => ({ + version: "1.0.0", + metamodelVersion: "1.0", + resolvedFrom: { path: `../${name}/metaobjects` }, + artifact: `${name}.metaobjects.json`, + // Hashed from the very bytes written below — a hand-pinned hash here would + // make this test a snapshot of the corpus rather than a test of ordering. + integrity: sha256Integrity(text), + packages: [pkg], + nodes: [`${pkg}::Address`, `${pkg}::Audited`, `${pkg}::Customer`], + }); + + /** A consumer of both dependencies, declaring them in `order`. */ + function consumer(sub: string, order: readonly string[]): string { + write(`${sub}/metaobjects/meta.app.json`, { + "metadata.root": { package: "app", children: [ + { "object.entity": { name: "Order", children: [{ "field.string": { name: "id" } }] } }] }, + }); + writeText(`${sub}/.metaobjects/deps/acme-common/acme-common.metaobjects.json`, COMMON); + writeText(`${sub}/.metaobjects/deps/acme-extra/acme-extra.metaobjects.json`, EXTRA); + writeText(`${sub}/.metaobjects/config.json`, JSON.stringify({ + schema_version: 1, + sources: [], + dependencies: order.map((name) => ({ name, path: `../${name}/metaobjects` })), + })); + writeText(`${sub}/.metaobjects/deps.lock.json`, JSON.stringify({ + schema_version: 1, + // Lock keys are sorted by contract, so the lock cannot carry the ordering + // under test — only the config's `dependencies` array can. + dependencies: { + "acme-common": lockEntry("acme-common", COMMON, "acme::common"), + "acme-extra": lockEntry("acme-extra", EXTRA, "acme::extra"), + }, + })); + return join(root, sub); + } + + test("permuting the declared dependencies leaves files and fileIds identical", async () => { + const commonFirst = consumer("common-first", ["acme-common", "acme-extra"]); + const extraFirst = consumer("extra-first", ["acme-extra", "acme-common"]); + + const a = await resolveCollection(commonFirst, { explicitDir: commonFirst }); + const b = await resolveCollection(extraFirst, { explicitDir: extraFirst }); + + const rel = (c: { configDir: string }, p: string): string => p.slice(c.configDir.length + 1); + expect(b.files.map((f) => rel(b, f))).toEqual(a.files.map((f) => rel(a, f))); + expect([...b.fileIds].map(([p, id]) => [rel(b, p), id])).toEqual( + [...a.fileIds].map(([p, id]) => [rel(a, p), id]), + ); + // Not a vacuous comparison: both artifacts and the own file are in there, and + // the artifacts lead (the bases load first). + expect(a.files.map((f) => rel(a, f))).toEqual([ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + ".metaobjects/deps/acme-extra/acme-extra.metaobjects.json", + "metaobjects/meta.app.json", + ]); + }); +}); From df129849973dd227a379cba112e0bb2496028948 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 21:07:36 -0400 Subject: [PATCH 18/62] fix(sdk): DependencyManifest ends the barrel's two-Manifest collision; the requirements line's dependency count is conditional (FR-023) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two adjudicated fixes from the Task 8 review. 1. `verify`'s requirements summary appends `, N from dependencies` only when there ARE dependencies. A project that declares none prints the sentence it printed before metadata dependencies existed, ending at `counted over N metadata file(s).` — the byte-identical no-dependency guarantee covers this surface like any other. The covering assertion now includes the trailing period, so it can tell the two sentences apart; without it the test passed just as happily on the ungated output. 2. FR-023's manifest type is `DependencyManifest` / `DependencyManifestSchema`. The package barrel already published an agent-context `Manifest` through an `export *`, and an explicit export shadows a star one, so the bare name silently re-typed every consumer asking the root barrel for the agent-context manifest. FR-023's is the newcomer, and the longer name reads right beside its own siblings (`DependencySpec`, `LockEntry`, `Lock`, `ResolvedDependency`). The previous commit's subpath fix in `agent-context-staleness.ts` stands on its own and is unchanged. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../typescript/packages/cli/src/commands/verify.ts | 10 ++++++++-- .../cli/test/verify-requirements-e2e.test.ts | 8 +++++++- server/typescript/packages/sdk/src/dependencies.ts | 13 ++++++++++--- server/typescript/packages/sdk/src/index.ts | 6 ++++-- .../packages/sdk/test/dependencies.test.ts | 8 ++++---- 5 files changed, 33 insertions(+), 12 deletions(-) diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index 893447278..ec9c70361 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -618,8 +618,14 @@ export async function verifyCommand( `meta verify — requirements: ${s.total} entries (${s.functional} functional, ` + `${s.architectural} architectural) — ${parts.join(", ")}; ` + `${s.entitiesClaimed}/${s.entitiesTotal} entities claimed, ` + - `counted over ${collection.files.length} metadata file(s), ` + - `${collection.dependencies.length} from dependencies.`, + `counted over ${collection.files.length} metadata file(s)` + + // FR-023 — only when there ARE dependencies. A project that declares none + // must print the sentence it printed before dependencies existed, to the + // byte: "0 from dependencies" is noise on every existing project, and this + // line is a SURFACE the no-dependency guarantee covers like any other. + (collection.dependencies.length > 0 + ? `, ${collection.dependencies.length} from dependencies.` + : `.`), ); if (s.undecided > 0) { say( diff --git a/server/typescript/packages/cli/test/verify-requirements-e2e.test.ts b/server/typescript/packages/cli/test/verify-requirements-e2e.test.ts index 16bb4c840..5ab5e8e1f 100644 --- a/server/typescript/packages/cli/test/verify-requirements-e2e.test.ts +++ b/server/typescript/packages/cli/test/verify-requirements-e2e.test.ts @@ -111,7 +111,13 @@ describe("meta verify — requirements exit-code contract", () => { // an estate reports the covered half as fully claimed. No check can see a tree it // was never pointed at; publishing the count it was taken over is what makes a // wrong denominator noticeable. Two files here: the entities and the requirements. - expect(summary).toContain("counted over 2 metadata file(s)"); + // The trailing PERIOD is part of the assertion (FR-023): a project that + // declares no dependencies prints the sentence it printed before metadata + // dependencies existed, ending here. Without the period this passes just as + // happily on "… file(s), 0 from dependencies.", which is the one thing the + // no-dependency guarantee forbids on this surface. The dependency count is + // appended only when there IS one (asserted where that case is built). + expect(summary).toContain("counted over 2 metadata file(s)."); }, TIMEOUT_MS); test("a model with NO requirements exits 0 — opt-in by declaration", async () => { diff --git a/server/typescript/packages/sdk/src/dependencies.ts b/server/typescript/packages/sdk/src/dependencies.ts index ccae3d920..a559e0633 100644 --- a/server/typescript/packages/sdk/src/dependencies.ts +++ b/server/typescript/packages/sdk/src/dependencies.ts @@ -103,7 +103,7 @@ function isSortedAscending(values: readonly string[]): boolean { /** * A non-empty string array that must already be in ascending order — the - * shared shape of `packages` and `nodes` on both {@link ManifestSchema} and + * shared shape of `packages` and `nodes` on both {@link DependencyManifestSchema} and * {@link LockEntrySchema}. Sortedness is validated, never imposed here: the * writer (the publisher's `sharedModelFile()` generator, and `meta deps * sync` copying the manifest into the lock) is responsible for producing @@ -118,11 +118,18 @@ function sortedStringArray(fieldName: string) { /** * `metaobjects.pkg.json` (DESIGN §3.2) — generated by the publisher's * `sharedModelFile()` and read by the consumer's `meta deps sync`. + * + * NOT named `Manifest`/`DependencyManifestSchema`, deliberately: the package barrel + * already publishes an agent-context `Manifest` (`scaffold.ts`) through an + * `export *`, and an explicit export SHADOWS a star one — so the bare name + * here silently re-typed every consumer asking the root barrel for the + * agent-context manifest. `DependencyManifest` also reads right beside its own + * siblings (`DependencySpec`, `LockEntry`, `Lock`, `ResolvedDependency`). * `.strict()`: an unknown key — in particular a resurrected `mode` (FR-023 * §11.3, removed by Task 6 and not reintroduced) — is a hard parse error * rather than one zod silently strips. */ -export const ManifestSchema = z +export const DependencyManifestSchema = z .object({ schema_version: z.literal(1), name: DependencyName, @@ -149,7 +156,7 @@ export const ManifestSchema = z }) .strict(); -export type Manifest = z.infer; +export type DependencyManifest = z.infer; /** * The one transport a lock entry's `resolvedFrom` carries — the declared diff --git a/server/typescript/packages/sdk/src/index.ts b/server/typescript/packages/sdk/src/index.ts index 43707165d..20b4306b0 100644 --- a/server/typescript/packages/sdk/src/index.ts +++ b/server/typescript/packages/sdk/src/index.ts @@ -33,7 +33,7 @@ export { DependencySpecSchema, dependencyName, IntegritySchema, - ManifestSchema, + DependencyManifestSchema, LockEntrySchema, LockSchema, sha256Integrity, @@ -45,7 +45,9 @@ export { importedNodesOf, explicitlyIncludes, } from "./dependencies.js"; -export type { DependencySpec, Manifest, LockEntry, Lock, ResolvedDependency } from "./dependencies.js"; +// `DependencyManifest`, not `Manifest`: the agent-context re-export below already +// publishes that name, and an explicit export shadows a star one. +export type { DependencySpec, DependencyManifest, LockEntry, Lock, ResolvedDependency } from "./dependencies.js"; // Meta Forge metadata types + attribute name constants (registered into a // TypeRegistry to let Loader parse decision/principle/etc. children + the diff --git a/server/typescript/packages/sdk/test/dependencies.test.ts b/server/typescript/packages/sdk/test/dependencies.test.ts index 1346e1fbb..8a84fad0e 100644 --- a/server/typescript/packages/sdk/test/dependencies.test.ts +++ b/server/typescript/packages/sdk/test/dependencies.test.ts @@ -13,7 +13,7 @@ import { explicitlyIncludes, LOCK_FILE, LockSchema, - ManifestSchema, + DependencyManifestSchema, readLock, sha256Integrity, writeLock, @@ -40,9 +40,9 @@ test("manifest and lock schemas: sorted arrays, sorted keys, one transport", () packages: ["acme::common"], nodes: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"], }; - expect(() => ManifestSchema.parse(manifest)).not.toThrow(); - expect(() => ManifestSchema.parse({ ...manifest, nodes: ["b", "a"] })).toThrow(/sorted/); - expect(() => ManifestSchema.parse({ ...manifest, mode: "reference" })).toThrow(); + expect(() => DependencyManifestSchema.parse(manifest)).not.toThrow(); + expect(() => DependencyManifestSchema.parse({ ...manifest, nodes: ["b", "a"] })).toThrow(/sorted/); + expect(() => DependencyManifestSchema.parse({ ...manifest, mode: "reference" })).toThrow(); const { schema_version: _s, name: _n, ...rest } = manifest; const entry = { ...rest, resolvedFrom: { path: "x" } }; From dc1ada81ce843408076022e29808e1699cc4e7e4 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 21:09:35 -0400 Subject: [PATCH 19/62] docs(fr-023): the refusal names object and package; DependencyManifest in the docs Two controller rulings from Task 8's concerns, synced into the documents that later tasks read. - The ownership refusal names the object and the package, not the dependency. The load-time option shape carries flat package and node sets with no package-to-dependency-name map, and the package is what a user acts on since scope.include takes a package. The design claimed otherwise; the code was right and the design was wrong. - The dependency manifest type is DependencyManifest / DependencyManifestSchema after the rename that ended the sdk barrel's two-Manifest collision. The plan and the design still named the old symbol, and the design is a living spec a later task reads. Prose naming the manifest FILE is deliberately unchanged, as is the unrelated readManifestDir. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 10 +++++----- .../2026-09-11-fr-023-metadata-dependencies-design.md | 9 ++++++--- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 5d187c952..b338cbf07 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -671,14 +671,14 @@ refactor(deps): retire `mode`, the local override and three superseded error cod **Read first:** DESIGN §3.2, §3.3 (minus `mode`), §11.2 row "`.metaobjects/config.json`". The pinned hashes in the corpus README. `sdk/src/config.ts` for the `.strict()`-everywhere convention. **Interfaces:** -- Produces TS: `IntegritySchema` (`/^sha256-[0-9a-f]{64}$/`); `ManifestSchema` (`schema_version: z.literal(1)`, `name: DependencyName`, `version: z.string().min(1)`, `metamodelVersion: z.string().regex(/^\d+\.\d+$/)`, `artifact: z.string().endsWith(ARTIFACT_SUFFIX)`, `integrity`, `packages` / `nodes` as `z.array(z.string().min(1)).refine(sorted)`); `LockEntrySchema` = the manifest fields minus `schema_version`/`name` plus `resolvedFrom` (a strict union `{ path } | { npm, dir? } | { python, dir? }`); `LockSchema = { schema_version: 1, dependencies: z.record(DependencyName, LockEntrySchema) }.refine(keys sorted, { message: "deps.lock.json: dependency keys must be sorted" })`; `type Manifest`, `type Lock`, `type LockEntry`; `type ResolvedDependency = { name; version; packages: readonly string[]; nodes: readonly string[]; artifactPath: string; sourceId: string }`; `sha256Integrity(bytes: Uint8Array | string): string`; `readLock(configDir): Promise`; `writeLock(configDir, lock): Promise` (sorted keys, 2-space indent, trailing newline); `dependencySourceId(name, artifact)` = `` `${DEPENDENCY_SOURCE_ID_PREFIX}${name}/${artifact}` ``. +- Produces TS: `IntegritySchema` (`/^sha256-[0-9a-f]{64}$/`); `DependencyManifestSchema` (`schema_version: z.literal(1)`, `name: DependencyName`, `version: z.string().min(1)`, `metamodelVersion: z.string().regex(/^\d+\.\d+$/)`, `artifact: z.string().endsWith(ARTIFACT_SUFFIX)`, `integrity`, `packages` / `nodes` as `z.array(z.string().min(1)).refine(sorted)`); `LockEntrySchema` = the manifest fields minus `schema_version`/`name` plus `resolvedFrom` (a strict union `{ path } | { npm, dir? } | { python, dir? }`); `LockSchema = { schema_version: 1, dependencies: z.record(DependencyName, LockEntrySchema) }.refine(keys sorted, { message: "deps.lock.json: dependency keys must be sorted" })`; `type DependencyManifest`, `type Lock`, `type LockEntry`; `type ResolvedDependency = { name; version; packages: readonly string[]; nodes: readonly string[]; artifactPath: string; sourceId: string }`; `sha256Integrity(bytes: Uint8Array | string): string`; `readLock(configDir): Promise`; `writeLock(configDir, lock): Promise` (sorted keys, 2-space indent, trailing newline); `dependencySourceId(name, artifact)` = `` `${DEPENDENCY_SOURCE_ID_PREFIX}${name}/${artifact}` ``. - Produces Python: `validate_manifest(obj) -> dict`, `validate_lock(obj) -> dict`, `sha256_integrity(data: bytes) -> str`, `read_lock(config_dir: Path) -> dict | None`, `dependency_source_id(name, artifact)`; `ResolvedDependency` dataclass (same fields). Hand-validated dicts with the same rules; a violation raises `ParseError(code=ErrorCode.ERR_DEPENDENCY_MANIFEST_INVALID)` for manifest shape and `ERR_DEPENDENCY_SNAPSHOT_STALE` for lock shape. - [ ] **Step 1: Failing TS tests** (`sdk/test/dependencies.test.ts`): ```ts import { readFile } from "node:fs/promises"; import { resolve } from "node:path"; -import { LockSchema, ManifestSchema, sha256Integrity } from "../src/dependencies.js"; +import { LockSchema, DependencyManifestSchema, sha256Integrity } from "../src/dependencies.js"; const CORPUS = resolve(import.meta.dir, "../../../../../fixtures/dependency-conformance/artifacts"); test("sha256Integrity of the pinned v1 artifact equals the README value", async () => { expect(sha256Integrity(await readFile(`${CORPUS}/acme-common-v1.json`))).toBe("sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d"); @@ -686,9 +686,9 @@ test("sha256Integrity of the pinned v1 artifact equals the README value", async test("manifest and lock schemas: sorted arrays, sorted keys, one transport", () => { const manifest = { schema_version: 1, name: "acme-common", version: "1.0.0", metamodelVersion: "1.0", artifact: "acme-common.metaobjects.json", integrity: "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", packages: ["acme::common"], nodes: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"] }; - expect(() => ManifestSchema.parse(manifest)).not.toThrow(); - expect(() => ManifestSchema.parse({ ...manifest, nodes: ["b", "a"] })).toThrow(/sorted/); - expect(() => ManifestSchema.parse({ ...manifest, mode: "reference" })).toThrow(); + expect(() => DependencyManifestSchema.parse(manifest)).not.toThrow(); + expect(() => DependencyManifestSchema.parse({ ...manifest, nodes: ["b", "a"] })).toThrow(/sorted/); + expect(() => DependencyManifestSchema.parse({ ...manifest, mode: "reference" })).toThrow(); const { schema_version: _s, name: _n, ...rest } = manifest; const entry = { ...rest, resolvedFrom: { path: "x" } }; expect(() => LockSchema.parse({ schema_version: 1, dependencies: { "acme-common": entry } })).not.toThrow(); diff --git a/docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md b/docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md index 7bb0ef7b1..c96b1a4c6 100644 --- a/docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md +++ b/docs/superpowers/specs/2026-09-11-fr-023-metadata-dependencies-design.md @@ -1199,8 +1199,11 @@ The plan (`docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md`) is re-cut to m - **A consumer may not declare a new top-level node into a dependency's package.** Post load, a top-level object whose `packageOf(fqn)` is a dependency package and whose FQN is NOT in that dependency's `nodes` was declared locally into a package the consumer - does not own: `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, naming the object, the package and - the dependency, with the fix ("declare it in your own package and `extends` the + does not own: `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, naming the object and the package (NOT the + dependency — ruled 2026-09-11: the load-time option shape carries flat + `importedPackages` / `importedNodes` sets with no package-to-dependency-name map, + and the package is what the user acts on, since `scope.include` takes a package), + with the fix ("declare it in your own package and `extends` the dependency's node, or overlay the dependency's node with `overlay: true`"). An OVERLAY merges into the existing node, whose FQN *is* in `nodes`, so it passes untouched — only a genuinely new object is refused. This is what keeps a @@ -1278,7 +1281,7 @@ The plan (`docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md`) is re-cut to m |---|---| | `.metaobjects/config.json` | `dependencies: [{ name, path }]`; `npm` / `python` arms stay in the schema and are refused by `deps sync` (`ERR_DEPENDENCY_UNRESOLVED`, "not supported by this toolchain yet") — the `sources` `resource`/`package` precedent | | `sdk` `Collection` | `dependencies`, `ownFiles`, `fileIds`, `importedPackages`, `importedNodes`, `imported(fqn)` (package-keyed); `inScope` and `inMigrateScope` become the composed predicates; `declaredMigrateScope` is the user's alone | -| `metadata` | `LockSchema`/`ManifestSchema` live in sdk; `declaredTopLevelKeys(source)` exported from the loader for the overlay lint | +| `metadata` | `LockSchema`/`DependencyManifestSchema` live in sdk; `declaredTopLevelKeys(source)` exported from the loader for the overlay lint | | `codegen-ts` | `sharedModelFile()`; `GenContext.registry` / `sourceFiles`; `renderSharedEnumsFile(root, { select })` | | `migrate-ts` | `scopeExpectedSchema(built, inScope, { imported })` — the `declaredSchemas` correction | | `cli` | `meta deps sync \| check \| list`; `verify --deps`; the overlay lint; `gen` refusal by name; `init` no longer scaffolds `package.meta.json` | From 605935e4fdfd47a18bf4f26378e11488fcf1885d Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 21:21:27 -0400 Subject: [PATCH 20/62] test(deps): the loader's existing errors are the first cross-repo drift gate (FR-023) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Appends 7 load-time failure cases to fixtures/dependency-conformance/cases.json (DESIGN §2.5's table, plus the sibling whole-object-overlay case and the malformed-artifact case from §11.1 item 4) — no source change. Every one demonstrates that when an upstream dependency's snapshot changes underneath a consumer, an existing loader error (ERR_OVERLAY_NO_TARGET, ERR_UNRESOLVED_SUPER, ERR_UNRESOLVED_OBJECT_REF, ERR_EXTENDS_TARGET_MISMATCH, ERR_MERGE_CONFLICT, ERR_MALFORMED_JSON) already fires — proven by running each case, not merely asserted. README documents the mapping table and the confirmed load-order guarantee (loader errors before the package-ownership refusal) the whole-object-overlay case depends on. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- fixtures/dependency-conformance/README.md | 37 +++ fixtures/dependency-conformance/cases.json | 317 +++++++++++++++++++++ 2 files changed, 354 insertions(+) diff --git a/fixtures/dependency-conformance/README.md b/fixtures/dependency-conformance/README.md index 38f34178b..f75fdc95a 100644 --- a/fixtures/dependency-conformance/README.md +++ b/fixtures/dependency-conformance/README.md @@ -107,6 +107,43 @@ If a hash differs, fix the bytes (whitespace, trailing newline, CRLF) — never pinned values; a hash change here is a change to what every port's `sync`/manifest test asserts against. +## Load-time failure cases — the loader's existing errors are the first drift gate + +DESIGN §2.5 says that once `meta deps sync` replaces a stale snapshot, a consumer construct +that pointed at something upstream removed or changed fails through the loader's *existing* +errors — no new machinery. The seven cases below exercise that table end-to-end: the +dependency's artifact loads first, the consumer's own file loads second and fails exactly +where §2.5 predicts. + +| Upstream change / consumer construct | Case | Error (existing) | +|---|---|---| +| a node removed; local `overlay: true` targets it (field-level) | `an-overlay-whose-target-was-removed-fails` | `ERR_OVERLAY_NO_TARGET` | +| a whole object the artifact never exports; local `overlay: true` targets it | `an-overlay-of-a-whole-object-absent-from-the-artifact-fails` | `ERR_OVERLAY_NO_TARGET` | +| a node removed/never existed; local `extends` targets it | `an-extends-whose-target-was-removed-fails` | `ERR_UNRESOLVED_SUPER` | +| a node removed/never existed; local `field.object @objectRef` targets it | `a-reference-whose-target-was-removed-fails` | `ERR_UNRESOLVED_OBJECT_REF` | +| a member's subtype changed (v1 `Customer.email` is `field.string`); local dotted `extends: Customer.email` from a `field.int` | `a-dotted-extends-whose-member-changed-subtype-fails` | `ERR_EXTENDS_TARGET_MISMATCH` | +| an attr the consumer's overlay also sets is now set differently upstream (base `@maxLength: 120`, overlay `@maxLength: 80`) | `an-overlay-attr-the-base-now-sets-differently-conflicts` | `ERR_MERGE_CONFLICT` | +| the dependency's own artifact file fails to parse | `a-dependency-file-error-names-the-dependency` | `ERR_MALFORMED_JSON`, naming `dep:acme-common/acme-common.metaobjects.json` | + +All seven pass with **no source change** (`sdk/test/dependency-conformance.test.ts`). +`an-overlay-whose-target-was-removed-fails` is the one case that rests on a genuine +upstream removal rather than "never existed" — it loads +`acme-common-v2-email-removed.json` (v1's `Customer` minus `email`) as the dependency +snapshot and pins that artifact's hash (`sha256-c4ba…ce7a`, printed above) in the case's +`lock`; every other case here loads the unmodified `acme-common-v1.json`. + +**Ordering note (verified, not merely asserted).** +`an-overlay-of-a-whole-object-absent-from-the-artifact-fails` overlays a whole top-level +object (`Invoice`) into `acme::common` — a package the dependency owns but does not +export `Invoice` from. That input also matches the post-load ownership refusal +`ERR_DEPENDENCY_PACKAGE_NOT_OWNED` (§11.1 item 2, "a consumer may not declare a new +top-level node into a dependency's package"). `sdk/src/memory.ts`'s `loadMemory` throws +the loader's own errors (`result.errors`) BEFORE it runs the ownership walk +(`refuseUnownedPackages`) — confirmed by running this exact case, not merely read off the +source — so `ERR_OVERLAY_NO_TARGET` wins. If this case ever reports +`ERR_DEPENDENCY_PACKAGE_NOT_OWNED` instead, that is an ordering regression to fix, not a +corpus expectation to update. + ## Which arms each port runs - **TypeScript** — resolution, load-time failure, lock/snapshot integrity. diff --git a/fixtures/dependency-conformance/cases.json b/fixtures/dependency-conformance/cases.json index 31de23905..ee4e23442 100644 --- a/fixtures/dependency-conformance/cases.json +++ b/fixtures/dependency-conformance/cases.json @@ -734,6 +734,323 @@ "app/.metaobjects/deps/acme-common/acme-common.metaobjects.json", "app/metaobjects/meta.app.json" ] + }, + { + "name": "an-overlay-whose-target-was-removed-fails", + "tree": { + "metaobjects/meta.ov.json": "{\"metadata.root\":{\"package\":\"acme::common\",\"children\":[{\"object.entity\":{\"name\":\"Customer\",\"overlay\":true,\"children\":[{\"field.string\":{\"name\":\"email\",\"overlay\":true,\"children\":[{\"view.text\":{\"name\":\"emailView\"}}]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v2-email-removed.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-c4ba364bff0071b164b127326c095d6d32915b57781271e0be64a7003c27ce7a", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.ov.json" + ], + "expectLoadError": "ERR_OVERLAY_NO_TARGET" + }, + { + "name": "an-overlay-of-a-whole-object-absent-from-the-artifact-fails", + "tree": { + "metaobjects/meta.ov-invoice.json": "{\"metadata.root\":{\"package\":\"acme::common\",\"children\":[{\"object.entity\":{\"name\":\"Invoice\",\"overlay\":true,\"children\":[{\"field.string\":{\"name\":\"number\"}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.ov-invoice.json" + ], + "expectLoadError": "ERR_OVERLAY_NO_TARGET" + }, + { + "name": "an-extends-whose-target-was-removed-fails", + "tree": { + "metaobjects/meta.vip.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Vip\",\"extends\":\"acme::common::Gone\",\"children\":[{\"source.rdb\":{\"@table\":\"vips\"}},{\"field.long\":{\"name\":\"id\"}},{\"field.string\":{\"name\":\"tier\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.vip.json" + ], + "expectLoadError": "ERR_UNRESOLVED_SUPER" + }, + { + "name": "a-reference-whose-target-was-removed-fails", + "tree": { + "metaobjects/meta.vip2.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Vip\",\"children\":[{\"source.rdb\":{\"@table\":\"vips\"}},{\"field.long\":{\"name\":\"id\"}},{\"field.object\":{\"name\":\"addr\",\"@objectRef\":\"acme::common::Missing\",\"@storage\":\"jsonb\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.vip2.json" + ], + "expectLoadError": "ERR_UNRESOLVED_OBJECT_REF" + }, + { + "name": "a-dotted-extends-whose-member-changed-subtype-fails", + "tree": { + "metaobjects/meta.args.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.value\":{\"name\":\"VipArgs\",\"children\":[{\"field.int\":{\"name\":\"email\",\"extends\":\"acme::common::Customer.email\"}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.args.json" + ], + "expectLoadError": "ERR_EXTENDS_TARGET_MISMATCH" + }, + { + "name": "an-overlay-attr-the-base-now-sets-differently-conflicts", + "tree": { + "metaobjects/meta.ov-conflict.json": "{\"metadata.root\":{\"package\":\"acme::common\",\"children\":[{\"object.entity\":{\"name\":\"Customer\",\"overlay\":true,\"children\":[{\"field.string\":{\"name\":\"email\",\"overlay\":true,\"@maxLength\":80}}]}}]}}" + }, + "treeFiles": { + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "artifacts/acme-common-v1.json" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": [ + "acme::common" + ], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer" + ] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.ov-conflict.json" + ], + "expectLoadError": "ERR_MERGE_CONFLICT" + }, + { + "name": "a-dependency-file-error-names-the-dependency", + "tree": { + "metaobjects/meta.app.json": "{\"metadata.root\":{\"package\":\"app\",\"children\":[{\"object.entity\":{\"name\":\"Order\",\"children\":[{\"source.rdb\":{\"@table\":\"orders\"}},{\"field.long\":{\"name\":\"id\"}},{\"identity.primary\":{\"name\":\"pk\",\"@fields\":[\"id\"]}}]}}]}}", + ".metaobjects/deps/acme-common/acme-common.metaobjects.json": "{\"metadata.root\":" + }, + "config": { + "schema_version": 1, + "sources": [], + "dependencies": [ + { + "name": "acme-common", + "path": "../acme-common/metaobjects" + } + ] + }, + "lock": { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": { + "path": "../acme-common/metaobjects" + }, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-b8f7f146735efb3ed6d5beca1407ea99009d76c6eda670fc5f0b704340f4f102", + "packages": [], + "nodes": [] + } + } + }, + "expectFiles": [ + ".metaobjects/deps/acme-common/acme-common.metaobjects.json", + "metaobjects/meta.app.json" + ], + "expectLoadError": "ERR_MALFORMED_JSON", + "expectErrorFiles": [ + "dep:acme-common/acme-common.metaobjects.json" + ] } ] } From 9887a57696e66fe72dbdb42f42b92b0ce45ef1f3 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 21:22:44 -0400 Subject: [PATCH 21/62] docs(fr-023): the shared-enums predicate must reach both call sites Pre-flight check before the codegen-selection task: the plan's file list named only one of renderSharedEnumsFile's two call sites. The other is the reference template meta init copies eagerly into every scaffolded project, so threading the selection predicate through the library path alone would leave every consumer's own generated copy emitting enums for imported entities that the library path correctly excludes. Both call sites must honour it, and a diff updating only one is an Important finding rather than a nit. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index b338cbf07..8d5f07a1b 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -805,6 +805,7 @@ test(deps): the loader's existing errors are the first cross-repo drift gate (FR - Modify: `server/typescript/packages/cli/src/commands/gen.ts` (the pre-check refusing a positional that names an excluded import) - Modify: `server/typescript/packages/codegen-ts/src/templates/enums-file.ts` (`renderSharedEnumsFile(root, opts?: { select?: (fqn: string) => boolean })`) - Modify: `server/typescript/packages/codegen-ts/src/generators/entity-file.ts` (pass `select` from `ctx`) +- Modify: `server/typescript/packages/codegen-ts/src/reference/entity.ts` (the SECOND `renderSharedEnumsFile(ctx.loadedRoot)` call site, at ~:178 — pass `select` here too). **Added 2026-09-11 by pre-flight check.** This file is a reference template, and `meta init` copies it EAGERLY — verified: it is one of the FIVE generators `init.ts` writes into every scaffolded project's `codegen/generators/` (ADR-0034 scaffold-and-own; `eject.ts` names this exact path for `meta eject`). So this is the DEFAULT path, not an opt-in one: threading `select` only through `generators/entity-file.ts` would leave EVERY `meta init` project emitting shared enums for imported entities that the library path correctly excludes — silent wrong output, the exact failure mode the default-exclusion rule exists to prevent. Both call sites must honour the predicate, and a diff that updates only one of them is an Important finding, not a nit. - Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (`RunGenOpts.scope` — the real exported name is `RunGenOpts`, NOT `RunGenOptions`, verified at `runner.ts:45`; pre-flight ruling 2026-09-11 — is threaded into `GenContext` as `select` for the templates that render whole-model artifacts — read how `entityFile` reaches `ctx.loadedRoot` and add `ctx.select`) - Modify: `server/typescript/packages/codegen-ts/src/generator.ts` (`GenContext.select?: (fqn: string) => boolean`) - Test: `cli/test/gen-imported-nodes.test.ts` (new), `codegen-ts/test/shared-enums-imported.test.ts` (new) From 33d9dd13a560ebef0c734f05aec18507d369cfaa Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 21:24:24 -0400 Subject: [PATCH 22/62] docs(fr-023): the load-failure task adds seven cases, not six MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The task's file list said six while its own step enumerates six bullets whose first carries an embedded sibling case — seven in total. The implementer caught the disagreement and followed the step, which was right; the file line would have misled the next reader the same way it misled me when I summarised it into the dispatch. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 8d5f07a1b..1e064f1e6 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -776,7 +776,7 @@ feat(sdk): resolveCollection loads dependency snapshots first and composes the d ### Task 9: The load-time failure corpus cases (no new machinery) **Files:** -- Modify: `fixtures/dependency-conformance/cases.json` (append the six cases) +- Modify: `fixtures/dependency-conformance/cases.json` (append the SEVEN cases — Step 1 enumerates six bullets, but the first carries an EMBEDDED sibling case, so seven cases are added, not six. Corrected 2026-09-11 after the Task 9 implementer caught the Files line and Step 1 disagreeing.) - Modify: `fixtures/dependency-conformance/README.md` (a table: upstream change → consumer construct → existing error code) - Test: `sdk/test/dependency-conformance.test.ts` From 902e8dc88fc29f1695e4061003035993b550d664 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 21:40:18 -0400 Subject: [PATCH 23/62] =?UTF-8?q?feat(codegen-ts):=20imported=20nodes=20ar?= =?UTF-8?q?e=20not=20generated=20unless=20scope.include=20names=20their=20?= =?UTF-8?q?package=20(FR-023=20=C2=A711)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 10: `meta gen`'s existing scope wiring (Task 8) already narrows per-entity output, but the shared-enums module renders once over the whole loaded root and never runs a generator's per-entity filter — so it needed its own `select` knob threaded through GenContext/RunGenOpts to honour the same default exclusion of imported metadata. Also adds a `meta gen ` pre-check that refuses (exit 2) a positional naming only objects imported from a dependency and excluded from output, distinct from the existing "no entities matched" warning. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../packages/cli/src/commands/gen.ts | 55 +++++ .../cli/test/gen-imported-nodes.test.ts | 204 ++++++++++++++++++ .../packages/codegen-ts/src/enum-shared.ts | 20 +- .../packages/codegen-ts/src/generator.ts | 10 + .../codegen-ts/src/generators/entity-file.ts | 8 +- .../codegen-ts/src/reference/entity.ts | 11 +- .../packages/codegen-ts/src/runner.ts | 6 + .../codegen-ts/src/templates/enums-file.ts | 15 +- .../test/shared-enums-imported.test.ts | 88 ++++++++ 9 files changed, 410 insertions(+), 7 deletions(-) create mode 100644 server/typescript/packages/cli/test/gen-imported-nodes.test.ts create mode 100644 server/typescript/packages/codegen-ts/test/shared-enums-imported.test.ts diff --git a/server/typescript/packages/cli/src/commands/gen.ts b/server/typescript/packages/cli/src/commands/gen.ts index d522e70f4..d38100e2f 100644 --- a/server/typescript/packages/cli/src/commands/gen.ts +++ b/server/typescript/packages/cli/src/commands/gen.ts @@ -17,6 +17,9 @@ import { import { loadMemory, resolveCollection } from "@metaobjectsdev/sdk"; import { runGen, listGenerators } from "@metaobjectsdev/codegen-ts"; import type { WriteStatus } from "@metaobjectsdev/codegen-ts"; +import { packageOfResolutionKey } from "@metaobjectsdev/metadata"; +import type { MetaRoot } from "@metaobjectsdev/metadata"; +import type { Collection } from "@metaobjectsdev/sdk"; import { reportLoadError } from "../lib/load-error.js"; /** @@ -131,6 +134,19 @@ export async function genCommand(args: string[], cwd: string, fmt: OutputFormat return 2; } + // FR-023 §11.1 item 2 — refuse a positional that names ONLY objects imported + // from a dependency and excluded from output by the default exclusion rule. + // Distinct from the runner's existing "no entities matched" WARNING + // (runner.ts): that path covers a name matching nothing loaded at all; this + // one covers a name that matches something real, just not something this + // project generates. A name matching nothing falls through untouched — the + // runner's own warning still covers it. + const importedRefusal = refuseImportedPositional(cliConfig.entities, metadata, genCollection); + if (importedRefusal !== undefined) { + log.error(importedRefusal); + return 2; + } + let result; try { result = await runGen({ @@ -221,6 +237,45 @@ export async function genCommand(args: string[], cwd: string, fmt: OutputFormat return hasFailure ? 1 : 0; } +/** + * FR-023 §11.1 item 2 — the error message for `meta gen ` when EVERY + * loaded object named `` is imported from a dependency and excluded from + * output (imported metadata is load-only by default; only the consumer's own + * `scope.include` naming the package literally opts it in). Returns undefined + * for every other case, INCLUDING a name matching nothing at all — that stays + * the runner's existing "no entities matched" warning, unchanged. + * + * A bare positional is a NAME, not a fully-qualified one (`cliConfig.entities` + * is matched against `entity.name` elsewhere in this file the same way), so two + * packages declaring the same short name are both candidates; the refusal + * fires only when NONE of them would generate. + */ +function refuseImportedPositional( + entityNames: readonly string[], + metadata: MetaRoot, + collection: Collection, +): string | undefined { + if (entityNames.length === 0) return undefined; + const allObjects = metadata.objects(); + for (const name of entityNames) { + const matches = allObjects.filter((o) => o.name === name); + if (matches.length === 0) continue; // nothing loaded by this name — the runner's own warning covers it + const allExcludedImports = matches.every((o) => { + const fqn = o.resolutionKey(); + return collection.imported(fqn) && !collection.inScope(fqn); + }); + if (!allExcludedImports) continue; + const fqn = matches[0]!.resolutionKey(); + const pkg = packageOfResolutionKey(fqn); + const dependencyName = collection.dependencies.find((d) => d.packages.includes(pkg))?.name ?? pkg; + return ( + `meta gen: '${name}' (${fqn}) is imported from dependency '${dependencyName}' and is not ` + + `generated here — add its package to scope.include in .metaobjects/config.json to generate it` + ); + } + return undefined; +} + /** * Run the advisory verify-as-teacher scan (same pass `meta verify` runs): surface * authored source that hand-rolls what the metadata could model. `gen` is the diff --git a/server/typescript/packages/cli/test/gen-imported-nodes.test.ts b/server/typescript/packages/cli/test/gen-imported-nodes.test.ts new file mode 100644 index 000000000..b6ad6b03f --- /dev/null +++ b/server/typescript/packages/cli/test/gen-imported-nodes.test.ts @@ -0,0 +1,204 @@ +// FR-023 §11.1 item 2 — codegen selection honours the default exclusion of +// imported metadata: a node loaded from a dependency's synced snapshot is +// load-only unless the consumer's OWN `scope.include` names its package +// literally. `meta gen`'s output already narrows via `genCollection.inScope` +// (Task 8); this gate covers what selection alone does not — a positional +// naming an excluded import must REFUSE (exit 2) rather than silently +// generate nothing or silently generate the import. +// +// The scaffold: a consumer package `app` (declares `Order`) depends on +// `acme-common` (a synced snapshot exporting `acme::common::Address` and +// `acme::common::Customer`) — the same fixture shape the shared +// dependency-conformance corpus (fixtures/dependency-conformance/) uses for +// "a-dependency-adds-its-artifact-to-the-resolved-set". +import { describe, test, expect, beforeEach, afterEach } from "bun:test"; +import { mkdtempSync, mkdirSync, rmSync, readdirSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { METAMODEL_VERSION } from "@metaobjectsdev/metadata"; +import { sha256Integrity } from "@metaobjectsdev/sdk"; +import { run } from "../src/index.js"; + +const WORKSPACE_TMP = resolve(import.meta.dirname, "fixtures", "__tmp__"); + +const DEP_NAME = "acme-common"; +const ARTIFACT_BASENAME = "acme-common.metaobjects.json"; + +/** The dependency's synced snapshot: one value object, one entity. */ +const ARTIFACT = JSON.stringify({ + "metadata.root": { + children: [ + { + "object.value": { + name: "Address", + package: "acme::common", + children: [{ "field.string": { name: "city" } }], + }, + }, + { + "object.entity": { + name: "Customer", + package: "acme::common", + children: [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "email" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +const APP = JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Order", + children: [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +function genOutDir(root: string): string { + return join(root, "generated", "db"); +} + +/** Scaffold a consumer of `acme-common`, optionally declaring `scope.include`. */ +function setupRepo(opts: { scopeInclude?: string[] } = {}): string { + mkdirSync(WORKSPACE_TMP, { recursive: true }); + const root = mkdtempSync(join(WORKSPACE_TMP, "forge-gen-imported-")); + + mkdirSync(join(root, "metaobjects"), { recursive: true }); + writeFileSync(join(root, "metaobjects", "meta.app.json"), APP, "utf8"); + + const depDir = join(root, ".metaobjects", "deps", DEP_NAME); + mkdirSync(depDir, { recursive: true }); + writeFileSync(join(depDir, ARTIFACT_BASENAME), ARTIFACT, "utf8"); + + const config: Record = { + schema_version: 1, + sources: [], + dependencies: [{ name: DEP_NAME, path: "../acme-common/metaobjects" }], + }; + if (opts.scopeInclude) { + config.scope = { include: opts.scopeInclude }; + } + mkdirSync(join(root, ".metaobjects"), { recursive: true }); + writeFileSync(join(root, ".metaobjects", "config.json"), JSON.stringify(config), "utf8"); + + writeFileSync( + join(root, ".metaobjects", "deps.lock.json"), + JSON.stringify({ + schema_version: 1, + dependencies: { + [DEP_NAME]: { + version: "1.0.0", + metamodelVersion: METAMODEL_VERSION, + resolvedFrom: { path: "../acme-common/metaobjects" }, + artifact: ARTIFACT_BASENAME, + integrity: sha256Integrity(ARTIFACT), + packages: ["acme::common"], + nodes: ["acme::common::Address", "acme::common::Customer"], + }, + }, + }), + "utf8", + ); + + writeFileSync( + join(root, "metaobjects.config.ts"), + ` +import { defineConfig } from "@metaobjectsdev/codegen-ts"; +export default defineConfig({ + outDir: ${JSON.stringify(genOutDir(root))}, + dialect: "sqlite", + dbImport: "~/db", + extStyle: "none", + generators: ["entity"], +}); +`, + ); + return root; +} + +let out: string[]; +let err: string[]; +let origLog: typeof console.log; +let origErr: typeof console.error; + +beforeEach(() => { + out = []; + err = []; + origLog = console.log; + origErr = console.error; + console.log = (...a: unknown[]) => { out.push(a.map(String).join(" ")); }; + console.error = (...a: unknown[]) => { err.push(a.map(String).join(" ")); }; +}); +afterEach(() => { + console.log = origLog; + console.error = origErr; +}); + +describe("meta gen — imported nodes are excluded by default (FR-023)", () => { + test("(a) an unscoped consumer generates its own entity, never the import", async () => { + const root = setupRepo(); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + const files = readdirSync(genOutDir(root)); + expect(files).toContain("Order.ts"); + expect(files).not.toContain("Customer.ts"); + expect(files).not.toContain("Address.ts"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test("(b) scope.include naming the dependency's package generates it too", async () => { + const root = setupRepo({ scopeInclude: ["app::**", "acme::common::**"] }); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + const files = readdirSync(genOutDir(root)); + expect(files).toContain("Customer.ts"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test("(c) naming an excluded import positionally refuses with exit 2", async () => { + const root = setupRepo(); + try { + const exit = await run(["gen", "Customer", "--cwd", root]); + expect(exit).toBe(2); + const all = err.join("\n"); + expect(all).toContain("imported from dependency 'acme-common'"); + expect(all).toContain("scope.include"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + test("(d) verify --codegen is green afterward and reports no drift for the absent import", async () => { + const root = setupRepo(); + try { + expect(await run(["gen", "--cwd", root])).toBe(0); + out = []; + err = []; + const exit = await run(["verify", "--cwd", root, "--codegen"]); + const all = [...out, ...err].join("\n"); + expect(exit).toBe(0); + expect(all).not.toContain("Customer.ts"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/server/typescript/packages/codegen-ts/src/enum-shared.ts b/server/typescript/packages/codegen-ts/src/enum-shared.ts index 54f0a46be..25f0cf632 100644 --- a/server/typescript/packages/codegen-ts/src/enum-shared.ts +++ b/server/typescript/packages/codegen-ts/src/enum-shared.ts @@ -77,10 +77,21 @@ export function sharedEnumForField(field: MetaField): SharedEnum | undefined { * actually CONSUMED by at least one concrete entity field — keyed by the * materialized type name. A declaration nobody extends is not materialized (no * dangling type). Deterministic order: first-consumption order across entities. + * + * `select` (FR-023 §11.1 item 2), when given, narrows which CONSUMING entities + * count toward "is this enum used" — an entity for which `select(entity. + * resolutionKey())` is false contributes nothing, even if it extends the shared + * declaration. It does not touch which declarations EXIST, only which usages are + * counted. Absent `select` ⇒ every entity counts, byte-identical to before this + * parameter existed. */ -export function collectSharedEnums(root: MetaRoot): Map { +export function collectSharedEnums( + root: MetaRoot, + select?: (fqn: string) => boolean, +): Map { const out = new Map(); for (const entity of root.objects()) { + if (select !== undefined && !select(entity.resolutionKey())) continue; for (const field of entity.fields()) { const shared = sharedEnumForField(field); if (shared === undefined) continue; @@ -91,8 +102,11 @@ export function collectSharedEnums(root: MetaRoot): Map { } /** The shared enums that metaobjects MATERIALIZES (non-@provided, consumed). */ -export function materializedSharedEnums(root: MetaRoot): SharedEnum[] { - return [...collectSharedEnums(root).values()].filter((e) => !e.provided); +export function materializedSharedEnums( + root: MetaRoot, + select?: (fqn: string) => boolean, +): SharedEnum[] { + return [...collectSharedEnums(root, select).values()].filter((e) => !e.provided); } /** Whether any shared enum (materialized or provided) is consumed in the model. */ diff --git a/server/typescript/packages/codegen-ts/src/generator.ts b/server/typescript/packages/codegen-ts/src/generator.ts index 334b74f9e..b7e35347f 100644 --- a/server/typescript/packages/codegen-ts/src/generator.ts +++ b/server/typescript/packages/codegen-ts/src/generator.ts @@ -20,6 +20,16 @@ export interface GenContext { * filter is set). Always call this from helpers; do not call generator.filter * directly. */ matches: (entity: MetaObject) => boolean; + /** FR-023 §11.1 item 2 — the model-wide output-scope predicate over a node's + * `resolutionKey()` (the runner's own `RunGenOpts.scope`, verbatim). A + * DIFFERENT knob from `matches`: `matches` is the per-generator `filter` + * (ANDed into what THIS generator emits), while `select` is the whole-model + * scope every generator shares — for a template that renders once over + * `ctx.loadedRoot` rather than per-entity (the shared-enums module is the + * first such template), `matches` never runs at all, so that template must + * read `select` directly to honour the same exclusion. Undefined ⇒ every + * node is in scope, byte-identical to a project with no `scope` declared. */ + select?: (fqn: string) => boolean; config: ResolvedGenConfig; /** Pre-built by the runner for built-in generators that wrap existing * templates. Third-party generators typically don't need this. Always diff --git a/server/typescript/packages/codegen-ts/src/generators/entity-file.ts b/server/typescript/packages/codegen-ts/src/generators/entity-file.ts index 8ca01454b..49ea2f7b9 100644 --- a/server/typescript/packages/codegen-ts/src/generators/entity-file.ts +++ b/server/typescript/packages/codegen-ts/src/generators/entity-file.ts @@ -59,7 +59,13 @@ export const entityFile = function entityFile(opts?: EntityFileOpts): Generator // FR-019: emit the shared-enums module ONCE per run, into the entity-module // target root. Returns null (no file) when the model uses no materialized // shared enums — keeping the inline-enum default byte-identical (no new file). - const sharedEnums = renderSharedEnumsFile(ctx.loadedRoot); + // FR-023 §11.1 item 2: `select` (never `ctx.matches` — this renders from the + // WHOLE root, not this generator's filtered subset) excludes an enum used + // only by an imported, out-of-scope entity. + const sharedEnums = renderSharedEnumsFile( + ctx.loadedRoot, + ctx.select !== undefined ? { select: ctx.select } : undefined, + ); if (sharedEnums !== null) { files.push({ path: `${SHARED_ENUMS_BASENAME}.ts`, diff --git a/server/typescript/packages/codegen-ts/src/reference/entity.ts b/server/typescript/packages/codegen-ts/src/reference/entity.ts index 0189349d2..b37d09e23 100644 --- a/server/typescript/packages/codegen-ts/src/reference/entity.ts +++ b/server/typescript/packages/codegen-ts/src/reference/entity.ts @@ -175,7 +175,16 @@ export const entityFile = function entityFile(opts?: EntityFileOpts): Generator generate: async (ctx: GenContext): Promise => { const files = await perEntityEmit(ctx); // FR-019: emit the shared-enums module once per run (null → no file). - const sharedEnums = renderSharedEnumsFile(ctx.loadedRoot); + // FR-023 §11.1 item 2: this is a REFERENCE TEMPLATE `meta init` copies into + // every scaffolded project's codegen/generators/ (ADR-0034) — the DEFAULT + // path, not an opt-in one. `select` (never `ctx.matches`, which never runs + // for a whole-root render like this one) excludes an enum used only by an + // imported, out-of-scope entity; without it every `meta init` project would + // silently emit shared enums the library path correctly excludes. + const sharedEnums = renderSharedEnumsFile( + ctx.loadedRoot, + ctx.select !== undefined ? { select: ctx.select } : undefined, + ); if (sharedEnums !== null) { files.push({ path: `${SHARED_ENUMS_BASENAME}.ts`, content: await formatTs(sharedEnums) }); } diff --git a/server/typescript/packages/codegen-ts/src/runner.ts b/server/typescript/packages/codegen-ts/src/runner.ts index b7e8f45e8..f0ee37229 100644 --- a/server/typescript/packages/codegen-ts/src/runner.ts +++ b/server/typescript/packages/codegen-ts/src/runner.ts @@ -605,6 +605,12 @@ export async function runGen(opts: RunGenOpts): Promise { entities: safeEntities, loadedRoot: root, matches: (e) => generator.filter?.(e) ?? true, + // FR-023 §11.1 item 2 — the SAME `scope` predicate that already narrowed + // `entities` above, re-exposed for a template that renders once over the + // WHOLE loaded root (the shared-enums module, so far the only one) rather + // than per matched entity, where `matches` never runs. Absent scope ⇒ no + // `select` at all, byte-identical to a project with no `scope` declared. + ...(scope !== undefined && { select: scope }), config: { outDir: selfTarget.outDir, extStyle: config.extStyle, diff --git a/server/typescript/packages/codegen-ts/src/templates/enums-file.ts b/server/typescript/packages/codegen-ts/src/templates/enums-file.ts index e18d8aa85..1c202eabd 100644 --- a/server/typescript/packages/codegen-ts/src/templates/enums-file.ts +++ b/server/typescript/packages/codegen-ts/src/templates/enums-file.ts @@ -37,9 +37,20 @@ export const ${sharedEnumZodConstName(e.name)} = ${z}.enum([${members}]); /** * The full shared-enums module body, or null when the model has no materialized * shared enums (so the generator emits no file at all). + * + * `opts.select` (FR-023 §11.1 item 2) — the model-wide scope predicate over a + * CONSUMING entity's `resolutionKey()`: an enum extended only by entities + * `select` excludes is not materialized. This module renders once per run from + * the WHOLE loaded root (never per-generator-filtered — see #266), so it is the + * one shared-enums call site that must be threaded the scope predicate + * separately from `ctx.matches`. Omitted ⇒ every entity counts, byte-identical + * to a caller that never passes `opts` at all. */ -export function renderSharedEnumsFile(root: MetaRoot): string | null { - const enums = materializedSharedEnums(root); +export function renderSharedEnumsFile( + root: MetaRoot, + opts?: { select?: (fqn: string) => boolean }, +): string | null { + const enums = materializedSharedEnums(root, opts?.select); if (enums.length === 0) return null; const body = joinCode(enums.map(renderOneSharedEnum), { on: "\n" }).toString(); diff --git a/server/typescript/packages/codegen-ts/test/shared-enums-imported.test.ts b/server/typescript/packages/codegen-ts/test/shared-enums-imported.test.ts new file mode 100644 index 000000000..b716133d2 --- /dev/null +++ b/server/typescript/packages/codegen-ts/test/shared-enums-imported.test.ts @@ -0,0 +1,88 @@ +// FR-023 §11.1 item 2 — the shared-enums module must honour the same +// model-wide `select` predicate codegen selection uses everywhere else. +// +// `renderSharedEnumsFile` walks the WHOLE loaded root (it has to — the module +// is emitted once per run, not per generator invocation), so without a +// `select` knob it would materialize a shared enum consumed ONLY by an +// imported, out-of-scope entity — the exact silent-wrong-output failure mode +// FR-023's default-exclusion rule exists to prevent (see task-10-brief.md). +// +// `select(entity.resolutionKey())` decides which CONSUMING entities count +// toward "is this enum actually used" — not which enum declarations exist. +import { describe, test, expect } from "bun:test"; +import { MetaDataLoader, InMemoryStringSource, type MetaRoot } from "@metaobjectsdev/metadata"; +import { renderSharedEnumsFile } from "../src/templates/enums-file.js"; + +/** + * Two packages, one shared enum: `lib::Status` (abstract, root-level) is + * extended by a field on BOTH `lib::Thing` and `app::Order` — so the enum is + * "used" from two different packages, and `select` can admit one while + * excluding the other. + */ +async function loadRoot(): Promise { + const lib = { + "metadata.root": { + package: "lib", + children: [ + { "field.enum": { name: "Status", abstract: true, "@values": ["a", "b"] } }, + { + "object.entity": { + name: "Thing", + children: [ + { "field.long": { name: "id" } }, + { "field.enum": { name: "status", extends: "lib::Status" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, + }; + const app = { + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Order", + children: [ + { "field.long": { name: "id" } }, + { "field.enum": { name: "status", extends: "lib::Status" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, + }; + const result = await new MetaDataLoader().load([ + new InMemoryStringSource(JSON.stringify(lib)), + new InMemoryStringSource(JSON.stringify(app)), + ]); + expect(result.errors).toEqual([]); + return result.root; +} + +describe("renderSharedEnumsFile — select (FR-023)", () => { + test("a select admitting only the consumer that uses the enum still materializes it", async () => { + const root = await loadRoot(); + const out = renderSharedEnumsFile(root, { select: (fqn) => fqn === "app::Order" }); + expect(out).not.toBeNull(); + expect(out).toContain("Status"); + }); + + test("a select admitting nothing materializes no shared enum at all", async () => { + const root = await loadRoot(); + const out = renderSharedEnumsFile(root, { select: () => false }); + expect(out).toBeNull(); + }); + + test("no select is byte-identical to today's (unfiltered) output", async () => { + const root = await loadRoot(); + const withoutOpts = renderSharedEnumsFile(root); + const withUndefinedSelect = renderSharedEnumsFile(root, {}); + expect(withoutOpts).not.toBeNull(); + expect(withoutOpts).toContain("Status"); + expect(withUndefinedSelect).toBe(withoutOpts); + }); +}); From 75c25db59484243d0d87ee1155ca0f13c359eb0b Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 21:41:43 -0400 Subject: [PATCH 24/62] docs(fr-023): the import predicate must reach the two indirect scope call sites MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-flight check before the schema-tooling task: its file list names four CLI sites, but only three of them call scopeExpectedSchema directly. The offline migrate pipeline reaches it through planOffline, and verify --db reaches it through computeDriftFromActual. Threading the option only at the direct sites would leave both of those performing no import exclusion at all, with every test still passing — that task's own Postgres test calls the function directly and cannot see the gap. Both option bags now carry it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 1e064f1e6..99487b2dc 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -836,6 +836,10 @@ feat(codegen-ts): imported nodes are not generated unless scope.include names th **Files:** - Modify: `server/typescript/packages/migrate-ts/src/scope.ts` (`scopeExpectedSchema(built, inScope, opts?: { imported?: ObjectScopePredicate })`) +- Modify: `server/typescript/packages/migrate-ts/src/snapshot/plan.ts` (`PlanOfflineArgs` gains `imported?: ObjectScopePredicate`; `planOffline` forwards it into its `scopeExpectedSchema(...)` call at ~:54). **Added 2026-09-11 by pre-flight check.** +- Modify: `server/typescript/packages/migrate-ts/src/drift/drift.ts` (`ComputeDriftOptions` gains `imported?: ObjectScopePredicate`; `computeDriftFromActual` forwards it into its `scopeExpectedSchema(...)` call at ~:100). **Added 2026-09-11 by pre-flight check.** + + **Why these two are NOT optional.** The Files list below names four CLI sites, and all four are real — but only THREE of them call `scopeExpectedSchema` directly (`migrate.ts:683` live/Kysely, `migrate.ts:1486` D1, `verify.ts:557` committed-snapshot gate). The other two CLI paths reach it ONE LAYER DOWN and would receive nothing: the OFFLINE migrate pipeline passes `inScope` into `planOffline` (`migrate.ts:1248/1259`), which calls `scopeExpectedSchema` itself at `plan.ts:54`; and `verify --db` passes `inScope` into `computeDriftFromActual` (`verify.ts:1001` sqlite/postgres, `verify.ts:1087` D1), which calls it at `drift.ts:100`. Threading `{ imported }` only at the direct call sites yields a diff whose tests all pass while `verify --db` and offline `migrate` perform NO import exclusion at all — and this task's own Postgres test calls `scopeExpectedSchema` directly, so it cannot catch the hole. A diff that updates the direct sites but not these two option bags is an Important finding. - Modify: `server/typescript/packages/cli/src/lib/migrate-scope.ts` (`dependencyNote(command, names)`; `migrateScopeMismatch` reads `collection.declaredMigrateScope`) - Modify: `server/typescript/packages/cli/src/commands/migrate.ts` (the three sites: `scopeExpectedSchema(built, collection.inMigrateScope)` at the online + apply pipelines and `offlineScope` for the offline pipeline — each passes `{ imported: collection.imported }`; print `dependencyNote` when any excluded import declared a table or view) - Modify: `server/typescript/packages/cli/src/commands/verify.ts` (`schemaScope` site: same `{ imported }`; the replay path's `inScope`; the note) From a64592586776ba703a01c3a5219dc6db19c5e7a7 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 21:45:22 -0400 Subject: [PATCH 25/62] docs(fr-023): the codegen-selection task's run commands name real test files Its step ran three test paths that are not in the repo. The implementer trusted the source over the example and ran the real analogs, which was right. Corrected to the real paths, including the integration subdirectory a flat listing of the test folder hides. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 99487b2dc..4b5e959b3 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -819,8 +819,8 @@ test(deps): the loader's existing errors are the first cross-repo drift gate (FR - [ ] **Step 2: Implement.** Run: ```bash -cd server/typescript/packages/codegen-ts && bun test test/shared-enums-imported.test.ts test/enums-file.test.ts -cd server/typescript/packages/cli && bun test test/gen-imported-nodes.test.ts test/gen.test.ts test/gen-list.test.ts test/verify-codegen.test.ts +cd server/typescript/packages/codegen-ts && bun test test/shared-enums-imported.test.ts test/enum-shared-provided.test.ts test/reference-byte-identical.test.ts +cd server/typescript/packages/cli && bun test test/gen-imported-nodes.test.ts test/gen-list.test.ts test/integration/gen-scope.test.ts test/integration/verify-codegen-scope.test.ts bun run --filter '@metaobjectsdev/codegen-ts' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck ``` Expected: PASS. (Golden gate: `cd server/typescript && bun test packages/codegen-ts/test` must show no golden change — `select` is absent for every golden project.) From 9d1e5d34456c81585af9b20f18eea5828bfeb727 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 21:46:04 -0400 Subject: [PATCH 26/62] docs(fr-023): point the scaffold example at a test that exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The codegen-selection task told an implementer to build its consumer scaffold the way a test does that is not in the repo — a pattern to copy rather than just a command to run, so it would have sent someone hunting. Now names the real gen test that scaffolds a config with the owned entity and barrel generators, verified to contain that shape before the edit was written. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 4b5e959b3..6a564acbc 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -815,7 +815,7 @@ test(deps): the loader's existing errors are the first cross-repo drift gate (FR **Interfaces:** - Produces: `renderSharedEnumsFile(root, { select })` emits an abstract enum iff at least one entity with `select(entity.resolutionKey())` true has a field that resolves to it (`resolveSharedEnumDecl`-style, via the field's `extends` chain — read `materializedSharedEnums`); with no `select`, byte-identical to today. `gen.ts`: before `runGen`, for each `cliConfig.entities` name, find the loaded objects with that bare name; if EVERY match is `imported && !inScope` → `log.error("meta gen: '' () is imported from dependency '' and is not generated here — add its package to scope.include in .metaobjects/config.json to generate it")`, exit 2. -- [ ] **Step 1: Failing tests.** `cli/test/gen-imported-nodes.test.ts`: scaffold a consumer (`APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1`, a `metaobjects.config.ts` with the owned `entityFile()` + `barrel()` as `gen.test.ts` does). (a) `meta gen` writes `Order.ts` and NOT `Customer.ts` / `Address.ts`. (b) With `"scope": { "include": ["app::**", "acme::common::**"] }`: `Customer.ts` IS written. (c) `meta gen Customer` (no include) → exit 2, stderr contains `imported from dependency 'acme-common'` and `scope.include`. (d) `meta verify --codegen` after (a) is green and reports no drift for the absent `Customer.ts`. `codegen-ts/test/shared-enums-imported.test.ts`: a root with an abstract `field.enum` `lib::Status` (`@values: ["a","b"]`) and two entities — `app::Order` with a field `extends: "lib::Status"` and `lib::Thing` with the same — `renderSharedEnumsFile(root, { select: (f) => f === "app::Order" })` contains `Status`; `select: () => false` returns `null`; no `select` equals today's output. Run → FAIL. +- [ ] **Step 1: Failing tests.** `cli/test/gen-imported-nodes.test.ts`: scaffold a consumer (`APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1`, a `metaobjects.config.ts` with the owned `entityFile()` + `barrel()` as `cli/test/gen-split-tree-single-import.test.ts` does (corrected 2026-09-11 — `gen.test.ts` does NOT exist; this is the real gen test that scaffolds a `metaobjects.config.ts` with the owned `entityFile()` + `barrel()`)). (a) `meta gen` writes `Order.ts` and NOT `Customer.ts` / `Address.ts`. (b) With `"scope": { "include": ["app::**", "acme::common::**"] }`: `Customer.ts` IS written. (c) `meta gen Customer` (no include) → exit 2, stderr contains `imported from dependency 'acme-common'` and `scope.include`. (d) `meta verify --codegen` after (a) is green and reports no drift for the absent `Customer.ts`. `codegen-ts/test/shared-enums-imported.test.ts`: a root with an abstract `field.enum` `lib::Status` (`@values: ["a","b"]`) and two entities — `app::Order` with a field `extends: "lib::Status"` and `lib::Thing` with the same — `renderSharedEnumsFile(root, { select: (f) => f === "app::Order" })` contains `Status`; `select: () => false` returns `null`; no `select` equals today's output. Run → FAIL. - [ ] **Step 2: Implement.** Run: ```bash From d28244ebb70dcf1571be93f44b3a3f1ab082fdf3 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 22:04:28 -0400 Subject: [PATCH 27/62] =?UTF-8?q?feat(migrate):=20imported=20tables=20leav?= =?UTF-8?q?e=20both=20sides=20of=20the=20schema=20diff,=20and=20never=20wi?= =?UTF-8?q?den=20its=20schema=20scope=20(FR-023=20=C2=A711)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A dependency's nodes are loaded so the consumer's own model can resolve against them. Nobody governs them here, so `migrate` and `verify --db` must exclude them on BOTH sides: out of the expected schema (no create, no alter) and suppressed on the actual side via `unmanagedNames` (no drop). Doing only the first half is strictly worse than doing nothing — the publisher's table exists in the shared database, so removing it from `expected` alone turns it into a proposed DROP TABLE against data this consumer does not own. The correction that makes it safe is an ORDERING one. `scopeExpectedSchema` pins `diff`'s schema scope to the unscoped model on purpose ("a scope narrows objects, never schemas"), which is right for `migrate.scope` — the same model declared into that schema — and wrong for an import: the consumer never declared into the publisher's schema, so leaving it pinned makes every table the publisher did NOT export a drop candidate. So an excluded import now leaves the expected side BEFORE `declaredSchemas` is computed, and the value is derived from the remainder. Everything `migrate.scope` excluded is still in that remainder, so the older rule is untouched. The predicate reaches all five CLI paths, including the two that call `scopeExpectedSchema` one layer down: `PlanOfflineArgs` (the offline pipeline, the default `meta migrate`) and `ComputeDriftOptions` (`verify --db`, both dialect paths). Threading only the direct sites would leave those performing no import exclusion while every test still passed. Also: the "your migrate.scope matched nothing" refusal now reads the DECLARED scope rather than the composed predicate. The composed one also excludes imports, so a consumer whose only table-declaring objects are imported — and who declared no scope at all — would have been refused, citing patterns that do not exist. `scopeExpectedSchema` without the new option is byte-identical: same snapshot object, no `declaredSchemas`, and the CLI supplies no predicate at all for a project with no dependencies. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../packages/cli/src/commands/migrate.ts | 33 +-- .../packages/cli/src/commands/verify.ts | 20 +- .../packages/cli/src/lib/migrate-scope.ts | 63 +++++- .../cli/test/migrate-imported-nodes.test.ts | 199 ++++++++++++++++++ .../test/dependency-imported-table-pg.test.ts | 173 +++++++++++++++ .../packages/migrate-ts/src/drift/drift.ts | 21 ++ .../packages/migrate-ts/src/scope.ts | 111 ++++++++-- .../packages/migrate-ts/src/snapshot/plan.ts | 24 ++- .../test/expected-schema-scope.test.ts | 128 ++++++++++- 9 files changed, 728 insertions(+), 44 deletions(-) create mode 100644 server/typescript/packages/cli/test/migrate-imported-nodes.test.ts create mode 100644 server/typescript/packages/integration-tests/test/dependency-imported-table-pg.test.ts diff --git a/server/typescript/packages/cli/src/commands/migrate.ts b/server/typescript/packages/cli/src/commands/migrate.ts index 01eda14d1..91ad03822 100644 --- a/server/typescript/packages/cli/src/commands/migrate.ts +++ b/server/typescript/packages/cli/src/commands/migrate.ts @@ -14,7 +14,7 @@ import { log } from "../lib/log.js"; import { loadMemory, resolveCollection, resolveConfigDir, type Collection } from "@metaobjectsdev/sdk"; import { loadMemoryOptionsFrom, loadMetaobjectsConfig, resolveGenConfigDir } from "../lib/load-metaobjects-config.js"; import { collectionLoadOptions } from "../lib/collection-load-options.js"; -import { migrateScopeMismatch, outOfScopeNote } from "../lib/migrate-scope.js"; +import { exclusionNotes, importedOption, migrateScopeMismatch } from "../lib/migrate-scope.js"; import { allowOptionFor, buildExpectedSchemaWithProvenance, @@ -202,7 +202,8 @@ function warnIfLedgerRelocated(cwd: string, resolvedOutDir: string): void { } /** - * Report what a declared `migrate.scope` left out (wording: `outOfScopeNote`). + * Report what this run did not govern — what a declared `migrate.scope` left out, and + * what a DEPENDENCY owns (wording: `exclusionNotes`). * * STDOUT in text format, STDERR otherwise. `--format json` / `--format toon` put a * single machine-readable document on stdout, and a prose line ahead of it breaks @@ -211,11 +212,15 @@ function warnIfLedgerRelocated(cwd: string, resolvedOutDir: string): void { * (`resolveFormat`): suppressing it outright would silence the note for every * piped and CI run, which is most of them. */ -function logOutOfScope(names: readonly string[], fmt: OutputFormat): void { - if (names.length === 0) return; - const msg = outOfScopeNote("migrate", names); - if (fmt === "text") log.info(msg); - else log.warn(msg); +function logOutOfScope( + names: readonly string[], + fromDependencies: readonly string[], + fmt: OutputFormat, +): void { + for (const msg of exclusionNotes("migrate", names, fromDependencies)) { + if (fmt === "text") log.info(msg); + else log.warn(msg); + } } function emitStructuredError(error: string, hint: string, fmt: OutputFormat): void { @@ -680,9 +685,9 @@ export async function migrateCommand( // leave the expected schema here and are suppressed on the actual side below — // dropping them from `expected` ALONE would propose DROP TABLE for every one of // them that exists in the database. - const scoped = scopeExpectedSchema(built, collection.inMigrateScope); + const scoped = scopeExpectedSchema(built, collection.inMigrateScope, importedOption(collection)); const expected = scoped.snapshot; - logOutOfScope(scoped.outOfScope, fmt); + logOutOfScope(scoped.outOfScope, scoped.importedOutOfScope ?? [], fmt); let actual; try { actual = await introspect(kysely.db, kysely.dialect); @@ -1257,6 +1262,10 @@ export async function runOfflineGenerate( views: offlineViews, // Per-command scope — narrows BOTH sides of the offline diff (see planOffline). ...(offlineScope !== undefined ? { inScope: offlineScope } : {}), + // ...and the import exclusion, which planOffline threads into the SAME call. + // Passing it only at the direct call sites would leave the offline pipeline — + // the default `meta migrate` — performing no import exclusion at all. + ...importedOption(collection), allow: tokensToAllowOptions(config.allow), onAmbiguous: async (a) => { collectedAmbiguous.push(a); @@ -1281,7 +1290,7 @@ export async function runOfflineGenerate( // entries); `expected` is the governed side the emitter renders against. Equal // for an unscoped run. const { diff: diffResult, nextSnapshot, expected: governedExpected } = plan; - logOutOfScope(plan.outOfScope, fmt); + logOutOfScope(plan.outOfScope, plan.importedOutOfScope ?? [], fmt); if (diffResult.blocked.length > 0) { log.error(`migrate: ${diffResult.blocked.length} destructive change(s) blocked; re-run with --allow `); @@ -1483,9 +1492,9 @@ async function runD1Migrate( const scopeRc = refuseScopeMismatch(collection, () => built.provenance, fmt); if (scopeRc !== undefined) return scopeRc; // Per-command scope — both-sided, exactly as on the Kysely path above. - const scoped = scopeExpectedSchema(built, collection.inMigrateScope); + const scoped = scopeExpectedSchema(built, collection.inMigrateScope, importedOption(collection)); const expected = scoped.snapshot; - logOutOfScope(scoped.outOfScope, fmt); + logOutOfScope(scoped.outOfScope, scoped.importedOutOfScope ?? [], fmt); let actual; try { actual = await introspectD1({ diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index ec9c70361..17e2922d6 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -71,7 +71,7 @@ import { type DriftResult, } from "@metaobjectsdev/migrate-ts"; import { loadMemory, resolveCollection } from "@metaobjectsdev/sdk"; -import { migrateScopeMismatch, outOfScopeNote } from "../lib/migrate-scope.js"; +import { exclusionNotes, importedOption, migrateScopeMismatch } from "../lib/migrate-scope.js"; import { TYPE_TEMPLATE, TEMPLATE_SUBTYPE_PROMPT, @@ -554,7 +554,7 @@ export async function verifyCommand( columnNamingStrategy: viewStrategy, views: buildProjectionViews(root, { dialect, columnNamingStrategy: viewStrategy }), }); - governed = scopeExpectedSchema(built, schemaScope); + governed = scopeExpectedSchema(built, schemaScope, importedOption(collection)); } // `verifyReplay` calls `applyPending` itself. That is NOT a second replay: the @@ -1002,6 +1002,9 @@ export async function verifyCommand( allow, views: expectedViews, ...(schemaScope !== undefined ? { inScope: schemaScope } : {}), + // The import exclusion reaches scopeExpectedSchema through this option bag — + // there is no direct call on this path to attach it to. + ...importedOption(collection), }); } catch (err) { log.error(`verify: failed to introspect ${kysely.displayUrl}: ${(err as Error).message}`); @@ -1088,6 +1091,8 @@ export async function verifyCommand( allow, views: expectedViews, ...(schemaScope !== undefined ? { inScope: schemaScope } : {}), + // Same option bag on the D1 path, for the same reason. + ...importedOption(collection), }); } catch (err) { log.error(`verify: ${(err as Error).message}`); @@ -1200,11 +1205,12 @@ export async function verifyCommand( ); } - // Same reasoning for the per-command scope: an object `migrate.scope` excluded - // was NOT checked, and silence would misreport it as checked-and-clean. Shared - // wording with `meta migrate` — one declaration, one sentence about it. - if (driftResult.outOfScope.length > 0) { - say(outOfScopeNote("verify", driftResult.outOfScope)); + // Same reasoning for the per-command scope and for imported metadata: an object + // excluded either way was NOT checked, and silence would misreport it as + // checked-and-clean. Shared wording with `meta migrate` — one declaration, one + // sentence about it — and each object named once (`exclusionNotes`). + for (const note of exclusionNotes("verify", driftResult.outOfScope, driftResult.importedOutOfScope ?? [])) { + say(note); } const changes = driftResult.changes; diff --git a/server/typescript/packages/cli/src/lib/migrate-scope.ts b/server/typescript/packages/cli/src/lib/migrate-scope.ts index 8d2d814b2..3f5878f8f 100644 --- a/server/typescript/packages/cli/src/lib/migrate-scope.ts +++ b/server/typescript/packages/cli/src/lib/migrate-scope.ts @@ -10,7 +10,7 @@ // or a refusal that drifted between them would be drift the user reads. import type { Collection } from "@metaobjectsdev/sdk"; -import type { SchemaProvenance } from "@metaobjectsdev/migrate-ts"; +import type { ObjectScopePredicate, SchemaProvenance } from "@metaobjectsdev/migrate-ts"; /** * Say what a declared `migrate.scope` left out, for `migrate` and `verify --db` @@ -32,6 +32,56 @@ export function outOfScopeNote(command: string, names: readonly string[]): strin ); } +/** + * Say what a DEPENDENCY owns, for `migrate` and `verify --db` alike (FR-023). + * + * A dependency's nodes are loaded so this project's own model can resolve against + * them, and are governed by nobody here. That is not the same sentence as + * `outOfScopeNote`'s: an excluded import is usually not the result of anything the + * author wrote — a project with no `migrate.scope` at all still excludes its imports — + * so reporting it as "outside migrate.scope" would name a key that does not exist. + * It also carries the opt-in, which is the whole remedy: name the package. + */ +export function dependencyNote(command: string, names: readonly string[]): string { + return ( + `meta ${command} — ${names.length} object(s) from dependencies not governed here ` + + `(name the package in migrate.scope to own them): ${names.join(", ")}` + ); +} + +/** + * Every exclusion a run owes the reader, each object named exactly ONCE. + * + * `outOfScope` is the full suppression set and `fromDependencies` a subset of it, so + * reporting both as-is would print an imported table twice under two different + * explanations. The partition happens here, once, rather than in each command. + */ +export function exclusionNotes( + command: string, + outOfScope: readonly string[], + fromDependencies: readonly string[], +): string[] { + const notes: string[] = []; + const imported = new Set(fromDependencies); + const declared = outOfScope.filter((name) => !imported.has(name)); + if (declared.length > 0) notes.push(outOfScopeNote(command, declared)); + if (fromDependencies.length > 0) notes.push(dependencyNote(command, fromDependencies)); + return notes; +} + +/** + * The import predicate a schema run threads into migrate-ts, or NOTHING. + * + * Nothing is the load-bearing half. A project with no dependencies must reach + * `scopeExpectedSchema` exactly as it always did — passing a predicate that happens to + * answer `false` for everything is not the same thing, because supplying one at all + * leaves the untouched same-object path and changes `declaredSchemas` from absent to + * derived. One helper so all five call sites make that decision identically. + */ +export function importedOption(collection: Collection): { imported?: ObjectScopePredicate } { + return collection.dependencies.length > 0 ? { imported: collection.imported } : {}; +} + /** How many loaded FQNs to name in the refusal below — enough to show the shape * an author's patterns have to match, short enough to stay readable. */ const EXAMPLE_FQN_CAP = 3; @@ -70,8 +120,13 @@ export function migrateScopeMismatch( */ provenance: () => SchemaProvenance, ): string | undefined { - const { inMigrateScope, migrateScopePatterns } = collection; - if (inMigrateScope === undefined) return undefined; + // The DECLARED `migrate.scope` alone, never the composed `inMigrateScope` (FR-023). + // The composed predicate also excludes imports, so a consumer whose only + // table-declaring objects come from a dependency — and who declared no scope at all + // — would be refused here, citing patterns that do not exist. There is nothing wrong + // with that project: its imports are excluded by design and it governs no tables yet. + const { declaredMigrateScope, migrateScopePatterns } = collection; + if (declaredMigrateScope === undefined) return undefined; // The declaring FQNs of every table and view the UNSCOPED model contributes — // the same provenance `scopeExpectedSchema` decides scope on, so the refusal @@ -87,7 +142,7 @@ export function migrateScopeMismatch( // nothing for a pattern to govern, scoped or not, and an empty schema has its // own (much louder) failure modes downstream. if (fqns.length === 0) return undefined; - if (fqns.some(inMigrateScope)) return undefined; + if (fqns.some(declaredMigrateScope)) return undefined; const patterns = JSON.stringify(migrateScopePatterns ?? []); const examples = fqns.slice(0, EXAMPLE_FQN_CAP).join(", "); diff --git a/server/typescript/packages/cli/test/migrate-imported-nodes.test.ts b/server/typescript/packages/cli/test/migrate-imported-nodes.test.ts new file mode 100644 index 000000000..3bf636bb2 --- /dev/null +++ b/server/typescript/packages/cli/test/migrate-imported-nodes.test.ts @@ -0,0 +1,199 @@ +/** + * FR-023 §11.1 item 2 — `meta migrate` never touches a dependency's tables. + * + * A consumer loads a dependency's nodes so its OWN model can resolve against them. + * Those nodes are load-only: the publisher owns their tables, and this consumer + * never declared them. So the suppression has to be two-sided, exactly as + * `migrate.scope`'s is — the imported table leaves the EXPECTED side (no CREATE, + * no ALTER) and is suppressed on the ACTUAL side (no DROP). + * + * Doing only the first half is strictly worse than doing nothing: the publisher's + * table exists in the shared database, so removing it from `expected` alone turns + * it into a proposed `DROP TABLE` — against data this consumer does not own. Case + * (b) is that hazard. + * + * The opt-in is the declaration that already exists: naming the dependency's + * package in `migrate.scope` says "I own these tables here" (case (c)). + */ +import { describe, test, expect, beforeEach, afterEach, afterAll } from "bun:test"; +import { mkdtemp, rm, mkdir, writeFile, readdir, readFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { METAMODEL_VERSION } from "@metaobjectsdev/metadata"; +import { sha256Integrity } from "@metaobjectsdev/sdk"; +import { snapshotPath, writeSnapshot } from "@metaobjectsdev/migrate-ts"; +import { runBaseline, runOfflineGenerate } from "../src/commands/migrate.js"; + +const DEP_NAME = "acme-common"; +const ARTIFACT_BASENAME = "acme-common.metaobjects.json"; +const MIGRATIONS_DIR = ".metaobjects/migrations"; + +/** The dependency's synced snapshot — the publisher's `customers` table. */ +const SNAP = JSON.stringify({ + "metadata.root": { + children: [ + { + "object.entity": { + name: "Customer", + package: "acme::common", + children: [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "email", "@maxLength": 120 } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +/** The consumer's own model. */ +const APP = JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Order", + children: [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +const dirs: string[] = []; +afterAll(async () => { for (const d of dirs) await rm(d, { recursive: true, force: true }); }); + +/** A consumer of `acme-common`, optionally declaring `migrate.scope`. */ +async function project(opts: { migrateScope?: string[] } = {}): Promise { + const root = await mkdtemp(join(tmpdir(), "migrate-imported-")); + dirs.push(root); + + await mkdir(join(root, "metaobjects"), { recursive: true }); + await writeFile(join(root, "metaobjects", "meta.app.json"), APP, "utf8"); + + const depDir = join(root, ".metaobjects", "deps", DEP_NAME); + await mkdir(depDir, { recursive: true }); + await writeFile(join(depDir, ARTIFACT_BASENAME), SNAP, "utf8"); + + const config: Record = { + schema_version: 1, + sources: [], + dependencies: [{ name: DEP_NAME, path: "../acme-common/metaobjects" }], + }; + if (opts.migrateScope) config["migrate"] = { scope: opts.migrateScope }; + await writeFile(join(root, ".metaobjects", "config.json"), JSON.stringify(config), "utf8"); + + await writeFile( + join(root, ".metaobjects", "deps.lock.json"), + JSON.stringify({ + schema_version: 1, + dependencies: { + [DEP_NAME]: { + version: "1.0.0", + metamodelVersion: METAMODEL_VERSION, + resolvedFrom: { path: "../acme-common/metaobjects" }, + artifact: ARTIFACT_BASENAME, + integrity: sha256Integrity(SNAP), + packages: ["acme::common"], + nodes: ["acme::common::Customer"], + }, + }, + }), + "utf8", + ); + return root; +} + +/** A greenfield reference snapshot: the file exists and records nothing. */ +async function emptySnapshot(root: string): Promise { + await writeSnapshot(snapshotPath(join(root, MIGRATIONS_DIR), "sqlite"), { tables: [], views: [] }); +} + +const cfg = () => + ({ dialect: "sqlite", outDir: `./${MIGRATIONS_DIR}`, onAmbiguous: "abort", + allow: [], slug: "auto", dryRun: false } as never); + +const migrationDirs = async (root: string): Promise => + (await readdir(join(root, MIGRATIONS_DIR))).filter((e) => !e.startsWith(".")); + +async function upSql(root: string): Promise { + const [dir] = await migrationDirs(root); + expect(dir).toBeDefined(); + return await readFile(join(root, MIGRATIONS_DIR, dir!, "up.sql"), "utf8"); +} + +let out: string[]; +let err: string[]; +let origLog: typeof console.log; +let origErr: typeof console.error; + +beforeEach(() => { + out = []; + err = []; + origLog = console.log; + origErr = console.error; + console.log = (...a: unknown[]) => { out.push(a.map(String).join(" ")); }; + console.error = (...a: unknown[]) => { err.push(a.map(String).join(" ")); }; +}); +afterEach(() => { + console.log = origLog; + console.error = origErr; +}); + +describe("meta migrate — imported tables are not governed (FR-023)", () => { + test("(a) the consumer's own table is created; the import is neither created nor named as drift", async () => { + const root = await project(); + await emptySnapshot(root); + + expect(await runOfflineGenerate(cfg(), root)).toBe(0); + const up = await upSql(root); + expect(up).toContain(`CREATE TABLE "orders"`); + expect(up).not.toContain("customers"); + + // The exclusion is REPORTED: an object silently dropped from the comparison is + // indistinguishable from one that was checked and found clean. (Which STREAM it + // lands on is the output format's decision, not this feature's — `logOutOfScope` + // puts narration on stdout in text format and on stderr in the structured ones, + // where stdout must carry exactly one document.) + const reported = [...out, ...err].join("\n"); + expect(reported).toContain("1 object(s) from dependencies not governed here"); + expect(reported).toContain("public.customers"); + // ...and it is NOT reported as a `migrate.scope` exclusion: this project declares + // no such key, so naming one would send the reader to a setting that isn't there. + expect(reported).not.toContain("outside migrate.scope"); + }); + + test("(b) the publisher's existing table is NOT proposed for drop", async () => { + const root = await project(); + // An unscoped baseline records what the shared database holds — both owners' + // tables — exactly as `baseline --from-db` would. + expect(await runBaseline(cfg(), root)).toBe(0); + out = []; + err = []; + + // Nothing changed, and `customers` must not become a DROP just because this + // consumer does not govern it. A proposed drop would be blocked (`allow` is + // empty) and exit 1; an unsuppressed ACTUAL side is what would produce it. + expect(await runOfflineGenerate(cfg(), root)).toBe(0); + expect(await migrationDirs(root)).toHaveLength(0); + }); + + test("(c) naming the dependency's package in migrate.scope opts INTO owning its tables", async () => { + const root = await project({ migrateScope: ["app::**", "acme::common::**"] }); + await emptySnapshot(root); + + expect(await runOfflineGenerate(cfg(), root)).toBe(0); + const up = await upSql(root); + expect(up).toContain(`CREATE TABLE "orders"`); + expect(up).toContain(`CREATE TABLE "customers"`); + // Nothing was excluded, so there is nothing to report. + expect([...out, ...err].join("\n")).not.toContain("from dependencies not governed here"); + }); +}); diff --git a/server/typescript/packages/integration-tests/test/dependency-imported-table-pg.test.ts b/server/typescript/packages/integration-tests/test/dependency-imported-table-pg.test.ts new file mode 100644 index 000000000..091d710c7 --- /dev/null +++ b/server/typescript/packages/integration-tests/test/dependency-imported-table-pg.test.ts @@ -0,0 +1,173 @@ +/** + * FR-023 §11.1 item 2 — the schema-scope half of the import exclusion, against a + * REAL Postgres, because it is the only place the bug is visible. + * + * `scopeExpectedSchema` pins `diff`'s schema scope to the UNSCOPED model on purpose: + * "a scope narrows objects, never schemas". That is right for `migrate.scope` — the + * same model declared into that schema — and wrong for an import. The consumer never + * declared into the publisher's schema, so leaving it pinned makes every table the + * publisher did NOT export a proposed `DROP TABLE` against the publisher's own data. + * + * SQLite cannot see this: it has no schema concept, so every object normalizes to one + * prefix and the pin is a constant. Postgres can, which is why this lives here. + * + * The consumer declares `app.orders` (its own schema) and imports `acme::common::Customer` + * (`public.customers`). The database also holds `public.invoices` — a publisher table the + * artifact does not export, and which this consumer has therefore never heard of. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from "bun:test"; +import { + buildExpectedSchemaWithProvenance, + collectUnmanagedNames, + diff, + introspectPostgres, + scopeExpectedSchema, + scopedDiffInputs, +} from "@metaobjectsdev/migrate-ts"; +import type { Change } from "@metaobjectsdev/migrate-ts"; +import { MetaDataLoader, InMemoryStringSource, type MetaRoot } from "@metaobjectsdev/metadata"; +import { Kysely, PostgresDialect, sql } from "kysely"; +import { Pool } from "pg"; +import { startPostgres, type RunningPg } from "../src/postgres-container.ts"; + +/** The dependency's published artifact (the committed v1 corpus shape): no root + * package, each top-level node carrying its own. */ +const ARTIFACT = JSON.stringify({ + "metadata.root": { + children: [ + { + "object.entity": { + name: "Customer", + package: "acme::common", + children: [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "email", "@maxLength": 120 } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +/** The consumer's own model, in its own database schema. */ +const APP = JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Order", + children: [ + { "source.rdb": { "@table": "orders", "@schema": "app" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +const ownOnly = (fqn: string): boolean => fqn === "app::Order"; +const imported = (fqn: string): boolean => fqn.startsWith("acme::common::"); + +let runningPg: RunningPg; +let pool: Pool; +let k: Kysely>; +let root: MetaRoot; + +beforeAll(async () => { + runningPg = await startPostgres(); + pool = new Pool({ connectionString: runningPg.connectionUri }); + k = new Kysely>({ dialect: new PostgresDialect({ pool }) }); + const loaded = await new MetaDataLoader().load([ + new InMemoryStringSource(ARTIFACT), + new InMemoryStringSource(APP), + ]); + expect(loaded.errors).toHaveLength(0); + root = loaded.root; +}, 180_000); + +afterAll(async () => { + await k?.destroy(); + await runningPg?.stop(); +}); + +beforeEach(async () => { + await sql.raw(`DROP TABLE IF EXISTS public.customers CASCADE;`).execute(k); + await sql.raw(`DROP TABLE IF EXISTS public.invoices CASCADE;`).execute(k); + await sql.raw(`DROP SCHEMA IF EXISTS app CASCADE;`).execute(k); + // The publisher's database, as the publisher made it: the exported table, and one + // it never exported. + await sql.raw(`CREATE TABLE public.customers (id bigint primary key, email varchar(120));`).execute(k); + await sql.raw(`CREATE TABLE public.invoices (id bigint primary key);`).execute(k); +}); + +/** Every change the scoped run proposes, with drops ALLOWED so a proposed drop shows + * up in `changes` rather than being filed under `blocked` and missed. */ +async function plan(inScope: (fqn: string) => boolean, withImportExclusion: boolean): Promise { + const built = buildExpectedSchemaWithProvenance(root, { dialect: "postgres" }); + const scoped = withImportExclusion + ? scopeExpectedSchema(built, inScope, { imported }) + : scopeExpectedSchema(built, inScope); + const actual = await introspectPostgres(k); + const result = await diff({ + ...scopedDiffInputs(scoped, collectUnmanagedNames(root)), + actual, + dialect: "postgres", + allow: { dropTable: true }, + }); + return result.changes; +} + +const touching = (ops: Change[], table: string): Change[] => + ops.filter((c) => ("table" in c ? (typeof c.table === "string" ? c.table : c.table.name) === table : false)); + +describe("an imported table leaves both sides of the schema diff — real Postgres", () => { + test("the consumer's own table is created, and the publisher's schema is never governed", async () => { + const ops = await plan(ownOnly, true); + + // The consumer's own table, in its own schema. + const creates = ops.filter((c) => c.kind === "create-table"); + expect(creates).toHaveLength(1); + const [create] = creates; + if (create?.kind !== "create-table") throw new Error("expected a create-table"); + expect(create.table.name).toBe("orders"); + expect(create.table.schema).toBe("app"); + + // THE assertion. `public.invoices` is a publisher table this consumer has never + // heard of. If the import stayed on the expected side long enough to pin `public` + // as a declared schema, this is a DROP TABLE against the publisher's data. + expect(touching(ops, "invoices")).toEqual([]); + + // And the imported table itself is untouched in either direction. + expect(touching(ops, "customers")).toEqual([]); + }, 180_000); + + test("counter-assert: a consumer that OWNS the package governs that schema, visibly", async () => { + // `migrate.scope` naming `acme::common` is the opt-in — the consumer took the + // publisher's package, so `public` is legitimately its schema now, and an + // undeclared table in it is a drop candidate exactly as it would be in its own. + const ops = await plan(() => true, true); + + const drops = touching(ops, "invoices"); + expect(drops).toHaveLength(1); + expect(drops[0]?.kind).toBe("drop-table"); + + // The exported table matches the metadata, so it produces nothing either way. + expect(touching(ops, "customers")).toEqual([]); + }, 180_000); + + test("the hazard is real: with no import predicate the same run proposes the destructive drop", async () => { + // Today's behaviour, pinned as the counter-assertion: the consumer governs only + // its own object, yet `public` is still pinned by the import it never declared — + // so the publisher's unexported table is proposed for DROP. + const ops = await plan(ownOnly, false); + + const drops = touching(ops, "invoices"); + expect(drops).toHaveLength(1); + expect(drops[0]?.kind).toBe("drop-table"); + }, 180_000); +}); diff --git a/server/typescript/packages/migrate-ts/src/drift/drift.ts b/server/typescript/packages/migrate-ts/src/drift/drift.ts index eb068c02a..f515283bf 100644 --- a/server/typescript/packages/migrate-ts/src/drift/drift.ts +++ b/server/typescript/packages/migrate-ts/src/drift/drift.ts @@ -58,6 +58,18 @@ export interface ComputeDriftOptions { * failing on tables migrate does not own is incoherent. */ inScope?: ObjectScopePredicate; + /** + * FR-023 — objects a DEPENDENCY owns. Such an object is loaded so this project's own + * model can resolve against it and is governed by nobody here unless `inScope` + * admits it: it leaves the expected side, is suppressed on the actual side, and — + * unlike an `inScope` exclusion — takes the publisher's database schema out of the + * run's scope with it, so a table the publisher never exported is not reported as an + * extra table in a schema this project does not manage (migrate-ts `scope.ts`). + * + * `verify --db` and `migrate` share this declaration for the same reason they share + * `inScope`. Omit for a project with no dependencies (unchanged behavior). + */ + imported?: ObjectScopePredicate; } export interface DriftResult extends DiffResult { @@ -77,6 +89,13 @@ export interface DriftResult extends DiffResult { * `GovernedScope`, which is what `excludeFromSnapshot` takes. */ declaredSchemas: readonly string[] | undefined; + /** + * The subset of `outOfScope` a DEPENDENCY contributed (FR-023). Reported for the + * same reason `declaredSchemas` is: so the caller states WHY an object was left out + * without re-deriving the decision from a second expected-schema build, which could + * come to disagree with the one this comparison actually made. + */ + importedOutOfScope: readonly string[] | undefined; } /** @@ -106,6 +125,7 @@ export async function computeDriftFromActual( ...(opts?.views !== undefined ? { views: opts.views } : {}), }), opts?.inScope, + opts?.imported !== undefined ? { imported: opts.imported } : undefined, ); const result = await diff({ // The three scoped-diff obligations as one value (see scope.ts's header): @@ -122,6 +142,7 @@ export async function computeDriftFromActual( ...result, outOfScope: scoped.outOfScope, declaredSchemas: scoped.declaredSchemas, + importedOutOfScope: scoped.importedOutOfScope, }; } diff --git a/server/typescript/packages/migrate-ts/src/scope.ts b/server/typescript/packages/migrate-ts/src/scope.ts index 0fa36ed27..b6f4dbb78 100644 --- a/server/typescript/packages/migrate-ts/src/scope.ts +++ b/server/typescript/packages/migrate-ts/src/scope.ts @@ -34,6 +34,18 @@ // survivors instead is precisely the inversion above. Declaring a scope is not a way // to hand a schema over; removing the objects from the model is. // +// FR-023 — AN IMPORT IS THE ONE EXCLUSION THAT MUST MOVE THE SCHEMA SET. The rule +// above is stated for `migrate.scope`, where it is right: that scope narrows a model +// which DID declare into the schema, so the schema stays this model's to manage. A +// dependency's node is the opposite case — the consumer never declared it, and the +// publisher owns the schema it sits in. Leaving that schema pinned would make every +// table the publisher did NOT export, tables this consumer has never heard of, a +// proposed DROP against the publisher's data. +// +// So `scopeExpectedSchema`'s optional `imported` predicate partitions those objects +// out FIRST, and `declaredSchemas` is derived from what remains. Everything +// `migrate.scope` excluded is still in that remainder, so the rule above is untouched. +// // `scopedDiffInputs` exists so no caller has to remember any of this: it returns all // three obligations as one object, and every scoped `diff` call goes through it. @@ -74,6 +86,16 @@ export interface ScopedExpectedSchema { * (nothing to derive from; `diff`'s legacy whole-DB fallback is preserved). */ declaredSchemas?: string[]; + /** + * The subset of `outOfScope` a DEPENDENCY contributed (FR-023), so a caller can say + * why an object was excluded. `outOfScope` deliberately stays the FULL set — it is + * what feeds `unmanagedNames`, and splitting it would silently drop half of the + * actual-side suppression. + * + * `undefined` when no `imported` predicate was supplied, which is every project that + * declares no dependencies. + */ + importedOutOfScope?: string[]; } /** @@ -217,45 +239,98 @@ export function declaredSchemasOf(snapshot: SchemaSnapshot): string[] { ].sort(); } +/** Optional exclusions layered on top of `inScope`. */ +export interface ScopeExpectedSchemaOptions { + /** + * FR-023 — is this object's declaring FQN owned by one of the project's + * DEPENDENCIES? Such an object is loaded so the consumer's own model can resolve + * against it, and is governed by nobody here unless the consumer's own + * `migrate.scope` names its package (which `inScope` then admits). + * + * Supplied only by a project that declares dependencies: absent, this function + * behaves exactly as it always has. + */ + imported?: ObjectScopePredicate; +} + /** * Narrow an expected schema to the objects inside `inScope`. * - * An undefined predicate returns the input untouched — the SAME snapshot object, - * not an equal copy — so a project that declares no `migrate.scope` reaches the - * diff, the emitter and the committed snapshot through an unchanged value. + * An undefined predicate with no `opts` returns the input untouched — the SAME + * snapshot object, not an equal copy — so a project that declares no `migrate.scope` + * and no dependencies reaches the diff, the emitter and the committed snapshot + * through an unchanged value. + * + * A table or view with NO recorded provenance is KEPT, by both exclusions. Scope + * decides on the declaring object's FQN, and an object whose FQN is unknown was never + * proven to be anyone else's; dropping it would silently un-manage it (and, worse, + * suppressing its name on the actual side would hide real drift). * - * A table or view with NO recorded provenance is KEPT. Scope decides on the - * declaring object's FQN, and an object whose FQN is unknown was never proven to be - * anyone else's; dropping it would silently un-manage it (and, worse, suppressing - * its name on the actual side would hide real drift). + * With `opts.imported`, an imported object the scope does not admit leaves the + * expected side BEFORE `declaredSchemas` is computed — see the module header for why + * that ordering is the whole point. */ export function scopeExpectedSchema( built: ExpectedSchemaWithProvenance, inScope: ObjectScopePredicate | undefined, + opts?: ScopeExpectedSchemaOptions, ): ScopedExpectedSchema { - if (inScope === undefined) return { snapshot: built.snapshot, outOfScope: [] }; + const imported = opts?.imported; + if (inScope === undefined && imported === undefined) { + return { snapshot: built.snapshot, outOfScope: [] }; + } + + // PASS 1 — the import partition. An imported object survives only when the + // consumer's own scope NAMES it, which is how a consumer opts into owning a + // dependency's tables (DESIGN §11.1 item 2: "explicit include replaces `mode`"). + // + // An `inScope` of `undefined` does NOT rescue it. That combination cannot arise + // from the CLI — `inMigrateScope` is defined whenever dependencies exist — and + // resolving it the other way would be the unsafe direction: excluding an import + // costs nothing, while keeping one risks DDL against the publisher's database. + const importedOutOfScope: string[] = []; + const declaredHere = (obj: T): boolean => { + if (imported === undefined) return true; + const qualified = qualifiedDbName(obj); + const fqn = built.provenance.get(qualified); + if (fqn === undefined || !imported(fqn)) return true; + if (inScope?.(fqn) === true) return true; + importedOutOfScope.push(qualified); + return false; + }; + const remainder: SchemaSnapshot = { + ...built.snapshot, + tables: built.snapshot.tables.filter(declaredHere), + views: built.snapshot.views.filter(declaredHere), + }; - // Computed from `built.snapshot` — the UNSCOPED side — deliberately, and before - // the filter below runs. Deriving it from the survivors would reproduce exactly - // the defect this exists to close. - const declared = declaredSchemasOf(built.snapshot); + // Computed from the REMAINDER — everything this model actually declares — and + // before the scope filter below. Deriving it from the scope's survivors instead + // would reproduce exactly the defect the module header describes; deriving it from + // the unpartitioned side would pin the publisher's schema. Both halves matter. + const declared = declaredSchemasOf(remainder); - const outOfScope: string[] = []; + // PASS 2 — the per-command scope, unchanged. + const scopeOutOfScope: string[] = []; const governed = (obj: T): boolean => { + if (inScope === undefined) return true; const qualified = qualifiedDbName(obj); const fqn = built.provenance.get(qualified); if (fqn === undefined || inScope(fqn)) return true; - outOfScope.push(qualified); + scopeOutOfScope.push(qualified); return false; }; return { snapshot: { - ...built.snapshot, - tables: built.snapshot.tables.filter(governed), - views: built.snapshot.views.filter(governed), + ...remainder, + tables: remainder.tables.filter(governed), + views: remainder.views.filter(governed), }, - outOfScope, + // ONE suppression set: both exclusions must reach `unmanagedNames`, or the + // objects they removed from `expected` come back as proposed drops. + outOfScope: [...importedOutOfScope, ...scopeOutOfScope], ...(declared.length > 0 ? { declaredSchemas: declared } : {}), + ...(imported !== undefined ? { importedOutOfScope } : {}), }; } diff --git a/server/typescript/packages/migrate-ts/src/snapshot/plan.ts b/server/typescript/packages/migrate-ts/src/snapshot/plan.ts index 01ad1787d..34ff6d084 100644 --- a/server/typescript/packages/migrate-ts/src/snapshot/plan.ts +++ b/server/typescript/packages/migrate-ts/src/snapshot/plan.ts @@ -22,6 +22,15 @@ export interface PlanOfflineArgs extends Pick { expect(provenance.get("public.matches")).toBe("arena::Match"); }); }); + +/** + * FR-023 §11.1 item 2 — an EXCLUDED IMPORT leaves the expected side BEFORE + * `declaredSchemas` is computed. + * + * `migrate.scope` and an import exclude for opposite reasons, and the schema pin is + * where the difference bites. A `migrate.scope` narrows a model that DID declare into + * the schema, so the schema stays pinned ("a scope narrows objects, never schemas"). + * A dependency's node was never declared by this consumer at all: the publisher owns + * that schema, and pinning it would make every table the publisher did NOT export — + * tables this consumer has never heard of — a proposed DROP. + * + * So the import partition runs FIRST, and `declaredSchemas` is derived from what is + * left. Everything the *scope* excluded is still in that remainder, so the older rule + * is untouched. + */ +function scopeTable(name: string, schema: string): TableDescriptor { + return { + name, + schema, + columns: [ + { name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false, identity: "increment" }, + ], + indexes: [], + foreignKeys: [], + primaryKey: ["id"], + checks: [], + }; +} + +/** + * A consumer's own `app.orders`, a dependency's `public.customers`, and a co-owner's + * `reporting.matches` — one object per exclusion REASON, so the two can be told apart. + */ +function builtWithImport(): ExpectedSchemaWithProvenance { + return { + snapshot: { + tables: [ + scopeTable("orders", "app"), + scopeTable("customers", "public"), + scopeTable("matches", "reporting"), + ], + views: [], + }, + provenance: new Map([ + ["app.orders", "app::Order"], + ["public.customers", "acme::common::Customer"], + ["reporting.matches", "arena::Match"], + ]), + }; +} + +const ownOnly = (fqn: string): boolean => fqn === "app::Order"; +const importedCustomer = (fqn: string): boolean => fqn === "acme::common::Customer"; + +describe("scopeExpectedSchema — excluded imports (FR-023)", () => { + test("an excluded import leaves the expected side and never widens the schema scope", () => { + const scoped = scopeExpectedSchema(builtWithImport(), ownOnly, { imported: importedCustomer }); + + expect(scoped.snapshot.tables.map((t) => t.name)).toEqual(["orders"]); + expect(scoped.outOfScope.sort()).toEqual(["public.customers", "reporting.matches"]); + + // THE assertion this task exists for. `public` is the PUBLISHER's schema — this + // consumer never declared into it — so it must be gone. `reporting` was excluded + // by the SCOPE, over an object this model does declare, so it stays pinned. + expect(scoped.declaredSchemas).toEqual(["app", "reporting"]); + expect(scoped.declaredSchemas).not.toContain("public"); + }); + + test("the hazard is real: without the import predicate the publisher's schema stays pinned", () => { + // Today's behaviour, kept as the counter-assertion. `public` is pinned, so every + // table the publisher did not export becomes a drop candidate in it. + const scoped = scopeExpectedSchema(builtWithImport(), ownOnly); + expect(scoped.declaredSchemas).toEqual(["app", "public", "reporting"]); + }); + + test("an import the scope NAMES is governed like any other object", () => { + const scoped = scopeExpectedSchema(builtWithImport(), () => true, { imported: importedCustomer }); + + // `inScope` admits it, so the partition does not take it: the consumer opted in + // (`migrate.scope` naming the package) and owns those tables. + expect(scoped.snapshot.tables.map((t) => t.name)).toEqual(["orders", "customers", "matches"]); + expect(scoped.outOfScope).toEqual([]); + expect(scoped.declaredSchemas).toEqual(["app", "public", "reporting"]); + expect(scoped.importedOutOfScope).toEqual([]); + }); + + test("the imported half is reported separately, so the CLI can word its note", () => { + const scoped = scopeExpectedSchema(builtWithImport(), ownOnly, { imported: importedCustomer }); + // `outOfScope` stays the FULL suppression set (it feeds `unmanagedNames`); this is + // only the subset a dependency contributed. + expect(scoped.importedOutOfScope).toEqual(["public.customers"]); + }); + + test("a table with no provenance is never treated as imported", () => { + const built = builtWithImport(); + const withStranger: ExpectedSchemaWithProvenance = { + snapshot: { ...built.snapshot, tables: [...built.snapshot.tables, scopeTable("stranger", "public")] }, + provenance: built.provenance, + }; + const scoped = scopeExpectedSchema(withStranger, () => true, { imported: () => true }); + // Scope never guesses: an unknown FQN was not proven to be anyone else's. + expect(scoped.snapshot.tables.map((t) => t.name)).toContain("stranger"); + expect(scoped.importedOutOfScope).toEqual([]); + }); + + test("an undefined scope still partitions imports (defensive — Task 8 never produces it)", () => { + // `inMigrateScope` is defined whenever dependencies exist, so this cannot arise + // from the CLI. It resolves fail-SAFE: excluding an import risks nothing, while + // keeping one risks proposing DROP against the publisher's database. + const scoped = scopeExpectedSchema(builtWithImport(), undefined, { imported: importedCustomer }); + expect(scoped.snapshot.tables.map((t) => t.name)).toEqual(["orders", "matches"]); + expect(scoped.outOfScope).toEqual(["public.customers"]); + expect(scoped.declaredSchemas).toEqual(["app", "reporting"]); + }); + + test("no opts ⇒ byte-identical: the untouched path is the SAME object", () => { + const built = builtWithImport(); + const scoped = scopeExpectedSchema(built, undefined); + expect(scoped.snapshot).toBe(built.snapshot); + expect(scoped.outOfScope).toEqual([]); + expect(scoped.declaredSchemas).toBeUndefined(); + expect(scoped.importedOutOfScope).toBeUndefined(); + }); +}); From 27749df472b27508c96876b483367276ee82ad7c Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 22:06:40 -0400 Subject: [PATCH 28/62] docs(fr-023): the requirements task's helper reference, and both its call sites MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-flight check before the ledger task: its read-first line cited a line number off by over two hundred, and described coverableEntities as a single call site when it has two. Gating one and not the other would leave imported entities still counted, with every test green — the third time this plan has named one call site where there are several. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 6a564acbc..9dde72400 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -882,7 +882,7 @@ feat(migrate): imported tables leave both sides of the schema diff, and never wi - Modify: `server/typescript/packages/cli/src/commands/verify.ts` (pass `{ coverable: collection.inScope }`) - Test: `cli/test/verify-requirements-imported.test.ts` (new) -**Read first:** DESIGN §2.7 row "requirements ledger" (as amended: the denominator excludes excluded imports; a local `@implementedBy` may still name an imported member — resolution runs over the whole loaded model). Pre-flight ruling T20: NO `governs` filter on `verify --templates` (dead — exports never carry `template.*`); only the ledger changes. `requirement-check.ts:77 coverableEntities`. +**Read first:** DESIGN §2.7 row "requirements ledger" (as amended: the denominator excludes excluded imports; a local `@implementedBy` may still name an imported member — resolution runs over the whole loaded model). Pre-flight ruling T20: NO `governs` filter on `verify --templates` (dead — exports never carry `template.*`); only the ledger changes. `requirement-check.ts` — **corrected 2026-09-11 by pre-flight check**: `scanRequirements` is at **:254** (it takes NO options bag today, so `opts?: { coverable? }` is cleanly additive) and `coverableEntities` at **:311**, NOT :77. **And `coverableEntities` has TWO call sites — :541 and :599 — not one.** Apply the predicate INSIDE `coverableEntities` so both inherit it; filtering at one call site silently leaves the other counting imported entities. A diff that gates only one call site is an Important finding.. - [ ] **Step 1: Failing test.** Consumer = `APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1` plus a `metaobjects/meta.req.json` with one `requirement.functional` `@implementedBy: "app::Order"`. `meta verify` (default run): the requirements summary counts 1 coverable entity, 0 uncovered; NO `WARN_REQUIREMENT_OBJECT_UNCLAIMED` for `acme::common::Customer`; the summary line says `, 1 from dependencies`. With `"scope": { "include": ["app::**", "acme::common::**"] }`: `Customer` IS reported unclaimed. Run → FAIL. From d4ce356cffe12ac49c2a6217a5046f6113835395 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 22:08:02 -0400 Subject: [PATCH 29/62] docs(fr-023): remove a worked-out-loud false start from the schema task's step Its first step named a test file that does not exist, showed a builder call with the wrong signature, and then talked itself out of its own example mid-sentence before giving the right one. A step an implementer is told to follow verbatim cannot contain the author's discarded first attempt. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 9dde72400..88f9f52d3 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -851,7 +851,7 @@ feat(codegen-ts): imported nodes are not generated unless scope.include names th - Produces migrate-ts: `scopeExpectedSchema(built, inScope, opts?)` — when `opts.imported` is given, first partition `built.snapshot`'s tables and views by `imported(fqn) && !inScope(fqn)` (via `built.provenance`); the excluded half goes to `outOfScope`; `declared = declaredSchemasOf()`; then the existing filter over the remainder. Without `opts`, byte-identical. An `inScope` of `undefined` with `opts.imported` given still performs the import partition (Task 8 guarantees `inMigrateScope` is defined whenever dependencies exist, so this branch is defensive). - Produces cli: `dependencyNote(command, names)` = `` `meta ${command} — ${names.length} object(s) from dependencies not governed here (name the package in migrate.scope to own them): ${names.join(", ")}` ``. -- [ ] **Step 1: Failing migrate-ts unit test** (`migrate-ts/test/scope.test.ts`, extend): build an `ExpectedSchemaWithProvenance` by hand with `app.orders` (fqn `app::Order`) and `public.customers` (fqn `acme::common::Customer`); `scopeExpectedSchema(built, () => true, { imported: (f) => f === "acme::common::Customer" })` — hmm, `inScope` returns true for the import here, so nothing is excluded; use `inScope: (f) => f === "app::Order"`: `snapshot.tables` = `[app.orders]`, `outOfScope` = `["public.customers"]`, `declaredSchemas` = `["app"]` (NOT `["app","public"]`). And with `inScope: () => true`: `declaredSchemas` = `["app","public"]`, `outOfScope` = `[]`. Run → FAIL. +- [ ] **Step 1: Failing migrate-ts unit test** (`migrate-ts/test/expected-schema-scope.test.ts`, extend — **corrected 2026-09-11**: `migrate-ts/test/scope.test.ts` does NOT exist): build an `ExpectedSchemaWithProvenance` by hand with `app.orders` (fqn `app::Order`) and `public.customers` (fqn `acme::common::Customer`); **corrected 2026-09-11** — this step previously showed `scopeExpectedSchema(built, () => true, {...})` and then talked itself out of it mid-sentence (“hmm, `inScope` returns true for the import here”). The wrong form is removed; use `inScope: (f) => f === "app::Order"`: `snapshot.tables` = `[app.orders]`, `outOfScope` = `["public.customers"]`, `declaredSchemas` = `["app"]` (NOT `["app","public"]`). And with `inScope: () => true`: `declaredSchemas` = `["app","public"]`, `outOfScope` = `[]`. Run → FAIL. - [ ] **Step 2: Failing sqlite CLI test** (`cli/test/migrate-imported-nodes.test.ts`): consumer = `APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1`, `migrate.dialect: "sqlite"`. (a) `runOfflineGenerate` proposes `CREATE TABLE orders` and NOT `customers`; stderr contains `1 object(s) from dependencies not governed here`. (b) Baseline with both tables present; re-run: no `DROP TABLE customers`. (c) `"migrate": { "scope": ["app::**", "acme::common::**"] }`: `CREATE TABLE customers` IS proposed and the note is absent. Run → FAIL. From 6692fc65da02ed0f79f119ab78043aa1803e6b7a Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 22:09:39 -0400 Subject: [PATCH 30/62] docs(fr-023): the dependency note lands on stdout, not stderr The schema task's second step told an implementer to assert the note on stderr. Under the default text format it goes through the info logger, which writes to standard output; only the non-text branch uses the error stream. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 88f9f52d3..3f76cf8dc 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -853,7 +853,7 @@ feat(codegen-ts): imported nodes are not generated unless scope.include names th - [ ] **Step 1: Failing migrate-ts unit test** (`migrate-ts/test/expected-schema-scope.test.ts`, extend — **corrected 2026-09-11**: `migrate-ts/test/scope.test.ts` does NOT exist): build an `ExpectedSchemaWithProvenance` by hand with `app.orders` (fqn `app::Order`) and `public.customers` (fqn `acme::common::Customer`); **corrected 2026-09-11** — this step previously showed `scopeExpectedSchema(built, () => true, {...})` and then talked itself out of it mid-sentence (“hmm, `inScope` returns true for the import here”). The wrong form is removed; use `inScope: (f) => f === "app::Order"`: `snapshot.tables` = `[app.orders]`, `outOfScope` = `["public.customers"]`, `declaredSchemas` = `["app"]` (NOT `["app","public"]`). And with `inScope: () => true`: `declaredSchemas` = `["app","public"]`, `outOfScope` = `[]`. Run → FAIL. -- [ ] **Step 2: Failing sqlite CLI test** (`cli/test/migrate-imported-nodes.test.ts`): consumer = `APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1`, `migrate.dialect: "sqlite"`. (a) `runOfflineGenerate` proposes `CREATE TABLE orders` and NOT `customers`; stderr contains `1 object(s) from dependencies not governed here`. (b) Baseline with both tables present; re-run: no `DROP TABLE customers`. (c) `"migrate": { "scope": ["app::**", "acme::common::**"] }`: `CREATE TABLE customers` IS proposed and the note is absent. Run → FAIL. +- [ ] **Step 2: Failing sqlite CLI test** (`cli/test/migrate-imported-nodes.test.ts`): consumer = `APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1`, `migrate.dialect: "sqlite"`. (a) `runOfflineGenerate` proposes `CREATE TABLE orders` and NOT `customers`; stdout — **corrected 2026-09-11**: under the default `fmt: "text"` the note goes through `log.info` -> `console.log`, i.e. standard output; only the non-text branch uses the error stream (verified at `migrate.ts:215-224` and `lib/log.ts:1-4`) contains `1 object(s) from dependencies not governed here`. (b) Baseline with both tables present; re-run: no `DROP TABLE customers`. (c) `"migrate": { "scope": ["app::**", "acme::common::**"] }`: `CREATE TABLE customers` IS proposed and the note is absent. Run → FAIL. - [ ] **Step 3: Failing Postgres test** (`integration-tests/test/dependency-imported-table-pg.test.ts`) — the one that can see schemas: load a consumer model in-memory (`InMemoryStringSource`) consisting of the v1 artifact text PLUS a local root `{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"Order","children":[{"source.rdb":{"@table":"orders","@schema":"app"}},{"field.long":{"name":"id"}},{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}`. On a fresh Postgres run raw SQL: `CREATE TABLE public.customers (id bigint primary key, email varchar(120))` (the imported table, as the publisher created it) and `CREATE TABLE public.invoices (id bigint primary key)` (a publisher table the artifact does NOT export). `built = buildExpectedSchemaWithProvenance(root, "postgres")`; `scoped = scopeExpectedSchema(built, (f) => f === "app::Order", { imported: (f) => f.startsWith("acme::common::") })`; `actual = introspectPostgres(...)`; `ops = diff({ ...scopedDiffInputs(scoped, collectUnmanagedNames(root)), actual, dialect: "postgres" })`. Assert: `ops` contains a create for `app.orders`; contains NO drop of `public.invoices`; contains NO create/drop/alter touching `public.customers`. Counter-assert (the legacy own case): `scopeExpectedSchema(built, () => true, { imported })` → `ops` contains a DROP of `public.invoices` (the consumer took `public` by including the package — correct and visible) and nothing for `public.customers` (it matches). Run `cd server/typescript/packages/integration-tests && bun test test/dependency-imported-table-pg.test.ts` → FAIL (the first form proposes `DROP TABLE public.invoices` today). From 4afbd7efdb1b6669b256ae56f7788e8ee831af07 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 22:21:45 -0400 Subject: [PATCH 31/62] =?UTF-8?q?feat(cli):=20the=20requirements=20ledger?= =?UTF-8?q?=20does=20not=20count=20imported=20entities=20the=20project=20d?= =?UTF-8?q?oes=20not=20include=20(FR-023=20=C2=A711)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Collection.inScope narrows coverableEntities() so an entity loaded from a dependency's synced snapshot only enters the object-coverage denominator when the consumer's own scope.include names its package literally. Applied INSIDE coverableEntities so both call sites (the gate and the summary) inherit it — scanRequirements(root, opts?: { coverable }) threads the predicate through RequirementScan, additively (no options bag = unchanged behaviour). verify.ts passes { coverable: collection.inScope }. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../packages/cli/src/commands/verify.ts | 2 +- .../packages/cli/src/lib/requirement-check.ts | 40 +++- .../test/verify-requirements-imported.test.ts | 180 ++++++++++++++++++ 3 files changed, 214 insertions(+), 8 deletions(-) create mode 100644 server/typescript/packages/cli/test/verify-requirements-imported.test.ts diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index 17e2922d6..5d7c096ef 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -595,7 +595,7 @@ export async function verifyCommand( // ONE scan for all three passes. The gate and the summary each used to walk the // model AND resolve every @implementedBy claim for themselves — the resolution // being the expensive half — and the lint added a third walk on top. - const scan = scanRequirements(root); + const scan = scanRequirements(root, { coverable: collection.inScope }); const diags = [...checkRequirements(root, scan)]; // Printed on EVERY run, clean or not — a gate that says nothing when it diff --git a/server/typescript/packages/cli/src/lib/requirement-check.ts b/server/typescript/packages/cli/src/lib/requirement-check.ts index 90202378b..6c396a2a9 100644 --- a/server/typescript/packages/cli/src/lib/requirement-check.ts +++ b/server/typescript/packages/cli/src/lib/requirement-check.ts @@ -249,11 +249,25 @@ export function collectRequirements(root: MetaData): MetaRequirement[] { export interface RequirementScan { readonly addressed: readonly AddressedRequirement[]; readonly claimedObjects: ReadonlySet; + /** FR-023 — narrows which entities `coverableEntities` counts. Undefined + * means "every non-abstract entity is coverable", exactly as before this + * option existed: a project with no dependencies gets no change at all. + * Threaded through so BOTH `coverableEntities` call sites (the gate and the + * summary) inherit the same narrowing — see `coverableEntities`. */ + readonly coverable?: (fqn: string) => boolean; } -export function scanRequirements(root: MetaData): RequirementScan { +export function scanRequirements( + root: MetaData, + opts?: { coverable?: (fqn: string) => boolean }, +): RequirementScan { const addressed = collectAddressedRequirements(root); - return { addressed, claimedObjects: claimedObjectKeys(root, addressed.map((r) => r.node)) }; + return { + addressed, + claimedObjects: claimedObjectKeys(root, addressed.map((r) => r.node)), + // `exactOptionalPropertyTypes` — an omitted key, never an explicit `undefined`. + ...(opts?.coverable !== undefined ? { coverable: opts.coverable } : {}), + }; } /** @@ -307,10 +321,22 @@ function claimedObjectKeys(root: MetaData, reqs: MetaRequirement[]): Set * An ABSTRACT entity is shape, not data: there is no table and no rows, so * demanding a capability claim for it is the same category error as demanding * one for an object.value. It is exempt for the same reason. + * + * FR-023: an entity loaded from a dependency's synced snapshot is load-only — + * the publisher owns it, and this project never declared it — unless the + * project's own `scope.include` names the dependency's package literally + * (`Collection.inScope`, threaded in as `coverable`). Demanding a capability + * claim for an import nobody here opted into owning would report a project as + * failing to claim entities it does not own. `coverable` is applied INSIDE + * this function, once, so every call site inherits it — the alternative + * (filtering at a call site) is exactly the shape of bug this task exists to + * avoid: the other call site would keep counting imports. */ -function coverableEntities(root: MetaData): MetaData[] { +function coverableEntities(root: MetaData, coverable?: (fqn: string) => boolean): MetaData[] { return root.children().filter( - (n) => n.type === TYPE_OBJECT && n.subType === OBJECT_SUBTYPE_ENTITY && !n.isAbstract, + (n) => + n.type === TYPE_OBJECT && n.subType === OBJECT_SUBTYPE_ENTITY && !n.isAbstract && + (coverable === undefined || coverable(n.resolutionKey())), ); } @@ -324,7 +350,7 @@ function coverableEntities(root: MetaData): MetaData[] { */ export function checkRequirements(root: MetaData, scan: RequirementScan = scanRequirements(root)): Diagnostic[] { const out: Diagnostic[] = []; - const { addressed, claimedObjects } = scan; + const { addressed, claimedObjects, coverable } = scan; if (addressed.length === 0) return out; // opt-in by declaration — no requirements, nothing to say for (const { node: req, path: reqPath } of addressed) { @@ -538,7 +564,7 @@ export function checkRequirements(root: MetaData, scan: RequirementScan = scanRe // // So a green run means "every entity is claimed by something", not "every node is // described". The stronger reading would be false. - for (const ent of coverableEntities(root)) { + for (const ent of coverableEntities(root, coverable)) { const key = ent.resolutionKey(); if (!claimedObjects.has(key)) { out.push({ @@ -596,7 +622,7 @@ export function summariseRequirements( // summary cannot disagree with the diagnostics printed beneath it — previously // the same helper, now literally the same result. const claimed = scan.claimedObjects; - for (const ent of coverableEntities(root)) { + for (const ent of coverableEntities(root, scan.coverable)) { summary.entitiesTotal++; if (claimed.has(ent.resolutionKey())) summary.entitiesClaimed++; } diff --git a/server/typescript/packages/cli/test/verify-requirements-imported.test.ts b/server/typescript/packages/cli/test/verify-requirements-imported.test.ts new file mode 100644 index 000000000..e66c5fc82 --- /dev/null +++ b/server/typescript/packages/cli/test/verify-requirements-imported.test.ts @@ -0,0 +1,180 @@ +// FR-023 §11 — the requirements ledger must not count imported entities the +// project does not own. +// +// The object-coverage pass (`coverableEntities` in `requirement-check.ts`) +// demands a capability claim for every non-abstract entity in the loaded +// model. A dependency's synced snapshot loads its entities so the consumer's +// OWN model can resolve against them (`extends`, relationships) — but the +// publisher owns those entities, and the consumer never declared them. Left +// unfiltered, adding a dependency makes every one of its entities count +// against THIS project's ledger, reporting a consumer as failing to claim +// things it does not own. +// +// The fix narrows the denominator with `Collection.inScope` — the same +// default-exclusion-of-imports predicate `meta gen` and `meta migrate` +// already use — so an import counts only when the consumer's own +// `scope.include` names its package literally. +import { describe, test, expect, beforeEach, afterEach, afterAll } from "bun:test"; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { METAMODEL_VERSION } from "@metaobjectsdev/metadata"; +import { sha256Integrity } from "@metaobjectsdev/sdk"; +import { run } from "../src/index.js"; + +const DEP_NAME = "acme-common"; +const ARTIFACT_BASENAME = "acme-common.metaobjects.json"; + +/** The dependency's synced snapshot — one entity the publisher owns. */ +const SNAP = JSON.stringify({ + "metadata.root": { + children: [ + { + "object.entity": { + name: "Customer", + package: "acme::common", + children: [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "email" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +/** The consumer's own model: one entity it owns and claims. */ +const APP = JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Order", + children: [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +/** One requirement claiming the consumer's own entity by FQN. */ +const REQUIREMENTS = JSON.stringify({ + "metadata.root": { + children: [ + { + "requirement.functional": { + name: "orderRecord", + "@level": 4, + "@status": "live", + "@statement": "An order is a durable record.", + "@counterexample": "An order vanishes on restart.", + "@implementedBy": ["app::Order"], + }, + }, + ], + }, +}); + +const dirs: string[] = []; +afterAll(() => { for (const d of dirs) rmSync(d, { recursive: true, force: true }); }); + +/** A consumer of `acme-common`, optionally declaring `scope.include`. */ +function project(opts: { scopeInclude?: string[] } = {}): string { + const root = mkdtempSync(join(tmpdir(), "vreq-imported-")); + dirs.push(root); + + mkdirSync(join(root, "metaobjects"), { recursive: true }); + writeFileSync(join(root, "metaobjects", "meta.app.json"), APP, "utf8"); + writeFileSync(join(root, "metaobjects", "meta.req.json"), REQUIREMENTS, "utf8"); + + const depDir = join(root, ".metaobjects", "deps", DEP_NAME); + mkdirSync(depDir, { recursive: true }); + writeFileSync(join(depDir, ARTIFACT_BASENAME), SNAP, "utf8"); + + const config: Record = { + schema_version: 1, + sources: [], + dependencies: [{ name: DEP_NAME, path: "../acme-common/metaobjects" }], + }; + if (opts.scopeInclude) config.scope = { include: opts.scopeInclude }; + writeFileSync(join(root, ".metaobjects", "config.json"), JSON.stringify(config), "utf8"); + + writeFileSync( + join(root, ".metaobjects", "deps.lock.json"), + JSON.stringify({ + schema_version: 1, + dependencies: { + [DEP_NAME]: { + version: "1.0.0", + metamodelVersion: METAMODEL_VERSION, + resolvedFrom: { path: "../acme-common/metaobjects" }, + artifact: ARTIFACT_BASENAME, + integrity: sha256Integrity(SNAP), + packages: ["acme::common"], + nodes: ["acme::common::Customer"], + }, + }, + }), + "utf8", + ); + return root; +} + +let out: string[]; +let err: string[]; +let origLog: typeof console.log; +let origErr: typeof console.error; + +beforeEach(() => { + out = []; + err = []; + origLog = console.log; + origErr = console.error; + console.log = (...a: unknown[]) => { out.push(a.map(String).join(" ")); }; + console.error = (...a: unknown[]) => { err.push(a.map(String).join(" ")); }; +}); +afterEach(() => { + console.log = origLog; + console.error = origErr; +}); + +describe("meta verify — the requirements ledger excludes out-of-scope imports (FR-023)", () => { + test("an unscoped consumer's ledger counts only its own entity — the import is invisible", async () => { + const root = project(); + const exit = await run(["verify", "--format", "text", "--cwd", root]); + expect(exit).toBe(0); + + const all = [...out, ...err].join("\n"); + // The denominator is 1 (Order only) and it is fully claimed — Customer, an + // import this project never opted into owning, does not enter the count. + expect(all).toContain("1/1 entities claimed"); + // And it is never named as an unclaimed entity, even though nothing here + // claims it either — it is not COVERABLE, so it is not counted at all. + expect(all).not.toContain("WARN_REQUIREMENT_OBJECT_UNCLAIMED"); + expect(all).not.toContain("acme::common::Customer"); + // The dependency count on the summary line is unaffected by this change — + // it still reports the one declared dependency. + expect(all).toContain(", 1 from dependencies."); + }); + + test("naming the dependency's package in scope.include makes its entity coverable — and unclaimed", async () => { + const root = project({ scopeInclude: ["app::**", "acme::common::**"] }); + const exit = await run(["verify", "--format", "text", "--cwd", root]); + // Coverage is advisory (WARN), so an unclaimed import still exits 0 — + // exactly as an unclaimed entity of the consumer's own always has. + expect(exit).toBe(0); + + const all = [...out, ...err].join("\n"); + expect(all).toContain("1/2 entities claimed"); + expect(all).toContain("WARN_REQUIREMENT_OBJECT_UNCLAIMED"); + expect(all).toContain("acme::common::Customer"); + expect(all).toContain(", 1 from dependencies."); + }); +}); From 3f804d7d5ddc1bba4799ee15d89b3e14d1ff0d2b Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 22:23:40 -0400 Subject: [PATCH 32/62] docs(fr-023): the ledger task's regression test lives under the unit directory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Its run command pointed at the package test root. That is the fourth wrong test path in this plan, and the second where the file was one directory deeper than the reference assumed — worth resolving such paths with a find rather than assuming the conventional location. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 3f76cf8dc..c6bd5e683 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -886,7 +886,7 @@ feat(migrate): imported tables leave both sides of the schema diff, and never wi - [ ] **Step 1: Failing test.** Consumer = `APP` + `SNAP` + `CONFIG_REF` + `LOCK_V1` plus a `metaobjects/meta.req.json` with one `requirement.functional` `@implementedBy: "app::Order"`. `meta verify` (default run): the requirements summary counts 1 coverable entity, 0 uncovered; NO `WARN_REQUIREMENT_OBJECT_UNCLAIMED` for `acme::common::Customer`; the summary line says `, 1 from dependencies`. With `"scope": { "include": ["app::**", "acme::common::**"] }`: `Customer` IS reported unclaimed. Run → FAIL. -- [ ] **Step 2: Implement.** Run `cd server/typescript/packages/cli && bun test test/verify-requirements-imported.test.ts test/verify-requirements-e2e.test.ts test/requirement-check.test.ts` and `bun run --filter '@metaobjectsdev/cli' typecheck` → PASS. +- [ ] **Step 2: Implement.** Run `cd server/typescript/packages/cli && bun test test/verify-requirements-imported.test.ts test/verify-requirements-e2e.test.ts test/unit/requirement-check.test.ts` and `bun run --filter '@metaobjectsdev/cli' typecheck` → PASS. - [ ] **Step 3: Commit** ``` From a9c4fc14d93c96a456c6f5e668579b0d89bd5fb9 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 22:25:48 -0400 Subject: [PATCH 33/62] docs(fr-023): two more run commands name tests that exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by sweeping every test path across the remaining tasks at once rather than discovering them one dispatch at a time — four of the first ten defects in this plan were wrong test paths, so the sweep was worth doing as a batch. One was a second instance of a name already corrected elsewhere; the other now points at the suite that genuinely covers the templates subverb. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index c6bd5e683..ff1bf04d3 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -915,7 +915,7 @@ feat(cli): the requirements ledger does not count imported entities the project - [ ] **Step 1: Failing tests** (`codegen-ts/test/shared-model-file.test.ts`): a publisher project whose two source files declare `acme::common::{Address, Audited, Customer}` exactly as `acme-common-v1.json` does plus an internal `acme::common::Secret` entity and a `requirement.functional`; `sharedModelFile({ name: "acme-common", include: ["acme::common::**"], exclude: ["acme::common::Secret"], version: "1.0.0" })` through `runGen` emits (a) `acme-common.metaobjects.json` BYTE-EQUAL to `fixtures/dependency-conformance/artifacts/acme-common-v1.json` (fix the publisher fixture until it is — the pinned artifact is the contract), (b) a manifest equal to the Global Constraints example with the pinned v1 hash; (c) determinism: two runs, identical bytes; (d) closure: `include: ["acme::common::Customer"]` alone fails naming `acme::common::Customer → acme::common::Address` (Customer's `field.object` → Address in the fixture; read the artifact to confirm the real edge); (e) a `files` subset that omits the file declaring `Customer` exports two nodes; (f) a node carrying a consumer-provider attr (register a throwaway provider in the test) fails the core re-load with the message above. Run → FAIL. -- [ ] **Step 2: Implement.** Run `cd server/typescript/packages/codegen-ts && bun test test/shared-model-file.test.ts test/generator-registry.test.ts`, `cd server/typescript/packages/cli && bun test test/eject.test.ts test/gen.test.ts`, `bun run --filter '@metaobjectsdev/codegen-ts' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck` → PASS. Also `cd server/python && uv run --extra integration pytest tests/conformance/test_generator_registry_conformance.py -q` → PASS (the registry entry says `typescript` only). +- [ ] **Step 2: Implement.** Run `cd server/typescript/packages/codegen-ts && bun test test/shared-model-file.test.ts test/generator-registry.test.ts`, `cd server/typescript/packages/cli && bun test test/eject.test.ts test/gen-list.test.ts test/integration/gen-scope.test.ts`, `bun run --filter '@metaobjectsdev/codegen-ts' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck` → PASS. Also `cd server/python && uv run --extra integration pytest tests/conformance/test_generator_registry_conformance.py -q` → PASS (the registry entry says `typescript` only). - [ ] **Step 3: Commit** ``` @@ -973,7 +973,7 @@ feat(cli): meta deps sync and list — a committed snapshot pinned by a sha256 l - [ ] **Step 2: Implement.** Run ```bash -cd server/typescript/packages/cli && bun test test/deps-check.test.ts test/deps-sync.test.ts test/unit/args-verify.test.ts test/help-lists-every-flag.test.ts test/verify-templates.test.ts +cd server/typescript/packages/cli && bun test test/deps-check.test.ts test/deps-sync.test.ts test/unit/args-verify.test.ts test/help-lists-every-flag.test.ts test/integration/verify-subverbs.test.ts bun run --filter '@metaobjectsdev/cli' typecheck ``` Expected: PASS. From 79c6c256e7a1158af7b85c048f3817342b10474f Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 22:27:36 -0400 Subject: [PATCH 34/62] docs(fr-023): the overlay task's run list, and a coverage gap worth knowing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Its named test does not exist, and the discovery command the step suggests finds nothing because no test names the private walk it is about. The obvious substitute turned out to be a six-line placeholder with no tests in it, which would have added a vacuously passing file to a must-stay-green list. Now names the two suites that genuinely exercise the overlay-merge path, and records that nothing currently pins the ordering guarantee the task generalizes — so a regression there would be silent. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index ff1bf04d3..08acf19b3 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -1013,11 +1013,11 @@ revisit: DESIGN §11.4 (an app that loads one collection but must serve only par - [ ] **Step 2: Implement.** Run ```bash -cd server/typescript/packages/metadata && bun test test/declared-top-level-keys.test.ts test/loader-overlay-partition.test.ts +cd server/typescript/packages/metadata && bun test test/declared-top-level-keys.test.ts test/fr5c-merge-attribution.test.ts test/round-trip.test.ts cd server/typescript/packages/cli && bun test test/verify-overlay-lint.test.ts test/help-lists-every-flag.test.ts test/verify-requirements-e2e.test.ts bun run --filter '@metaobjectsdev/metadata' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck ``` -Expected: PASS (`loader-overlay-partition` or whichever existing test covers `_partitionOverlayLast` must stay green — find it with `grep -rl partitionOverlay metadata/test`). +Expected: PASS (`loader-overlay-partition` or whichever existing test covers `_partitionOverlayLast` must stay green — **corrected 2026-09-11 by pre-flight sweep** — that discovery command finds NOTHING: no test in `metadata/test` names the private partition walk at all. `overlay.test.ts` is a 6-line PLACEHOLDER with zero tests (and its own comment points at a `loader.test.ts` that does not exist), so naming it would have added a vacuously-passing file to this run list. The loader's overlay-merge path is really exercised by `fr5c-merge-attribution.test.ts` (contributor tracking through the merge phase) and `round-trip.test.ts` (multi-file order: common -> vehicle -> overlay); run those. **Be aware NO existing test pins the ordering guarantee itself** — this task generalizes that walk, so a regression in it would be SILENT. Consider adding an order-independence assertion (load an overlay-only source BEFORE its base, assert the merge still happens)). - [ ] **Step 3: Commit** ``` From 34eb5c3eeb6ce72f997f77fe8ec1702ea91696e5 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 22:29:36 -0400 Subject: [PATCH 35/62] docs(fr-023): qualify the unchanged-behaviour guarantee on the second axis MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The guarantee promised identical behaviour for any project with no dependencies, on every surface, without qualification. That holds for the dependency axis it was written about, but the design deliberately routes the requirements denominator through the same predicate that applies a project's own declared scope — so a project that declares one, with no dependencies at all, does see its denominator narrow. Intended behaviour, overreaching wording; qualified so a later reader does not read it as a contradiction. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 08acf19b3..97e3dbcb8 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -67,7 +67,7 @@ with `CONFIG_REF` = `{ "schema_version": 1, "sources": [], "dependencies": [{ "n - `Collection.inScope(fqn)` (codegen, `verify --codegen`, shared enums, `meta gen `, the ledger) = `matchesScope(fqn, scope) && (!imported(fqn) || explicitlyIncluded(packageOf(fqn), scope.include))`. - `Collection.inMigrateScope(fqn)` (`migrate`, `verify --db`, offline generate, replay) = `(declaredMigrateScope?.(fqn) ?? true) && (!imported(fqn) || explicitlyIncluded(packageOf(fqn), migrate.scope))`; **`undefined` iff the project declares no `migrate.scope` AND resolves no dependencies** (the byte-identical path). `Collection.declaredMigrateScope` is the user's `migrate.scope` alone (in lockstep with `migrateScopePatterns`) and is what the "scope matched nothing" refusal reads. - `packageOf(fqn)` is `packageOfResolutionKey` (`metadata/src/naming.ts`, Task 2) / `package_of_resolution_key`. -- A project with no dependencies resolves and behaves byte-identically to today on every surface. +- A project with no dependencies **and no declared `scope`** resolves and behaves identically to today on every surface. **Qualified 2026-09-11**: the unqualified form overreached on a second axis. DESIGN §11.1 deliberately lists the requirements ledger's denominator among the surfaces `inScope` governs, and `inScope` is `matchesScope(fqn, scope) && (...)` — so a project that declares `scope.include` for its own codegen, with zero dependencies, now also sees its ledger denominator narrow to that declared scope. That is intended (coverable = your own declared responsibility), but it is a real change for such a project and belongs in the CHANGELOG. **Two-sided schema exclusion (DESIGN §11.1 item 2).** `scopeExpectedSchema(built, inScope, { imported })` removes an object with `imported(fqn) && !inScope(fqn)` from the expected side BEFORE `declaredSchemas` is computed, adds its qualified name to `outOfScope` (so `scopedDiffInputs` suppresses it on the actual side), and then runs the existing user-scope filter over the remainder unchanged. Importing a table-backed entity must NEVER make `migrate` or `verify --db` propose a `DROP` of the publisher's other tables, and must never propose creating the imported table — unless `migrate.scope` names the package, in which case the consumer governs it (the legacy own-the-shared-model case). From d6ff3d277ddcc1063eb958eb4b873f8a6c4005df Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 23:17:09 -0400 Subject: [PATCH 36/62] feat(codegen-ts): sharedModelFile() emits a publisher's flattened shared-model artifact and manifest (FR-023) Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../registry.json | 5 + .../packages/cli/src/commands/gen.ts | 4 + .../codegen-ts/src/generator-registry.ts | 11 + .../packages/codegen-ts/src/generator.ts | 22 +- .../codegen-ts/src/generators/index.ts | 2 + .../src/generators/shared-model-file.ts | Bin 0 -> 13024 bytes .../packages/codegen-ts/src/runner.ts | 23 +- .../test/generator-registry.test.ts | 1 + .../codegen-ts/test/shared-model-file.test.ts | 290 ++++++++++++++++++ 9 files changed, 356 insertions(+), 2 deletions(-) create mode 100644 server/typescript/packages/codegen-ts/src/generators/shared-model-file.ts create mode 100644 server/typescript/packages/codegen-ts/test/shared-model-file.test.ts diff --git a/fixtures/generator-registry-conformance/registry.json b/fixtures/generator-registry-conformance/registry.json index 9825251aa..c887cedf8 100644 --- a/fixtures/generator-registry-conformance/registry.json +++ b/fixtures/generator-registry-conformance/registry.json @@ -133,6 +133,11 @@ "tier": "native", "ports": ["kotlin"] }, + "shared-model": { + "concept": "FR-023: a publisher's flattened shared-model artifact + manifest for a consumer's `meta deps sync`.", + "tier": "native", + "ports": ["typescript"] + }, "docs": { "concept": "Neutral per-entity / per-template Markdown documentation pages.", "tier": "neutral", diff --git a/server/typescript/packages/cli/src/commands/gen.ts b/server/typescript/packages/cli/src/commands/gen.ts index d38100e2f..f94279b4f 100644 --- a/server/typescript/packages/cli/src/commands/gen.ts +++ b/server/typescript/packages/cli/src/commands/gen.ts @@ -162,6 +162,10 @@ export async function genCommand(args: string[], cwd: string, fmt: OutputFormat // passed: an unconfigured project's predicate admits everything, so this // is a no-op for the common case, not a behavior change. scope: genCollection.inScope, + // FR-023 §4.3 — the collection's own source files (never a dependency's + // snapshot artifact), so sharedModelFile() defaults its `files` selection + // to exactly what this project itself declares. + sourceFiles: genCollection.ownFiles, ...(cliConfig.entities.length > 0 ? { entityFilter: cliConfig.entities } : {}), }); } catch (err) { diff --git a/server/typescript/packages/codegen-ts/src/generator-registry.ts b/server/typescript/packages/codegen-ts/src/generator-registry.ts index 39595fbfe..0b1959c2e 100644 --- a/server/typescript/packages/codegen-ts/src/generator-registry.ts +++ b/server/typescript/packages/codegen-ts/src/generator-registry.ts @@ -39,6 +39,7 @@ import { docsFile, templateGenerator, traceHelperFile, + sharedModelFile, } from "./generators/index.js"; export type GeneratorTier = "native" | "neutral"; @@ -180,6 +181,16 @@ export const generatorRegistry: Record = { factory: () => traceHelperFile(), options: "outDir?, target?", }, + "shared-model": { + name: "shared-model", + description: "FR-023: a publisher's flattened shared-model artifact + manifest for a consumer's `meta deps sync`.", + tier: "native", + // `name`/`include` are required at run time (an empty include matches + // everything, so a placeholder here constructs without throwing — real use + // always supplies both, same as `template`'s templatePrimitive() above). + factory: () => sharedModelFile({ name: "shared-model", include: [] }), + options: "name, include, exclude?, files?, version?, target?", + }, // ----- Tier-2 neutral (owned by the `meta docs` engine — D1 / ADR-0020) --- docs: { diff --git a/server/typescript/packages/codegen-ts/src/generator.ts b/server/typescript/packages/codegen-ts/src/generator.ts index b7e35347f..954865736 100644 --- a/server/typescript/packages/codegen-ts/src/generator.ts +++ b/server/typescript/packages/codegen-ts/src/generator.ts @@ -1,4 +1,4 @@ -import type { MetaObject, MetaRoot } from "@metaobjectsdev/metadata"; +import type { MetaObject, MetaRoot, TypeRegistry } from "@metaobjectsdev/metadata"; import type { RenderContext } from "./render-context.js"; import type { ResolvedGenConfig } from "./metaobjects-config.js"; import type { OrphanPolicy } from "./reconcile-orphans.js"; @@ -45,6 +45,26 @@ export interface GenContext { * in a sub-directory. Undefined only when the runner was driven * programmatically without an explicit projectRoot. */ projectRoot?: string; + /** + * FR-023 §4.3 — the run's composed registry (core providers plus whatever this + * project's `metaobjects.config.ts` `providers` adds), filled by the runner. The + * SAME vocabulary `opts.metadata` was loaded with. `sharedModelFile()`'s + * standalone re-loads of a `files` subset use this so a publisher project's own + * consumer-supplied vocabulary is honoured identically to the main load. Optional + * on the type so tests and custom callers don't need a placeholder; always + * present at run time when invoked via `runGen()`. + */ + registry?: TypeRegistry; + /** + * FR-023 §4.3 — the collection's own source files (never a dependency's + * artifact), filled by the runner from `RunGenOpts.sourceFiles`. Distinct from + * `loadedRoot`, which is already fully loaded and merged: `sharedModelFile()` + * needs the raw file LIST so it can re-load a `files:`-narrowed subset of it + * standalone. Undefined when the caller never supplied `RunGenOpts.sourceFiles` + * (a generator relying on it should default sensibly, as `sharedModelFile()`'s + * own `files` option does). + */ + sourceFiles?: readonly string[]; warn: (msg: string) => void; } diff --git a/server/typescript/packages/codegen-ts/src/generators/index.ts b/server/typescript/packages/codegen-ts/src/generators/index.ts index 6c7e73d08..c7ad6d339 100644 --- a/server/typescript/packages/codegen-ts/src/generators/index.ts +++ b/server/typescript/packages/codegen-ts/src/generators/index.ts @@ -35,6 +35,8 @@ export { type TemplateFormat, } from "./template-generator.js"; export { traceHelperFile, type TraceHelperOpts } from "./trace-helper-file.js"; +// FR-023 §4.3 — the publisher's flattened shared-model artifact + manifest generator. +export { sharedModelFile, type SharedModelFileOpts } from "./shared-model-file.js"; export type { EntityDocData, StorageFieldDoc, diff --git a/server/typescript/packages/codegen-ts/src/generators/shared-model-file.ts b/server/typescript/packages/codegen-ts/src/generators/shared-model-file.ts new file mode 100644 index 0000000000000000000000000000000000000000..e37673ad998c0af5fa6e5fca76407953c374f8f5 GIT binary patch literal 13024 zcmb_j>yq0>cFu1-MK4RW24xbXi{;8v9?Q%O$DZ}vGUVFXl2ikbY?81*0KsK0JDIBd zO8#%<9r7M|6hBG6?{qf+awx4U8`oMgfyU|6r@!-^OZV>WsiVvO^OvvG|NQq?{wwwC zzyFsiR$;E^r|DeBM^UW3?m(3*t+HwwN5x9#UleK)hh?b~J-37VYmQc7UPg;>R{Br& z_V7P7Y(7$1p8ld|rMii7T|i(|s3j!iVVUNtOjV&{z7ADUO$%MBbfKcMQ0Yyg*18Pm zVHr}SH@y>Wig2w}l+5C4uJ`n<{H=;vn(2N<-SR{&^KiWma}~(rK!tIx!}*=63SD$n zn9P-4MSylRlH2|s9ddEUtkNnJ;+pPKG*qxYg6iluf`>o8^&p(Rh))%8aG&AZ;OB* zFTyG=%_iqn0%d9RBCYb7Rx+xmf?0X%n@7Y%5p<1gh`)gj=6$Z0QBmf15K3DZS+jJV zr3H}U-qyAn4^KzxI-P~nDh_km)>^1Vi`B*Lf+!DBoD%EAT#*4A)Od+i8VHqX){pfy z@R$IV#Xu$a0T|@^Un_u*`{zFk)Se1Tz0QDS&7WS_fc1-_RwUN*B|auvpwl`7B;g&9 zOsKZ@7fzD_Ry`TH4$IjpN|u6D8UQAn*CKbKmOG18yzsVcfai!}yvElG6u@CG=yI{ds zQ_IU>su`vTlM>#st^jH{Y8qc_dS|G{qm$9WWOP`Iq>!G{ae)EYZ6iEN69vzvdGKJ- zhlR>8B2jQQj+<6dJXeb_iZLu)TPxL4y<9JJp6k44SK3us80Cev&sW^#JPPCJFWb`^ z92V~FSXk`PI&2`Q%flODyB~*l*waOtuY2{xjfFF;Tj$q$4uqw73EjatVvDNe`abGA ztIca)UFyDg6qiOi;qaOzipK3kyMyz~5n*&4!8_q*Nqpx}Ecyv)00pW>mLMK1LEeO$ z9Z$y1_XcaBx>$wSgBMEQW>D8`9XBA+&;@$cOKdYZXbzhLm^BV)7xlHTMp%dnzQOkB zZ+$vnIwf}({x3zE1nNt}X6|A3acN&P)CJB0Sw(PtXmqb;VBS*y5Ed)-pce2W)ggQa zv-~bA)6Sb_q`n1(6k||i4>k|yWE=k4{BBXSznMolRqd%?Qc&N<1c(Kz*3c(Pm3Ns| zr&!Y=B1*6Q;1U$i2kzwwm=ZX|ljG!6uhaZ)48lp4V_`-A*x_&SdtKDOce^G> zTGu)kyI$%ApM3(le3gZ>kKt0EFD~hl0>dBm9p79G4}KheKe`%^E{2!G$@wK89{g~8 za(KnrHy%&^d@;h)fBw_)<>+*DHsMftHW{5>oD3%;J~$mshNtI;qm!$jMwjE`^RwO) zXVLFkP?+oMJ^o}|ZW&m~ewonmw?5k={@UNB$xL79{IW{m{M@GRU4qd3J^-PB3+)2} z6&}Y)>wWkAy*jY~F5466#R4#nuJwhnXZssCZNb!GMU>+sdd&n*ZPOzc>G$dgU#4l? zUvOrS+%fkG71o(hnJJ%qv65iYW%08qvLL$D!AmGOl@*z=; zOs1xI*~0_oE|e$vB;xlB;fy4YX*6SYN=e|Wb(H7K{8mT~Ot30IcF4TqJ74{%b;dM? zsgnPMVYXBo3s(E8(?X-~I0zeOMsHbz;~z0uJ$Tzr9F2hf^7Yq!jM;nk%&yBMV|4=K zFoC>trPrZNRuGg6KmoP@lOfiZhVUc-lr;G#Ga&HQv%T%LjSqepoerJrL5~+eq9t~u ziaJMnt(A#wb%uV9fMUXQ9%@mMRy{5Mt3Y zrKX55q*$aQ@Hk-eQ?WtTq!`#Smibw-d}9?8R;-hAn*=G86r};2Wb{Ee!>^-_$aZq)_nA(_b9y%7FPW#vxsI*U=wx5PG?m6 z8MN92z!^Y+jI%1Us!3A;f!efeYb?@=I1ANPf9!sso#_2n<1yNuPBlQh(&g!j+*> z&kxkb8j}!SJQxIhD~&w<1=O3(svOWb6>c?a%<||OxtEbsz!}N|N}nJB4@j{Ad=-1! z$0{VcD37_Lp&UK$zG;;nHiZ{aC_R__`a5QG!CvrPy5OJJQHVmjERo%^OcMLiEq7-^ zvLhl6`eO|_DxJ`oRg3_fL~-K`HUdG{{uH34XK*r>9Im=I7MpT*yK@+AQf0z1%j6>? zR{#+eG#xa>7nFl-=%<+w%%TE)Gv$Yf^BXiPSe-anqb*|aE!1nGP8=5H7@=#QU-*JXvKN%Ota04sqGTu%{4GwnS`WV4gtL+gx(IxAPmhUk1`3*#wL zXA(7x8RtKrF^swzvj;n)3dI}LiD)z6OGF^?qE~+qOcbH9K1l;bo6!3Vfe6XRCsO6W zh9@0@vNx(->z35P6n`Sk+gbDx|4oC79Wj&DNc4YQ=P(Ah?PA@OUb1|Bi#8Z=e?!WV zpgXPWL#$j}U*6Rkh`5`y0v&)nBEpWjP%EM^OhOtkO1EN2wi$=4sAn1m=rfRi3Jt!3 z+e1KPLKnyLTWfG<9TP}upi_Au9UOq_tgeIrbypO<`_O_rxP0TyzIw+GCHR7om1pp6 zJQ>`{mp=AV7%10?*X=SM_*qq~Jm2@-?D9dk4NHs?tTj21ZIfJ}IUpLbSd>i#Ql)|5 z+>^dhzcj(5d1c{SZ!mkHMF|f^BPM9dcL{Gb#jxN9D2r+~1II5aZj&cpy%2QB4w#Te z=UJr4uax7LWrdAFCIuQ7htjkZWj3@BMJkNPWI00FMPACn z0?kY;Pg<*>&Jg7U39g3Ko*_ZfdlNzc1F9zyncVy8XN#<}^NDI{C|CpL2sm4^+U^mN z1p0R{LN!{L_+LXw`d4ja#c=o{6N>Inv0P)2W&)aKOb3iKS0JLEx0h{8}`Z05>! zBTTUdbe@sXsyUBZf|cGcUrs?#CTo9_@(=fnv+pvFkB ziCZ*>ff(d#zo;@)-2lk%!1=5N7VOw{nN(N_#zVz@-C(f{#h})6w5>ix*>Ddee+rBw zw1(z&QLh=T_-T5rv3+4;7XXpRXFrTkxsDDQc{YRnRHKo-1Pe+jDMToKv&n{Jah@_X zBGLW+qA-~N&@1qS6W*+_@s3a(JK{aJ1g}Z|WDhzOoD%sB2RA~ESBT3o6Y;u%@pQ}N zcqG+8Jp+`K4MlN=^oT3KTw+)&vxN={WYSE;?M|Q?tT?nt+m}GLR6C?nPEfsPX7kx& zfoS^)_P%3q(s{h~GN%6g;jymLT3xgt^H{Z-K$sm#XSr^FfjxmWIiL_qpp6iQm~e$R z;8Zagn!@mbuFs`8sGEf}lD7bi#5zhU{e~cQ38JM>>@A2*dbcLTFLY9%H6&eGyFO=B z6E%!mySaAzw(~1T5acT#{aLRT0sn?y8@0BkK~RAwnrVWk_ioX6@bC{$?`sJU!RG1< ztrMA}{(k2iG7PB&nzVMBdOz70%C|E(-Jgz-#}R{PkoY4h`~3VkErmu8SY!08Ecd2ebTcae)WpFbMp z_JFJ5U2|g&4t$ihOGC#=6!zd4g z(#av8sHCN`F?EM68S2`4v(B*J+xFy7rcp%aka--BH_8ZT#J$@A)c;lTF=77munB+5Vvz2; zO(m!rE^}yX1~>r^;47TewKQcgOE41^45#SKwpY*enPDY#O1W+=3FISNmJAx)=PL6B z=*At~pVr8LGft5Axl!zs(abHg6777ACGdswr8+Dd-SG+%%ve5Yd0lHH&~JN(bykM9 z=EmWEir(R1S3?mXtK08lAftWVooT<`w-)4b2GTmk9Oh0yhmNS+Q9s@`O@1$Adslf^ z?46frZli=LZ$<6_7!qB6=x~AdLR9E)TAluPAHD^V)SG7Xl13Oz!u9y(2XdmVQ){M4 zCh2oLEhB9aEjEAdJc1+26?$!w6OP!dW)|!I3APQ{d3rB%(*_S6XrACsL+cEk!M=_* z%-hv_)6{#9PCYV0^hE8wus3fn7+%+ z;*E&D)m~23?Qtl8sa9zMn;|w!x#X&7tOPTonr4p%Y}P1Cg=`BFYhUB2Su|xChRlnX z1I&gQkeE^mC}EoK3XBsu=JW1r%cBtW2%P%D!bVz*o(voc3^sAqF)`OMQLFoU0-?Ft>hEcjSWGJiNsN9<~$yR{k|N@pyI$ zHrzy+TMZKM0${Km3(r0Fg)S097u!WjV|Bwlc7h``*)b2rZFFrmfyRb0@94?J82evT zqK+b@pe>1RL$^rCG9TIuGN)Ol)55I45MH3v(|h(&Ou6u&e*KTX$`ey0JOo`|Qq|7; z#O0>sq`xb03mwS%*kZFeAJ=8u0eBCZ5S3WP?*L7P>=~yqRaxMo4=4z!wz;Xu76M|u z^K{PSmNs~wJiSNnP8ScqDMEi`BDA~igzH@PdgG_=EWHt);{m(o<-sCt*ig)chTh{# zVwQhx7Ti;$;hJZWq~&OH1u=1RhEs}ajyrMbEe+w{KSd(pi*yJI%fWMIrndO>;D~4^mZ1)vXqu-f| ziva(-X~kA!+-;sTI=~3JkTa#RNpa@sE>_`=7po|HvvuPSc`t8kGB-1L>bU*q?!9c{jYl&}Tu5f`8(`bMQ}gijzPSbH+3Pj9 z2vue5Q9s^m*9P$C(Gr&vyv|DBcDma*wZz0I@(y|X=YpXfx^;)nOCHC_?gw0TM$@VW zlmUrJ9gu$6*!THJyHiBnpfDMq`-zNicjss8h6U?aV1}no{_JLs<1>d%hF&L)$y40~ z`2Q?t-2~w^Xh^yv%${v6YhS*{8SE|U(piM#Ph3IU&Zlgp8==>V&?qio|8D{4JK?A4 zyvYrey2}EZ!)J0zAW*agpJEyEiE@_S>H?C3D=^X&dFIO2mDL)jQSLh3<_)Z-RyMtj znnIfwRz&k)jWVWv(F?*K->EbT;-ZKpvrSrSgwNk`(6#v-ow1* zYHKD2I>zbdv87w};3O&#ulcgJ5-%MxnYW)xQ_r+Eqs5))ERHCJ!_l64*#&DbMQdc* kiqp`ys7wfpc};4!eHc|Yo@^r+yV)8|Kd|eq2eSM7zlZX&bN~PV literal 0 HcmV?d00001 diff --git a/server/typescript/packages/codegen-ts/src/runner.ts b/server/typescript/packages/codegen-ts/src/runner.ts index f0ee37229..8210b8b0b 100644 --- a/server/typescript/packages/codegen-ts/src/runner.ts +++ b/server/typescript/packages/codegen-ts/src/runner.ts @@ -6,7 +6,10 @@ import { tmpdir } from "node:os"; import { fileURLToPath } from "node:url"; import { existsSync, readFileSync } from "node:fs"; import type { MetaData, MetaObject } from "@metaobjectsdev/metadata"; -import { isMetaRoot, OBJECT_SUBTYPE_VALUE, FIELD_SUBTYPE_TIMESTAMP, FIELD_ATTR_FILTERABLE } from "@metaobjectsdev/metadata"; +import { + isMetaRoot, OBJECT_SUBTYPE_VALUE, FIELD_SUBTYPE_TIMESTAMP, FIELD_ATTR_FILTERABLE, + composeRegistry, coreProviders, +} from "@metaobjectsdev/metadata"; import { assignEmittedNames } from "./naming/collision-names.js"; import { isAbstract } from "./instance-artifacts.js"; import { dbEmittingObjects, missingDialectMessage } from "./db-emitting.js"; @@ -106,6 +109,15 @@ export interface RunGenOpts { * they pass straight through. */ scope?: (fqn: string) => boolean; + /** + * FR-023 §4.3 — the collection's own source files (`Collection.ownFiles`; never + * a dependency's snapshot artifact), threaded onto `GenContext.sourceFiles` for + * `sharedModelFile()`'s default `files` selection. The CLI's `gen` command + * always passes `genCollection.ownFiles`; a programmatic caller that never + * wires `sharedModelFile()` (or always passes that generator's own `files` + * option explicitly) can omit this with no behavior change elsewhere. + */ + sourceFiles?: readonly string[]; } export interface RunGenResult { @@ -374,6 +386,13 @@ export async function runGen(opts: RunGenOpts): Promise { // 2. Resolve targets + entity-module target. const config = normalizeConfig(opts.config); + // FR-023 §4.3 — the run's composed registry, threaded onto every GenContext. + // Mirrors sdk's `loadMemory` exactly (`defaultLoadMemoryProviders` = core + // providers, then the project's own `config.providers` appended) so + // `sharedModelFile()`'s standalone re-load of a `files` subset sees the SAME + // vocabulary `opts.metadata` was originally loaded with. + const registry = composeRegistry([...coreProviders, ...(config.providers ?? [])]); + // The compile break (`value` → `fetcher` on the provider) sends an adopter to the right // line; this names the value to put there, once, for the projects that had one. if (shouldNoteBaseUrlMove(config.apiPrefix, recordedEngine)) { @@ -623,6 +642,8 @@ export async function runGen(opts: RunGenOpts): Promise { }, renderContext, ...(projectRoot !== undefined && { projectRoot }), + registry, + ...(opts.sourceFiles !== undefined && { sourceFiles: opts.sourceFiles }), warn: (msg) => warnings.push(`[${generator.name}] ${msg}`), }; diff --git a/server/typescript/packages/codegen-ts/test/generator-registry.test.ts b/server/typescript/packages/codegen-ts/test/generator-registry.test.ts index bdeaf8171..6f0d86fa4 100644 --- a/server/typescript/packages/codegen-ts/test/generator-registry.test.ts +++ b/server/typescript/packages/codegen-ts/test/generator-registry.test.ts @@ -29,6 +29,7 @@ const EXPECTED_NATIVE = [ "template", "api-docs", "trace-helper", + "shared-model", ] as const; // Neutral / `meta docs`-owned (Tier-2). Present in the registry for identity + diff --git a/server/typescript/packages/codegen-ts/test/shared-model-file.test.ts b/server/typescript/packages/codegen-ts/test/shared-model-file.test.ts new file mode 100644 index 000000000..bfb3259fb --- /dev/null +++ b/server/typescript/packages/codegen-ts/test/shared-model-file.test.ts @@ -0,0 +1,290 @@ +// FR-023 §4.3 — sharedModelFile(): the publisher's flattened shared-model artifact. +// +// A publisher project's two source files declare `acme::common::{Address, Audited, +// Customer}` exactly as the pinned cross-port corpus artifact +// (fixtures/dependency-conformance/artifacts/acme-common-v1.json) does, plus an +// INTERNAL `acme::common::Secret` entity (never published in the byte-identical +// scenarios below) and a `requirement.functional` (always excluded). Secret exists +// to exercise the closure check (a real `field.object` ref to Address) and the +// core-provider re-load check (a throwaway common attr no core provider ships) — +// see the report for why these use Secret rather than Customer (the pinned +// artifact's actual Customer/Address pair carries no ref between them at all). +import { describe, test, expect, beforeEach, afterEach } from "bun:test"; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync, readFileSync, existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { runGen } from "../src/runner.js"; +import { defineConfig } from "../src/metaobjects-config.js"; +import { sharedModelFile } from "../src/generators/shared-model-file.js"; +import { + MetaDataLoader, + ATTR_SUBTYPE_STRING, + type MetaDataTypeProvider, +} from "@metaobjectsdev/metadata"; +import { FileSource } from "@metaobjectsdev/metadata/core"; + +function findRepoRoot(start: string): string { + let dir = start; + for (;;) { + if (existsSync(join(dir, "fixtures")) && existsSync(join(dir, "server"))) return dir; + const parent = dirname(dir); + if (parent === dir) throw new Error("could not locate repo root (dir containing fixtures/ and server/)"); + dir = parent; + } +} + +const REPO_ROOT = findRepoRoot(import.meta.dir); +const PINNED_ARTIFACT = join(REPO_ROOT, "fixtures", "dependency-conformance", "artifacts", "acme-common-v1.json"); +const PINNED_HASH = "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d"; + +// A throwaway provider registering ONE common attr no core provider ships — +// stands in for "a consumer-provider attr" (registerCommonAttrs, not a whole new +// subtype, per ADR-0050: common attrs project onto every type, so this is the +// simplest real vocabulary a downstream provider could add). +function secretHandlingProvider(): MetaDataTypeProvider { + return { + id: "test-secret-handling", + dependencies: ["metaobjects-core-types"], + registerTypes(registry) { + registry.registerCommonAttrs([ + { + name: "secretHandling", + valueType: ATTR_SUBTYPE_STRING, + required: false, + description: "test-only throwaway attr — not core vocabulary.", + }, + ]); + }, + }; +} + +// File A: Address (object.value) + Audited (abstract object.entity) — exactly the +// shape of the pinned artifact's Address/Audited. +const FILE_A = { + "metadata.root": { + package: "acme::common", + children: [ + { + "object.value": { + name: "Address", + children: [ + { "field.string": { name: "city" } }, + { "field.string": { name: "street" } }, + ], + }, + }, + { + "object.entity": { + name: "Audited", + abstract: true, + children: [{ "field.timestamp": { name: "createdAt" } }], + }, + }, + ], + }, +}; + +// File B: Customer (exactly the pinned artifact's shape), Secret (internal — a real +// field.object ref to Address, plus the throwaway @secretHandling attr), and a +// requirement.functional (always excluded regardless of include/exclude). +const FILE_B = { + "metadata.root": { + package: "acme::common", + children: [ + { + "object.entity": { + name: "Customer", + children: [ + { "source.rdb": { "@table": "customers" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "email", "@maxLength": 120 } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + { + "object.entity": { + name: "Secret", + "@secretHandling": "vault", + children: [ + { "source.rdb": { "@table": "secrets" } }, + { "field.long": { name: "id" } }, + { "field.object": { name: "billingAddress", "@objectRef": "Address" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + { + "requirement.functional": { + name: "KeepSecretsSafe", + "@level": 5, + "@status": "live", + "@statement": "Every secret's billing address is stored encrypted at rest.", + "@counterexample": "A secret row with a plaintext billing address.", + }, + }, + ], + }, +}; + +let tmp: string; +let fileA: string; +let fileB: string; + +beforeEach(() => { + tmp = mkdtempSync(join(tmpdir(), "shared-model-file-")); + mkdirSync(join(tmp, "metaobjects"), { recursive: true }); + fileA = join(tmp, "metaobjects", "meta.common-base.json"); + fileB = join(tmp, "metaobjects", "meta.common-customer.json"); + writeFileSync(fileA, JSON.stringify(FILE_A, null, 2)); + writeFileSync(fileB, JSON.stringify(FILE_B, null, 2)); +}); + +afterEach(() => { + rmSync(tmp, { recursive: true, force: true }); +}); + +/** Loads the outer `opts.metadata` MetaRoot `runGen` requires (non-strict — the + * strict, provider-aware load under test is entirely INSIDE sharedModelFile()'s + * own standalone re-load of `ctx.sourceFiles`/`opts.files`). */ +async function loadOuterRoot() { + const loader = new MetaDataLoader(); + const { root, errors } = await loader.load([new FileSource(fileA), new FileSource(fileB)]); + expect(errors).toEqual([]); + return root; +} + +async function gen(generator: ReturnType, outDir: string) { + const metadata = await loadOuterRoot(); + return runGen({ + config: defineConfig({ + outDir, + extStyle: "none", + dbImport: "../db", + dialect: "sqlite", + generators: [generator], + // Needed because Secret (loaded on every run — it lives in fileB, always + // among ctx.sourceFiles) carries @secretHandling; sharedModelFile()'s own + // standalone load is STRICT and uses `ctx.registry`, which the runner + // composes from exactly this list (mirrors sdk's loadMemory). + providers: [secretHandlingProvider()], + }), + metadata, + projectRoot: outDir, + sourceFiles: [fileA, fileB], + }); +} + +describe("sharedModelFile()", () => { + test("(a) emits the artifact byte-equal to the pinned corpus artifact", async () => { + const result = await gen( + sharedModelFile({ + name: "acme-common", + include: ["acme::common::**"], + exclude: ["acme::common::Secret"], + version: "1.0.0", + }), + tmp, + ); + expect(result.conflicts).toEqual([]); + const artifactPath = join(tmp, "acme-common.metaobjects.json"); + const actual = readFileSync(artifactPath, "utf-8"); + const expected = readFileSync(PINNED_ARTIFACT, "utf-8"); + expect(actual).toBe(expected); + }); + + test("(b) the manifest matches the Global Constraints example, pinned hash included", async () => { + await gen( + sharedModelFile({ + name: "acme-common", + include: ["acme::common::**"], + exclude: ["acme::common::Secret"], + version: "1.0.0", + }), + tmp, + ); + const manifest = JSON.parse(readFileSync(join(tmp, "metaobjects.pkg.json"), "utf-8")); + expect(manifest).toEqual({ + schema_version: 1, + name: "acme-common", + version: "1.0.0", + metamodelVersion: "1.0", + artifact: "acme-common.metaobjects.json", + integrity: PINNED_HASH, + packages: ["acme::common"], + nodes: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"], + }); + }); + + test("(c) determinism — two runs produce identical bytes", async () => { + const opts = () => + sharedModelFile({ + name: "acme-common", + include: ["acme::common::**"], + exclude: ["acme::common::Secret"], + version: "1.0.0", + }); + const dirA = join(tmp, "run-a"); + const dirB = join(tmp, "run-b"); + mkdirSync(dirA, { recursive: true }); + mkdirSync(dirB, { recursive: true }); + await gen(opts(), dirA); + await gen(opts(), dirB); + expect(readFileSync(join(dirA, "acme-common.metaobjects.json"), "utf-8")) + .toBe(readFileSync(join(dirB, "acme-common.metaobjects.json"), "utf-8")); + expect(readFileSync(join(dirA, "metaobjects.pkg.json"), "utf-8")) + .toBe(readFileSync(join(dirB, "metaobjects.pkg.json"), "utf-8")); + }); + + // Closure: the pinned artifact's REAL Customer/Address pair carries no ref + // between them at all (Customer's fields are id/email/pk only — confirmed by + // reading the pinned artifact; see the report). Secret's real `field.object + // @objectRef: Address` is the genuine edge this repo's fixtures provide, so the + // closure failure is exercised on `Secret -> Address` instead of the brief's + // `Customer -> Address` example. Same algorithm, same message shape. + test("(d) closure failure — an unincluded referenced target fails naming the pair", async () => { + await expect( + gen(sharedModelFile({ name: "acme-common", include: ["acme::common::Secret"] }), tmp), + ).rejects.toThrow( + /include it or exclude the referrer:[\s\S]*acme::common::Secret → acme::common::Address/, + ); + }); + + test("(e) a files subset omitting Customer's file exports exactly two nodes", async () => { + await gen( + sharedModelFile({ + name: "acme-common-partial", + include: ["acme::common::**"], + files: [fileA], + version: "1.0.0", + }), + tmp, + ); + const manifest = JSON.parse(readFileSync(join(tmp, "metaobjects.pkg.json"), "utf-8")); + expect(manifest.nodes).toEqual(["acme::common::Address", "acme::common::Audited"]); + }); + + // Core-provider re-load: closure passes (Address is included alongside Secret), + // but Secret's @secretHandling attr is registered by the run's own provider + // (needed just to LOAD it at all, strict) and not by any core provider — so the + // re-load-with-core-providers-only step must fail it. + test("(f) a node carrying a consumer-provider attr fails the core-only re-load", async () => { + await expect( + gen( + sharedModelFile({ + name: "acme-common", + include: ["acme::common::Secret", "acme::common::Address"], + }), + tmp, + ), + ).rejects.toThrow( + /needs a provider this toolchain does not ship; Phase 1 exports must load with core vocabulary/, + ); + }); + + test("empty selection fails with a clear error", async () => { + await expect( + gen(sharedModelFile({ name: "acme-common", include: ["acme::nothing::**"] }), tmp), + ).rejects.toThrow(/selected no nodes/); + }); +}); From eb9f0034da2b603b3c3583b699489dfd3fad2a06 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 23:34:27 -0400 Subject: [PATCH 37/62] fix(codegen-ts): replace a raw NUL byte in shared-model-file.ts's dedupe key with the \0 escape (FR-023) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A literal 0x00 control byte in the closure-check dedupe key made git treat the file as binary — diffs showed no content, and the public-repo-hygiene pre-commit grep never enumerated its added lines, silently bypassing that guard for this file. The `\0` escape produces the identical runtime string with byte-identical source. Also adds a targeted test for a top-level node whose own explicit `package` differs from its file's default: the closure check's effectivePackage()-derived referrer package agrees with the loader's own threaded reference resolution here (fileDefaultPackage is captured at parse time from the nearest PACKAGED ancestor, not the file root), so this closes the reviewer-flagged unknown rather than a live defect. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../src/generators/shared-model-file.ts | Bin 13024 -> 13959 bytes .../codegen-ts/test/shared-model-file.test.ts | 69 ++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/server/typescript/packages/codegen-ts/src/generators/shared-model-file.ts b/server/typescript/packages/codegen-ts/src/generators/shared-model-file.ts index e37673ad998c0af5fa6e5fca76407953c374f8f5..e35d89d16aff5695f0e3dfb5858dcfa6929f3152 100644 GIT binary patch delta 959 zcmY*XOKTHR6yCXLH*Ta0H-{pal)QF=q7d4tilwbhiwK3z&CHpZYwp}T+(*+O5?8tx ze}Uk_ozOod{Uv^Pk_avmlFK>Y`CfncYyJD@_Hwwlw?%t2;YL{DjimygG^;q9?JMJg z<@B+mob%8y&3G}OFD`xaX}3r7N@f+! zO;9ECF5rmN zB`g#^YA=JZd7Goq3o3#~Gl_E;;@2&Ung?_a_3Sc%wl>z}A%nc=NKB}Y_JLm}R7A7^ zzj2?ON72e>WHc_v*~!Vp*?vNYua6QCkDLDt&MKixKz5K=-gYx8LEO&cA;Ovhh7;FQ zZk>tAY#P!XU|_g~C+F!PIk-4Y9$dXi-d|0U)A8B({N#+D)9&SH_x8v3z56?lKCWi| MANh3q>*14~zcGbNp8x;= delta 19 acmZqAeUQ3gDfebGer5qihRp%;=HdWMW(F1j diff --git a/server/typescript/packages/codegen-ts/test/shared-model-file.test.ts b/server/typescript/packages/codegen-ts/test/shared-model-file.test.ts index bfb3259fb..33d6258dd 100644 --- a/server/typescript/packages/codegen-ts/test/shared-model-file.test.ts +++ b/server/typescript/packages/codegen-ts/test/shared-model-file.test.ts @@ -287,4 +287,73 @@ describe("sharedModelFile()", () => { gen(sharedModelFile({ name: "acme-common", include: ["acme::nothing::**"] }), tmp), ).rejects.toThrow(/selected no nodes/); }); + + // Review ⚠️ — effectivePackage() (docs-paths.ts) derives a node's package from ITS + // OWN resolutionKey()/fileDefaultPackage. A nested field never carries its own + // `package`, so its fileDefaultPackage is stamped from the FILE's root default — + // NOT from an enclosing top-level object's own (different) explicit `package`. The + // loader's OWN generic reference resolution (validation-registry.ts's `walk()`) + // instead THREADS the referrer package down the tree, so a nested field resolves + // its bare refs against the ENCLOSING TOP-LEVEL OBJECT's actual package — correctly, + // by construction. This test drives that exact shape (file default "acme::common", + // a top-level node explicitly overriding to "acme::other", a nested field's bare + // ref that only resolves correctly under the override) through the closure check, + // to find out whether `effectivePackage()`-derived referrerPkg agrees. + test("package-override: a top-level node's own explicit package governs its nested field's bare-ref resolution in the closure check", async () => { + const fileC = join(tmp, "metaobjects", "meta.pkg-override.json"); + writeFileSync( + fileC, + JSON.stringify( + { + "metadata.root": { + package: "acme::common", // the FILE default + children: [ + { + "object.value": { + name: "Gadget", + package: "acme::other", // explicit override, DIFFERENT from the file default + children: [{ "field.object": { name: "part", "@objectRef": "Thing" } }], + }, + }, + { + "object.value": { + name: "Thing", + package: "acme::other", + children: [{ "field.string": { name: "label" } }], + }, + }, + ], + }, + }, + null, + 2, + ), + ); + + const loader = new MetaDataLoader(); + const { root: metadata, errors } = await loader.load([new FileSource(fileC)]); + expect(errors).toEqual([]); + + // include ONLY Gadget: a CORRECTLY package-scoped bare ref resolves "Thing" to + // acme::other::Thing (Gadget's own package), which is excluded here — closure + // must fail naming that pair. Were the referrer package wrongly computed as the + // file default "acme::common", the bare ref would resolve to NOTHING (no + // acme::common::Thing exists), and `checkClosure`'s defensive + // `if (resolved === undefined) continue` (written to trust the loader already + // validated every ref) would silently skip it instead — no error at all. + await expect( + runGen({ + config: defineConfig({ + outDir: tmp, + extStyle: "none", + dbImport: "../db", + dialect: "sqlite", + generators: [sharedModelFile({ name: "pkg-override", include: ["acme::other::Gadget"] })], + }), + metadata, + projectRoot: tmp, + sourceFiles: [fileC], + }), + ).rejects.toThrow(/acme::other::Gadget.*acme::other::Thing/s); + }); }); From ded60c2081eea3e2cf575b38ef36e594578c9b74 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Fri, 11 Sep 2026 23:40:32 -0400 Subject: [PATCH 38/62] docs(fr-023): correct three stale references in the phase-1a plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three reference defects found while resuming execution at Task 13. Each was verified against the tree before editing; the source won in every case. - Task 13's file list named `RunGenOptions.sourceFiles?`. The exported type is `RunGenOpts` (runner.ts:45) and `RunGenOptions` exists nowhere in the workspace. The plan had already diagnosed this name in Task 10's notes and left the wrong one standing here. - Tasks 6 and 21 both grep for deleted vocabulary expecting no output, using a bare `OVERLAY_IMPLICIT`. That pattern also matches `WARN_OVERLAY_IMPLICIT` — the diagnostic code Task 17 introduces, and which Task 20 then documents under docs/features and agent-context, both of which Task 21's grep covers. As written, the gate would fail on the plan's own deliverables. Both are anchored to `ERR_DEPENDENCY_OVERLAY_IMPLICIT`, which is what the deletion check means. The remaining alternatives are unambiguous: `SCHEMA_NOT_OWNED` does not match the live `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`. No behaviour change; plan text only. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 97e3dbcb8..663ec5c7c 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -650,7 +650,7 @@ cd server/typescript/packages/sdk && bun test test/config.test.ts test/dependenc cd server/typescript/packages/metadata && bun test test/errors.test.ts test/scope-conformance.test.ts cd server/python && uv run --extra integration pytest tests/config tests/unit/test_errors.py tests/conformance/test_scope_conformance.py -q bun run --filter '@metaobjectsdev/sdk' typecheck && bun run --filter '@metaobjectsdev/metadata' typecheck -grep -rn 'OVERLAY_IMPLICIT\|SCHEMA_NOT_OWNED\|BREAKING_CHANGE\|DEPENDENCY_MODES\|LOCAL_OVERRIDE_FILE\|deps.local.json' server fixtures --include=*.ts --include=*.py --include=*.json --include=*.md | grep -v superpowers +grep -rn 'ERR_DEPENDENCY_OVERLAY_IMPLICIT\|SCHEMA_NOT_OWNED\|BREAKING_CHANGE\|DEPENDENCY_MODES\|LOCAL_OVERRIDE_FILE\|deps.local.json' server fixtures --include=*.ts --include=*.py --include=*.json --include=*.md | grep -v superpowers ``` Expected: tests PASS; typecheck PASS; the grep prints NOTHING (the corpus README and runner included). The one-case corpus (`no-dependencies-resolves-exactly-as-before`) fails on `collection.imported` not existing — expected until Task 8. @@ -900,7 +900,7 @@ feat(cli): the requirements ledger does not count imported entities the project **Files:** - Create: `server/typescript/packages/codegen-ts/src/generators/shared-model-file.ts` - Modify: `server/typescript/packages/codegen-ts/src/generator.ts` (`GenContext.registry?: TypeRegistry`, `GenContext.sourceFiles?: readonly string[]`) -- Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (fill both; `RunGenOptions.sourceFiles?`) +- Modify: `server/typescript/packages/codegen-ts/src/runner.ts` (fill both; `RunGenOpts.sourceFiles?`) - Modify: `server/typescript/packages/codegen-ts/src/generator-registry.ts` (entry `shared-model`, `options: "name, include, exclude?, files?, version?, target?"`) - Modify: `server/typescript/packages/codegen-ts/src/generators/index.ts` (export) - Modify: `fixtures/generator-registry-conformance/registry.json` (entry `shared-model`, `"ports": ["typescript"]`, `tier: "native"`) @@ -1119,7 +1119,7 @@ Expected: all three green. `gates` includes `check-metamodel-version.mjs` (no di - [ ] **Step 2:** Confirm the deletions held: ```bash -grep -rn 'OVERLAY_IMPLICIT\|SCHEMA_NOT_OWNED\|BREAKING_CHANGE\|DEPENDENCY_MODES\|deps.local.json\|foreignOwner\|governs(' server fixtures docs/features agent-context --include=*.ts --include=*.py --include=*.json --include=*.md +grep -rn 'ERR_DEPENDENCY_OVERLAY_IMPLICIT\|SCHEMA_NOT_OWNED\|BREAKING_CHANGE\|DEPENDENCY_MODES\|deps.local.json\|foreignOwner\|governs(' server fixtures docs/features agent-context --include=*.ts --include=*.py --include=*.json --include=*.md ``` Expected: no output. From 69762ea8be38eaef657515792f42cea2d2d10878 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 00:07:56 -0400 Subject: [PATCH 39/62] =?UTF-8?q?feat(cli):=20meta=20deps=20sync=20and=20l?= =?UTF-8?q?ist=20=E2=80=94=20a=20committed=20snapshot=20pinned=20by=20a=20?= =?UTF-8?q?sha256=20lock=20(FR-023)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../packages/cli/src/commands/deps.ts | 161 ++++++ server/typescript/packages/cli/src/index.ts | 36 +- .../typescript/packages/cli/src/lib/args.ts | 60 +++ .../packages/cli/src/lib/dependency-sync.ts | 461 ++++++++++++++++++ .../cli/test/__snapshots__/cli.test.ts.snap | 5 +- .../packages/cli/test/deps-sync.test.ts | 304 ++++++++++++ .../packages/cli/test/unit/args-deps.test.ts | 61 +++ 7 files changed, 1086 insertions(+), 2 deletions(-) create mode 100644 server/typescript/packages/cli/src/commands/deps.ts create mode 100644 server/typescript/packages/cli/src/lib/dependency-sync.ts create mode 100644 server/typescript/packages/cli/test/deps-sync.test.ts create mode 100644 server/typescript/packages/cli/test/unit/args-deps.test.ts diff --git a/server/typescript/packages/cli/src/commands/deps.ts b/server/typescript/packages/cli/src/commands/deps.ts new file mode 100644 index 000000000..52d1d74b6 --- /dev/null +++ b/server/typescript/packages/cli/src/commands/deps.ts @@ -0,0 +1,161 @@ +// server/typescript/packages/cli/src/commands/deps.ts +// +// FR-023 Phase 1a, Task 14 — `meta deps sync` (`path` transport) and +// `meta deps list`. `meta deps check` parses (see `parseDepsArgs`) but is +// Task 15's — this task refuses it rather than guessing at its report shape. +import { join } from "node:path"; +import { + DEFAULT_METAOBJECTS_DIR, + discoverCollectionRoot, + loadConfig, + loadMemory, + readLock, + resolveCollection, + type DependencySpec, + type Lock, +} from "@metaobjectsdev/sdk"; +import { parseDepsArgs, type DepsFlags } from "../lib/args.js"; +import { log } from "../lib/log.js"; +import { emitStructured, type OutputFormat } from "../lib/format.js"; +import { reportLoadError } from "../lib/load-error.js"; +import { collectionLoadOptions } from "../lib/collection-load-options.js"; +import { applySync, planSync } from "../lib/dependency-sync.js"; + +/** The declared `dependencies` for the config governing `cwd` — read + * DIRECTLY via `loadConfig`, never through `resolveCollection`/`Collection`: + * that resolver VERIFIES the snapshot against the lock (`verifySnapshot`) + * and throws `ERR_DEPENDENCY_SNAPSHOT_STALE` the moment they disagree — the + * exact condition `meta deps sync` exists to fix. A project with no config + * at all declares none (`resolveCollection`'s own default: no config ⇒ no + * dependencies), which `discoverCollectionRoot`'s `hasConfig` already knows + * without a doomed `loadConfig` call. */ +async function declaredDependencies(cwd: string): Promise<{ configDir: string; specs: readonly DependencySpec[] }> { + const { dir: configDir, hasConfig } = await discoverCollectionRoot(cwd); + if (!hasConfig) return { configDir, specs: [] }; + const cfg = await loadConfig(join(configDir, DEFAULT_METAOBJECTS_DIR)); + return { configDir, specs: cfg.dependencies }; +} + +async function runSync(configDir: string, specs: readonly DependencySpec[], flags: DepsFlags, fmt: OutputFormat): Promise { + const lock: Lock | undefined = await readLock(configDir); + + let plan; + try { + plan = await planSync(configDir, specs, lock, flags.names); + } catch (err) { + log.error((err as Error).message); + return 1; + } + + let applied; + try { + applied = await applySync(configDir, plan, { dryRun: flags.dryRun }); + } catch (err) { + log.error((err as Error).message); + return 1; + } + + if (fmt === "text") { + for (const w of plan.warnings) log.info(w); + if (applied.report.length === 0) { + log.info("meta deps sync: nothing to do — no dependencies declared."); + } else { + for (const r of applied.report) log.info(r.line); + } + } else { + emitStructured( + { + warnings: plan.warnings, + dependencies: applied.report.map((r) => ({ name: r.name, action: r.action, message: r.line })), + }, + fmt, + ); + } + + // A dry run wrote nothing to validate — the plan already stands for what + // WOULD load. + if (flags.dryRun) return 0; + + // DESIGN §4.1's closing sentence: load the full collection once, so a sync + // that produces an unloadable model (a collision the plan missed because it + // only compares declared node LISTS, a scope that now excludes something + // load-bearing, …) fails in THIS command with the loader's own error, + // rather than surfacing on the next unrelated `meta gen`/`verify`. + let collection; + try { + collection = await resolveCollection(configDir); + } catch (err) { + log.error(`meta deps sync: the updated dependency set does not resolve: ${(err as Error).message}`); + return 1; + } + try { + await loadMemory(collection.configDir, collectionLoadOptions(collection)); + } catch (err) { + reportLoadError(log, "meta deps sync: the updated dependency set does not load", err); + return 1; + } + + return 0; +} + +async function runList(configDir: string, fmt: OutputFormat): Promise { + const lock = await readLock(configDir); + const entries = Object.entries(lock?.dependencies ?? {}); + + if (fmt === "text") { + if (entries.length === 0) { + log.info("meta deps list: no dependencies locked — run `meta deps sync`."); + return 0; + } + for (const [name, entry] of entries) { + const hash8 = entry.integrity.slice("sha256-".length, "sha256-".length + 8); + log.info( + `${name} ${entry.version} ${hash8} ${entry.nodes.length} node(s) ${entry.packages.join(", ")}`, + ); + } + } else { + emitStructured( + { dependencies: entries.map(([name, entry]) => ({ name, ...entry })) }, + fmt, + ); + } + return 0; +} + +export async function depsCommand(args: string[], cwd: string, fmt: OutputFormat): Promise { + let flags: DepsFlags; + try { + flags = parseDepsArgs(args); + } catch (err) { + log.error((err as Error).message); + return 2; + } + + let configDir: string; + let specs: readonly DependencySpec[]; + try { + ({ configDir, specs } = await declaredDependencies(cwd)); + } catch (err) { + // Mirrors `resolveCollection`'s own convention for a config.json that + // EXISTS but fails to load (malformed JSON, a schema violation): exit 2, + // the same code every other routed command uses for that class of + // failure (gen.ts, migrate.ts, docs.ts) — distinct from the exit 1 a + // dependency-resolution failure gets below, because this is a config + // problem, not a `sync` verdict. + log.error((err as Error).message); + return 2; + } + + switch (flags.subverb) { + case "sync": + return runSync(configDir, specs, flags, fmt); + case "list": + return runList(configDir, fmt); + case "check": + log.error( + "meta deps check is not implemented in this release. It will compare each declared " + + "dependency's INSTALLED artifact against the committed lock (ERR_DEPENDENCY_UPSTREAM_DRIFT).", + ); + return 1; + } +} diff --git a/server/typescript/packages/cli/src/index.ts b/server/typescript/packages/cli/src/index.ts index ae633914d..b2b94dc06 100644 --- a/server/typescript/packages/cli/src/index.ts +++ b/server/typescript/packages/cli/src/index.ts @@ -17,7 +17,7 @@ const VERSION = cliVersion(); * human text. It is named ONCE and used by both the warning below and the help * text above, so the two cannot drift apart. */ -const FORMAT_AWARE_COMMANDS: readonly string[] = ["gen", "verify", "migrate", "types"]; +const FORMAT_AWARE_COMMANDS: readonly string[] = ["gen", "verify", "migrate", "types", "deps"]; const HELP_TEXT = `meta — MetaObjects CLI (v${VERSION}) @@ -32,6 +32,9 @@ COMMANDS: gen [...] Codegen TS targets from your declared metadata eject Copy a reference generator into codegen/generators/ to own it (any time after init) eject --list List every ejectable generator name, grouped by package + deps sync [...] Resolve declared dependencies (path transport): sync the committed + snapshot + sha256 lock (.metaobjects/deps/, .metaobjects/deps.lock.json) + deps list One line per locked dependency: name, version, hash, node/package summary types [query] Search the metadata vocabulary (types, subtypes, @attrs) by name or description export Flatten loaded metadata to one canonical JSON artifact docs [] --out Generate neutral metadata documentation (entity + template pages; --site for HTML site) @@ -191,6 +194,33 @@ the same operation for ANY generator — one you skipped at init time, a UI-tier generator like form/hooks/grid, or one a package gains later. It prints the import line to paste into metaobjects.config.ts, and it never overwrites a file you already own unless you pass --force. +`, + deps: `meta deps — sync a declared metadata dependency's committed snapshot + +USAGE: + meta deps sync [...] Resolve each declared dependency (path transport only in + this release — npm/python refuse by name), validate its + manifest + artifact, copy it into .metaobjects/deps//, + and pin it in .metaobjects/deps.lock.json. Naming one or more + s syncs only those; bare 'sync' syncs everything + declared. A dependency removed from config is PRUNED from + the lock and its snapshot dir deleted, every run. + meta deps list One line per locked dependency: name, version, hash8, + node count, packages. + +FLAGS: + --dry-run Plan and report; write nothing (sync only) + --format Output format (global flag; default toon off-TTY) + --help, -h Print this help + +Only the "path" transport resolves in this release — a "npm"/"python" dependency spec is +valid config (reserved for a future toolchain) but 'meta deps sync' refuses it by name: +"transport \`npm\` is not supported by this toolchain yet; use \`path\`". + +The lock is the ONLY thing 'meta deps sync' writes to besides the snapshot directory — +sync never touches your own metadata files. 'meta deps check' (a future release) compares +the lock against what is installed right now; this command only ever compares against +what config DECLARES. `, verify: `meta verify — drift gate (templates / DB schema / codegen / migration replay) @@ -530,6 +560,10 @@ export async function run(argv: string[]): Promise { const { ejectCommand } = await import("./commands/eject.js"); return ejectCommand(rest, cwd); } + case "deps": { + const { depsCommand } = await import("./commands/deps.js"); + return depsCommand(rest, cwd, fmt); + } case "export": { const { exportCommand } = await import("./commands/export.js"); return exportCommand(rest, cwd); diff --git a/server/typescript/packages/cli/src/lib/args.ts b/server/typescript/packages/cli/src/lib/args.ts index d292ead43..c7a92d7e1 100644 --- a/server/typescript/packages/cli/src/lib/args.ts +++ b/server/typescript/packages/cli/src/lib/args.ts @@ -607,3 +607,63 @@ export function parseEjectArgs(argv: string[]): EjectFlags { force: !!values.force, }; } + +// --------------------------------------------------------------------------- +// deps flags — FR-023 Phase 1a Task 14 +// --------------------------------------------------------------------------- + +/** `meta deps`'s subcommand. `sync` and `list` are this task's; `check` is + * parsed (so the grammar and its usage errors are stable now) but not yet + * implemented — Task 15 wires its behavior. */ +export type DepsSubverb = "sync" | "check" | "list"; + +const DEPS_SUBVERBS: readonly DepsSubverb[] = ["sync", "check", "list"]; + +export interface DepsFlags { + subverb: DepsSubverb; + /** Dependency names to narrow to — `sync`'s `[…]`. Empty means "all + * declared dependencies." Accepted (and ignored) by every subverb rather + * than rejected outright: the positional grammar is one shape for all + * three, and only `sync` gives the list meaning today. */ + names: string[]; + /** `sync` only: plan and report, write nothing. */ + dryRun: boolean; +} + +/** The flag table `parseDepsArgs` parses. Exported so the help text can be + * gated against it. Deliberately carries no "format" key: `--format` is a + * GLOBAL flag `index.ts` strips from argv before any command's parser ever + * runs (see `MIGRATE_OPTIONS`'s "--migration-format, not --format" note + * above) — `meta deps` becomes format-aware by index.ts adding "deps" to + * `FORMAT_AWARE_COMMANDS` and passing the resolved `fmt` through, exactly as + * gen/verify/migrate do, not by re-declaring the flag here. */ +export const DEPS_OPTIONS = { + "dry-run": { type: "boolean", default: false }, +} as const; + +export function parseDepsArgs(argv: string[]): DepsFlags { + const { values, positionals } = parseArgs({ + args: argv, + options: DEPS_OPTIONS, + strict: true, + allowPositionals: true, + }); + + const [subverbRaw, ...names] = positionals; + if (subverbRaw === undefined) { + throw new Error( + "meta deps requires a subcommand: sync | check | list. Try `meta deps sync`.", + ); + } + if (!DEPS_SUBVERBS.includes(subverbRaw as DepsSubverb)) { + throw new Error( + `meta deps: unknown subcommand "${subverbRaw}"; expected one of: ${DEPS_SUBVERBS.join(", ")}.`, + ); + } + + return { + subverb: subverbRaw as DepsSubverb, + names, + dryRun: !!values["dry-run"], + }; +} diff --git a/server/typescript/packages/cli/src/lib/dependency-sync.ts b/server/typescript/packages/cli/src/lib/dependency-sync.ts new file mode 100644 index 000000000..d77e9108e --- /dev/null +++ b/server/typescript/packages/cli/src/lib/dependency-sync.ts @@ -0,0 +1,461 @@ +// server/typescript/packages/cli/src/lib/dependency-sync.ts +// +// FR-023 Phase 1a, Task 14 — the algorithm `meta deps sync` runs (DESIGN §4.1 +// steps 1-5, 7-8; step 6, the usage-aware classifier, is superseded — DESIGN +// §11.3, "the committed artifact's diff in the consumer's history is the +// review"). Five functions, split so `--dry-run` is "run everything except +// the write": `readManifestDir` / `validateAgainstSpec` / `standaloneLoadCheck` +// read and validate one already-resolved dependency directory; +// `planSync` composes them over every declared dependency (transport +// resolution, per-dependency validation, the cross-dependency node-collision +// check); `applySync` is the only function that touches the filesystem. +import { existsSync, statSync } from "node:fs"; +import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; +import { + codeSource, + InMemoryStringSource, + MetaDataLoader, + METAMODEL_VERSION, + packageOfResolutionKey, + ParseError, +} from "@metaobjectsdev/metadata"; +import { + DEPS_DIR, + DEFAULT_METAOBJECTS_DIR, + MANIFEST_FILE, + DependencyManifestSchema, + dependencyName, + sha256Integrity, + writeLock, + type DependencyManifest, + type DependencySpec, + type Lock, + type LockEntry, +} from "@metaobjectsdev/sdk"; + +/** `dependency-sync.ts`'s own `ParseError` factory for every + * `ERR_DEPENDENCY_MANIFEST_INVALID` case (DESIGN §4.1 step 2 and step 4): + * absent/invalid manifest, name mismatch, missing artifact, hash mismatch, + * a standalone load failure, or a `nodes`/`packages` mismatch against what + * the artifact actually declares. One thrower so every case names the + * dependency and carries the same code. */ +function manifestInvalid(name: string, caller: string, detail: string): never { + throw new ParseError(`dependency "${name}": ${detail}`, { + code: "ERR_DEPENDENCY_MANIFEST_INVALID", + source: codeSource(caller), + }); +} + +/** The MAJOR half of a `major.minor` metamodel version string — mirrors the + * private helper of the same name in `sdk/src/dependencies.ts` (not + * exported there; the contract, "compare on the major alone" per ADR-0035 + * Amendment 2, is small enough to restate rather than widen that module's + * public surface for one caller). */ +function metamodelMajor(version: string): string { + return version.split(".")[0] ?? version; +} + +/** The MINOR half, as a number — 0 when absent (defensive; the manifest + * schema's `^\d+\.\d+$` regex already guarantees a minor is present). */ +function metamodelMinor(version: string): number { + return Number(version.split(".")[1] ?? 0); +} + +function sameSortedArray(a: readonly string[], b: readonly string[]): boolean { + if (a.length !== b.length) return false; + const left = [...a].sort(); + const right = [...b].sort(); + return left.every((v, i) => v === right[i]); +} + +/** + * DESIGN §4.1 step 1 — resolve a dependency spec's transport to a directory. + * `npm` / `python` refuse by design (Phase 1a ships `path` only — DESIGN + * §11.3's transports table); a `path` that does not resolve to a directory on + * disk is the SAME code (`ERR_DEPENDENCY_UNRESOLVED`) with a different + * detail, because both are "the transport could not locate a directory + * holding metaobjects.pkg.json" — the manifest read is a separate step + * (`readManifestDir`) that only runs once a directory is in hand. + */ +export function resolveDependencyDir(configDir: string, spec: DependencySpec): string { + const name = dependencyName(spec); + if ("path" in spec) { + const dir = resolve(configDir, spec.path); + if (!existsSync(dir) || !statSync(dir).isDirectory()) { + throw new ParseError( + `dependency "${name}": the "path" transport resolved to ${dir}, but no directory exists there`, + { code: "ERR_DEPENDENCY_UNRESOLVED", source: codeSource("resolveDependencyDir") }, + ); + } + return dir; + } + const kind = "npm" in spec ? "npm" : "python"; + throw new ParseError( + `dependency "${name}": transport \`${kind}\` is not supported by this toolchain yet; use \`path\``, + { code: "ERR_DEPENDENCY_UNRESOLVED", source: codeSource("resolveDependencyDir") }, + ); +} + +export interface ManifestRead { + readonly manifest: DependencyManifest; + readonly artifactContent: string; +} + +/** + * DESIGN §4.1 step 2 — read `/metaobjects.pkg.json`, strict, and verify + * it is internally consistent with the artifact sitting beside it. Every + * failure here is `ERR_DEPENDENCY_MANIFEST_INVALID`: absent, not JSON, fails + * the schema, names a missing artifact, or the artifact's bytes don't hash + * to the recorded `integrity` — a publisher who bypassed `sharedModelFile()` + * is caught here, before the (more expensive) standalone load in + * {@link standaloneLoadCheck}. + */ +export async function readManifestDir(dir: string, name: string): Promise { + const manifestPath = join(dir, MANIFEST_FILE); + let raw: string; + try { + raw = await readFile(manifestPath, "utf8"); + } catch { + manifestInvalid(name, "readManifestDir", `no ${MANIFEST_FILE} found at ${manifestPath}`); + } + + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch (err) { + manifestInvalid(name, "readManifestDir", `${manifestPath} is not valid JSON: ${(err as Error).message}`); + } + + const result = DependencyManifestSchema.safeParse(parsed); + if (!result.success) { + const issues = result.error.issues + .map((i) => `${i.path.join(".") || "(root)"}: ${i.message}`) + .join("; "); + manifestInvalid(name, "readManifestDir", `${manifestPath} failed schema validation — ${issues}`); + } + const manifest = result.data; + + const artifactPath = join(dir, manifest.artifact); + let artifactContent: string; + try { + artifactContent = await readFile(artifactPath, "utf8"); + } catch { + manifestInvalid( + name, + "readManifestDir", + `the manifest names artifact "${manifest.artifact}", but it is missing at ${artifactPath}`, + ); + } + + const actual = sha256Integrity(artifactContent); + if (actual !== manifest.integrity) { + manifestInvalid( + name, + "readManifestDir", + `${artifactPath} hashes to ${actual}, but the manifest records ${manifest.integrity}`, + ); + } + + return { manifest, artifactContent }; +} + +/** + * DESIGN §4.1 step 2 (the `name ≠ spec.name` arm) + step 3 (`metamodelVersion` + * major/minor). Returns an advisory warning string when the dependency was + * published against a newer metamodel MINOR (the metadata contract still + * loads; a MAJOR mismatch throws — ADR-0035 Amendment 2: the metadata + * contract is promised on the major alone), `undefined` otherwise. + */ +export function validateAgainstSpec(manifest: DependencyManifest, spec: DependencySpec): string | undefined { + const name = dependencyName(spec); + if (manifest.name !== name) { + manifestInvalid( + name, + "validateAgainstSpec", + `the manifest declares name "${manifest.name}", but it is declared in config as "${name}"`, + ); + } + + const manifestMajor = metamodelMajor(manifest.metamodelVersion); + const toolchainMajor = metamodelMajor(METAMODEL_VERSION); + if (manifestMajor !== toolchainMajor) { + throw new ParseError( + `dependency "${name}" was published against metamodel ${manifest.metamodelVersion}; this ` + + `toolchain speaks ${METAMODEL_VERSION}. A different metamodel MAJOR is a different metadata ` + + `contract — upgrade the toolchain, or use a release of "${name}" built against it.`, + { code: "ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE", source: codeSource("validateAgainstSpec") }, + ); + } + + if (metamodelMinor(manifest.metamodelVersion) > metamodelMinor(METAMODEL_VERSION)) { + return ( + `dependency "${name}" was published against metamodel ${manifest.metamodelVersion}, newer than ` + + `this toolchain's ${METAMODEL_VERSION} — vocabulary it uses that this toolchain does not yet ` + + `have may fail to load.` + ); + } + return undefined; +} + +/** + * DESIGN §4.1 step 4 — load the artifact standalone with CORE providers only + * (a consumer loads it with ITS OWN providers, never the publisher's; a + * `MetaDataLoader` built with no `registry` option composes core providers by + * default), strict, and check it declares EXACTLY the manifest's `nodes` and + * `packages` — a publisher who hand-edited the manifest (or bypassed + * `sharedModelFile()` entirely) is caught here. + */ +export async function standaloneLoadCheck(manifest: DependencyManifest, artifactContent: string): Promise { + const loader = new MetaDataLoader({ strict: true }); + const loaded = await loader.load([ + new InMemoryStringSource(artifactContent, { id: manifest.artifact }), + ]); + if (loaded.errors.length > 0) { + manifestInvalid( + manifest.name, + "standaloneLoadCheck", + `the artifact does not load standalone: ${loaded.errors[0]!.message}`, + ); + } + + // ADR-0039 sanctioned own-only case: `MetaRoot` never `extends`, so its own + // children ARE its effective children (the same reasoning + // `sharedModelFile()`'s step 2 documents on the publisher side). + const nodes = loaded.root.ownChildren().map((n) => n.resolutionKey()); + const packages = [...new Set(nodes.map((n) => packageOfResolutionKey(n)))]; + + if (!sameSortedArray(nodes, manifest.nodes)) { + manifestInvalid( + manifest.name, + "standaloneLoadCheck", + `the artifact's top-level nodes (${[...nodes].sort().join(", ") || "(none)"}) do not match the ` + + `manifest's "nodes" (${[...manifest.nodes].sort().join(", ") || "(none)"})`, + ); + } + if (!sameSortedArray(packages, manifest.packages)) { + manifestInvalid( + manifest.name, + "standaloneLoadCheck", + `the artifact's packages (${[...packages].sort().join(", ") || "(none)"}) do not match the ` + + `manifest's "packages" (${[...manifest.packages].sort().join(", ") || "(none)"})`, + ); + } +} + +/** The declared spec's transport, verbatim, minus `name` — a lock entry's + * `resolvedFrom` (DESIGN §3.3). */ +function resolvedFromOf(spec: DependencySpec): LockEntry["resolvedFrom"] { + if ("path" in spec) return { path: spec.path }; + if ("npm" in spec) return spec.dir === undefined ? { npm: spec.npm } : { npm: spec.npm, dir: spec.dir }; + return spec.dir === undefined ? { python: spec.python } : { python: spec.python, dir: spec.dir }; +} + +/** One dependency this run will (re)sync — either brand new or a changed/ + * unchanged re-sync of one already in the lock. */ +export interface SyncedDependencyPlan { + readonly name: string; + readonly action: "sync" | "unchanged"; + readonly manifest: DependencyManifest; + readonly artifactContent: string; + readonly resolvedFrom: LockEntry["resolvedFrom"]; + readonly oldEntry: LockEntry | undefined; +} + +/** A lock entry whose dependency is no longer declared in config. */ +export interface PrunedDependencyPlan { + readonly name: string; + readonly entry: LockEntry; +} + +export interface SyncPlan { + /** Target dependencies (all declared, or the requested subset), in name order. */ + readonly plans: readonly SyncedDependencyPlan[]; + /** Declared-but-not-targeted lock entries (only when a name filter was + * given) — carried into the new lock untouched. */ + readonly carryForward: ReadonlyMap; + /** Lock entries whose dependency config no longer declares, in name order. */ + readonly prunes: readonly PrunedDependencyPlan[]; + /** Advisory metamodel-minor-ahead warnings, one per affected dependency. */ + readonly warnings: readonly string[]; +} + +/** + * DESIGN §4.1 steps 1-5, for every declared dependency (or the requested + * `filterNames` subset), in name order. Read-only: resolves each transport, + * validates its manifest and artifact, and checks the WHOLE resulting node + * set (freshly-resolved targets plus whatever untouched lock entries carry + * forward) for collisions — nothing is written here, which is what makes + * `--dry-run` "run this and skip `applySync`." + */ +export async function planSync( + configDir: string, + specs: readonly DependencySpec[], + lock: Lock | undefined, + filterNames: readonly string[] = [], +): Promise { + const allNames = specs.map(dependencyName); + const declared = new Set(allNames); + + let targets: DependencySpec[]; + if (filterNames.length === 0) { + targets = [...specs]; + } else { + const unknown = filterNames.filter((n) => !declared.has(n)); + if (unknown.length > 0) { + throw new ParseError( + `meta deps sync: not declared in ${DEFAULT_METAOBJECTS_DIR}/config.json: ${unknown.join(", ")}` + + (allNames.length > 0 + ? ` (declared: ${[...declared].sort().join(", ")})` + : " (no dependencies declared)"), + { code: "ERR_DEPENDENCY_UNRESOLVED", source: codeSource("planSync") }, + ); + } + targets = specs.filter((s) => filterNames.includes(dependencyName(s))); + } + targets.sort((a, b) => dependencyName(a).localeCompare(dependencyName(b))); + + const targetNames = new Set(targets.map(dependencyName)); + const oldEntries = lock?.dependencies ?? {}; + + // Node ownership seeds from untouched (carried-forward) lock entries, so a + // freshly-resolved target's nodes are checked against what already stands, + // not just against each other. + const nodeOwner = new Map(); + const carryForward = new Map(); + for (const [name, entry] of Object.entries(oldEntries)) { + if (declared.has(name) && !targetNames.has(name)) { + carryForward.set(name, entry); + for (const node of entry.nodes) nodeOwner.set(node, name); + } + } + + const warnings: string[] = []; + const plans: SyncedDependencyPlan[] = []; + for (const spec of targets) { + const name = dependencyName(spec); + const dir = resolveDependencyDir(configDir, spec); + const { manifest, artifactContent } = await readManifestDir(dir, name); + const warning = validateAgainstSpec(manifest, spec); + if (warning !== undefined) warnings.push(warning); + await standaloneLoadCheck(manifest, artifactContent); + + for (const node of manifest.nodes) { + const prior = nodeOwner.get(node); + if (prior !== undefined && prior !== name) { + throw new ParseError( + `dependencies "${prior}" and "${name}" both export "${node}" — one fully-qualified node ` + + `cannot come from two places, and whichever loaded second would silently win`, + { code: "ERR_DEPENDENCY_NODE_COLLISION", source: codeSource("planSync") }, + ); + } + nodeOwner.set(node, name); + } + + const oldEntry = oldEntries[name]; + const action: "sync" | "unchanged" = + oldEntry !== undefined && oldEntry.integrity === manifest.integrity ? "unchanged" : "sync"; + plans.push({ name, action, manifest, artifactContent, resolvedFrom: resolvedFromOf(spec), oldEntry }); + } + + const prunes: PrunedDependencyPlan[] = Object.entries(oldEntries) + .filter(([name]) => !declared.has(name)) + .map(([name, entry]) => ({ name, entry })) + .sort((a, b) => a.name.localeCompare(b.name)); + + return { plans, carryForward, prunes, warnings }; +} + +export interface DependencyReportLine { + readonly name: string; + readonly action: "sync" | "unchanged" | "prune"; + readonly line: string; +} + +export interface ApplySyncResult { + readonly lock: Lock; + readonly report: readonly DependencyReportLine[]; +} + +/** First 8 hex chars after the `sha256-` prefix — the report format's ``. */ +function hash8(integrity: string): string { + const prefix = "sha256-"; + return integrity.slice(prefix.length, prefix.length + 8); +} + +/** + * DESIGN §4.1 steps 7-8. `opts.dryRun` runs the exact same reporting pass and + * writes nothing — no directory emptied, no artifact copied, no lock touched + * — which is what makes `--dry-run` a true preview of `planSync`'s verdict + * rather than a second code path that could drift from it. + */ +export async function applySync( + configDir: string, + plan: SyncPlan, + opts: { readonly dryRun: boolean }, +): Promise { + const dependencies: Record = {}; + for (const [name, entry] of plan.carryForward) dependencies[name] = entry; + + const report: DependencyReportLine[] = []; + + for (const p of plan.plans) { + const newEntry: LockEntry = { + version: p.manifest.version, + metamodelVersion: p.manifest.metamodelVersion, + artifact: p.manifest.artifact, + integrity: p.manifest.integrity, + packages: p.manifest.packages, + nodes: p.manifest.nodes, + resolvedFrom: p.resolvedFrom, + }; + dependencies[p.name] = newEntry; + + if (p.action === "unchanged") { + report.push({ name: p.name, action: "unchanged", line: `unchanged ${p.name} ${p.manifest.version}` }); + continue; + } + + const oldVersion = p.oldEntry?.version ?? "(new)"; + const oldHash = p.oldEntry !== undefined ? hash8(p.oldEntry.integrity) : "(new)"; + const newHash = hash8(p.manifest.integrity); + const verb = opts.dryRun ? "would sync" : "synced"; + report.push({ + name: p.name, + action: "sync", + line: `${verb} ${p.name} ${oldVersion}→${p.manifest.version} (${oldHash}→${newHash})`, + }); + + if (!opts.dryRun) { + const depDir = join(configDir, DEFAULT_METAOBJECTS_DIR, DEPS_DIR, p.name); + await rm(depDir, { recursive: true, force: true }); + await mkdir(depDir, { recursive: true }); + await writeFile(join(depDir, p.manifest.artifact), p.artifactContent, "utf8"); + } + } + + for (const pr of plan.prunes) { + report.push({ + name: pr.name, + action: "prune", + line: opts.dryRun ? `would prune ${pr.name}` : `pruned ${pr.name}`, + }); + if (!opts.dryRun) { + const depDir = join(configDir, DEFAULT_METAOBJECTS_DIR, DEPS_DIR, pr.name); + await rm(depDir, { recursive: true, force: true }); + } + } + + const lock: Lock = { schema_version: 1, dependencies }; + // Nothing to reconcile (no target dependency and no prune) is a true no-op — + // a project declaring no dependencies (and never having synced any) must + // not gain a `deps.lock.json` from a bare `meta deps sync`. `plan.prunes` + // being non-empty is what still writes when EVERY dependency was just + // removed from config: that run must produce a lock with an EMPTY + // `dependencies` map, not leave the stale one standing. + if (!opts.dryRun && (plan.plans.length > 0 || plan.prunes.length > 0)) { + await writeLock(configDir, lock); + } + + return { lock, report }; +} diff --git a/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap b/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap index 8041fce81..5cdd79801 100644 --- a/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap +++ b/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap @@ -14,6 +14,9 @@ COMMANDS: gen [...] Codegen TS targets from your declared metadata eject Copy a reference generator into codegen/generators/ to own it (any time after init) eject --list List every ejectable generator name, grouped by package + deps sync [...] Resolve declared dependencies (path transport): sync the committed + snapshot + sha256 lock (.metaobjects/deps/, .metaobjects/deps.lock.json) + deps list One line per locked dependency: name, version, hash, node/package summary types [query] Search the metadata vocabulary (types, subtypes, @attrs) by name or description export Flatten loaded metadata to one canonical JSON artifact docs [] --out Generate neutral metadata documentation (entity + template pages; --site for HTML site) @@ -27,7 +30,7 @@ COMMANDS: GLOBAL OPTIONS: --cwd , -C Run as if launched from (default: current directory) --format Output format (default: toon on non-TTY, text on TTY). - Honored by gen, verify, migrate, types; every other + Honored by gen, verify, migrate, types, deps; every other command prints text and says so if you pass it. \`types\` is the one exception to the default: it prints TEXT unless you ask for a format, on a TTY or not, because its diff --git a/server/typescript/packages/cli/test/deps-sync.test.ts b/server/typescript/packages/cli/test/deps-sync.test.ts new file mode 100644 index 000000000..4ff5e38ec --- /dev/null +++ b/server/typescript/packages/cli/test/deps-sync.test.ts @@ -0,0 +1,304 @@ +// FR-023 Phase 1a, Task 14 — `meta deps sync` (`path` transport) and +// `meta deps list`. +// +// The scaffold: a PUBLISHER directory (`acme-common/metaobjects/`) holding a +// hand-written manifest (`metaobjects.pkg.json`) beside the pinned +// `dependency-conformance` artifact, and a CONSUMER directory +// (`consumer/.metaobjects/config.json`) declaring a `path` dependency on it — +// laid out as SIBLINGS under one temp root, exactly as the Global Constraints +// `CONFIG_REF` (`"path": "../acme-common/metaobjects"`) expects. +// +// `meta deps sync` is the thing that turns that declaration into a committed +// snapshot (`.metaobjects/deps/acme-common/acme-common.metaobjects.json`) and +// a sha256-pinned lock (`.metaobjects/deps.lock.json`); `meta deps list` +// reports what the lock holds. +import { describe, test, expect, beforeEach, afterEach, afterAll } from "bun:test"; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { METAMODEL_VERSION } from "@metaobjectsdev/metadata"; +import { sha256Integrity } from "@metaobjectsdev/sdk"; +import { run } from "../src/index.js"; + +const DEP_NAME = "acme-common"; +const ARTIFACT_BASENAME = "acme-common.metaobjects.json"; +const MANIFEST_BASENAME = "metaobjects.pkg.json"; +const NODES = ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"]; + +// The corpus this repo already pins (fixtures/dependency-conformance/README.md, +// and the Global Constraints "Hash format" block) — read as raw bytes so the +// snapshot-equality assertions below compare byte-for-byte, never a re-typed +// (and possibly re-formatted) copy. +const CORPUS_ARTIFACTS = resolve( + import.meta.dirname, + "../../../../../fixtures/dependency-conformance/artifacts", +); +const V1_BYTES = readFileSync(join(CORPUS_ARTIFACTS, "acme-common-v1.json")); +const WIDENED_BYTES = readFileSync(join(CORPUS_ARTIFACTS, "acme-common-v1-widened.json")); +const V1_HASH = sha256Integrity(V1_BYTES); +const WIDENED_HASH = sha256Integrity(WIDENED_BYTES); + +/** First 8 hex chars after the "sha256-" prefix — `meta deps sync`'s report format. */ +function hash8(integrity: string): string { + return integrity.slice("sha256-".length, "sha256-".length + 8); +} + +/** The consumer's own model: one entity it owns. Mirrors the shape + * `gen-imported-nodes.test.ts` / `verify-requirements-imported.test.ts` use — + * present so the post-sync full-collection load (brief step 8) has a real + * combined model (own + dependency) to prove loadable, not an empty tree. */ +const APP = JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Order", + children: [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +/** Global Constraints `CONFIG_REF`, verbatim. */ +const CONFIG_REF = JSON.stringify({ + schema_version: 1, + sources: [], + dependencies: [{ name: DEP_NAME, path: "../acme-common/metaobjects" }], +}); + +function manifestJson(opts: { version: string; integrity: string; nodes: string[] }): string { + return JSON.stringify( + { + schema_version: 1, + name: DEP_NAME, + version: opts.version, + metamodelVersion: METAMODEL_VERSION, + artifact: ARTIFACT_BASENAME, + integrity: opts.integrity, + packages: ["acme::common"], + nodes: opts.nodes, + }, + null, + 2, + ); +} + +/** Global Constraints `LOCK_V1`, using the live `METAMODEL_VERSION` constant + * rather than a hardcoded "1.0" (same rationale as the sibling FR-023 + * fixtures already in this package: `gen-imported-nodes.test.ts` etc.). */ +const LOCK_V1 = { + schema_version: 1, + dependencies: { + [DEP_NAME]: { + version: "1.0.0", + metamodelVersion: METAMODEL_VERSION, + resolvedFrom: { path: "../acme-common/metaobjects" }, + artifact: ARTIFACT_BASENAME, + integrity: V1_HASH, + packages: ["acme::common"], + nodes: NODES, + }, + }, +}; + +const dirs: string[] = []; +afterAll(() => { + for (const d of dirs) rmSync(d, { recursive: true, force: true }); +}); + +/** A publisher + consumer pair, laid out as siblings under one temp root. */ +function setupProject(): { root: string; consumerRoot: string; publisherDir: string } { + const root = mkdtempSync(join(tmpdir(), "deps-sync-")); + dirs.push(root); + + const consumerRoot = join(root, "consumer"); + mkdirSync(join(consumerRoot, "metaobjects"), { recursive: true }); + writeFileSync(join(consumerRoot, "metaobjects", "meta.app.json"), APP, "utf8"); + mkdirSync(join(consumerRoot, ".metaobjects"), { recursive: true }); + writeFileSync(join(consumerRoot, ".metaobjects", "config.json"), CONFIG_REF, "utf8"); + + const publisherDir = join(root, "acme-common", "metaobjects"); + mkdirSync(publisherDir, { recursive: true }); + writeFileSync(join(publisherDir, ARTIFACT_BASENAME), V1_BYTES); + writeFileSync( + join(publisherDir, MANIFEST_BASENAME), + manifestJson({ version: "1.0.0", integrity: V1_HASH, nodes: NODES }), + "utf8", + ); + + return { root, consumerRoot, publisherDir }; +} + +let out: string[]; +let err: string[]; +let origLog: typeof console.log; +let origErr: typeof console.error; + +beforeEach(() => { + out = []; + err = []; + origLog = console.log; + origErr = console.error; + console.log = (...a: unknown[]) => { + out.push(a.map(String).join(" ")); + }; + console.error = (...a: unknown[]) => { + err.push(a.map(String).join(" ")); + }; +}); +afterEach(() => { + console.log = origLog; + console.error = origErr; +}); + +describe("meta deps sync — path transport (FR-023 Phase 1a Task 14)", () => { + test("(a) sync creates a byte-equal snapshot and a lock equal to LOCK_V1", async () => { + const { consumerRoot } = setupProject(); + + const exit = await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(0); + + const snapPath = join(consumerRoot, ".metaobjects", "deps", DEP_NAME, ARTIFACT_BASENAME); + expect(readFileSync(snapPath)).toEqual(V1_BYTES); + + const lock = JSON.parse( + readFileSync(join(consumerRoot, ".metaobjects", "deps.lock.json"), "utf8"), + ); + expect(lock).toEqual(LOCK_V1); + }); + + test("(b) a second sync prints 'unchanged' and leaves the lock bytes identical", async () => { + const { consumerRoot } = setupProject(); + await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + const lockPath = join(consumerRoot, ".metaobjects", "deps.lock.json"); + const before = readFileSync(lockPath, "utf8"); + + out = []; + err = []; + const exit = await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(0); + expect([...out, ...err].join("\n")).toContain("unchanged"); + + expect(readFileSync(lockPath, "utf8")).toBe(before); + }); + + test("(c) --dry-run writes nothing", async () => { + const { consumerRoot } = setupProject(); + + const exit = await run(["deps", "sync", "--dry-run", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(0); + expect(existsSync(join(consumerRoot, ".metaobjects", "deps.lock.json"))).toBe(false); + expect(existsSync(join(consumerRoot, ".metaobjects", "deps", DEP_NAME))).toBe(false); + }); + + test("(d) a widened artifact syncs and reports the hash change", async () => { + const { consumerRoot, publisherDir } = setupProject(); + await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + + writeFileSync(join(publisherDir, ARTIFACT_BASENAME), WIDENED_BYTES); + writeFileSync( + join(publisherDir, MANIFEST_BASENAME), + manifestJson({ version: "1.1.0", integrity: WIDENED_HASH, nodes: NODES }), + "utf8", + ); + + out = []; + err = []; + const exit = await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(0); + const printed = [...out, ...err].join("\n"); + expect(printed).toContain("synced acme-common"); + expect(printed).toContain(`(${hash8(V1_HASH)}→${hash8(WIDENED_HASH)})`); + + const lock = JSON.parse( + readFileSync(join(consumerRoot, ".metaobjects", "deps.lock.json"), "utf8"), + ); + expect(lock.dependencies[DEP_NAME].integrity).toBe(WIDENED_HASH); + expect( + readFileSync(join(consumerRoot, ".metaobjects", "deps", DEP_NAME, ARTIFACT_BASENAME)), + ).toEqual(WIDENED_BYTES); + }); + + test("(e) removing the dependency from config prunes the lock entry and the snapshot dir", async () => { + const { consumerRoot } = setupProject(); + await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + expect(existsSync(join(consumerRoot, ".metaobjects", "deps", DEP_NAME))).toBe(true); + + writeFileSync( + join(consumerRoot, ".metaobjects", "config.json"), + JSON.stringify({ schema_version: 1, sources: [], dependencies: [] }), + "utf8", + ); + + const exit = await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(0); + + const lock = JSON.parse( + readFileSync(join(consumerRoot, ".metaobjects", "deps.lock.json"), "utf8"), + ); + expect(lock.dependencies).toEqual({}); + expect(existsSync(join(consumerRoot, ".metaobjects", "deps", DEP_NAME))).toBe(false); + }); + + test('(f) a manifest whose "nodes" omits Customer fails the sync', async () => { + const { consumerRoot, publisherDir } = setupProject(); + // The ARTIFACT is untouched (still declares Customer) — only the + // manifest's declared "nodes" is wrong, so this exercises the + // standalone-load node-set check, not the hash check. + writeFileSync( + join(publisherDir, MANIFEST_BASENAME), + manifestJson({ + version: "1.0.0", + integrity: V1_HASH, + nodes: ["acme::common::Address", "acme::common::Audited"], + }), + "utf8", + ); + + const exit = await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(1); + const printed = [...out, ...err].join("\n"); + expect(printed).toContain("acme::common::Customer"); + expect(existsSync(join(consumerRoot, ".metaobjects", "deps.lock.json"))).toBe(false); + }); + + test("(g) an npm transport spec refuses — 'not supported by this toolchain yet'", async () => { + const root = mkdtempSync(join(tmpdir(), "deps-sync-npm-")); + dirs.push(root); + mkdirSync(join(root, "metaobjects"), { recursive: true }); + writeFileSync(join(root, "metaobjects", "meta.app.json"), APP, "utf8"); + mkdirSync(join(root, ".metaobjects"), { recursive: true }); + writeFileSync( + join(root, ".metaobjects", "config.json"), + JSON.stringify({ + schema_version: 1, + sources: [], + dependencies: [{ name: "x", npm: "@acme/model" }], + }), + "utf8", + ); + + const exit = await run(["deps", "sync", "--format", "text", "--cwd", root]); + expect(exit).toBe(1); + expect([...out, ...err].join("\n")).toContain("not supported by this toolchain yet"); + }); + + test("(h) list after a sync prints one summary line per lock entry", async () => { + const { consumerRoot } = setupProject(); + await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + + out = []; + err = []; + const exit = await run(["deps", "list", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(0); + expect([...out, ...err].join("\n")).toContain( + `acme-common 1.0.0 ${hash8(V1_HASH)} 3 node(s) acme::common`, + ); + }); +}); diff --git a/server/typescript/packages/cli/test/unit/args-deps.test.ts b/server/typescript/packages/cli/test/unit/args-deps.test.ts new file mode 100644 index 000000000..a72508bb6 --- /dev/null +++ b/server/typescript/packages/cli/test/unit/args-deps.test.ts @@ -0,0 +1,61 @@ +// FR-023 Phase 1a, Task 14 — `parseDepsArgs` subverb + flag parsing. +// +// `meta deps` takes a required subverb positional (`sync | check | list`), any +// further positionals are dependency NAMES (a filter — only meaningful to +// `sync` in this task; `check` is Task 15's), and `--dry-run` is the one flag +// this task adds. `--format` is a GLOBAL flag `index.ts` strips before a +// command ever sees its argv (see `args.ts`'s `MIGRATE_OPTIONS` comment: "the +// flag is --migration-format, not --format: `--format` is already consumed by +// the top-level parser") — so, unlike the task brief's shorthand, DEPS_OPTIONS +// carries no "format" key; `meta deps` is format-aware the same way +// gen/verify/migrate are, via the `fmt` parameter `index.ts` passes to the +// command, not via its own parser. +import { describe, test, expect } from "bun:test"; +import { parseDepsArgs } from "../../src/lib/args.js"; + +describe("parseDepsArgs", () => { + test("sync with no names — dry-run defaults false", () => { + expect(parseDepsArgs(["sync"])).toEqual({ subverb: "sync", names: [], dryRun: false }); + }); + + test("sync with one or more dependency names", () => { + expect(parseDepsArgs(["sync", "acme-common"])).toEqual({ + subverb: "sync", + names: ["acme-common"], + dryRun: false, + }); + expect(parseDepsArgs(["sync", "acme-common", "other-lib"])).toEqual({ + subverb: "sync", + names: ["acme-common", "other-lib"], + dryRun: false, + }); + }); + + test("--dry-run", () => { + expect(parseDepsArgs(["sync", "--dry-run"])).toEqual({ + subverb: "sync", + names: [], + dryRun: true, + }); + }); + + test("list", () => { + expect(parseDepsArgs(["list"])).toEqual({ subverb: "list", names: [], dryRun: false }); + }); + + test("check — parses; this task does not implement its behavior", () => { + expect(parseDepsArgs(["check"])).toEqual({ subverb: "check", names: [], dryRun: false }); + }); + + test("no subverb at all is a usage error", () => { + expect(() => parseDepsArgs([])).toThrow(/subcommand/); + }); + + test("an unknown subverb is a usage error", () => { + expect(() => parseDepsArgs(["bogus"])).toThrow(/unknown subcommand "bogus"/); + }); + + test("an unknown flag is a usage error", () => { + expect(() => parseDepsArgs(["sync", "--foo"])).toThrow(/Unknown option '--foo'/); + }); +}); From 06490cccfe8ba300d1c886607ab8d5862063c7d5 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 00:25:20 -0400 Subject: [PATCH 40/62] =?UTF-8?q?fix(cli):=20deps=20sync=20=E2=80=94=20gua?= =?UTF-8?q?rd=20a=20corrupted=20lock,=20gate=20deps=20flags=20in=20--help,?= =?UTF-8?q?=20dedupe=20hash8,=20cover=20the=20name=20filter=20(FR-023)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix round 1 on Task 14: catch readLock's throw in `deps sync`/`deps list` (a corrupted committed lock previously crashed via an unhandled rejection — the same failure class already guarded for a bad config.json); add `deps` to help-lists-every-flag.test.ts's COMMANDS table so its flags are actually gated going forward; export one `hash8` helper from dependency-sync.ts (built on INTEGRITY_PREFIX) instead of three independent inlined copies; and add coverage for `meta deps sync `'s carry-forward + cross- dependency collision seeding (no defect found — confirms the existing `planSync` behavior). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../packages/cli/src/commands/deps.ts | 45 +++- .../packages/cli/src/lib/dependency-sync.ts | 11 +- .../packages/cli/test/deps-sync.test.ts | 214 +++++++++++++++++- .../cli/test/help-lists-every-flag.test.ts | 2 + 4 files changed, 260 insertions(+), 12 deletions(-) diff --git a/server/typescript/packages/cli/src/commands/deps.ts b/server/typescript/packages/cli/src/commands/deps.ts index 52d1d74b6..86d29453a 100644 --- a/server/typescript/packages/cli/src/commands/deps.ts +++ b/server/typescript/packages/cli/src/commands/deps.ts @@ -9,6 +9,7 @@ import { discoverCollectionRoot, loadConfig, loadMemory, + LOCK_FILE, readLock, resolveCollection, type DependencySpec, @@ -19,7 +20,27 @@ import { log } from "../lib/log.js"; import { emitStructured, type OutputFormat } from "../lib/format.js"; import { reportLoadError } from "../lib/load-error.js"; import { collectionLoadOptions } from "../lib/collection-load-options.js"; -import { applySync, planSync } from "../lib/dependency-sync.js"; +import { applySync, hash8, planSync } from "../lib/dependency-sync.js"; + +/** + * `readLock`, converted into a diagnostic that NAMES the file rather than + * letting a corrupted committed lock's raw `JSON.parse`/`ZodError` surface as + * an unhandled rejection. `deps.lock.json` is checked-in, hand-editable, and + * merge-conflictable — a realistic way for it to break — and `bin/meta.ts`'s + * `run(...).then((code) => process.exit(code))` has no top-level `.catch()`, + * so an uncaught throw here would crash the process with a stack trace + * instead of the clean exit code every other failure in this command gets. + */ +async function readLockOrThrow(configDir: string): Promise { + try { + return await readLock(configDir); + } catch (err) { + throw new Error( + `${DEFAULT_METAOBJECTS_DIR}/${LOCK_FILE} is corrupted and could not be read: ${(err as Error).message}. ` + + "Fix it by hand, or delete it and re-run `meta deps sync` to regenerate it.", + ); + } +} /** The declared `dependencies` for the config governing `cwd` — read * DIRECTLY via `loadConfig`, never through `resolveCollection`/`Collection`: @@ -37,7 +58,16 @@ async function declaredDependencies(cwd: string): Promise<{ configDir: string; s } async function runSync(configDir: string, specs: readonly DependencySpec[], flags: DepsFlags, fmt: OutputFormat): Promise { - const lock: Lock | undefined = await readLock(configDir); + let lock: Lock | undefined; + try { + lock = await readLockOrThrow(configDir); + } catch (err) { + // Same convention as `depsCommand`'s `declaredDependencies` catch below: a + // committed project file that fails to parse is a config-class problem, + // exit 2 — distinct from the exit-1 a `sync` verdict failure gets. + log.error((err as Error).message); + return 2; + } let plan; try { @@ -99,7 +129,13 @@ async function runSync(configDir: string, specs: readonly DependencySpec[], flag } async function runList(configDir: string, fmt: OutputFormat): Promise { - const lock = await readLock(configDir); + let lock: Lock | undefined; + try { + lock = await readLockOrThrow(configDir); + } catch (err) { + log.error((err as Error).message); + return 2; + } const entries = Object.entries(lock?.dependencies ?? {}); if (fmt === "text") { @@ -108,9 +144,8 @@ async function runList(configDir: string, fmt: OutputFormat): Promise { return 0; } for (const [name, entry] of entries) { - const hash8 = entry.integrity.slice("sha256-".length, "sha256-".length + 8); log.info( - `${name} ${entry.version} ${hash8} ${entry.nodes.length} node(s) ${entry.packages.join(", ")}`, + `${name} ${entry.version} ${hash8(entry.integrity)} ${entry.nodes.length} node(s) ${entry.packages.join(", ")}`, ); } } else { diff --git a/server/typescript/packages/cli/src/lib/dependency-sync.ts b/server/typescript/packages/cli/src/lib/dependency-sync.ts index d77e9108e..1158f68df 100644 --- a/server/typescript/packages/cli/src/lib/dependency-sync.ts +++ b/server/typescript/packages/cli/src/lib/dependency-sync.ts @@ -23,6 +23,7 @@ import { import { DEPS_DIR, DEFAULT_METAOBJECTS_DIR, + INTEGRITY_PREFIX, MANIFEST_FILE, DependencyManifestSchema, dependencyName, @@ -377,10 +378,12 @@ export interface ApplySyncResult { readonly report: readonly DependencyReportLine[]; } -/** First 8 hex chars after the `sha256-` prefix — the report format's ``. */ -function hash8(integrity: string): string { - const prefix = "sha256-"; - return integrity.slice(prefix.length, prefix.length + 8); +/** First 8 hex chars after the `sha256-` (`INTEGRITY_PREFIX`) prefix — the + * report format's ``. Exported so every caller — `deps.ts`'s `deps + * list` rendering, tests — shares this ONE definition rather than each + * reimplementing the slice with its own inlined `"sha256-"` literal. */ +export function hash8(integrity: string): string { + return integrity.slice(INTEGRITY_PREFIX.length, INTEGRITY_PREFIX.length + 8); } /** diff --git a/server/typescript/packages/cli/test/deps-sync.test.ts b/server/typescript/packages/cli/test/deps-sync.test.ts index 4ff5e38ec..a903cf2d9 100644 --- a/server/typescript/packages/cli/test/deps-sync.test.ts +++ b/server/typescript/packages/cli/test/deps-sync.test.ts @@ -17,7 +17,7 @@ import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; import { METAMODEL_VERSION } from "@metaobjectsdev/metadata"; -import { sha256Integrity } from "@metaobjectsdev/sdk"; +import { INTEGRITY_PREFIX, sha256Integrity } from "@metaobjectsdev/sdk"; import { run } from "../src/index.js"; const DEP_NAME = "acme-common"; @@ -38,9 +38,12 @@ const WIDENED_BYTES = readFileSync(join(CORPUS_ARTIFACTS, "acme-common-v1-widene const V1_HASH = sha256Integrity(V1_BYTES); const WIDENED_HASH = sha256Integrity(WIDENED_BYTES); -/** First 8 hex chars after the "sha256-" prefix — `meta deps sync`'s report format. */ +/** First 8 hex chars after the `INTEGRITY_PREFIX` — `meta deps sync`'s report + * format. Mirrors (deliberately not imports — this is a CLI test, not a + * caller of `dependency-sync.ts`'s internals) the shared `hash8` helper + * exported from `src/lib/dependency-sync.ts`. */ function hash8(integrity: string): string { - return integrity.slice("sha256-".length, "sha256-".length + 8); + return integrity.slice(INTEGRITY_PREFIX.length, INTEGRITY_PREFIX.length + 8); } /** The consumer's own model: one entity it owns. Mirrors the shape @@ -302,3 +305,208 @@ describe("meta deps sync — path transport (FR-023 Phase 1a Task 14)", () => { ); }); }); + +// Fix round 1, FIX 4 — `meta deps sync ` (the positional name filter) +// had zero automated coverage: nothing proved that syncing ONE declared +// dependency by name (a) leaves every OTHER dependency's lock entry +// byte-identical, untouched, and (b) still seeds node-ownership from those +// untouched entries, so a newly-resolved target can still collide against a +// dependency the run never targeted. Task 15's `check` is specified to +// resolve "exactly as sync steps 1-2 do", so this carry-forward/collision +// seeding is machinery a later task builds on. +describe("meta deps sync — the name filter (FR-023 Phase 1a Task 14, fix round 1)", () => { + const SECOND_NAME = "acme-extra"; + const SECOND_ARTIFACT_BASENAME = "acme-extra.metaobjects.json"; + const SECOND_PACKAGES = ["acme::extra"]; + const SECOND_NODES = ["acme::extra::Widget"]; + + // A second, independent publisher — one object.value with a single field, + // small enough to hand-write rather than borrow from the shared corpus + // (which has no second, unrelated dependency fixture; this test needs two). + const SECOND_ARTIFACT_CONTENT = + JSON.stringify( + { + "metadata.root": { + children: [ + { + "object.value": { + name: "Widget", + package: "acme::extra", + children: [{ "field.string": { name: "label" } }], + }, + }, + ], + }, + }, + null, + 2, + ) + "\n"; + const SECOND_HASH = sha256Integrity(SECOND_ARTIFACT_CONTENT); + + /** A manifest with fully overridable `name`/`artifact`/`packages`/`nodes` — + * unlike the top-level `manifestJson`, which is hardcoded to `acme-common` + * in package `acme::common`. Scoped to this describe block only: the + * colliding-manifest scenario below needs `acme-common`'s OWN manifest to + * (incorrectly) declare `acme::extra`'s package/node, which the shared + * helper cannot express. */ + function genericManifest(opts: { + name: string; + artifact: string; + version: string; + integrity: string; + nodes: string[]; + packages: string[]; + }): string { + return JSON.stringify( + { + schema_version: 1, + name: opts.name, + version: opts.version, + metamodelVersion: METAMODEL_VERSION, + artifact: opts.artifact, + integrity: opts.integrity, + packages: opts.packages, + nodes: opts.nodes, + }, + null, + 2, + ); + } + + function setupTwoDependencyProject(): { + consumerRoot: string; + commonPublisherDir: string; + extraPublisherDir: string; + } { + const root = mkdtempSync(join(tmpdir(), "deps-sync-filter-")); + dirs.push(root); + + const consumerRoot = join(root, "consumer"); + mkdirSync(join(consumerRoot, "metaobjects"), { recursive: true }); + writeFileSync(join(consumerRoot, "metaobjects", "meta.app.json"), APP, "utf8"); + mkdirSync(join(consumerRoot, ".metaobjects"), { recursive: true }); + writeFileSync( + join(consumerRoot, ".metaobjects", "config.json"), + JSON.stringify({ + schema_version: 1, + sources: [], + dependencies: [ + { name: DEP_NAME, path: "../acme-common/metaobjects" }, + { name: SECOND_NAME, path: "../acme-extra/metaobjects" }, + ], + }), + "utf8", + ); + + const commonPublisherDir = join(root, "acme-common", "metaobjects"); + mkdirSync(commonPublisherDir, { recursive: true }); + writeFileSync(join(commonPublisherDir, ARTIFACT_BASENAME), V1_BYTES); + writeFileSync( + join(commonPublisherDir, MANIFEST_BASENAME), + manifestJson({ version: "1.0.0", integrity: V1_HASH, nodes: NODES }), + "utf8", + ); + + const extraPublisherDir = join(root, "acme-extra", "metaobjects"); + mkdirSync(extraPublisherDir, { recursive: true }); + writeFileSync(join(extraPublisherDir, SECOND_ARTIFACT_BASENAME), SECOND_ARTIFACT_CONTENT, "utf8"); + writeFileSync( + join(extraPublisherDir, MANIFEST_BASENAME), + genericManifest({ + name: SECOND_NAME, + artifact: SECOND_ARTIFACT_BASENAME, + version: "1.0.0", + integrity: SECOND_HASH, + nodes: SECOND_NODES, + packages: SECOND_PACKAGES, + }), + "utf8", + ); + + return { consumerRoot, commonPublisherDir, extraPublisherDir }; + } + + test("syncing one dependency by name leaves the other's lock entry byte-identical, and still guards against collision", async () => { + const { consumerRoot, commonPublisherDir } = setupTwoDependencyProject(); + + // Baseline: sync everything declared. + expect(await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot])).toBe(0); + + const lockPath = join(consumerRoot, ".metaobjects", "deps.lock.json"); + const afterInitial = JSON.parse(readFileSync(lockPath, "utf8")); + const extraEntryBefore = afterInitial.dependencies[SECOND_NAME]; + expect(extraEntryBefore).toBeDefined(); + + // Widen acme-common ONLY — acme-extra's publisher is never touched again + // in this test. + writeFileSync(join(commonPublisherDir, ARTIFACT_BASENAME), WIDENED_BYTES); + writeFileSync( + join(commonPublisherDir, MANIFEST_BASENAME), + manifestJson({ version: "1.1.0", integrity: WIDENED_HASH, nodes: NODES }), + "utf8", + ); + + out = []; + err = []; + const filteredExit = await run([ + "deps", + "sync", + DEP_NAME, + "--format", + "text", + "--cwd", + consumerRoot, + ]); + expect(filteredExit).toBe(0); + expect([...out, ...err].join("\n")).toContain("synced acme-common"); + + const afterFiltered = JSON.parse(readFileSync(lockPath, "utf8")); + expect(afterFiltered.dependencies[DEP_NAME].integrity).toBe(WIDENED_HASH); + // The untargeted dependency's entry is BYTE-IDENTICAL to what it was + // before this run — the carry-forward path, not a silent re-validation + // or re-copy of something nobody asked to sync. + expect(afterFiltered.dependencies[SECOND_NAME]).toEqual(extraEntryBefore); + + // Now replace acme-common's publisher with an artifact that (incorrectly) + // exports the EXACT fully-qualified node acme-extra already owns + // ("acme::extra::Widget"). A name-filtered `sync acme-common` must still + // catch this against acme-extra's UNTOUCHED lock entry — proving + // `planSync` seeds node ownership from carried-forward entries, not only + // from the targets this run resolves. + const collidingHash = sha256Integrity(SECOND_ARTIFACT_CONTENT); + writeFileSync(join(commonPublisherDir, ARTIFACT_BASENAME), SECOND_ARTIFACT_CONTENT, "utf8"); + writeFileSync( + join(commonPublisherDir, MANIFEST_BASENAME), + genericManifest({ + name: DEP_NAME, + artifact: ARTIFACT_BASENAME, + version: "1.2.0", + integrity: collidingHash, + nodes: SECOND_NODES, + packages: SECOND_PACKAGES, + }), + "utf8", + ); + + out = []; + err = []; + const collisionExit = await run([ + "deps", + "sync", + DEP_NAME, + "--format", + "text", + "--cwd", + consumerRoot, + ]); + expect(collisionExit).toBe(1); + const collisionPrinted = [...out, ...err].join("\n"); + expect(collisionPrinted).toContain(SECOND_NAME); + expect(collisionPrinted).toContain(DEP_NAME); + expect(collisionPrinted).toContain("acme::extra::Widget"); + + // A collision is caught during PLANNING (read-only) — the lock must be + // completely untouched by the failed attempt. + expect(JSON.parse(readFileSync(lockPath, "utf8"))).toEqual(afterFiltered); + }); +}); diff --git a/server/typescript/packages/cli/test/help-lists-every-flag.test.ts b/server/typescript/packages/cli/test/help-lists-every-flag.test.ts index 1881cf0bf..2ec479833 100644 --- a/server/typescript/packages/cli/test/help-lists-every-flag.test.ts +++ b/server/typescript/packages/cli/test/help-lists-every-flag.test.ts @@ -22,6 +22,7 @@ import { describe, expect, test } from "bun:test"; import { AGENT_DOCS_OPTIONS, + DEPS_OPTIONS, EJECT_OPTIONS, EXPORT_OPTIONS, GEN_OPTIONS, @@ -51,6 +52,7 @@ const COMMANDS: ReadonlyArray ["prompt-snapshot", PROMPT_SNAPSHOT_OPTIONS], ["migrate", MIGRATE_OPTIONS], ["eject", EJECT_OPTIONS], + ["deps", DEPS_OPTIONS], ]; describe("every accepted flag appears in its command's --help", () => { From 851938627e42e5d56123f6aca61ba23dc392b4dd Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 00:34:44 -0400 Subject: [PATCH 41/62] test(cli): make the deps sync name-filter test discriminate on filtering itself (FR-023) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix round 2 on Task 14: the round-1 name-filter test exercised real code but could not fail under a mutant that deletes filtering entirely (targets = [...specs] unconditionally) — the untargeted dependency's untouched publisher made "carried forward" and "re-resolved fresh" produce identical results. Adds a test that deletes the untargeted dependency's publisher directory outright before a filtered sync, so re-resolving it (the mutant's behavior) fails while correct filtering succeeds. Verified against the mutant by hand: fails under it, passes reverted. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../packages/cli/test/deps-sync.test.ts | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/server/typescript/packages/cli/test/deps-sync.test.ts b/server/typescript/packages/cli/test/deps-sync.test.ts index a903cf2d9..a211daf53 100644 --- a/server/typescript/packages/cli/test/deps-sync.test.ts +++ b/server/typescript/packages/cli/test/deps-sync.test.ts @@ -509,4 +509,57 @@ describe("meta deps sync — the name filter (FR-023 Phase 1a Task 14, fi // completely untouched by the failed attempt. expect(JSON.parse(readFileSync(lockPath, "utf8"))).toEqual(afterFiltered); }); + + // Fix round 2 — the test above exercises real code but does not + // DISCRIMINATE: a mutant that deletes filtering (`targets = [...specs]` + // unconditionally, ignoring `filterNames`) passes every assertion in it, + // because the untargeted dependency's publisher is untouched either way, so + // "carried forward" and "re-resolved fresh" produce byte-identical results. + // + // This test makes the untargeted dependency IMPOSSIBLE to process + // successfully (its publisher directory is deleted outright), then proves a + // name-filtered sync succeeds anyway. Under correct filtering, the deleted + // dependency is never touched, so its absence is irrelevant. Under the + // mutant above, BOTH dependencies are processed every run, so + // `resolveDependencyDir` throws `ERR_DEPENDENCY_UNRESOLVED` for the deleted + // one and the whole sync fails — the one asymmetry the test above lacked. + test("a filtered sync never re-resolves the untargeted dependency, even when its publisher is gone", async () => { + const { consumerRoot, extraPublisherDir } = setupTwoDependencyProject(); + + // Baseline: sync everything declared. + expect(await run(["deps", "sync", "--format", "text", "--cwd", consumerRoot])).toBe(0); + + const lockPath = join(consumerRoot, ".metaobjects", "deps.lock.json"); + const baseline = JSON.parse(readFileSync(lockPath, "utf8")); + expect(baseline.dependencies[SECOND_NAME]).toBeDefined(); + expect(baseline.dependencies[DEP_NAME]).toBeDefined(); + + // acme-extra can no longer be resolved by ANY path — its whole publisher + // directory is gone. It is still DECLARED in the consumer's config, so a + // correctly-filtered `sync acme-common` must carry its lock entry forward + // untouched rather than attempt to re-resolve it. + rmSync(extraPublisherDir, { recursive: true, force: true }); + + out = []; + err = []; + const exit = await run([ + "deps", + "sync", + DEP_NAME, // acme-extra is never named + "--format", + "text", + "--cwd", + consumerRoot, + ]); + + expect(exit).toBe(0); + + const after = JSON.parse(readFileSync(lockPath, "utf8")); + // acme-common's own manifest never changed since the baseline sync — this + // filtered run reports it "unchanged." The load-bearing assertion is + // below: acme-extra's entry survives BYTE-IDENTICAL despite its publisher + // directory not existing on disk at all. + expect(after.dependencies[DEP_NAME]).toEqual(baseline.dependencies[DEP_NAME]); + expect(after.dependencies[SECOND_NAME]).toEqual(baseline.dependencies[SECOND_NAME]); + }); }); From f6d86489654c403ee4586c6c7cb731a73b53af12 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 00:38:12 -0400 Subject: [PATCH 42/62] docs(fr-023): the deps option table carries no format key MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 14's file list specified `DEPS_OPTIONS = { "dry-run": boolean, format: string }`. `--format` is a GLOBAL flag: `index.ts` extracts it from anywhere in argv before any command parser runs, and validates it once for every command. No command's option table declares a `format` key — the apparent counter-example, `MIGRATE_OPTIONS`, matches only the unrelated `"migration-format"`. `parseDepsArgs` also runs `parseArgs` with `strict: true`, so the two facts have to hold together: listing `format` there would be dead at best, and omitting it is only safe because the global extraction happens first. Both verified before correcting. Found by the Task 14 implementer against the source, not by the plan's author. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 663ec5c7c..83cfa6760 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -929,7 +929,7 @@ feat(codegen-ts): sharedModelFile() emits a publisher's flattened shared-model a **Files:** - Create: `server/typescript/packages/cli/src/commands/deps.ts` - Create: `server/typescript/packages/cli/src/lib/dependency-sync.ts` (pure functions over a resolved directory: `readManifestDir`, `validateAgainstSpec`, `standaloneLoadCheck`, `planSync`, `applySync`) -- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`DEPS_OPTIONS = { "dry-run": boolean, format: string }`, `parseDepsArgs` with positionals `sync | check | list […]`) +- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`DEPS_OPTIONS = { "dry-run": boolean }` (NOT `format` — corrected 2026-09-12: `--format` is a GLOBAL flag, extracted from argv by `index.ts:391` and validated once at `:450` before any command parser runs; no command option table carries a `format` key, and `parseArgs` is `strict: true`, so listing it here would be dead at best), `parseDepsArgs` with positionals `sync | check | list […]`) - Modify: `server/typescript/packages/cli/src/index.ts` (register `deps` beside `eject`; help text; `FORMAT_AWARE_COMMANDS` gains `"deps"`) - Test: `cli/test/deps-sync.test.ts`, `cli/test/unit/args-deps.test.ts`, `cli/test/help-lists-every-flag.test.ts` (existing — must stay green) From 051469658905844f750d9db082a16e901a563705 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 00:56:00 -0400 Subject: [PATCH 43/62] feat(cli): meta deps check and verify --deps fail when the installed dependency differs from the lock (FR-023) Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../packages/cli/src/commands/deps.ts | 97 ++++--- .../packages/cli/src/commands/verify.ts | 67 ++++- server/typescript/packages/cli/src/index.ts | 23 +- .../typescript/packages/cli/src/lib/args.ts | 20 +- .../packages/cli/src/lib/dependency-sync.ts | 178 ++++++++++++ .../cli/test/__snapshots__/cli.test.ts.snap | 6 +- .../packages/cli/test/deps-check.test.ts | 259 ++++++++++++++++++ .../cli/test/unit/args-verify.test.ts | 22 +- 8 files changed, 626 insertions(+), 46 deletions(-) create mode 100644 server/typescript/packages/cli/test/deps-check.test.ts diff --git a/server/typescript/packages/cli/src/commands/deps.ts b/server/typescript/packages/cli/src/commands/deps.ts index 86d29453a..ab54264e7 100644 --- a/server/typescript/packages/cli/src/commands/deps.ts +++ b/server/typescript/packages/cli/src/commands/deps.ts @@ -1,16 +1,13 @@ // server/typescript/packages/cli/src/commands/deps.ts // -// FR-023 Phase 1a, Task 14 — `meta deps sync` (`path` transport) and -// `meta deps list`. `meta deps check` parses (see `parseDepsArgs`) but is -// Task 15's — this task refuses it rather than guessing at its report shape. +// FR-023 Phase 1a — Task 14 shipped `meta deps sync` (`path` transport) and +// `meta deps list`; Task 15 shipped `meta deps check` (this file's `runCheck`). import { join } from "node:path"; import { DEFAULT_METAOBJECTS_DIR, discoverCollectionRoot, loadConfig, loadMemory, - LOCK_FILE, - readLock, resolveCollection, type DependencySpec, type Lock, @@ -20,27 +17,15 @@ import { log } from "../lib/log.js"; import { emitStructured, type OutputFormat } from "../lib/format.js"; import { reportLoadError } from "../lib/load-error.js"; import { collectionLoadOptions } from "../lib/collection-load-options.js"; -import { applySync, hash8, planSync } from "../lib/dependency-sync.js"; - -/** - * `readLock`, converted into a diagnostic that NAMES the file rather than - * letting a corrupted committed lock's raw `JSON.parse`/`ZodError` surface as - * an unhandled rejection. `deps.lock.json` is checked-in, hand-editable, and - * merge-conflictable — a realistic way for it to break — and `bin/meta.ts`'s - * `run(...).then((code) => process.exit(code))` has no top-level `.catch()`, - * so an uncaught throw here would crash the process with a stack trace - * instead of the clean exit code every other failure in this command gets. - */ -async function readLockOrThrow(configDir: string): Promise { - try { - return await readLock(configDir); - } catch (err) { - throw new Error( - `${DEFAULT_METAOBJECTS_DIR}/${LOCK_FILE} is corrupted and could not be read: ${(err as Error).message}. ` + - "Fix it by hand, or delete it and re-run `meta deps sync` to regenerate it.", - ); - } -} +import { + applySync, + checkDependencies, + ERR_DEPENDENCY_UPSTREAM_DRIFT, + formatCheckLine, + hash8, + planSync, + readLockOrThrow, +} from "../lib/dependency-sync.js"; /** The declared `dependencies` for the config governing `cwd` — read * DIRECTLY via `loadConfig`, never through `resolveCollection`/`Collection`: @@ -157,6 +142,60 @@ async function runList(configDir: string, fmt: OutputFormat): Promise { return 0; } +/** + * `meta deps check` (Task 15) — re-resolve each declared dependency exactly + * as `sync` steps 1-2 do and compare its installed artifact's TRUE hash + * against what `.metaobjects/deps.lock.json` pinned. Read-only: never touches + * the lock or the snapshot (that is `sync`'s job). A `drifted` or + * `unresolved` dependency fails the command — a check that cannot check must + * not pass. + */ +async function runCheck( + configDir: string, + specs: readonly DependencySpec[], + fmt: OutputFormat, +): Promise { + let lock: Lock | undefined; + try { + lock = await readLockOrThrow(configDir); + } catch (err) { + log.error((err as Error).message); + return 2; + } + + if (specs.length === 0) { + if (fmt === "text") { + log.info("meta deps check: nothing to check — no dependencies declared."); + } else { + emitStructured({ dependencies: [] }, fmt); + } + return 0; + } + + const results = await checkDependencies(configDir, specs, lock); + const failing = results.filter((r) => r.status !== "current"); + + if (fmt === "text") { + for (const r of results) log.info(formatCheckLine(r)); + if (failing.length > 0) { + log.error( + `meta deps check: ${failing.length} of ${results.length} dependenc` + + `${results.length === 1 ? "y" : "ies"} drifted or unresolved (${ERR_DEPENDENCY_UPSTREAM_DRIFT}).`, + ); + } + } else { + emitStructured( + { + dependencies: results.map((r) => ({ ...r, line: formatCheckLine(r) })), + ...(failing.length > 0 ? { code: ERR_DEPENDENCY_UPSTREAM_DRIFT } : {}), + }, + fmt, + ); + } + + return failing.length > 0 ? 1 : 0; +} + export async function depsCommand(args: string[], cwd: string, fmt: OutputFormat): Promise { let flags: DepsFlags; try { @@ -187,10 +226,6 @@ export async function depsCommand(args: string[], cwd: string, fmt: OutputFormat case "list": return runList(configDir, fmt); case "check": - log.error( - "meta deps check is not implemented in this release. It will compare each declared " + - "dependency's INSTALLED artifact against the committed lock (ERR_DEPENDENCY_UPSTREAM_DRIFT).", - ); - return 1; + return runCheck(configDir, specs, fmt); } } diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index 5d7c096ef..51590b004 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -70,7 +70,14 @@ import { type D1Runner, type DriftResult, } from "@metaobjectsdev/migrate-ts"; -import { loadMemory, resolveCollection } from "@metaobjectsdev/sdk"; +import { + DEFAULT_METAOBJECTS_DIR, + loadConfig, + loadMemory, + resolveCollection, + type DependencySpec, +} from "@metaobjectsdev/sdk"; +import { checkDependencies, ERR_DEPENDENCY_UPSTREAM_DRIFT, formatCheckLine, readLockOrThrow } from "../lib/dependency-sync.js"; import { exclusionNotes, importedOption, migrateScopeMismatch } from "../lib/migrate-scope.js"; import { TYPE_TEMPLATE, @@ -164,11 +171,15 @@ export async function verifyCommand( // D1 has no URL connection); that check lives inside runSchemaVerify. const runCodegen = flags.codegen; const runDocs = flags.docs; + // Task 15 — same shape as --codegen/--docs: selected ONLY by its own flag, + // never folded into the bare-verify default (it needs the publisher + // reachable, which CI may not have — see the VerifyFlags doc on `deps`). + const runDeps = flags.deps; if (!flags.anyExplicit) { say( "meta verify — running --templates (default). Explicit subverbs: " + "--templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), " + - "--docs (docs drift), " + + "--docs (docs drift), --deps (dependency drift), " + "--replay/--replay-snapshot (the committed migration chain replays from empty).", ); } @@ -342,6 +353,7 @@ export async function verifyCommand( const schemaExit = await runSchemaVerify(); const codegenExit = runCodegen ? await runCodegenVerify() : 0; const docsExit = runDocs ? await runDocsVerify() : 0; + const depsExit = runDeps ? await runDepsVerify() : 0; // Requirements have no subverb: `requirement.*` nodes are metadata, so they // are checked on every `meta verify`. Opt-in by DECLARATION — a model with no // requirement nodes is silent, not in drift. @@ -363,6 +375,7 @@ export async function verifyCommand( schemaExit, codegenExit, docsExit, + depsExit, requirementExit, replayExit, ); @@ -379,6 +392,7 @@ export async function verifyCommand( { gate: "schema", ran: ranSchemaGate, ok: schemaExit === 0 }, { gate: "codegen", ran: runCodegen, ok: codegenExit === 0 }, { gate: "docs", ran: runDocs, ok: docsExit === 0 }, + { gate: "deps", ran: runDeps, ok: depsExit === 0 }, { gate: "requirements", ran: true, ok: requirementExit === 0 }, { gate: "replay", ran: flags.replay || flags.replaySnapshot, ok: replayExit === 0 }, ], @@ -1404,6 +1418,55 @@ export async function verifyCommand( } return 1; } + + // -- dependency drift (Task 15, FR-023) -------------------------------------- + // Gated on --deps. Re-resolves each declared dependency exactly as `meta deps + // check` does — never part of the bare-verify default (it needs the publisher + // reachable, which CI may not have). + async function runDepsVerify(): Promise { + // Read RAW declared specs (never `collection.dependencies`, which is the + // already-`ResolvedDependency[]` the LOCK produced — it carries no + // transport info, so it cannot be re-resolved). A project with no + // config.json at all declares no dependencies. + let depSpecs: readonly DependencySpec[] = []; + try { + const cfg = await loadConfig(join(collection.configDir, DEFAULT_METAOBJECTS_DIR)); + depSpecs = cfg.dependencies; + } catch { + depSpecs = []; + } + + if (depSpecs.length === 0) { + say("verify --deps: no dependencies declared — nothing to check."); + return 0; + } + + let lock: Awaited>; + try { + lock = await readLockOrThrow(collection.configDir); + } catch (err) { + log.error(`verify --deps: ${(err as Error).message}`); + return 2; + } + + const results = await checkDependencies(collection.configDir, depSpecs, lock); + const failing = results.filter((r) => r.status !== "current"); + + for (const r of results) { + if (r.status === "current") say(formatCheckLine(r)); + else log.error(formatCheckLine(r)); + } + + if (failing.length > 0) { + log.error( + `verify --deps: ${failing.length} of ${results.length} dependenc` + + `${results.length === 1 ? "y" : "ies"} drifted or unresolved (${ERR_DEPENDENCY_UPSTREAM_DRIFT}).`, + ); + return 1; + } + say("verify --deps: every dependency's installed artifact matches the lock."); + return 0; + } } /** diff --git a/server/typescript/packages/cli/src/index.ts b/server/typescript/packages/cli/src/index.ts index b2b94dc06..22ba1d956 100644 --- a/server/typescript/packages/cli/src/index.ts +++ b/server/typescript/packages/cli/src/index.ts @@ -38,7 +38,7 @@ COMMANDS: types [query] Search the metadata vocabulary (types, subtypes, @attrs) by name or description export Flatten loaded metadata to one canonical JSON artifact docs [] --out Generate neutral metadata documentation (entity + template pages; --site for HTML site) - verify Drift gate — subverbs: --templates / --db / --codegen / --docs (bare = --templates) + verify Drift gate — subverbs: --templates / --db / --codegen / --docs / --deps (bare = --templates) upgrade Rewrite retired metadata vocabulary (previews; --apply writes) prompt-snapshot Snapshot rendered template.* output; --check gates drift migrate Diff metadata vs live DB; emit migration SQL files @@ -97,6 +97,10 @@ VERIFY FLAGS (ADR-0021 D2 — explicit subverbs; combine any; exit 1 on ANY drif run would emit; never reports an extra file, because docs.outDir holds hand-written documentation MetaObjects did not write. Needs metaobjects.config.ts; exit 2 if absent. + --deps Dependency drift — re-resolve each declared dependency and compare + its installed artifact's hash against .metaobjects/deps.lock.json + (the same check 'meta deps check' runs). Never part of the bare-verify + default — it needs the publisher reachable, which CI may not have. --db Schema drift — live DB URL enables the schema-drift gate. Supports: file:, libsql:, postgres:, postgresql:. Omit to skip. D1 has no URL — use --dialect d1 / --d1 instead. @@ -207,6 +211,11 @@ USAGE: the lock and its snapshot dir deleted, every run. meta deps list One line per locked dependency: name, version, hash8, node count, packages. + meta deps check Re-resolve each declared dependency (path transport, same + as sync steps 1-2) and compare its INSTALLED artifact's hash + against .metaobjects/deps.lock.json. Read-only — never + touches the lock or the snapshot. Exits 1 if any dependency + has drifted or cannot be resolved (ERR_DEPENDENCY_UPSTREAM_DRIFT). FLAGS: --dry-run Plan and report; write nothing (sync only) @@ -218,9 +227,9 @@ valid config (reserved for a future toolchain) but 'meta deps sync' refuses it b "transport \`npm\` is not supported by this toolchain yet; use \`path\`". The lock is the ONLY thing 'meta deps sync' writes to besides the snapshot directory — -sync never touches your own metadata files. 'meta deps check' (a future release) compares -the lock against what is installed right now; this command only ever compares against -what config DECLARES. +sync never touches your own metadata files. 'meta deps check' compares the lock against +what is installed right now (the same check 'meta verify --deps' runs); 'meta deps sync' +only ever compares against what config DECLARES. `, verify: `meta verify — drift gate (templates / DB schema / codegen / migration replay) @@ -240,6 +249,12 @@ FLAGS: a namespace MetaObjects owns); the count of exempt pages is reported either way, and a run where every page is ignored is refused rather than reported clean. + --deps Dependency drift — re-resolve each declared dependency exactly as + 'meta deps check' does and compare its installed artifact's hash + against .metaobjects/deps.lock.json. Exits 1 if any dependency has + drifted or cannot be resolved (ERR_DEPENDENCY_UPSTREAM_DRIFT). NEVER + part of the bare-verify default — it needs the publisher reachable, + which CI may not have. --db Schema drift — live DB URL enables the schema-drift gate. Supports: file:, libsql:, postgres:, postgresql: D1 has no URL — use --dialect d1 / --d1 instead. diff --git a/server/typescript/packages/cli/src/lib/args.ts b/server/typescript/packages/cli/src/lib/args.ts index c7a92d7e1..cb8a056e7 100644 --- a/server/typescript/packages/cli/src/lib/args.ts +++ b/server/typescript/packages/cli/src/lib/args.ts @@ -296,7 +296,17 @@ export interface VerifyFlags { * read as that flag's opposite rather than as a replay depth. */ replaySnapshot: boolean; - /** Whether ANY explicit subverb flag (--templates/--db/--codegen/--docs/--replay*) was passed. */ + /** + * Dependency drift (Task 15, FR-023) — re-resolve each declared dependency and + * compare its installed artifact's hash against `.metaobjects/deps.lock.json` + * (the same comparison `meta deps check` runs). Deliberately NEVER part of the + * bare-verify default, unlike --templates: it needs the publisher reachable, + * which CI may not have. It is still counted in `anyExplicit`, same as + * --codegen/--docs, so a lone `verify --deps` does not ALSO run the template + * gate. + */ + deps: boolean; + /** Whether ANY explicit subverb flag (--templates/--db/--codegen/--docs/--deps/--replay*) was passed. */ anyExplicit: boolean; /** Suppress the advisory anti-pattern (verify-as-teacher) pass. */ noAntipatterns: boolean; @@ -342,6 +352,7 @@ export const VERIFY_OPTIONS = { templates: { type: "boolean", default: false }, codegen: { type: "boolean", default: false }, docs: { type: "boolean", default: false }, + deps: { type: "boolean", default: false }, replay: { type: "boolean", default: false }, "replay-snapshot": { type: "boolean", default: false }, "no-antipatterns": { type: "boolean", default: false }, @@ -397,15 +408,17 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { const templates = !!values.templates; const codegen = !!values.codegen; const docs = !!values.docs; + const deps = !!values.deps; const replay = !!values.replay; const replaySnapshot = !!values["replay-snapshot"]; // --db is itself an explicit subverb selector: passing a connection URL means // "run the schema-drift mode". So is `--dialect d1` (D1 has no --db connection // URL — see the `d1` field doc above). The replay flags are subverbs too, and // must be listed here or `meta verify --replay` would ALSO run the template gate - // as the bare-verify default. + // as the bare-verify default. --deps joins the same list for the same reason — + // it must NOT also be part of that default (see the VerifyFlags doc on `deps`). const anyExplicit = - templates || codegen || docs || values.db !== undefined || dialect === "d1" || replay || replaySnapshot; + templates || codegen || docs || deps || values.db !== undefined || dialect === "d1" || replay || replaySnapshot; return { prompts: values.prompts, @@ -416,6 +429,7 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { templates, codegen, docs, + deps, replay, replaySnapshot, anyExplicit, diff --git a/server/typescript/packages/cli/src/lib/dependency-sync.ts b/server/typescript/packages/cli/src/lib/dependency-sync.ts index 1158f68df..d12a97f06 100644 --- a/server/typescript/packages/cli/src/lib/dependency-sync.ts +++ b/server/typescript/packages/cli/src/lib/dependency-sync.ts @@ -19,14 +19,17 @@ import { METAMODEL_VERSION, packageOfResolutionKey, ParseError, + type ErrorCode, } from "@metaobjectsdev/metadata"; import { DEPS_DIR, DEFAULT_METAOBJECTS_DIR, INTEGRITY_PREFIX, + LOCK_FILE, MANIFEST_FILE, DependencyManifestSchema, dependencyName, + readLock, sha256Integrity, writeLock, type DependencyManifest, @@ -462,3 +465,178 @@ export async function applySync( return { lock, report }; } + +// --------------------------------------------------------------------------- +// Task 15 — `meta deps check` / `verify --deps`: does the INSTALLED artifact +// (re-resolved live, exactly as `sync` steps 1-2 do) still hash to what the +// LOCK pinned? (DESIGN §2.3 "the lock pins bytes", §4.1 as amended by §11 — +// no usage-aware classification, just current/drifted/unresolved.) +// --------------------------------------------------------------------------- + +/** + * `readLock`, converted into a diagnostic that NAMES the file rather than + * letting a corrupted committed lock's raw `JSON.parse`/`ZodError` surface as + * an unhandled rejection. `deps.lock.json` is checked-in, hand-editable, and + * merge-conflictable — a realistic way for it to break — and `bin/meta.ts`'s + * `run(...).then((code) => process.exit(code))` has no top-level `.catch()`, + * so an uncaught throw here would crash the process with a stack trace + * instead of the clean exit code every other failure in this command gets. + * + * The ONE call site for the sdk's `readLock` in `cli/src` — `deps.ts`'s + * `sync`/`list`/`check` handlers and `verify.ts`'s `--deps` gate all call + * THIS wrapper instead, so the diagnostic can never go stale in one of them. + */ +export async function readLockOrThrow(configDir: string): Promise { + try { + return await readLock(configDir); + } catch (err) { + throw new Error( + `${DEFAULT_METAOBJECTS_DIR}/${LOCK_FILE} is corrupted and could not be read: ${(err as Error).message}. ` + + "Fix it by hand, or delete it and re-run `meta deps sync` to regenerate it.", + ); + } +} + +/** + * `meta deps check` / `verify --deps`'s classification for one declared + * dependency (DESIGN §4.1 as amended by §11 — no usage-aware classifier, just + * "does the installed hash match the lock"): + * - `current` — resolved fine, and its installed artifact hashes to + * exactly what the lock pinned. + * - `drifted` — resolved fine, but the hash differs. + * - `unresolved` — step 1 (`resolveDependencyDir`) or step 2 + * (`readManifestDir`) failed, OR the dependency is declared + * but the lock has no entry for it at all (nothing to + * compare against — "a check that cannot check must not + * pass"). `detail` carries WHY. + * + * `lockVersion`/`lockIntegrity` are `undefined` exactly when there is no lock + * entry; `installedVersion`/`installedIntegrity` are `undefined` exactly when + * resolution itself failed. + */ +export interface DependencyCheckResult { + readonly name: string; + readonly status: "current" | "drifted" | "unresolved"; + readonly lockVersion: string | undefined; + readonly lockIntegrity: string | undefined; + readonly installedVersion: string | undefined; + readonly installedIntegrity: string | undefined; + /** Populated for `unresolved` only — the reason, with the redundant + * `dependency "": ` prefix (`manifestInvalid`/`resolveDependencyDir`'s + * own convention) stripped, since `formatCheckLine` already names the + * dependency once in its own lead-in. */ + readonly detail: string | undefined; +} + +/** Strips the `dependency "": ` prefix `resolveDependencyDir` and + * `readManifestDir`'s errors always carry — `formatCheckLine` names the + * dependency itself, so repeating it verbatim inside `detail` would read + * twice. */ +function stripDependencyPrefix(name: string, message: string): string { + const prefix = `dependency "${name}": `; + return message.startsWith(prefix) ? message.slice(prefix.length) : message; +} + +/** + * DESIGN §4.1 as amended by §11 — resolve each declared dependency EXACTLY as + * `sync` steps 1-2 do (`resolveDependencyDir` then `readManifestDir`; never + * step 3's `validateAgainstSpec` — `check` compares hashes, it does not + * re-validate the spec), then compare the freshly-read artifact's TRUE hash + * (re-hashed here from the bytes `readManifestDir` returned, never the + * installed manifest's `integrity` field taken on faith) against the lock's. + * + * Read-only: nothing here writes the lock, the snapshot, or touches + * `.metaobjects/deps/` — that is `sync`'s job. Every declared dependency gets + * exactly one result, in name order. + */ +export async function checkDependencies( + configDir: string, + specs: readonly DependencySpec[], + lock: Lock | undefined, +): Promise { + const entries = lock?.dependencies ?? {}; + const sorted = [...specs].sort((a, b) => dependencyName(a).localeCompare(dependencyName(b))); + const results: DependencyCheckResult[] = []; + + for (const spec of sorted) { + const name = dependencyName(spec); + const lockEntry = entries[name]; + + try { + const dir = resolveDependencyDir(configDir, spec); + const { manifest, artifactContent } = await readManifestDir(dir, name); + // Never trust the manifest's own `integrity` field for the comparison — + // re-hash the bytes actually sitting on disk right now. + const installedIntegrity = sha256Integrity(artifactContent); + + if (lockEntry === undefined) { + results.push({ + name, + status: "unresolved", + lockVersion: undefined, + lockIntegrity: undefined, + installedVersion: manifest.version, + installedIntegrity, + detail: "declared, and resolves, but is not in the lock yet", + }); + continue; + } + + results.push({ + name, + status: installedIntegrity === lockEntry.integrity ? "current" : "drifted", + lockVersion: lockEntry.version, + lockIntegrity: lockEntry.integrity, + installedVersion: manifest.version, + installedIntegrity, + detail: undefined, + }); + } catch (err) { + results.push({ + name, + status: "unresolved", + lockVersion: lockEntry?.version, + lockIntegrity: lockEntry?.integrity, + installedVersion: undefined, + installedIntegrity: undefined, + detail: stripDependencyPrefix(name, (err as Error).message), + }); + } + } + + return results; +} + +/** The one code every `checkDependencies` failure (`drifted` or `unresolved`) + * is reported under — `deps.ts`'s `check` handler and `verify.ts`'s `--deps` + * gate both surface it, so it is named once here rather than inlined twice. */ +export const ERR_DEPENDENCY_UPSTREAM_DRIFT: ErrorCode = "ERR_DEPENDENCY_UPSTREAM_DRIFT"; + +/** + * `meta deps check` / `verify --deps`'s one report line per dependency. The + * `drifted` shape is pinned exactly by the task brief: + * `: drifted — lock (), installed (); run meta + * deps sync and review the artifact diff`. + */ +export function formatCheckLine(result: DependencyCheckResult): string { + const lockDesc = `${result.lockVersion ?? "(none)"} (${ + result.lockIntegrity !== undefined ? hash8(result.lockIntegrity) : "n/a" + })`; + const installedDesc = `${result.installedVersion ?? "(none)"} (${ + result.installedIntegrity !== undefined ? hash8(result.installedIntegrity) : "n/a" + })`; + switch (result.status) { + case "current": + return `${result.name}: current — ${lockDesc}`; + case "drifted": + return ( + `${result.name}: drifted — lock ${lockDesc}, installed ${installedDesc}; ` + + "run meta deps sync and review the artifact diff" + ); + case "unresolved": + return ( + `${result.name}: unresolved — ${result.detail ?? "could not resolve the dependency"}; ` + + "run meta deps sync once the dependency is reachable" + ); + } +} diff --git a/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap b/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap index 5cdd79801..b576a112a 100644 --- a/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap +++ b/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap @@ -20,7 +20,7 @@ COMMANDS: types [query] Search the metadata vocabulary (types, subtypes, @attrs) by name or description export Flatten loaded metadata to one canonical JSON artifact docs [] --out Generate neutral metadata documentation (entity + template pages; --site for HTML site) - verify Drift gate — subverbs: --templates / --db / --codegen / --docs (bare = --templates) + verify Drift gate — subverbs: --templates / --db / --codegen / --docs / --deps (bare = --templates) upgrade Rewrite retired metadata vocabulary (previews; --apply writes) prompt-snapshot Snapshot rendered template.* output; --check gates drift migrate Diff metadata vs live DB; emit migration SQL files @@ -79,6 +79,10 @@ VERIFY FLAGS (ADR-0021 D2 — explicit subverbs; combine any; exit 1 on ANY drif run would emit; never reports an extra file, because docs.outDir holds hand-written documentation MetaObjects did not write. Needs metaobjects.config.ts; exit 2 if absent. + --deps Dependency drift — re-resolve each declared dependency and compare + its installed artifact's hash against .metaobjects/deps.lock.json + (the same check 'meta deps check' runs). Never part of the bare-verify + default — it needs the publisher reachable, which CI may not have. --db Schema drift — live DB URL enables the schema-drift gate. Supports: file:, libsql:, postgres:, postgresql:. Omit to skip. D1 has no URL — use --dialect d1 / --d1 instead. diff --git a/server/typescript/packages/cli/test/deps-check.test.ts b/server/typescript/packages/cli/test/deps-check.test.ts new file mode 100644 index 000000000..a12393d0d --- /dev/null +++ b/server/typescript/packages/cli/test/deps-check.test.ts @@ -0,0 +1,259 @@ +// FR-023 Phase 1a, Task 15 — `meta deps check` and `meta verify --deps`. +// +// Same publisher/consumer scaffold as `deps-sync.test.ts` (Task 14): a PUBLISHER +// directory (`acme-common/metaobjects/`) holding a manifest beside the pinned +// `dependency-conformance` artifact, and a CONSUMER (`consumer/.metaobjects/config.json`) +// declaring a `path` dependency on it. Every test here starts from an +// ALREADY-SYNCED consumer (a real `deps sync` run, so the lock is real) and then +// perturbs the PUBLISHER only — proving `check`/`verify --deps` compare the +// installed artifact against the lock without ever re-running `sync` itself. +import { describe, test, expect, afterAll } from "bun:test"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { METAMODEL_VERSION } from "@metaobjectsdev/metadata"; +import { INTEGRITY_PREFIX, sha256Integrity } from "@metaobjectsdev/sdk"; +import { run } from "../src/index.js"; + +const DEP_NAME = "acme-common"; +const ARTIFACT_BASENAME = "acme-common.metaobjects.json"; +const MANIFEST_BASENAME = "metaobjects.pkg.json"; +const NODES = ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"]; + +// The corpus this repo already pins (fixtures/dependency-conformance/README.md, +// and the Global Constraints "Hash format" block) — read as raw bytes so the +// hash assertions below are computed off the real corpus, never a re-typed copy. +const CORPUS_ARTIFACTS = resolve( + import.meta.dirname, + "../../../../../fixtures/dependency-conformance/artifacts", +); +const V1_BYTES = readFileSync(join(CORPUS_ARTIFACTS, "acme-common-v1.json")); +const WIDENED_BYTES = readFileSync(join(CORPUS_ARTIFACTS, "acme-common-v1-widened.json")); +const V1_HASH = sha256Integrity(V1_BYTES); +const WIDENED_HASH = sha256Integrity(WIDENED_BYTES); + +/** First 8 hex chars after the `INTEGRITY_PREFIX` — mirrors the shared `hash8` + * helper exported from `src/lib/dependency-sync.ts` (deliberately not + * imported — this is a CLI test, not a caller of that module's internals). */ +function hash8(integrity: string): string { + return integrity.slice(INTEGRITY_PREFIX.length, INTEGRITY_PREFIX.length + 8); +} + +const APP = JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Order", + children: [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +/** Global Constraints `CONFIG_REF`, verbatim. */ +const CONFIG_REF = JSON.stringify({ + schema_version: 1, + sources: [], + dependencies: [{ name: DEP_NAME, path: "../acme-common/metaobjects" }], +}); + +function manifestJson(opts: { version: string; integrity: string; nodes: string[] }): string { + return JSON.stringify( + { + schema_version: 1, + name: DEP_NAME, + version: opts.version, + metamodelVersion: METAMODEL_VERSION, + artifact: ARTIFACT_BASENAME, + integrity: opts.integrity, + packages: ["acme::common"], + nodes: opts.nodes, + }, + null, + 2, + ); +} + +const dirs: string[] = []; +afterAll(() => { + for (const d of dirs) rmSync(d, { recursive: true, force: true }); +}); + +/** A publisher + consumer pair, laid out as siblings under one temp root — + * identical layout to `deps-sync.test.ts`'s `setupProject()`. */ +function setupProject(): { root: string; consumerRoot: string; publisherDir: string } { + const root = mkdtempSync(join(tmpdir(), "deps-check-")); + dirs.push(root); + + const consumerRoot = join(root, "consumer"); + mkdirSync(join(consumerRoot, "metaobjects"), { recursive: true }); + writeFileSync(join(consumerRoot, "metaobjects", "meta.app.json"), APP, "utf8"); + mkdirSync(join(consumerRoot, ".metaobjects"), { recursive: true }); + writeFileSync(join(consumerRoot, ".metaobjects", "config.json"), CONFIG_REF, "utf8"); + + const publisherDir = join(root, "acme-common", "metaobjects"); + mkdirSync(publisherDir, { recursive: true }); + writeFileSync(join(publisherDir, ARTIFACT_BASENAME), V1_BYTES); + writeFileSync( + join(publisherDir, MANIFEST_BASENAME), + manifestJson({ version: "1.0.0", integrity: V1_HASH, nodes: NODES }), + "utf8", + ); + + return { root, consumerRoot, publisherDir }; +} + +/** Runs `argv`, capturing console.log/console.error separately, mirroring + * `advisory-structured-output.test.ts`'s `capture()` — needed here (rather + * than `deps-sync.test.ts`'s inline out/err arrays) because the `verify + * --deps --format json` case needs `out` isolated from stderr narration to + * `JSON.parse` it. */ +async function capture(argv: string[]): Promise<{ exit: number; out: string; err: string }> { + const outLines: string[] = []; + const errLines: string[] = []; + const origLog = console.log; + const origErr = console.error; + console.log = (...a: unknown[]) => { outLines.push(a.map(String).join(" ")); }; + console.error = (...a: unknown[]) => { errLines.push(a.map(String).join(" ")); }; + let exit: number; + try { + exit = await run(argv); + } finally { + console.log = origLog; + console.error = origErr; + } + return { exit, out: outLines.join("\n"), err: errLines.join("\n") }; +} + +interface VerifyGateRow { + gate: string; + ran: boolean; + ok: boolean; +} + +describe("meta deps check — path transport (FR-023 Phase 1a Task 15)", () => { + test("(a) after sync, check exits 0 and reports current", async () => { + const { consumerRoot } = setupProject(); + expect((await capture(["deps", "sync", "--format", "text", "--cwd", consumerRoot])).exit).toBe(0); + + const { exit, out, err } = await capture(["deps", "check", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(0); + expect(`${out}\n${err}`).toContain(`${DEP_NAME}: current — 1.0.0 (${hash8(V1_HASH)})`); + }); + + test("(b) a widened publisher: check exits 1 with the pinned drift line, and verify --deps exits 1 with a deps row", async () => { + const { consumerRoot, publisherDir } = setupProject(); + await capture(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + + writeFileSync(join(publisherDir, ARTIFACT_BASENAME), WIDENED_BYTES); + writeFileSync( + join(publisherDir, MANIFEST_BASENAME), + manifestJson({ version: "1.1.0", integrity: WIDENED_HASH, nodes: NODES }), + "utf8", + ); + + const checkResult = await capture(["deps", "check", "--format", "text", "--cwd", consumerRoot]); + expect(checkResult.exit).toBe(1); + expect(`${checkResult.out}\n${checkResult.err}`).toContain( + `${DEP_NAME}: drifted — lock 1.0.0 (${hash8(V1_HASH)}), installed 1.1.0 (${hash8(WIDENED_HASH)}); ` + + "run meta deps sync and review the artifact diff", + ); + // Pinned exactly, per the task brief — the literal example, not just the + // interpolation of the same values. + expect(`${checkResult.out}\n${checkResult.err}`).toContain( + "acme-common: drifted — lock 1.0.0 (10fbf886), installed 1.1.0 (fa00b9f3); " + + "run meta deps sync and review the artifact diff", + ); + + const verifyResult = await capture(["verify", "--deps", "--format", "json", "--cwd", consumerRoot]); + expect(verifyResult.exit).toBe(1); + const payload = JSON.parse(verifyResult.out.trim()) as { verify: VerifyGateRow[] }; + const depsRow = payload.verify.find((g) => g.gate === "deps"); + expect(depsRow).toEqual({ gate: "deps", ran: true, ok: false }); + }); + + test("(c) the publisher directory is gone: check reports unresolved and exits 1", async () => { + const { consumerRoot, publisherDir } = setupProject(); + await capture(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + rmSync(publisherDir, { recursive: true, force: true }); + + const { exit, out, err } = await capture(["deps", "check", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(1); + const printed = `${out}\n${err}`; + expect(printed).toContain(`${DEP_NAME}: unresolved`); + expect(printed).toContain("no directory exists there"); + }); + + test("(d) a bare 'meta verify' (no --deps) exits 0 regardless of dependency drift", async () => { + const { consumerRoot, publisherDir } = setupProject(); + await capture(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + // The publisher is now unreachable — 'deps check'/'verify --deps' would + // both fail against it. A bare 'verify' must never notice. + rmSync(publisherDir, { recursive: true, force: true }); + + const { exit } = await capture(["verify", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(0); + }); + + // Mutation-catching (per the task's own note): an implementation that just + // compares the LOCK's declared `integrity` string to the INSTALLED + // manifest's declared `integrity` string — without ever re-hashing the + // artifact bytes actually on disk — would read this as "current", because + // neither manifest's OWN `integrity` field changed. The correct + // implementation re-hashes (via `readManifestDir`, which itself refuses a + // manifest whose declared integrity no longer matches its artifact) and so + // must report this as `unresolved`, never `current`. + test("(e) the installed ARTIFACT bytes changed without regenerating its manifest: never reported current", async () => { + const { consumerRoot, publisherDir } = setupProject(); + await capture(["deps", "sync", "--format", "text", "--cwd", consumerRoot]); + + // Swap the artifact bytes only. The manifest.pkg.json beside it (untouched) + // still declares the OLD v1 integrity — a publisher who forgot to + // regenerate it after editing the artifact by hand. + writeFileSync(join(publisherDir, ARTIFACT_BASENAME), WIDENED_BYTES); + + const { exit, out, err } = await capture(["deps", "check", "--format", "text", "--cwd", consumerRoot]); + const printed = `${out}\n${err}`; + expect(printed).not.toContain(`${DEP_NAME}: current`); + expect(printed).toContain(`${DEP_NAME}: unresolved`); + expect(exit).toBe(1); + }); + + // A declared dependency that resolves fine but was never synced (no lock + // entry at all) has nothing to compare against — "a check that cannot + // check must not pass" applies here too, not only to a resolution failure. + test("(g) a declared dependency with no lock entry yet is unresolved, not current", async () => { + const { consumerRoot } = setupProject(); + // Deliberately no 'deps sync' — the lock file does not exist at all. + + const { exit, out, err } = await capture(["deps", "check", "--format", "text", "--cwd", consumerRoot]); + expect(exit).toBe(1); + const printed = `${out}\n${err}`; + expect(printed).toContain(`${DEP_NAME}: unresolved`); + expect(printed).not.toContain(`${DEP_NAME}: current`); + }); + + test("(f) no dependencies declared: check exits 0 and does nothing", async () => { + const root = mkdtempSync(join(tmpdir(), "deps-check-none-")); + dirs.push(root); + mkdirSync(join(root, "metaobjects"), { recursive: true }); + writeFileSync(join(root, "metaobjects", "meta.app.json"), APP, "utf8"); + mkdirSync(join(root, ".metaobjects"), { recursive: true }); + writeFileSync( + join(root, ".metaobjects", "config.json"), + JSON.stringify({ schema_version: 1, sources: [], dependencies: [] }), + "utf8", + ); + + const { exit, out, err } = await capture(["deps", "check", "--format", "text", "--cwd", root]); + expect(exit).toBe(0); + expect(`${out}\n${err}`).toContain("nothing to check"); + }); +}); diff --git a/server/typescript/packages/cli/test/unit/args-verify.test.ts b/server/typescript/packages/cli/test/unit/args-verify.test.ts index 1894482d2..d61199c42 100644 --- a/server/typescript/packages/cli/test/unit/args-verify.test.ts +++ b/server/typescript/packages/cli/test/unit/args-verify.test.ts @@ -6,7 +6,7 @@ describe("parseVerifyArgs", () => { test("defaults: prompts/db/dialect undefined, allow empty, skipSchema false, no explicit subverb", () => { expect(parseVerifyArgs([])).toEqual({ prompts: undefined, db: undefined, dialect: undefined, allow: [], skipSchema: false, - templates: false, codegen: false, docs: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, lax: false, + templates: false, codegen: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -14,7 +14,7 @@ describe("parseVerifyArgs", () => { test("--prompts is captured", () => { expect(parseVerifyArgs(["--prompts", "templates"])).toEqual({ prompts: "templates", db: undefined, dialect: undefined, allow: [], skipSchema: false, - templates: false, codegen: false, docs: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, lax: false, + templates: false, codegen: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -22,7 +22,7 @@ describe("parseVerifyArgs", () => { test("--db / --dialect / --skip-schema are captured", () => { expect(parseVerifyArgs(["--db", "file:x.db", "--dialect", "sqlite", "--skip-schema"])).toEqual({ prompts: undefined, db: "file:x.db", dialect: "sqlite", allow: [], skipSchema: true, - templates: false, codegen: false, docs: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, lax: false, + templates: false, codegen: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -32,7 +32,7 @@ describe("parseVerifyArgs", () => { test("--dialect d1 --d1 --remote are captured; --dialect d1 alone is an explicit subverb", () => { expect(parseVerifyArgs(["--dialect", "d1", "--d1", "DB", "--remote"])).toEqual({ prompts: undefined, db: undefined, dialect: "d1", allow: [], skipSchema: false, - templates: false, codegen: false, docs: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, lax: false, + templates: false, codegen: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, lax: false, replay: false, replaySnapshot: false, d1: "DB", remote: true, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -56,7 +56,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs(["--allow", "drop-column,drop-table"])).toEqual({ prompts: undefined, db: undefined, dialect: undefined, allow: ["drop-column", "drop-table"], skipSchema: false, - templates: false, codegen: false, docs: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, lax: false, + templates: false, codegen: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -110,6 +110,18 @@ describe("parseVerifyArgs", () => { expect(f.codegen).toBe(true); expect(f.anyExplicit).toBe(true); }); + // Task 15 — --deps is its own subverb, same shape as --codegen/--docs: it + // counts toward anyExplicit (so a lone `verify --deps` does not ALSO run the + // bare-verify template default) but is never itself part of that default. + test("--deps sets the explicit-subverb flags and nothing else", () => { + const f = parseVerifyArgs(["--deps"]); + expect(f.deps).toBe(true); + expect(f.templates).toBe(false); + expect(f.codegen).toBe(false); + expect(f.docs).toBe(false); + expect(f.anyExplicit).toBe(true); + expect(parseVerifyArgs([]).deps).toBe(false); + }); test("--no-requirement-lint mutes the advisory lint, and defaults off", () => { expect(parseVerifyArgs([]).noRequirementLint).toBe(false); expect(parseVerifyArgs(["--no-requirement-lint"]).noRequirementLint).toBe(true); From 382eadbbf163fd4d751d2f1e386d57e0576eead0 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 01:09:20 -0400 Subject: [PATCH 44/62] docs(cli): document the readManifestDir invariant checkDependencies's re-hash leans on (FR-023) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 15 fix round 1: readManifestDir already guarantees sha256Integrity(artifactContent) === manifest.integrity before checkDependencies's own re-hash can run, so the re-hash cannot currently observe a divergence. Comment only — records the invariant and the risk if readManifestDir's check is ever relaxed, so a future reader isn't left to trace it themselves. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../packages/cli/src/lib/dependency-sync.ts | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/server/typescript/packages/cli/src/lib/dependency-sync.ts b/server/typescript/packages/cli/src/lib/dependency-sync.ts index d12a97f06..1f50a95a2 100644 --- a/server/typescript/packages/cli/src/lib/dependency-sync.ts +++ b/server/typescript/packages/cli/src/lib/dependency-sync.ts @@ -567,6 +567,20 @@ export async function checkDependencies( const { manifest, artifactContent } = await readManifestDir(dir, name); // Never trust the manifest's own `integrity` field for the comparison — // re-hash the bytes actually sitting on disk right now. + // + // Right now this cannot observe a divergence from `manifest.integrity`: + // `readManifestDir` above (§4.1 step 2, :152-159) already throws + // `ERR_DEPENDENCY_MANIFEST_INVALID` unless `sha256Integrity(artifactContent) + // === manifest.integrity`, so by the time this line runs the two values are + // provably equal — this call is redundant WORK, not redundant SAFETY. The + // safety it buys is against `readManifestDir` changing out from under this + // function: if that check is ever relaxed, moved, or made conditional, this + // line becomes the ONLY thing standing between `check` and trusting a + // self-declared field, with no test that would fail to say so (the + // divergent state is unreachable through `checkDependencies`'s public + // signature today, precisely because `readManifestDir` forecloses it + // first). Read `manifest.integrity` here only if you have also confirmed + // `readManifestDir` still enforces the hash match unconditionally. const installedIntegrity = sha256Integrity(artifactContent); if (lockEntry === undefined) { From 3b813451f1e3a95857db26a15c68f804e7aa74f9 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 01:36:13 -0400 Subject: [PATCH 45/62] =?UTF-8?q?feat(cli):=20meta=20verify=20reports=20an?= =?UTF-8?q?=20unflagged=20cross-file=20redeclaration=20as=20an=20overlay?= =?UTF-8?q?=20authoring=20finding=20(FR-023=20=C2=A711)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../packages/cli/src/commands/verify.ts | 94 ++++++- server/typescript/packages/cli/src/index.ts | 11 + .../typescript/packages/cli/src/lib/args.ts | 12 + .../packages/cli/src/lib/overlay-lint.ts | 129 +++++++++ .../cli/test/__snapshots__/cli.test.ts.snap | 2 + .../cli/test/unit/args-verify.test.ts | 10 +- .../cli/test/verify-overlay-lint.test.ts | 258 ++++++++++++++++++ .../typescript/packages/metadata/src/index.ts | 6 +- .../metadata/src/loader/meta-data-loader.ts | 155 ++++++++--- .../test/declared-top-level-keys.test.ts | 160 +++++++++++ 10 files changed, 776 insertions(+), 61 deletions(-) create mode 100644 server/typescript/packages/cli/src/lib/overlay-lint.ts create mode 100644 server/typescript/packages/cli/test/verify-overlay-lint.test.ts create mode 100644 server/typescript/packages/metadata/test/declared-top-level-keys.test.ts diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index 51590b004..69f1a1633 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -31,6 +31,8 @@ import { checkRequirements, summariseRequirements, scanRequirements, type Diagnostic, } from "../lib/requirement-check.js"; import { lintRequirements } from "../lib/requirement-lint.js"; +import { lintOverlays } from "../lib/overlay-lint.js"; +import { FileSource } from "@metaobjectsdev/metadata/core"; import { resolveD1Config, resolveMigrateConfig } from "../lib/config.js"; import { buildWranglerExecuteArgs, @@ -326,7 +328,7 @@ export async function verifyCommand( const promptsDir = join(projectRoot, flags.prompts ?? DEFAULT_PROMPTS_DIR); const provider = new FileProvider(promptsDir); - // The two advisory sections, captured as the gates run so the structured payload + // The advisory sections, captured as the gates run so the structured payload // can carry them IN FULL. They were previously formatted straight to stderr and // existed nowhere else, which is why 96% of a 239-finding report was unreachable // by any flag, env var or format. @@ -334,6 +336,12 @@ export async function verifyCommand( skippedSection("the requirement pass did not run"); let antiPatternSection: AdvisorySection = skippedSection("the advisory anti-pattern pass did not run"); + // FR-023 §11.1 item 4 (Task 17) — the overlay authoring lint. Its own section + // (never folded into `requirements`, which is about a different kind of node + // entirely) so the structured payload names what ran and what didn't the same + // way every other advisory pass does. + let overlaySection: AdvisorySection = + skippedSection("the overlay lint did not run"); // The ledger counts `meta verify` prints on every run. Undefined for a project // declaring no requirement.* node at all (opt-in by declaration) — the payload // then omits the block rather than reporting zeroes that would read as an empty @@ -364,6 +372,12 @@ export async function verifyCommand( // would parse cleanly and do nothing at all. const replayExit = flags.replay || flags.replaySnapshot ? await runReplayVerify() : 0; + // FR-023 §11.1 item 4 (Task 17) — the overlay authoring lint. Runs on every + // `meta verify`, not gated on any subverb (an unflagged cross-file + // redeclaration is a risk regardless of which drift gates were selected). + // Warnings ONLY — never changes the exit code. + await runOverlayLintAdvisory(); + // Advisory verify-as-teacher pass: surface hand-rolled work the metadata could // model. Warnings ONLY — never changes the exit code (bias to under-flagging). // Suppressed with --no-antipatterns or META_NO_ANTIPATTERNS=1 for the rare @@ -400,6 +414,7 @@ export async function verifyCommand( requirements: requirementSection, requirementCounts, antiPatterns: antiPatternSection, + overlays: overlaySection, }), fmt, ); @@ -667,18 +682,13 @@ export async function verifyCommand( const errors = diags.filter((d) => d.severity === "error"); const warns = diags.filter((d) => d.severity === "warn"); - // Named `fmtDiag`, not `fmt`: `fmt` is this command's OUTPUT FORMAT parameter, - // and a shadow of it inside the one function that must not confuse the two is - // how a structured run quietly reverts to text. - const fmtDiag = (d: Diagnostic): string => - ` ${d.code}${d.path !== undefined ? ` [${d.path}]` : ""}: ${d.message}`; // Capped per SECTION, never across them: a ledger of a few hundred entries can // produce hundreds of prose findings, and a shared budget would let the advisory // lint push every gate warning off the end. The cap VALUE is now one shared // constant (`--limit`), so raising it cannot miss a section — errors stay // uncapped, as they always were. - for (const d of errors) log.error(fmtDiag(d)); - warnCapped(warns.map(fmtDiag), flags.limit, { structured }); + for (const d of errors) log.error(formatDiagnostic(d)); + warnCapped(warns.map(formatDiagnostic), flags.limit, { structured }); // -- the authoring lint: its own section, its own cap ---------------------- // Separate from the gate above because it makes a different claim. The gate @@ -707,7 +717,7 @@ export async function verifyCommand( `meta verify — requirements: ${lint.length} authoring warning(s) ` + `(advisory — does not fail the build):`, ); - warnCapped(lint.map(fmtDiag), flags.limit, { structured }); + warnCapped(lint.map(formatDiagnostic), flags.limit, { structured }); } // Everything this pass found, uncapped, for the structured payload — gate @@ -725,6 +735,44 @@ export async function verifyCommand( return 0; } + // -- overlay authoring lint (FR-023 §11.1 item 4, Task 17) ------------------ + // Its own section, its own cap — same discipline as the requirement lint + // above: a noisy section must not push another section's findings off the + // end of a capped run. This lint has NO gate half at all (no + // "ledger disagrees with the model" claim to make), so nothing here can + // ever reach the exit code — records its result either way, same as the + // anti-pattern pass below. + // + // Reads raw file content (via `lintOverlays` + `declaredTopLevelKeys`), never + // the loaded/merged model: the merge has already lost which FILE contributed + // which declaration, and that is exactly what this lint needs to name. + async function runOverlayLintAdvisory(): Promise { + if (flags.noOverlayLint) { + overlaySection = skippedSection("suppressed by --no-overlay-lint"); + return; + } + if (process.env.META_NO_OVERLAY_LINT === "1") { + overlaySection = skippedSection("suppressed by META_NO_OVERLAY_LINT=1"); + return; + } + let findings: Diagnostic[]; + try { + findings = await lintOverlays(collection, (path) => new FileSource(path)); + } catch (err) { + // Never let an advisory scan break verify — and never report it as clean. + overlaySection = skippedSection(`the overlay lint failed: ${(err as Error).message}`); + return; + } + overlaySection = ranSection(findings.map((d) => toDiagnosticRow(d, "lint"))); + if (findings.length > 0) { + log.warn( + `meta verify — overlays: ${findings.length} unflagged cross-file redeclaration(s) ` + + `(advisory — does not fail the build):`, + ); + warnCapped(findings.map(formatDiagnostic), flags.limit, { structured }); + } + } + // -- verify-as-teacher (advisory) ------------------------------------------ // Records its result EITHER WAY — a skip carries its reason rather than looking // like a clean scan. Warnings only; nothing here reaches the exit code (the @@ -1537,6 +1585,17 @@ interface VerifyGateRow { ok: boolean; } +/** + * Format one `Diagnostic` as a printed TEXT line. Named `formatDiagnostic`, not + * `fmt`: `fmt` is this command's OUTPUT FORMAT parameter, and a shadow of it + * anywhere near this code is how a structured run quietly reverts to text. + * Shared by the requirement gate/lint AND the overlay lint — every advisory or + * gate pass that speaks in `Diagnostic` prints it identically. + */ +function formatDiagnostic(d: Diagnostic): string { + return ` ${d.code}${d.path !== undefined ? ` [${d.path}]` : ""}: ${d.message}`; +} + /** Project a requirement diagnostic into a payload row. */ function toDiagnosticRow(d: Diagnostic, source: "gate" | "lint"): AdvisoryDiagnosticRow { return { @@ -1568,6 +1627,7 @@ function buildVerifyPayload(input: { requirements: AdvisorySection; requirementCounts: RequirementCounts | undefined; antiPatterns: AdvisorySection; + overlays: AdvisorySection; }): Record { const ran = input.gates.filter((g) => g.ran); const failed = ran.filter((g) => !g.ok); @@ -1582,6 +1642,9 @@ function buildVerifyPayload(input: { if (input.requirements.total > 0) { parts.push(`${input.requirements.total} requirement diagnostic(s)`); } + if (input.overlays.status === "ran" && input.overlays.total > 0) { + parts.push(`${input.overlays.total} overlay authoring finding(s)`); + } const help: string[] = []; if (failed.length > 0) { @@ -1594,7 +1657,17 @@ function buildVerifyPayload(input: { `${input.antiPatterns.total} authored site(s) hand-roll what MetaObjects can model — see antiPatterns.rows[] and run \`meta types \``, ); } - if (failed.length === 0 && input.antiPatterns.total === 0 && input.requirements.total === 0) { + if (input.overlays.total > 0) { + help.push( + `${input.overlays.total} unflagged cross-file redeclaration(s) — see overlays.rows[]; add overlay: true so a renamed or removed target fails loudly instead of silently becoming a new object`, + ); + } + if ( + failed.length === 0 && + input.antiPatterns.total === 0 && + input.requirements.total === 0 && + input.overlays.total === 0 + ) { help.push("no drift and nothing advisory to answer — nothing to do"); } @@ -1605,6 +1678,7 @@ function buildVerifyPayload(input: { help, antiPatterns: input.antiPatterns, requirements: input.requirements, + overlays: input.overlays, ...(input.requirementCounts !== undefined ? { requirementCounts: input.requirementCounts } : {}), // The honest boundary. Everything named here is REACHABLE — it is printed as // text on stderr — but it is not in this document, and a reader must not have diff --git a/server/typescript/packages/cli/src/index.ts b/server/typescript/packages/cli/src/index.ts index 22ba1d956..0d76b4378 100644 --- a/server/typescript/packages/cli/src/index.ts +++ b/server/typescript/packages/cli/src/index.ts @@ -115,6 +115,8 @@ VERIFY FLAGS (ADR-0021 D2 — explicit subverbs; combine any; exit 1 on ANY drif --no-antipatterns Suppress the advisory "you hand-rolled what MetaObjects can model" pass (aggregate/currency/enum hints; warnings only) --no-requirement-lint Suppress the advisory requirement AUTHORING lint (not the gate) + --no-overlay-lint Suppress the advisory overlay-redeclaration AUTHORING lint + (never a gate — this lint can't fail the build) --limit How many advisory lines TEXT output prints per section before it truncates (default 20). Never applies to --format toon/json, which carry every finding and every diagnostic. @@ -282,6 +284,8 @@ FLAGS: migrating a model onto a registered provider. --no-antipatterns Suppress the advisory "hand-rolled what MetaObjects can model" pass --no-requirement-lint Suppress the advisory requirement AUTHORING lint (not the gate) + --no-overlay-lint Suppress the advisory overlay-redeclaration AUTHORING lint — + never a gate; this lint can't fail the build --limit Advisory lines TEXT output prints PER SECTION before truncating (default 20; per-section so the authoring lint can never push the gate's own warnings off the end) @@ -303,6 +307,13 @@ lint in its own section: names that are not addressable, prose slots holding one sentence twice, content written where no surface reads it. Warnings only — it can never fail the build. Opt out with --no-requirement-lint or META_NO_REQUIREMENT_LINT=1. The requirements GATE itself (dangling refs, link floor, levels) always runs. + +verify also prints an ADVISORY overlay authoring lint, in its own section: a +top-level declaration redeclared in two or more files where more than one +redeclaration lacks 'overlay: true' — today's default merge rule reuses it +silently, but the same unflagged redeclaration silently becomes a NEW object +the day the target is renamed or removed upstream. Warnings only — it can +never fail the build. Opt out with --no-overlay-lint or META_NO_OVERLAY_LINT=1. `, export: `meta export — flatten loaded metadata to one canonical JSON artifact diff --git a/server/typescript/packages/cli/src/lib/args.ts b/server/typescript/packages/cli/src/lib/args.ts index cb8a056e7..d9864e3b3 100644 --- a/server/typescript/packages/cli/src/lib/args.ts +++ b/server/typescript/packages/cli/src/lib/args.ts @@ -318,6 +318,16 @@ export interface VerifyFlags { * the half that CAN fail a build switched on. Same shape as --no-antipatterns. */ noRequirementLint: boolean; + /** + * Suppress the advisory overlay AUTHORING lint (FR-023 §11.1 item 4) — the + * finding that an unflagged cross-file redeclaration works today only + * because the parser's default merge rule reuses the existing node by + * (type, resolutionKey); it silently becomes a NEW object the day the + * target is renamed or removed upstream. Same shape as + * --no-requirement-lint: mutes the advisory half only, never a gate (this + * lint has no gate half at all — it can never fail the build). + */ + noOverlayLint: boolean; /** * ADR-0023 strict-attr load opt-OUT (#96). `verify` is strict-by-default — an * undeclared/typo'd own `@attr` fails verify (ERR_UNKNOWN_ATTR). `--lax` @@ -357,6 +367,7 @@ export const VERIFY_OPTIONS = { "replay-snapshot": { type: "boolean", default: false }, "no-antipatterns": { type: "boolean", default: false }, "no-requirement-lint": { type: "boolean", default: false }, + "no-overlay-lint": { type: "boolean", default: false }, lax: { type: "boolean", default: false }, "d1": { type: "string" }, "remote": { type: "boolean", default: false }, @@ -435,6 +446,7 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { anyExplicit, noAntipatterns: !!values["no-antipatterns"], noRequirementLint: !!values["no-requirement-lint"], + noOverlayLint: !!values["no-overlay-lint"], lax: !!values.lax, d1: values.d1 as string | undefined, remote: !!values.remote, diff --git a/server/typescript/packages/cli/src/lib/overlay-lint.ts b/server/typescript/packages/cli/src/lib/overlay-lint.ts new file mode 100644 index 000000000..42221b004 --- /dev/null +++ b/server/typescript/packages/cli/src/lib/overlay-lint.ts @@ -0,0 +1,129 @@ +// server/typescript/packages/cli/src/lib/overlay-lint.ts +// +// `meta verify` — the overlay AUTHORING lint (FR-023 §11.1 item 4). +// +// The parser's own merge rule (parser-core.ts, "Default: no operator → +// silently reuse existing or create new") means a top-level redeclaration of +// an existing node works TODAY whether or not it carries `overlay: true` — +// as long as the target still exists under the same (type, resolutionKey). +// That is exactly the trap: a consumer who redeclares an imported node +// without the flag gets today's silent merge for free, and gets a silent NEW +// OBJECT the day the upstream node is renamed or removed — no error, just a +// second, disconnected node under the same name. `overlay: true` turns that +// into a loud `ERR_OVERLAY_NO_TARGET` instead. +// +// This lint does not change that parser behavior (nor could it — it reads +// raw file content, never the parser's merge decisions). It tells the AUTHOR: +// every top-level (type, resolutionKey) declared in two or more of the +// collection's files where more than one declaration lacks the flag. Exactly +// one unflagged declaration is the base and is fine; every additional +// unflagged one is a finding. +// +// Advisory only, like its sibling in requirement-lint.ts: never fails +// `meta verify`, never changes load behavior. Deliberately NO severity +// constant to flip — see requirement-lint.ts's header for why that promise +// would be false here too. + +import { declaredTopLevelKeys, type DeclaredTopLevelKey, type MetaDataSource } from "@metaobjectsdev/metadata"; +import type { Collection } from "@metaobjectsdev/sdk"; +import { relPosix } from "./rel-posix.js"; +import type { Diagnostic } from "./requirement-check.js"; + +export const WARN_OVERLAY_IMPLICIT = "WARN_OVERLAY_IMPLICIT"; + +/** + * Resolve one collection file path to the `MetaDataSource` `lintOverlays` + * reads it through. Injected so the lint never touches the filesystem + * directly: the production wiring in `verify.ts` backs this with + * `FileSource` (from `@metaobjectsdev/metadata/core`); a test can supply an + * in-memory source instead. + */ +export type ReadSource = (path: string) => MetaDataSource; + +function warn(code: string, message: string): Diagnostic { + return { severity: "warn", code, message }; +} + +/** One file's declaration of a given (type, resolutionKey). */ +interface Declaration { + readonly file: string; + readonly overlay: boolean; +} + +/** + * Lint the collection's files for unflagged cross-file redeclarations. + * + * Reads every file in `collection.files` — dependency artifacts first, since + * `Collection.files` already orders them that way and they are the BASES a + * consumer's own declarations amend — and structurally scans each with + * {@link declaredTopLevelKeys} (never the loaded/merged model: the merged + * tree has already lost which FILE contributed which declaration). + * + * Groups by (type, resolutionKey) — mirroring the parser's own merge lookup, + * which matches on type AND resolutionKey (parser-core.ts) — and reports + * every unflagged declaration after the first one for a key with 2+ + * declarations. The first unflagged declaration is the base. + * + * `file` in each finding's message is the collection-relative path + * (`relPosix`), or the dependency's `dep:/` id when the + * declaration came from an imported artifact. + * + * Unreadable or unparsable files are skipped — the loader itself reports + * those (as a load error / `meta verify`'s own failure path); duplicating + * that diagnosis here would just be a second, worse-informed version of the + * same message. + */ +export async function lintOverlays( + collection: Collection, + readSource: ReadSource, +): Promise { + // type -> resolutionKey -> every declaration seen for it, in file order. + const byType = new Map>(); + + for (const path of collection.files) { + let declared: ReadonlyArray; + try { + const source = readSource(path); + const content = await source.read(); + declared = await declaredTopLevelKeys(content, source.format); + } catch { + continue; + } + const file = collection.fileIds.get(path) ?? relPosix(collection.configDir, path); + for (const decl of declared) { + let byKey = byType.get(decl.type); + if (byKey === undefined) { + byKey = new Map(); + byType.set(decl.type, byKey); + } + const list = byKey.get(decl.key) ?? []; + list.push({ file, overlay: decl.overlay }); + byKey.set(decl.key, list); + } + } + + const diags: Diagnostic[] = []; + for (const byKey of byType.values()) { + for (const [fqn, declarations] of byKey) { + if (declarations.length < 2) continue; + let baseSeen = false; + for (const decl of declarations) { + if (decl.overlay) continue; + if (!baseSeen) { + // The first unflagged declaration is the base — nothing to report. + baseSeen = true; + continue; + } + diags.push( + warn( + WARN_OVERLAY_IMPLICIT, + `${fqn} is redeclared in ${decl.file} without overlay: true — add the flag ` + + `so a removed or renamed target fails loudly (ERR_OVERLAY_NO_TARGET) instead ` + + `of silently becoming a new object`, + ), + ); + } + } + } + return diags; +} diff --git a/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap b/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap index b576a112a..80da2b40b 100644 --- a/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap +++ b/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap @@ -97,6 +97,8 @@ VERIFY FLAGS (ADR-0021 D2 — explicit subverbs; combine any; exit 1 on ANY drif --no-antipatterns Suppress the advisory "you hand-rolled what MetaObjects can model" pass (aggregate/currency/enum hints; warnings only) --no-requirement-lint Suppress the advisory requirement AUTHORING lint (not the gate) + --no-overlay-lint Suppress the advisory overlay-redeclaration AUTHORING lint + (never a gate — this lint can't fail the build) --limit How many advisory lines TEXT output prints per section before it truncates (default 20). Never applies to --format toon/json, which carry every finding and every diagnostic. diff --git a/server/typescript/packages/cli/test/unit/args-verify.test.ts b/server/typescript/packages/cli/test/unit/args-verify.test.ts index d61199c42..d70ab1f54 100644 --- a/server/typescript/packages/cli/test/unit/args-verify.test.ts +++ b/server/typescript/packages/cli/test/unit/args-verify.test.ts @@ -6,7 +6,7 @@ describe("parseVerifyArgs", () => { test("defaults: prompts/db/dialect undefined, allow empty, skipSchema false, no explicit subverb", () => { expect(parseVerifyArgs([])).toEqual({ prompts: undefined, db: undefined, dialect: undefined, allow: [], skipSchema: false, - templates: false, codegen: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, lax: false, + templates: false, codegen: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, noOverlayLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -14,7 +14,7 @@ describe("parseVerifyArgs", () => { test("--prompts is captured", () => { expect(parseVerifyArgs(["--prompts", "templates"])).toEqual({ prompts: "templates", db: undefined, dialect: undefined, allow: [], skipSchema: false, - templates: false, codegen: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, lax: false, + templates: false, codegen: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, noOverlayLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -22,7 +22,7 @@ describe("parseVerifyArgs", () => { test("--db / --dialect / --skip-schema are captured", () => { expect(parseVerifyArgs(["--db", "file:x.db", "--dialect", "sqlite", "--skip-schema"])).toEqual({ prompts: undefined, db: "file:x.db", dialect: "sqlite", allow: [], skipSchema: true, - templates: false, codegen: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, lax: false, + templates: false, codegen: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, noOverlayLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -32,7 +32,7 @@ describe("parseVerifyArgs", () => { test("--dialect d1 --d1 --remote are captured; --dialect d1 alone is an explicit subverb", () => { expect(parseVerifyArgs(["--dialect", "d1", "--d1", "DB", "--remote"])).toEqual({ prompts: undefined, db: undefined, dialect: "d1", allow: [], skipSchema: false, - templates: false, codegen: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, lax: false, + templates: false, codegen: false, docs: false, deps: false, anyExplicit: true, noAntipatterns: false, noRequirementLint: false, noOverlayLint: false, lax: false, replay: false, replaySnapshot: false, d1: "DB", remote: true, limit: DEFAULT_ADVISORY_LIMIT, }); @@ -56,7 +56,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs(["--allow", "drop-column,drop-table"])).toEqual({ prompts: undefined, db: undefined, dialect: undefined, allow: ["drop-column", "drop-table"], skipSchema: false, - templates: false, codegen: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, lax: false, + templates: false, codegen: false, docs: false, deps: false, anyExplicit: false, noAntipatterns: false, noRequirementLint: false, noOverlayLint: false, lax: false, replay: false, replaySnapshot: false, d1: undefined, remote: false, limit: DEFAULT_ADVISORY_LIMIT, }); diff --git a/server/typescript/packages/cli/test/verify-overlay-lint.test.ts b/server/typescript/packages/cli/test/verify-overlay-lint.test.ts new file mode 100644 index 000000000..631b2e2b1 --- /dev/null +++ b/server/typescript/packages/cli/test/verify-overlay-lint.test.ts @@ -0,0 +1,258 @@ +// `meta verify` — the overlay authoring lint (FR-023 §11.1 item 4, Task 17). +// +// The parser's own merge rule (parser-core.ts: "Default: no operator → +// silently reuse existing or create new") means an unflagged top-level +// redeclaration of an EXISTING node works today, whether or not it carries +// `overlay: true` — as long as the target still exists under the same +// (type, resolutionKey). The lint exists because that silent success hides a +// real risk: the same unflagged redeclaration becomes a silent NEW object, +// with no error at all, the day the target is renamed or removed. This suite +// drives `meta verify` end-to-end (temp dirs + `run()`, same pattern as +// `verify-requirements-imported.test.ts` / `gen-imported-nodes.test.ts`) +// rather than unit-testing `lintOverlays` directly, because the finding is a +// property of an on-disk collection of files, not of one loaded model. +import { describe, test, expect, beforeEach, afterEach, afterAll } from "bun:test"; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync, cpSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { run } from "../src/index.js"; + +const DEP_NAME = "acme-common"; +const ARTIFACT_BASENAME = "acme-common.metaobjects.json"; +// The real, committed dependency-conformance fixture — its bytes hash to the +// pinned `sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d` +// (global-constraints.md's LOCK_V1), and it declares acme::common::Address, +// acme::common::Audited and acme::common::Customer. Reused verbatim rather than +// hand-rolled so this test's snapshot+lock agree with the pinned hash by +// construction, not by a second, easy-to-drift computation. +const ARTIFACT_FIXTURE = resolve( + import.meta.dirname, + "../../../../../fixtures/dependency-conformance/artifacts/acme-common-v1.json", +); + +/** The reference lock (global-constraints.md's LOCK_V1), verbatim. */ +const LOCK_V1 = JSON.stringify({ + schema_version: 1, + dependencies: { + [DEP_NAME]: { + version: "1.0.0", + metamodelVersion: "1.0", + resolvedFrom: { path: "../acme-common/metaobjects" }, + artifact: ARTIFACT_BASENAME, + integrity: "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + packages: ["acme::common"], + nodes: ["acme::common::Address", "acme::common::Audited", "acme::common::Customer"], + }, + }, +}); + +/** The reference config (global-constraints.md's CONFIG_REF), verbatim. */ +const CONFIG_REF = JSON.stringify({ + schema_version: 1, + sources: [], + dependencies: [{ name: DEP_NAME, path: "../acme-common/metaobjects" }], +}); + +/** The consumer's own model: one entity it owns (global-constraints.md's APP). */ +const APP = JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Order", + children: [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, +}); + +/** The consumer redeclaring the dependency's Customer, adding a view.text + * child — with or without `overlay: true` on the top-level declaration. */ +function customerOverlayFile(flagged: boolean): string { + return JSON.stringify({ + "metadata.root": { + package: "acme::common", + children: [ + { + "object.entity": { + name: "Customer", + ...(flagged ? { overlay: true } : {}), + children: [ + { + "field.string": { + name: "email", + children: [{ "view.text": { name: "emailView" } }], + }, + }, + ], + }, + }, + ], + }, + }); +} + +const dirs: string[] = []; +afterAll(() => { + for (const d of dirs) rmSync(d, { recursive: true, force: true }); +}); + +/** A consumer of acme-common, with `metaobjects/meta.ov.json` redeclaring + * acme::common::Customer — flagged or not. */ +function overlayProject(flagged: boolean): string { + const root = mkdtempSync(join(tmpdir(), "vov-")); + dirs.push(root); + + mkdirSync(join(root, "metaobjects"), { recursive: true }); + writeFileSync(join(root, "metaobjects", "meta.app.json"), APP, "utf8"); + writeFileSync(join(root, "metaobjects", "meta.ov.json"), customerOverlayFile(flagged), "utf8"); + + const depDir = join(root, ".metaobjects", "deps", DEP_NAME); + mkdirSync(depDir, { recursive: true }); + cpSync(ARTIFACT_FIXTURE, join(depDir, ARTIFACT_BASENAME)); + + writeFileSync(join(root, ".metaobjects", "config.json"), CONFIG_REF, "utf8"); + writeFileSync(join(root, ".metaobjects", "deps.lock.json"), LOCK_V1, "utf8"); + return root; +} + +/** Three own files all declaring app::Subscriber, none flagged. Alphabetical + * basename order (a, b, c) is the discovery order the loader/collection use. */ +function threeWayProject(): string { + const root = mkdtempSync(join(tmpdir(), "vov-3way-")); + dirs.push(root); + mkdirSync(join(root, "metaobjects"), { recursive: true }); + writeFileSync( + join(root, "metaobjects", "meta.a.json"), + JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Subscriber", + children: [ + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + ], + }, + }), + "utf8", + ); + writeFileSync( + join(root, "metaobjects", "meta.b.json"), + JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { "object.entity": { name: "Subscriber", children: [{ "field.string": { name: "email" } }] } }, + ], + }, + }), + "utf8", + ); + writeFileSync( + join(root, "metaobjects", "meta.c.json"), + JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { "object.entity": { name: "Subscriber", children: [{ "field.string": { name: "name" } }] } }, + ], + }, + }), + "utf8", + ); + return root; +} + +let out: string[]; +let err: string[]; +let origLog: typeof console.log; +let origErr: typeof console.error; + +beforeEach(() => { + out = []; + err = []; + origLog = console.log; + origErr = console.error; + console.log = (...a: unknown[]) => { out.push(a.map(String).join(" ")); }; + console.error = (...a: unknown[]) => { err.push(a.map(String).join(" ")); }; +}); +afterEach(() => { + console.log = origLog; + console.error = origErr; +}); + +describe("meta verify — the overlay authoring lint (FR-023 §11.1 item 4)", () => { + test("(a) an unflagged redeclaration of an imported node is an advisory finding, exit 0", async () => { + const root = overlayProject(false); + const exit = await run(["verify", "--format", "text", "--cwd", root]); + expect(exit).toBe(0); + + const all = [...out, ...err].join("\n"); + expect(all).toContain( + "acme::common::Customer is redeclared in metaobjects/meta.ov.json without overlay: true", + ); + }); + + test("(b) the same redeclaration WITH overlay: true produces no finding", async () => { + const root = overlayProject(true); + const exit = await run(["verify", "--format", "text", "--cwd", root]); + expect(exit).toBe(0); + + const all = [...out, ...err].join("\n"); + expect(all).not.toContain("is redeclared in"); + expect(all).not.toContain("acme::common::Customer"); + }); + + test("(c) three own files declaring the same node unflagged: two findings, naming the 2nd and 3rd files", async () => { + const root = threeWayProject(); + const exit = await run(["verify", "--format", "text", "--cwd", root]); + expect(exit).toBe(0); + + const all = [...out, ...err].join("\n"); + expect(all).toContain("app::Subscriber is redeclared in metaobjects/meta.b.json without overlay: true"); + expect(all).toContain("app::Subscriber is redeclared in metaobjects/meta.c.json without overlay: true"); + // The FIRST unflagged declaration is the base — never reported as a redeclaration. + expect(all).not.toContain("redeclared in metaobjects/meta.a.json"); + // Exactly two findings, not more (e.g. no third finding produced by comparing + // b against c a second time). + const occurrences = all.split("app::Subscriber is redeclared in").length - 1; + expect(occurrences).toBe(2); + }); + + test("(d) --no-overlay-lint silences the finding from (a)", async () => { + const root = overlayProject(false); + const exit = await run(["verify", "--format", "text", "--cwd", root, "--no-overlay-lint"]); + expect(exit).toBe(0); + + const all = [...out, ...err].join("\n"); + expect(all).not.toContain("is redeclared in"); + }); + + test("META_NO_OVERLAY_LINT=1 also silences the finding from (a)", async () => { + const root = overlayProject(false); + const prev = process.env.META_NO_OVERLAY_LINT; + process.env.META_NO_OVERLAY_LINT = "1"; + try { + const exit = await run(["verify", "--format", "text", "--cwd", root]); + expect(exit).toBe(0); + } finally { + if (prev === undefined) delete process.env.META_NO_OVERLAY_LINT; + else process.env.META_NO_OVERLAY_LINT = prev; + } + + const all = [...out, ...err].join("\n"); + expect(all).not.toContain("is redeclared in"); + }); +}); diff --git a/server/typescript/packages/metadata/src/index.ts b/server/typescript/packages/metadata/src/index.ts index 00c1ed70a..2822af835 100644 --- a/server/typescript/packages/metadata/src/index.ts +++ b/server/typescript/packages/metadata/src/index.ts @@ -245,8 +245,10 @@ export { resolveSuperRef } from "./super-resolve.js"; export { expandRef, isRelativeRef, refMatchesObject, resolveObjectRef, didYouMeanHint, REF_BEARING_ATTR_NAMES } from "./naming-refs.js"; // Loader hierarchy -export { MetaDataLoader } from "./loader/meta-data-loader.js"; -export type { LoadOptions, LoadResult, LoadingState, DirectoryFactoryOptions } from "./loader/meta-data-loader.js"; +export { MetaDataLoader, declaredTopLevelKeys } from "./loader/meta-data-loader.js"; +export type { + LoadOptions, LoadResult, LoadingState, DirectoryFactoryOptions, DeclaredTopLevelKey, +} from "./loader/meta-data-loader.js"; export { InMemoryStringSource } from "./loader/meta-data-source.js"; export type { MetaDataSource, MetaDataFormat } from "./loader/meta-data-source.js"; diff --git a/server/typescript/packages/metadata/src/loader/meta-data-loader.ts b/server/typescript/packages/metadata/src/loader/meta-data-loader.ts index f918bdf84..799466a0e 100644 --- a/server/typescript/packages/metadata/src/loader/meta-data-loader.ts +++ b/server/typescript/packages/metadata/src/loader/meta-data-loader.ts @@ -13,7 +13,14 @@ import { TypeId, TypeRegistry } from "../registry.js"; import { coreProviders } from "../core-types.js"; import { composeRegistry } from "../provider.js"; import { TYPE_METADATA, SUBTYPE_ROOT } from "../shared/base-types.js"; -import { RESERVED_KEY_CHILDREN, RESERVED_KEY_OVERLAY } from "../shared/structural.js"; +import { + PACKAGE_SEPARATOR, + RESERVED_KEY_CHILDREN, + RESERVED_KEY_NAME, + RESERVED_KEY_OVERLAY, + RESERVED_KEY_PACKAGE, + TYPE_SUBTYPE_SEPARATOR, +} from "../shared/structural.js"; import { ParseError } from "../errors.js"; import type { LoaderWarning } from "../source.js"; import { codeSource, resolvedSource } from "../source.js"; @@ -97,6 +104,104 @@ function makeSyntheticRoot(): MetaRoot { return new MetaRoot(new TypeId(TYPE_METADATA, SUBTYPE_ROOT), ""); } +// --------------------------------------------------------------------------- +// declaredTopLevelKeys — structural pre-parse walk (#160, generalized) +// --------------------------------------------------------------------------- + +/** One root-level declaration as `declaredTopLevelKeys` structurally scans it — + * before registry-driven parsing, super resolution, or merge. */ +export interface DeclaredTopLevelKey { + /** The wrapper key's TYPE segment (e.g. "object" for "object.entity"). A bare + * wrapper key with no "." is its own type — this walk never consults the + * registry for a default subType, because it never needs the subType. */ + type: string; + /** The resolution key the declaration would carry once parsed: the child's + * own `package` if set, else the root's `package`, then `::`, then its + * `name`. Mirrors `rootChildResolutionKey` (parser-core.ts) minus relative + * (`::`-prefixed) package-path expansion, which no root-level declaration + * in this walk's callers (the overlay-only partition, the overlay lint) + * needs — both compare whole declared packages, never relative ones. */ + key: string; + /** Whether the declaration's body carries `overlay: true`. */ + overlay: boolean; +} + +/** + * Structurally scan a source's raw content (JSON via `JSON.parse`; sigil-free + * authoring YAML via the raw YAML walker — `overlay: true` is a bare key + * before desugar) and report every top-level declaration under + * `metadata.root.children`: its type, the resolution key it would carry once + * parsed, and whether it carries `overlay: true`. + * + * A declaration with no (string, non-empty) `name` is skipped — a resolution + * key can't be computed for it, and the real parser will report it as a + * loader error rather than silently reusing or creating a node. Malformed + * root shapes (root missing, not an object, `children` absent or not an + * array) return `[]` rather than throwing — the real parse loop is where a + * genuine structural error surfaces; this walk must never crash a caller + * that is only trying to answer "what does this file declare". + * + * Generalizes what `_rootIsOverlayOnly` used to do just for the overlay-only + * partition (#160): `_isOverlayOnlySource` is now expressed in terms of this + * function, and the overlay authoring lint (`meta verify`) is its second + * caller. + */ +export async function declaredTopLevelKeys( + content: string, + format: MetaDataFormat, +): Promise> { + // Strip UTF-8 BOM if present (mirrors parseJson / parseYaml). + const normalized = content.charCodeAt(0) === 0xfeff ? content.slice(1) : content; + let parsed: unknown; + if (format === "json") { + parsed = JSON.parse(normalized); + } else if (format === "yaml") { + const { parseYamlWithPositions } = await import("../core/yaml-positions-walker.js"); + parsed = parseYamlWithPositions(normalized).value; + } else { + return []; + } + return declaredTopLevelKeysFromParsedRoot(parsed); +} + +/** The structural walk `declaredTopLevelKeys` performs once content is + * structurally parsed — split out so it takes no format/BOM concerns. */ +function declaredTopLevelKeysFromParsedRoot(parsed: unknown): ReadonlyArray { + if (typeof parsed !== "object" || parsed === null) return []; + const parsedRecord = parsed as Record; + // Canonical JSON always fuses the subType onto the root wrapper key + // ("metadata.root"); sigil-free YAML authoring writes the bare type + // ("metadata") and leaves the default subType to the registry — this + // structural walk runs BEFORE desugar/registry resolution, so it must + // accept both spellings rather than only ever matching JSON's. + const explicitRootKey = `${TYPE_METADATA}${TYPE_SUBTYPE_SEPARATOR}${SUBTYPE_ROOT}`; // "metadata.root" + const rootBody = parsedRecord[explicitRootKey] ?? parsedRecord[TYPE_METADATA]; + if (typeof rootBody !== "object" || rootBody === null) return []; + const rawRootPkg = (rootBody as Record)[RESERVED_KEY_PACKAGE]; + const rootPkg = typeof rawRootPkg === "string" ? rawRootPkg : ""; + const children = (rootBody as Record)[RESERVED_KEY_CHILDREN]; + if (!Array.isArray(children)) return []; + + const declared: DeclaredTopLevelKey[] = []; + for (const child of children) { + if (typeof child !== "object" || child === null) continue; + // Each child is a single-key wrapper: { "object.entity": { ... } }. + for (const [wrapperKey, body] of Object.entries(child as Record)) { + if (typeof body !== "object" || body === null) continue; + const bodyRecord = body as Record; + const name = bodyRecord[RESERVED_KEY_NAME]; + if (typeof name !== "string" || name === "") continue; + const dotIdx = wrapperKey.indexOf(TYPE_SUBTYPE_SEPARATOR); + const type = dotIdx < 0 ? wrapperKey : wrapperKey.slice(0, dotIdx); + const rawOwnPkg = bodyRecord[RESERVED_KEY_PACKAGE]; + const pkg = typeof rawOwnPkg === "string" && rawOwnPkg !== "" ? rawOwnPkg : rootPkg; + const key = pkg !== "" ? `${pkg}${PACKAGE_SEPARATOR}${name}` : name; + declared.push({ type, key, overlay: bodyRecord[RESERVED_KEY_OVERLAY] === true }); + } + } + return declared; +} + // --------------------------------------------------------------------------- // MetaDataLoader class // --------------------------------------------------------------------------- @@ -366,54 +471,16 @@ export class MetaDataLoader { } /** - * Structurally scan a source's raw content (JSON via JSON.parse; sigil-free - * authoring YAML via the raw YAML walker — `overlay: true` is a bare key - * before desugar) and report whether every top-level object declaration under - * `metadata.root.children` carries `overlay: true` (and there is at least one). + * Whether every top-level declaration in a source's raw content carries + * `overlay: true` (and there is at least one) — re-expressed over the + * generalized structural walk, {@link declaredTopLevelKeys}. */ private static async _isOverlayOnlySource( content: string, format: MetaDataFormat, ): Promise { - // Strip UTF-8 BOM if present (mirrors parseJson / parseYaml). - const normalized = content.charCodeAt(0) === 0xfeff ? content.slice(1) : content; - let parsed: unknown; - if (format === "json") { - parsed = JSON.parse(normalized); - } else if (format === "yaml") { - const { parseYamlWithPositions } = await import( - "../core/yaml-positions-walker.js" - ); - parsed = parseYamlWithPositions(normalized).value; - } else { - return false; - } - return MetaDataLoader._rootIsOverlayOnly(parsed); - } - - /** True when the structurally-parsed root has ≥1 child and every top-level - * child node carries `overlay: true` (declares no base objects). */ - private static _rootIsOverlayOnly(parsed: unknown): boolean { - if (typeof parsed !== "object" || parsed === null) return false; - const rootKey = `${TYPE_METADATA}.${SUBTYPE_ROOT}`; // "metadata.root" - const rootBody = (parsed as Record)[rootKey]; - if (typeof rootBody !== "object" || rootBody === null) return false; - const children = (rootBody as Record)[RESERVED_KEY_CHILDREN]; - if (!Array.isArray(children) || children.length === 0) return false; - return children.every((child) => { - if (typeof child !== "object" || child === null) return false; - // Each child is a single-key wrapper: { "object.projection": { ... } }. - const bodies = Object.values(child as Record); - return ( - bodies.length > 0 && - bodies.every( - (body) => - typeof body === "object" && - body !== null && - (body as Record)[RESERVED_KEY_OVERLAY] === true, - ) - ); - }); + const declared = await declaredTopLevelKeys(content, format); + return declared.length > 0 && declared.every((d) => d.overlay); } // --------------------------------------------------------------------------- diff --git a/server/typescript/packages/metadata/test/declared-top-level-keys.test.ts b/server/typescript/packages/metadata/test/declared-top-level-keys.test.ts new file mode 100644 index 000000000..947c4fa4e --- /dev/null +++ b/server/typescript/packages/metadata/test/declared-top-level-keys.test.ts @@ -0,0 +1,160 @@ +// declaredTopLevelKeys — the structural pre-parse walk `_isOverlayOnlySource` +// used to perform privately (#160), generalized to report every top-level +// declaration's (type, resolutionKey, overlay) rather than just a single +// "is this source overlay-only" boolean. +// +// Two callers exist post-generalization: the loader's own overlay-only-source +// partition (re-expressed over this function — see meta-data-loader.ts) and +// the `meta verify` overlay authoring lint (cli package, Task 17). +// +// FR-023 §11 (overlay authoring lint) task 17. + +import { describe, test, expect } from "bun:test"; +import { + declaredTopLevelKeys, MetaDataLoader, InMemoryStringSource, TYPE_OBJECT, TYPE_FIELD, +} from "../src/index.js"; + +describe("declaredTopLevelKeys — JSON", () => { + test("reports type, resolution key and overlay flag for each top-level child", async () => { + const content = JSON.stringify({ + "metadata.root": { + package: "app", + children: [ + { + "object.entity": { + name: "Order", + }, + }, + { + "object.entity": { + name: "Order", + package: "acme::common", + overlay: true, + }, + }, + ], + }, + }); + + const result = await declaredTopLevelKeys(content, "json"); + + expect(result).toEqual([ + { type: "object", key: "app::Order", overlay: false }, + { type: "object", key: "acme::common::Order", overlay: true }, + ]); + }); + + test("a source with no package falls back to the bare name as the key", async () => { + const content = JSON.stringify({ + "metadata.root": { + children: [{ "object.entity": { name: "Standalone" } }], + }, + }); + + const result = await declaredTopLevelKeys(content, "json"); + + expect(result).toEqual([{ type: "object", key: "Standalone", overlay: false }]); + }); + + test("a declaration with no name is skipped — no resolution key can be computed", async () => { + const content = JSON.stringify({ + "metadata.root": { + package: "app", + children: [{ "object.entity": {} }, { "object.entity": { name: "Kept" } }], + }, + }); + + const result = await declaredTopLevelKeys(content, "json"); + + expect(result).toEqual([{ type: "object", key: "app::Kept", overlay: false }]); + }); + + test("a malformed root (no metadata.root, children absent) returns an empty list rather than throwing", async () => { + expect(await declaredTopLevelKeys(JSON.stringify({}), "json")).toEqual([]); + expect( + await declaredTopLevelKeys(JSON.stringify({ "metadata.root": { package: "app" } }), "json"), + ).toEqual([]); + }); +}); + +describe("declaredTopLevelKeys — YAML (sigil-free authoring)", () => { + test("a bare `overlay: true` child reports overlay: true, before desugar", async () => { + const content = [ + "metadata:", + " package: app", + " children:", + " - object.entity:", + " name: Base", + " - object.entity:", + " name: Overlaid", + " overlay: true", + "", + ].join("\n"); + + const result = await declaredTopLevelKeys(content, "yaml"); + + expect(result).toEqual([ + { type: "object", key: "app::Base", overlay: false }, + { type: "object", key: "app::Overlaid", overlay: true }, + ]); + }); +}); + +describe("declaredTopLevelKeys — order-independence of the overlay-only partition", () => { + // #160's overlay-only-source partition (re-expressed over declaredTopLevelKeys + // by this task) exists precisely so an overlay-only source presented BEFORE + // its base still merges. Nothing in the suite pinned that ordering guarantee + // before this task generalized the walk it depends on — a regression here + // (e.g. dropping the stable-partition call, or a declaredTopLevelKeys bug + // that misreports a source as NOT overlay-only) would be silent. + test("an overlay-only source loaded BEFORE its base still merges into it", async () => { + const base = new InMemoryStringSource( + JSON.stringify({ + "metadata.root": { + package: "acme", + children: [ + { + "object.entity": { + name: "Widget", + children: [{ "field.string": { name: "sku" } }], + }, + }, + ], + }, + }), + { id: "base.json", format: "json" }, + ); + // Every top-level declaration in this source carries overlay: true, so it + // is overlay-only — declaredTopLevelKeys is what tells the partition that. + const overlay = new InMemoryStringSource( + JSON.stringify({ + "metadata.root": { + package: "acme", + children: [ + { + "object.entity": { + name: "Widget", + overlay: true, + children: [{ "field.string": { name: "warranty" } }], + }, + }, + ], + }, + }), + { id: "overlay.json", format: "json" }, + ); + + const loader = new MetaDataLoader({ freeze: false }); + // Overlay source FIRST in the array — the stable-partition must still + // move it to the end so it merges into `base` rather than raising + // ERR_OVERLAY_NO_TARGET (no existing "Widget" would be visible yet if the + // sources were parsed in the array's literal order). + const { root, errors } = await loader.load([overlay, base]); + + expect(errors).toEqual([]); + const widget = root.ownChildByTypeAndName(TYPE_OBJECT, "Widget"); + expect(widget).toBeDefined(); + expect(widget!.ownChildByTypeAndName(TYPE_FIELD, "sku")).toBeDefined(); + expect(widget!.ownChildByTypeAndName(TYPE_FIELD, "warranty")).toBeDefined(); + }); +}); From a58fc5b506b3fd4518303620da71d3d76f1c28d3 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 01:52:30 -0400 Subject: [PATCH 46/62] fix(metadata): declaredTopLevelKeys expands a relative package the same way the real parser does Reuses parser-core.ts's expandPackageForPath (now exported) instead of using an own `package` verbatim, so a `::`-prefixed package on a root-level declaration resolves to the same key MetaData.resolutionKey() produces on the loaded side. Fixes a silent mis-grouping risk in the overlay lint (Task 17 fix round 1). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../metadata/src/loader/meta-data-loader.ts | 21 +++++--- .../packages/metadata/src/parser-core.ts | 8 ++- .../test/declared-top-level-keys.test.ts | 52 +++++++++++++++++++ 3 files changed, 74 insertions(+), 7 deletions(-) diff --git a/server/typescript/packages/metadata/src/loader/meta-data-loader.ts b/server/typescript/packages/metadata/src/loader/meta-data-loader.ts index 799466a0e..702e80dd8 100644 --- a/server/typescript/packages/metadata/src/loader/meta-data-loader.ts +++ b/server/typescript/packages/metadata/src/loader/meta-data-loader.ts @@ -42,6 +42,7 @@ import { validateAttrSchema } from "../attr-schema-validate.js"; import type { MetaDataFormat, MetaDataSource } from "./meta-data-source.js"; import { InMemoryStringSource } from "./meta-data-source.js"; import type { ParseOptions, ParseResult } from "../parser-core.js"; +import { expandPackageForPath } from "../parser-core.js"; // Local mirror of DirectorySource's options shape. Deliberately inlined here // (instead of `import type`'d from ./sources/directory-source.js) so the @@ -116,11 +117,16 @@ export interface DeclaredTopLevelKey { * registry for a default subType, because it never needs the subType. */ type: string; /** The resolution key the declaration would carry once parsed: the child's - * own `package` if set, else the root's `package`, then `::`, then its - * `name`. Mirrors `rootChildResolutionKey` (parser-core.ts) minus relative - * (`::`-prefixed) package-path expansion, which no root-level declaration - * in this walk's callers (the overlay-only partition, the overlay lint) - * needs — both compare whole declared packages, never relative ones. */ + * own `package` if set (expanded against the root's `package` via + * `expandPackageForPath` when it's a relative, `::`-prefixed path — e.g. + * root `acme` + own `::garage` → `acme::garage`), else the root's + * `package` verbatim, then `::`, then its `name`. Exactly + * `rootChildResolutionKey` (parser-core.ts) — including the relative-path + * expansion, reusing `expandPackageForPath` rather than reimplementing it, + * because a second copy is exactly how this walk and the real parser + * silently disagreed on a relative package (task 17 fix-round-1: this + * function used to return `::garage::Garage` for a fixture the real + * loader resolves to `acme::garage::Garage`). */ key: string; /** Whether the declaration's body carries `overlay: true`. */ overlay: boolean; @@ -194,7 +200,10 @@ function declaredTopLevelKeysFromParsedRoot(parsed: unknown): ReadonlyArray { test("reports type, resolution key and overlay flag for each top-level child", async () => { @@ -158,3 +167,46 @@ describe("declaredTopLevelKeys — order-independence of the overlay-only partit expect(widget!.ownChildByTypeAndName(TYPE_FIELD, "warranty")).toBeDefined(); }); }); + +describe("declaredTopLevelKeys — key agrees with the real loader on a relative package", () => { + // fix-round-1: this walk used to return "::garage::Garage" for this exact + // fixture (root package "acme", Garage's own "package": "::garage") — the + // relative-package expansion `rootChildResolutionKey` (parser-core.ts) + // performs via `expandPackageForPath` was missing here. The real loader + // resolves the same declaration to "acme::garage::Garage". Both sides of + // that comparison were wrong the same way before the fix, which is why + // nothing in the existing suite caught it — this test computes the + // expected value from the REAL loader rather than a hand-typed literal, so + // it cannot pass by both sides encoding the same mistake. + test("Garage's key (own package '::garage', root package 'acme') matches root.resolutionKey()", async () => { + const path = fixturePath("acme-vehicle-metadata.json"); + + // acme-vehicle-metadata.json's fields extend acme::common bases, so + // acme-common-metadata.json must load first — same two-file order + // round-trip.test.ts uses for this fixture pair. + const loader = new MetaDataLoader({ freeze: false }); + const { root, errors } = await loader.load([ + new FileSource(fixturePath("acme-common-metadata.json")), + new FileSource(path), + ]); + expect(errors).toEqual([]); + const garage = root.ownChildByTypeAndName(TYPE_OBJECT, "Garage"); + expect(garage).toBeDefined(); + const expectedKey = garage!.resolutionKey(); + // The fixture's relative spelling means a bug here still lands "some + // string" that ends in "::Garage" — assert the real loader actually + // performed the expansion too (i.e. this isn't vacuously comparing two + // wrong-but-equal strings), then compare declaredTopLevelKeys against it. + expect(expectedKey).toBe(`acme${PACKAGE_SEPARATOR}garage${PACKAGE_SEPARATOR}Garage`); + + const content = await readFile(path, "utf-8"); + const declared = await declaredTopLevelKeys(content, "json"); + // Located by the AUTHORED name (independent of the key-computation this + // test exists to check), not by re-deriving the expected key. + const garageDecl = declared.find( + (d) => d.type === TYPE_OBJECT && d.key.endsWith(`${PACKAGE_SEPARATOR}Garage`), + ); + expect(garageDecl).toBeDefined(); + expect(garageDecl!.key).toBe(expectedKey); + }); +}); From ab5ffb6c0accd3a845ddf2af99ad432e83541152 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 01:58:29 -0400 Subject: [PATCH 47/62] docs(fr-023): correct two stale claims in the phase-1a plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both found while executing Task 17; both verified against the tree before editing. - Task 17's Step 2 asserted "NO existing test pins the ordering guarantee itself" for the overlay-last source partition. False: `sdk/test/order-independence.test.ts` permutes all six orderings of three real files — overlay-before-base included — through the real loader, and its own comment records that disabling the partition makes three of the six throw ERR_OVERLAY_NO_TARGET. The earlier sweep missed it by scoping the search to `metadata/test` when the test lives in `sdk/test`; the plan's own suggested discovery command would have found it at repo scope. - Task 20 told the implementer to regenerate the agent-context corpus with `node scripts/bundle-agent-context.mjs`, citing `agent-context/bundle.test.ts` as the assertion. Wrong on three counts: that path does not exist from the repo root, it is a different script that does not refresh the corpus, and the cited test is a ten-line existence check that cannot detect stale prose. The gate that pins prose is `agent-context-conformance.test.ts`, regenerated by `bun scripts/regen-agent-context-conformance.ts` from the sdk package. Plan text only; no behaviour change. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md index 83cfa6760..069b16206 100644 --- a/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md +++ b/docs/superpowers/plans/2026-09-11-fr-023-phase-1a.md @@ -1017,7 +1017,7 @@ cd server/typescript/packages/metadata && bun test test/declared-top-level-keys. cd server/typescript/packages/cli && bun test test/verify-overlay-lint.test.ts test/help-lists-every-flag.test.ts test/verify-requirements-e2e.test.ts bun run --filter '@metaobjectsdev/metadata' typecheck && bun run --filter '@metaobjectsdev/cli' typecheck ``` -Expected: PASS (`loader-overlay-partition` or whichever existing test covers `_partitionOverlayLast` must stay green — **corrected 2026-09-11 by pre-flight sweep** — that discovery command finds NOTHING: no test in `metadata/test` names the private partition walk at all. `overlay.test.ts` is a 6-line PLACEHOLDER with zero tests (and its own comment points at a `loader.test.ts` that does not exist), so naming it would have added a vacuously-passing file to this run list. The loader's overlay-merge path is really exercised by `fr5c-merge-attribution.test.ts` (contributor tracking through the merge phase) and `round-trip.test.ts` (multi-file order: common -> vehicle -> overlay); run those. **Be aware NO existing test pins the ordering guarantee itself** — this task generalizes that walk, so a regression in it would be SILENT. Consider adding an order-independence assertion (load an overlay-only source BEFORE its base, assert the merge still happens)). +Expected: PASS (`loader-overlay-partition` or whichever existing test covers `_partitionOverlayLast` must stay green — **corrected 2026-09-11 by pre-flight sweep** — that discovery command finds NOTHING: no test in `metadata/test` names the private partition walk at all. `overlay.test.ts` is a 6-line PLACEHOLDER with zero tests (and its own comment points at a `loader.test.ts` that does not exist), so naming it would have added a vacuously-passing file to this run list. The loader's overlay-merge path is really exercised by `fr5c-merge-attribution.test.ts` (contributor tracking through the merge phase) and `round-trip.test.ts` (multi-file order: common -> vehicle -> overlay); run those. **CORRECTED 2026-09-12: an existing test DOES pin it** — `server/typescript/packages/sdk/test/order-independence.test.ts:126-180` permutes all six orderings of three real files (overlay-before-base included) through the real loader, and its comment states it catches `_partitionOverlayLast` being disabled (3 of 6 permutations throw `ERR_OVERLAY_NO_TARGET`). The earlier sweep missed it by scoping the search to `metadata/test` when the test lives in `sdk/test` — this task generalizes that walk, so a regression in it would be SILENT. Consider adding an order-independence assertion (load an overlay-only source BEFORE its base, assert the merge still happens)). - [ ] **Step 3: Commit** ``` @@ -1088,7 +1088,7 @@ feat(cli): meta init tracks .metaobjects/deps and deps.lock.json and no longer s - Modify: `spec/roadmap.md` (FR-023 status cell → Phase 1a shipped per DESIGN §11; the "doc-first quick wins" bullet closes; the §299 description row is updated to the B+ shape) - Modify: `agent-context/skills/metaobjects-authoring/SKILL.md` (the `overlay` section: extending and overlaying a dependency's nodes is the expected way to build on a shared model; `overlay: true` on every such amendment; what fails when upstream changes; a local node in the dependency's package is the consumer's own), `metaobjects-codegen/SKILL.md` (`sharedModelFile()`; the default-exclusion rule and `scope.include`), `metaobjects-verify/SKILL.md` (`verify --deps`, the stale-snapshot error, the overlay lint) - Modify: `CHANGELOG.md` `[Unreleased]` — Added (`dependencies`, `meta deps sync|check|list`, `verify --deps`, `sharedModelFile()`, `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, the overlay lint, `declaredTopLevelKeys`, `serializeSharedDocument`, `FileSource` `id`); Changed (`Collection` members; `inScope` / `inMigrateScope` compose the default-exclusion rule — byte-identical for a project without dependencies; scope grammar moved to `metadata`; `scopeExpectedSchema` third argument; `scanRequirements` option); Deprecated (`package.meta.json` scaffold; sdk `package.ts` / `workspace.ts` exports — removal at 2.0). `metamodelVersion` stays `1.0` — say so. -- Regenerate the agent-context corpus in the SAME commit (`node scripts/bundle-agent-context.mjs` or the documented command — the sdk `agent-context/bundle.test.ts` asserts it). +- Regenerate the agent-context corpus in the SAME commit. **CORRECTED 2026-09-12:** the gate that pins skill PROSE is `sdk/test/agent-context-conformance.test.ts`, which byte-compares the per-stack `expected/` trees under `fixtures/agent-context-conformance/` (five stacks), and it is regenerated by `cd server/typescript/packages/sdk && bun scripts/regen-agent-context-conformance.ts`. The command previously named here was wrong twice: `scripts/bundle-agent-context.mjs` does not exist from the repo root (the real path is `server/typescript/packages/sdk/scripts/bundle-agent-context.mjs`), and that is a DIFFERENT script — running it does not refresh the corpus. `test/agent-context/bundle.test.ts`, cited as the assertion, is a 10-line EXISTENCE check that cannot detect stale prose; it fails only when the gitignored bundle was never built. The Step 2 filter `bun test test/agent-context` DOES reach the real gate (it matches the `agent-context/` dir plus `agent-context-conformance.test.ts` and `agent-context-capability-grounding.test.ts`), so a stale corpus shows red — previously with the wrong remedy named beside it. **Read first:** DESIGN §5.3, §11. `docs/features/metadata-sources.md` §"Vendoring" and §"retired". The skills' existing `overlay` sections (grep `overlay`). Public-repo hygiene: no private names, no home paths. From e449517f5bec690524c7a6d1b0186dd98c5f9958 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 02:38:14 -0400 Subject: [PATCH 48/62] =?UTF-8?q?feat(python):=20dependencies=20resolve,?= =?UTF-8?q?=20load=20first=20and=20are=20excluded=20from=20gen=20unless=20?= =?UTF-8?q?scope.include=20names=20their=20package=20(FR-023=20=C2=A711)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- server/python/src/metaobjects/cli.py | 236 ++++++++++++--- .../python/src/metaobjects/codegen/runner.py | 27 +- .../src/metaobjects/config/dependencies.py | 272 +++++++++++++++++- .../src/metaobjects/config/neutral_config.py | 72 ++++- .../src/metaobjects/config/source_resolver.py | 121 +++++++- .../tests/codegen/test_cli_dependencies.py | 150 ++++++++++ .../python/tests/config/test_dependencies.py | 221 ++++++++++++++ .../test_dependency_conformance.py | 178 ++++++++++++ .../test_source_resolution_conformance.py | 14 +- 9 files changed, 1226 insertions(+), 65 deletions(-) create mode 100644 server/python/tests/codegen/test_cli_dependencies.py create mode 100644 server/python/tests/conformance/test_dependency_conformance.py diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index 25722fa32..7155b9b5e 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -49,9 +49,14 @@ import sys import tempfile from pathlib import Path +from typing import Callable from metaobjects import MetaDataLoader from metaobjects.errors import ParseError +from metaobjects.config.dependencies import Collection, imported_from, refuse_unowned_packages +from metaobjects.loader.meta_data_loader import LoadResult +from metaobjects.loader.sources import FileSource +from metaobjects.meta.core.object.meta_object import MetaObject from metaobjects.agent_context import ( AGENT_CONTEXT_MANIFEST_PATH, agent_context_staleness, @@ -93,14 +98,14 @@ _resolve_payload_vo, ) from metaobjects.meta.template import template_constants as tc -from metaobjects.naming import COLUMN_NAMING_STRATEGIES, DEFAULT_COLUMN_NAMING +from metaobjects.naming import COLUMN_NAMING_STRATEGIES, DEFAULT_COLUMN_NAMING, package_of_resolution_key from metaobjects.render.filesystem_provider import FilesystemProvider from metaobjects.render.verify import ( ERR_REQUIRED_SLOT_UNUSED, PayloadField, verify as render_verify, ) -from metaobjects.shared.base_types import TYPE_TEMPLATE +from metaobjects.shared.base_types import TYPE_OBJECT, TYPE_TEMPLATE from metaobjects.shared.separators import PACKAGE_SEP @@ -296,33 +301,64 @@ def _load_root( return result.root, [] -def _load_root_from_paths( - paths: list[str], +def _load_collection_result( + collection: Collection, strict: bool = False, providers: list[object] | None = None, libraries: list[str] | None = None, -) -> tuple[MetaData | None, list[str]]: - """Load metadata from an explicit file list rather than a single directory. - - The source-resolution ladder's ``.metaobjects/config.json`` rung - (:func:`resolve_metadata_location`) can resolve to several directories or - individual files, which a single ``from_directory`` call cannot express — - so this loads each resolved file as its own ``file://`` source via - :meth:`MetaDataLoader.from_uris`. Mirrors :func:`_load_root`'s ``strict``/ - ``providers``/``libraries`` contract exactly. +) -> LoadResult: + """Load a resolved `Collection`'s files — its dependency artifacts leading, + each under its `dep:/` source id (FR-023) — then run the + post-load ownership refusal (`ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, DESIGN + §11.5). Mirrors the TS `loadMemory` (`sdk/src/memory.ts`). + + Lower-level than :func:`_load_root_from_collection`: this returns the raw + `LoadResult` (structured `MetaError`s, a code + provenance on each) rather + than the CLI's flattened `(root, message-strings)` contract, so a caller + that needs the CODE or the offending FILE — the dependency corpus runner — + can read them. ``refuse_unowned_packages`` runs AFTER the loader's own + errors, never before: an unflagged overlay whose target the upstream + removed fails first, with its own coded error. """ - uris = [Path(p).resolve().as_uri() for p in paths] + lib_sources: list[object] = [] + if libraries: + from metaobjects.library import library_sources + + lib_sources = library_sources(libraries) + + sources = [FileSource(p, id=collection.file_ids.get(p)) for p in collection.files] + if providers: from metaobjects.core_types import core_providers - result = MetaDataLoader.from_uris( - uris, - providers=[*core_providers, *providers], - strict=strict, - libraries=libraries, - ) + loader = MetaDataLoader(providers=[*core_providers, *providers], strict=strict) else: - result = MetaDataLoader.from_uris(uris, strict=strict, libraries=libraries) + loader = MetaDataLoader(strict=strict) + + result = loader.load([*lib_sources, *sources]) + if not result.errors: + refuse_unowned_packages(result.root, collection.imported_packages, collection.imported_nodes) + return result + + +def _load_root_from_collection( + collection: Collection, + strict: bool = False, + providers: list[object] | None = None, + libraries: list[str] | None = None, +) -> tuple[MetaData | None, list[str]]: + """Load a resolved `Collection` — the source-resolution ladder's + ``.metaobjects/config.json`` rung (:func:`resolve_metadata_location`), which + can resolve to several directories, individual files, AND a dependency's + snapshot artifact, none of which a single ``from_directory`` call can + express. Mirrors :func:`_load_root`'s ``strict``/``providers``/``libraries`` + contract exactly (CLI-facing: flattened error messages, never a raised + exception for a plain loader error). + """ + try: + result = _load_collection_result(collection, strict=strict, providers=providers, libraries=libraries) + except ParseError as exc: + return None, [f"{exc.code}: {exc}"] if result.errors: msgs = [f"{e.code}: {e.message}" for e in result.errors] return None, msgs @@ -369,6 +405,51 @@ def _parse_entities(value: str | None) -> list[str] | None: return names or None +def _refuse_imported_entities( + entity_names: list[str] | None, + root: MetaData, + collection: Collection, +) -> str | None: + """FR-023 §11.1 item 2 — the error message for a target's ``entities:`` (or + ``--entities``) naming ONLY objects imported from a dependency and excluded + from output (imported metadata is load-only by default; only the project's + own `scope.include` naming the package literally opts it in). Returns + `None` for every other case, INCLUDING a name matching nothing at all — + that stays the runner's own "no entities matched" warning, unchanged. + Mirrors the TS `refuseImportedPositional` (`cli/src/commands/gen.ts`). + + A named entity is a bare NAME, not a fully-qualified one, so two packages + declaring the same short name are both candidates; the refusal fires only + when NONE of them would generate. + """ + if not entity_names: + return None + # ADR-0039 sanctioned own: top-level object scan on the loader ROOT + # (metadata.root is never extended, so own == effective) — mirrors + # `codegen.runner._objects`. + all_objects = [ + c for c in root.own_children() if c.type == TYPE_OBJECT and isinstance(c, MetaObject) + ] + for name in entity_names: + matches = [o for o in all_objects if o.name == name] + if not matches: + continue # nothing loaded by this name — the runner's own warning covers it + all_excluded = all( + collection.imported(o.resolution_key()) and not collection.in_scope(o.resolution_key()) + for o in matches + ) + if not all_excluded: + continue + fqn = matches[0].resolution_key() + pkg = package_of_resolution_key(fqn) + dep_name = imported_from(pkg, collection.dependencies) or pkg + return ( + f"error: '{name}' ({fqn}) is imported from dependency '{dep_name}' and is not " + "generated here — add its package to scope.include in .metaobjects/config.json" + ) + return None + + def _run_suite( root: MetaData, out_dir: str, @@ -380,6 +461,7 @@ def _run_suite( project_root: str | None = None, baseline: str = "default", refused_out: list[str] | None = None, + select: Callable[[str], bool] | None = None, ) -> list[str]: """Run a generator suite against an ALREADY-LOADED ``root`` into ``out_dir``. @@ -388,6 +470,12 @@ def _run_suite( the ``--template-spec`` pass). See :func:`_generate` for the parameter semantics. Returns the written paths. + ``select`` (FR-023 §11.1 item 2) — normally a resolved `Collection`'s + `in_scope`: the output filter that excludes an imported dependency's + objects from codegen unless the project's own `scope.include` names their + package. `None` (the default) admits everything, byte-identical to a + project with no dependencies. + NOTE: ``run_gen``'s output-path collision guard is per-pass — a ``--template-spec`` ``outputPattern`` that collides with a default-suite path is not flagged across the two passes (it would overwrite, since default files @@ -420,7 +508,7 @@ def _run_suite( baseline=baseline, ) suite = generators if generators is not None else _default_generators() - result = run_gen(config, root, generators=suite, entity_filter=entity_filter) + result = run_gen(config, root, generators=suite, entity_filter=entity_filter, select=select) for warning in result.warnings: print(f"warning: {warning}") # A refusal is a FAILED gate, and the return value carries only what was WRITTEN — so @@ -557,8 +645,11 @@ def template_spec_generators( def resolve_metadata_location( config: ProjectConfig | None, root: Path, -) -> list[str]: +) -> Collection: """The precedence ladder for where metadata lives, rungs 2-4. First match wins. + Returns the full FR-023-aware `Collection` at EVERY rung — `dependencies` (and + `scope`/`migrate.scope`) are read from `root`'s neutral `.metaobjects/config.json` + regardless of which rung supplied the OWN files (DESIGN §2.3). 2. This port's native surface — ``metadata`` in ``metaobjects.config.yaml``. 3. ``sources`` in the port-neutral ``.metaobjects/config.json``. @@ -583,23 +674,27 @@ def resolve_metadata_location( through to the next rung. See `docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §5. """ - from metaobjects.config.source_resolver import resolve_collection, resolve_sources + from metaobjects.config.source_resolver import ( + build_collection, + resolve_collection_full, + resolve_sources, + ) if config is not None: # `config.metadata_dir()` is already resolved to an absolute path # (`ProjectConfig._resolve_under`), so the base passed here is - # likewise irrelevant. - return [ - str(p) for p in resolve_sources(root, [{"path": config.metadata_dir()}]) - ] + # likewise irrelevant to resolving OWN files — but `root` is still + # where `.metaobjects/config.json` (dependencies/scope) is read from. + own_files = resolve_sources(root, [{"path": config.metadata_dir()}]) + return build_collection(root, own_files) - # Rungs 3 and 4 both live in `resolve_collection`. - return [str(p) for p in resolve_collection(root)] + # Rungs 3 and 4 both live in `resolve_collection_full`. + return resolve_collection_full(root) def _resolve_metadata_location_or_print_error( config: ProjectConfig | None, root: Path -) -> list[str] | None: +) -> Collection | None: """``resolve_metadata_location`` for the no-explicit-``metadata_dir`` CLI paths (``docs``, and the ``gen``/``verify --codegen`` neutral fallbacks), translating a raised ``ParseError`` into this CLI's print-and-return-1 @@ -657,8 +752,8 @@ def _cmd_docs(args: argparse.Namespace) -> int: except ConfigError as exc: print(f"error: {exc}", file=sys.stderr) return 1 - paths = _resolve_metadata_location_or_print_error(config, root_dir) - if paths is None: + collection = _resolve_metadata_location_or_print_error(config, root_dir) + if collection is None: return 1 docs_libraries: list[str] | None = None if config is not None: @@ -673,8 +768,8 @@ def _cmd_docs(args: argparse.Namespace) -> int: return 1 providers = [*providers, *config_providers] docs_libraries = config.libraries - root, errors = _load_root_from_paths( - paths, providers=providers, libraries=docs_libraries + root, errors = _load_root_from_collection( + collection, providers=providers, libraries=docs_libraries ) project_default = root_dir.name else: @@ -873,6 +968,7 @@ def _run_gen_targets( project_root: str | None = None, baseline: str = "default", refused_out: list[str] | None = None, + select: Callable[[str], bool] | None = None, ) -> tuple[list[str], list[str]]: """Run each target's suite into its ``outDir``. Returns (all_written, errors). @@ -887,6 +983,9 @@ def _run_gen_targets( path across targets and record an error when two targets emit the same one. (Detection is post-write — the colliding file may already be on disk — but the command still fails, so a misconfigured gate is caught in CI.) + + ``select`` (FR-023 §11.1 item 2) — see :func:`_run_suite`; the SAME predicate + applies to every target, since a `Collection`'s scope is project-wide. """ all_written: list[str] = [] seen: dict[str, str] = {} # full path -> target name @@ -903,6 +1002,7 @@ def _run_gen_targets( written = _run_suite( root, out_dir, gens, t.entities, gen_state_dir=gen_state_dir, project_root=project_root, baseline=baseline, refused_out=refused_out, + select=select, ) except ValueError as exc: # intra-target run_gen collision → clean error errors.append(f"target '{t.name}': {exc}") @@ -942,8 +1042,8 @@ def _cmd_gen_neutral_fallback(args: argparse.Namespace) -> int: return 2 root_dir = Path.cwd() - paths = _resolve_metadata_location_or_print_error(None, root_dir) - if paths is None: + collection = _resolve_metadata_location_or_print_error(None, root_dir) + if collection is None: return 1 generators: list[Generator] | None = None @@ -960,19 +1060,25 @@ def _cmd_gen_neutral_fallback(args: argparse.Namespace) -> int: if not providers_ok: return 1 - root, load_errors = _load_root_from_paths(paths, providers=providers) + root, load_errors = _load_root_from_collection(collection, providers=providers) if root is None: print("error: failed to load metadata:", file=sys.stderr) for msg in load_errors: print(f" {msg}", file=sys.stderr) return 1 + imported_refusal = _refuse_imported_entities(entities, root, collection) + if imported_refusal is not None: + print(imported_refusal, file=sys.stderr) + return 2 + gen_state = str(root_dir.resolve() / ".metaobjects" / ".gen-state") column_naming = getattr(args, "column_naming", None) or DEFAULT_COLUMN_NAMING refused: list[str] = [] written = _run_suite( root, args.out, generators, entities, gen_state_dir=gen_state, column_naming=column_naming, + select=collection.in_scope, project_root=str(root_dir.resolve()), baseline=getattr(args, "baseline", None) or "default", refused_out=refused, @@ -1038,8 +1144,17 @@ def _cmd_gen_config(args: argparse.Namespace) -> int: if not providers_ok: return 1 - root, load_errors = _load_root( - config.metadata_dir(), providers=providers, libraries=config.libraries + # FR-023: rung 2 (this native `metaobjects.config.yaml` surface) still reads + # `dependencies`/`scope`/`migrate.scope` from the project's neutral + # `.metaobjects/config.json` (DESIGN §2.3 — dependencies apply at EVERY rung). + collection = _resolve_metadata_location_or_print_error( + config, project_root_for(config.metadata_dir()) + ) + if collection is None: + return 1 + + root, load_errors = _load_root_from_collection( + collection, providers=providers, libraries=config.libraries ) if root is None: print("error: failed to load metadata:", file=sys.stderr) @@ -1047,6 +1162,12 @@ def _cmd_gen_config(args: argparse.Namespace) -> int: print(f" {msg}", file=sys.stderr) return 1 + for t in targets: + imported_refusal = _refuse_imported_entities(t.entities, root, collection) + if imported_refusal is not None: + print(imported_refusal, file=sys.stderr) + return 2 + refused: list[str] = [] written, errors = _run_gen_targets( config, targets, root, @@ -1054,6 +1175,7 @@ def _cmd_gen_config(args: argparse.Namespace) -> int: project_root=str(project_root_for(config.metadata_dir())), baseline=getattr(args, "baseline", None) or "default", refused_out=refused, + select=collection.in_scope, ) if errors: for msg in errors: @@ -1323,8 +1445,8 @@ def _verify_codegen_neutral_fallback(args: argparse.Namespace) -> int: return 2 root_dir = Path.cwd() - paths = _resolve_metadata_location_or_print_error(None, root_dir) - if paths is None: + collection = _resolve_metadata_location_or_print_error(None, root_dir) + if collection is None: return 1 strict = not getattr(args, "lax", False) @@ -1338,7 +1460,7 @@ def _verify_codegen_neutral_fallback(args: argparse.Namespace) -> int: with tempfile.TemporaryDirectory() as tmp: entities = _parse_entities(getattr(args, "entities", None)) - root, load_errors = _load_root_from_paths(paths, strict=strict, providers=providers) + root, load_errors = _load_root_from_collection(collection, strict=strict, providers=providers) if root is None: print("error: failed to load metadata:", file=sys.stderr) for msg in load_errors: @@ -1347,8 +1469,14 @@ def _verify_codegen_neutral_fallback(args: argparse.Namespace) -> int: print(_strict_load_hint(), file=sys.stderr) return 1 + imported_refusal = _refuse_imported_entities(entities, root, collection) + if imported_refusal is not None: + print(imported_refusal, file=sys.stderr) + return 2 + _run_suite( - root, tmp, None, entities, gen_state_dir=None, column_naming=column_naming + root, tmp, None, entities, gen_state_dir=None, column_naming=column_naming, + select=collection.in_scope, ) expected = _relative_set(Path(tmp)) committed = _relative_set(Path(args.out)) @@ -1429,8 +1557,16 @@ def _verify_codegen_config(args: argparse.Namespace) -> int: if not providers_ok: return 1 - root, load_errors = _load_root( - config.metadata_dir(), strict=strict, providers=providers, libraries=config.libraries + # FR-023: see the matching comment in `_cmd_gen_config` — dependencies apply + # at this rung too. + collection = _resolve_metadata_location_or_print_error( + config, project_root_for(config.metadata_dir()) + ) + if collection is None: + return 1 + + root, load_errors = _load_root_from_collection( + collection, strict=strict, providers=providers, libraries=config.libraries ) if root is None: print("error: failed to load metadata:", file=sys.stderr) @@ -1440,6 +1576,12 @@ def _verify_codegen_config(args: argparse.Namespace) -> int: print(_strict_load_hint(), file=sys.stderr) return 1 + for t in closure: + imported_refusal = _refuse_imported_entities(t.entities, root, collection) + if imported_refusal is not None: + print(imported_refusal, file=sys.stderr) + return 2 + with tempfile.TemporaryDirectory() as temp_root: # Map each unique real outDir to ONE temp slot (shared real dir => shared slot). temp_for: dict[str, str] = {} @@ -1456,7 +1598,7 @@ def _verify_codegen_config(args: argparse.Namespace) -> int: # gen_state_dir stays None: this regenerates into a temp tree purely to # diff, so recording a manifest would mutate the user's project from a # read-only drift check, keyed to a directory deleted seconds later. - _written, errors = _run_gen_targets(config, remapped, root) + _written, errors = _run_gen_targets(config, remapped, root, select=collection.in_scope) if errors: for msg in errors: print(f"error: {msg}", file=sys.stderr) diff --git a/server/python/src/metaobjects/codegen/runner.py b/server/python/src/metaobjects/codegen/runner.py index 49bf3bbbd..9ab1de352 100644 --- a/server/python/src/metaobjects/codegen/runner.py +++ b/server/python/src/metaobjects/codegen/runner.py @@ -60,8 +60,22 @@ def run_gen( *, generators: list[Generator], entity_filter: list[str] | None = None, + select: Callable[[str], bool] | None = None, merge_strategy: str = "overwrite", ) -> RunGenResult: + """Mirrors ``codegen-ts``'s ``runner.ts``. + + ``select`` (FR-023 §11.1 item 2) — applied AFTER ``entity_filter``, over each + object's ``resolution_key()`` (never the bare name — two packages may declare + the same short name). Normally a resolved ``Collection``'s ``in_scope``: the + output filter that excludes an imported dependency's objects from codegen + unless the project's own ``scope.include`` names their package. ``None`` (the + default) admits everything, byte-identical to a project with no dependencies. + A generator that renders once over the WHOLE selected set (the shared-enums + module, so far the only one) reads it off ``ctx.entities`` like any other + generator — no separate wiring needed, since that list is already the + post-``select`` set (parity with the TS ``codegen-ts`` runner's ``select``). + """ result = RunGenResult() if not isinstance(metadata, MetaRoot): raise ValueError("run_gen: metadata must be a loaded MetaRoot.") @@ -69,13 +83,16 @@ def run_gen( objs = _objects(metadata) if entity_filter is not None: objs = [o for o in objs if o.name in entity_filter] + if select is not None: + objs = [o for o in objs if select(o.resolution_key())] if not objs: - reason = ( - "no object children match the provided entity_filter" - if entity_filter is not None - else "root has no object children" - ) + if entity_filter is not None: + reason = "no object children match the provided entity_filter" + elif select is not None: + reason = "no object children survived the collection's scope" + else: + reason = "root has no object children" result.warnings.append(f"No entities to generate — {reason}.") return result diff --git a/server/python/src/metaobjects/config/dependencies.py b/server/python/src/metaobjects/config/dependencies.py index 1abac781a..d9fa100ed 100644 --- a/server/python/src/metaobjects/config/dependencies.py +++ b/server/python/src/metaobjects/config/dependencies.py @@ -12,9 +12,13 @@ import re from dataclasses import dataclass from pathlib import Path -from typing import Any +from typing import Any, Callable, Sequence from metaobjects.errors import ErrorCode, ParseError +from metaobjects.meta.meta_data import MetaData +from metaobjects.naming import package_of_resolution_key +from metaobjects.registry_manifest import METAMODEL_VERSION +from metaobjects.shared.base_types import TYPE_OBJECT #: Directory (under `.metaobjects/`) holding the synced snapshot artifacts, #: one subdirectory per dependency name: `.metaobjects/deps//`. @@ -311,3 +315,269 @@ class ResolvedDependency: nodes: tuple[str, ...] artifact_path: str source_id: str + + +def _metamodel_major(version: str) -> str: + """The MAJOR half of a `major.minor` metamodel version. The metadata + contract is promised on the major alone (ADR-0035 Amendment 2).""" + return version.split(".")[0] + + +def _stale(detail: str) -> ParseError: + """Every stale-snapshot refusal, in one place so every one of them ends + with the command that fixes it. Returns rather than raises (unlike the TS + `never`-typed sibling) — Python has no control-flow narrowing on `raise + fn()`, so the call site still writes `raise _stale(...)`.""" + return ParseError(f"{detail}; run `meta deps sync`", code=ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE) + + +def verify_snapshot( + config_dir: Path, + specs: list[dict[str, Any]], + lock: dict[str, Any] | None, +) -> list[ResolvedDependency]: + """Verify a project's committed snapshot against its lock, and resolve the + dependencies the collection will load FIRST (DESIGN §4.2 step 2-3). Mirrors + the TS `verifySnapshot` (`sdk/src/dependencies.ts`) exactly. + + ``lock`` is the ALREADY-VALIDATED dict :func:`read_lock` / :func:`validate_lock` + return (``dependencies`` a plain dict keyed by name), never raw JSON. + + Two failures are NOT staleness and get their own codes: a dependency published + against a different metamodel MAJOR (``ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE``) + and two dependencies exporting the same fully-qualified node + (``ERR_DEPENDENCY_NODE_COLLISION``). + + Result order is dependency NAME order, never the config's declaration order — + the artifacts lead the loaded file list, so declaration order must not decide + what the loader sees first. + + A project with no dependencies and no lock resolves to ``[]`` without touching + the filesystem — the byte-identical path. + """ + declared = [d["name"] for d in specs] + + if lock is None: + # No dependencies AND no lock is the untouched project, not a stale one. + if not declared: + return [] + raise _stale( + f"{len(declared)} dependenc{'y is' if len(declared) == 1 else 'ies are'} declared " + f"({', '.join(declared)}) but there is no {_METAOBJECTS_DIR}/{LOCK_FILE}" + ) + + entries: dict[str, Any] = lock["dependencies"] + declared_names = set(declared) + for name in entries: + if name not in declared_names: + raise _stale( + f'{_METAOBJECTS_DIR}/{LOCK_FILE} locks dependency "{name}", which ' + f"{_METAOBJECTS_DIR}/config.json no longer declares" + ) + + resolved: list[ResolvedDependency] = [] + for name in sorted(declared_names): + entry = entries.get(name) + if entry is None: + raise _stale( + f'dependency "{name}" is declared but {_METAOBJECTS_DIR}/{LOCK_FILE} has no ' + "entry for it" + ) + + artifact_path = config_dir / _METAOBJECTS_DIR / DEPS_DIR / name / entry["artifact"] + try: + data = artifact_path.read_bytes() + except OSError: + raise _stale( + f'the committed snapshot for "{name}" is missing (expected {artifact_path})' + ) from None + + actual = sha256_integrity(data) + if actual != entry["integrity"]: + raise _stale( + f'the committed snapshot for "{name}" does not match the lock — {artifact_path} ' + f'hashes to {actual}, the lock records {entry["integrity"]}' + ) + + if _metamodel_major(entry["metamodelVersion"]) != _metamodel_major(METAMODEL_VERSION): + raise ParseError( + f'dependency "{name}" was published against metamodel {entry["metamodelVersion"]}; ' + f"this toolchain speaks {METAMODEL_VERSION}. A different metamodel MAJOR is a " + f'different metadata contract — upgrade the toolchain, or use a release of "{name}" ' + "built against it.", + code=ErrorCode.ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE, + ) + + resolved.append( + ResolvedDependency( + name=name, + version=entry["version"], + packages=tuple(entry["packages"]), + nodes=tuple(entry["nodes"]), + artifact_path=str(artifact_path), + source_id=dependency_source_id(name, entry["artifact"]), + ) + ) + + # Collision is checked across the WHOLE resolved set, so the error names the + # two dependencies in name order however the config declared them. + owner: dict[str, str] = {} + for dep in resolved: + for node in dep.nodes: + prior = owner.get(node) + if prior is not None: + raise ParseError( + f'dependencies "{prior}" and "{dep.name}" both export "{node}" — one ' + "fully-qualified node cannot come from two places, and whichever loaded " + "second would silently win", + code=ErrorCode.ERR_DEPENDENCY_NODE_COLLISION, + ) + owner[node] = dep.name + + return resolved + + +def explicitly_includes(patterns: Sequence[str] | None, pkg: str) -> bool: + """Does some pattern in `patterns` name `pkg` LITERALLY (DESIGN §11.1 item 2)? + + Drop the pattern's final segment — which names the node — and what remains + must be wildcard-free and equal to `pkg`. So `acme::common::**` and + `acme::common::Address` both name `acme::common`; `acme::**` and `**` reach + its nodes but name nothing, and an absent or empty list names nothing. + + Matching is therefore NOT `matches_scope` — a pattern that MATCHES a + package's nodes is a weaker statement than one that NAMES the package. + Mirrors the TS `explicitlyIncludes` (`sdk/src/dependencies.ts`) exactly. + """ + if not patterns or pkg == "": + return False + for pattern in patterns: + named = package_of_resolution_key(pattern) + if named and "*" not in named and named == pkg: + return True + return False + + +def imported_from(pkg: str, dependencies: Sequence[ResolvedDependency]) -> str | None: + """The name of the dependency that owns package `pkg`, or `None` when no + resolved dependency exports it. Used by the CLI's `meta gen `-style + refusal message (FR-023 §11.1 item 2) to name the offending dependency.""" + for dep in dependencies: + if pkg in dep.packages: + return dep.name + return None + + +def refuse_unowned_packages( + root: MetaData, + imported_packages: frozenset[str] | None, + imported_nodes: frozenset[str] | None, +) -> None: + """FR-023 §11.5 — a consumer may not declare a NEW top-level node into a + package one of its dependencies owns. + + This is what keeps the package-keyed exclusion rule from failing silently. + Imported-ness is decided by PACKAGE, so such a node would be excluded from + this project's own codegen, migrate and ledger — producing no output and no + error. An overlay of the dependency's own node is untouched: its resolution + key is in `imported_nodes`, because the node it merged into came from the + artifact. Mirrors the TS `refuseUnownedPackages` (`sdk/src/memory.ts`) — + called AFTER the loader's own errors, never before (an unflagged overlay + whose target the upstream removed fails first, with its own coded error). + + No-op when nothing is imported, which is every project that declares no + dependencies. + """ + if not imported_packages: + return + nodes = imported_nodes or frozenset() + + # ADR-0039 SANCTIONED own-accessor case: a root-level scan. `MetaRoot` has + # no super, so own and effective children are the same set here, and the + # question asked is precisely "what did this tree declare at the top + # level", which is the own layer by definition. + for node in root.own_children(): + if node.type != TYPE_OBJECT: + continue + key = node.resolution_key() + pkg = package_of_resolution_key(key) + if pkg not in imported_packages or key in nodes: + continue + raise ParseError( + f'"{key}" is declared here, but the package "{pkg}" belongs to a metadata ' + f'dependency this project imports, and "{key}" is not one of the nodes that ' + "dependency exports. Declare it in a package this project owns and 'extends' " + "the dependency's node if it needs its shape; if it was meant to AMEND the " + "dependency's node, give it that node's name and 'overlay: true'; if this " + f'project really does own "{pkg}", name it in \'scope.include\' and stop ' + "importing it.", + code=ErrorCode.ERR_DEPENDENCY_PACKAGE_NOT_OWNED, + ) + + +@dataclass(frozen=True) +class Collection: + """Everything the FR-023-aware source ladder resolved for one project: + its own metadata files, its dependencies' snapshot artifacts (leading the + file list — see `files`), and the predicates every action surface reads to + decide whether an imported object should be excluded (DESIGN §11.1 item 2). + Mirrors the TS `Collection` interface (`sdk/src/collection.ts`), the + Python-relevant subset. + """ + + #: Canonically-ordered absolute file paths: dependency artifacts (in + #: dependency-NAME order) followed by this project's own files, in + #: `resolve_sources`'s canonical (content) order. Identical to `own_files` + #: when nothing is imported. + files: tuple[Path, ...] + + #: `files` minus the dependency artifacts — this project's own metadata. + own_files: tuple[Path, ...] + + #: The `FileSource` id each dependency artifact loads under + #: (`dep:/`), keyed by its path in `files`. Own files are + #: absent from this map and keep the default `basename(path)`. + file_ids: dict[Path, str] + + #: The resolved dependencies, in dependency-NAME order. Empty for a + #: project that declares none. + dependencies: tuple[ResolvedDependency, ...] + + #: THE exclusion key (DESIGN §11.5): the union of every dependency's + #: `packages`. A `frozenset`, not a sorted sequence — nothing here reads it + #: in order. + imported_packages: frozenset[str] + + #: The union of every dependency's `nodes` — read only by + #: `refuse_unowned_packages`, to tell an overlay of an imported node from a + #: genuinely new local declaration in the dependency's package. NOT the + #: exclusion key. + imported_nodes: frozenset[str] + + #: The user's declared `scope.include` patterns, for `in_scope`'s + #: explicit-include rule. + scope_include: tuple[str, ...] + + #: `migrate.scope`-governed predicate, or `None` when the project declares + #: no `migrate.scope` AND resolves no dependencies (mirrors TS's + #: `inMigrateScope`; nothing in the Python CLI consumes this today — schema + #: is TS-owned, ADR-0015). + in_migrate_scope: Callable[[str], bool] | None + + def imported(self, fqn: str) -> bool: + """Is `fqn` in a package one of this project's dependencies owns?""" + return package_of_resolution_key(fqn) in self.imported_packages + + def in_scope(self, fqn: str) -> bool: + """Output filter for codegen (`run_gen`'s `select`) and `verify --codegen`. + + `(not imported(fqn)) or explicitly_includes(scope_include, packageOf(fqn))` + — UNLIKE TypeScript this does NOT also apply `matches_scope` to the + project's OWN objects: the Python CLI has never applied `scope` to its + own generated output (T18 ruling, 2026-09-11) — only the import- + exclusion rule is new behaviour here. + """ + pkg = package_of_resolution_key(fqn) + if pkg not in self.imported_packages: + return True + return explicitly_includes(self.scope_include, pkg) diff --git a/server/python/src/metaobjects/config/neutral_config.py b/server/python/src/metaobjects/config/neutral_config.py index 73d4deb54..70ab9ce7e 100644 --- a/server/python/src/metaobjects/config/neutral_config.py +++ b/server/python/src/metaobjects/config/neutral_config.py @@ -34,6 +34,24 @@ class NeutralConfig: #: `dir` only beside `npm`/`python`. dependencies: list[dict[str, str]] + #: FR-023 §11.1 item 2 — the user's declared `scope.include` patterns, READ + #: ONLY for the explicit-include rule (`explicitly_includes`): a dependency's + #: package survives selection only when one of these patterns NAMES it + #: literally. Unlike TypeScript, the Python CLI has never applied `scope` to + #: its OWN objects (no caller of `matches_scope`/`compile_scope` exists in + #: `server/python/src` before this) — `scope.exclude` is therefore not read + #: at all, and adding that would be new behaviour, not parity (T18 ruling). + #: Empty when `scope` / `scope.include` is absent. + scope_include: list[str] + + #: FR-023 §11.1 item 2 — the user's declared `migrate.scope` patterns + #: (include-only, same grammar as `scope.include`), or `None` when the key + #: is absent. Mirrors TS `migrate.scope` for completeness — nothing in the + #: Python CLI's own `gen`/`verify --codegen` path consumes it (schema is + #: TS-owned, ADR-0015); it exists so `Collection.in_migrate_scope` can be + #: built with full TS parity. + migrate_scope: list[str] | None + def read_neutral_config(config_dir: Path) -> NeutralConfig | None: """Read the neutral subset from ``config_dir/.metaobjects/config.json``. @@ -110,8 +128,60 @@ def read_neutral_config(config_dir: Path) -> NeutralConfig | None: code=ErrorCode.ERR_COLLECTION_NOT_FOUND, ) + scope_include = _read_scope_include(raw, path) + migrate_scope = _read_migrate_scope(raw, path) + # Unknown top-level keys are IGNORED by design — see the module docstring. - return NeutralConfig(sources=[dict(s) for s in sources], dependencies=dependencies) + return NeutralConfig( + sources=[dict(s) for s in sources], + dependencies=dependencies, + scope_include=scope_include, + migrate_scope=migrate_scope, + ) + + +def _string_array_or_raise(value: object, field: str, path: Path) -> list[str]: + if not isinstance(value, list) or not all(isinstance(v, str) and v.strip() for v in value): + raise ParseError( + f"{path}: '{field}' must be an array of non-empty strings", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + return list(value) + + +def _read_scope_include(raw: dict[str, object], path: Path) -> list[str]: + """FR-023 §11.1 item 2 — `scope.include`, read only for the explicit-include + rule. `scope.exclude` and every other key inside `scope` are ignored, same + tolerance the module docstring already grants unknown TOP-level keys: this + port models the neutral subset it actually consumes, not the whole + TypeScript-owned `scope` vocabulary. + """ + scope = raw.get("scope") + if scope is None: + return [] + if not isinstance(scope, dict): + raise ParseError(f"{path}: 'scope' must be an object", code=ErrorCode.ERR_COLLECTION_NOT_FOUND) + include = scope.get("include") + if include is None: + return [] + return _string_array_or_raise(include, "scope.include", path) + + +def _read_migrate_scope(raw: dict[str, object], path: Path) -> list[str] | None: + """FR-023 §11.1 item 2 — `migrate.scope`, read for completeness + (`Collection.in_migrate_scope`). Every other key inside `migrate` + (`dialect`, `outDir`, ...) is TypeScript-owned and ignored — see + `test_unknown_top_level_keys_are_ignored`. + """ + migrate = raw.get("migrate") + if migrate is None: + return None + if not isinstance(migrate, dict): + raise ParseError(f"{path}: 'migrate' must be an object", code=ErrorCode.ERR_COLLECTION_NOT_FOUND) + scope = migrate.get("scope") + if scope is None: + return None + return _string_array_or_raise(scope, "migrate.scope", path) def _validate_dependency_spec(dep: object, path: Path) -> dict[str, str]: diff --git a/server/python/src/metaobjects/config/source_resolver.py b/server/python/src/metaobjects/config/source_resolver.py index fd7ea7d6b..46165d753 100644 --- a/server/python/src/metaobjects/config/source_resolver.py +++ b/server/python/src/metaobjects/config/source_resolver.py @@ -2,11 +2,21 @@ import os from pathlib import Path +from typing import Callable from metaobjects.errors import ErrorCode, ParseError from metaobjects.loader.sources import DirectorySource +from metaobjects.naming import package_of_resolution_key +from metaobjects.scope import compile_scope, matches_scope -from .neutral_config import DEFAULT_METADATA_DIR, read_neutral_config +from .dependencies import ( + Collection, + ResolvedDependency, + explicitly_includes, + read_lock, + verify_snapshot, +) +from .neutral_config import DEFAULT_METADATA_DIR, NeutralConfig, read_neutral_config def _list_metadata_files(directory: Path) -> list[Path]: @@ -101,11 +111,99 @@ def resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path] return list(seen) -def resolve_collection(root: Path) -> list[Path]: - """The full ladder: declared `sources`, else the default directory. +def _make_in_migrate_scope( + migrate_scope: list[str] | None, + imported_packages: frozenset[str], + has_dependencies: bool, +) -> Callable[[str], bool] | None: + """Build `Collection.in_migrate_scope` — mirrors the TS `inMigrateScope` + for completeness (T18 ruling: nothing in the Python CLI's own `gen`/ + `verify --codegen` path consumes this; schema is TS-owned, ADR-0015). - Only the DEFAULT may be absent — a declared source that does not resolve is - `ERR_SOURCE_UNRESOLVED`, a louder failure. + `None` iff the project declares no `migrate.scope` AND resolves no + dependencies — the byte-identical path a caller reads as "admits + everything" (`collection.in_migrate_scope(fqn) if ... else True`). + """ + if migrate_scope is None and not has_dependencies: + return None + compiled = compile_scope(include=migrate_scope) + + def predicate(fqn: str) -> bool: + if not matches_scope(fqn, compiled): + return False + pkg = package_of_resolution_key(fqn) + if pkg not in imported_packages: + return True + return explicitly_includes(migrate_scope, pkg) + + return predicate + + +def _collection_from_own_files( + root: Path, own_files: list[Path], cfg: NeutralConfig | None +) -> Collection: + """The tail shared by `resolve_collection_full` (own files via the source + ladder) and `build_collection` (own files from an external surface, e.g. + a native `metaobjects.config.yaml` `metadata:` key) — dependencies, scope + and migrate.scope come from `root`'s neutral `.metaobjects/config.json` + regardless of where `own_files` came from (DESIGN §2.3: dependencies are + read at EVERY rung of the source ladder). + """ + dependency_specs: list[dict[str, str]] = cfg.dependencies if cfg is not None else [] + scope_include: list[str] = cfg.scope_include if cfg is not None else [] + migrate_scope: list[str] | None = cfg.migrate_scope if cfg is not None else None + + lock = read_lock(root) + dependencies: list[ResolvedDependency] = ( + [] if not dependency_specs and lock is None else verify_snapshot(root, dependency_specs, lock) + ) + + imported_packages: frozenset[str] = frozenset( + pkg for dep in dependencies for pkg in dep.packages + ) + imported_nodes: frozenset[str] = frozenset(node for dep in dependencies for node in dep.nodes) + + # The artifacts LEAD the file list, in dependency-NAME order (verify_snapshot's + # own return order) — see `Collection.files`. + dep_paths = [Path(dep.artifact_path) for dep in dependencies] + file_ids: dict[Path, str] = {p: dep.source_id for p, dep in zip(dep_paths, dependencies)} + own_files_t = tuple(own_files) + + return Collection( + files=tuple(dep_paths) + own_files_t, + own_files=own_files_t, + file_ids=file_ids, + dependencies=tuple(dependencies), + imported_packages=imported_packages, + imported_nodes=imported_nodes, + scope_include=tuple(scope_include), + in_migrate_scope=_make_in_migrate_scope( + migrate_scope, imported_packages, has_dependencies=bool(dependencies) + ), + ) + + +def build_collection(root: Path, own_files: list[Path]) -> Collection: + """Build a full `Collection` for an EXTERNALLY-determined own-files set — + e.g. rung 2 of the CLI's source-resolution ladder (`metadata:` in a native + `metaobjects.config.yaml`) — while dependencies/scope/migrate.scope still + come from `root`'s neutral `.metaobjects/config.json` (DESIGN §2.3: + dependencies are read by every port at EVERY rung of the source ladder, + not just the declared-`sources`/default-directory rungs `resolve_collection_full` + covers on its own). + """ + root = root.resolve() + return _collection_from_own_files(root, own_files, read_neutral_config(root)) + + +def resolve_collection_full(root: Path) -> Collection: + """The full ladder, FR-023-aware: declared `sources` (else the default + directory) for this project's OWN files, plus its resolved dependencies + (DESIGN §4.2) leading the file list, plus the `scope`/`migrate.scope` + predicates every action surface reads (DESIGN §11.1 item 2). + + Only the DEFAULT source directory may be absent — a declared source that + does not resolve is `ERR_SOURCE_UNRESOLVED`, a louder failure. """ root = root.resolve() cfg = read_neutral_config(root) @@ -122,4 +220,15 @@ def resolve_collection(root: Path) -> list[Path]: ) specs = [{"path": DEFAULT_METADATA_DIR}] - return resolve_sources(root, specs) + own_files = resolve_sources(root, specs) + return _collection_from_own_files(root, own_files, cfg) + + +def resolve_collection(root: Path) -> list[Path]: + """The full ladder: declared `sources`, else the default directory. + + A thin projection of `resolve_collection_full` (T18 ruling) — this + project's OWN files only, never a dependency's snapshot artifact, so its + public shape (and every pre-FR-023 caller) is unchanged. + """ + return list(resolve_collection_full(root).own_files) diff --git a/server/python/tests/codegen/test_cli_dependencies.py b/server/python/tests/codegen/test_cli_dependencies.py new file mode 100644 index 000000000..fb30daeed --- /dev/null +++ b/server/python/tests/codegen/test_cli_dependencies.py @@ -0,0 +1,150 @@ +"""FR-023 Task 18 — the Python CLI's collection-aware `gen`/`verify --codegen`: +a dependency's objects are excluded from codegen selection by default +(DESIGN §11.1 item 2); `scope.include` naming the dependency's package opts it +in; a target's `entities:` (or `--entities`) naming ONLY an excluded import +refuses with exit 2; `verify --codegen` shares the same selection as `gen`. + +Uses the plan's reference fixtures verbatim (`APP`, `LOCK_V1`, the pinned +`acme-common-v1.json` artifact) — see +`.superpowers/sdd/2026-09-11-fr-023-phase-1a/global-constraints.md`. +""" +from __future__ import annotations + +import json +import shutil +from pathlib import Path + +from metaobjects.cli import main + +_CORPUS_ARTIFACT = ( + Path(__file__).resolve().parents[4] + / "fixtures" + / "dependency-conformance" + / "artifacts" + / "acme-common-v1.json" +) + +_APP_JSON = ( + '{"metadata.root":{"package":"app","children":[{"object.entity":{"name":"Order",' + '"children":[{"source.rdb":{"@table":"orders"}},{"field.long":{"name":"id"}},' + '{"identity.primary":{"name":"pk","@fields":["id"]}}]}}]}}' +) + +_LOCK_V1 = { + "schema_version": 1, + "dependencies": { + "acme-common": { + "version": "1.0.0", + "metamodelVersion": "1.0", + "resolvedFrom": {"path": "../acme-common/metaobjects"}, + "artifact": "acme-common.metaobjects.json", + "integrity": "sha256-10fbf886e22faceca32c56e5e647c3ff1c82f503e638cba3bd3aa9390f7c409d", + "packages": ["acme::common"], + "nodes": [ + "acme::common::Address", + "acme::common::Audited", + "acme::common::Customer", + ], + } + }, +} + + +def _consumer( + tmp_path: Path, + *, + scope_include: list[str] | None = None, + target_entities: list[str] | None = None, +) -> Path: + """A consumer project: APP + SNAP + CONFIG_REF + LOCK_V1, with one + `metaobjects.config.yaml` target (`generators: [names]`).""" + (tmp_path / "metaobjects").mkdir() + (tmp_path / "metaobjects" / "meta.app.json").write_text(_APP_JSON) + + metaobjects_dir = tmp_path / ".metaobjects" + deps_dir = metaobjects_dir / "deps" / "acme-common" + deps_dir.mkdir(parents=True) + shutil.copyfile(_CORPUS_ARTIFACT, deps_dir / "acme-common.metaobjects.json") + + config_json: dict[str, object] = { + "schema_version": 1, + "sources": [], + "dependencies": [{"name": "acme-common", "path": "../acme-common/metaobjects"}], + } + if scope_include is not None: + config_json["scope"] = {"include": scope_include} + (metaobjects_dir / "config.json").write_text(json.dumps(config_json)) + (metaobjects_dir / "deps.lock.json").write_text(json.dumps(_LOCK_V1)) + + entities_line = f"\n entities: {json.dumps(target_entities)}" if target_entities else "" + (tmp_path / "metaobjects.config.yaml").write_text( + "metadata: metaobjects\n" + "targets:\n" + " main:\n" + " outDir: gen\n" + f" generators: [names]{entities_line}\n" + ) + return tmp_path / "metaobjects.config.yaml" + + +def test_gen_excludes_the_dependencys_entities_by_default(tmp_path: Path) -> None: + cfg = _consumer(tmp_path) + rc = main(["gen", "--config", str(cfg)]) + assert rc == 0 + assert (tmp_path / "gen" / "order_names.py").exists() + assert not (tmp_path / "gen" / "customer_names.py").exists() + + +def test_gen_scope_include_naming_the_package_opts_it_in(tmp_path: Path) -> None: + cfg = _consumer(tmp_path, scope_include=["acme::common::**"]) + rc = main(["gen", "--config", str(cfg)]) + assert rc == 0 + assert (tmp_path / "gen" / "order_names.py").exists() + assert (tmp_path / "gen" / "customer_names.py").exists() + + +def test_gen_a_wildcard_scope_include_does_not_opt_the_package_in(tmp_path: Path) -> None: + # `acme::**` REACHES the package's nodes without NAMING it literally + # (`explicitly_includes`) — the default exclusion still applies. + cfg = _consumer(tmp_path, scope_include=["acme::**"]) + rc = main(["gen", "--config", str(cfg)]) + assert rc == 0 + assert (tmp_path / "gen" / "order_names.py").exists() + assert not (tmp_path / "gen" / "customer_names.py").exists() + + +def test_gen_target_naming_an_excluded_import_refuses_with_exit_2( + tmp_path: Path, capsys +) -> None: + cfg = _consumer(tmp_path, target_entities=["Customer"]) + rc = main(["gen", "--config", str(cfg)]) + assert rc == 2 + err = capsys.readouterr().err + assert "'Customer'" in err + assert "acme-common" in err + assert "scope.include" in err + assert not (tmp_path / "gen").exists() + + +def test_gen_target_naming_an_excluded_import_is_fine_once_scope_includes_it( + tmp_path: Path, +) -> None: + cfg = _consumer(tmp_path, scope_include=["acme::common::**"], target_entities=["Customer"]) + rc = main(["gen", "--config", str(cfg)]) + assert rc == 0 + assert (tmp_path / "gen" / "customer_names.py").exists() + assert not (tmp_path / "gen" / "order_names.py").exists() + + +def test_verify_codegen_shares_the_selection_with_gen(tmp_path: Path) -> None: + cfg = _consumer(tmp_path) + assert main(["gen", "--config", str(cfg)]) == 0 + # Fresh gen, same selection → verify --codegen must see no drift, and must + # NOT regenerate the excluded customer_names.py and report it "missing". + assert main(["verify", "--codegen", "--config", str(cfg)]) == 0 + + +def test_verify_codegen_shares_the_selection_after_scope_widens(tmp_path: Path) -> None: + cfg = _consumer(tmp_path, scope_include=["acme::common::**"]) + assert main(["gen", "--config", str(cfg)]) == 0 + assert main(["verify", "--codegen", "--config", str(cfg)]) == 0 diff --git a/server/python/tests/config/test_dependencies.py b/server/python/tests/config/test_dependencies.py index 6850aa2a9..667e64dd5 100644 --- a/server/python/tests/config/test_dependencies.py +++ b/server/python/tests/config/test_dependencies.py @@ -13,12 +13,19 @@ from metaobjects.config.dependencies import ( LOCK_FILE, + Collection, + ResolvedDependency, dependency_source_id, + explicitly_includes, + imported_from, read_lock, + refuse_unowned_packages, sha256_integrity, validate_lock, validate_manifest, + verify_snapshot, ) +from metaobjects import MetaDataLoader from metaobjects.errors import ErrorCode, ParseError CORPUS = Path(__file__).resolve().parents[4] / "fixtures" / "dependency-conformance" / "artifacts" @@ -140,3 +147,217 @@ def test_read_lock_raises_on_a_shape_violation(tmp_path: Path) -> None: with pytest.raises(ParseError) as e: read_lock(tmp_path) assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + + +# --------------------------------------------------------------------------- +# FR-023 Task 18 — verify_snapshot, explicitly_includes, imported_from, +# refuse_unowned_packages, Collection. The dependency-conformance corpus +# (`tests/conformance/test_dependency_conformance.py`) exercises these +# end-to-end via `resolve_collection_full`; these are the focused unit tests. +# --------------------------------------------------------------------------- + + +def _write_snapshot(tmp_path: Path, name: str, artifact_name: str, source: Path) -> None: + d = tmp_path / ".metaobjects" / "deps" / name + d.mkdir(parents=True) + (d / artifact_name).write_bytes(source.read_bytes()) + + +def test_verify_snapshot_no_dependencies_no_lock_returns_empty(tmp_path: Path) -> None: + # A mutation deleting the `lock is None and not declared` short-circuit + # would instead raise (or crash on `lock["dependencies"]` with lock=None) — + # this pins the byte-identical untouched-project path. + assert verify_snapshot(tmp_path, [], None) == [] + + +def test_verify_snapshot_declared_but_no_lock_is_stale(tmp_path: Path) -> None: + with pytest.raises(ParseError) as e: + verify_snapshot(tmp_path, [{"name": "acme-common", "path": "x"}], None) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + + +def test_verify_snapshot_declared_name_missing_from_lock_entries_is_stale(tmp_path: Path) -> None: + lock = validate_lock({"schema_version": 1, "dependencies": {}}) + with pytest.raises(ParseError) as e: + verify_snapshot(tmp_path, [{"name": "acme-common", "path": "x"}], lock) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + + +def test_verify_snapshot_lock_entry_without_a_declared_dependency_is_stale(tmp_path: Path) -> None: + _write_snapshot(tmp_path, "acme-common", "acme-common.metaobjects.json", CORPUS / "acme-common-v1.json") + lock = validate_lock({"schema_version": 1, "dependencies": {"acme-common": _entry()}}) + # Declares NOTHING, but the lock has an "acme-common" entry. + with pytest.raises(ParseError) as e: + verify_snapshot(tmp_path, [], lock) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + + +def test_verify_snapshot_resolves_the_pinned_artifact(tmp_path: Path) -> None: + _write_snapshot(tmp_path, "acme-common", "acme-common.metaobjects.json", CORPUS / "acme-common-v1.json") + lock = validate_lock({"schema_version": 1, "dependencies": {"acme-common": _entry()}}) + resolved = verify_snapshot(tmp_path, [{"name": "acme-common", "path": "../acme-common"}], lock) + + assert len(resolved) == 1 + dep = resolved[0] + assert dep.name == "acme-common" + assert dep.version == "1.0.0" + assert dep.packages == ("acme::common",) + assert dep.nodes == ("acme::common::Address", "acme::common::Audited", "acme::common::Customer") + assert dep.source_id == "dep:acme-common/acme-common.metaobjects.json" + assert dep.artifact_path.endswith("acme-common.metaobjects.json") + + +def test_verify_snapshot_orders_results_by_dependency_name_not_declaration_order( + tmp_path: Path, +) -> None: + _write_snapshot(tmp_path, "acme-common", "acme-common.metaobjects.json", CORPUS / "acme-common-v1.json") + entry_b = {**_entry(), "packages": ["b::pkg"], "nodes": ["b::pkg::Thing"]} + lock = validate_lock( + {"schema_version": 1, "dependencies": {"acme-common": _entry(), "b-dep": entry_b}} + ) + # `b-dep`'s artifact is never read (its packages/nodes are fabricated) because + # dependency name order is "acme-common" < "b-dep" and — for THIS assertion — + # only the ORDER of the resolved list matters, not that b-dep resolves too. + # Declare in the OPPOSITE order to prove result order is name-sorted, not + # declaration-sorted; b-dep would fail on its (fabricated) artifact if this + # test's dependency list actually needed it to resolve, so keep it absent + # and assert failure lands on it specifically, past acme-common. + specs = [{"name": "b-dep", "path": "y"}, {"name": "acme-common", "path": "x"}] + with pytest.raises(ParseError) as e: + verify_snapshot(tmp_path, specs, lock) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + assert "b-dep" in str(e.value) + + +def test_verify_snapshot_missing_artifact_is_stale(tmp_path: Path) -> None: + lock = validate_lock({"schema_version": 1, "dependencies": {"acme-common": _entry()}}) + with pytest.raises(ParseError) as e: + verify_snapshot(tmp_path, [{"name": "acme-common", "path": "x"}], lock) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + + +def test_verify_snapshot_hash_mismatch_is_stale(tmp_path: Path) -> None: + d = tmp_path / ".metaobjects" / "deps" / "acme-common" + d.mkdir(parents=True) + (d / "acme-common.metaobjects.json").write_text("{}") # wrong bytes + lock = validate_lock({"schema_version": 1, "dependencies": {"acme-common": _entry()}}) + with pytest.raises(ParseError) as e: + verify_snapshot(tmp_path, [{"name": "acme-common", "path": "x"}], lock) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE + + +def test_verify_snapshot_metamodel_major_mismatch(tmp_path: Path) -> None: + _write_snapshot(tmp_path, "acme-common", "acme-common.metaobjects.json", CORPUS / "acme-common-v1.json") + entry = {**_entry(), "metamodelVersion": "99.0"} + lock = validate_lock({"schema_version": 1, "dependencies": {"acme-common": entry}}) + with pytest.raises(ParseError) as e: + verify_snapshot(tmp_path, [{"name": "acme-common", "path": "x"}], lock) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE + + +def test_verify_snapshot_node_collision(tmp_path: Path) -> None: + _write_snapshot(tmp_path, "acme-common", "acme-common.metaobjects.json", CORPUS / "acme-common-v1.json") + _write_snapshot(tmp_path, "acme-common-2", "acme-common.metaobjects.json", CORPUS / "acme-common-v1.json") + lock = validate_lock( + { + "schema_version": 1, + "dependencies": {"acme-common": _entry(), "acme-common-2": _entry()}, + } + ) + specs = [{"name": "acme-common", "path": "x"}, {"name": "acme-common-2", "path": "y"}] + with pytest.raises(ParseError) as e: + verify_snapshot(tmp_path, specs, lock) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_NODE_COLLISION + + +def test_explicitly_includes_names_the_package_literally() -> None: + assert explicitly_includes(["acme::common::**"], "acme::common") is True + assert explicitly_includes(["acme::common::Address"], "acme::common") is True + # A wildcard REACHES the package's nodes without NAMING the package. + assert explicitly_includes(["acme::**"], "acme::common") is False + assert explicitly_includes(["**"], "acme::common") is False + assert explicitly_includes(None, "acme::common") is False + assert explicitly_includes([], "acme::common") is False + # A root-level (package-less) fqn is never named — fail-closed. + assert explicitly_includes(["acme::common::**"], "") is False + + +def test_imported_from_finds_the_owning_dependency() -> None: + deps = [ + ResolvedDependency( + name="acme-common", version="1.0.0", packages=("acme::common",), + nodes=(), artifact_path="x", source_id="dep:x", + ), + ResolvedDependency( + name="acme-extra", version="1.0.0", packages=("acme::extra",), + nodes=(), artifact_path="y", source_id="dep:y", + ), + ] + assert imported_from("acme::common", deps) == "acme-common" + assert imported_from("acme::extra", deps) == "acme-extra" + assert imported_from("acme::other", deps) is None + assert imported_from("acme::other", []) is None + + +def _collection(**overrides: object) -> Collection: + base: dict[str, object] = dict( + files=(), own_files=(), file_ids={}, dependencies=(), + imported_packages=frozenset(), imported_nodes=frozenset(), + scope_include=(), in_migrate_scope=None, + ) + base.update(overrides) + return Collection(**base) # type: ignore[arg-type] + + +def test_collection_imported_is_package_keyed() -> None: + c = _collection(imported_packages=frozenset({"acme::common"})) + assert c.imported("acme::common::Customer") is True + assert c.imported("app::Order") is False + + +def test_collection_in_scope_excludes_imported_by_default() -> None: + c = _collection(imported_packages=frozenset({"acme::common"})) + assert c.in_scope("app::Order") is True # own object — never excluded + assert c.in_scope("acme::common::Customer") is False # imported, no explicit include + + +def test_collection_in_scope_explicit_include_opts_the_package_in() -> None: + c = _collection( + imported_packages=frozenset({"acme::common"}), + scope_include=("acme::common::**",), + ) + assert c.in_scope("acme::common::Customer") is True + + +def test_refuse_unowned_packages_noop_when_nothing_is_imported() -> None: + result = MetaDataLoader.from_string('{"metadata.root":{"children":[]}}') + assert not result.errors + refuse_unowned_packages(result.root, frozenset(), frozenset()) + refuse_unowned_packages(result.root, None, None) # type: ignore[arg-type] + + +def test_refuse_unowned_packages_raises_for_a_new_local_node_in_a_dependency_package() -> None: + doc = ( + '{"metadata.root":{"package":"acme::common","children":[' + '{"object.value":{"name":"Note","children":[{"field.string":{"name":"text"}}]}}' + "]}}" + ) + result = MetaDataLoader.from_string(doc) + assert not result.errors + with pytest.raises(ParseError) as e: + refuse_unowned_packages(result.root, frozenset({"acme::common"}), frozenset()) + assert e.value.code == ErrorCode.ERR_DEPENDENCY_PACKAGE_NOT_OWNED + + +def test_refuse_unowned_packages_allows_a_node_already_in_imported_nodes() -> None: + # The overlay-merge outcome: the node's key IS one of the dependency's own + # `nodes` (it merged into the imported node, rather than declaring a new one). + doc = ( + '{"metadata.root":{"package":"acme::common","children":[' + '{"object.value":{"name":"Note","children":[{"field.string":{"name":"text"}}]}}' + "]}}" + ) + result = MetaDataLoader.from_string(doc) + refuse_unowned_packages( + result.root, frozenset({"acme::common"}), frozenset({"acme::common::Note"}) + ) # no raise diff --git a/server/python/tests/conformance/test_dependency_conformance.py b/server/python/tests/conformance/test_dependency_conformance.py new file mode 100644 index 000000000..e84a9532b --- /dev/null +++ b/server/python/tests/conformance/test_dependency_conformance.py @@ -0,0 +1,178 @@ +"""Runs the shared dependency corpus (FR-023) against the Python reference +implementation. Every port ships an equivalent runner reading this same file +(`fixtures/dependency-conformance/`, see its README for the case schema). + +The predicates under test are the COMPOSED ones (DESIGN §11.1 item 2): +`collection.imported` keys on the lock's `packages`, and `in_scope` / +`in_migrate_scope` exclude what a dependency owns unless the project's own +scope NAMES that package. Mirrors +`server/typescript/packages/sdk/test/dependency-conformance.test.ts`. +""" +from __future__ import annotations + +import json +import shutil +from pathlib import Path + +import pytest + +from metaobjects.cli import _load_collection_result +from metaobjects.config.dependencies import LOCK_FILE +from metaobjects.config.source_resolver import resolve_collection_full +from metaobjects.errors import ParseError +from metaobjects.shared.base_types import TYPE_OBJECT + +_CORPUS_DIR = ( + Path(__file__).resolve().parents[4] / "fixtures" / "dependency-conformance" +) +_CORPUS = _CORPUS_DIR / "cases.json" +_CASES = json.loads(_CORPUS.read_text())["cases"] + +# These two corpus cases use `view.text` purely as incidental content on an +# overlaid field (to exercise the overlay/dependency machinery, not the view +# subtype itself). `view.text` is one of the 11 TS-web-presentation-only view +# subtypes DEREGISTERED in this port — "no backend/codegen/render consumer in +# Python, so they are dead vocab here" (`meta/presentation/view/view_constants.py`) +# — a decision that predates FR-023. Loading either fixture fails with +# `ERR_UNKNOWN_SUBTYPE` before the overlay-merge pass this case is actually +# testing ever runs. Not a collection-resolver defect: `expectFiles` (pure +# resolution, untouched by the load) is still asserted normally below; only the +# LOAD-dependent assertions are replaced with the documented, known failure +# mode — asserted precisely, so a future registration of `view.text` fails this +# loudly and flags the exemption (and the corpus case, for Python) for review. +_VIEW_TEXT_NOT_REGISTERED_IN_PYTHON = frozenset( + { + "an-overlay-of-a-dependency-node-is-not-refused", + "an-overlay-whose-target-was-removed-fails", + } +) + + +def test_corpus_is_non_empty() -> None: + """A silent zero-case run is a failed gate, not a pass. + + `@pytest.mark.parametrize` over an empty list simply collects zero tests — + pytest reports that as a SKIP, not a failure, so a corpus that quietly lost + its cases would report green here with nothing actually checked. Mirrors + the TS runner's identically-named guard. + """ + assert len(_CASES) > 0 + + +def _materialize(case: dict, root: Path) -> Path: + """Materializes `case["tree"]` (and `case["treeFiles"]`, copied byte-for-byte + from the corpus dir) under `root`, then writes `config` / `lock` (each when + present) under `/.metaobjects/`. Returns the directory + resolution must be invoked against. Mirrors the TS runner's `materialize`. + """ + for rel, content in case["tree"].items(): + p = root / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(content) + + for rel, corpus_rel in case.get("treeFiles", {}).items(): + p = root / rel + p.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(_CORPUS_DIR / corpus_rel, p) + + resolve_dir = root / case.get("resolveFrom", ".") + metaobjects_dir = resolve_dir / ".metaobjects" + + if case["config"] is not None: + metaobjects_dir.mkdir(parents=True, exist_ok=True) + (metaobjects_dir / "config.json").write_text(json.dumps(case["config"], indent=2)) + if "lock" in case: + metaobjects_dir.mkdir(parents=True, exist_ok=True) + (metaobjects_dir / LOCK_FILE).write_text(json.dumps(case["lock"], indent=2)) + + return resolve_dir + + +def _top_level_fqns(root) -> list[str]: + """Every loaded top-level object's resolution key. + + ADR-0039 sanctioned own: `root.own_children()` is a ROOT-LEVEL scan + (`metadata.root` is never extended, so own and effective children are the + same set here) — exactly the "every loaded top-level object" the corpus's + `expectImported`/`expectSelected`/`expectMigrateGoverned` arms are defined + over (README: "EXHAUSTIVE set... over every loaded top-level object"). + """ + return [c.resolution_key() for c in root.own_children() if c.type == TYPE_OBJECT] + + +@pytest.mark.parametrize("case", _CASES, ids=[c["name"] for c in _CASES]) +def test_dependency_conformance_case(case: dict, tmp_path: Path) -> None: + resolve_dir = _materialize(case, tmp_path) + + if "expectError" in case: + with pytest.raises(ParseError) as e: + resolve_collection_full(resolve_dir) + assert e.value.code == case["expectError"] + return + + # A case with neither expectFiles nor expectError is a malformed corpus + # entry, not "expect zero files" — fail loudly rather than silently + # passing it (same discipline as source-resolution-conformance). + assert "expectFiles" in case, ( + f'corpus case "{case["name"]}" has neither expectFiles nor expectError' + ) + + collection = resolve_collection_full(resolve_dir) + root = tmp_path.resolve() + got = {p.relative_to(root).as_posix() for p in collection.files} + assert got == set(case["expectFiles"]) + # A set comparison alone cannot see a duplicate emission — assert the RAW + # list length too, before it is thrown away by the set conversion. + assert len(collection.files) == len(case["expectFiles"]) + + if case["name"] in _VIEW_TEXT_NOT_REGISTERED_IN_PYTHON: + result = _load_collection_result(collection) + assert result.errors and result.errors[0].code == "ERR_UNKNOWN_SUBTYPE", ( + "view.text started resolving in Python — remove this case from " + "_VIEW_TEXT_NOT_REGISTERED_IN_PYTHON and exercise it normally" + ) + return + + needs_load = ( + "expectImported" in case or "expectSelected" in case or "expectMigrateGoverned" in case + ) + if needs_load: + result = _load_collection_result(collection) + assert not result.errors, [f"{e.code}: {e.message}" for e in result.errors] + top_level = _top_level_fqns(result.root) + + if "expectImported" in case: + imported = sorted(fqn for fqn in top_level if collection.imported(fqn)) + assert imported == sorted(case["expectImported"]) + if "expectSelected" in case: + selected = sorted(fqn for fqn in top_level if collection.in_scope(fqn)) + assert selected == sorted(case["expectSelected"]) + if "expectMigrateGoverned" in case: + governed = sorted( + fqn + for fqn in top_level + if (collection.in_migrate_scope(fqn) if collection.in_migrate_scope else True) + ) + assert governed == sorted(case["expectMigrateGoverned"]) + + if "expectLoadError" in case: + # Two arms produce a load-time failure: a loader-native `MetaError` in + # `result.errors` (parse/merge/validation), or the post-load ownership + # refusal (`ERR_DEPENDENCY_PACKAGE_NOT_OWNED`), which — mirroring the TS + # `refuseUnownedPackages` — RAISES rather than populating `result.errors`. + try: + result = _load_collection_result(collection) + except ParseError as exc: + assert exc.code == case["expectLoadError"] + assert "expectErrorFiles" not in case, ( + "the ownership-refusal ParseError carries no file provenance to assert" + ) + else: + assert result.errors, "expected a load error but the collection loaded cleanly" + first = result.errors[0] + assert first.code == case["expectLoadError"] + if "expectErrorFiles" in case: + files = getattr(getattr(first, "envelope", None), "files", None) or ( + (first.source,) if first.source else () + ) + assert list(files) == case["expectErrorFiles"] diff --git a/server/python/tests/conformance/test_source_resolution_conformance.py b/server/python/tests/conformance/test_source_resolution_conformance.py index da5d71725..59c4997b9 100644 --- a/server/python/tests/conformance/test_source_resolution_conformance.py +++ b/server/python/tests/conformance/test_source_resolution_conformance.py @@ -110,8 +110,11 @@ def test_cli_falls_back_to_neutral_config(tmp_path: Path, monkeypatch) -> None: from metaobjects.cli import resolve_metadata_location monkeypatch.chdir(tmp_path) - got = resolve_metadata_location(config=None, root=tmp_path) - assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in got} == { + # FR-023: resolve_metadata_location now returns the full `Collection` at + # every rung — `.files` is the resolved file list a no-dependencies project + # still gets byte-identically (own files only, nothing led by an artifact). + collection = resolve_metadata_location(config=None, root=tmp_path) + assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in collection.files} == { "model/meta.a.json" } @@ -147,9 +150,10 @@ def test_docs_with_metaobjects_config_yaml_honors_declared_libraries( ) -> None: """`docs`'s no-positional branch loads `metaobjects.config.yaml` (rung 2 of the ladder) purely to read its `metadata` key, then reloads via - `_load_root_from_paths` — which dropped both `config.providers` and - `config.libraries`, even though `config` was already sitting right there. - A project declaring `libraries: ["ai"]` and `extends: + `_load_root_from_collection` (formerly `_load_root_from_paths`) — which + used to drop both `config.providers` and `config.libraries`, even though + `config` was already sitting right there. A project declaring + `libraries: ["ai"]` and `extends: metaobjects::ai::LlmCallBase` must resolve through `docs` exactly as it already does through `gen` / `verify --codegen` (see `test_shipped_library_ai.py::TestTheCliCanLoadTheLibrary`). From 77f0f0e77dc83f53744bb40fbdd859476b0ddd70 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 02:52:26 -0400 Subject: [PATCH 49/62] =?UTF-8?q?test(python):=20add=20the=20stale-snapsho?= =?UTF-8?q?t=20CLI=20scenario=20and=20fix=20a=20non-discriminating=20order?= =?UTF-8?q?ing=20test=20(FR-023=20=C2=A711=20fix=20round=201)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- .../tests/codegen/test_cli_dependencies.py | 25 +++++++++++++ .../python/tests/config/test_dependencies.py | 35 ++++++++++++------- 2 files changed, 48 insertions(+), 12 deletions(-) diff --git a/server/python/tests/codegen/test_cli_dependencies.py b/server/python/tests/codegen/test_cli_dependencies.py index fb30daeed..b0b0d75c3 100644 --- a/server/python/tests/codegen/test_cli_dependencies.py +++ b/server/python/tests/codegen/test_cli_dependencies.py @@ -148,3 +148,28 @@ def test_verify_codegen_shares_the_selection_after_scope_widens(tmp_path: Path) cfg = _consumer(tmp_path, scope_include=["acme::common::**"]) assert main(["gen", "--config", str(cfg)]) == 0 assert main(["verify", "--codegen", "--config", str(cfg)]) == 0 + + +def test_gen_a_stale_snapshot_is_refused_before_generation(tmp_path: Path, capsys) -> None: + """Step 2's fifth scenario: editing one byte of the committed snapshot must + make `resolve_metadata_location` -> `build_collection` -> `verify_snapshot` + raise `ERR_DEPENDENCY_SNAPSHOT_STALE` BEFORE anything is loaded or + generated — never a partial/stale `gen/` directory.""" + cfg = _consumer(tmp_path) + artifact = tmp_path / ".metaobjects" / "deps" / "acme-common" / "acme-common.metaobjects.json" + data = bytearray(artifact.read_bytes()) + data[0] ^= 0xFF # one bit-flipped byte -> the lock's pinned sha256 no longer matches + artifact.write_bytes(bytes(data)) + + rc = main(["gen", "--config", str(cfg)]) + assert rc != 0 + err = capsys.readouterr().err + # `_resolve_metadata_location_or_print_error` prints `str(exc)` — a + # ParseError's message, not its `.code` — so the STALE code itself is not + # a substring; the message text IS the coded diagnostic here. + assert "does not match the lock" in err + assert "run `meta deps sync`" in err + # The "before generation" half: no gen/ directory at all, not even a + # partial one — the failure must happen in resolution, before run_gen ever + # gets a root to generate from. + assert not (tmp_path / "gen").exists() diff --git a/server/python/tests/config/test_dependencies.py b/server/python/tests/config/test_dependencies.py index 667e64dd5..6548415f6 100644 --- a/server/python/tests/config/test_dependencies.py +++ b/server/python/tests/config/test_dependencies.py @@ -210,23 +210,34 @@ def test_verify_snapshot_resolves_the_pinned_artifact(tmp_path: Path) -> None: def test_verify_snapshot_orders_results_by_dependency_name_not_declaration_order( tmp_path: Path, ) -> None: + # BOTH dependencies must resolve successfully — a fabricated/absent second + # artifact can only ever fail, so whichever position it's visited in it is + # "the one that failed," and a `"name" in str(error)` assertion would pass + # under EITHER declaration-order or name-order iteration (this is exactly + # what fix-round-1 caught: the prior version of this test proved nothing). _write_snapshot(tmp_path, "acme-common", "acme-common.metaobjects.json", CORPUS / "acme-common-v1.json") - entry_b = {**_entry(), "packages": ["b::pkg"], "nodes": ["b::pkg::Thing"]} + b_dep_bytes = ( + b'{"metadata.root":{"children":[{"object.value":{"name":"Thing",' + b'"package":"b::pkg","children":[]}}]}}' + ) + b_dep_dir = tmp_path / ".metaobjects" / "deps" / "b-dep" + b_dep_dir.mkdir(parents=True) + (b_dep_dir / "b-dep.metaobjects.json").write_bytes(b_dep_bytes) + entry_b = { + **_entry(), + "artifact": "b-dep.metaobjects.json", + "integrity": sha256_integrity(b_dep_bytes), + "packages": ["b::pkg"], + "nodes": ["b::pkg::Thing"], + } lock = validate_lock( {"schema_version": 1, "dependencies": {"acme-common": _entry(), "b-dep": entry_b}} ) - # `b-dep`'s artifact is never read (its packages/nodes are fabricated) because - # dependency name order is "acme-common" < "b-dep" and — for THIS assertion — - # only the ORDER of the resolved list matters, not that b-dep resolves too. - # Declare in the OPPOSITE order to prove result order is name-sorted, not - # declaration-sorted; b-dep would fail on its (fabricated) artifact if this - # test's dependency list actually needed it to resolve, so keep it absent - # and assert failure lands on it specifically, past acme-common. + # Declared in the OPPOSITE of name order — resolution must still come back + # NAME-ordered ("acme-common" before "b-dep"), never declaration-ordered. specs = [{"name": "b-dep", "path": "y"}, {"name": "acme-common", "path": "x"}] - with pytest.raises(ParseError) as e: - verify_snapshot(tmp_path, specs, lock) - assert e.value.code == ErrorCode.ERR_DEPENDENCY_SNAPSHOT_STALE - assert "b-dep" in str(e.value) + resolved = verify_snapshot(tmp_path, specs, lock) + assert [d.name for d in resolved] == ["acme-common", "b-dep"] def test_verify_snapshot_missing_artifact_is_stale(tmp_path: Path) -> None: From 1d7fd882bfd0129adb8149ea78bbdebd6c70e4b1 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 03:00:51 -0400 Subject: [PATCH 50/62] feat(cli): meta init tracks .metaobjects/deps and deps.lock.json and no longer scaffolds package.meta.json (FR-023) Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- server/typescript/packages/cli/README.md | 2 +- .../packages/cli/src/commands/init.ts | 29 ++++++------ .../typescript/packages/cli/test/init.test.ts | 45 ++++++++++++++++--- 3 files changed, 52 insertions(+), 24 deletions(-) diff --git a/server/typescript/packages/cli/README.md b/server/typescript/packages/cli/README.md index 45a8feb90..4e0005e0d 100644 --- a/server/typescript/packages/cli/README.md +++ b/server/typescript/packages/cli/README.md @@ -86,7 +86,7 @@ Running `meta` with no arguments prints a concise status line (whether a `metaob ### `meta init` -Scaffolds `metaobjects/` (visible entity declarations, with a placeholder `meta.common.json`), `.metaobjects/` (hidden tool state: `config.json`, `package.meta.json`, `AGENTS.md`, `CLAUDE.md`, `.gitignore`, `.gen-state/`), the **owned codegen generators** at `codegen/generators/{entity,queries,routes,barrel}.ts`, and `metaobjects.config.ts` at the repo root. +Scaffolds `metaobjects/` (visible entity declarations, with a placeholder `meta.common.json`), `.metaobjects/` (hidden tool state: `config.json`, `AGENTS.md`, `CLAUDE.md`, `.gitignore`, `.gen-state/`), the **owned codegen generators** at `codegen/generators/{entity,queries,routes,barrel}.ts`, and `metaobjects.config.ts` at the repo root. The generators are copied from the codegen reference templates and are **yours to edit** (ADR-0034 scaffold-and-own); the scaffolded `metaobjects.config.ts` imports them locally, and `meta gen` runs from those local copies — not from the package. Each generator file is written only if absent, so re-running with `--force` never clobbers a hand-edited generator. diff --git a/server/typescript/packages/cli/src/commands/init.ts b/server/typescript/packages/cli/src/commands/init.ts index 972887875..076f34405 100644 --- a/server/typescript/packages/cli/src/commands/init.ts +++ b/server/typescript/packages/cli/src/commands/init.ts @@ -1,9 +1,9 @@ import { mkdir, writeFile, readFile, readdir, stat, rm } from "node:fs/promises"; import { join } from "node:path"; -import { basename, dirname } from "node:path"; +import { dirname } from "node:path"; import { existsSync as existsSyncWrap, readFileSync as readFileSyncWrap } from "node:fs"; import { createRequire } from "node:module"; -import { DEFAULT_CONFIG, ConfigSchema, saveConfig, PACKAGE_MANIFEST_FILE, DEFAULT_METADATA_DIR, DEFAULT_METAOBJECTS_DIR } from "@metaobjectsdev/sdk"; +import { DEFAULT_CONFIG, ConfigSchema, saveConfig, PACKAGE_MANIFEST_FILE, DEFAULT_METADATA_DIR, DEFAULT_METAOBJECTS_DIR, DEPS_DIR, LOCK_FILE } from "@metaobjectsdev/sdk"; import { assemble, resolveAgentContextRoot, planScaffold, AGENT_CONTEXT_MANIFEST_PATH, type Manifest, type Stack, @@ -85,7 +85,8 @@ const METAOBJECTS_GITIGNORE_BODY = `# The codegen merge base. The snapshot BODIE # These ARE meant to be tracked — keep them even if a broad pattern matches. !migrations/ !config.json -!package.meta.json +!${DEPS_DIR}/ +!${LOCK_FILE} `; // A minimal root .gitignore for a fresh project — only written when none exists, @@ -703,19 +704,15 @@ export async function init(opts: InitOptions): Promise { await writeFile(join(agentDir, ".gitignore"), METAOBJECTS_GITIGNORE_BODY, "utf8"); result.created.push(".metaobjects/.gitignore"); - // .metaobjects/package.meta.json — scaffold v0.3 package manifest if absent - const manifestPath = join(agentDir, PACKAGE_MANIFEST_FILE); - if (!(await fileExists(manifestPath))) { - const defaultPackageName = basename(opts.cwd); - const manifestBody = { - name: defaultPackageName, - version: "0.1.0", - extends: [] as string[], - }; - await writeFile(manifestPath, JSON.stringify(manifestBody, null, 2) + "\n", "utf8"); - result.created.push(`.metaobjects/${PACKAGE_MANIFEST_FILE}`); - } else { - result.preserved.push(`.metaobjects/${PACKAGE_MANIFEST_FILE}`); + // .metaobjects/package.meta.json — the v0.3 prototype manifest is deprecated + // (nothing reads it; FR-023 metadata dependencies is its replacement), so + // `meta init` no longer scaffolds it. A pre-existing one is left untouched; + // just note the deprecation so an existing project sees it. + const legacyManifestPath = join(agentDir, PACKAGE_MANIFEST_FILE); + if (await fileExists(legacyManifestPath)) { + result.warnings.push( + `note: .metaobjects/${PACKAGE_MANIFEST_FILE} is deprecated (nothing reads it; removed in 2.0) — see docs/features/metadata-dependencies.md`, + ); } await writeAgentContext(opts, result); diff --git a/server/typescript/packages/cli/test/init.test.ts b/server/typescript/packages/cli/test/init.test.ts index 5e9e01393..0b8c69c4e 100644 --- a/server/typescript/packages/cli/test/init.test.ts +++ b/server/typescript/packages/cli/test/init.test.ts @@ -116,14 +116,45 @@ describe("init() — happy path", () => { expect(existsSync(join(cwd, ".metaobjects", ".gen-state"))).toBe(true); }); - test("scaffolds package.meta.json under .metaobjects/ with three-field manifest", async () => { - await init({ cwd }); + // FR-023 — the v0.3 package.meta.json prototype is deprecated (nothing reads + // it); `meta init` must no longer scaffold it. Fails if the removed scaffold + // block (writing a { name, version, extends } manifest when absent) is restored. + test("does NOT scaffold .metaobjects/package.meta.json (FR-023 — deprecated v0.3 manifest)", async () => { + const result = await init({ cwd }); const manifestPath = join(cwd, ".metaobjects", "package.meta.json"); - expect(existsSync(manifestPath)).toBe(true); - const manifest = JSON.parse(readFileSync(manifestPath, "utf8")); - expect(manifest.name).toBeDefined(); - expect(manifest.version).toBe("0.1.0"); - expect(manifest.extends).toEqual([]); + expect(existsSync(manifestPath)).toBe(false); + expect(result.created).not.toContain(".metaobjects/package.meta.json"); + }); + + // A pre-existing package.meta.json (from a project scaffolded before FR-023) + // must be left exactly as it was — neither overwritten nor deleted — with its + // deprecation surfaced as a warning. Fails if init overwrites or deletes the + // file, or if the deprecation note regresses in wording or is dropped. + test("leaves a pre-existing .metaobjects/package.meta.json untouched and warns it is deprecated", async () => { + mkdirSync(join(cwd, ".metaobjects"), { recursive: true }); + const manifestPath = join(cwd, ".metaobjects", "package.meta.json"); + const existingManifest = JSON.stringify({ name: "legacy-pkg", version: "0.1.0", extends: [] }, null, 2) + "\n"; + writeFileSync(manifestPath, existingManifest, "utf8"); + + const result = await init({ cwd, force: true }); + + expect(readFileSync(manifestPath, "utf8")).toBe(existingManifest); + expect(result.created).not.toContain(".metaobjects/package.meta.json"); + expect(result.warnings).toContain( + "note: .metaobjects/package.meta.json is deprecated (nothing reads it; removed in 2.0) — see docs/features/metadata-dependencies.md", + ); + }); + + // The two new dependency-tooling files (deps.lock.json + the deps/ snapshot + // directory, both written only by `meta deps sync`, never by init) must be + // tracked rather than swept up by the per-target-shadow ignore pattern. Fails + // if either negation is missing, or if the old package.meta.json negation lingers. + test("scaffolded .metaobjects/.gitignore tracks deps/ and deps.lock.json, not package.meta.json", async () => { + await init({ cwd }); + const ignore = readFileSync(join(cwd, ".metaobjects", ".gitignore"), "utf8"); + expect(ignore).toContain("!deps/"); + expect(ignore).toContain("!deps.lock.json"); + expect(ignore).not.toContain("!package.meta.json"); }); test("writes a valid default config.json under .metaobjects/", async () => { From 5403df4491e1944eeb4c0cfef856632cccbebade Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 03:05:48 -0400 Subject: [PATCH 51/62] fix(cli): drop the orphaned package.meta.json entry from meta init's --print-only forecast (FR-023 task 19 fix round 1) Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- server/typescript/packages/cli/src/commands/init.ts | 1 - server/typescript/packages/cli/test/init.test.ts | 10 ++++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/server/typescript/packages/cli/src/commands/init.ts b/server/typescript/packages/cli/src/commands/init.ts index 076f34405..c623634ed 100644 --- a/server/typescript/packages/cli/src/commands/init.ts +++ b/server/typescript/packages/cli/src/commands/init.ts @@ -674,7 +674,6 @@ export async function init(opts: InitOptions): Promise { "metaobjects/meta.common.json", ".metaobjects/config.json", ".metaobjects/.gitignore", - `.metaobjects/${PACKAGE_MANIFEST_FILE}`, ); result.created.push(".metaobjects/AGENTS.md", ".metaobjects/CLAUDE.md", ".claude/skills/metaobjects-*", AGENT_CONTEXT_MANIFEST_PATH); for (const name of SCAFFOLDED_GENERATOR_NAMES) result.created.push(`${OWNED_GENERATORS_DIR}/${name}.ts`); diff --git a/server/typescript/packages/cli/test/init.test.ts b/server/typescript/packages/cli/test/init.test.ts index 0b8c69c4e..2f9e8ff98 100644 --- a/server/typescript/packages/cli/test/init.test.ts +++ b/server/typescript/packages/cli/test/init.test.ts @@ -157,6 +157,16 @@ describe("init() — happy path", () => { expect(ignore).not.toContain("!package.meta.json"); }); + // FR-023 fix round 1 — `--print-only` previews what a real run will create; the + // scaffold-removal above left an orphaned forecast entry at the printOnly branch + // (a second, separate site from the removed scaffold block) that still named + // .metaobjects/package.meta.json even though the real path never writes it. Fails + // if that push is restored, since a real init run never produces this file. + test("--print-only does not forecast .metaobjects/package.meta.json", async () => { + const result = await init({ cwd, printOnly: true }); + expect(result.created).not.toContain(".metaobjects/package.meta.json"); + }); + test("writes a valid default config.json under .metaobjects/", async () => { await init({ cwd }); const config = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); From b8c2d1242219f84457818191a39b60ed0c9228d2 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Sat, 12 Sep 2026 03:28:25 -0400 Subject: [PATCH 52/62] =?UTF-8?q?docs:=20metadata=20dependencies=20?= =?UTF-8?q?=E2=80=94=20declare,=20sync,=20build=20on,=20and=20what=20fails?= =?UTF-8?q?=20when=20upstream=20moves=20(FR-023=20=C2=A711)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01ApQWqotP1jaKs85XhKVXG4 --- CHANGELOG.md | 88 ++++ .../skills/metaobjects-authoring/SKILL.md | 26 + .../skills/metaobjects-codegen/SKILL.md | 38 ++ .../skills/metaobjects-verify/SKILL.md | 31 +- docs/CONFORMANCE.md | 48 +- docs/README.md | 2 + docs/features/abstracts-and-inheritance.md | 18 + docs/features/cli.md | 8 + docs/features/entities.md | 8 + docs/features/metadata-dependencies.md | 453 ++++++++++++++++++ docs/features/metadata-sources.md | 24 + .../skills/metaobjects-authoring/SKILL.md | 26 + .../skills/metaobjects-codegen/SKILL.md | 38 ++ .../skills/metaobjects-verify/SKILL.md | 31 +- .../skills/metaobjects-authoring/SKILL.md | 26 + .../skills/metaobjects-codegen/SKILL.md | 38 ++ .../skills/metaobjects-verify/SKILL.md | 31 +- .../skills/metaobjects-authoring/SKILL.md | 26 + .../skills/metaobjects-codegen/SKILL.md | 38 ++ .../skills/metaobjects-verify/SKILL.md | 31 +- .../skills/metaobjects-authoring/SKILL.md | 26 + .../skills/metaobjects-codegen/SKILL.md | 38 ++ .../skills/metaobjects-verify/SKILL.md | 31 +- .../skills/metaobjects-authoring/SKILL.md | 26 + .../skills/metaobjects-codegen/SKILL.md | 38 ++ .../skills/metaobjects-verify/SKILL.md | 31 +- fixtures/scope-conformance/README.md | 24 +- scripts/site/counts.test.ts | 1 + spec/roadmap.md | 30 +- 29 files changed, 1245 insertions(+), 29 deletions(-) create mode 100644 docs/features/metadata-dependencies.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 302d43de6..c60344004 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,94 @@ here.** ## [Unreleased] +### Added + +- **Metadata dependencies (FR-023, Phase 1a — TypeScript + Python).** A project can + now build on a metadata model published by another repository or package instead + of copy-pasting it: + - **`dependencies`** in `.metaobjects/config.json` — `[{ name, path }]` today (the + `npm`/`python` transport keys are reserved in the schema and refused by `sync` + with `ERR_DEPENDENCY_UNRESOLVED`, "not supported by this toolchain yet"). + - **`meta deps sync […] [--dry-run]` / `meta deps check` / `meta deps + list`** — `sync` resolves each dependency's `path`, validates its manifest and + artifact hash, re-loads the artifact standalone with core providers, and writes + the committed snapshot (`.metaobjects/deps//`) + lock + (`.metaobjects/deps.lock.json`); `check` compares the currently-installed + artifact's hash against the lock, read-only; `list` prints the lock. + - **`verify --deps`** — the same drift comparison as `meta deps check`, as a + gated `verify` subverb (needs the publisher's `path` reachable, so it is never + part of the bare-`verify` default). `drifted` or `unresolved` fails with the new + `ERR_DEPENDENCY_UPSTREAM_DRIFT`. + - **`sharedModelFile()`** (`@metaobjectsdev/codegen-ts`) — the publisher-side + generator: selects a subset of a project's own metadata by the `scope` + pattern grammar, closure-checks it, and emits one canonical-JSON artifact + + `metaobjects.pkg.json` manifest. Registered and discoverable (`meta gen + --list`), but deliberately **not** offered by `meta eject --list` — the + artifact is a contract whose bytes a cross-port corpus pins and whose hash + consumers verify, so a user-owned editable copy would invite a silent break. + - **An overlay authoring lint** in `meta verify` — advisory, runs on every + invocation (no subverb): a top-level `(type, resolutionKey)` declared in two or + more collection files (dependency artifacts included) where more than one + declaration lacks `overlay: true`. Never fails the build; mute with + `--no-overlay-lint` / `META_NO_OVERLAY_LINT=1`. + - **`ERR_DEPENDENCY_PACKAGE_NOT_OWNED`** — a project may `extends` or `overlay: + true` a dependency's node freely, but declaring a *brand-new* top-level node + into a package a dependency owns is refused by name, naming the fix. + - New error codes altogether: `ERR_DEPENDENCY_UNRESOLVED`, + `ERR_DEPENDENCY_MANIFEST_INVALID`, `ERR_DEPENDENCY_SNAPSHOT_STALE`, + `ERR_DEPENDENCY_NODE_COLLISION`, `ERR_DEPENDENCY_METAMODEL_INCOMPATIBLE`, + `ERR_DEPENDENCY_UPSTREAM_DRIFT`, `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`. + - New exports: `declaredTopLevelKeys` (`@metaobjectsdev/metadata` — the raw, + pre-parse walk the overlay lint and the loader's own overlay-only partition + share) and `serializeSharedDocument` (the canonical shared-artifact + serializer `sharedModelFile()` targets; Python's `serialize_shared_document` + is byte-identical, though Python has no publisher CLI wired to it yet — + `sharedModelFile()` itself is TypeScript-only in Phase 1a). `FileSource`'s + constructor takes an optional `{ id }`, so a dependency's synced snapshot + loads with `dep:/` provenance instead of reading like a + local file. + - Deliberately **not** built in Phase 1a — see `docs/features/metadata-dependencies.md` + ("Deferred"): the `npm`/`python`/`maven`/`nuget` transports, a local + co-development override, a usage-aware breaking-change classifier, cross-boundary + codegen imports (`packageBindings`), a runtime `ObjectManager` scope predicate, + and Java/Kotlin/C# as dependency consumers or publishers (Phase 2). + - No registered vocabulary changed — `metamodelVersion` stays `1.0`. + Docs: `docs/features/metadata-dependencies.md`. Corpus: + `fixtures/dependency-conformance/` (23 cases, TS + Python). + +### Changed + +- **A project that declares `scope.include` now sees its requirements-ledger + denominator narrow to that scope — even with zero dependencies.** The ledger's + ["coverable" object count](docs/features/requirements.md) is computed by + `Collection.inScope`, which is `matchesScope(fqn, scope) && …` regardless of + whether the project has any `dependencies` at all. Previously the ledger counted + every non-abstract entity in the loaded model; a project that scopes its own + codegen with `scope.include` now has its ledger coverage counted only over that + declared scope too. This is a real behavior change for an existing project that + already declares `scope.include` and reads its `meta verify` coverage numbers — + they may drop even though nothing was deleted. +- **`Collection`** (`@metaobjectsdev/sdk`) gains `dependencies`, `ownFiles`, + `fileIds`, `importedPackages`, `importedNodes`, `imported(fqn)`, + `declaredMigrateScope`; `inScope` and `inMigrateScope` are now the COMPOSED + predicates described above (default-exclusion of imported metadata) — byte-for-byte + identical to their prior behavior for a project with no `dependencies` and no + declared `scope`/`migrate.scope`. +- **The scope-pattern grammar (`compileScope`/`matchesScope`/`compilePattern`) + moved from `@metaobjectsdev/sdk` to `@metaobjectsdev/metadata`** — pure, + browser-safe string code, so `@metaobjectsdev/codegen-ts` (the `sharedModelFile()` + publisher side) can use it without a `sdk` dependency. `sdk` re-exports it + unchanged; existing importers of `compileScope`/`matchesScope` from `sdk` are + unaffected. +- **`scopeExpectedSchema`** (`@metaobjectsdev/migrate-ts`) takes a third, optional + `{ imported }` argument: an imported object the scope does not admit is removed + from the *expected* schema before `declaredSchemas` is computed — so importing a + table-backed entity can never turn the publisher's other tables into `DROP` + candidates, and never proposes creating the imported table. +- **`scanRequirements`** (`@metaobjectsdev/cli`) takes an optional second argument, + `{ coverable }` — the predicate the ledger denominator change above threads + through. + ### Deprecated - **`@metaobjectsdev/sdk`: the v0.3 `package.meta.json` prototype and workspace discovery** — diff --git a/agent-context/skills/metaobjects-authoring/SKILL.md b/agent-context/skills/metaobjects-authoring/SKILL.md index ca8f93f2a..86bfc2dfd 100644 --- a/agent-context/skills/metaobjects-authoring/SKILL.md +++ b/agent-context/skills/metaobjects-authoring/SKILL.md @@ -1111,6 +1111,32 @@ across files (same `package` + same `name` → merged; last-writer-wins on attr conflicts, structural children accumulate). Use `extends` to share shape between distinct entities; use `overlay` to split one entity's declaration across files. +**Extending and overlaying a metadata dependency's nodes is the expected way to +build on a shared model.** A project may declare `dependencies` in +`.metaobjects/config.json` and `meta deps sync` a publisher's metadata into a +committed snapshot — that snapshot loads BEFORE your own files, so a foreign +abstract resolves via `extends` and a foreign node re-opens via `overlay: true` +exactly like a local one. Two rules are specific to that boundary: + +- **Say `overlay: true` on every amendment to a node you don't own.** Within one + project the parser merges a same-`(type, package::name)` redeclaration whether + or not it carries the flag; only the flagged form fails loudly + (`ERR_OVERLAY_NO_TARGET`) when the target is gone. Skip the flag on a + dependency's node and its removal upstream silently becomes a new, disconnected + local object instead of a build failure — always flag a contribution to a node + you did not declare. +- **A brand-new top-level node declared into a dependency's package is refused**, + not silently excluded: `ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, naming the object, + the package, and the fix (declare it in your own package and `extends` the + dependency's node, or give it that node's exact name with `overlay: true` if you + meant to amend it). A *local node in your own package* is unaffected — this + refusal fires only for a package a dependency owns. + +What fails when upstream changes (a node removed/renamed, a member's shape +changed) is the loader's existing errors, unchanged — see +`docs/features/metadata-dependencies.md` for the full table and what `meta deps +sync`/`verify --deps` check that the loader cannot. + ## Discriminator inheritance (TPH) When several concrete entities are variants of one thing and should share a diff --git a/agent-context/skills/metaobjects-codegen/SKILL.md b/agent-context/skills/metaobjects-codegen/SKILL.md index ce26d4474..6b44eaa0c 100644 --- a/agent-context/skills/metaobjects-codegen/SKILL.md +++ b/agent-context/skills/metaobjects-codegen/SKILL.md @@ -138,6 +138,44 @@ instance/write artifacts regardless. Per-entity opt-outs exist (e.g. skipping client-side artifacts for a given entity) and are set as attributes on the entity in metadata, not in code. +## A dependency's metadata is load-only by default — codegen excludes it + +A project may declare `dependencies` in `.metaobjects/config.json` and `meta deps +sync` a publisher's metadata into a committed snapshot (TypeScript + Python, +Phase 1a). That snapshot's nodes load so your own model can resolve against them +(`extends`, `overlay: true`, plain FQN references) — but codegen (and `verify +--codegen`, and the requirements ledger's denominator) **excludes them by +default**. A node is "imported" when its metadata *package* is one a dependency +owns; an imported node is generated only when your own `scope.include` names that +package **literally** (`acme::common::**` or `acme::common::Address` name +`acme::common`; a bare `acme::**` or `**` do not — they match the package's nodes, +which is weaker than naming it). Naming the package in `scope.include` (and, if +you own its tables, `migrate.scope`) is how a consumer takes over a shared model — +the "I instantiate this metadata myself" case, no separate mode needed. + +Running `meta gen ` (or a Python `entities: [...]`) on a name that resolves +to nothing but excluded imports is refused by name (exit 2) rather than silently +generating nothing — the message names the dependency and the `scope.include` fix. + +## Publishing a shared model: `sharedModelFile()` + +The other side of the same feature: `@metaobjectsdev/codegen-ts` ships +`sharedModelFile({ name, include, exclude?, files?, version?, target? })`, a +generator a publisher wires to select a subset of its own metadata (by the same +scope-pattern grammar as `scope`) and emit it as one canonical-JSON artifact + +manifest — the thing a consumer's `meta deps sync` copies. It closure-checks the +selection (every reference from a selected node must resolve to another selected +node, or the build fails naming the pair) and re-loads the emitted artifact with +core providers only, so a Phase 1a export needing non-core vocabulary fails at +publish time. It is registered and shows up in `meta gen --list` like any other +generator — but **it is deliberately not offered by `meta eject --list`** (unlike +the four ADR-0034 scaffold-and-own generators). The artifact is a contract whose +bytes a cross-port corpus pins and whose hash consumers verify; a user-owned, +editable copy would invite an artifact that silently stops matching what +consumers expect. Only TypeScript can run it in Phase 1a — every port can +*consume* a dependency, but only the TypeScript toolchain can publish one. Full +detail: `docs/features/metadata-dependencies.md`. + ## You don't have to generate everything — pick your layers Codegen is **granular and à la carte, not all-or-nothing.** The most powerful diff --git a/agent-context/skills/metaobjects-verify/SKILL.md b/agent-context/skills/metaobjects-verify/SKILL.md index b4aeb6191..ac896864d 100644 --- a/agent-context/skills/metaobjects-verify/SKILL.md +++ b/agent-context/skills/metaobjects-verify/SKILL.md @@ -105,9 +105,29 @@ If this project declares `requirement.functional` / `requirement.architectural` `meta verify` — there is no subverb — and the severity of a broken link depends on the requirement's `@status`, which is the part that surprises people reading a failure. +## The overlay authoring lint runs on every run too + +If this project declares `dependencies` in `.metaobjects/config.json` +(`docs/features/metadata-dependencies.md`), `meta verify` also runs an ADVISORY +overlay lint on every invocation, no subverb needed: a top-level `(type, +resolutionKey)` declared in two or more collection files — dependency artifacts +included, since they are the base a consumer's own files amend — where MORE THAN +ONE declaration lacks `overlay: true`. Exactly one unflagged declaration is the +base and is fine; every additional one is a finding naming its file, because it is +exactly the state that turns loud (`ERR_OVERLAY_NO_TARGET`) the day the target is +renamed or removed, instead of quietly forking into a second, disconnected object. +Never fails the build; mute it with `--no-overlay-lint` or +`META_NO_OVERLAY_LINT=1` the same way the anti-pattern pass is muted. + +**A stale dependency snapshot is a load-time failure, not a `verify` finding.** +If `.metaobjects/deps//` disagrees with `.metaobjects/deps.lock.json` — a +missing snapshot, a hand-edited one, or a lock entry with no matching config +declaration — every command (not just `verify`) fails to even LOAD the metadata, +with `ERR_DEPENDENCY_SNAPSHOT_STALE` naming the fix: run `meta deps sync`. + ## The `verify` subverbs -`verify` has three drift checks. Run them in CI. +`verify` has four drift checks. Run the ones your project uses in CI. - **`--db`** — schema drift. Introspects the live database and fails if it has diverged from metadata. This is a **schema concern, so it is the Node toolchain's @@ -128,6 +148,15 @@ requirement's `@status`, which is the part that surprises people reading a failu if any reference isn't on the payload VO. This is the build-time gate for the prompt-construction pillar. +- **`--deps`** — dependency drift (FR-023, **Node `meta` only**, TS + Python + consumers). Re-resolves each declared dependency's `path` right now and compares + its installed artifact's hash against `.metaobjects/deps.lock.json` — the same + comparison `meta deps check` runs. Never part of the bare-`verify` default: it + needs the publisher's `path` reachable, which CI checking out only your own repo + may not have. Reports `current` / `drifted` / `unresolved` per dependency; + `drifted` or `unresolved` fails with `ERR_DEPENDENCY_UPSTREAM_DRIFT`, fix is + "review the artifact diff, then `meta deps sync`." + **Only `--db` is Node-universal.** `--codegen` / `--templates` run through each port's own build tool, not the Node `meta`: diff --git a/docs/CONFORMANCE.md b/docs/CONFORMANCE.md index 26c471e7a..0c10d3a49 100644 --- a/docs/CONFORMANCE.md +++ b/docs/CONFORMANCE.md @@ -43,8 +43,8 @@ regenerate with `ls -d fixtures//*/ | wc -l`. | [`fixtures/generator-registry-conformance/`](../fixtures/generator-registry-conformance/) | 1 canonical manifest | ✓ | ✓ | ✓ | ✓ | ✓ | | [`fixtures/provider-composition-conformance/`](../fixtures/provider-composition-conformance/) | 9 (5 error-shape + 4 compose-load) | ✓ | ✓ | — (JVM registry via Java) | ✓ | ✓ | | [`fixtures/source-resolution-conformance/`](../fixtures/source-resolution-conformance/) | 25 cases | ✓ (reference implementation) | ✓ | inherits via Java | ✓ | ✓ | -| [`fixtures/scope-conformance/`](../fixtures/scope-conformance/) | 10 cases | ✓ (reference implementation) | — | — | — | — | -| [`fixtures/dependency-conformance/`](../fixtures/dependency-conformance/) | 1 case (FR-023 Phase 1a, in progress — cases land with the implementation) | runner in place (`sdk/test/dependency-conformance.test.ts`), red by design until the resolver lands | — (Phase 2) | — (Phase 2) | — (Phase 2) | — (no runner yet) | +| [`fixtures/scope-conformance/`](../fixtures/scope-conformance/) | 10 cases | ✓ (reference implementation) | — | — | — | ✓ | +| [`fixtures/dependency-conformance/`](../fixtures/dependency-conformance/) | 23 cases | ✓ (reference implementation) | — (Phase 2) | — (Phase 2) | — (Phase 2) | ✓ (2 of 23 assert a documented Python-only vocabulary gap instead of the corpus's full contract — see below) | | [`fixtures/agent-context-conformance/`](../fixtures/agent-context-conformance/) | 4 | ✓ (the emitter is TS-owned) | — | — | — | — | | [`fixtures/metamodel-docs/`](../fixtures/metamodel-docs/) | 1 | ✓ (docs emit is TS-owned) | — | — | — | — | @@ -230,20 +230,42 @@ matching is case-sensitive.** These are exactly the rules four independent implementations would otherwise each get slightly wrong — the failure mode that produced the cross-port `LIKE`/`ILIKE` divergence fixed in 0.21.6. -**TypeScript is the only port with a runner today.** The reference implementation is -[`server/typescript/packages/sdk/src/scope.ts`](../server/typescript/packages/sdk/src/scope.ts) -(`compilePattern` / `compileScope` / `matchesScope`), and the corpus was authored -against it. Java, Kotlin, C# and Python have no runner yet; when each gains one, this corpus is -what it implements against — it exists now precisely so those four land on one -grammar rather than four. +**TypeScript and Python run it today.** The reference implementation is +[`server/typescript/packages/metadata/src/scope.ts`](../server/typescript/packages/metadata/src/scope.ts) +(`compilePattern` / `compileScope` / `matchesScope`; `@metaobjectsdev/sdk` re-exports +it unchanged, since it moved there from `sdk` when FR-023 needed it from +`codegen-ts` without a `sdk` dependency), and the corpus was authored against it. +Python's port (`server/python/src/metaobjects/scope.py`, `matches_scope` using +`re.fullmatch`) runs the same corpus. Java, Kotlin and C# have no runner yet; when +each gains one, this corpus is what it implements against — it exists now precisely +so those three land on one grammar rather than three. + +### `fixtures/dependency-conformance/` (23 cases) + +All 23 cases → [features/metadata-dependencies.md](features/metadata-dependencies.md) +(declaring a dependency, `meta deps sync`, the committed snapshot + lock, default +exclusion of imported metadata, overlay/extends across the boundary, and every load- +and resolution-time failure). File-shaped like `scope-conformance/` above: one +committed `cases.json`, no per-port fixture, no ledger. + +**TypeScript (the reference implementation) and Python both run all 23 cases.** +Python's runner asserts every case's `expectFiles` (pure resolution) normally; for +the 2 cases that incidentally use `view.text` (a TS-web-presentation-only view +subtype this port does not register) as overlay content unrelated to what the case +tests, it asserts the documented `ERR_UNKNOWN_SUBTYPE` load failure in place of the +full `expectImported`/`expectSelected`/`expectMigrateGoverned` assertions the corpus +defines for them — see `test_dependency_conformance.py`'s own comment for the +exemption and the condition that retires it. Java, Kotlin and C# have no runner — +Phase 1a is TypeScript + Python only; those three ports arrive in Phase 2. ## Orphaned fixtures (tested but not yet documented) -The fixtures in the eight corpora mapped above (metamodel 314 + yaml 15 + verify 31 -+ render 15 + persistence 33 + api-contract 41 + source-resolution 25 + scope 10) each -map to a feature doc. None are orphaned today. The remaining corpora in the totals table gate tooling -contracts (registry manifests, provider composition, agent context, docs emit) -rather than user-facing metamodel behaviour, so they have no feature-doc row. +The fixtures in the nine corpora mapped above (metamodel 314 + yaml 15 + verify 31 ++ render 15 + persistence 33 + api-contract 41 + source-resolution 25 + scope 10 + +dependency 23) each map to a feature doc. None are orphaned today. The remaining +corpora in the totals table gate tooling contracts (registry manifests, provider +composition, agent context, docs emit) rather than user-facing metamodel behaviour, +so they have no feature-doc row. If you add a new fixture and don't see a clear home for it, either: diff --git a/docs/README.md b/docs/README.md index 7903aad34..e645de4e2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -38,6 +38,7 @@ docs/ │ ├── generated-mutations.md # generated POST/PATCH mutation surface │ ├── image-upload.md # view.image form control (TS-web) │ ├── metadata-sources.md # where metadata comes from: sources, scope, discovery +│ ├── metadata-dependencies.md # building on a metadata model published elsewhere (FR-023) │ └── own-your-codegen.md # scaffold-and-own generator ownership (ADR-0034) └── ports/ # one file per language/framework port ├── typescript.md @@ -60,6 +61,7 @@ this tree is documentation, not the source of truth. | Compare what TS vs Java vs Kotlin vs C# vs Python emit for the same metadata | any [`features/*.md`](features/) — every feature shows all five ports side-by-side | | Author metadata in YAML instead of JSON | [`features/yaml-authoring.md`](features/yaml-authoring.md) | | Point the toolchain at metadata that lives somewhere other than `metaobjects/`, or scope what a project generates and migrates | [`features/metadata-sources.md`](features/metadata-sources.md) | +| Build on a metadata model another repository publishes (`dependencies`, `meta deps sync`, overlay/extend across the boundary) | [`features/metadata-dependencies.md`](features/metadata-dependencies.md) | | Record what the system is supposed to do, and stop agents reviving retired features | [`features/requirements.md`](features/requirements.md) | | Wire prompt construction (FR-004) | [`features/templates-and-payloads.md`](features/templates-and-payloads.md) | | Share a metadata shape across multiple instances (abstracts, `extends:`) | [`features/abstracts-and-inheritance.md`](features/abstracts-and-inheritance.md) | diff --git a/docs/features/abstracts-and-inheritance.md b/docs/features/abstracts-and-inheritance.md index 4a005cb90..a067e28b5 100644 --- a/docs/features/abstracts-and-inheritance.md +++ b/docs/features/abstracts-and-inheritance.md @@ -283,6 +283,24 @@ the same entity's declaration is split across files (e.g., domain code in one file, persistence overlay in another). See [`loaders.md`](loaders.md) for the overlay merge semantics. +### The same rule, across a repository boundary + +Nothing above changes when the base you `extends` or the node you `overlay` lives in +a [metadata dependency](metadata-dependencies.md) rather than your own tree — +dependency files load before yours in the same `loader.load(...)`, so a foreign +abstract resolves and a foreign node re-opens exactly like a local one. **One rule +is stricter across that boundary, though: say `overlay: true` on every amendment to +a node you do not own.** Within one project the parser merges a same-`(type, +package::name)` redeclaration whether or not it carries the flag; only the flagged +form fails loudly (`ERR_OVERLAY_NO_TARGET`) when the target disappears. Skip the flag +on a dependency's node and an upstream removal turns your amendment into a silent +new local object under the same name instead of a build failure — the flag is what +makes that loud. Declaring a brand-new top-level node into a dependency's package +(rather than amending one it already exports) is refused outright +(`ERR_DEPENDENCY_PACKAGE_NOT_OWNED`); see +[`metadata-dependencies.md`](metadata-dependencies.md) for the full rule and what +fails when the base you extended, or the node you overlaid, changes upstream. + ## When to use abstracts vs. new subtypes vs. attr extensions This is the same decision as diff --git a/docs/features/cli.md b/docs/features/cli.md index 6c783ac1b..fc6ef8dfc 100644 --- a/docs/features/cli.md +++ b/docs/features/cli.md @@ -33,6 +33,7 @@ command surface splits in two: | **Template/prompt drift** (`verify --templates`) | **Node `meta`** | `meta verify --templates` | TS reference (ADR-0021 D2) — `{{field}}`↔payload; the bare-`verify` default | | **Vocabulary upgrade** (`upgrade`) | **Node `meta`** | `meta upgrade [--to ] [--apply]` | **any backend** — rewrites RETIRED metadata vocabulary (`@violation` → `@counterexample`, `@readOnly` → `@mutability`, dropping `@verifiedBy`) and resolves ATTRIBUTE CONTRADICTIONS (`@fields` beside `@expr` on an index key). Node-only because it edits the metadata documents themselves, which every port shares; a non-TS project runs `npx meta upgrade` against its own `metaobjects/`. **Canonical JSON and YAML alike.** Previews by default. Retirements needing a human decision are refused and the run exits non-zero, so CI cannot record a partial migration as finished | | **Vocabulary search** (`types`) | **Node `meta`** | `meta types [query]` | **any backend** — apropos/`kubectl explain` over the live metamodel registry (names + descriptions + when-to-use); the vocabulary is cross-port identical (registry-conformance) | +| **Dependency sync** (`deps`) | **Node `meta`** | `meta deps sync \| check \| list` | **any backend** — resolves a declared metadata dependency's `path` transport into a committed snapshot + lock (FR-023); every port then LOADS that snapshot at every rung of the source ladder, so `sync`/`check`/`list` themselves stay Node-only. See [`metadata-dependencies.md`](metadata-dependencies.md) | | TS codegen | Node `meta` | `meta gen` | TS projects. **No `--template-spec` flag, deliberately** — `metaobjects.config.ts` already takes generator VALUES, so a declarative template generator is declared there (`templateGenerator()`, or `templateSpecToGenerators(parseTemplateSpec(spec))` to reuse a C#/Python spec file). Keeping it in the config is what lets `meta verify --codegen` regenerate WITH it, since that gate re-runs the config's generator list; see [declarative template scopes](codegen-concepts.md#declarative-template-scopes) | | C# codegen | `dotnet meta` | `dotnet meta gen` / `verify --templates` / `verify --codegen` | a .NET tool (`ToolCommandName=dotnet-meta`); invoked `dotnet meta` so it never shadows the Node `meta`; ships the ADR-0021 D2 subverbs (`--db` rejected, exit 2; bare `verify` = `--templates`). `gen` also accepts `--template-spec ` (+ `--template-root `, default `templates`) — the declarative Mustache template-codegen surface (the cross-port JSON contract shared with Python), **auto-discovered at `/template-spec.json`** when the flag is absent. Prefer the conventional path: `verify --codegen` takes no `--template-spec`, so discovery is how the drift gate sees your template generators at all; see [declarative template scopes](codegen-concepts.md#declarative-template-scopes) | | Java/Kotlin codegen | Maven plugin | `mvn metaobjects:generate` (`metaobjects:generate`) | Kotlin generators run through the same goal — see below. **No `--template-spec` flag, deliberately** — `` already loads a consumer class from the project classpath, so the declarative surface is `com.metaobjects.generator.template.TemplateScopeGenerator` wired as an ordinary `` with `