Skip to content

CONFORMANCE.md

Latest commit

 

History

History
355 lines (300 loc) · 25.6 KB

CONFORMANCE.md

File metadata and controls

355 lines (300 loc) · 25.6 KB
Raw

Conformance coverage

The MetaObjects standard ships 22 shared conformance corpora under fixtures/. Every port runs every corpus that is applicable to it and asserts the same expected behaviour against the same fixtures. This page is the inverse index: fixture → feature doc + per-port pass status, and it is the single place per-corpus counts are maintained (the port READMEs deliberately point here instead of restating them).

Not every corpus is a five-port corpus, and that asymmetry is deliberate rather than a gap — schema migration is TypeScript-owned (ADR-0015), Kotlin runs on the JVM and inherits the Java loader/render/extract engines instead of re-implementing them, and a few corpora gate tooling that only one port ships. The dark cells below say which is which.

If you are coming from a feature doc's ## Verified by section, you are in the right place. If you are wondering whether a particular fixture has a human-readable explanation somewhere, look it up in the "Fixture-to-doc mapping" table below.

Per-corpus totals

Counts are fixture directories (or scenario files, where a corpus is file-shaped); regenerate with ls -d fixtures/<corpus>/*/ | wc -l.

Corpus Fixtures TS Java Kotlin C# Python
fixtures/conformance/ (metamodel) 329 ✓ ✓ inherits via metadata-ktx ✓ ✓
fixtures/yaml-conformance/ 16 16 / 16 15 / 16 (1 ledgered: yaml-quoted-leading-zero — Java pipeline strips quotes off "007") inherits via Java 15 / 16 (1 ledgered: error-yaml-coerced-hex-in-string — YamlDotNet doesn't coerce 0xFF) 16 / 16
fixtures/verify-conformance/ 31 ✓ ✓ inherits via Java ✓ ✓
fixtures/verify-strict-conformance/ 1 ✓ — — — ✓
fixtures/render-conformance/ 15 ✓ ✓ inherits via Java ✓ ✓
fixtures/extract-conformance/ 33 ✓ ✓ inherits the shared JVM engine ✓ ✓
fixtures/output-prompt-conformance/ 14 ✓ ✓ ✓ ✓ ✓
fixtures/persistence-conformance/ 33 (27 query + 6 migration) all 33 27 query (migrations TS-only, ADR-0015) 27 query (via Exposed) 27 query 27 query
fixtures/api-contract-conformance/ 41 (26 core + 8 tph + 3 m2m + 2 jsonb + 2 write-through) ✓ (Fastify reference + generated lane) ✓ (embedded HTTP + JDBC) ✓ (embedded HTTP + Exposed) ✓ (HttpListener + Npgsql) ✓ (FastAPI + pg8000)
fixtures/validation-conformance/ 16 cases ✓ ✓ ✓ ✓ ✓
fixtures/registry-conformance/ 1 canonical manifest ✓ (reference emitter) ✓ ✓ ✓ ✓
fixtures/object-model-conformance/ 1 shared metadata fixture (per-port scenarios) ✓ ✓ ✓ ✓ ✓
fixtures/codegen-conformance/ 4 ✓ ✓ ✓ ✓ ✓
fixtures/template-codegen-conformance/ 3 ✓ ✓ ✓ ✓ ✓
fixtures/template-output-render-conformance/ 5 ✓ ✓ ✓ ✓ ✓
fixtures/generator-registry-conformance/ 1 canonical manifest ✓ ✓ ✓ ✓ ✓
fixtures/provider-composition-conformance/ 9 (5 error-shape + 4 compose-load) ✓ ✓ — (JVM registry via Java) ✓ ✓
fixtures/source-resolution-conformance/ 25 cases ✓ (reference implementation) ✓ inherits via Java ✓ ✓
fixtures/scope-conformance/ 10 cases ✓ (reference implementation) — — — ✓
fixtures/dependency-conformance/ 23 cases ✓ (reference implementation) — (Phase 2) — (Phase 2) — (Phase 2) ✓ (2 of 23 assert a documented Python-only vocabulary gap instead of the corpus's full contract — see below)
fixtures/agent-context-conformance/ 4 ✓ (the emitter is TS-owned) — — — —
fixtures/metamodel-docs/ 1 ✓ (docs emit is TS-owned) — — — —

A ✓ means the port runs the corpus green; an explicit n / m is used where a port carries a ledgered divergence. The two ledgered YAML fixtures are documented library-vs-pipeline divergences (see the _comment block in each port's yaml-conformance-expected-failures.json for the full reconciliation note). They are tracked as known-gaps rather than silently patched — the runner treats listed fixtures as passing, but a future port-level reconciliation pass would close them.

Split coverage: where a feature is gated, and where it deliberately is not

The matrix is corpus-shaped, so a feature whose coverage splits — across several corpora, or between its vocabulary and its behaviour — never gets a row of its own, and its boundary ends up discoverable only from the feature doc. The split ones are recorded here instead.

Capability requirements (features/requirements.md):

  • Vocabulary — gated in all five ports. The types, their attributes and their closed enums travel in registry-conformance's byte-matched manifest, which every port reproduces exactly, and accept/reject behaviour is pinned by requirement-* fixtures in fixtures/conformance/. A port that drifts on what it will load fails.
  • Checks — TypeScript only, by decision. The meta verify diagnostics over requirements ship in the TypeScript CLI; the other ports load and validate and stop there. Same call as ADR-0015: one implementation of a build-time gate rather than five, where the gate is not a per-port runtime concern. verify-conformance therefore holds no requirement cases, and that absence is deliberate rather than a gap.

Stated as mechanisms rather than as a list of attribute names on purpose — the requirement vocabulary has a breaking change scheduled (FR-038), which moves what the manifest contains without moving the boundary between the two halves.

Does the generated code compile (codegen-compile-conformance, all five ports):

  • Not a corpus, so it has no row. It reuses fixtures/persistence-conformance/canonical/meta.fitness.json rather than adding a kitchen sink beside the one that already exists — 16 entities, two view-backed projections, two M:N junctions including a self-join, a 4-entity TPH hierarchy, jsonb, isArray, currency, decimal, and AllTypes carrying every persistable field subtype. A second such model would drift from the first, and this one is maintained by the other corpora already.
  • Why it exists. Every corpus above gates BEHAVIOUR. None asks whether the emitted code builds — so four "generated code does not compile" defects shipped in 1.0.4 with the whole matrix green, because gen exits 0 in all four cases and the adopter's build is the first thing that disagrees. The gate found five more across the ports on its first runs.
  • Each port uses its own compiler, so it is the same question rather than the same code: ts.createProgram, Roslyn, javac via ToolProvider, KotlinCompilation, and — Python having no static compiler — importing the generated package (which is what resolves the imports BETWEEN modules) plus ruff F821 for the branches importing never executes.
  • Every port excludes its framework-bound route tier — TS routesFile, C# RoutesGenerator, Java SpringControllerGenerator, Kotlin KotlinSpringControllerGenerator. Those imports are not on an in-memory compile's classpath and stubbing them drowns the signal; that tier is compiled and booted over real HTTP in the api-contract integration lane instead. The boundary is one cross-port rule, not four local concessions.
  • The prompt tier is out of scope here because the fitness corpus declares no template.* nodes. Including those generators would emit nothing and read as coverage that is not there.

Do features work in combination, judged independently (TypeScript only, the integration-tests package, real Postgres):

  • Why it exists. Every corpus above exercises each feature on its own, and every one was green while the first real application found nine defects in an afternoon — each where two features meet (a reference onto a TPH subtype, a required default on one, an M:N declared on one). Two of them were invisible for a structural reason as well: meta verify --db diffs a database against the same buildExpectedSchema that wrote its migration, so a dropped FK looked in sync; and a dropped .references() compiles.
  • The independent oracle (src/independent-oracle.ts) restates, from the metadata alone, which table stores each object's rows, which columns and FKs that table has, and which routes the API serves — and reads the answer from pg_catalog, from Drizzle's own getTableConfig, and from the booted Fastify app, never through migrate-ts or codegen. independent-oracle-pg.test.ts asks it of a hand-written TPH model, including an M:N traversal against real rows, and of the committed canonical/schema.postgres.sql every port's query runner executes.
  • The feature-combination gate (feature-combinations-pg.test.ts) enumerates feature axes — hierarchy shape, where a link is declared, what it points at, link kind, link-table key, a required or defaulted field — and runs a pairwise covering set of models (23 today) through generate, tsc with routes included, migrate, and boot, judged by the oracle. It found three defects on its first run that no other gate could see.
  • Known defects are a ledger, not a skip. A check that fails for a recorded defect passes only if every failure line matches that defect's signature, and a recorded defect that stops reproducing fails its case. The ledger can only shrink.

How to tell a deliberate split from a real parity gap, since the two look identical in the matrix — both show one port covered and four blank. Ask what the uncovered ports claim. Here they claim nothing: they load requirement vocabulary and stop, exactly as the feature doc says. Contrast {{#hasField}} in 0.23.1, where the JVM emitted has<Field>() onto generated payload records and verify accepted the section, while no render engine in any port implemented the other half — two ports shipping halves of one promise, with no fixture that could see it. A split is deliberate when no port makes a claim the corpus would have to check; it is a gap when some port already made one.

A hazard for whoever removes an attribute — and it has now fired once. Contract prose lives inside attr descriptions in expected-registry.json, where five ports byte-match it and nothing else gates it. @trackedBy still carries "NOT resolved by verify, which does not reach the network". @verifiedBy carried "verify checks each name EXISTS and is not skipped; it never runs them" — and that attribute was retired in 0.24.0 (FR-038), which deleted the only gated statement of that guarantee along with it.

That removal was done to the rule: the sentence was rehomed in the same change, to spec/capability-ledger.md (§ "verifiedBy — RETIRED") and to the migration guide, before the description was deleted. Do the same for the next one. Sweep every description being removed for contract prose and rehome it in the same change, not after — the same class as the ADR-0047 renumbering trap, where a string that reads like prose is actually a gated artifact.

Per-port runners + commands:

Port Metamodel + YAML + render + verify Persistence API contract
TypeScript cd server/typescript && bun test (per-package, ~3s) scripts/integration-test.sh ts (needs Docker) cd server/typescript/packages/integration-tests && bun test test/api-contract.test.ts (needs Docker)
Java cd server/java && mvn -pl metadata test (and per-tier -pl render, etc.) scripts/integration-test.sh java (needs Docker) mvn -f server/java/integration-tests/pom.xml test -Dtest=ApiContractConformanceTest (needs Docker)
Kotlin cd server/java && mvn -pl codegen-kotlin test (snapshot suite) mvn -f server/java/integration-tests-kotlin/pom.xml test (needs Docker) mvn -f server/java/integration-tests-kotlin/pom.xml test -Dtest=ApiContractConformanceTest (needs Docker)
C# dotnet test (per project) scripts/integration-test.sh csharp (needs Docker) dotnet test server/csharp/MetaObjects.IntegrationTests/MetaObjects.IntegrationTests.csproj --filter "FullyQualifiedName~ApiContractConformanceTest" (needs Docker)
Python pytest (per package) scripts/integration-test.sh python (needs Docker) cd server/python && uv run --extra integration pytest tests/integration/test_api_contract.py (needs Docker)

The persistence-conformance corpus is intentionally on-demand — none of the unit-test runners (bun test, dotnet test, pytest, mvn test) pull Docker. scripts/integration-test.sh is the entry point and is wired into docs/RELEASING.md §2b as the pre-latest gate.

Fixture-to-doc mapping

fixtures/conformance/ — metamodel loader + canonical serializer (329)

Fixture prefix Feature doc
loader-basic-*, error-parse-* features/loaders.md
extends-*, error-extends-* features/entities.md (inheritance)
identity-*, subtype-entity-*, subtype-value-* features/entities.md (identity / entity vs value)
attr-*, error-attr-*, error-reserved-word-as-attr features/entities.md (attributes)
doc-common-attrs-* features/entities.md (documentation common attrs)
auto-set-on-* features/entities.md (auto-set timestamps)
attr-filter-*, loader-filterable-*, warning-filterable-*, layout-data-grid-*, error-data-grid-* features/entities.md (filter / sort / grid)
overlay-* features/entities.md (overlay / merge)
merge-three-way-no-conflict, error-merge-conflict-attr, merge-conflict-unmarked-attr-redeclaration, warning-duplicate-declaration features/loaders.md (multi-file merge attribution, FR5c)
field-string-*, field-decimal-*, field-object-storage-*, error-field-object-storage-* features/field-types.md
currency-* features/field-types.md (currency)
enum-*, error-enum-*, warning-enum-* features/field-types.md (enum)
source-rdb-*, source-db-table-*, source-db-view-*, source-multi-source-*, error-source-* features/source-kinds.md
relationship-*, error-unknown-relationship-* features/relationships.md
template-*, error-template-* features/templates-and-payloads.md
origin-*, error-origin-* features/templates-and-payloads.md (payload origins)
projection-*, error-projection-*, field-readonly-on-view-projection features/source-kinds.md (projections + the object taxonomy, ADR-0028)
requirement-*, error-unknown-attr-requirement features/requirements.md (vocabulary only — the meta verify checks are TS-owned; see "Split coverage" above)
smoke-empty-metadata features/entities.md

fixtures/yaml-conformance/ (16)

All 16 fixtures → features/yaml-authoring.md. The corpus splits into 9 happy-path fixtures (sigil-free attrs, array suffix, anchor/alias, block scalars, mixed bare-and-prefixed, quoted leading zero, an overlay declared before its base in one file, etc.) and 7 error-yaml-* fixtures that pin the YAML 1.1 coercion guards (bool / null / hex in string contexts; numeric in enum contexts; reserved-as-attr).

fixtures/render-conformance/ (15)

All 15 fixtures → features/templates-and-payloads.md (render engine output section). 4 are end-to-end shape examples (prompt / email / spreadsheet / CSV-injection escape); 10 pin Mustache-engine semantics — dotted-path lookup, parent-context fallthrough, falsy/empty-array section behavior, inverted sections, nested partials, standalone-tag whitespace stripping, raw-HTML bypass, trailing-newline preservation, and unicode multibyte handling.

fixtures/verify-conformance/ (31)

All 31 fixtures → features/migrations-and-drift.md (template drift section — Renderer.verify).

fixtures/persistence-conformance/ (33 — 27 query + 6 migration)

fixtures/api-contract-conformance/ (41)

All 41 scenarios → features/api-contract.md (cross-port REST API URL grammar + JSON wire format). Verifies every backend's emitted CRUD routes answer identically over HTTP — list / get / create / patch+put / delete, plus pagination (limit/offset), sort (sort=field:dir), the withCount=1 envelope, the not_found / invalid_sort error envelopes, and the 201 / 204 status codes.

The corpus also covers the 9 cross-port filter operators (eq, ne, gt, gte, lt, lte, in, like, isNull) plus the implicit-AND combinator and 2 error shapes (invalid_filter_field / invalid_filter_op) under the URL grammar ?filter[<field>][<op>]=<value> (FR-009). On top of the 26 core scenarios the corpus carries four sub-corpora — tph/ (8, single-table inheritance), m2m/ (3), jsonb/ (2, typed value-object columns) and write-through/ (2, table-write + view-read entities). All 5 ports — TS, Java, Kotlin, C#, Python — run it in BOTH lanes: a hand-rolled reference server and the port's own GENERATED API artifact booted over HTTP.

fixtures/source-resolution-conformance/ (25 cases)

All 25 cases → features/metadata-sources.md (how a declared sources set resolves to a file list). Companion to scope-conformance/ below — sources decides which files are read, scope filters what is emitted from them. The corpus is file-shaped: one committed cases.json, read directly by every port's runner, with no per-port fixture and no ledger.

It pins the resolved file SET for a declared sources list — the default directory, replacement-not-merge, the relative-path base (the directory HOLDING .metaobjects/, never the process cwd), recursive directory walking, case-insensitive extension matching, union-with-de-duplication, and every error condition (an unresolvable path, an unsupported resource/package kind, and a malformed config — "expectError": true pins only that resolution RAISES, since which error code it raises with is deliberately NOT a cross-port contract; see the corpus README). All four CLI surfaces run it — TypeScript (the reference implementation), C#, Python, and Java (Kotlin inherits it, since Kotlin has no CLI entry point of its own and runs through the same Maven plugin as Java): server/typescript/packages/sdk/test/source-resolution-conformance.test.ts, server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs, server/python/tests/conformance/test_source_resolution_conformance.py, and server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java.

fixtures/scope-conformance/ (10 cases)

All 10 cases → features/metadata-sources.md (the scope pattern grammar). The corpus is file-shaped: one committed cases.json, read directly by every port's runner, with no per-port fixture and no ledger.

It pins the semantics of a consumer's include/exclude scope over fully-qualified names — * matches within one :: segment and never crosses it; a segment that is exactly ** matches one or more whole segments (so acme::** does not match the bare acme); every other character is literal, regex metacharacters included; an absent or empty include means everything; multiple include patterns are a union and exclude is applied after it; and matching is case-sensitive. These are exactly the rules four independent implementations would otherwise each get slightly wrong — the failure mode that produced the cross-port LIKE/ILIKE divergence fixed in 0.21.6.

TypeScript and Python run it today. The reference implementation is server/typescript/packages/metadata/src/scope.ts (compilePattern / compileScope / matchesScope; @metaobjectsdev/sdk re-exports it unchanged, since it moved there from sdk when FR-023 needed it from codegen-ts without a sdk dependency), and the corpus was authored against it. Python's port (server/python/src/metaobjects/scope.py, matches_scope using re.fullmatch) runs the same corpus. Java, Kotlin and C# have no runner yet; when each gains one, this corpus is what it implements against — it exists now precisely so those three land on one grammar rather than three.

fixtures/dependency-conformance/ (23 cases)

All 23 cases → features/metadata-dependencies.md (declaring a dependency, meta deps sync, the committed snapshot + lock, default exclusion of imported metadata, overlay/extends across the boundary, and every load- and resolution-time failure). File-shaped like scope-conformance/ above: one committed cases.json, no per-port fixture, no ledger.

TypeScript (the reference implementation) and Python both run all 23 cases, identically — every case's expectFiles/expectImported/expectSelected/ expectMigrateGoverned/expectLoadError assertions apply to both runners with no exemption. The two overlay cases that need a view child as incidental content use view.currency (the one concrete view.* subtype registered cross-port), not view.text — a prior revision used view.text, which Python doesn't register, and carried a since-discharged allowlist for it. Java, Kotlin and C# have no runner — Phase 1a is TypeScript + Python only; those three ports arrive in Phase 2.

ADR-0055 (deferred overlay application) — SHIPPED in every port. The nine cases added for it (overlay-mixed-file-base-in-later-file, overlay-mixed-file-base-in-earlier-file, overlay-same-file-before-base, overlay-nested-under-plain-parent-base-later, overlay-applies-after-all-plain-declarations, overlay-two-overlays-source-order, error-overlay-no-target, error-overlay-no-target-nested, and the YAML twin yaml-overlay-before-base-same-file) are green on TypeScript, C#, Java, Kotlin (via the JVM loader) and Python, with no entry in any expected-failures ledger — they ran rather than being excused. The fixtures were committed while three ports were still red, and the red was recorded here rather than in a ledger, because ledgering a fixture the design intends to pass is how a port quietly stays behind.

Orphaned fixtures (tested but not yet documented)

The fixtures in the nine corpora mapped above (metamodel 329 + yaml 16 + verify 31

  • render 15 + persistence 33 + api-contract 41 + source-resolution 25 + scope 10 + dependency 23) each map to a feature doc. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider composition, agent context, docs emit) rather than user-facing metamodel behaviour, so they have no feature-doc row.

If you add a new fixture and don't see a clear home for it, either:

  1. Add a section to the closest matching feature doc and reference it from its ## Verified by block, or
  2. Open an issue if the fixture exercises behavior the feature docs haven't yet described.

Orphaned docs (documented but no fixture coverage)

None of the 8 feature docs lack fixture coverage. The closest case is the high-level overview material in each doc (e.g., the "Anatomy of an entity" explainer in entities.md) — these are pedagogical, not behavioral, and don't require pinning.

See also