From b331165e0aafbedcfb8aaaf32eb958b0b8bd8bd1 Mon Sep 17 00:00:00 2001 From: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com> Date: Sun, 28 Jun 2026 13:38:58 +0800 Subject: [PATCH 1/2] =?UTF-8?q?docs(adr):=20ADR-0076=20=E2=80=94=20objectq?= =?UTF-8?q?l=20layering=20(lean=20subpath=20+=20enforced=20boundary=20+=20?= =?UTF-8?q?trigger-gated=20repo=20split)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Data-driven v12 assessment of how to let a thin host (objectbase gateway) embed only the objectql engine without the whole platform. Decision (sequenced, based on measured engine↔platform coupling): - Now (additive, non-breaking): a `@objectstack/objectql/core` subpath lean entry; an engine↔platform boundary ratchet (core must not import protocol/plugin/kernel); a capability/profile contract (derive required caps from object schema, default fail-closed, `trusted` via two-key). - Later (trigger-gated): extract a standalone repo on the objectui model once the cross-package commit ratio (currently 88%) falls. Rationale recorded in-ADR: knex is a query builder objectql already uses; splitting protocol saves ≈0 deps; 88% of last month's engine/registry commits were cross-package, so the engine API is not freezable/separable yet. Co-Authored-By: Claude Opus 4.8 --- docs/adr/0076-objectql-core-tiering.md | 150 +++++++++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100644 docs/adr/0076-objectql-core-tiering.md diff --git a/docs/adr/0076-objectql-core-tiering.md b/docs/adr/0076-objectql-core-tiering.md new file mode 100644 index 0000000000..14942d8807 --- /dev/null +++ b/docs/adr/0076-objectql-core-tiering.md @@ -0,0 +1,150 @@ +# ADR-0076: objectql layering — a lean embeddable surface, an enforced engine↔platform boundary, and a trigger-gated path to a standalone repo + +**Status**: Proposed (2026-06-28, rev. 3 — data-driven) — feasibility + value assessment for the next major (v12) +**Deciders**: ObjectStack Protocol Architects +**Builds on**: [ADR-0025](./0025-plugin-package-distribution.md) (plugin package distribution), [ADR-0033](./0033-ai-assisted-metadata-authoring.md) (open-core boundary), [ADR-0048](./0048-cross-package-metadata-collision.md) (package id is the addressing unit), [ADR-0066](./0066-unified-authorization-model.md) (secure-by-default `access`, `requiredPermissions`, posture-gated RLS bypass), [ADR-0070](./0070-package-first-authoring.md) (package-first authoring) +**Consumers**: `@objectstack/objectql` (gains a `./core` subpath entry; no package split), `@objectstack/formula`, `@objectstack/plugin-security`, `@objectstack/plugin-sharing`, `@objectstack/spec` (shared object model), the host boot path (`createObjectQLKernel`), and out-of-tree embedders — notably `../objectbase` (its `gateway`). + +**Premise**: objectql was conceived as "a metadata-driven ORM that replaces TypeORM." It has outgrown that framing: it is a metadata-driven **data engine** that sits *on top of* knex (`driver-sql` depends on `knex@^3.2.10`) with an application platform layered above (RLS, sharing, field permissions, formula, protocol/metadata management, federation). The "ORM ambition" is not the source of complexity; the platform above the engine is — and that platform is the product value. + +> **Trigger**: `../objectbase` (its `gateway` — a thin, latency-sensitive host for some core algorithms) wants to use *only* the objectql engine without dragging in the whole platform; an agent inspecting the source recommended dropping objectql for raw knex. Investigation showed this is **half perception** (the engine already runs standalone; `protocol.ts` — 268KB — is never loaded for basic CRUD) and **half a productization gap** (no lean entry point, no standalone example). The team previously ran objectql as a *separate repo* (`@objectql/core`) and merged it back due to maintenance overhead, but notes `../objectui` proves a viable separate-repo model (sibling checkout, local link, no publishing). This ADR decides the layering for v12 **based on measured engine↔platform coupling**, not on aesthetics. + +--- + +## TL;DR + +1. **knex is not an ORM; it is a SQL query builder — and objectql already uses it.** "Use knex instead of objectql" is a category error; objectql is a *consumer* of knex, not a competitor. Retreating to TypeORM/knex forfeits the one-library / one-object-model goal and eliminates none of the felt complexity. +2. **Splitting protocol into its own package saves ≈ 0 dependencies** (verified: `protocol.ts` adds no third-party or workspace dep; engine+registry's only external import is `node:async_hooks`). The footprint argument for a package split does not hold. +3. **The engine API is NOT stable enough to freeze or separate today.** Measured over the last month: of 50 commits touching `engine.ts`/`registry.ts`, **44 (88%) also changed files outside the objectql package** — high-frequency cross-package co-evolution. Separating now would convert ~44 atomic commits/month into two-repo coordinated PRs. +4. **Decision (sequenced):** *Now* — stay in the monorepo; ship the lean value **additively** (a `./core` subpath entry, a standalone example) and **impose discipline** (an engine↔platform boundary ratchet + a capability/profile contract). *Later* — when the cross-package ratio falls (engine stabilizes), extract a standalone `objectql` repo on the `objectui` model. The boundary ratchet is the forcing function that drives that ratio down. +5. **One object model everywhere:** object definitions live in `@objectstack/spec` (zero runtime deps); gateway and backend import the *same* `*.object.ts` and call the *same* engine — only their installed capability set differs. +6. **Formula stays in core** (used by validation, L1 hooks, and seed loader — not just formula fields). `cel-js` is small and is not the complexity driver; the 268KB `protocol.ts` is. + +## Context + +### The conflated mental model + +| Layer | What it is | Where it lives | Complexity | +|---|---|---|---| +| **SQL generation** | dialect, pooling, binding | `knex` (in `driver-sql`) | outsourced — not ours | +| **Data engine ("the ORM")** | QueryAST → driver; CRUD; hooks; validation; formula | `engine.ts`, `registry.ts`, `validation/`, `hook-wrappers.ts` | small, *should be* stable — what the gateway wants | +| **Application platform** | RLS, sharing, field perms, protocol/metadata mgmt, federation, REST/GraphQL, UI gen | `plugin-security`, `plugin-sharing`, `protocol.ts`, rest, … | large — **the product value** | + +The "it keeps getting more complex" feeling maps to row 3. The cure is an **explicit, enforced boundary** between rows 2 and 3 — not abandoning the engine, and not (yet) physically separating it. + +### What the gateway needs + +`formula` + `hooks` wanted; permissions/RLS **optional**, assembled per host by third parties; same library; ideally the same object definitions; multi-backend; future merge into the platform. + +## Decision data (the deciding measurement) + +Last month, commits touching `engine.ts` or `registry.ts`: + +``` +total: 50 +also touched files OUTSIDE the objectql package: 44 (88%) +isolated to the objectql package: 6 (12%) +``` + +Representative titles are inherently cross-package atomic commits: `feat(objectql,spec)`, `feat(spec,objectql,platform-objects)`, `feat(driver-sql,objectql)`, `feat(automation,objectql)`, `feat(objectql,webhooks)`. **The engine is in active co-evolution with spec/security/analytics/automation/driver-sql/webhooks — its API cannot be frozen, and therefore cannot be cheaply separated, right now.** This is *the* fact that orders the decision below. + +## Decision + +### D1 — Lean entry via a `./core` subpath, not a package split [new] + +Add `@objectstack/objectql/core` exporting only the engine surface (engine + registry + hooks + validation + formula + driver contract). The gateway imports `@objectstack/objectql/core`; `@objectstack/objectql` (root) keeps re-exporting everything for backward compatibility. No new package, no breaking change. Tree-shaking (ESM + `sideEffects:false`) keeps `protocol.ts` out of the core entry's graph; in a Node host the unused file is effectively free. + +### D2 — Enforce the engine↔platform boundary with a ratchet [new — the keystone] + +A CI test (custom import-scan or dependency-cruiser) asserts core modules (`engine`, `registry`, `hooks`, `validation`) **never import** `protocol` / `plugin` / `kernel`. Today this invariant holds only *accidentally* (`engine.ts` does not import `protocol.ts`; protocol imports the engine *type* only). The ratchet makes it durable and — crucially — creates **backpressure against the engine absorbing platform concerns**, which is the root cause of the complexity that prompted this ADR. Fits the team's existing ratchet culture (liveness / api-surface / authz-matrix). + +### D3 — Capability + profile contract for graceful degradation [new, extends ADR-0066] + +The hard case: the same `account.object.ts` declares RLS / field permissions; the backend enforces them; the gateway deliberately does not. + +- **Required capabilities are *derived*** from what the object declares (RLS block → `rls`; object/field `requiredPermissions` → `permissions`; sharing rules → `sharing`). No hand-maintained list. +- Plugins **declare what they provide**: a `provides: ['rls', ...]` field (today they carry only `name`/`dependencies`). +- The host **validates derived-vs-provided capabilities at boot**. **Default profile = fail-closed** (a derived authz capability with no provider → refuse to load; consistent with ADR-0066). +- A host may set `profile: 'trusted' | 'internal'` to legitimately run without the authz surface. **Two-key**: build-time absence of the authz plugin (no enforcement code to bypass) *plus* an explicit runtime `profile:'trusted'` assertion the boot validator requires. Never a single runtime boolean that disables an *installed* enforcement; production refuses `trusted` unless env-allowlisted. This lifts ADR-0066's posture-gated bypass from the *runtime* to the *assembly* layer. + +### D4 — Formula stays in core (NOT optional) [ruled] + +`@objectstack/formula` is used in the core path in four places — `engine.ts` (formula fields), `validation/rule-validator.ts` (conditional/format validations), `hook-wrappers.ts` (L1 expression hooks), `seed-loader.ts` (dynamic seed values). It is therefore a hard dependency of the core surface. `cel-js` is small and not the complexity driver. + +### D5 — One object model, authored once [existing → ruled] + +Object definitions stay in `@objectstack/spec` (zero runtime deps). Gateway (`/core`) and backend (`objectql`) import the **same** `*.object.ts` and register via the same `registry.registerObject(...)`. + +### D6 — Capabilities attach uniformly across assembly modes [existing → ruled] + +RLS/permissions/sharing already attach as engine **middleware** (`ql.registerMiddleware(...)`) plus optional hooks — kernel-independent. Ratify `registerMiddleware` + hooks as the **public, supported** capability-attachment API so third parties compose plugins in a lean host without the heavy kernel. + +### D7 — Repo extraction is a future, trigger-gated phase [new — sequencing] + +A standalone `objectql` repo (sibling-link, no publish, the `objectui` model; consumed by both framework and objectbase; carrying `spec`/object-definition root) is the right *destination* but is **gated on a measured trigger**: the cross-package ratio of engine/registry commits (currently **88%**) falling to a low, stable level. Rationale: `objectui` separates cleanly because it is a stable *downstream leaf*; the engine is an *upstream foundation* with 88% cross-cutting churn — separating it now reproduces the maintenance overhead that caused the earlier merge-back. The D1/D2/D3 groundwork makes the eventual extraction mechanical. + +## What stays unchanged + +- knex remains the SQL backend under `driver-sql`. No driver rewrite. Drivers stay independent packages; multi-backend unchanged. +- `engine.ts` has no dependency on `protocol.ts`; RLS/sharing/permissions remain middleware/hook plugins; the data path is untouched. +- ADR-0066 secure-by-default semantics preserved (and become the *default* profile). + +## Feasibility (verified against current source) + +| Seam | Status | Evidence | +|---|---|---| +| Engine runs standalone | ✅ | `new ObjectQL()` + `registerDriver` + `registry.registerObject(inline)` = full CRUD, no kernel | +| Core surface separable as subpath | ✅ | `engine.ts` never imports `protocol.ts`; protocol holds only the engine *type* | +| Capabilities detachable | ✅ | `plugin-security`/`plugin-sharing` attach via `ql.registerMiddleware(...)` + hooks; not installing = lean host | +| Splitting protocol saves deps | ❌ | `protocol.ts` adds no third-party/workspace dep; the only external import in engine+registry is `node:async_hooks` | +| Engine API freezable now | ❌ | 88% of engine/registry commits last month were cross-package (see *Decision data*) | +| Profile precedent exists | ✅ | ADR-0066 schema posture cached in `getObjectSecurityMeta()` and enforced in security middleware | + +**Conclusion: the lean value is achievable today *additively* (subpath + example) plus discipline (ratchet + capability contract). Physical separation (package or repo) is premature given 88% cross-package churn and a ≈0 dependency win.** + +## Worked example + +```ts +// Gateway (objectbase) — lean subpath entry, trusted profile, no authz surface +import { ObjectQL } from '@objectstack/objectql/core'; +import { MemoryDriver } from '@objectstack/driver-memory'; +import { Account } from '@shared/objects/account.object'; // SAME definition as the backend + +const engine = new ObjectQL({ profile: 'trusted' }); // explicit, auditable bypass +engine.registerDriver(new MemoryDriver(), true); +await engine.init(); +engine.registry.registerObject(Account); // declares RLS — tolerated only under 'trusted' +await engine.find('account', { where: { active: true } }); + +// Full backend — same object, full capability set, default (fail-closed) profile +import { createObjectQLKernel } from '@objectstack/objectql'; +import { SecurityPlugin } from '@objectstack/plugin-security'; +import { SharingPlugin } from '@objectstack/plugin-sharing'; +// boots protocol + security + sharing; Account's RLS is enforced +``` + +## Phasing + +**Now (additive, non-breaking):** +1. **P1 — `./core` subpath export** (`package.json` exports + a `src/core.ts` barrel that omits protocol/plugin/kernel). +2. **P2 — Boundary ratchet** test (core must not import protocol/plugin/kernel) + an export-surface ratchet for the core barrel. +3. **P3 — Capability/profile contract** (plugins `provides`; derive required caps; boot-time validator; `profile` flag; default fail-closed; prod env-gates `trusted`). *The only substantial new code.* +4. **P4 — Standalone embed example + smoke** under `examples/`. + +**Later (trigger-gated, D7):** extract standalone `objectql` repo when the cross-package commit ratio drops. + +## Consequences + +- **+** Gateway unblocked; one library, one object model; safe optional permissions; the boundary that prevents future sprawl is enforced — all without a breaking change. +- **+** Entirely additive now; reversible; keeps atomic cross-package commits while the engine is still co-evolving. +- **−** Subpath + ratchet are weaker isolation than a package boundary (a ratchet can be mis-configured). Accepted deliberately: the package/repo boundary is deferred to D7, not forgone. +- **Risk (highest)**: capability/profile under-specified → silent authz bypass. Mitigation: default fail-closed; `trusted` two-key + prod env-gate; wire into authz-matrix ratchet. +- **Risk**: core-barrel drift (something pulls protocol into the core entry). Mitigation: P2 ratchets. + +## Open questions (with recommended positions) + +1. **Naming** — resolved by D1: no `-core` package; the lean entry is the `@objectstack/objectql/core` **subpath**; `@objectstack/objectql` stays canonical (npm norm: `react`/`react-dom`). +2. **Capability granularity** — *derive* from object declarations (do not overload per-user `requiredPermissions` for per-deployment host capability); keep an optional deployment-level explicit override as fallback. +3. **`trusted` profile** — two-key: build-time plugin absence + explicit runtime assertion; prod env-allowlisted. Never a lone runtime boolean disabling installed enforcement. +4. **Does the gateway need protocol-lite?** — no: compile-time `registry.registerObject(import)` suffices and the registry already lives in the core surface; full protocol (sys_metadata, draft/publish, ownership) stays out of the gateway. +5. **D7 trigger threshold** — what cross-package ratio (currently 88%) and over what window signals "stable enough to extract"? To be set when first reviewed; track the metric in CI. From 0979e30b207ffa2fa2ac5e0da86d941ccaf91b6a Mon Sep 17 00:00:00 2001 From: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com> Date: Sun, 28 Jun 2026 13:45:51 +0800 Subject: [PATCH 2/2] =?UTF-8?q?docs(adr):=20ADR-0076=20rev.4=20=E2=80=94?= =?UTF-8?q?=20relocate=20protocol=20to=20@objectstack/metadata-protocol=20?= =?UTF-8?q?(it's=20metadata-domain,=20mis-located=20in=20objectql)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrects the centerpiece. protocol.ts implements ObjectStackProtocol (contract in spec), manages sys_metadata/locks/commits/ownership — metadata-domain work that only uses the engine as storage via a 5-method SysMetadataEngine interface. Measured: protocol.ts co-changes with engine.ts only 6% of the time vs 43% with metadata-core/metadata/spec; move blast radius = 2 source files (plugin.ts, index.ts). So the right "now" move is relocating protocol into a new @objectstack/metadata-protocol package — not the rev.3 ./core subpath. objectql becomes a lean data engine by construction (268KB leaves the package); gateway depends on it with no protocol. Boundary ratchet keeps it pure. Engine repo-split stays a separate, trigger-gated (88% cross-package) future phase. Co-Authored-By: Claude Opus 4.8 --- docs/adr/0076-objectql-core-tiering.md | 154 +++++++++++-------------- 1 file changed, 66 insertions(+), 88 deletions(-) diff --git a/docs/adr/0076-objectql-core-tiering.md b/docs/adr/0076-objectql-core-tiering.md index 14942d8807..2a2b520670 100644 --- a/docs/adr/0076-objectql-core-tiering.md +++ b/docs/adr/0076-objectql-core-tiering.md @@ -1,150 +1,128 @@ -# ADR-0076: objectql layering — a lean embeddable surface, an enforced engine↔platform boundary, and a trigger-gated path to a standalone repo +# ADR-0076: objectql is the data engine — relocate metadata management (protocol) out of it; enforce the boundary; defer the engine repo-split -**Status**: Proposed (2026-06-28, rev. 3 — data-driven) — feasibility + value assessment for the next major (v12) +**Status**: Proposed (2026-06-28, rev. 4 — data-driven; centerpiece corrected to protocol relocation) — v12 assessment **Deciders**: ObjectStack Protocol Architects -**Builds on**: [ADR-0025](./0025-plugin-package-distribution.md) (plugin package distribution), [ADR-0033](./0033-ai-assisted-metadata-authoring.md) (open-core boundary), [ADR-0048](./0048-cross-package-metadata-collision.md) (package id is the addressing unit), [ADR-0066](./0066-unified-authorization-model.md) (secure-by-default `access`, `requiredPermissions`, posture-gated RLS bypass), [ADR-0070](./0070-package-first-authoring.md) (package-first authoring) -**Consumers**: `@objectstack/objectql` (gains a `./core` subpath entry; no package split), `@objectstack/formula`, `@objectstack/plugin-security`, `@objectstack/plugin-sharing`, `@objectstack/spec` (shared object model), the host boot path (`createObjectQLKernel`), and out-of-tree embedders — notably `../objectbase` (its `gateway`). +**Builds on**: [ADR-0005](./0005-metadata-customization-overlay.md) (sys_metadata overlay substrate), [ADR-0025](./0025-plugin-package-distribution.md) (plugin package distribution), [ADR-0033](./0033-ai-assisted-metadata-authoring.md) (open-core boundary), [ADR-0048](./0048-cross-package-metadata-collision.md) (package id is the addressing unit), [ADR-0066](./0066-unified-authorization-model.md) (secure-by-default, posture-gated bypass) +**Consumers**: **new** `@objectstack/metadata-protocol` (receives `protocol` + `sys-metadata-repository` + `metadata-diagnostics`), `@objectstack/objectql` (loses protocol → becomes a lean data engine; keeps a back-compat re-export), `@objectstack/metadata-core` (gains the `SysMetadataEngine` interface), `@objectstack/plugin-security`, `@objectstack/plugin-sharing`, `@objectstack/spec`, and out-of-tree embedders — notably `../objectbase` (its `gateway`). -**Premise**: objectql was conceived as "a metadata-driven ORM that replaces TypeORM." It has outgrown that framing: it is a metadata-driven **data engine** that sits *on top of* knex (`driver-sql` depends on `knex@^3.2.10`) with an application platform layered above (RLS, sharing, field permissions, formula, protocol/metadata management, federation). The "ORM ambition" is not the source of complexity; the platform above the engine is — and that platform is the product value. +**Premise**: objectql was conceived as "a metadata-driven ORM that replaces TypeORM." It has outgrown that framing: it is a metadata-driven **data engine** sitting on top of knex (`driver-sql` → `knex@^3.2.10`). The complexity the team feels is not the engine; it is everything layered above and *beside* it — including `protocol.ts` (268KB of sys_metadata management / draft-publish / package ownership / locks) which lives *inside the objectql package by historical accident*, not by design. -> **Trigger**: `../objectbase` (its `gateway` — a thin, latency-sensitive host for some core algorithms) wants to use *only* the objectql engine without dragging in the whole platform; an agent inspecting the source recommended dropping objectql for raw knex. Investigation showed this is **half perception** (the engine already runs standalone; `protocol.ts` — 268KB — is never loaded for basic CRUD) and **half a productization gap** (no lean entry point, no standalone example). The team previously ran objectql as a *separate repo* (`@objectql/core`) and merged it back due to maintenance overhead, but notes `../objectui` proves a viable separate-repo model (sibling checkout, local link, no publishing). This ADR decides the layering for v12 **based on measured engine↔platform coupling**, not on aesthetics. +> **Trigger**: `../objectbase`'s `gateway` wants to embed only the objectql engine without the platform. While scoping that, a sharper observation surfaced: **`protocol.ts` is metadata-domain code mis-located in the data-engine package.** Measurement confirms it (below). Relocating it — not a subpath, not a repo split — is the correct, cheap, *now* move that makes objectql lean by construction. --- ## TL;DR -1. **knex is not an ORM; it is a SQL query builder — and objectql already uses it.** "Use knex instead of objectql" is a category error; objectql is a *consumer* of knex, not a competitor. Retreating to TypeORM/knex forfeits the one-library / one-object-model goal and eliminates none of the felt complexity. -2. **Splitting protocol into its own package saves ≈ 0 dependencies** (verified: `protocol.ts` adds no third-party or workspace dep; engine+registry's only external import is `node:async_hooks`). The footprint argument for a package split does not hold. -3. **The engine API is NOT stable enough to freeze or separate today.** Measured over the last month: of 50 commits touching `engine.ts`/`registry.ts`, **44 (88%) also changed files outside the objectql package** — high-frequency cross-package co-evolution. Separating now would convert ~44 atomic commits/month into two-repo coordinated PRs. -4. **Decision (sequenced):** *Now* — stay in the monorepo; ship the lean value **additively** (a `./core` subpath entry, a standalone example) and **impose discipline** (an engine↔platform boundary ratchet + a capability/profile contract). *Later* — when the cross-package ratio falls (engine stabilizes), extract a standalone `objectql` repo on the `objectui` model. The boundary ratchet is the forcing function that drives that ratio down. -5. **One object model everywhere:** object definitions live in `@objectstack/spec` (zero runtime deps); gateway and backend import the *same* `*.object.ts` and call the *same* engine — only their installed capability set differs. -6. **Formula stays in core** (used by validation, L1 hooks, and seed loader — not just formula fields). `cel-js` is small and is not the complexity driver; the 268KB `protocol.ts` is. +1. **knex is a SQL query builder objectql already uses**, not an ORM to switch to. Retreating forfeits the one-library / one-object-model goal and removes none of the felt complexity. +2. **`protocol.ts` does not belong in the objectql package.** It implements `ObjectStackProtocol` (the contract lives in `@objectstack/spec`), manages sys_metadata, locks, commits, package ownership — pure metadata-domain work. It uses the engine only as storage, through a **5-method `SysMetadataEngine` interface** injected at runtime. +3. **Measured coupling makes the relocation cheap.** Last month: `protocol.ts` changed 47×; only **3 (6%)** also touched `engine.ts`, while **20 (43%)** touched `metadata-core`/`metadata`/`spec`. Blast radius of the move is **2 source files** (`plugin.ts` wiring, `index.ts` re-export). Its two helpers (`sys-metadata-repository`, `metadata-diagnostics`) depend only on `metadata-core`/`spec`/`types` and move with it. +4. **Decision (now):** relocate `protocol` + helpers into a **new `@objectstack/metadata-protocol`** package between `metadata-core` (pure contracts) and `metadata` (plugin). objectql becomes a lean data engine **by construction** — the 268KB genuinely leaves the package; the gateway depends on objectql with no protocol. Add a **boundary ratchet** so the engine stays pure. Add the **capability/profile** contract for optional permissions. Formula stays in core. +5. **Decision (later, separate concern):** extracting the *engine itself* into a standalone repo remains **trigger-gated** on its cross-package commit ratio (currently **88%** for `engine.ts`/`registry.ts`). That is orthogonal to — and unblocked by — the protocol relocation. -## Context +## Context: the layers -### The conflated mental model - -| Layer | What it is | Where it lives | Complexity | +| Layer | What it is | Where it lives today | Where it should live | |---|---|---|---| -| **SQL generation** | dialect, pooling, binding | `knex` (in `driver-sql`) | outsourced — not ours | -| **Data engine ("the ORM")** | QueryAST → driver; CRUD; hooks; validation; formula | `engine.ts`, `registry.ts`, `validation/`, `hook-wrappers.ts` | small, *should be* stable — what the gateway wants | -| **Application platform** | RLS, sharing, field perms, protocol/metadata mgmt, federation, REST/GraphQL, UI gen | `plugin-security`, `plugin-sharing`, `protocol.ts`, rest, … | large — **the product value** | - -The "it keeps getting more complex" feeling maps to row 3. The cure is an **explicit, enforced boundary** between rows 2 and 3 — not abandoning the engine, and not (yet) physically separating it. - -### What the gateway needs +| SQL generation | dialect/pool/binding | `knex` (in `driver-sql`) | unchanged | +| **Data engine** | QueryAST→driver; CRUD; hooks; validation; formula | `objectql/src/{engine,registry,validation,hook-wrappers}` | unchanged (this is `@objectstack/objectql`) | +| **Metadata management** | `ObjectStackProtocol` impl; sys_metadata CRUD; draft/publish; locks; ownership; diagnostics | **`objectql/src/protocol.ts` (mis-located)** + `metadata-core` + `metadata` plugin | **new `@objectstack/metadata-protocol`** | +| Governance | RLS, sharing, field perms | `plugin-security`, `plugin-sharing` (already separate) | unchanged | -`formula` + `hooks` wanted; permissions/RLS **optional**, assembled per host by third parties; same library; ideally the same object definitions; multi-backend; future merge into the platform. +## Decision data -## Decision data (the deciding measurement) - -Last month, commits touching `engine.ts` or `registry.ts`: - -``` -total: 50 -also touched files OUTSIDE the objectql package: 44 (88%) -isolated to the objectql package: 6 (12%) ``` +protocol.ts commits last month: 47 + also touched engine.ts: 3 (6%) ← protocol↔engine ≈ decoupled + also touched metadata-core/metadata/spec: 20 (43%) ← its real neighbors +import sites of ObjectStackProtocolImplementation (source): plugin.ts, index.ts (+ co-located helpers) +engine surface protocol needs: SysMetadataEngine = { find, findOne, insert, update, delete, transaction? } (injected) -Representative titles are inherently cross-package atomic commits: `feat(objectql,spec)`, `feat(spec,objectql,platform-objects)`, `feat(driver-sql,objectql)`, `feat(automation,objectql)`, `feat(objectql,webhooks)`. **The engine is in active co-evolution with spec/security/analytics/automation/driver-sql/webhooks — its API cannot be frozen, and therefore cannot be cheaply separated, right now.** This is *the* fact that orders the decision below. +(for contrast) engine.ts/registry.ts commits last month: 50, of which 44 (88%) cross-package → engine itself NOT separable yet +``` ## Decision -### D1 — Lean entry via a `./core` subpath, not a package split [new] - -Add `@objectstack/objectql/core` exporting only the engine surface (engine + registry + hooks + validation + formula + driver contract). The gateway imports `@objectstack/objectql/core`; `@objectstack/objectql` (root) keeps re-exporting everything for backward compatibility. No new package, no breaking change. Tree-shaking (ESM + `sideEffects:false`) keeps `protocol.ts` out of the core entry's graph; in a Node host the unused file is effectively free. +### D1 — Relocate protocol out of objectql into `@objectstack/metadata-protocol` [new — the centerpiece] -### D2 — Enforce the engine↔platform boundary with a ratchet [new — the keystone] +- Move `protocol.ts`, `sys-metadata-repository.ts`, `metadata-diagnostics.ts` from `@objectstack/objectql` into a **new `@objectstack/metadata-protocol`** package. +- Move the `SysMetadataEngine` interface into `@objectstack/metadata-core`; `metadata-protocol` depends on `metadata-core` + `spec` + `core` (for `IDataEngine`) + `types` — **not on the objectql package**. The concrete engine is injected at runtime (as today). +- `@objectstack/objectql` re-exports `ObjectStackProtocolImplementation` from the new package for back-compat (the only two source importers — `plugin.ts`, `index.ts` — are updated; `plugin.ts` keeps wiring it, now imported from `metadata-protocol`). +- Result: objectql is a lean data engine **by construction** — 268KB of metadata logic physically leaves the engine package. The gateway depends on `@objectstack/objectql` and never pulls protocol. (This **replaces** the rev.3 `./core` subpath proposal, which was a workaround for the false premise that protocol must stay in objectql. A subpath remains an optional later polish to also exclude the small `ObjectQLPlugin`/`kernel-factory` glue.) -A CI test (custom import-scan or dependency-cruiser) asserts core modules (`engine`, `registry`, `hooks`, `validation`) **never import** `protocol` / `plugin` / `kernel`. Today this invariant holds only *accidentally* (`engine.ts` does not import `protocol.ts`; protocol imports the engine *type* only). The ratchet makes it durable and — crucially — creates **backpressure against the engine absorbing platform concerns**, which is the root cause of the complexity that prompted this ADR. Fits the team's existing ratchet culture (liveness / api-surface / authz-matrix). +### D2 — Boundary ratchet keeps the engine pure [new — keystone] -### D3 — Capability + profile contract for graceful degradation [new, extends ADR-0066] +A CI test asserts `@objectstack/objectql` does **not** import `@objectstack/metadata-protocol` / `plugin` / `kernel` (the dependency points one way: `metadata-protocol → objectql-public-interface`, never back). This makes the relocation durable and creates backpressure against the engine re-absorbing metadata/platform concerns — the root cause of the original complexity. Fits the team's existing ratchet culture (liveness / api-surface / authz-matrix). -The hard case: the same `account.object.ts` declares RLS / field permissions; the backend enforces them; the gateway deliberately does not. +### D3 — Capability + profile contract for optional governance [new, extends ADR-0066] -- **Required capabilities are *derived*** from what the object declares (RLS block → `rls`; object/field `requiredPermissions` → `permissions`; sharing rules → `sharing`). No hand-maintained list. -- Plugins **declare what they provide**: a `provides: ['rls', ...]` field (today they carry only `name`/`dependencies`). -- The host **validates derived-vs-provided capabilities at boot**. **Default profile = fail-closed** (a derived authz capability with no provider → refuse to load; consistent with ADR-0066). -- A host may set `profile: 'trusted' | 'internal'` to legitimately run without the authz surface. **Two-key**: build-time absence of the authz plugin (no enforcement code to bypass) *plus* an explicit runtime `profile:'trusted'` assertion the boot validator requires. Never a single runtime boolean that disables an *installed* enforcement; production refuses `trusted` unless env-allowlisted. This lifts ADR-0066's posture-gated bypass from the *runtime* to the *assembly* layer. +Required capabilities are **derived** from object declarations (RLS block → `rls`; `requiredPermissions` → `permissions`; sharing → `sharing`). Plugins declare `provides: [...]`. The host validates at boot; **default fail-closed**. A host may run without the authz surface via an explicit `profile: 'trusted' | 'internal'` — **two-key**: build-time plugin absence + explicit runtime assertion; production env-gated. Lifts ADR-0066 posture-gated bypass to the assembly layer. -### D4 — Formula stays in core (NOT optional) [ruled] +### D4 — Formula stays in core [ruled] -`@objectstack/formula` is used in the core path in four places — `engine.ts` (formula fields), `validation/rule-validator.ts` (conditional/format validations), `hook-wrappers.ts` (L1 expression hooks), `seed-loader.ts` (dynamic seed values). It is therefore a hard dependency of the core surface. `cel-js` is small and not the complexity driver. +`@objectstack/formula` is used by `engine.ts` (formula fields), `validation/rule-validator.ts`, `hook-wrappers.ts`, `seed-loader.ts` — a hard dependency of the engine. `cel-js` is small and not the complexity driver. ### D5 — One object model, authored once [existing → ruled] -Object definitions stay in `@objectstack/spec` (zero runtime deps). Gateway (`/core`) and backend (`objectql`) import the **same** `*.object.ts` and register via the same `registry.registerObject(...)`. +Definitions live in `@objectstack/spec` (zero runtime deps); gateway and backend import the same `*.object.ts` and call the same engine; only the installed capability set differs. ### D6 — Capabilities attach uniformly across assembly modes [existing → ruled] -RLS/permissions/sharing already attach as engine **middleware** (`ql.registerMiddleware(...)`) plus optional hooks — kernel-independent. Ratify `registerMiddleware` + hooks as the **public, supported** capability-attachment API so third parties compose plugins in a lean host without the heavy kernel. - -### D7 — Repo extraction is a future, trigger-gated phase [new — sequencing] +RLS/permissions/sharing attach via engine **middleware** (`ql.registerMiddleware(...)`) + hooks — kernel-independent. Ratify this as the public, supported capability-attachment API. -A standalone `objectql` repo (sibling-link, no publish, the `objectui` model; consumed by both framework and objectbase; carrying `spec`/object-definition root) is the right *destination* but is **gated on a measured trigger**: the cross-package ratio of engine/registry commits (currently **88%**) falling to a low, stable level. Rationale: `objectui` separates cleanly because it is a stable *downstream leaf*; the engine is an *upstream foundation* with 88% cross-cutting churn — separating it now reproduces the maintenance overhead that caused the earlier merge-back. The D1/D2/D3 groundwork makes the eventual extraction mechanical. +### D7 — Engine repo-split is a separate, trigger-gated future phase [sequencing] -## What stays unchanged - -- knex remains the SQL backend under `driver-sql`. No driver rewrite. Drivers stay independent packages; multi-backend unchanged. -- `engine.ts` has no dependency on `protocol.ts`; RLS/sharing/permissions remain middleware/hook plugins; the data path is untouched. -- ADR-0066 secure-by-default semantics preserved (and become the *default* profile). +Extracting the *engine* into a standalone repo (the `objectui` sibling-link, no-publish model) is gated on the engine's cross-package commit ratio (currently **88%**) falling. This ADR's relocation (D1) is independent of and unblocked by D7. `objectui` separates cleanly because it is a stable downstream leaf; the engine is an upstream foundation still in 88% cross-cutting co-evolution — separating it now would reproduce the overhead that caused the earlier `@objectql/core` merge-back. ## Feasibility (verified against current source) -| Seam | Status | Evidence | +| Claim | Status | Evidence | |---|---|---| -| Engine runs standalone | ✅ | `new ObjectQL()` + `registerDriver` + `registry.registerObject(inline)` = full CRUD, no kernel | -| Core surface separable as subpath | ✅ | `engine.ts` never imports `protocol.ts`; protocol holds only the engine *type* | -| Capabilities detachable | ✅ | `plugin-security`/`plugin-sharing` attach via `ql.registerMiddleware(...)` + hooks; not installing = lean host | -| Splitting protocol saves deps | ❌ | `protocol.ts` adds no third-party/workspace dep; the only external import in engine+registry is `node:async_hooks` | -| Engine API freezable now | ❌ | 88% of engine/registry commits last month were cross-package (see *Decision data*) | -| Profile precedent exists | ✅ | ADR-0066 schema posture cached in `getObjectSecurityMeta()` and enforced in security middleware | - -**Conclusion: the lean value is achievable today *additively* (subpath + example) plus discipline (ratchet + capability contract). Physical separation (package or repo) is premature given 88% cross-package churn and a ≈0 dependency win.** +| protocol is metadata-domain, not engine | ✅ | implements `ObjectStackProtocol` (`spec/api`); manages sys_metadata/locks/commits/ownership | +| protocol decoupled from the engine | ✅ | 6% commit co-change with `engine.ts`; needs only the 5-method `SysMetadataEngine` (injected) | +| relocation blast radius is tiny | ✅ | only `plugin.ts` + `index.ts` import the impl; helpers depend only on metadata-core/spec/types | +| splitting protocol saves engine deps | ✅ (for the engine package) | protocol's deps (spec/*, core, metadata-core) leave with it; engine keeps only its own | +| engine itself separable now | ❌ | 88% of engine/registry commits cross-package → D7 deferred | ## Worked example ```ts -// Gateway (objectbase) — lean subpath entry, trusted profile, no authz surface -import { ObjectQL } from '@objectstack/objectql/core'; +// Gateway (objectbase) — depends on the (now lean) engine; protocol is simply not in the package +import { ObjectQL } from '@objectstack/objectql'; import { MemoryDriver } from '@objectstack/driver-memory'; import { Account } from '@shared/objects/account.object'; // SAME definition as the backend -const engine = new ObjectQL({ profile: 'trusted' }); // explicit, auditable bypass +const engine = new ObjectQL({ profile: 'trusted' }); engine.registerDriver(new MemoryDriver(), true); await engine.init(); -engine.registry.registerObject(Account); // declares RLS — tolerated only under 'trusted' +engine.registry.registerObject(Account); await engine.find('account', { where: { active: true } }); -// Full backend — same object, full capability set, default (fail-closed) profile +// Backend — adds metadata management + governance on top of the same engine import { createObjectQLKernel } from '@objectstack/objectql'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; import { SecurityPlugin } from '@objectstack/plugin-security'; -import { SharingPlugin } from '@objectstack/plugin-sharing'; -// boots protocol + security + sharing; Account's RLS is enforced ``` ## Phasing -**Now (additive, non-breaking):** -1. **P1 — `./core` subpath export** (`package.json` exports + a `src/core.ts` barrel that omits protocol/plugin/kernel). -2. **P2 — Boundary ratchet** test (core must not import protocol/plugin/kernel) + an export-surface ratchet for the core barrel. -3. **P3 — Capability/profile contract** (plugins `provides`; derive required caps; boot-time validator; `profile` flag; default fail-closed; prod env-gates `trusted`). *The only substantial new code.* +1. **P1 — Relocate protocol** to `@objectstack/metadata-protocol` (move 3 files; move `SysMetadataEngine` into metadata-core; update `plugin.ts` import + `index.ts` re-export). Behavior unchanged. +2. **P2 — Boundary ratchet** (objectql must not import metadata-protocol/plugin/kernel) + export-surface ratchet. +3. **P3 — Capability/profile contract** (the only substantial new code). 4. **P4 — Standalone embed example + smoke** under `examples/`. - -**Later (trigger-gated, D7):** extract standalone `objectql` repo when the cross-package commit ratio drops. +5. **Later (D7)** — engine repo-split when the cross-package ratio drops. ## Consequences -- **+** Gateway unblocked; one library, one object model; safe optional permissions; the boundary that prevents future sprawl is enforced — all without a breaking change. -- **+** Entirely additive now; reversible; keeps atomic cross-package commits while the engine is still co-evolving. -- **−** Subpath + ratchet are weaker isolation than a package boundary (a ratchet can be mis-configured). Accepted deliberately: the package/repo boundary is deferred to D7, not forgone. -- **Risk (highest)**: capability/profile under-specified → silent authz bypass. Mitigation: default fail-closed; `trusted` two-key + prod env-gate; wire into authz-matrix ratchet. -- **Risk**: core-barrel drift (something pulls protocol into the core entry). Mitigation: P2 ratchets. +- **+** objectql is lean by construction; protocol lands in its proper domain; real enforced package boundary (not a convention); gateway unblocked; all without breaking downstream (re-export shim). +- **+** Cheap and low-risk now (6% coupling, 2-file blast radius, narrow injected interface). +- **−** One new package in the fixed version group; a major-version bump for the moved export (mitigated by the re-export shim). +- **Risk (highest)**: capability/profile under-specified → silent authz bypass. Mitigation: default fail-closed; `trusted` two-key + prod env-gate; authz-matrix ratchet. +- **Risk**: tsup DTS stricter than tsc / turbo build-order during the package move. Mitigation: incremental, watch DTS. ## Open questions (with recommended positions) -1. **Naming** — resolved by D1: no `-core` package; the lean entry is the `@objectstack/objectql/core` **subpath**; `@objectstack/objectql` stays canonical (npm norm: `react`/`react-dom`). -2. **Capability granularity** — *derive* from object declarations (do not overload per-user `requiredPermissions` for per-deployment host capability); keep an optional deployment-level explicit override as fallback. -3. **`trusted` profile** — two-key: build-time plugin absence + explicit runtime assertion; prod env-allowlisted. Never a lone runtime boolean disabling installed enforcement. -4. **Does the gateway need protocol-lite?** — no: compile-time `registry.registerObject(import)` suffices and the registry already lives in the core surface; full protocol (sys_metadata, draft/publish, ownership) stays out of the gateway. -5. **D7 trigger threshold** — what cross-package ratio (currently 88%) and over what window signals "stable enough to extract"? To be set when first reviewed; track the metric in CI. +1. **protocol's new home** — resolved: a dedicated **`@objectstack/metadata-protocol`** between `metadata-core` (kept pure) and the `metadata` plugin (avoids dragging chokidar/fs deps onto pure runtime protocol logic). +2. **`SysMetadataEngine` interface home** — recommend `@objectstack/metadata-core` (it is a contract; metadata-protocol depends on metadata-core anyway). +3. **Capability granularity** — derive from object declarations; do not overload per-user `requiredPermissions`. +4. **`trusted` profile** — two-key (build-time absence + explicit runtime assertion), prod env-allowlisted. +5. **D7 trigger threshold** — what cross-package ratio (from 88%) over what window signals "extract the engine"? Track in CI; set on first review.