Skip to content

docs(adr): ADR-0076 — relocate metadata protocol out of objectql (lean engine, enforced boundary, deferred repo-split) #2414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
os-zhuang merged 2 commits into from
Jun 28, 2026
Merged

docs(adr): ADR-0076 — relocate metadata protocol out of objectql (lean engine, enforced boundary, deferred repo-split) #2414

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b331165
docs(adr): ADR-0076 — objectql layering (lean subpath + enforced boun…
os-zhuang Jun 28, 2026
0979e30
docs(adr): ADR-0076 rev.4 — relocate protocol to @objectstack/metadat…
os-zhuang Jun 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 128 additions & 0 deletions docs/adr/0076-objectql-core-tiering.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# ADR-0076: objectql is the data engine — relocate metadata management (protocol) out of it; enforce the boundary; defer the engine repo-split

**Status**: Proposed (2026-06-28, rev. 4 — data-driven; centerpiece corrected to protocol relocation) — v12 assessment
**Deciders**: ObjectStack Protocol Architects
**Builds on**: [ADR-0005](./0005-metadata-customization-overlay.md) (sys_metadata overlay substrate), [ADR-0025](./0025-plugin-package-distribution.md) (plugin package distribution), [ADR-0033](./0033-ai-assisted-metadata-authoring.md) (open-core boundary), [ADR-0048](./0048-cross-package-metadata-collision.md) (package id is the addressing unit), [ADR-0066](./0066-unified-authorization-model.md) (secure-by-default, posture-gated bypass)
**Consumers**: **new** `@objectstack/metadata-protocol` (receives `protocol` + `sys-metadata-repository` + `metadata-diagnostics`), `@objectstack/objectql` (loses protocol → becomes a lean data engine; keeps a back-compat re-export), `@objectstack/metadata-core` (gains the `SysMetadataEngine` interface), `@objectstack/plugin-security`, `@objectstack/plugin-sharing`, `@objectstack/spec`, and out-of-tree embedders — notably `../objectbase` (its `gateway`).

**Premise**: objectql was conceived as "a metadata-driven ORM that replaces TypeORM." It has outgrown that framing: it is a metadata-driven **data engine** sitting on top of knex (`driver-sql` → `knex@^3.2.10`). The complexity the team feels is not the engine; it is everything layered above and *beside* it — including `protocol.ts` (268KB of sys_metadata management / draft-publish / package ownership / locks) which lives *inside the objectql package by historical accident*, not by design.

> **Trigger**: `../objectbase`'s `gateway` wants to embed only the objectql engine without the platform. While scoping that, a sharper observation surfaced: **`protocol.ts` is metadata-domain code mis-located in the data-engine package.** Measurement confirms it (below). Relocating it — not a subpath, not a repo split — is the correct, cheap, *now* move that makes objectql lean by construction.

---

## TL;DR

1. **knex is a SQL query builder objectql already uses**, not an ORM to switch to. Retreating forfeits the one-library / one-object-model goal and removes none of the felt complexity.
2. **`protocol.ts` does not belong in the objectql package.** It implements `ObjectStackProtocol` (the contract lives in `@objectstack/spec`), manages sys_metadata, locks, commits, package ownership — pure metadata-domain work. It uses the engine only as storage, through a **5-method `SysMetadataEngine` interface** injected at runtime.
3. **Measured coupling makes the relocation cheap.** Last month: `protocol.ts` changed 47×; only **3 (6%)** also touched `engine.ts`, while **20 (43%)** touched `metadata-core`/`metadata`/`spec`. Blast radius of the move is **2 source files** (`plugin.ts` wiring, `index.ts` re-export). Its two helpers (`sys-metadata-repository`, `metadata-diagnostics`) depend only on `metadata-core`/`spec`/`types` and move with it.
4. **Decision (now):** relocate `protocol` + helpers into a **new `@objectstack/metadata-protocol`** package between `metadata-core` (pure contracts) and `metadata` (plugin). objectql becomes a lean data engine **by construction** — the 268KB genuinely leaves the package; the gateway depends on objectql with no protocol. Add a **boundary ratchet** so the engine stays pure. Add the **capability/profile** contract for optional permissions. Formula stays in core.
5. **Decision (later, separate concern):** extracting the *engine itself* into a standalone repo remains **trigger-gated** on its cross-package commit ratio (currently **88%** for `engine.ts`/`registry.ts`). That is orthogonal to — and unblocked by — the protocol relocation.

## Context: the layers

| Layer | What it is | Where it lives today | Where it should live |
|---|---|---|---|
| SQL generation | dialect/pool/binding | `knex` (in `driver-sql`) | unchanged |
| **Data engine** | QueryAST→driver; CRUD; hooks; validation; formula | `objectql/src/{engine,registry,validation,hook-wrappers}` | unchanged (this is `@objectstack/objectql`) |
| **Metadata management** | `ObjectStackProtocol` impl; sys_metadata CRUD; draft/publish; locks; ownership; diagnostics | **`objectql/src/protocol.ts` (mis-located)** + `metadata-core` + `metadata` plugin | **new `@objectstack/metadata-protocol`** |
| Governance | RLS, sharing, field perms | `plugin-security`, `plugin-sharing` (already separate) | unchanged |

## Decision data

```
protocol.ts commits last month: 47
also touched engine.ts: 3 (6%) ← protocol↔engine ≈ decoupled
also touched metadata-core/metadata/spec: 20 (43%) ← its real neighbors
import sites of ObjectStackProtocolImplementation (source): plugin.ts, index.ts (+ co-located helpers)
engine surface protocol needs: SysMetadataEngine = { find, findOne, insert, update, delete, transaction? } (injected)

(for contrast) engine.ts/registry.ts commits last month: 50, of which 44 (88%) cross-package → engine itself NOT separable yet
```

## Decision

### D1 — Relocate protocol out of objectql into `@objectstack/metadata-protocol` [new — the centerpiece]

- Move `protocol.ts`, `sys-metadata-repository.ts`, `metadata-diagnostics.ts` from `@objectstack/objectql` into a **new `@objectstack/metadata-protocol`** package.
- Move the `SysMetadataEngine` interface into `@objectstack/metadata-core`; `metadata-protocol` depends on `metadata-core` + `spec` + `core` (for `IDataEngine`) + `types` — **not on the objectql package**. The concrete engine is injected at runtime (as today).
- `@objectstack/objectql` re-exports `ObjectStackProtocolImplementation` from the new package for back-compat (the only two source importers — `plugin.ts`, `index.ts` — are updated; `plugin.ts` keeps wiring it, now imported from `metadata-protocol`).
- Result: objectql is a lean data engine **by construction** — 268KB of metadata logic physically leaves the engine package. The gateway depends on `@objectstack/objectql` and never pulls protocol. (This **replaces** the rev.3 `./core` subpath proposal, which was a workaround for the false premise that protocol must stay in objectql. A subpath remains an optional later polish to also exclude the small `ObjectQLPlugin`/`kernel-factory` glue.)

### D2 — Boundary ratchet keeps the engine pure [new — keystone]

A CI test asserts `@objectstack/objectql` does **not** import `@objectstack/metadata-protocol` / `plugin` / `kernel` (the dependency points one way: `metadata-protocol → objectql-public-interface`, never back). This makes the relocation durable and creates backpressure against the engine re-absorbing metadata/platform concerns — the root cause of the original complexity. Fits the team's existing ratchet culture (liveness / api-surface / authz-matrix).

### D3 — Capability + profile contract for optional governance [new, extends ADR-0066]

Required capabilities are **derived** from object declarations (RLS block → `rls`; `requiredPermissions` → `permissions`; sharing → `sharing`). Plugins declare `provides: [...]`. The host validates at boot; **default fail-closed**. A host may run without the authz surface via an explicit `profile: 'trusted' | 'internal'` — **two-key**: build-time plugin absence + explicit runtime assertion; production env-gated. Lifts ADR-0066 posture-gated bypass to the assembly layer.

### D4 — Formula stays in core [ruled]

`@objectstack/formula` is used by `engine.ts` (formula fields), `validation/rule-validator.ts`, `hook-wrappers.ts`, `seed-loader.ts` — a hard dependency of the engine. `cel-js` is small and not the complexity driver.

### D5 — One object model, authored once [existing → ruled]

Definitions live in `@objectstack/spec` (zero runtime deps); gateway and backend import the same `*.object.ts` and call the same engine; only the installed capability set differs.

### D6 — Capabilities attach uniformly across assembly modes [existing → ruled]

RLS/permissions/sharing attach via engine **middleware** (`ql.registerMiddleware(...)`) + hooks — kernel-independent. Ratify this as the public, supported capability-attachment API.

### D7 — Engine repo-split is a separate, trigger-gated future phase [sequencing]

Extracting the *engine* into a standalone repo (the `objectui` sibling-link, no-publish model) is gated on the engine's cross-package commit ratio (currently **88%**) falling. This ADR's relocation (D1) is independent of and unblocked by D7. `objectui` separates cleanly because it is a stable downstream leaf; the engine is an upstream foundation still in 88% cross-cutting co-evolution — separating it now would reproduce the overhead that caused the earlier `@objectql/core` merge-back.

## Feasibility (verified against current source)

| Claim | Status | Evidence |
|---|---|---|
| protocol is metadata-domain, not engine | ✅ | implements `ObjectStackProtocol` (`spec/api`); manages sys_metadata/locks/commits/ownership |
| protocol decoupled from the engine | ✅ | 6% commit co-change with `engine.ts`; needs only the 5-method `SysMetadataEngine` (injected) |
| relocation blast radius is tiny | ✅ | only `plugin.ts` + `index.ts` import the impl; helpers depend only on metadata-core/spec/types |
| splitting protocol saves engine deps | ✅ (for the engine package) | protocol's deps (spec/*, core, metadata-core) leave with it; engine keeps only its own |
| engine itself separable now | ❌ | 88% of engine/registry commits cross-package → D7 deferred |

## Worked example

```ts
// Gateway (objectbase) — depends on the (now lean) engine; protocol is simply not in the package
import { ObjectQL } from '@objectstack/objectql';
import { MemoryDriver } from '@objectstack/driver-memory';
import { Account } from '@shared/objects/account.object'; // SAME definition as the backend

const engine = new ObjectQL({ profile: 'trusted' });
engine.registerDriver(new MemoryDriver(), true);
await engine.init();
engine.registry.registerObject(Account);
await engine.find('account', { where: { active: true } });

// Backend — adds metadata management + governance on top of the same engine
import { createObjectQLKernel } from '@objectstack/objectql';
import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol';
import { SecurityPlugin } from '@objectstack/plugin-security';
```

## Phasing

1. **P1 — Relocate protocol** to `@objectstack/metadata-protocol` (move 3 files; move `SysMetadataEngine` into metadata-core; update `plugin.ts` import + `index.ts` re-export). Behavior unchanged.
2. **P2 — Boundary ratchet** (objectql must not import metadata-protocol/plugin/kernel) + export-surface ratchet.
3. **P3 — Capability/profile contract** (the only substantial new code).
4. **P4 — Standalone embed example + smoke** under `examples/`.
5. **Later (D7)** — engine repo-split when the cross-package ratio drops.

## Consequences

- **+** objectql is lean by construction; protocol lands in its proper domain; real enforced package boundary (not a convention); gateway unblocked; all without breaking downstream (re-export shim).
- **+** Cheap and low-risk now (6% coupling, 2-file blast radius, narrow injected interface).
- **−** One new package in the fixed version group; a major-version bump for the moved export (mitigated by the re-export shim).
- **Risk (highest)**: capability/profile under-specified → silent authz bypass. Mitigation: default fail-closed; `trusted` two-key + prod env-gate; authz-matrix ratchet.
- **Risk**: tsup DTS stricter than tsc / turbo build-order during the package move. Mitigation: incremental, watch DTS.

## Open questions (with recommended positions)

1. **protocol's new home** — resolved: a dedicated **`@objectstack/metadata-protocol`** between `metadata-core` (kept pure) and the `metadata` plugin (avoids dragging chokidar/fs deps onto pure runtime protocol logic).
2. **`SysMetadataEngine` interface home** — recommend `@objectstack/metadata-core` (it is a contract; metadata-protocol depends on metadata-core anyway).
3. **Capability granularity** — derive from object declarations; do not overload per-user `requiredPermissions`.
4. **`trusted` profile** — two-key (build-time absence + explicit runtime assertion), prod env-allowlisted.
5. **D7 trigger threshold** — what cross-package ratio (from 88%) over what window signals "extract the engine"? Track in CI; set on first review.