From 131721d764c893c1813793f8295a582812d08cda Mon Sep 17 00:00:00 2001 From: g0 Date: Thu, 17 Sep 2026 02:40:38 +0800 Subject: [PATCH 1/3] docs: document timestamp column units (epoch ms) in ARCHITECTURE.md and fresh-install schema --- ARCHITECTURE.md | 31 +++++++++++++++++++ .../src/features/magic-context/storage-db.ts | 24 +++++++------- 2 files changed, 43 insertions(+), 12 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 9fbf502a8..9862f8851 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -147,6 +147,37 @@ Background maintenance (V2: per-task cron scheduling). A process-wide 15-min tim `storage-db.ts` creates the schema and runs versioned migrations (`migrations.ts`, currently v1–v82). `LATEST_SUPPORTED_VERSION` is a schema fence — it MUST be bumped with every new migration (a unit test asserts it equals the highest migration), and a stale value makes the DB refuse to open after the migration applies. Open-time migrations guard against live-process schema corruption via `enforceMigrationOnOpenGuard`, refusing to migrate when live OpenCode or Pi processes hold the database (scoped conservatively across all live instances for the default shared `context.db`, and scoped strictly to same-data-directory PIDs for non-default paths). Current-schema migration checks no longer acquire `BEGIN IMMEDIATE` when no migrations are pending, avoiding contention behind writer instances under dual-instance boot. SQLite `busy_timeout` (5s) is configured before the first fence read on both backends (`src/features/magic-context/storage-db.ts`). Historian and wrapup lease releases in TypeScript and Pi are best-effort, catching `SQLITE_BUSY` errors on delete and letting orphaned rows expire on their TTL. For replay stability across compaction marker advances, stripped placeholder IDs are bounded (4096) and cleared on `message.removed`. Legacy open fences and column healing (`ensureColumn()` + `healAllNullColumns()`, defined in `storage-schema-helpers.ts` to prevent cycles between `storage-db` and `migrations`) backfill upgraded DBs before orphan-sweep index creation so pre-migration tables do not wedge on startup. Migration v71 rebuilds authority guard triggers to durable `context_privilege_state` state-table form, eliminating connection-local UDF dependencies (`mc_privileged_writer`) so non-registering SQLite connections execute guarded writes safely. Migration v78 adds the `migration_pending` recovery journal to track in-flight cross-harness session migrations (OpenCode → Pi/OMP) crash-safely without exposing raw `session_id` columns to `clearSession()`. Migration v81 persists Last Known Good transform snapshots in `lkg_slots` for crash-safe recovery across restarts. Migration v82 adds `memory_verifications.mapping_origin` to distinguish mapper independence from host-rejected fallback sentinels (mirrored in Rust store migration 51). In Rust mode, memory mirror projections and NULL-field repairs execute in atomic write transactions with SQL-level recency guards against host `updated_at` and `classified_at` to prevent stale module snapshots from rolling back newer host lifecycle or classification states, and authority prepare/mirror routines rebind stable source identities and prebind page tombstones so host memory IDs survive round-trips. New session-scoped tables must be added to `clearSession()`. A bulletproof `MAGIC_CONTEXT_TEST_DATA_DIR` guard keeps the test suite off the live DB (running `bun test` once migrated a live DB and fail-closed running binaries). SQLite binds must use SPREAD positional args, never the array form (`bun:sqlite` binds a lone array positionally; `node:sqlite` reads it as named params and throws). For branch forks in Pi, copy durable session state (compartments, tags, pending operations, and session metadata) to the new session via `copySessionStateForClone()` in `src/features/magic-context/storage-clone.ts`. Run this copy inside an immediate SQLite transaction, filtering and mapping message ordinals and tag composite keys to the copied branch entries, and clearing cached cache bytes to trigger fresh rematerialization. Slow SQLite write transactions (held for ≥1000ms) are logged post-commit with site and duration attribution (`src/shared/write-transaction-timing.ts`) across historian publish, recomp, lease-guarded writes, clear-session, marker-drain, and migration runs without affecting transaction success. Under Rust `mc-store`, `cortexkit-store 0.2.0` logs a store-ahead migration outcome instead of a silent no-op, without refusing startup so older binary rollbacks remain safe. +### Timestamp Columns + +Timestamp columns in the fresh-install schema (`storage-db.ts`) are stamped in **epoch milliseconds**. Each entry below was verified against its writer; the unit is epoch ms on every verified path. Rows whose writer is not TypeScript `Date.now()` (or that have more than one writer) are called out explicitly, and the `storage-db.ts` schema comment `-- epoch ms (Date.now())` is only placed on columns where `Date.now()` is the actual writer. + +| Table | Column | Unit | Writer | Verified | +|-------|--------|------|--------|----------| +| source_contents | created_at | epoch ms | `Date.now()` (`storage-source.ts`) | ✅ | +| compartments | created_at | epoch ms | `Date.now()` (`compartment-storage.ts`) | ✅ | +| compartment_chunk_embeddings | created_at | epoch ms | `Date.now()` (`compartment-chunk-embedding.ts`) | ✅ | +| compartment_events | created_at | epoch ms | `Date.now()` (`compartment-events.ts`) | ✅ | +| session_facts | created_at | epoch ms | `Date.now()` (`compartment-storage.ts`) | ✅ | +| session_facts | updated_at | epoch ms | `Date.now()` (`compartment-storage.ts`) | ✅ | +| primer_candidates | created_at | epoch ms | `Date.now()` (`storage-primers.ts`) | ✅ | +| primers | created_at | epoch ms | `Date.now()` (`storage-primers.ts`) | ✅ | +| primers | updated_at | epoch ms | `Date.now()` (`storage-primers.ts`) | ✅ | +| synapse_batch_ledger | created_at | epoch ms | `Date.now()` (`storage-embedding-measurements.ts`) | ✅ | +| synapse_batch_ledger | updated_at | epoch ms | `Date.now()` (`storage-embedding-measurements.ts`) | ✅ | +| embedding_measurement_corpus | created_at | epoch ms | `Date.now()` (`storage-embedding-measurements.ts`) | ✅ | +| historian_runs | created_at | epoch ms | `Date.now()` (`storage-historian-runs.ts`) | ✅ | +| recomp_compartments | created_at | epoch ms | `Date.now()` (`compartment-storage.ts`) | ✅ | +| recomp_facts | created_at | epoch ms | `Date.now()` (`compartment-storage.ts`) | ✅ | +| memories | created_at | epoch ms | `Date.now()` (host, `storage-memory.ts`) / Rust store via mirror (`context-authority.ts`) | ⚠️ mixed writer | +| memories | updated_at | epoch ms | `Date.now()` (host) / Rust store via mirror (`context-authority.ts`) | ⚠️ mixed writer | +| workspaces | created_at | epoch ms | `now_millis()` (dashboard Rust, `workspaces.rs`) | ✅ | +| workspaces | updated_at | epoch ms | `now_millis()` (dashboard Rust, `workspaces.rs`) | ✅ | + +Notes: +- `memories` rows created by the Rust-module mirror first allocate a placeholder with `created_at`/`updated_at` = 0 (`context-authority.ts`), then apply the module snapshot's own epoch-ms stamp — so `Date.now()` is not the sole writer, and the schema comment deliberately omits `memories`. +- `workspaces` is stamped by the dashboard Rust backend (`chrono::Utc::now().timestamp_millis()`), not by TypeScript. +- This table covers the `created_at`/`updated_at` columns audited for this change; it is not an exhaustive inventory of every timestamp column in the schema. + ## Session modes There are three effective runtime surfaces. Primary sessions and subagents use the normal context-management pipeline with different feature sets. Compaction-off mode keeps additive knowledge surfaces but removes Magic Context's context-window management. The primary reduce surface is additionally gated by the session's actual `ctx_reduce` tool availability. diff --git a/packages/plugin/src/features/magic-context/storage-db.ts b/packages/plugin/src/features/magic-context/storage-db.ts index 60c35d908..fbadad6a0 100644 --- a/packages/plugin/src/features/magic-context/storage-db.ts +++ b/packages/plugin/src/features/magic-context/storage-db.ts @@ -939,7 +939,7 @@ export function initializeDatabase( tag_id INTEGER, session_id TEXT, content TEXT, - created_at INTEGER, + created_at INTEGER, -- epoch ms (Date.now()) harness TEXT NOT NULL DEFAULT 'opencode', PRIMARY KEY(session_id, tag_id) ); @@ -963,7 +963,7 @@ export function initializeDatabase( p1_embedding BLOB, p1_embedding_model_id TEXT, legacy INTEGER NOT NULL DEFAULT 0, - created_at INTEGER NOT NULL, + created_at INTEGER NOT NULL, -- epoch ms (Date.now()) harness TEXT NOT NULL DEFAULT 'opencode', UNIQUE(session_id, sequence) ); @@ -982,7 +982,7 @@ export function initializeDatabase( model_id TEXT NOT NULL, dims INTEGER NOT NULL, vector BLOB NOT NULL, - created_at INTEGER NOT NULL, + created_at INTEGER NOT NULL, -- epoch ms (Date.now()) UNIQUE(compartment_id, model_id, window_index) ); CREATE INDEX IF NOT EXISTS idx_cce_session ON compartment_chunk_embeddings(session_id); @@ -1005,7 +1005,7 @@ export function initializeDatabase( kind TEXT NOT NULL, at_compartment INTEGER, fields_json TEXT NOT NULL DEFAULT '{}', - created_at INTEGER NOT NULL, + created_at INTEGER NOT NULL, -- epoch ms (Date.now()) harness TEXT NOT NULL DEFAULT 'opencode' ); CREATE INDEX IF NOT EXISTS idx_compartment_events_session @@ -1034,7 +1034,7 @@ export function initializeDatabase( session_id TEXT NOT NULL, category TEXT NOT NULL, content TEXT NOT NULL, - created_at INTEGER NOT NULL, + created_at INTEGER NOT NULL, -- epoch ms (Date.now()) updated_at INTEGER NOT NULL, harness TEXT NOT NULL DEFAULT 'opencode' ); @@ -1053,7 +1053,7 @@ export function initializeDatabase( source_message_time INTEGER NOT NULL, question_embedding BLOB, question_embedding_model_id TEXT, - created_at INTEGER NOT NULL, + created_at INTEGER NOT NULL, -- epoch ms (Date.now()) UNIQUE(project_path, harness, session_id, source_start_message_id, source_end_message_id) ); CREATE INDEX IF NOT EXISTS idx_primer_candidates_project_time @@ -1076,7 +1076,7 @@ export function initializeDatabase( answer_refreshed_at INTEGER, source_candidate_ids TEXT NOT NULL DEFAULT '[]', source_candidate_provenance TEXT, - created_at INTEGER NOT NULL, + created_at INTEGER NOT NULL, -- epoch ms (Date.now()) updated_at INTEGER NOT NULL ); CREATE INDEX IF NOT EXISTS idx_primers_project_status_observed @@ -1189,7 +1189,7 @@ export function initializeDatabase( job_id TEXT, cursor TEXT, status TEXT NOT NULL DEFAULT 'pending', - created_at INTEGER NOT NULL DEFAULT 0, + created_at INTEGER NOT NULL DEFAULT 0, -- epoch ms (Date.now()) updated_at INTEGER NOT NULL DEFAULT 0, UNIQUE(session_id, request_key) ); @@ -1230,7 +1230,7 @@ export function initializeDatabase( shadow_epoch INTEGER NOT NULL DEFAULT 0, corpus_hash TEXT NOT NULL DEFAULT '', coverage_json TEXT NOT NULL DEFAULT '{}', - created_at INTEGER NOT NULL DEFAULT 0, + created_at INTEGER NOT NULL DEFAULT 0, -- epoch ms (Date.now()) UNIQUE(dedup_key, cohort_key) ); CREATE INDEX IF NOT EXISTS idx_embedding_measurement_session @@ -1655,7 +1655,7 @@ CREATE INDEX IF NOT EXISTS idx_dream_queue_pending ON dream_queue(started_at, en importance_avg REAL, discarded_last INTEGER NOT NULL DEFAULT 0, legacy INTEGER NOT NULL DEFAULT 0, - created_at INTEGER NOT NULL + created_at INTEGER NOT NULL -- epoch ms (Date.now()) ); CREATE INDEX IF NOT EXISTS idx_historian_runs_session ON historian_runs(session_id, created_at DESC); @@ -1708,7 +1708,7 @@ CREATE INDEX IF NOT EXISTS idx_dream_queue_pending ON dream_queue(started_at, en importance INTEGER NOT NULL DEFAULT 50, episode_type TEXT, pass_number INTEGER NOT NULL, - created_at INTEGER NOT NULL, + created_at INTEGER NOT NULL, -- epoch ms (Date.now()) harness TEXT NOT NULL DEFAULT 'opencode', UNIQUE(session_id, sequence) ); @@ -1719,7 +1719,7 @@ CREATE INDEX IF NOT EXISTS idx_dream_queue_pending ON dream_queue(started_at, en category TEXT NOT NULL, content TEXT NOT NULL, pass_number INTEGER NOT NULL, - created_at INTEGER NOT NULL, + created_at INTEGER NOT NULL, -- epoch ms (Date.now()) harness TEXT NOT NULL DEFAULT 'opencode' ); From 5b2d8a05ad64774694b14c0e88da1da4523f90f5 Mon Sep 17 00:00:00 2001 From: g0 Date: Thu, 17 Sep 2026 03:23:52 +0800 Subject: [PATCH 2/3] docs: qualify timestamp provenance (caller overrides, clone preservation) Address Greptile/cubic review: - ARCHITECTURE.md: flag caller-supplied timestamp paths in the table (row.createdAt for compartment_chunk_embeddings, candidate.createdAt for primer_candidates, input.now for primers); add Notes covering the session-clone path (copySessionStateForClone preserves created_at verbatim) and the NOT NULL DEFAULT 0 backstop on ledger columns. - storage-db.ts: source_contents.created_at comment now reads "epoch ms; Date.now() on source writes, preserved on session clones". Docs/comments only; no behavior or migration changes. --- ARCHITECTURE.md | 12 +++++++----- .../plugin/src/features/magic-context/storage-db.ts | 2 +- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 9862f8851..7a9f3e732 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -149,19 +149,19 @@ Background maintenance (V2: per-task cron scheduling). A process-wide 15-min tim ### Timestamp Columns -Timestamp columns in the fresh-install schema (`storage-db.ts`) are stamped in **epoch milliseconds**. Each entry below was verified against its writer; the unit is epoch ms on every verified path. Rows whose writer is not TypeScript `Date.now()` (or that have more than one writer) are called out explicitly, and the `storage-db.ts` schema comment `-- epoch ms (Date.now())` is only placed on columns where `Date.now()` is the actual writer. +Timestamp columns in the fresh-install schema (`storage-db.ts`) are stamped in **epoch milliseconds**. Each entry below was verified against its writer; the unit is epoch ms on every verified path. Rows whose writer is not TypeScript `Date.now()` (or that have more than one writer) are called out explicitly, and the `storage-db.ts` schema comment `-- epoch ms (Date.now())` is only placed on columns where `Date.now()` is the actual writer. Caller-supplied overrides and clone-preserved copies are called out per row and in the Notes below. | Table | Column | Unit | Writer | Verified | |-------|--------|------|--------|----------| | source_contents | created_at | epoch ms | `Date.now()` (`storage-source.ts`) | ✅ | | compartments | created_at | epoch ms | `Date.now()` (`compartment-storage.ts`) | ✅ | -| compartment_chunk_embeddings | created_at | epoch ms | `Date.now()` (`compartment-chunk-embedding.ts`) | ✅ | +| compartment_chunk_embeddings | created_at | epoch ms | `Date.now()` default; caller `row.createdAt` honored (`compartment-chunk-embedding.ts`) | ✅ | | compartment_events | created_at | epoch ms | `Date.now()` (`compartment-events.ts`) | ✅ | | session_facts | created_at | epoch ms | `Date.now()` (`compartment-storage.ts`) | ✅ | | session_facts | updated_at | epoch ms | `Date.now()` (`compartment-storage.ts`) | ✅ | -| primer_candidates | created_at | epoch ms | `Date.now()` (`storage-primers.ts`) | ✅ | -| primers | created_at | epoch ms | `Date.now()` (`storage-primers.ts`) | ✅ | -| primers | updated_at | epoch ms | `Date.now()` (`storage-primers.ts`) | ✅ | +| primer_candidates | created_at | epoch ms | `Date.now()` default; caller `candidate.createdAt` honored (`storage-primers.ts`) | ✅ | +| primers | created_at | epoch ms | `Date.now()` default; caller `input.now` honored (`storage-primers.ts`) | ✅ | +| primers | updated_at | epoch ms | `Date.now()` default; caller `input.now` honored (`storage-primers.ts`) | ✅ | | synapse_batch_ledger | created_at | epoch ms | `Date.now()` (`storage-embedding-measurements.ts`) | ✅ | | synapse_batch_ledger | updated_at | epoch ms | `Date.now()` (`storage-embedding-measurements.ts`) | ✅ | | embedding_measurement_corpus | created_at | epoch ms | `Date.now()` (`storage-embedding-measurements.ts`) | ✅ | @@ -176,6 +176,8 @@ Timestamp columns in the fresh-install schema (`storage-db.ts`) are stamped in * Notes: - `memories` rows created by the Rust-module mirror first allocate a placeholder with `created_at`/`updated_at` = 0 (`context-authority.ts`), then apply the module snapshot's own epoch-ms stamp — so `Date.now()` is not the sole writer, and the schema comment deliberately omits `memories`. - `workspaces` is stamped by the dashboard Rust backend (`chrono::Utc::now().timestamp_millis()`), not by TypeScript. +- Session clones (`copySessionStateForClone()`, `storage-clone.ts`) copy `created_at` verbatim from the source row for `source_contents`, `compartments`, and `session_facts` — the original timestamp is preserved, not re-stamped with `Date.now()`. +- `synapse_batch_ledger` and `embedding_measurement_corpus` declare `NOT NULL DEFAULT 0` as a backstop; every current writer passes `Date.now()` explicitly, so the default never fires on verified write paths. - This table covers the `created_at`/`updated_at` columns audited for this change; it is not an exhaustive inventory of every timestamp column in the schema. ## Session modes diff --git a/packages/plugin/src/features/magic-context/storage-db.ts b/packages/plugin/src/features/magic-context/storage-db.ts index fbadad6a0..3d4c599e0 100644 --- a/packages/plugin/src/features/magic-context/storage-db.ts +++ b/packages/plugin/src/features/magic-context/storage-db.ts @@ -939,7 +939,7 @@ export function initializeDatabase( tag_id INTEGER, session_id TEXT, content TEXT, - created_at INTEGER, -- epoch ms (Date.now()) + created_at INTEGER, -- epoch ms; Date.now() on source writes, preserved on session clones harness TEXT NOT NULL DEFAULT 'opencode', PRIMARY KEY(session_id, tag_id) ); From 8d76de6e5e7e47474cd1518e35de6a9dc151ab86 Mon Sep 17 00:00:00 2001 From: g0 Date: Thu, 17 Sep 2026 03:46:03 +0800 Subject: [PATCH 3/3] docs: note refreshedAt override on primers.updated_at updatePrimerAnswer() honors caller-supplied refreshedAt for primers.updated_at (default Date.now()); the table row previously listed only input.now. Docs only. --- ARCHITECTURE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 7a9f3e732..90e99e04b 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -161,7 +161,7 @@ Timestamp columns in the fresh-install schema (`storage-db.ts`) are stamped in * | session_facts | updated_at | epoch ms | `Date.now()` (`compartment-storage.ts`) | ✅ | | primer_candidates | created_at | epoch ms | `Date.now()` default; caller `candidate.createdAt` honored (`storage-primers.ts`) | ✅ | | primers | created_at | epoch ms | `Date.now()` default; caller `input.now` honored (`storage-primers.ts`) | ✅ | -| primers | updated_at | epoch ms | `Date.now()` default; caller `input.now` honored (`storage-primers.ts`) | ✅ | +| primers | updated_at | epoch ms | `Date.now()` default; caller `input.now` / `refreshedAt` honored (`storage-primers.ts`) | ✅ | | synapse_batch_ledger | created_at | epoch ms | `Date.now()` (`storage-embedding-measurements.ts`) | ✅ | | synapse_batch_ledger | updated_at | epoch ms | `Date.now()` (`storage-embedding-measurements.ts`) | ✅ | | embedding_measurement_corpus | created_at | epoch ms | `Date.now()` (`storage-embedding-measurements.ts`) | ✅ |