feat(v2.10): Float32 BLOB vectors — 5x smaller, lossless, migrated in bounded slices

[MXAntian]

Summary

• memories.content_vector moves from a JSON array to a Float32 BLOB (host byte order, same assumption as sqlite-vec). Measured on a 10,920-row library: 226 MB → 43 MB, and lossless — 2,048,000 / 2,048,000 sampled stored values are float32-exact (Math.fround(x) === x), i.e. the API already returns float32.
• No schema change: the column keeps TEXT affinity; SQLite stores BLOBs verbatim and typeof() separates the eras. Existing IS NOT NULL AND != '' checks keep matching.
• Migration 013 converts in place in 200-row transactions, as a background loop in the long-lived MCP server (yielding between batches) — never in hook children or ordinary CLI calls, which run under a spawn timeout. Resumable; both UPDATEs re-check typeof so a concurrent converter/re-embedder is never overwritten. PRAGMA user_version = 13 marks completion; the server honours it (skipIfComplete) so later starts are one pragma read. Unparseable text → NULL so the self-heal sweep re-embeds it.
• node index.mjs --migrate-vectors finishes it in one go and deliberately ignores the marker, so legacy rows that reappear (restored backup, out-of-band import) get converted. VACUUM is left to the operator (exclusive lock).
• Readers decode both formats; a half-migrated library is fully functional.

On the retired "JSON for cross-tool visibility" comment

A project-wide grep for content_vector found 10 files outside this checkout:

• ariel-workspace/memory-engram/index.mjs — a sibling engine copy for another agent; opens its own engram-ariel.db, never this file.
• 6 ad-hoc backfill / NULL-check scripts in chinatsu-workspace/.tmp/ and skill-audit/ — reference the column name, none JSON.parse it.
• 3 files in a git worktree of this same repo — not external.

So the accurate claim is no other process reads this database's column, not "no reader anywhere". The commit message and the code comment say that.

Review fixes (fanout on the draft — all five P1s taken, plus five P2s)

finding	fix
P1 trailing COUNT(*) full scan on every call	removed; remaining is 0 when drained, null when cut short, exact only with count: true (tested)
P1 migration ran inside hook children (spawnSync kills at ~2.8 s)	removed from initMemory(); runs in the MCP server's background loop + CLI only
P1 encodeVector accepted NaN/Infinity/float32 overflow	rejects non-finite, symmetric with decode (tested)
P1 --migrate-vectors had no try/catch	file-standard try/catch, exit 1 with a re-run hint on SQLITE_BUSY
P1 write-gate ranker untested on mixed formats	rankNearDuplicateCandidates extracted from findNearDuplicates; tested with JSON + BLOB + junk + orthogonal candidates
P2 UPDATE by rowid could overwrite a concurrent re-embed	both UPDATEs carry AND typeof(content_vector) = 'text'
P2 endianness implicit	documented in the codec header
P2 schema comment implied sqlite-vec reads this column	reworded: only application-layer cosine reads it
P2 one-way marker unexplained	README callout + CLI ignores marker (tested)
P2 conversations.content_vector still JSON	writer now encodes too (no reader exists, confirmed by the review)

Test plan

• vector-blob.integration.test.mjs — 49 checks: codec (both directions reject non-finite; misaligned Buffer; every column state); migration bounded, remaining=null without count, resume, junk→NULL, marker only when drained, idempotent, skipIfComplete honoured only when asked; memory-health and the write-gate ranker each pair a legacy JSON row with a BLOB row
• all 18 existing suites pass locally unchanged, including hooks/hooks.test.mjs which boots the modified server
• server smoke: 450 seeded legacy rows → 450 BLOBs + user_version=13 within 4 s of boot; runner log line present
• CLI smoke: 5 legacy + 1 junk with marker pre-set → {"converted":5,"skipped":1,"drained":true}, exit 0
• CI on the amended head
• after merge (2026-09-04): runtime fork rebased (6 private commits, 2 expected conflicts); server restarted, up in 105 ms; background runner converted 10,925 / 10,925 rows, 0 unparseable, in 2.4 s; VACUUM locked 2.2 s, 412 → 223 MB (vector bytes 226 → 42.7 MB); integrity ok. Follow-up test: load the libsimple tokenizer on every raw test connection #40 fixes a test-harness portability gap found while verifying (no such tokenizer: simple where libsimple is present).

🤖 Generated with Claude Code