Skip to content

[WIP] Plan TraceDecay V2 brain rewrite#421

Draft
ScriptedAlchemy wants to merge 665 commits into
masterfrom
codex/tracedecay-total-redesign-plan
Draft

[WIP] Plan TraceDecay V2 brain rewrite#421
ScriptedAlchemy wants to merge 665 commits into
masterfrom
codex/tracedecay-total-redesign-plan

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Jul 10, 2026

Copy link
Copy Markdown
Owner

WIP: plan only

This PR contains only the TraceDecay V2 total-rewrite design: one master plan, one plan-set index, and 28 linked subsystem/crate/product plans (30 documentation files). It deliberately contains no implementation.

Unified product and architecture

  • Rebuild TraceDecay as one coherent profile-wide brain across projects, repositories, worktrees, branches, threads, turns, agents, workflows, tools, artifacts, code, Git, facts, memories, tasks, plans, delivery, and time.
  • Make an immutable sanitized observation/event stream the write authority; derive versioned projections, graph/read models, search indexes, analytics, hints, and UI views through fenced replayable projectors with checkpoints, dead letters, repair, and provenance.
  • Replace ambiguous project-only identity and split stores with canonical profile/project/repository/worktree/session/message/agent/task/artifact/code identities, explicit source bindings, temporal validity, variant safety, and protected consolidation/quarantine workflows.
  • Define crate ownership, dependency/release DAGs, anti-corruption boundaries, migration/deletion gates, negative-code budgets, and shared kernels so the rewrite converges instead of recreating fragmented subsystems.

Multi-machine shared Brain

  • Adds plan 28 for local-only, remote-authority, cached-client, read-replica, standby, and hybrid-placement operation under one BrainId.
  • Keeps every SQLite/WAL family host-local. Remote nodes exchange sanitized observations, signed snapshots/tails, immutable packs, and typed commands through the authenticated application/API contract; no network-mounted database or database URL is exposed.
  • Correlates equivalent clones across machines through credential-free Git remote evidence plus verified object/commit/tree/ancestry evidence. Paths and worktrees stay node-scoped; forks, shallow/rewritten histories, and contradictions remain explicit candidates.
  • Uses exactly one writable authority per mutable shard, signed/expiring node and cache grants, bounded causal frontiers, authority/placement epochs, offline idempotent capture, visible lag/pending/conflict state, and positive external fencing before standby promotion.
  • Makes Tailscale an optional connectivity profile only. Ordinary authenticated HTTPS/mTLS, LAN/VPN, reverse proxy, or tunnel deployments use the same TraceDecay identity, authorization, privacy, revocation, and fencing rules.
  • Adds wrapped-key disaster recovery after total authority-node/keyring loss, explicit remote schemas/retention, node/store/repository topology UI, Sync Observatory, Brain Settings, CLI/API/SDK bindings, compact default MCP status, operator-only MCP administration, and multi-machine fault/RPO/RTO gates.

Product surfaces and task execution

  • Replace the project-only dashboard with an All / Brain workspace plus scoped lenses; interactive Git, code, thread, turn, agent, task, plan, memory, impact, and timeline graphs; reasoning/code/artifact lineage; synchronized explorers; and accessible table/outline parity.
  • Add hermetic replay/debug labs for search, LCM, facts/memory, hooks, hints, context scouting, redaction, curation, automation, scheduling, configuration, and experiments with current-versus-historical policy comparisons and reproducible receipts.
  • Port and redesign the Hermes Kanban concept as a native TraceDecay task/plan graph—not an adapter—with federated initiatives/boards, DAG dependencies, multi-project work bundles, model/effort routing, executor/provider capabilities, offers/admission, leases, grants, work packets, nearby-agent awareness, and duplicate/conflict prevention.
  • Add validated sharded frontmatter/CommonMark task-graph edit workspaces for large agent edits, authoritative validate/diff/rebase/submit flows, safe temporary cleanup, exact candidate refs, and atomic owner-shard publication.

Search, hints, automation, APIs, CLI, and MCP

  • Redesign hybrid lexical/vector/graph/temporal retrieval, supersession-aware ranking, contextual retrieval IDs, cross-project scope, explainability, qrels/judgments, real-session replay evaluation, and precision/recall promotion gates.
  • Make adaptive hint arbitration and optional incremental model-backed context scouting compact, material, deduplicated, privacy-safe, nearby-work aware, and measurable against useful-silence baselines.
  • Make autonomous memory/skill curation and self-improvement event-driven and frontier-fenced: unchanged/no-activity inputs create no repeated runs or model/tool work.
  • Publish one typed application contract through an official HTTP API, Rust/TypeScript/Python SDKs, CLI, optional MCP, dashboard, hooks, and executors; centralize typed views, Markdown/JSON rendering, errors, pagination, and capability discovery.
  • Split MCP into independently installable context/work/operator trust-boundary registrations backed by one binary/daemon/catalog. Skills plus CLI remain the complete MCP-free baseline; eager clients must pass the same security/context budgets as deferred clients.

Claude Code, Codex, and Cursor bundles

  • Adds plan 27 with primary-source research for official Claude Code, Codex, and Cursor plugin, skill, command, agent/subagent, hook, MCP, configuration, marketplace, permission, and progressive-disclosure behavior.
  • Keeps one canonical HostIntegrationManifestV1; PR 22A owns its catalog/source IR, PR 22I deterministically emits unsigned HostBundlePayloadV1 trees, PR 36R independently rebuilds/scans/conformance-tests/attests/signs HostBundleManifestV1, and application alone binds a signed release to current host capability state.
  • Uses host capability evidence and stock-host conformance instead of assuming cross-host parity. Unsupported/version-gated behavior gets an explicit omission and skill/CLI/API fallback.
  • Ships a small skills-first core plus optional least-privilege MCP companions; commands remain thin compatibility/explicit-user affordances, hooks remain bounded, and specialist roles receive exact task/scope/worktree/retrieval-anchor handoffs.

Privacy and control plane

  • Add ingest/retrieval/export/presentation secret detection and redaction with policy versions, sensitivity labels, grants, audit receipts, protected quarantine, leak-regression tests, and UI/CLI controls.
  • Put configuration—including redaction, retention, providers/models, hints, hooks, search, automation, executors, task routing, host integrations, shared-Brain placement/sync/cache/failover, and sharing—in one versioned control plane available through UI, CLI, API, and optional MCP.
  • Preserve fully autonomous curation semantics; there is no generic preview/apply/rollback proposal queue.

Accepted-base reconciliation and evidence

  • Current with master at 273f50c0372f063b97f4755563a3ded65ef324d5 (chore: release v0.0.53 (#437)); branch is not behind.
  • Incorporates accepted changes through fix(doctor): isolate registry diff conflicts #440, including fix(storage): add explicit split-store consolidation #425 split-store consolidation and fix(doctor): derive orphan stores from registry reconstruction #439/fix(doctor): isolate registry diff conflicts #440 registry-reconstruction truth/conflict isolation. At publication, this draft [WIP] Plan TraceDecay V2 brain rewrite #421 is the only open PR.
  • Planning used TraceDecay LCM/session/message/fact/code/Git/analytics/diagnostic/automation surfaces, supported CLI/provider-local fallbacks when context tools failed, several independent subagent audits, Hermes source analysis, and primary-source/web/arXiv research.
  • Private chronological exports remain outside Git under /fast/tracedecay-redesign-research (0700; files 0600): 34,352 secret-scanned role-user records and a 9,988-record best-effort human subset through 2026-07-11T01:04:10.875Z.
  • Retained sanitized-content SHA-256: broad 0c56ebd9c54edfc76cda10b62de05e0a9443004c3ddce960daf300208bbdf340; human 942f6662c77f820c4d8a2b7063889c4521cecee68bc0258c7c7c12ddc6309dc2.
  • No transcript, database, generated fixture, or private research artifact is committed or indexed by this PR.

Validation

  • 30/30 intended documentation files and no other changed paths.
  • Internal relative links valid; Markdown fences balanced; git diff --check clean.
  • Failure modes FM-001 through FM-137 contiguous and unique.
  • Cross-plan type owners, signed sync/cache contracts, positive failover fencing, recovery keys, physical schemas, PR sequencing, closed Brain operation bindings, host compiler/signing/runtime boundaries, task-edit ceilings, configuration targets, and integration operations reconciled by independent final audits.
  • Corpus ordering, deduplication, timestamps, content hashes, permissions, and redacted-copy equality validated.
  • Direct plan scan, private-corpus scans, and staged-diff scan pass with gitleaks and zero findings.

Draft/WIP remains intentional until the plan is reviewed as one flowing architecture.

@changeset-bot

changeset-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 948728d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy ScriptedAlchemy added the WIP Work in progress; do not merge label Jul 10, 2026
@ScriptedAlchemy
ScriptedAlchemy force-pushed the codex/tracedecay-total-redesign-plan branch from e50f8f5 to 4f5ab66 Compare July 10, 2026 05:54
@ScriptedAlchemy

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 986f25cca6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/plans/2026-07-09-tracedecay-brain-rewrite.md Outdated
@ScriptedAlchemy
ScriptedAlchemy force-pushed the codex/tracedecay-total-redesign-plan branch 4 times, most recently from 5862bd6 to 53502de Compare July 11, 2026 16:52
Diagnose through file:?immutable=1&mode=ro without DatabaseAuthority,
WAL/SHM, schema, or worker side effects. Expand inventory to columns and
indexes; keep FTS repair as the sole explicit writer-lane effect.
Prove the path API leaves bytes, WAL/SHM, and authority lock files
unchanged; missing stores stay unavailable; index/column drift reports
partial migration_gap findings without repair.
Keep one bounded frame-read future alive while health-checking so
mid-frame pauses no longer drop consumed prefixes.
Share one monotonic Instant across daemon client stages so CLI
timeouts cancel in-flight work instead of dropping mid-frame state.
Fail closed on truncated MCP tool payloads so CLI stdout never
emits a different schema under the shared monotonic deadline path.
Keep the cold-Doctor path API warning-free until the transport route is
integrated.
Record truthful sanitization/provenance for the provider-native Codex
captures reused by the PR8 temporal harness.
Drive Codex admit, CanonicalSessionTemporalProjector materialize,
SessionRefreshService, and SessionRetrievalService under HOME and
TRACEDECAY_DATA_DIR RAII isolation with provisional attestation.
Mark the workload harness_ready with Codex receipts, keep evidence
provisional, and keep dry-run Cargo-free with Windows unsupported-run.
Keep estimate()-only implementors source-compatible via default
Whitespace policy, stream UTF-8/token scans without full output
materialization, and close substring concatenate and leftover tests.
Unblock temporal lib tests after UnauthorizedSource gained anchor_id.
Add knowledge_at and valid_time to copy records, defaulting legacy wires to
unknown validity so historical payloads remain deserializable.
Bump session-temporal schema to v3, recreate legacy copy tables with
knowledge_at/valid_time_json, and preserve unknown validity on backfill.
Derive ProviderLinkage when typed provider evidence permits, reject forged
assertion ids and non-canonical proofs, persist copy bitemporality through
projection digests, and cover restart/multi-batch guard paths.
Avoid GlobalDb::open_read_only_at on the cold Doctor route so authority,
WAL, and SHM artifacts are not created while preserving doctor_runtime JSON.
Keep estimate()-only implementors source-compatible via default
Whitespace policy, stream UTF-8/token scans without full output
materialization, and close substring concatenate and leftover tests.
Unblock temporal lib tests after UnauthorizedSource gained anchor_id.
ScriptedAlchemy and others added 30 commits July 22, 2026 10:29
Third import-race repair: lint sweep removed the then-unused constant
import; a later surface commit references it at handlers/mod.rs:1183.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backtick TraceDecay in model_catalog doc, flip if_not_else in feedback,
map_or(true,..)->is_none_or and manual_let_else conversions in model_lifecycle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fastembed_adapter.rs now imports super::model_catalog and crate::config,
which the path-including pr10 test crate must mirror at its own root.
Verified: cargo test --test pr10_artifact_runtime_prep_test --no-run
exits 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Format the committed content of profile_config.rs, which entered HEAD
unformatted when staged agent work was swept into 8dc0880.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pinned yaml-edit 0.2.3 cannot synthesize nested block or flow
collections: Mapping::set / Sequence::push render new members at column
0 because their indentation is derived from context lost through the
cloned get_mapping / get_sequence handles, and node-copying set ignores
the requested indent. The lossless-editing refactor therefore wrote
malformed ~/.hermes/config.yaml (nested keys flush-left), so
registration_state parsed plugins/memory/context as null, returned
Repairable, and install/uninstall surfaced "host bundle interrupted
operation requires recovery" or "unsupported Hermes plugins config".

Rewrite the enable/disable mutation path to locate byte spans with
yaml-edit and apply edits as text splices on the source, re-parsing
between operations. yaml-edit is used only to parse and locate; comments,
anchors, aliases, flow collections and CRLF line endings survive
byte-for-byte. Fixes lossless_profile_config_corpus,
enable_plugin_creates_missing_profile_config, and the downstream
lifecycle/host-bundle registration-state checks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The registry now annotates 16 additional tools with readOnlyHint=true
(affected_tests, diagnostics_read, feedback_*, file_metadata,
git_preview, health_read, qualified_name, session_lookup, source_*,
test_results). The cursor plugin README mcpAllowlist must list exactly
the read-only set so those tools skip the Auto-review classifier; the
missing entries failed readme_mcp_allowlist_matches_read_only_tools.
Regenerate the allowlist from the registry's readOnlyHint=true tools.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ensure_observation_authority ran the full-introspection schema contract
validation twice per first-open. The second pass re-validates an
unchanged shape: between the passes only rows are seeded and invariant
triggers are re-installed from the same constant definitions. Measured
~50ms per GlobalDb first-open (three opens per consolidate fixture),
paid by every non-daemon CLI/hook process and by tracedecay init.
Fail-closed validation is retained: every open still validates once
immediately after schema installation. Verified in a clean snapshot:
schema_contract 8/8, projection 7/7, migrations green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolve clippy findings across lane-C cold files: manual_let_else,
match/if-let rewrites, doc backticks, elided lifetimes, redundant
closures, map_or, default-ctor, large_futures boxing, must_use,
semicolons, and production expect_used -> unwrap_or_else(panic!).
Behavior preserved. Signature-changing lints (needless_pass_by_value,
too_many_arguments, struct_field_names, option_option,
trivially_copy_pass_by_ref) deferred to avoid co-editor conflicts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When disable removes the last child of a block mapping, the container
key is left with a null value (not an empty mapping), so the previous
collapse check — which required a mapping value — kept a dangling
`memory:` / `context:` / `tracedecay:` line. Treat a null or empty value
as collapsible and let remove_map_entry drop a null-valued entry by its
key line. Uninstall now removes the empty tracedecay-created memory and
context blocks, satisfying test_hermes_global_install_and_uninstall_plugin.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR12 configuration control plane installs each project's daemon
client into a process-global slot but never removed it, so every
non-daemon process that opened a project (init, CLI, hooks) retained the
opened store Arc — and its exclusive sessions.db writer lease — for its
entire lifetime, starving the managed daemon of the single-writer lock.
Release the registration when the runtime drops, Arc::ptr_eq-guarded so
a newer client installed by a live handle is never removed. Also seed
the anchor-authority test fixture through the writer broker with the
canonical tagged owner encoding; plain conn() is sealed query_only.

Snapshot-verified: memory_eval 13/13, retrieval_anchor_authority 5/5,
cli_branch_add_uses_managed_daemon_as_the_only_database_writer, and the
mcp_suite serve harness all pass with this change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restore missing BTreeMap/Arc/OperationBudgetUsage imports and the
unwrap_or(false) tails of two git_available helpers. This in-progress
test-module code entered HEAD via lane commits verified only with
cargo check --lib, which does not compile cfg(test) modules. Verified:
cargo test --lib --no-run exits 0 with zero errors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The semantic_code module compiled unconditionally but imported
feature-gated fastembed symbols, so --no-default-features could not
build (the last red clippy lens). Provide a feature-disabled stand-in
runtime whose operations fail with a typed IncompatibleRuntime error and
an uninhabited session type, so semantic retrieval degrades to its
documented fallback states. Un-gate fastembed_failure (it builds only
feature-independent error types) and the runtime factory. Verified:
cargo check --lib for both lenses exit 0; cargo clippy --lib
--no-default-features -D warnings exit 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Local codex/cursor installs failed with 'host bundle atomic filesystem
operation failed': the host-bundle catalog deployed raw template assets
while the compatibility registration adapter re-ran the legacy
installer, so the component-set digest verification saw two different
byte streams and rolled the install back. Deploy the installers' own
rendered inventories for the shared components, correct the codex and
cursor project-local registration paths, and surface the integration's
own error before it collapses into the generic storage failure.
Verified: agent_suite 511/512 (the one residual, codex update-plugin
bootstrap refresh, was already failing on CI and is tracked separately).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Close the last five clippy-gate errors: un-gate required_member_bytes
(its body reads member bytes through feature-independent store and
lifecycle pins, and the adapter tests call it in every profile), bind
the scheduler-test re-exports to the semantic-fastembed feature their
only consumers carry, and gate the mcp_suite project-session path
behind test-transport where its harness lives. Verified with all three
CI lenses (cargo clippy --workspace --all-targets -D warnings for
all-features, default, and no-default-features): zero errors each.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A git root with no published configuration snapshot fell into
allow_init=false, contradicting the schema default (auto-init enabled)
and leaving explicit initialize-roots repos unable to open; follow the
SyncConfig default instead. Verified:
initialize_roots_auto_initializes_unindexed_git_repo passes (was red).
Also repair four Windows-lens breaks invisible to Linux clippy: qualify
serde_json paths in the socket invocation test, cfg-split the maybe_dir
and open_project_file imports whose only users are platform-gated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Format 18 committed files left unformatted by the 224e6d6 convergence
push. Excludes tests/session_suite/temporal_derived_evidence.rs, which
carries active co-editor working-tree edits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collapse a nested if and drop a lazy Option::None closure in evaluation,
add the too_many_arguments allow (crate precedent), drop clone-on-Copy of
SummarySourceHorizonV1, and merge identical member-role arms in
session_derived. Verified: cargo clippy -p tracedecay-domain
--all-targets --all-features -D warnings exits 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
My prior commit 1e6bd2e swept a co-editor's in-flight removal of
OwnerDecisionEvidenceV1 from evaluation.rs, but its consumers in
search_eval_acceptance_contract.rs and tracedecay-search-eval.rs are
still present on the branch, leaving dangling references that broke the
build (Test/Dashboard/Windows/Hermes). Restore the definition (keeping
only my two intended clippy fixes) so the committed tree is
self-consistent again; the co-editor lands the complete feature removal
as its own coherent change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backtick TraceDecay in the module doc and match the unit closure arg
with |()| instead of |_| (ignored_unit_patterns). Verified:
cargo clippy --lib -D warnings exits 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

WIP Work in progress; do not merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants