Skip to content

feat(ingest): document ingestion adapters — docx + Confluence export (#192)#194

Merged
cdeust merged 1 commit into
mainfrom
feat/document-ingestion-192
Jul 26, 2026
Merged

feat(ingest): document ingestion adapters — docx + Confluence export (#192)#194
cdeust merged 1 commit into
mainfrom
feat/document-ingestion-192

Conversation

@cdeust

@cdeust cdeust commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Closes #192.

Summary

Adds ingest_document, a standalone (file-only, no upstream) MCP tool that ingests a .docx (OOXML zip) or a Confluence storage-format XHTML export into Cortex's memory/wiki store, on the same provenance + staleness path as code references. docx is parsed with the stdlib (zipfile + xml.etree) — no heavyweight dependency (§8). Standalone tool count 51 → 52.

Layer assignment (Clean Architecture §2.2)

Layer Files Role
core (pure, zero I/O) core/document_model.py, core/docx_parser.py, core/confluence_parser.py, core/document_normalizer.py parse bytes/strings → typed ParsedDocument → wiki/memory payloads
infrastructure infrastructure/document_reader.py file read + zip/XML unpacking; raises DocumentReadError
handlers (composition root) handlers/ingest_document.py, handlers/ingest_document_writers.py wire parsers + reader + wiki_write/remember
registration tool_registry_ingest.py register ingest_document unconditionally (file-only)

Import direction verified: no core → infrastructure/handlers, no infrastructure → core.

The shared seam #28 consumes

core.confluence_parser.parse_confluence_storage(xhtml)core.document_normalizer.normalize_document(doc, prov)handlers.wiki_write.write_governed_page + ingest_document_writers.write_document_memories. The live-Confluence REST connector (enterprise-backlog#28, OUT of scope here) reuses every line of parsing/normalization/writing — it swaps only the byte source (REST fetch instead of read_confluence_export) and the provenance (DocumentProvenance(source=page URL, version=page version, source_kind="confluence-api")).

Stakes (Move 7)

High — new public MCP tool surface + data-write path. Full discipline applied.

Edge cases (§13.1-A2 — each mapped to a test)

Case Behavior Test
Empty doc Ingested; explicit "no extractable text" notice; 0 section memories test_document_normalizer::TestEmptyDocument, test_ingest_document::TestEmptyDocument
Headings-only Headings rendered; no contentless memories test_docx_parser::test_headings_only_document, test_confluence_parser::test_headings_only
Tables Rendered as GFM markdown tables test_docx_parser::TestTables, test_confluence_parser::TestTables, test_document_normalizer::TestTableRendering
Embedded images Skipped with explicit notice (F1), asserted test_document_normalizer::TestImageNotice, test_ingest_document::TestImagesSkipped
Malformed zip/XML Loud error, no partial write (asserted no page/memory) test_document_reader::*, test_ingest_document::test_malformed_{zip,xml}_writes_nothing
Idempotent re-ingest (§13.1-A6) Same (source, version) → no new writes test_ingest_document::test_reingest_same_version_is_idempotent

Gates (quoted)

  • ruff checkAll checks passed!; ruff format --check1017 files already formatted
  • pyright (changed files) → 0 errors (tracked delta not increased)
  • Full pytest5559 passed, 12 skipped, 8 warnings in 2723.33s, exit 0
  • Mutation (mutmut, scoped per changed file) → 0 surviving non-equivalent mutants; every equivalent documented at the use site (UTF-8 codec-case, split/rsplit on single-} tags, get default behind an isinstance guard, the "auto" documentation default)

Completion Ledger

§13.1 item Evidence
A1 happy paths test_ingest_document::TestDocxHappyPath/TestConfluenceHappyPath (run output quoted above)
A2 edge cases table above — each mapped to a test
A3 failure paths (signal emission asserted) read/parse failures return ingested:false + error (exact-substring asserted); wiki-write failure logs exact message + sets wiki_path:null (test_wiki_write_failure_*)
A4 trust-boundary validation path/format validated in handler; malformed inputs tested
A5 invariants / partial-failure contracts on every new function; malformed input writes nothing (asserted)
A6 idempotency version-tag dedup; test_reingest_same_version_is_idempotent
B concurrency N/A — no shared mutable state, no async coordination introduced
C resources 1 MB read cap (sourced to AP MAX_PARSE_BYTES); single-pass parse
D security no injection surface (read-only parse); LLM/file content treated as untrusted; no secrets
E interfaces additive tool; tests_py/test_main.py + test_tool_profiles.py baselines bumped 51→52
F observability image-skip / empty / wiki-fail signals asserted; nominal path quiet
G tests mutation-tested; deterministic (unique tmp dirs); full suite green
H quality SOLID/layering/sizes (≤300 lines/file, ≤40 code-lines/method) / sourced constants / no anti-patterns; CHANGELOG + docs/mcp-tools.md + docs/module-inventory.md updated
H7 Boy-Scout no seen defects left untreated in touched files

Self-flagged risks

None blocking. The live-Confluence REST leg is enterprise-backlog#28 (explicitly out of scope); this PR delivers the offline export leg complete and the exact seam #28 will consume.

🤖 Generated with Claude Code

…192)

Add `ingest_document`, a standalone MCP tool that ingests a .docx (OOXML
zip, stdlib zipfile+xml — no heavyweight dependency) or a Confluence
storage-format XHTML export into the memory/wiki store.

Layering (Clean Architecture §2.2):
- core/ (pure, zero I/O): document_model, docx_parser, confluence_parser,
  document_normalizer — the shared parse→normalize seam.
- infrastructure/document_reader.py: file read + zip/XML unpacking.
- handlers/ingest_document.py (+_writers): composition root wiring the
  parsers to wiki_write/remember with provenance + idempotent re-ingest.

Every produced page/memory carries provenance (source + content-hash
version); re-ingesting the same version is a no-op. Headings, paragraphs,
tables extracted; embedded images skipped with an explicit notice (no OCR);
malformed zip/XML fails loudly, writing nothing.

The parse_confluence_storage → normalize_document → write path is the seam
the live-Confluence REST connector (enterprise-backlog#28) will consume.

Standalone tool count 51 → 52 (test_main, test_tool_profiles, mcp-tools.md,
module-inventory.md, CHANGELOG updated). Mutation-tested (mutmut) on all
changed files: 0 surviving non-equivalent mutants; equivalents documented
at the use site.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cdeust
cdeust merged commit 2c061b4 into main Jul 26, 2026
14 checks passed
@cdeust
cdeust deleted the feat/document-ingestion-192 branch July 26, 2026 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: document ingestion adapters — Confluence wiki and docx into memory/wiki pipeline

1 participant