feat(ingest): document ingestion adapters — docx + Confluence export (#192)#194
Merged
Conversation
…192) Add `ingest_document`, a standalone MCP tool that ingests a .docx (OOXML zip, stdlib zipfile+xml — no heavyweight dependency) or a Confluence storage-format XHTML export into the memory/wiki store. Layering (Clean Architecture §2.2): - core/ (pure, zero I/O): document_model, docx_parser, confluence_parser, document_normalizer — the shared parse→normalize seam. - infrastructure/document_reader.py: file read + zip/XML unpacking. - handlers/ingest_document.py (+_writers): composition root wiring the parsers to wiki_write/remember with provenance + idempotent re-ingest. Every produced page/memory carries provenance (source + content-hash version); re-ingesting the same version is a no-op. Headings, paragraphs, tables extracted; embedded images skipped with an explicit notice (no OCR); malformed zip/XML fails loudly, writing nothing. The parse_confluence_storage → normalize_document → write path is the seam the live-Confluence REST connector (enterprise-backlog#28) will consume. Standalone tool count 51 → 52 (test_main, test_tool_profiles, mcp-tools.md, module-inventory.md, CHANGELOG updated). Mutation-tested (mutmut) on all changed files: 0 surviving non-equivalent mutants; equivalents documented at the use site. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #192.
Summary
Adds
ingest_document, a standalone (file-only, no upstream) MCP tool that ingests a .docx (OOXML zip) or a Confluence storage-format XHTML export into Cortex's memory/wiki store, on the same provenance + staleness path as code references. docx is parsed with the stdlib (zipfile+xml.etree) — no heavyweight dependency (§8). Standalone tool count 51 → 52.Layer assignment (Clean Architecture §2.2)
core/document_model.py,core/docx_parser.py,core/confluence_parser.py,core/document_normalizer.pyParsedDocument→ wiki/memory payloadsinfrastructure/document_reader.pyDocumentReadErrorhandlers/ingest_document.py,handlers/ingest_document_writers.pywiki_write/remembertool_registry_ingest.pyingest_documentunconditionally (file-only)Import direction verified: no
core → infrastructure/handlers, noinfrastructure → core.The shared seam #28 consumes
core.confluence_parser.parse_confluence_storage(xhtml)→core.document_normalizer.normalize_document(doc, prov)→handlers.wiki_write.write_governed_page+ingest_document_writers.write_document_memories. The live-Confluence REST connector (enterprise-backlog#28, OUT of scope here) reuses every line of parsing/normalization/writing — it swaps only the byte source (REST fetch instead ofread_confluence_export) and the provenance (DocumentProvenance(source=page URL, version=page version, source_kind="confluence-api")).Stakes (Move 7)
High — new public MCP tool surface + data-write path. Full discipline applied.
Edge cases (§13.1-A2 — each mapped to a test)
test_document_normalizer::TestEmptyDocument,test_ingest_document::TestEmptyDocumenttest_docx_parser::test_headings_only_document,test_confluence_parser::test_headings_onlytest_docx_parser::TestTables,test_confluence_parser::TestTables,test_document_normalizer::TestTableRenderingtest_document_normalizer::TestImageNotice,test_ingest_document::TestImagesSkippedtest_document_reader::*,test_ingest_document::test_malformed_{zip,xml}_writes_nothingtest_ingest_document::test_reingest_same_version_is_idempotentGates (quoted)
ruff check→All checks passed!;ruff format --check→1017 files already formattedpyright(changed files) → 0 errors (tracked delta not increased)pytest→5559 passed, 12 skipped, 8 warnings in 2723.33s, exit 0}tags,getdefault behind an isinstance guard, the"auto"documentation default)Completion Ledger
test_ingest_document::TestDocxHappyPath/TestConfluenceHappyPath(run output quoted above)ingested:false+error(exact-substring asserted); wiki-write failure logs exact message + setswiki_path:null(test_wiki_write_failure_*)test_reingest_same_version_is_idempotentMAX_PARSE_BYTES); single-pass parsetests_py/test_main.py+test_tool_profiles.pybaselines bumped 51→52docs/mcp-tools.md+docs/module-inventory.mdupdatedSelf-flagged risks
None blocking. The live-Confluence REST leg is enterprise-backlog#28 (explicitly out of scope); this PR delivers the offline export leg complete and the exact seam #28 will consume.
🤖 Generated with Claude Code