[luisquintanilla]

Summary

Let MEDI process the shared Documents model without replacing IngestionDocument as its entry point or maintaining a second document model.

An extraction result can be passed directly into the existing ingestion pipeline:

ExtractionDocument extracted = await extractionClient.ExtractAsync(
    input, sourceId, mediaType, cancellationToken: cancellationToken);

var document = new IngestionDocument(ingestionId, extracted);
await pipeline.ProcessAsync(document, cancellationToken);

The ingestion identifier remains independent of the extracted document's identifier. The document graph is shared, not copied or converted through Markdown.

What changes

Area	Change
Ingestion root	Keep IngestionDocument, its existing string constructor, and the ingestion-root signatures on readers, processors, chunkers, chunks and pipeline results. Add a constructor accepting an ExtractionDocument and a get-only ExtractedDocument property.
Element model	Replace ingestion-specific sections/elements with the shared Documents primitives. Sections forwards the underlying List<ExtractionDocumentSection>; EnumerateContent() returns shared elements.
Built-in processing	Adapt all four public chunkers, image enrichment and Markdig/MarkItDown readers to the same graph. Preserve token budgets/overlap, table/image semantics and source attribution. Processor collections use concrete List<T>.
Extraction bridge	Add optional DocumentExtractionReader. It wraps the exact extraction result using the requested ingestion key, preserving inner object identity, borrowed input/client ownership, option snapshots and cancellation.
Direct ingestion and persistence	Allow an already-constructed document to run without a dummy reader. SourceElements and VectorStoreWriter<TRecord>.CreateRecord let applications persist source attribution with their records.

Dependency boundary: core ingestion references Documents, not Extraction. Only the optional extraction-reader package references the extraction contracts. Existing non-generic AIContent chunks, typed vector records/Guid keys and incremental writer behavior remain.

Compatibility

This retains the ingestion root, not complete legacy source/binary compatibility. The old ingestion-specific section/element types are replaced by shared primitives, and processor collection signatures change to concrete Lists. The original IngestionDocument(string identifier) constructor creates an empty shared document.

No separate legacy-mode tree, renderer or pipeline redesign is introduced.

Suggested review order

1. Envelope and public APIs: IngestionDocument, reader/processor/chunker signatures, pipeline overloads and regenerated API inventories.
2. Extraction boundary: DocumentExtractionReader, especially distinct ingestion/source identifiers and exact shared-graph identity.
3. Processing and persistence: built-in chunkers/processors, contributing elements and typed writer integration.

Merge gate: Documents #7835 and Extraction #7588 must reach data-ingestion-preview2. Current-head CI fails during restore because those prerequisite projects are missing from the target. This PR is not merge-ready until they are integrated and required checks pass.

Validation and package details

Source and API validation

• 642 offline test executions pass: 163/163/163/153 on net10/net9/net8/net462, with zero failed/skipped/live cases selected. net462 retains the existing SQLite exclusion.
• Production builds cover net10/net9/net8/net462/netstandard2.0. The nonincremental test-project build reports zero warnings/errors.
• All five API inventories were regenerated for the retained roots, new envelope and concrete Lists, and independently match packed binary inventories.
• Narrow CA1002 attributes document the explicitly approved concrete public Lists. Existing ApiChief native-delta string ApiStage limitations, tool-only warnings and support-project RT0000 notices are separate from owning source-build results.

Actual-package consumers

• Manual ingestion and the optional extraction bridge each pass on all four runnable targets: eight executions, with 40 output-DLL hashes matching archives, including20 canonical core matches.
• Manual ingestion exercises all four built-in chunkers and typed storage, with no Extraction package, output DLL or loaded assembly.
• Bridge checks use different request/provider identifiers and a second ingestion key over the same exact graph, alongside ownership, snapshots and cancellation checks.
• An independent package-only net10 consumer passes 38 envelope checks; five runtime DLL hashes match the actual owning/core/support archives. Consumer SDK10.0.403/runtime10.0.14.
• Existing net462 package support-policy notices remain unsuppressed: three manual/four bridge, with no compiler errors. No live services were exercised.

Private review packages

Five original owning MEDI projects use fresh immutable version 10.10.0-review.20261102.2. Owning dependency edges use .2; Documents/Extraction/AI/AI.Abstractions remain 10.10.0-review.20261102.1, including byte-for-byte unchanged AI main support.

Ten owning main/symbol archives, five-TFM nuspec dependency groups and 25 main-package DLL entries are sealed. Original projects/targets use a private DirectoryBuildTargetsPath import selecting PackageVersion per project. There is no archive rewriting, shadow project, core repack or committed version-manifest change.

Private owning nuspec RepositoryCommit records the packaging base; exact accepted source blobs bind to the published head rather than claiming uncommitted edits existed in that base. Earlier package bytes remain unchanged. These are review candidates, not a public feed release.

Prerequisites, local reproduction and CI diagnosis

Published source

Owning head: f5a1e88adb328aa594a91e6e4111b74f93a307b3.

Tree: ef9228908d9d3cc52b14fab378d37540b19a36ae.

Normal coauthored fast-forward publication from parent 426b44dedcfd7e30dd81fd9878f3c93934aeaecc: 54 focused files in this commit, 49 net owning files against the target. All 54 committed source/test/doc/API blobs match accepted inputs. No private prerequisite snapshots, review inputs, packs, notes or version manifests are committed.

Reproduce local validation before prerequisites land

The approved target is data-ingestion-preview2 at e124c123afeeda2f271f3b99a70eb3cfe187a471. Restore only these pinned prerequisite production paths into the worktree; do not commit them:

git fetch --no-tags https://github.com/luisquintanilla/extensions.git ee9ee634c9f1e649f409e0d2cf0efd9318437d90 0f69c5cd069e2a8bc53e5bc0021c6825b4d35726
git restore --worktree --source ee9ee634c9f1e649f409e0d2cf0efd9318437d90 -- src\Libraries\Microsoft.Extensions.Documents.Abstractions
git restore --worktree --source 0f69c5cd069e2a8bc53e5bc0021c6825b4d35726 -- src\Libraries\Microsoft.Extensions.DocumentExtraction.Abstractions

The 21 required production files match public Git blobs and remain outside the focused owning commit. Remove this temporary prerequisite-restoration step after the real dependencies reach the target, reconcile and rerun required checks. Private packages are not a substitute for integrating those public-source prerequisites.

Current-head CI

Azure build1660213 completed/failed. It evaluates PR merge revision f4d2a0694d5d6481ee1b5f219746b74d4c9162ae, with exact parents approved base e124c123afeeda2f271f3b99a70eb3cfe187a471 and owning head f5a1e88adb328aa594a91e6e4111b74f93a307b3.

Actual Ubuntu Restore log15, Windows Restore log27 and WarningsCheck Restore log32 each report two MSB4025 errors for missing Microsoft.Extensions.Documents.Abstractions.csproj and Microsoft.Extensions.DocumentExtraction.Abstractions.csproj, with zero MSBuild warnings. These jobs do not establish later build/test/AOT/pack success. This is new-head evidence, not relabeled previous builds1660161/1658900.

Documents build1660145 and Extraction build1660146 complete successfully at their published heads. Their CI does not integrate prerequisites into this target or waive this PR's required checks. No CI workaround or warning suppression for missing prerequisites is added.

Microsoft Reviewers: Open in CodeFlow

[luisquintanilla]

@adamsitnik - This builds on #7835 and #7588

But it'd be good to take a look as well to ensure you agree with these changes in MEDI.

These target the preview-2 branch