Repository navigation
Add document extraction providers for PdfPig, Azure, Mistral and VisionLanguageModel - #82
Merged
luisquintanilla merged 19 commits intoNov 7, 2026
Merged
Conversation
Implement PdfPig, Azure Content Understanding, Azure Document Intelligence, Mistral, and Vision adapters with offline acceptance and a pinned public-source core bootstrap. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 2, 2026
Consume the sealed 20261102.1 core, remove shared response-capture graph, preserve source frames and native inspection, and retain sole text payloads with independent image descriptions and resource safety. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Map explicit Azure cell children with validated deferred ownership. Migrate all five providers to DocumentExtractionClient while preserving service lookup and resource ownership. Document the preview break and pending public dependency coordination. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pin the integrated public source commit and tree, build four original core packages at fresh review version .4, and synchronize provider dependency and same-run artifact selection. Preserve native provider dependencies and document source-built preview provenance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…icitly Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Defer the entire advanced PdfPig companion, retain CU Markdown and strict Vision table mapping fixes, and document the public-source bootstrap and reviewer entry points. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
luisquintanilla
changed the title
luisquintanilla
changed the title
Retain net8/net10, add netstandard2.0 using Toolkit legacy support, and validate actual standard provider assets, package consumers and CTDE0001 diagnostics. Replace review-only guidance with product-facing setup and examples. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Rename the package, assembly, namespace, client, options, project, tests and bundled skill without compatibility aliases. Preserve extraction behavior and vision-llm source-frame identities. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
luisquintanilla
changed the title
Laurianti
reviewed
Nov 6, 2026
|
|
||
| if (!parents.ContainsKey(section)) | ||
| { | ||
| ProviderAdapterHelpers.GetSection(document, section.Sources).Elements.Add(section); |
Member
There was a problem hiding this comment.
EnumerateContent() is out of reading order when the result has sections. Both Azure providers go through this code.
Paragraphs that no section lists stay in their page section. This line then appends the top-level native section at the end of the page 1 section, because GetSection takes the first source page. In every Document Intelligence and Content Understanding layout result I checked, including the Azure SDK recordings, pageHeader, pageFooter and pageNumber paragraphs are not in sections[].elements. So this happens on every document with page headers or footers.
The Azure SDK recording of prebuilt-layout (GetAnalyzeResultFiguresAsync.json), replayed through DocumentIntelligenceExtractionClient:
- now: the page header, the 20 chart labels of the figure, the page footer and the page number come first. The title is at index 23.
- with the fix below: the page header, the title, the body in order with the chart labels right after the figure, then the page footer and the page number.
With two pages, and sections[0] listing the title and the two body paragraphs:
- expected:
Title | Body one | Page 1 of 2 | Header two | Body two - actual:
Page 1 of 2 | Title | Body one | Body two | Header two
Suggested fix: move each element that no section lists into the deepest native section that spans its offset, then order each page section by offset.
Tested on 8edcb9c, .NET 8 and .NET 10. Without the fix the test below fails for both providers. With the fix all tests pass, this one included: 109/109 for Document Intelligence and 108/108 for Content Understanding.
Fix
diff --git a/DocumentExtraction/src/AzureContentUnderstanding/ContentUnderstandingExtractionClient.cs b/DocumentExtraction/src/AzureContentUnderstanding/ContentUnderstandingExtractionClient.cs
--- a/DocumentExtraction/src/AzureContentUnderstanding/ContentUnderstandingExtractionClient.cs
+++ b/DocumentExtraction/src/AzureContentUnderstanding/ContentUnderstandingExtractionClient.cs
@@ -154,7 +154,7 @@ public ContentUnderstandingExtractionClient(
MapParagraphs(output, content, frames, elements);
MapTables(output, content, frames, elements);
MapFigures(output, content, frames, elements);
- OrderContent(output, content, elements);
+ IReadOnlyDictionary<ExtractionDocumentElement, int> offsets = OrderContent(output, content, elements);
if (content.Paragraphs.Count == 0)
{
foreach (var page in content.Pages)
@@ -174,6 +174,7 @@ public ContentUnderstandingExtractionClient(
}
}
MapSections(output, content, frames, elements);
+ SectionHierarchy.Place(output, offsets);
if (elements.Count == 0 && !content.Pages.Any(page => page.Lines.Count > 0) &&
!string.IsNullOrEmpty(content.Markdown))
{
@@ -305,7 +306,7 @@ private static void MapSections(
SectionHierarchy.Apply(output, elements, sections, cells);
}
- private static void OrderContent(
+ private static IReadOnlyDictionary<ExtractionDocumentElement, int> OrderContent(
ExtractionDocument output, DocumentContent content,
IReadOnlyDictionary<string, ExtractionDocumentElement> elements)
{
@@ -326,6 +327,7 @@ private static void OrderContent(
}
SectionHierarchy.Order(output, offsets);
+ return offsets;
}
private static IReadOnlyList<ExtractionDocumentSourceReference> ParseSources(
diff --git a/DocumentExtraction/src/AzureDocumentIntelligence/DocumentIntelligenceExtractionClient.cs b/DocumentExtraction/src/AzureDocumentIntelligence/DocumentIntelligenceExtractionClient.cs
--- a/DocumentExtraction/src/AzureDocumentIntelligence/DocumentIntelligenceExtractionClient.cs
+++ b/DocumentExtraction/src/AzureDocumentIntelligence/DocumentIntelligenceExtractionClient.cs
@@ -133,7 +133,7 @@ await MapImagesAsync(
frames,
elements,
cancellationToken).ConfigureAwait(false);
- OrderContent(output, result, elements);
+ IReadOnlyDictionary<ExtractionDocumentElement, int> offsets = OrderContent(output, result, elements);
if (result.Paragraphs.Count == 0)
{
foreach (var page in result.Pages)
@@ -147,6 +147,7 @@ await MapImagesAsync(
}
}
MapSections(output, result, frames, elements);
+ SectionHierarchy.Place(output, offsets);
cancellationToken.ThrowIfCancellationRequested();
return output;
@@ -299,7 +300,7 @@ private static void MapSections(
SectionHierarchy.Apply(output, elements, sections, cells);
}
- private static void OrderContent(
+ private static IReadOnlyDictionary<ExtractionDocumentElement, int> OrderContent(
ExtractionDocument output, AnalyzeResult result,
IReadOnlyDictionary<string, ExtractionDocumentElement> elements)
{
@@ -320,6 +321,7 @@ private static void OrderContent(
}
SectionHierarchy.Order(output, offsets);
+ return offsets;
}
diff --git a/DocumentExtraction/src/Shared/SectionHierarchy.cs b/DocumentExtraction/src/Shared/SectionHierarchy.cs
--- a/DocumentExtraction/src/Shared/SectionHierarchy.cs
+++ b/DocumentExtraction/src/Shared/SectionHierarchy.cs
@@ -24,6 +24,90 @@ public static void Order(ExtractionDocument document, IReadOnlyDictionary<Extrac
}
}
+ // Elements that no native section lists, such as page headers, footers and page numbers, stay in their
+ // page section, while each top-level native section sits in the section of its first page. Move them into
+ // the deepest native section that spans their offset, then order every page section by offset.
+ public static void Place(ExtractionDocument document, IReadOnlyDictionary<ExtractionDocumentElement, int> offsets)
+ {
+ var ranges = new Dictionary<ExtractionDocumentSection, (int Start, int End)>();
+ ExtractionDocumentSection[] natives = document.Sections
+ .SelectMany(root => root.Elements.OfType<ExtractionDocumentSection>())
+ .ToArray();
+ foreach (ExtractionDocumentSection native in natives)
+ {
+ Measure(native);
+ }
+
+ foreach (ExtractionDocumentSection root in document.Sections)
+ {
+ foreach (ExtractionDocumentElement element in root.Elements.ToArray())
+ {
+ if (element is ExtractionDocumentSection
+ || !offsets.TryGetValue(element, out int offset)
+ || offset == int.MaxValue
+ || natives.Select(native => Deepest(native, offset)).FirstOrDefault(found => found is not null)
+ is not { } target)
+ {
+ continue;
+ }
+
+ root.Elements.Remove(element);
+ int index = target.Elements.FindIndex(child => Position(child) > offset);
+ target.Elements.Insert(index < 0 ? target.Elements.Count : index, element);
+ }
+
+ ExtractionDocumentElement[] ordered = root.Elements.OrderBy(Position).ToArray();
+ root.Elements.Clear();
+ root.Elements.AddRange(ordered);
+ }
+
+ (int Start, int End)? Measure(ExtractionDocumentSection section)
+ {
+ (int Start, int End)? range = null;
+ foreach (ExtractionDocumentElement child in section.Elements)
+ {
+ (int Start, int End)? childRange = child is ExtractionDocumentSection nested ? Measure(nested)
+ : offsets.TryGetValue(child, out int offset) && offset != int.MaxValue ? (offset, offset) : null;
+ if (childRange is { } value)
+ {
+ range = range is { } current
+ ? (Math.Min(current.Start, value.Start), Math.Max(current.End, value.End))
+ : value;
+ }
+ }
+
+ if (range is { } measured)
+ {
+ ranges[section] = measured;
+ }
+
+ return range;
+ }
+
+ ExtractionDocumentSection? Deepest(ExtractionDocumentSection section, int offset)
+ {
+ if (!ranges.TryGetValue(section, out (int Start, int End) range) || offset < range.Start || offset > range.End)
+ {
+ return null;
+ }
+
+ foreach (ExtractionDocumentSection nested in section.Elements.OfType<ExtractionDocumentSection>())
+ {
+ if (Deepest(nested, offset) is { } found)
+ {
+ return found;
+ }
+ }
+
+ return section;
+ }
+
+ int Position(ExtractionDocumentElement element)
+ => element is ExtractionDocumentSection section && ranges.TryGetValue(section, out (int Start, int End) range)
+ ? range.Start
+ : offsets.TryGetValue(element, out int offset) ? offset : int.MaxValue;
+ }
+
public static void Apply(
ExtractionDocument document,
IReadOnlyDictionary<string, ExtractionDocumentElement> elements,Test for AzureProviderContractTests.cs
[Fact]
public async Task ExtractAsync_TwoPages_KeepsReadingOrder()
{
var root = AzureWireFixture.Create();
var content = AzureWireFixture.Content(root);
content["tables"] = new JsonArray();
content["figures"] = new JsonArray();
(string Text, string? Role, int Page, int Offset)[] paragraphs =
[
("Title", "title", 1, 0),
("Body one", null, 1, 10),
("Page 1 of 2", "pageFooter", 1, 25),
("Header two", "pageHeader", 2, 40),
("Body two", null, 2, 55),
];
char[] text = new string(' ', 80).ToCharArray();
var nodes = new JsonArray();
foreach (var (value, role, page, offset) in paragraphs)
{
value.CopyTo(0, text, offset, value.Length);
var paragraph = new JsonObject { ["content"] = value };
if (role is not null)
{
paragraph["role"] = role;
}
var span = new JsonObject { ["offset"] = offset, ["length"] = value.Length };
if (AzureWireFixture.IsContentUnderstanding)
{
paragraph["span"] = span;
paragraph["source"] = $"D({page},1,2,3,2,3,4,2,5,1,4)";
}
else
{
paragraph["spans"] = new JsonArray(span);
paragraph["boundingRegions"] = JsonSerializer.SerializeToNode(new[] { new { pageNumber = page, polygon = AzureWireFixture.Coordinates } });
}
nodes.Add(paragraph);
}
content["paragraphs"] = nodes;
content[AzureWireFixture.IsContentUnderstanding ? "markdown" : "content"] = new string(text);
var pages = content["pages"]!.AsArray();
var second = pages[0]!.DeepClone();
pages[0]!["pageNumber"] = 1;
pages[0]!["spans"] = JsonSerializer.SerializeToNode(new[] { new { offset = 0, length = 40 } });
second["pageNumber"] = 2;
second["spans"] = JsonSerializer.SerializeToNode(new[] { new { offset = 40, length = 40 } });
pages.Add(second);
// As the services return it: page headers, footers and numbers are not section elements.
content["sections"] = new JsonArray(AzureWireFixture.Section("/paragraphs/0", "/paragraphs/1", "/paragraphs/4"));
var transport = Replay(AzureWireFixture.Bytes(root));
using var client = Create(transport);
using var input = new MemoryStream([1, 2, 3]);
var result = await Extract(client, input);
Assert.Equal(
"Title | Body one | Page 1 of 2 | Header two | Body two",
string.Join(" | ", result.EnumerateContent().Select(element => ((TextContent)((ExtractionDocumentTextElement)element).Content!).Text)));
}
Comment on lines
+223
to
+296
| var replacements = new List<(int Start, int Length, ExtractionDocumentTableElement Table)>(); | ||
| var unplacedTables = new List<ExtractionDocumentTableElement>(); | ||
|
|
||
| if (page.TryGetProperty("tables", out JsonElement tables) | ||
| && tables.ValueKind == JsonValueKind.Array) | ||
| { | ||
| foreach (JsonElement table in tables.EnumerateArray()) | ||
| { | ||
| string? tableMarkdown = GetString(table, "content") ?? GetString(table, "markdown"); | ||
| string? tableId = GetString(table, "id"); | ||
| (int Start, int Length)? match = string.IsNullOrEmpty(tableId) | ||
| ? null | ||
| : FindUnique(markdown, $"[{tableId}]({tableId})"); | ||
| if (match is null && !string.IsNullOrEmpty(tableMarkdown)) | ||
| { | ||
| match = FindUnique(markdown, tableMarkdown!); | ||
| } | ||
|
|
||
| if (match is { } candidate && replacements.Any(replacement => | ||
| replacement.Start < candidate.Start + candidate.Length | ||
| && candidate.Start < replacement.Start + replacement.Length)) | ||
| { | ||
| match = null; | ||
| } | ||
|
|
||
| var extractionTable = new ExtractionDocumentTableElement(tableMarkdown is null ? null : new TextContent(tableMarkdown), 0, 0); | ||
| extractionTable.Sources.Add(new ExtractionDocumentSourceReference(sourcePage, rawRepresentation: table.Clone())); | ||
| if (match is { } position) | ||
| { | ||
| replacements.Add((position.Start, position.Length, extractionTable)); | ||
| } | ||
| else | ||
| { | ||
| unplacedTables.Add(extractionTable); | ||
| } | ||
|
|
||
| } | ||
| } | ||
|
|
||
| int cursor = 0; | ||
| foreach ((int start, int length, ExtractionDocumentTableElement table) in | ||
| replacements.OrderBy(item => item.Start)) | ||
| { | ||
| AddText(markdown.Substring(cursor, start - cursor)); | ||
| AddTable(table); | ||
| cursor = start + length; | ||
| } | ||
|
|
||
| AddText(markdown.Substring(cursor)); | ||
| foreach (ExtractionDocumentTableElement table in unplacedTables) | ||
| { | ||
| AddTable(table); | ||
| } | ||
|
|
||
| if (page.TryGetProperty("images", out JsonElement images) | ||
| && images.ValueKind == JsonValueKind.Array) | ||
| { | ||
| foreach (JsonElement image in images.EnumerateArray()) | ||
| { | ||
| ReadOnlyMemory<byte>? content = DecodeImage(image); | ||
| ExtractionDocumentSourceReference source = CreateImageSource(sourcePage, image); | ||
| ProviderAdapterHelpers.AddElement(section, new ExtractionDocumentImageElement() | ||
| { | ||
| Content = content is null ? null : new DataContent(content.Value, "image/png"), | ||
| }, [source]); | ||
| } | ||
| } | ||
|
|
||
| void AddTable(ExtractionDocumentTableElement table) | ||
| { | ||
| ExtractionDocumentSourceReference[] sources = table.Sources.ToArray(); | ||
| table.Sources.Clear(); | ||
| ProviderAdapterHelpers.AddElement(section, table, sources); | ||
| } |
Member
There was a problem hiding this comment.
Images are not placed where they are on the page, and their placeholder stays in the text.
Mistral marks each image in the page Markdown with a placeholder like . Tables are already split at their placeholder, images are not. So each image is added after all the text of its page, and the text keeps the placeholder.
I tested it on the live Mistral API (mistral-ocr-latest) with a one-page PDF: a title, a paragraph, a JPEG chart, a paragraph and a table. The response has  between the two paragraphs, and image_base64 starts with data:image/jpeg;base64,.
- With the current code,
EnumerateContent()returns one text element with the whole page, placeholder included, then the image asimage/png. - With this suggestion and the
DecodeImageone on line 429, it returns the text before the chart, the image asimage/jpeg, then the text after the chart.
Apply the two suggestions together: this one uses the new DecodeImage return type.
The test below uses the page Markdown of that response. Tested on 8edcb9c, .NET 8 and .NET 10: 154/155 without the fix (this test fails), 155/155 with it.
Test for MistralDocumentExtractionClientTests.cs
[Fact]
public async Task ExtractAsync_ImagePlaceholder_KeepsImageInReadingOrder()
{
// Page Markdown returned by mistral-ocr-latest for a PDF with a JPEG chart; image_base64 shortened.
const string markdown = "# Quarterly report\n\nSales grew in every quarter of the year. This paragraph comes before the chart.\n\n\n\nFigure 1: Revenue by quarter. This paragraph comes after the chart.\n\n## Details\n\nThe table below lists the revenue of each quarter.\n\n| Quarter | Revenue |\n| --- | --- |\n| Q1 | 120 |\n| Q2 | 180 |\n| Q3 | 150 |\n| Q4 | 220 |\n\nThis closing paragraph comes after the table.";
using var session = new OfflineSession(PageResponse(markdown, [], [new
{
id = "img-0.jpeg", image_base64 = "data:image/jpeg;base64,/9j/4AAQ",
top_left_x = 101, top_left_y = 184, bottom_right_x = 527, bottom_right_y = 440,
}]));
ExtractionDocument document = await Extract(session);
Assert.Collection(Assert.Single(document.Sections).Elements,
element => Assert.Equal("# Quarterly report\n\nSales grew in every quarter of the year. This paragraph comes before the chart.\n\n", Text(element)),
element => Assert.Equal("image/jpeg", Assert.IsType<DataContent>(Assert.IsType<ExtractionDocumentImageElement>(element).Content).MediaType),
element => Assert.Equal("\n\nFigure 1: Revenue by quarter. This paragraph comes after the chart.\n\n## Details\n\nThe table below lists the revenue of each quarter.\n\n| Quarter | Revenue |\n| --- | --- |\n| Q1 | 120 |\n| Q2 | 180 |\n| Q3 | 150 |\n| Q4 | 220 |\n\nThis closing paragraph comes after the table.", Text(element)));
await session.AssertCleanAndBorrowedAsync(1, 1);
}
Suggested change
| var replacements = new List<(int Start, int Length, ExtractionDocumentTableElement Table)>(); | |
| var unplacedTables = new List<ExtractionDocumentTableElement>(); | |
| if (page.TryGetProperty("tables", out JsonElement tables) | |
| && tables.ValueKind == JsonValueKind.Array) | |
| { | |
| foreach (JsonElement table in tables.EnumerateArray()) | |
| { | |
| string? tableMarkdown = GetString(table, "content") ?? GetString(table, "markdown"); | |
| string? tableId = GetString(table, "id"); | |
| (int Start, int Length)? match = string.IsNullOrEmpty(tableId) | |
| ? null | |
| : FindUnique(markdown, $"[{tableId}]({tableId})"); | |
| if (match is null && !string.IsNullOrEmpty(tableMarkdown)) | |
| { | |
| match = FindUnique(markdown, tableMarkdown!); | |
| } | |
| if (match is { } candidate && replacements.Any(replacement => | |
| replacement.Start < candidate.Start + candidate.Length | |
| && candidate.Start < replacement.Start + replacement.Length)) | |
| { | |
| match = null; | |
| } | |
| var extractionTable = new ExtractionDocumentTableElement(tableMarkdown is null ? null : new TextContent(tableMarkdown), 0, 0); | |
| extractionTable.Sources.Add(new ExtractionDocumentSourceReference(sourcePage, rawRepresentation: table.Clone())); | |
| if (match is { } position) | |
| { | |
| replacements.Add((position.Start, position.Length, extractionTable)); | |
| } | |
| else | |
| { | |
| unplacedTables.Add(extractionTable); | |
| } | |
| } | |
| } | |
| int cursor = 0; | |
| foreach ((int start, int length, ExtractionDocumentTableElement table) in | |
| replacements.OrderBy(item => item.Start)) | |
| { | |
| AddText(markdown.Substring(cursor, start - cursor)); | |
| AddTable(table); | |
| cursor = start + length; | |
| } | |
| AddText(markdown.Substring(cursor)); | |
| foreach (ExtractionDocumentTableElement table in unplacedTables) | |
| { | |
| AddTable(table); | |
| } | |
| if (page.TryGetProperty("images", out JsonElement images) | |
| && images.ValueKind == JsonValueKind.Array) | |
| { | |
| foreach (JsonElement image in images.EnumerateArray()) | |
| { | |
| ReadOnlyMemory<byte>? content = DecodeImage(image); | |
| ExtractionDocumentSourceReference source = CreateImageSource(sourcePage, image); | |
| ProviderAdapterHelpers.AddElement(section, new ExtractionDocumentImageElement() | |
| { | |
| Content = content is null ? null : new DataContent(content.Value, "image/png"), | |
| }, [source]); | |
| } | |
| } | |
| void AddTable(ExtractionDocumentTableElement table) | |
| { | |
| ExtractionDocumentSourceReference[] sources = table.Sources.ToArray(); | |
| table.Sources.Clear(); | |
| ProviderAdapterHelpers.AddElement(section, table, sources); | |
| } | |
| var replacements = new List<(int Start, int Length, ExtractionDocumentElement Element)>(); | |
| var unplaced = new List<ExtractionDocumentElement>(); | |
| if (page.TryGetProperty("tables", out JsonElement tables) | |
| && tables.ValueKind == JsonValueKind.Array) | |
| { | |
| foreach (JsonElement table in tables.EnumerateArray()) | |
| { | |
| string? tableMarkdown = GetString(table, "content") ?? GetString(table, "markdown"); | |
| string? tableId = GetString(table, "id"); | |
| (int Start, int Length)? match = string.IsNullOrEmpty(tableId) | |
| ? null | |
| : FindUnique(markdown, $"[{tableId}]({tableId})"); | |
| if (match is null && !string.IsNullOrEmpty(tableMarkdown)) | |
| { | |
| match = FindUnique(markdown, tableMarkdown!); | |
| } | |
| if (match is { } candidate && replacements.Any(replacement => | |
| replacement.Start < candidate.Start + candidate.Length | |
| && candidate.Start < replacement.Start + replacement.Length)) | |
| { | |
| match = null; | |
| } | |
| var extractionTable = new ExtractionDocumentTableElement(tableMarkdown is null ? null : new TextContent(tableMarkdown), 0, 0); | |
| extractionTable.Sources.Add(new ExtractionDocumentSourceReference(sourcePage, rawRepresentation: table.Clone())); | |
| if (match is { } position) | |
| { | |
| replacements.Add((position.Start, position.Length, extractionTable)); | |
| } | |
| else | |
| { | |
| unplaced.Add(extractionTable); | |
| } | |
| } | |
| } | |
| if (page.TryGetProperty("images", out JsonElement images) | |
| && images.ValueKind == JsonValueKind.Array) | |
| { | |
| foreach (JsonElement image in images.EnumerateArray()) | |
| { | |
| var extractionImage = new ExtractionDocumentImageElement() { Content = DecodeImage(image) }; | |
| extractionImage.Sources.Add(CreateImageSource(sourcePage, image)); | |
| string? imageId = GetString(image, "id"); | |
| (int Start, int Length)? match = string.IsNullOrEmpty(imageId) | |
| ? null | |
| : FindUnique(markdown, $""); | |
| if (match is { } position && !replacements.Any(replacement => | |
| replacement.Start < position.Start + position.Length | |
| && position.Start < replacement.Start + replacement.Length)) | |
| { | |
| replacements.Add((position.Start, position.Length, extractionImage)); | |
| } | |
| else | |
| { | |
| unplaced.Add(extractionImage); | |
| } | |
| } | |
| } | |
| int cursor = 0; | |
| foreach ((int start, int length, ExtractionDocumentElement element) in | |
| replacements.OrderBy(item => item.Start)) | |
| { | |
| AddText(markdown.Substring(cursor, start - cursor)); | |
| AddPlaced(element); | |
| cursor = start + length; | |
| } | |
| AddText(markdown.Substring(cursor)); | |
| foreach (ExtractionDocumentElement element in unplaced) | |
| { | |
| AddPlaced(element); | |
| } | |
| void AddPlaced(ExtractionDocumentElement element) | |
| { | |
| ExtractionDocumentSourceReference[] sources = element.Sources.ToArray(); | |
| element.Sources.Clear(); | |
| ProviderAdapterHelpers.AddElement(section, element, sources); | |
| } |
Comment on lines
+429
to
+440
| private static ReadOnlyMemory<byte>? DecodeImage(JsonElement image) | ||
| { | ||
| if (!image.TryGetProperty("image_base64", out JsonElement base64) | ||
| || base64.ValueKind != JsonValueKind.String) | ||
| { | ||
| return null; | ||
| } | ||
|
|
||
| string raw = base64.GetString()!; | ||
| int comma = raw.IndexOf(','); | ||
| return Convert.FromBase64String(comma >= 0 ? raw.Substring(comma + 1) : raw); | ||
| } |
Member
There was a problem hiding this comment.
image_base64 is a data URI that carries the media type, and DecodeImage drops it. Line 286 then sets image/png for every image. On the live API, the JPEG chart above comes back as data:image/jpeg;base64,... and is reported as image/png.
Suggested fix: pass data URIs to new DataContent(raw), which reads the media type. Invalid base64 is still rejected.
Suggested change
| private static ReadOnlyMemory<byte>? DecodeImage(JsonElement image) | |
| { | |
| if (!image.TryGetProperty("image_base64", out JsonElement base64) | |
| || base64.ValueKind != JsonValueKind.String) | |
| { | |
| return null; | |
| } | |
| string raw = base64.GetString()!; | |
| int comma = raw.IndexOf(','); | |
| return Convert.FromBase64String(comma >= 0 ? raw.Substring(comma + 1) : raw); | |
| } | |
| private static DataContent? DecodeImage(JsonElement image) | |
| { | |
| if (!image.TryGetProperty("image_base64", out JsonElement base64) | |
| || base64.ValueKind != JsonValueKind.String) | |
| { | |
| return null; | |
| } | |
| string raw = base64.GetString()!; | |
| return raw.StartsWith("data:", StringComparison.OrdinalIgnoreCase) | |
| ? new DataContent(raw) | |
| : new DataContent(Convert.FromBase64String(raw), "image/png"); | |
| } |
Laurianti
reviewed
Nov 6, 2026
| var image = new ExtractionDocumentImageElement() | ||
| { | ||
| Content = content is null ? null : new DataContent(content.Value, "image/png"), | ||
| AlternativeText = figure.Caption?.Content, |
Member
There was a problem hiding this comment.
ExtractionDocumentImageElement.Text is never set, and the text the service recognizes inside a figure comes out as loose paragraphs. Both Azure providers do this.
Document Intelligence and Content Understanding list that text (axis labels, values) in figures[].elements, and no section lists it. The abstraction says Text is "OCR or transcription text associated with the image", and the README says "Text is recognized image text, when supplied".
I tested it on the live services, with prebuilt-layout on Document Intelligence (2024-11-30) and on Content Understanding (2025-11-01), using a two-page PDF with a bar chart:
- now:
Textis null, and the 9 chart labels (Revenue by quarter (k EUR),220, ...,Q4) come out as 9 text elements. - with the fix below:
TextisRevenue by quarter (k EUR)\n220\n180\n150\n120\nQ1\nQ2\nQ3\nQ4, and the labels are no longer loose text.
Suggested fix: join the paragraphs that figures[].elements points to into Text, and take them out of the page flow.
Tested on 8edcb9c, .NET 8 and .NET 10. Without the fix the test below fails for both providers. With the fix all tests pass, this one included: 109/109 for Document Intelligence and 108/108 for Content Understanding. It also applies together with the reading-order fix in my other comment: with both, 110/110 and 109/109.
Fix
diff --git a/DocumentExtraction/src/AzureContentUnderstanding/ContentUnderstandingExtractionClient.cs b/DocumentExtraction/src/AzureContentUnderstanding/ContentUnderstandingExtractionClient.cs
--- a/DocumentExtraction/src/AzureContentUnderstanding/ContentUnderstandingExtractionClient.cs
+++ b/DocumentExtraction/src/AzureContentUnderstanding/ContentUnderstandingExtractionClient.cs
@@ -261,6 +261,7 @@ private static void MapFigures(
var element = new ExtractionDocumentImageElement()
{
AlternativeText = figure.Description,
+ Text = ProviderAdapterHelpers.TakeFigureText(output, figure.Elements, elements),
};
elements.Add($"/figures/{index}", element);
ProviderAdapterHelpers.AddElement(ProviderAdapterHelpers.GetSection(output, sources), element, sources);
diff --git a/DocumentExtraction/src/AzureDocumentIntelligence/DocumentIntelligenceExtractionClient.cs b/DocumentExtraction/src/AzureDocumentIntelligence/DocumentIntelligenceExtractionClient.cs
--- a/DocumentExtraction/src/AzureDocumentIntelligence/DocumentIntelligenceExtractionClient.cs
+++ b/DocumentExtraction/src/AzureDocumentIntelligence/DocumentIntelligenceExtractionClient.cs
@@ -254,6 +254,7 @@ private async Task MapImagesAsync(
{
Content = content is null ? null : new DataContent(content.Value, "image/png"),
AlternativeText = figure.Caption?.Content,
+ Text = ProviderAdapterHelpers.TakeFigureText(output, figure.Elements, elements),
};
elements.Add($"/figures/{index}", image);
diff --git a/DocumentExtraction/src/Shared/ProviderAdapterHelpers.cs b/DocumentExtraction/src/Shared/ProviderAdapterHelpers.cs
--- a/DocumentExtraction/src/Shared/ProviderAdapterHelpers.cs
+++ b/DocumentExtraction/src/Shared/ProviderAdapterHelpers.cs
@@ -1,6 +1,7 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
+using Microsoft.Extensions.AI;
using Microsoft.Extensions.DocumentExtraction;
using Microsoft.Extensions.Documents;
@@ -48,6 +49,26 @@ public static ExtractionDocumentSection GetSection(
return section;
}
+ // The paragraphs a provider lists inside a figure (axis labels, values) are the image's recognized text.
+ // Take them out of the page flow so they are not repeated as loose text.
+ public static string? TakeFigureText(
+ ExtractionDocument document, IEnumerable<string>? pointers, IReadOnlyDictionary<string, ExtractionDocumentElement> elements)
+ {
+ var lines = new List<string>();
+ foreach (string pointer in pointers ?? [])
+ {
+ if (pointer.StartsWith("/paragraphs/", StringComparison.Ordinal)
+ && elements.TryGetValue(pointer, out ExtractionDocumentElement? element)
+ && element is ExtractionDocumentTextElement { Content: TextContent text })
+ {
+ lines.Add(text.Text);
+ GetSection(document, element.Sources).Elements.Remove(element);
+ }
+ }
+
+ return lines.Count == 0 ? null : string.Join("\n", lines);
+ }
+
public static DocumentExtractionOptions NormalizeOptions(DocumentExtractionOptions? options)
=> options?.Clone() ?? new DocumentExtractionOptions();
Test for AzureProviderContractTests.cs
[Fact]
public async Task ExtractAsync_FigureElements_BecomeImageText()
{
var root = AzureWireFixture.Create();
var content = AzureWireFixture.Content(root);
content["tables"] = new JsonArray();
var paragraphs = content["paragraphs"]!.AsArray();
foreach (string label in new[] { "Q1", "Q2" })
{
var paragraph = paragraphs[0]!.DeepClone().AsObject();
paragraph["content"] = label;
paragraph.Remove("role");
paragraphs.Add(paragraph);
}
// As the services return it: the text inside a figure is listed in figures[].elements, not in any section.
content["figures"]![0]!["elements"] = new JsonArray("/paragraphs/1", "/paragraphs/2");
content["sections"] = new JsonArray(AzureWireFixture.Section("/paragraphs/0", "/figures/0"));
var transport = Replay(AzureWireFixture.Bytes(root));
using var client = Create(transport);
using var input = new MemoryStream([1, 2, 3]);
var result = await Extract(client, input);
var image = Assert.Single(result.EnumerateContent().OfType<ExtractionDocumentImageElement>());
Assert.Equal("Q1\nQ2", image.Text);
Assert.Equal(new[] { "Invoice heading" },
result.EnumerateContent().OfType<ExtractionDocumentTextElement>().Select(text => ((TextContent)text.Content!).Text));
}Preserve page-local native Azure hierarchy and conservative figure OCR ownership. Resolve Mistral image/table placeholders without ambiguous claims and retain declared data-URI MIME and exact bytes. Add offline regression coverage and document mapping boundaries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Thanks so much @Laurianti for your reviews. I've addressed all of the feedback. |
Keep net10.0 and netstandard2.0 provider assets, centralize extraction framework defaults, and validate both actual assets on .NET 10. Update package acceptance, CI, docs and bundled skills; preserve portable experimental metadata and document the Mistral status-code migration distinction. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
luisquintanilla
deleted the
luisquintanilla-toolkit-extraction-providers
branch
November 7, 2026 03:45
68 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add five independent
CommunityToolkit.DocumentExtraction.*providers over the sharedDocumentExtractionClientcontract. Applications choose an extraction engine and receive the same provider-neutralExtractionDocument, without adopting an ingestion pipeline or a second document model.PdfPigAzureContentUnderstandingAzureDocumentIntelligenceMistralVisionLanguageModelIChatClient, with free-form and strict structured modes.Each row is a separately consumable package. Installing one does not install the other providers or the advanced ONNX/rendering stack.
Using the providers
The examples below are alternative constructions. Choose one, then use the common extraction operation shown afterward. Shared imports:
PdfPig
No credentials, model files or native renderer are required. The PDF must already contain text; this provider does not recognize scanned text.
Azure Content Understanding
Supply an application-configured
Azure.AI.ContentUnderstanding.ContentUnderstandingClient, including its endpoint and authentication. The adapter also supports selecting the analyzer through its constructor.Alternatively, adapt the same SDK client with its factory extension:
Azure Document Intelligence
Supply an application-configured
Azure.AI.DocumentIntelligence.DocumentIntelligenceClient. The adapter maps the service's reported content rather than reconstructing a competing document hierarchy.Alternatively, use its factory extension:
Both Azure construction forms borrow the supplied SDK client and capture the extraction settings; adapting a client does not make a service request.
Mistral
For the direct Mistral endpoint, supply the application's API key and, optionally, a borrowed
HttpClient:Microsoft Foundry is supported through its endpoint/
TokenCredentialconstruction path; callers can instead provide the exact OCR URI through options when they need explicit routing.Vision language model
Supply an application-configured
IChatClient. For PDF input, the selected adapter, API and model must all support PDFs; image support alone is not sufficient. This example opts into structured output, which additionally requires schema-based JSON response support:Omit the options for the free-form mode. The provider sends the original input bytes; it does not render PDFs locally or silently make a second fallback inference request when structured output is unsupported.
The full naming family is
CommunityToolkit.DocumentExtraction.VisionLanguageModelfor the package, assembly and namespace, withVisionLanguageModelDocumentExtractionClientandVisionLanguageModelDocumentExtractionClientOptions. It replaces the former family outright; no old package/type aliases are added. The required suppliedIChatClientremains borrowed. Internalvision-llmsource-frame identities are intentionally unchanged. See the provider README.One extraction operation
After any of those constructions:
Construction does not make a service request.
ExtractAsyncis the explicit request for network-backed providers. The identifier belongs to the application, not to a provider's operation-tracking system.Mapping, errors and ownership
Providers preserve ordered content, text roles, table anchors/spans and mixed cell content, source frames and reported geometry without inventing unknown facts. Ancestors do not duplicate aggregate descendant text. Image transcription and alternative description remain separate facts.
Caller input streams and supplied SDK, HTTP and chat clients remain borrowed. Returned documents are data, not disposal boundaries. Native inspection does not promise original wire bytes or independence from the real provider lifetime. Unsupported configuration, provider failures and cancellation are surfaced rather than hidden behind another extraction engine.
Azure Content Understanding preserves literal whole-document Markdown when no structured items exist, without synthesizing tables. VisionLanguageModel validates its structured response, including anchors/spans, overlaps, dimensions and aggregate grid limits; null and empty text remain distinct.
Compatibility and experimental API policy
All five packages target
netstandard2.0andnet10.0. Dedicatednet8.0library assets and .NET 8 test/CI/consumer profiles have been removed. The portable asset broadens managed compatibility; retaining the modern asset preserves its newer API paths instead of forcing every consumer through legacy fallbacks. This does not add an explicitnet462target or a new .NET Framework support promise. A .NET 8 application may technically resolve the portable asset, but that is not a supported-runtime commitment; migrate the application to .NET 10. Underlying upstream dependencies may still ship .NET 8 assets; this change is limited to Toolkit extraction packages.Mistral HTTP failures preserve numeric status in the exception message on both assets. The .NET 10 asset also populates
HttpRequestException.StatusCode; the portable asset does not. An application previously using the dedicated .NET 8 asset must account for that distinction if it now resolves the portable asset. No exception-contract redesign is introduced.Provider assemblies carry the Toolkit-owned
CTDE0001experimental diagnostic on every target, covering provider clients, options and extensions. Consumers explicitly opt in; upstreamMEDE0001andEXTEXP0019remain separate. The compatibility attribute remains internal on the portable target, and Mistral's required/init contract is preserved. Experimental marking concerns API stability, not recognition accuracy.How this fits with the shared libraries
Merged dotnet/extensions#7835 owns the Documents model. dotnet/extensions#7588 owns extraction contracts, outer middleware, DI and logging. These providers do not redefine those contracts.
dotnet/extensions#7904 separately owns the optional MEDI reader bridge; no DataIngestion dependency is introduced here. CommunityToolkit/AI#83 separately owns the complete advanced PdfPig/ONNX companion, including rendering, local OCR, table inference, fluent configuration and model setup. This PR does not depend on that companion.
Package-delivered agent skills
This PR also bundles one provider-specific Agent Skill in each of the five NuGet packages, using
skills/<unique-package-scoped-name>/SKILL.md, following Delivering Skills in NuGet packages. These version-matched customer playbooks help coding agents with setup, a working integration, output checks and troubleshooting; they complement rather than replace the READMEs. Guidance covers credentials and consent, borrowed lifetimes, experimental opt-in and provider-specific limitations. VisionLanguageModel explicitly distinguishes completion from correctness and addressed tables from Markdown-only results. Its skill is ctk-document-extraction-vision-language-model.Flagging this addition for reviewer awareness and feedback on its scope and usefulness. The skills are passive package content: restore caches them but does not universally activate them. Customers deliberately inspect/install them with compatible tooling such as
dotnet-package-skills, then refresh installed copies when package versions change. There are no restore/build hooks, automatic model downloads or service calls. The component README documents discovery and its host/direct-dependency limitations.Actual archive/restored-file checks and compilation of the five packed examples cover delivery and API compatibility. Improved customer-task success, token cost and latency have not been benchmarked; no efficacy claim is made.
API review points
...ExtractionClientand...DocumentExtractionClientsuffixes intentional?CTDE0001the appropriate experimental boundary for this first provider API, including its options and construction helpers?net10.0plusnetstandard2.0. .NET 8 reaches end of support on November 10, 2026; technical compatibility through the portable asset does not imply support for an end-of-life runtime. Shared extraction defaults, tests, package consumers, CI and documentation now follow this decision.Current canonical-source validation (
c30c817)Bootstrap now uses canonical
https://github.com/dotnet/extensions.git, pinned commit5635a6ed7499739e2411f8288e1bd2b340c2c0e2, tree4ea4fe11e7fcc828012824eece4233a5b5f5e591, and a distinct core package identity11.0.0-review.20261106.1. The original AI Abstractions, Documents Abstractions, Extraction Abstractions and Extraction middleware owning projects are built unchanged using upstream SDK11.0.100-rc.3.26522.148, their actual imports/dependency bands and all six default framework assets (net11.0,net10.0,net9.0,net8.0,net462,netstandard2.0). Native compiler Features remaindebug-determinism; no blank-Features workaround or rewritten projects are used. The only explicit pack overrides are review package version, output directory and the original upstream NuGet configuration.core-bootstrap.jsonrecords source/project/SDK and archive provenance; config generation rejects missing, stale or changed canonical evidence. This supersedes the prior fork bootstrap; matching API baseline/project blobs were not treated as proof of implementation/build equivalence.Toolkit libraries remain
net10.0plusnetstandard2.0, with SDK/runtime 10 for Toolkit builds and applications. SDK 11 is needed only to build the original upstream projects. The portable dependency closure retains upstream System.Text.Json 11 prerelease rather than replacing it with host JSON 10; manually selected portable test/consumer profiles explicitly restore that closure. Provider names, client APIs, mapping algorithms, ownership and support policy are unchanged. Non-extraction/VectorData central dependency versions are untouched.Fresh local Release build passes with zero warnings/errors. All five offline suites pass against the new canonical packages on modern-on-10 and actual portable-on-10: DI 136, CU 136, Mistral 190, PdfPig 16 and VisionLanguageModel 171 per profile, 649 each / 1,298 total, zero failed/skipped. Four canonical core archives and five exact two-TFM provider archives pass source/nuspec/dependency/skill checks; all 15 owning/Toolkit/consumer dependency closures are recorded. Both fresh final-package consumers pass expected unsuppressed diagnostics, explicit opt-in, execution and exact selected-provider/four-core DLL hash/TFM bindings. Middleware is explicitly loaded for acceptance-only metadata/provenance inspection, not introduced as a production provider dependency. Ten stale/missing/mutated provenance/archive cases fail before config emission; archive matching also distinguishes PdfPig from its separately owned companion.
Exact-head Windows/Linux acceptance passed for
c30c8175d618ad7431d6fe1dc08a76da0090fdb5. The run rebuilt the original canonical core, restored dependencies using the committed normal public source configuration, ran modern and actual-portable suites, packed providers and executed both package-consumer profiles on each OS. Named same-run core and OS-specific provider artifacts are available through that run; they expire and are not a public feed.Local NuGet.org TLS handshake failed, and the public dotnet-public mirror lacks the unchanged Azure Document Intelligence SDK. Local validation therefore explicitly used its previously verified cache for that SDK only, never core packages. The committed hosted default uses NuGet.org for DI and dotnet-public for other non-core dependencies; the successful hosted gates did not use that local DI cache or service credentials. Earlier failed restore/portable-closure attempts are preserved as evidence, not reported as passing. No new service/model requests or resource/model changes were made for this migration. The temporary review identities are source-built artifacts; availability of equivalent released public packages remains unproven and is still a public-release dependency gate.
Historical support-removal validation (
2939447)Scoped cached-only restore and Release build pass with zero warnings/errors. All five offline suites pass on .NET 10 against both modern and actual
netstandard2.0provider DLLs: DI 136, CU 136, Mistral 190, PdfPig 16 and VisionLanguageModel 171 per profile, 649 each / 1,298 total, zero failed/skipped. Portable validation executes the real portable DLLs; it is not a compile proxy.Each of the five actual archives has exactly two lib framework folders,
net10.0andnetstandard2.0, matching nuspec dependency groups and no dedicated .NET 8 ref/runtime asset. Bundled skills/frontmatter/relative links pass the existing offline checker. The acceptance script now rejects old three-framework archives before restoring consumers. Fresh isolated final-package consumers pass modern-on-10 and portable-on-10, expected unsuppressedCTDE0001, explicit opt-in, offline execution, loaded-DLL identity/hash binding, internal portable experimental-shim visibility, friend assemblies and Mistral's required/init metadata.The solution filter lists projects rather than framework-specific profiles and remains unchanged. No unrelated VectorData/component or upstream core frameworks were changed. No service/model calls, model acquisition or dependency downloads were made for that support migration. This section describes its historical revision; current canonical-source results are above.
Historical validation and mapping follow-up
Windows and Linux acceptance passed an earlier pre-rename provider revision. Its .NET 8/.NET 10 profiles and the separate nine-example compilation are historical, not current support promises or validation of this support-removal commit. The earlier VisionLanguageModel rename also passed 684 local executions and four package-consumer profiles. Its optional README logging example was not recompiled because middleware dependency versions were absent from the offline cache; no dependencies were downloaded for it.
The four review findings were corrected in
fac9188without changing public APIs or defaults. Both Azure adapters project cross-page native sections into page-local continuations retaining original native sources. Native child order within each page and atomic tables/cells take precedence over a global span sort; unowned text is not adopted using offset envelopes. Figure OCR is associated only for exclusive plain paragraph references with measured containment, excluding caption/footnote/descriptive/shared ownership and retaining native paragraph sources. Mistral resolves image/table placeholders together, preserves ambiguous literal text, and retains declared data-URI MIME and exact bytes; raw-base64 PNG labelling remains an explicitly documented compatibility convention, not format detection.That earlier correction passed 2,596 local executions and four consumer profiles before dedicated .NET 8 support was removed. Its requested bounded live checks ran once each through the then-packed assets against the existing DI/CU/Mistral configurations using an original synthetic two-page PDF/JPEG chart: 3 analyses, 7 wire requests (3 POST + 4 normal Azure poll GET), no retries or diagnostic submissions. Exact native captures were replayed offline through baseline and fixed packages. Selected cross-page ordering improved on both Azure adapters with all 20 recognized paragraphs preserved exactly once; DI associated eight chart labels/values with image.Text, CU six. CU Q3/Q4 polygons slightly cross the reported figure boundary and remain original paragraphs rather than receiving invented geometric tolerance. Mistral replaced its unique marker at the correct text position and preserved the returned native JPEG MIME/bytes; baseline labelled those same bytes PNG. These are bounded prior-revision mapping observations, not live certification of every current asset or general recognition accuracy. No new live runs were performed for framework removal.
Same-page native sections intentionally stay atomic: the captured Azure section's trailing
After chartparagraph is therefore enumerated before an unowned root caption despite physical caption order. This documented structural-versus-global-order boundary is not a universal reading-order claim. Live responses did not exercise tables/shared refs/ambiguous markers; persistent offline tests cover those cases. The experimental upstream dependencies are reproduced from pinned public source until matching public package identities are available.