Skip to content

Fix native document fidelity and source-grounded audits - #1911

Merged
Paul Lizer (paullizer) merged 1 commit into
Developmentfrom
paullizer-native-document-fidelity
Nov 11, 2026
Merged

Paul Lizer (paullizer) merged 1 commit into
Developmentfrom
paullizer-native-document-fidelity

Conversation

@paullizer

Copy link
Copy Markdown
Collaborator

Summary

  • Native source information was being lost or flattened before reaching chat, and answers could invent unsupported source bindings. Preserve absent, present-empty, present-value and inaccessible states, plus PDF/Excel/PPTX content and object associations, while retaining working Word review comments.
  • Inspect retained authorized active PDF/PPTX sources locally with bounded, source/revision/schema-bound continuations. Preserve native supplements through screened publication and prevent stale original metadata from describing a different sanitized active representation.
  • Validate source-only native-audit fact references and render factual values server-side instead of accepting model substitutions. Add verified synthetic fixtures and parser, persistence, HTTP and saved-answer regressions; update documentation/release notes and application version to 0.261.424.
  • No deployment or bulk re-ingestion is performed. Existing indexed Search content still needs explicit processing to acquire new evidence. Unsupported structures remain qualified; these reports are not accessibility certification.

Linked issue

N/A. No linked issue; the user explicitly requested no issue creation.

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

Application: 0.261.423 -> 0.261.424. Deployer files were not changed. Native Excel inspection schema 3.0 is independent of the cell/table reader schema; old native inventories/cursors cannot certify the new evidence contract.

Testing / validation

  • Final committed-branch regression selection: 243 passed.
  • Optimized native/schema-cursor selection: 51 passed.
  • Core Cosmos contracts and affected real-owner SDK regression: 246 passed.
  • python .\scripts\check_cosmos_response_boundaries.py: passed, 1,094 owners, 425 doubles, zero review gaps.
  • python .\functional_tests\test_docs_app_surface_coverage.py: passed, 7/7; generated inventory remains current.
  • python .\functional_tests\test_docs_site_quality.py: passed, 6/6.
  • git -c core.whitespace=blank-at-eol,blank-at-eof,space-before-tab,cr-at-eol --no-pager diff --check: passed.
  • python .\scripts\check_cosmos_response_boundaries.py --run-contracts: the full registry-wide 225-target sweep was stopped without completion, not claimed passed. Core and affected-owner checks were executed separately.

External blob/model I/O is isolated and Python network access blocked. Existing dependency, Swagger and older test-return deprecation warnings remain. No live beta deployment/retest, real-provider evaluation, or customer re-ingestion was performed.

Exact regression and optimized commands
$env:PYTHONPATH="$PWD\application\single_app;$PWD\functional_tests"
$env:PYTHONDONTWRITEBYTECODE='1'

# Passed: 243 tests
python -m pytest -q --tb=short --disable-warnings -p no:cacheprovider -p test_support.cosmos_network_guard .\functional_tests\test_native_document_fidelity.py .\functional_tests\test_native_document_fidelity_pipeline.py .\functional_tests\test_native_document_fidelity_http.py .\functional_tests\test_document_accessibility_extraction.py .\functional_tests\test_document_style_extraction.py .\functional_tests\test_document_accessibility_report_action.py .\functional_tests\test_excel_document_accessibility_extraction.py .\functional_tests\test_excel_native_inspection_chat_evidence.py .\functional_tests\test_excel_inventory_persistence.py .\functional_tests\test_word_comment_extraction.py .\functional_tests\test_orchestration_attachment_native_evidence.py .\functional_tests\test_orchestration_native_citation_delivery.py .\functional_tests\test_orchestration_document_action_binding.py .\functional_tests\test_orchestration_document_action_lineage.py .\functional_tests\test_orchestration_document_action_end_to_end.py .\functional_tests\test_publication_native_processing.py

# Passed: 51 tests
python -O -m pytest -q --tb=short --disable-warnings -p no:cacheprovider -p test_support.cosmos_network_guard .\functional_tests\test_native_document_fidelity.py .\functional_tests\test_native_document_fidelity_pipeline.py .\functional_tests\test_native_document_fidelity_http.py .\functional_tests\test_excel_inventory_persistence.py::test_source_and_section_bound_cursors_cannot_be_replayed
Exact core Cosmos and affected-owner command
$env:PYTHONPATH="$PWD\application\single_app;$PWD\functional_tests"
$env:PYTHONDONTWRITEBYTECODE='1'

# Passed: 246 tests
python -m pytest -q --tb=short --disable-warnings -p no:cacheprovider -p test_support.cosmos_network_guard -o faulthandler_timeout=90 .\functional_tests\test_workflow_chat_delivery_save_guard.py .\functional_tests\test_orchestration_failure_telemetry.py .\functional_tests\test_orchestration_result_imports.py .\functional_tests\test_cosmos_document_boundaries.py .\functional_tests\test_workflow_chat_delivery_concurrency.py .\functional_tests\test_orchestration_operation_imports.py .\functional_tests\test_orchestration_operation_runtime.py .\functional_tests\test_m365_pending_authorization.py .\functional_tests\test_m365_cosmos_sdk_contract.py .\functional_tests\test_cosmos_contract_runner.py .\functional_tests\test_orchestration_harness_execution.py::test_catalog_and_scheduler_fixtures_share_current_transport_types .\functional_tests\test_orchestration_cosmos_response_contract.py .\functional_tests\test_orchestration_email_review_flow.py .\functional_tests\test_workflow_authoring_sdk_boundaries.py::test_search_upload_owners_consume_real_sdk_point_documents

Documentation

  • Release notes updated, or not needed
  • Feature documentation updated, or not needed
  • Fix documentation updated, or not needed

Updated the document-accessibility and Excel-native-inspection feature docs, Document Search reference, v0.261.424 release notes, and docs/explanation/fixes/NATIVE_DOCUMENT_FIDELITY_FIX.md, including migration behavior and unsupported/partial coverage.

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security()) (N/A: no new routes)
  • Settings sent to non-admin frontends use sanitize_settings_for_user() (N/A: no new settings payloads)
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS (no browser assets added)
  • No secrets, keys, connection strings, or local-only artifacts are included

The fixture ZIP contains only nine verified synthetic source documents. Private conversation exports/account metadata and session artifacts are not included. Native reads retain current source authorization and active-representation checks; no reviewer-only original fallback is added.

Preserve native PDF, Excel and PowerPoint source states and associations, add authorized source-backed inspection and validated native audit rendering, and retain Word review-comment behavior.

Add verified synthetic regressions and documentation/release notes for v0.261.424.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment on lines +272 to +273
"These are the inspected source records. Absent, present-empty and inaccessible are different states. "
"Graphics counts describe the returned native coverage, not a conformance certificate.",
"description": inaccessible_field("legacy_native_state_unavailable"),
"title": inaccessible_field("legacy_native_state_unavailable"),
})
for index, run in enumerate(obj.get("text_runs", [])):
@paullizer
Paul Lizer (paullizer) merged commit 302e2ae into Development Nov 11, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants