Conversation
The mode (opensearch-project#5657) narrowed an aggregation to the index subset where the group field is aggregatable and labelled the result with a warning. It is being dropped: returning a knowingly incomplete answer to make a query faster is the wrong default to offer, and gating it on a cluster setting, a per-request flag and a response-format check made three moving parts that each had to survive the security plugin's transport-to-worker handoff (opensearch-project#5739, opensearch-project#5743, opensearch-project#5758). Removed: PartialResultAggregatePushdown and its hook in CalciteLogicalIndexScan, plugins.query.partial_result.on_mapping_conflict.enabled, the per-request partial_result field, and the warnings-supported format gate that existed only to protect this feature. A conflicted aggregation once again returns the complete result over every index, which since opensearch-project#5646 pushes down as a _source script rather than failing. Kept whole: the warnings channel it introduced -- Warning, QueryResponse.warnings, CalcitePlanContext.addWarning/drainWarnings, and the SimpleJsonResponseFormatter plumbing. Nothing produces a warning after this commit; opensearch-project#5807 is the next producer, reporting a search that covered only some of its shards. Note for reviewers: dropping partial_result from TransportPPLQueryRequest changes that request's wire format. The class version-gates nothing and the field was appended the same way in 3.9, so this is symmetric with how it arrived, but it does mean a mixed 3.9/3.10 cluster must not route PPL across versions. Signed-off-by: Kai Huang <ahkcs@amazon.com>
PR Reviewer Guide 🔍(Review updated until commit 8313a0f)Here are some key observations to aid the review process:
|
getIndexMappings() on OpenSearchIndex and lastIndexMappings on the describe request existed only so the partitioner could see which index mapped a field which way. Nothing reads them now. The deep copy before merging stays: MergeRuleHelper rewrites the accumulated type's properties in place, so a fetched mapping would otherwise stop describing its own index. Its test now asserts that on the fetched mappings directly rather than through the removed accessor. Signed-off-by: Kai Huang <ahkcs@amazon.com>
|
Persistent review updated to latest commit 8313a0f |
PR Code Suggestions ✨Explore these optional code suggestions:
|
Codecov Report❌ Patch coverage is
❌ Your project check has failed because the head coverage (63.28%) is below the target coverage (99.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #5814 +/- ##
============================================
+ Coverage 63.23% 63.28% +0.04%
+ Complexity 8823 8803 -20
============================================
Files 938 937 -1
Lines 40236 40094 -142
Branches 4537 4514 -23
============================================
- Hits 25445 25374 -71
+ Misses 13966 13896 -70
+ Partials 825 824 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Description
Removes the opt-in partial-result mode for text/keyword mapping conflicts added in #5657, keeping the warnings channel that PR introduced.
Why. The mode traded correctness for speed: on a conflicted group-by it narrowed the aggregation to the index subset where the field is aggregatable and labelled the answer with a warning. Returning a knowingly incomplete result to make a query faster is not a good default to offer, and the safeguards around it were three moving parts — a cluster setting, a per-request flag, and a response-format check — each of which had to survive the security plugin's transport-to-worker handoff. That cost three follow-up fixes (#5739, #5743, #5758) without changing what the feature bought.
A conflicted aggregation now returns the complete result over every index again, which since #5646 pushes down as a per-document
_sourcescript rather than failing.Removed
PartialResultAggregatePushdown+ testCalciteLogicalIndexScan.tryPartialResultAggregate/resolvePartitionFields/allowPartialFallbackpushDownAggregateoverload that existed only to make it one-shotplugins.query.partial_result.on_mapping_conflict.enabledsettings.rstsectionpartial_resultPPLQueryRequest,TransportPPLQueryRequest,PPLQueryRequestFactory,PPLService,AbstractPlan,CalcitePlanContextwarningsSupportedgatePushDownContext.cloneWithOsIndexCalcitePartialResultOnMappingConflictIT,PartialResultSecurityITKept
The warnings channel, whole:
Warning(includingTYPE_PARTIAL_RESULT),QueryResponse.warnings,CalcitePlanContext.addWarning/drainWarningsand its per-query clearing,QueryResult, and theSimpleJsonResponseFormatteremission. Nothing produces a warning after this commit — #5807 is the next producer, reporting a search that covered only some of its shards — so the channel stays exercised by its own unit tests in the meantime.For reviewers
Wire format. Dropping
partial_resultfromTransportPPLQueryRequestchanges that request's serialization. The class version-gates nothing and the field was appended the same way in 3.9, so this is symmetric with how it arrived — but it does mean a mixed 3.9/3.10 cluster must not route PPL across versions. Say so if you would rather keep a placeholder read/write.Coordination with #5807. That PR adds a second warning type for shard coverage. With this feature gone there is only one meaning of "partial" left, so #5807 can collapse onto
PARTIAL_RESULTand OpenSearch Dashboards needs no change at all. Whichever lands second takes that rename; I would rather land #5807 first since it is reviewed and green.Testing
./gradlew build -x integTest -x tracingIntegTest -x yamlRestTest -x doctestgreen: 11,433 unit tests across core (5284), ppl (2141), opensearch (1710), legacy (1444), sql (711), protocol (71), plugin (50), common (22), 0 failures.Related Issues
Follows #5657. Coordinates with #5807.
Check List
settings.rstsection for the removed setting is deleted.--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.