Support multi-value keyword fields in PPL (type as ARRAY) + ITs - #5780
finnegancarroll wants to merge 6 commits into
Conversation
PR Reviewer Guide 🔍(Review updated until commit bf189c0)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to bf189c0 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit e242604
Suggestions up to commit e7e257a
Suggestions up to commit b9720fe
Suggestions up to commit 6966804
Suggestions up to commit daf4966
|
Codecov Report❌ Patch coverage is
❌ Your project check has failed because the head coverage (63.23%) is below the target coverage (99.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #5780 +/- ##
============================================
- Coverage 63.24% 63.23% -0.02%
- Complexity 8820 8823 +3
============================================
Files 938 938
Lines 40211 40240 +29
Branches 4530 4538 +8
============================================
+ Hits 25432 25446 +14
- Misses 13957 13968 +11
- Partials 822 826 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 8e68e61.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
|
Persistent review updated to latest commit 2a9c8aa |
2a9c8aa to
5873a83
Compare
|
Persistent review updated to latest commit 5873a83 |
5873a83 to
d12143a
Compare
|
Persistent review updated to latest commit d12143a |
d12143a to
75358d3
Compare
|
Persistent review updated to latest commit 75358d3 |
|
Persistent review updated to latest commit d6b9ead |
d6b9ead to
75358d3
Compare
|
Persistent review updated to latest commit 75358d3 |
|
Persistent review updated to latest commit daf4966 |
daf4966 to
6966804
Compare
|
Persistent review updated to latest commit 6966804 |
02e53de to
2d17f50
Compare
…n tests Types a keyword field mapped `multi_value: true` as `ExprCoreType.ARRAY` so the stock PPL multi-value operators bind to it and route to the analytics engine, and adds integration-test coverage exercised against the analytics engine (composite/parquet). - OpenSearchDataType.parseMapping: detect `multi_value: true` in the field mapping and surface the field as ARRAY (new `ofArray` factory), so array operators (array_length, mvjoin, mvindex, mvfind, mvdedup, mvappend, mvexpand, ...) type-check instead of Calcite rejecting them. - CalciteMultiValueKeywordOperatorIT: PPL operator coverage on a real multi_value keyword field (projection, array_length, mv* family, mvexpand incl. per-doc limit=N and single-element), with exact-row/id assertions. - CalciteMultiValueDistributedIT: 2-shard coverage of the coordinator reduce / cross-shard paths - cross-shard mvexpand group-by, PARTIAL/FINAL stage assertions via the profile API, backend-routing (analytics engine vs Lucene delegation), and cross-shard stored-LIST projection. - integ-test/build.gradle: analyticsEngineMultiValueIT task (AE-enabled cluster) + filters. Depends on the analytics-engine (core) multi-value execution support: the mvexpand Correlate/Uncollect planner marking (opensearch-project/OpenSearch#23054) and the multi-shard / LIST Arrow-schema fix (opensearch-project/OpenSearch#23040). The multi-shard tests pass only with #23040 applied. Signed-off-by: Finn Carroll <carrofin@amazon.com>
2d17f50 to
b9720fe
Compare
|
Persistent review updated to latest commit b9720fe |
…eQueryIT task Fold the multi-value keyword/distributed ITs into the existing analytics-engine non-security IT task instead of a separate task+cluster with an identical config. Rename analyticsEngineProfileIT -> analyticsEngineQueryIT and add the two CalciteMultiValue* classes to its filter; delete the duplicate analyticsEngineMultiValueIT task/cluster block. Keep the integTest / integTestRemote exclusions (the plain cluster has no composite data format). Collapse the two workflow run-steps into one analyticsEngineQueryIT step and drop the temporary base-distribution debug step. Signed-off-by: Finn Carroll <carrofin@amazon.com>
…mic mapping, mvzip/split/lambda) Fill the previously-empty mvzip/split/lambda-predicate placeholders and add two gaps flagged in review: - Implicit group-by on a multi_value field (stats ... by tags with no mvexpand): the AE expands array elements into per-element buckets. - Dynamic mapping: index array docs with no declared mapping, assert the field auto-promotes to multi_value:true in _mapping and that projection / array_length / implicit group-by behave like the explicitly-mapped field. - mvzip/split against the real multi_value field; exists/forall/filter lambda predicates over its elements. Also drop the explicit enableCalcite() call so the suite mirrors a customer using default cluster settings (the analytics-engine path relies on the Calcite engine). Signed-off-by: Finn Carroll <carrofin@amazon.com>
|
Persistent review updated to latest commit e7e257a |
…e lib The analytics-engine native library (libopensearch_native.so from the feature-datafusion build) is compiled against glibc 2.28 and cannot dlopen on the default al2 CI runner (glibc 2.26), so every node fails at boot with 'Cannot open library: .../libopensearch_native.so'. Request the almalinux8 CI image (glibc 2.28) so the node loads the native library at startup. Signed-off-by: Finn Carroll <carrofin@amazon.com>
|
Persistent review updated to latest commit e242604 |
…yword IT Higher-order lambda predicates are not supported on the analytics-engine route (backend rejects them: Function [exists] is not currently supported as a scalar function). Replace the three lambda tests with a note; they are not part of multi_value keyword support. Signed-off-by: Finn Carroll <carrofin@amazon.com>
|
Persistent review updated to latest commit bf189c0 |
… core The analytics-engine-compat CI runs against the published feature-datafusion core build, which does not yet contain several multi_value fixes. Remove the 8 tests that depend on unmerged core changes and replace each with a TODO documenting the needed fix and the expected results to restore: - implicit GROUP BY on a multi_value field (needs the ancestor-ref retype fix; draft opensearch-project/OpenSearch core change) - mvzip over a multi_value field (needs Utf8View element downcast in rust mvzip UDF) - dynamic multi_value auto-promotion (needs multi_value-all-types storage, #23063) Remaining cases (projection, array_length, mvjoin/mvindex/mvfind/mvdedup/mvappend, mvexpand + edge cases, split) pass against stock core. Signed-off-by: Finn Carroll <carrofin@amazon.com>
Description
Adds SQL/PPL support for querying a
multi_valuekeyword field via the analytics engine, plus integration-test coverage.The core change is small: a keyword field mapped
multi_value: trueis stored as a LIST column, so the frontend must surface it as a CalciteARRAY. Otherwise the stock PPL multi-value operators (array_length,mvjoin,mvindex,mvfind,mvdedup,mvappend, themvexpandcommand, …) fail to type-check against the field.Full test coverage dependent on upstream core PRs:
opensearch-project/OpenSearch#23040
Fixes schema mismatch on multi shard scenario
opensearch-project/OpenSearch#23054
Provides markable RelNodes for uncollect/corrolate
opensearch-project/OpenSearch#22905
Fixes sorting over MV fields
Test infrastructure note
The two new ITs require the composite/parquet analytics-engine cluster, so — following the existing convention for
AnalyticsEngineProfileIT— they are excluded from the default:integTestand run in the dedicated:integTest:analyticsEngineMultiValueITtask (which installs the analytics-engine plugin stack). LikeanalyticsEngineProfileIT, this task is not wired into a standard CI workflow; it is run against a locally-built analytics-engine (see the validation note above). The defaultintegTestintegration jobs therefore skip these classes rather than failing on theindex.composite.*settings.