Skip to content

Support multi-value keyword fields in PPL (type as ARRAY) + ITs - #5780

Open
finnegancarroll wants to merge 6 commits into
opensearch-project:mainfrom
finnegancarroll:feature/mv-sql-upstream
Open

finnegancarroll wants to merge 6 commits into
opensearch-project:mainfrom
finnegancarroll:feature/mv-sql-upstream

Conversation

@finnegancarroll

@finnegancarroll finnegancarroll commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds SQL/PPL support for querying a multi_value keyword field via the analytics engine, plus integration-test coverage.

The core change is small: a keyword field mapped multi_value: true is stored as a LIST column, so the frontend must surface it as a Calcite ARRAY. Otherwise the stock PPL multi-value operators (array_length, mvjoin, mvindex, mvfind, mvdedup, mvappend, the mvexpand command, …) fail to type-check against the field.

Full test coverage dependent on upstream core PRs:

opensearch-project/OpenSearch#23040
Fixes schema mismatch on multi shard scenario

opensearch-project/OpenSearch#23054
Provides markable RelNodes for uncollect/corrolate

opensearch-project/OpenSearch#22905
Fixes sorting over MV fields


Test infrastructure note

The two new ITs require the composite/parquet analytics-engine cluster, so — following the existing convention for AnalyticsEngineProfileIT — they are excluded from the default :integTest and run in the dedicated :integTest:analyticsEngineMultiValueIT task (which installs the analytics-engine plugin stack). Like analyticsEngineProfileIT, this task is not wired into a standard CI workflow; it is run against a locally-built analytics-engine (see the validation note above). The default integTest integration jobs therefore skip these classes rather than failing on the index.composite.* settings.

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

(Review updated until commit bf189c0)

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ Recommended focus areas for review

Possible Issue

The multi_value check uses Boolean.TRUE.equals(innerMap.get("multi_value")), which only matches if the value is exactly the Boolean object true. If the mapping JSON deserializes multi_value as a string "true" instead of a Boolean, this check silently fails and the field is not wrapped as ARRAY. This breaks array operator type-checking for fields that should be multi-value. Verify the actual deserialized type or use a more lenient check like Boolean.parseBoolean(String.valueOf(innerMap.get("multi_value"))).

if (Boolean.TRUE.equals(innerMap.get("multi_value"))) {
  fieldType = OpenSearchDataType.ofArray(fieldType);
}

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

PR Code Suggestions ✨

Latest suggestions up to bf189c0

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
General
Close input stream properly

The readAllBytes() method does not close the underlying input stream, which can lead
to resource leaks. Wrap the stream in a try-with-resources block or use
EntityUtils.toString() from the Apache HTTP client library to ensure proper resource
cleanup.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueDistributedIT.java [226-229]

 private static String entityAsString(Response response) throws IOException {
-  return new String(
-      response.getEntity().getContent().readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
+  try (var stream = response.getEntity().getContent()) {
+    return new String(stream.readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
+  }
 }
Suggestion importance[1-10]: 7

__

Why: The suggestion correctly identifies a potential resource leak where readAllBytes() doesn't automatically close the stream. Wrapping in try-with-resources ensures proper cleanup, though the impact is moderate since the stream is typically closed when the response is consumed.

Medium
Handle specific exception types

Catching and ignoring all exceptions can mask critical errors beyond the expected
"index not found" scenario. Catch only ResponseException with a 404 status code to
handle the specific case where the index doesn't exist, and let other exceptions
propagate.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueDistributedIT.java [53-56]

 private void provision() throws IOException {
   try {
     client().performRequest(new Request("DELETE", "/" + INDEX));
-  } catch (Exception ignored) {
+  } catch (org.opensearch.client.ResponseException e) {
+    if (e.getResponse().getStatusLine().getStatusCode() != 404) {
+      throw e;
+    }
   }
Suggestion importance[1-10]: 6

__

Why: The suggestion improves error handling by catching only the expected ResponseException with 404 status instead of all exceptions. This prevents masking unexpected errors, though the impact is moderate as this is test setup code where broad exception catching is sometimes acceptable.

Low

Previous suggestions

Suggestions up to commit e242604
CategorySuggestion                                                                                                                                    Impact
General
Close input stream properly

The readAllBytes() method does not close the underlying input stream, which can lead
to resource leaks. Wrap the stream in a try-with-resources block or use
EntityUtils.toString() from the Apache HTTP client library to ensure proper resource
cleanup.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueDistributedIT.java [226-229]

 private static String entityAsString(Response response) throws IOException {
-  return new String(
-      response.getEntity().getContent().readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
+  try (var stream = response.getEntity().getContent()) {
+    return new String(stream.readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
+  }
 }
Suggestion importance[1-10]: 7

__

Why: The suggestion correctly identifies a potential resource leak. While readAllBytes() closes the stream internally in modern Java versions, explicitly wrapping in try-with-resources is a best practice for clarity and ensures proper cleanup across all scenarios.

Medium
Suggestions up to commit e7e257a
CategorySuggestion                                                                                                                                    Impact
General
Close InputStream to prevent leaks

The InputStream from getContent() is not explicitly closed, which could lead to
resource leaks. Wrap the stream in a try-with-resources block to ensure proper
cleanup even if an exception occurs during reading.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueDistributedIT.java [226-229]

 private static String entityAsString(Response response) throws IOException {
-  return new String(
-      response.getEntity().getContent().readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
+  try (var stream = response.getEntity().getContent()) {
+    return new String(stream.readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
+  }
 }
Suggestion importance[1-10]: 7

__

Why: The suggestion correctly identifies a potential resource leak. While readAllBytes() typically closes the stream internally, explicitly using try-with-resources is a best practice that ensures proper cleanup in all scenarios and improves code clarity.

Medium
Avoid catching all exceptions silently

Catching and ignoring all exceptions can mask critical errors during test setup.
Consider logging the exception or catching only the specific ResponseException that
indicates the index doesn't exist, allowing other failures to surface properly.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueDistributedIT.java [55-56]

-} catch (Exception ignored) {
+} catch (org.opensearch.client.ResponseException e) {
+  // Index doesn't exist, which is expected on first run
 }
Suggestion importance[1-10]: 5

__

Why: The suggestion correctly identifies that catching all exceptions silently can mask errors. However, in test setup code where the goal is to ensure a clean state (deleting an index that may not exist), this pattern is acceptable. The improvement to catch only ResponseException is valid but has moderate impact.

Low
Handle exceptions more specifically

Silently catching all exceptions during index deletion can hide setup failures.
Catch only the specific ResponseException for non-existent indices, or log
unexpected exceptions to aid debugging when tests fail during provisioning.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueKeywordOperatorIT.java [54-55]

-} catch (Exception ignored) {
+} catch (org.opensearch.client.ResponseException e) {
+  // Index doesn't exist, which is expected
 }
Suggestion importance[1-10]: 5

__

Why: Similar to suggestion 1, this correctly identifies the broad exception catch. The suggestion to use ResponseException is valid for test setup cleanup code, but the impact is moderate since the current pattern is common in test provisioning where ensuring a clean slate is the priority.

Low
Suggestions up to commit b9720fe
CategorySuggestion                                                                                                                                    Impact
General
Handle string and boolean multi_value values

The multi_value property check assumes the value is a Boolean, but mapping values
can be strings ("true"/"false"). Consider handling both Boolean and String
representations to prevent the condition from silently failing when the mapping
contains "multi_value": "true" as a string.

opensearch/src/main/java/org/opensearch/sql/opensearch/data/type/OpenSearchDataType.java [140-142]

-if (Boolean.TRUE.equals(innerMap.get("multi_value"))) {
+Object multiValue = innerMap.get("multi_value");
+if (Boolean.TRUE.equals(multiValue) || "true".equalsIgnoreCase(String.valueOf(multiValue))) {
   fieldType = OpenSearchDataType.ofArray(fieldType);
 }
Suggestion importance[1-10]: 7

__

Why: This is a valid concern about type safety. OpenSearch mappings could potentially contain multi_value as either a boolean or string representation. The suggestion correctly identifies a potential bug where string values like "true" would not be recognized, causing the multi_value field to not be wrapped as an ARRAY type, breaking array operator functionality.

Medium
Handle index deletion exceptions properly

Catching and ignoring all exceptions during index deletion can mask critical errors.
Consider logging the exception or catching only the specific ResponseException for
404 (index not found) to ensure unexpected failures are visible during test setup.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueDistributedIT.java [53-56]

 private void provision() throws IOException {
   try {
     client().performRequest(new Request("DELETE", "/" + INDEX));
-  } catch (Exception ignored) {
+  } catch (ResponseException e) {
+    if (e.getResponse().getStatusLine().getStatusCode() != 404) {
+      throw e;
+    }
   }
Suggestion importance[1-10]: 5

__

Why: The suggestion correctly identifies that catching all exceptions can mask errors. However, the improvement is moderate since this is test setup code where ignoring deletion failures is a common pattern. The suggestion to catch only ResponseException with 404 check is a valid improvement for better error visibility.

Low
Suggestions up to commit 6966804
CategorySuggestion                                                                                                                                    Impact
General
Close input stream properly

The readAllBytes() method does not close the underlying input stream, which can lead
to resource leaks. Wrap the stream in a try-with-resources block or use
EntityUtils.toString() from the Apache HTTP client library to ensure proper resource
cleanup.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueDistributedIT.java [226-229]

 private static String entityAsString(Response response) throws IOException {
-  return new String(
-      response.getEntity().getContent().readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
+  try (var stream = response.getEntity().getContent()) {
+    return new String(stream.readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
+  }
 }
Suggestion importance[1-10]: 7

__

Why: While readAllBytes() does consume the stream, explicitly closing it in a try-with-resources ensures proper resource cleanup and follows best practices. However, the current code may not cause immediate issues since the HTTP client typically manages the stream lifecycle.

Medium
Handle specific exceptions only

Catching and silently ignoring all exceptions can mask critical errors during test
setup. Consider catching only the specific exception type (e.g., ResponseException
for 404 not found) to avoid suppressing unexpected failures that could indicate real
problems.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueDistributedIT.java [53-56]

 try {
   client().performRequest(new Request("DELETE", "/" + INDEX));
-} catch (Exception ignored) {
+} catch (ResponseException e) {
+  if (e.getResponse().getStatusLine().getStatusCode() != 404) {
+    throw e;
+  }
 }
Suggestion importance[1-10]: 6

__

Why: Catching specific exceptions like ResponseException for 404 errors is better practice than silently ignoring all exceptions. However, in test setup code where the intent is to ensure a clean state regardless of whether the index exists, the current approach is acceptable though not ideal.

Low
Suggestions up to commit daf4966
CategorySuggestion                                                                                                                                    Impact
General
Close input stream properly

The readAllBytes() method does not close the underlying input stream, which can lead
to resource leaks. Wrap the stream in a try-with-resources block or use
EntityUtils.toString() from the Apache HTTP client library to ensure proper resource
management.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueDistributedIT.java [226-229]

 private static String entityAsString(Response response) throws IOException {
-  return new String(
-      response.getEntity().getContent().readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
+  try (var stream = response.getEntity().getContent()) {
+    return new String(stream.readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
+  }
 }
Suggestion importance[1-10]: 7

__

Why: While readAllBytes() does consume the stream, explicitly closing it with try-with-resources is a best practice for resource management. However, the current code is not incorrect as the stream will eventually be closed by garbage collection, making this a moderate improvement rather than a critical fix.

Medium
Handle specific exceptions only

Catching and ignoring all exceptions can mask critical errors during test setup.
Consider catching only the specific exception type (e.g., ResponseException for 404
not found) to avoid suppressing unexpected failures that could indicate real
problems.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueDistributedIT.java [53-56]

 try {
   client().performRequest(new Request("DELETE", "/" + INDEX));
-} catch (Exception ignored) {
+} catch (ResponseException e) {
+  if (e.getResponse().getStatusLine().getStatusCode() != 404) {
+    throw e;
+  }
 }
Suggestion importance[1-10]: 6

__

Why: Catching specific exceptions like ResponseException for 404 errors is better practice than catching all exceptions. However, in test setup code where the goal is to ensure a clean state (deleting an index that may not exist), the current approach is acceptable and commonly used in integration tests.

Low

@codecov

codecov Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 40.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.23%. Comparing base (a29cf85) to head (8e68e61).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
...h/sql/opensearch/data/type/OpenSearchDataType.java 40.00% 2 Missing and 1 partial ⚠️

❌ Your project check has failed because the head coverage (63.23%) is below the target coverage (99.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5780      +/-   ##
============================================
- Coverage     63.24%   63.23%   -0.02%     
- Complexity     8820     8823       +3     
============================================
  Files           938      938              
  Lines         40211    40240      +29     
  Branches       4530     4538       +8     
============================================
+ Hits          25432    25446      +14     
- Misses        13957    13968      +11     
- Partials        822      826       +4     
Flag Coverage Δ
sql-engine 63.23% <40.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 8e68e61.

PathLineSeverityDescription
.github/workflows/analytics-engine-compat.yml21highCI execution environment changed from the default to 'almalinux8'. Changing the build platform alters which container image all CI steps execute inside, which is analogous to modifying a FROM directive in a Dockerfile. The stated reason (glibc 2.28 compatibility for a native library) is plausible, but the actual almalinux8 image tag resolved by the get-ci-image-tag workflow cannot be verified from this diff. Maintainers should confirm the image tag is pinned/trusted and that no additional packages or scripts are introduced by the new base image.

The table above displays the top 10 most important findings.

Total: 1 | Critical: 0 | High: 1 | Medium: 0 | Low: 0


Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.


⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 2a9c8aa

@finnegancarroll finnegancarroll added the enhancement New feature or request label Sep 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 5873a83

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit d12143a

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 75358d3

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit d6b9ead

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 75358d3

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit daf4966

@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 6966804

…n tests

Types a keyword field mapped `multi_value: true` as `ExprCoreType.ARRAY` so the stock
PPL multi-value operators bind to it and route to the analytics engine, and adds
integration-test coverage exercised against the analytics engine (composite/parquet).

- OpenSearchDataType.parseMapping: detect `multi_value: true` in the field mapping and
  surface the field as ARRAY (new `ofArray` factory), so array operators (array_length,
  mvjoin, mvindex, mvfind, mvdedup, mvappend, mvexpand, ...) type-check instead of Calcite
  rejecting them.
- CalciteMultiValueKeywordOperatorIT: PPL operator coverage on a real multi_value keyword
  field (projection, array_length, mv* family, mvexpand incl. per-doc limit=N and
  single-element), with exact-row/id assertions.
- CalciteMultiValueDistributedIT: 2-shard coverage of the coordinator reduce / cross-shard
  paths - cross-shard mvexpand group-by, PARTIAL/FINAL stage assertions via the profile API,
  backend-routing (analytics engine vs Lucene delegation), and cross-shard stored-LIST
  projection.
- integ-test/build.gradle: analyticsEngineMultiValueIT task (AE-enabled cluster) + filters.

Depends on the analytics-engine (core) multi-value execution support: the mvexpand
Correlate/Uncollect planner marking (opensearch-project/OpenSearch#23054) and the
multi-shard / LIST Arrow-schema fix (opensearch-project/OpenSearch#23040). The multi-shard
tests pass only with #23040 applied.

Signed-off-by: Finn Carroll <carrofin@amazon.com>
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit b9720fe

Comment thread integ-test/build.gradle Outdated
…eQueryIT task

Fold the multi-value keyword/distributed ITs into the existing analytics-engine
non-security IT task instead of a separate task+cluster with an identical config.
Rename analyticsEngineProfileIT -> analyticsEngineQueryIT and add the two
CalciteMultiValue* classes to its filter; delete the duplicate
analyticsEngineMultiValueIT task/cluster block. Keep the integTest /
integTestRemote exclusions (the plain cluster has no composite data format).
Collapse the two workflow run-steps into one analyticsEngineQueryIT step and
drop the temporary base-distribution debug step.

Signed-off-by: Finn Carroll <carrofin@amazon.com>
…mic mapping, mvzip/split/lambda)

Fill the previously-empty mvzip/split/lambda-predicate placeholders and add two
gaps flagged in review:
- Implicit group-by on a multi_value field (stats ... by tags with no mvexpand):
  the AE expands array elements into per-element buckets.
- Dynamic mapping: index array docs with no declared mapping, assert the field
  auto-promotes to multi_value:true in _mapping and that projection / array_length
  / implicit group-by behave like the explicitly-mapped field.
- mvzip/split against the real multi_value field; exists/forall/filter lambda
  predicates over its elements.

Also drop the explicit enableCalcite() call so the suite mirrors a customer using
default cluster settings (the analytics-engine path relies on the Calcite engine).

Signed-off-by: Finn Carroll <carrofin@amazon.com>
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit e7e257a

…e lib

The analytics-engine native library (libopensearch_native.so from the
feature-datafusion build) is compiled against glibc 2.28 and cannot dlopen on the
default al2 CI runner (glibc 2.26), so every node fails at boot with
'Cannot open library: .../libopensearch_native.so'. Request the almalinux8 CI image
(glibc 2.28) so the node loads the native library at startup.

Signed-off-by: Finn Carroll <carrofin@amazon.com>
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit e242604

…yword IT

Higher-order lambda predicates are not supported on the analytics-engine route
(backend rejects them: Function [exists] is not currently supported as a scalar
function). Replace the three lambda tests with a note; they are not part of
multi_value keyword support.

Signed-off-by: Finn Carroll <carrofin@amazon.com>
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit bf189c0

… core

The analytics-engine-compat CI runs against the published feature-datafusion core
build, which does not yet contain several multi_value fixes. Remove the 8 tests that
depend on unmerged core changes and replace each with a TODO documenting the needed
fix and the expected results to restore:
- implicit GROUP BY on a multi_value field (needs the ancestor-ref retype fix; draft
  opensearch-project/OpenSearch core change)
- mvzip over a multi_value field (needs Utf8View element downcast in rust mvzip UDF)
- dynamic multi_value auto-promotion (needs multi_value-all-types storage, #23063)

Remaining cases (projection, array_length, mvjoin/mvindex/mvfind/mvdedup/mvappend,
mvexpand + edge cases, split) pass against stock core.

Signed-off-by: Finn Carroll <carrofin@amazon.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants