Skip to content

concat_ws with array<string> arguments fails natively instead of flattening like Spark #5675

Description

@peterxcli

Describe the bug

Spark's ConcatWs accepts both StringType and ArrayType(StringType) children and flattens array arguments: concat_ws(',', array('a','b'), 'c') = a,b,c. CometConcatWs.getSupportLevel (spark/src/main/scala/org/apache/comet/serde/strings.scala:291-298) never inspects child types and lowers every non-foldable call to DataFusion's concat_ws, which accepts only Utf8 / LargeUtf8 / Utf8View and fails with Input was List(...) which is not a supported datatype for concat_ws function (datafusion-functions 54.1 concat_ws.rs). The query fails at native execution where Spark returns a result. There is no Comet test with an array argument.

Steps to reproduce

CREATE TABLE t USING parquet AS SELECT * FROM VALUES (array('a','b'), 'c') AS t(arr, s);
SELECT concat_ws(',', arr, s) FROM t;
-- also: SELECT concat_ws(',', split(s, ' ')) FROM t

Spark: a,b,c. Comet: native error Input was List(Field { name: "item", data_type: Utf8, ... }) which is not a supported datatype for concat_ws function.

Expected behavior

a,b,c — either flatten array arguments natively like Spark, or fall back to Spark for them.

Proposed solution

Short term: in getSupportLevel, return Unsupported (with a reason) when any child's dataType is an ArrayType, so the expression falls back (or goes through the codegen dispatcher). Longer term: a Spark-compatible native concat_ws that flattens list arguments (skipping null elements like Spark). Add a test with an array argument.

Additional context

Only the all-foldable case is declined today (and Spark constant-folds that anyway). Related: #3339 (null-separator crash, fixed by #3542 with a null-literal branch but no argument-type gate).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions