Describe the bug
Spark's ConcatWs accepts both StringType and ArrayType(StringType) children and flattens array arguments: concat_ws(',', array('a','b'), 'c') = a,b,c. CometConcatWs.getSupportLevel (spark/src/main/scala/org/apache/comet/serde/strings.scala:291-298) never inspects child types and lowers every non-foldable call to DataFusion's concat_ws, which accepts only Utf8 / LargeUtf8 / Utf8View and fails with Input was List(...) which is not a supported datatype for concat_ws function (datafusion-functions 54.1 concat_ws.rs). The query fails at native execution where Spark returns a result. There is no Comet test with an array argument.
Steps to reproduce
CREATE TABLE t USING parquet AS SELECT * FROM VALUES (array('a','b'), 'c') AS t(arr, s);
SELECT concat_ws(',', arr, s) FROM t;
-- also: SELECT concat_ws(',', split(s, ' ')) FROM t
Spark: a,b,c. Comet: native error Input was List(Field { name: "item", data_type: Utf8, ... }) which is not a supported datatype for concat_ws function.
Expected behavior
a,b,c — either flatten array arguments natively like Spark, or fall back to Spark for them.
Proposed solution
Short term: in getSupportLevel, return Unsupported (with a reason) when any child's dataType is an ArrayType, so the expression falls back (or goes through the codegen dispatcher). Longer term: a Spark-compatible native concat_ws that flattens list arguments (skipping null elements like Spark). Add a test with an array argument.
Additional context
Only the all-foldable case is declined today (and Spark constant-folds that anyway). Related: #3339 (null-separator crash, fixed by #3542 with a null-literal branch but no argument-type gate).
Describe the bug
Spark's
ConcatWsaccepts bothStringTypeandArrayType(StringType)children and flattens array arguments:concat_ws(',', array('a','b'), 'c')=a,b,c.CometConcatWs.getSupportLevel(spark/src/main/scala/org/apache/comet/serde/strings.scala:291-298) never inspects child types and lowers every non-foldable call to DataFusion'sconcat_ws, which accepts onlyUtf8/LargeUtf8/Utf8Viewand fails withInput was List(...) which is not a supported datatype for concat_ws function(datafusion-functions 54.1concat_ws.rs). The query fails at native execution where Spark returns a result. There is no Comet test with an array argument.Steps to reproduce
Spark:
a,b,c. Comet: native errorInput was List(Field { name: "item", data_type: Utf8, ... }) which is not a supported datatype for concat_ws function.Expected behavior
a,b,c— either flatten array arguments natively like Spark, or fall back to Spark for them.Proposed solution
Short term: in
getSupportLevel, returnUnsupported(with a reason) when any child'sdataTypeis anArrayType, so the expression falls back (or goes through the codegen dispatcher). Longer term: a Spark-compatible nativeconcat_wsthat flattens list arguments (skipping null elements like Spark). Add a test with an array argument.Additional context
Only the all-foldable case is declined today (and Spark constant-folds that anyway). Related: #3339 (null-separator crash, fixed by #3542 with a null-literal branch but no argument-type gate).