Describe the bug
spark_read_side_padding_internal (native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:225) iterates the length array with let length = length.unwrap();, so any NULL in the length column panics with called Option::unwrap() on a None value and fails the task with a CometNativeException. Spark's StringRPad / StringLPad are NullIntolerant and return NULL for that row.
The path is default-on: CometStringRPad / CometStringLPad only require the pad to be a literal, so rpad(s, len_col) runs natively. Both the 2-arg [Array, Array] arm (lines 144-162) and the 3-arg [Array, Array, Scalar] arm (163-183) reach the unwrap.
Steps to reproduce
CREATE TABLE t USING parquet AS SELECT * FROM VALUES ('abc', 5), ('abc', CAST(NULL AS INT)) AS t(s, l);
SELECT rpad(s, l) FROM t; -- same for lpad(s, l) and rpad(s, l, 'x')
Spark: abc , NULL. Comet: task fails — called Option::unwrap() on a None value at read_side_padding.rs:225.
Unit level: spark_rpad(["abc","abc"], lengths=[Some(4), None], " ") panics.
Expected behavior
NULL for every row whose length is NULL, like Spark.
Proposed solution
Replace the unwrap with a match that appends null (and continues) when either the string or the length is null, in both arms. Add a unit test with a NULL length and a NULL-length row to string_rpad.sql / testStringPadding.
Additional context
The fuzz test cannot catch this because FuzzDataGenerator never generates NULL integers (#5389). The unwrap was introduced in #2630; #2096 was a different, earlier rpad panic (non-literal length hitting the unsupported-arguments arm).
Describe the bug
spark_read_side_padding_internal(native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:225) iterates the length array withlet length = length.unwrap();, so any NULL in the length column panics withcalled Option::unwrap() on a None valueand fails the task with aCometNativeException. Spark'sStringRPad/StringLPadareNullIntolerantand return NULL for that row.The path is default-on:
CometStringRPad/CometStringLPadonly require the pad to be a literal, sorpad(s, len_col)runs natively. Both the 2-arg[Array, Array]arm (lines 144-162) and the 3-arg[Array, Array, Scalar]arm (163-183) reach the unwrap.Steps to reproduce
Spark:
abc,NULL. Comet: task fails —called Option::unwrap() on a None valueatread_side_padding.rs:225.Unit level:
spark_rpad(["abc","abc"], lengths=[Some(4), None], " ")panics.Expected behavior
NULL for every row whose length is NULL, like Spark.
Proposed solution
Replace the unwrap with a match that appends null (and continues) when either the string or the length is null, in both arms. Add a unit test with a NULL length and a NULL-length row to
string_rpad.sql/testStringPadding.Additional context
The fuzz test cannot catch this because
FuzzDataGeneratornever generates NULL integers (#5389). The unwrap was introduced in #2630; #2096 was a different, earlierrpadpanic (non-literal length hitting the unsupported-arguments arm).