Skip to content

rpad/lpad panic (Option::unwrap on None) when the length column contains NULL #5672

Description

@peterxcli

Describe the bug

spark_read_side_padding_internal (native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:225) iterates the length array with let length = length.unwrap();, so any NULL in the length column panics with called Option::unwrap() on a None value and fails the task with a CometNativeException. Spark's StringRPad / StringLPad are NullIntolerant and return NULL for that row.

The path is default-on: CometStringRPad / CometStringLPad only require the pad to be a literal, so rpad(s, len_col) runs natively. Both the 2-arg [Array, Array] arm (lines 144-162) and the 3-arg [Array, Array, Scalar] arm (163-183) reach the unwrap.

Steps to reproduce

CREATE TABLE t USING parquet AS SELECT * FROM VALUES ('abc', 5), ('abc', CAST(NULL AS INT)) AS t(s, l);
SELECT rpad(s, l) FROM t;      -- same for lpad(s, l) and rpad(s, l, 'x')

Spark: abc , NULL. Comet: task fails — called Option::unwrap() on a None value at read_side_padding.rs:225.

Unit level: spark_rpad(["abc","abc"], lengths=[Some(4), None], " ") panics.

Expected behavior

NULL for every row whose length is NULL, like Spark.

Proposed solution

Replace the unwrap with a match that appends null (and continues) when either the string or the length is null, in both arms. Add a unit test with a NULL length and a NULL-length row to string_rpad.sql / testStringPadding.

Additional context

The fuzz test cannot catch this because FuzzDataGenerator never generates NULL integers (#5389). The unwrap was introduced in #2630; #2096 was a different, earlier rpad panic (non-literal length hitting the unsupported-arguments arm).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions