You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
to_csv never runs inside Comet by default, unlike to_json / from_csv / schema_of_csv #5578
The serde has no CodegenDispatchFallback mixin, so with the default configuration the enclosing operator falls back to Spark for every to_csv call. The native path is reachable only with spark.comet.expression.StructsToCsv.allowIncompatible=true.
The rest of the structured-text family does not behave this way. to_json is Hybrid, and from_csv and schema_of_csv are both plain CometCodegenDispatch (serde/csv.scala:24, :26). to_csv is the odd one out.
There is also a docs bug here: docs/source/user-guide/latest/expressions.md lists to_csv as "✅ Native" with no note, which reads as "runs natively by default". It does not. Adding the mixin fixes the docs too, since GenerateDocs derives the Implementation column from the trait mixins and would then report it as Hybrid.
Fix: mix CodegenDispatchFallback into CometStructsToCsv. The complex-type arm dispatches fine — arrays, structs and maps are all in CometBatchKernelCodegen.isSupportedDataType as long as their leaves are.
Fixed by #5032, which mixes CodegenDispatchFallback into CometStructsToCsv so to_csv runs through the codegen dispatcher by default, and the expressions guide now lists it as Hybrid.
Activity
sam-1112 commentedon Sep 10, 2026
take
andygrove commentedon Sep 28, 2026
Fixed by #5032, which mixes CodegenDispatchFallback into CometStructsToCsv so to_csv runs through the codegen dispatcher by default, and the expressions guide now lists it as Hybrid.