You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Comet Native Executors: let the native planner reject every JVM callback, and drop the driver's list #7375
The driver (#7289) refuses a block whose plan calls back into the JVM by looking for message names in a hand-kept list (JvmScalarUdf, ArrowPythonUdf). The native planner separately refuses JVM calls when it plans a block with native inputs, through require_jvm, but not for ArrowPythonUdf. The two lists can drift apart, and the driver walks every block once more to check.
Describe the potential solution
Call require_jvm for ArrowPythonUdf in PhysicalPlanner, so the native planner refuses every JVM callback. The scheduler already plans every block when it checks a query, so the driver's list and its walk can go.
Additional context
The driver's list is jvmCallbacks in CometNativeExecutorsExec.scala.
Activity