Skip to content

Comet Native Executors: let the native planner reject every JVM callback, and drop the driver's list #7375

Open

Description

@andygrove

What is the problem the feature request solves?

The driver (#7289) refuses a block whose plan calls back into the JVM by looking for message names in a hand-kept list (JvmScalarUdf, ArrowPythonUdf). The native planner separately refuses JVM calls when it plans a block with native inputs, through require_jvm, but not for ArrowPythonUdf. The two lists can drift apart, and the driver walks every block once more to check.

Describe the potential solution

Call require_jvm for ArrowPythonUdf in PhysicalPlanner, so the native planner refuses every JVM callback. The scheduler already plans every block when it checks a query, so the driver's list and its walk can go.

Additional context

The driver's list is jvmCallbacks in CometNativeExecutorsExec.scala.

Part of #7288.

Willingness to contribute

Yes. I can contribute this feature independently.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions