Skip to content

Comet Native Executors: reuse Flight connections in the driver's client #7376

Open

Description

@andygrove

What is the problem the feature request solves?

The driver's Flight client (#7289) opens a new gRPC channel for every result partition it reads, even when many partitions live on the same few executors.

Describe the potential solution

Keep one channel per executor location in a process-wide cache in native/core/src/execution/native_executors/client.rs, and clone it for each partition. A tonic Channel is cheap to clone and multiplexes requests.

Additional context

This is a small efficiency gain, not a functional gap.

Part of #7288.

Willingness to contribute

Yes. I can contribute this feature independently.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions