Comet Native Executors: experiment with a native client and server for Spark's shuffle block transfer protocol #7384

What is the problem the feature request solves?

Comet Native Executors (#7288) run a query only if all of it can run natively. Running part of a query in Spark and part on the cluster needs a shuffle that both engines can read and write. Spark's block transfer protocol could be that shuffle, without a separate shuffle service. Unlike Spark's RPC control plane, which uses Java serialization, the block transfer protocol in network-common has a compact binary encoding and has been stable for many releases. Comet's native shuffle already writes Spark's data and index file layout.

Before designing any hybrid scheduling, it's worth finding out whether a native implementation of the data plane works against real Spark processes.

Describe the potential solution

An experiment, not production code:

1. Write a Rust client and server for the block fetch messages Spark uses to read shuffle blocks: OpenBlocks, FetchShuffleBlocks, StreamHandle, ChunkFetchRequest and ChunkFetchSuccess, and stream requests if needed, framed as network-common's MessageEncoder and MessageDecoder frame them.
2. Using the client, fetch the blocks of a shuffle that Comet's native shuffle wrote, from a Spark 3.5 executor and from the external shuffle service, and decode them with Comet's shuffle reader.
3. Using the server, serve shuffle files written in Spark's layout, and read them from a Spark task through a shuffle whose map outputs point at the server.

Authentication and encryption (SASL, spark.network.crypto) and push-based shuffle are out of scope. The experiment can require them to be off.

The result should say which messages are needed, how the protocol differs across Spark 3.4 to 4.x, and what registering native map outputs with MapOutputTrackerMaster would take.

Additional context

The design notes are in the contributor guide's Comet Native Executors page, under "Speaking Spark's protocols" (added in #7289). A native block fetch client could also move Comet on Spark's shuffle reads out of the JVM.

Part of #7288.

Willingness to contribute

Yes. I can contribute this feature independently.