docs: correct the plugin and shuffle sections of the plugin overview

[andygrove]

Which issue does this PR close?

No issue was filed. This is a small documentation correction found while checking what CometDriverPlugin still does after #6054.

Rationale for this change

The contributor guide's plugin overview makes four claims that are no longer true:

• "The plugin is loaded on the Spark driver and does not provide executor-side plugins." CometPlugin has provided CometExecutorPlugin since feat: release tokio runtime on driver/executor exit #4734, which uses it to shut down the native tokio runtime when an executor stops.
• "The plugin will update the current SparkConf with the extra configuration provided by Comet, such as executor memory configuration." fix: remove the ineffective spark.executor.memoryOverhead adjustment from the driver plugin #6054 removed the spark.executor.memoryOverhead adjustment, and the plugin no longer changes any executor memory setting.
• "we cannot avoid having one native plan to produce the shuffle input and another native plan for writing the batches to the shuffle file." Since perf: avoid FFI import/export between native subtree and ShuffleWriter #4507, the native shuffle writer runs the child's native operators and the writer in one plan.
• The shuffle reader "creates an ArrowReaderIterator to process the blocks using Arrow's StreamReader for decoding IPC batches." It no longer does. Shuffle blocks are decoded in native code, inside the consuming native plan by default.

What changes are included in this PR?

The "Comet SQL Plugin" section now describes what the plugin does today:

• CometDriverPlugin runs before any SparkSession exists, which is what lets it set static configuration.
• It sets spark.comet.version, then stops with a warning unless spark.memory.offHeap.enabled or spark.comet.exec.onHeap.enabled is true.
• Otherwise it registers the session extension, installs Comet's cache serializer when the in-memory cache is enabled, registers Comet's metrics when spark.comet.metrics.enabled=true, and logs warnings for problem settings such as an unset spark.executor.memoryOverhead.
• It does not change executor memory settings, and the section links to the tuning guide for sizing them.
• Both components shut down the native tokio runtime when the driver or an executor stops.
• The session extension can be registered without the plugin, which is how most of Comet's test suites and the Spark SQL tests enable Comet, so the driver plugin's steps do not run for them.

In the "Shuffle" section:

• "Shuffle Writes" now describes both shuffle implementations: native shuffle's single plan, including the case where the writer reads its input from the JVM, and JVM columnar shuffle. It links to the native and JVM shuffle guides for details.
• "Shuffle Reads" now describes the reader both implementations share, with its direct read and JVM decode paths, and links to the read path in the native shuffle guide.

How are these changes tested?

This is a documentation-only change. Each statement was checked against the code. prettier --check passes on the page, and a Sphinx build of docs/source produces no warnings from it.

[sunchao]

Summary

• Prior state and problem: plugin_overview.md incorrectly described executor plugins, automatic memory configuration, separate native shuffle plans, and JVM IPC decoding.
• Design approach: Updates the overview to match current code and links to the detailed shuffle and tuning guides.
• Correctness / compatibility analysis: Verified the changed claims against plugin initialization, shutdown, shuffle writers, reader dispatch, and native decoding. Checked SparkContext/SparkSession initialization ordering against Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0 sources. No introduced P1/P2 issues found within this review.
• Key design decisions: Clearly distinguishes native and JVM columnar shuffle, including the JVM-input fallback. Links keep implementation detail in the existing subsystem guides without adding abstraction or complexity.
• Implementation sketch: Both commits change only docs/source/contributor-guide/plugin_overview.md. The description matches Plugins.scala, CometNativeShuffleWriter, and the direct-read and JNI-decoding paths.
• Behavioral changes worth calling out: Documentation only. No execution behavior, compatibility, or runtime overhead changes.
• Suggested improvements: None at P1/P2 severity.

Reviewed the full diff from 67803a7a422c44de07af1e5d25c1dbeae8df68d4 to 3fb3d159878bc9777fda0d837e68f85852ad70d9. Confirmed non-draft status. The snapshot and live discussion endpoints contained no reviews, issue comments, inline comments, or review threads.

Routed skills: review-comet-pr, review-comet-shuffle-pr, and review-comet-memory-pr.

Exact-head CI: 7 checks passed and 15 were skipped, with no failures or unfinished checks. Preflight passed Markdown formatting and Mermaid validation. Runtime suites and documentation deployment were skipped.

Validation limits: Local git diff --check and checks of all five added links and their anchors passed. Sphinx and Prettier were unavailable locally, so no local documentation build or formatting run was performed. No runtime suites were run for this documentation-only diff. The checkout remains unchanged.

[andygrove]

Thanks @sunchao