chore: warn when a Compatible serde declines in convert

[andygrove]

Which issue does this PR close?

Related to #5574.

That issue is about the dispatcher being unreachable from convert. This PR does not fix that — it adds the diagnostic that makes the affected cases visible, so the cleanup in #5574 can be scoped from evidence rather than from a static sweep.

Rationale for this change

A serde whose getSupportLevel returns Compatible is promising that convert will succeed. When convert returns None instead, two things happen and neither is obvious:

• the enclosing operator falls back to Spark, and
• the JVM codegen dispatcher never gets a chance at the expression, because dispatchIfFallback is only reached from the Unsupported and Incompatible arms of exprToProtoInternal (QueryPlanSerde.scala:941 and :968).

So a check that lives in convert instead of getSupportLevel silently costs the user a whole-operator fallback that the dispatcher could have absorbed. Today nothing surfaces that.

What changes are included in this PR?

A warning on the Compatible arm when convert returns None, plus a new CometSerdeInvariantSuite.

The one piece of actual logic is the suppression rule. convert recurses into children, so a Compatible serde legitimately returns None when a child could not be serialized — getSupportLevel inspects the node, not the subtree. Without a guard, a single unsupported leaf would warn once for every ancestor on the way up.

The guard counts conversion failures on the planning thread and compares the counter across the convert call: if no nested exprToProtoInternal declined, the decline originated in this node. That is O(1). An earlier revision searched the subtree for a descendant carrying a fallback reason, which is O(n) per decline and therefore O(n·depth) along a failing path — a real cost here, since createBalancedBinaryExpr exists precisely because And/Or chains nest deeper than protobuf's 100-level recursion limit. The counter is also strictly more accurate: it catches children the serde synthesised inside convert (a Cast wrapper, say), which are not in the expression's subtree and so were invisible to the search. Verified equivalent on CometArrayExpressionSuite — one warning, same expression, before and after.

Two categories warn without being a serde bug, and both are documented in the scaladoc:

• getSupportLevel is not given inputs, so an inputs-dependent decline cannot be moved there. CometAttributeReference is the only one today.
• With spark.comet.exec.scalaUDF.codegen.enabled=false, every serde that reports Compatible and then routes to emitJvmCodegenDispatch warns, because the disabled-config check lives in the dispatcher rather than in getSupportLevel. Those are genuine instances of the invariant violation; moving that check up belongs with Codegen dispatcher is unreachable from convert, so serdes that decline there never get a dispatch attempt #5574, not here.

How are these changes tested?

New CometSerdeInvariantSuite (5 tests, registered in both PR workflows): warns on a real Compatible-then-None path, stays quiet when only a child failed, stays quiet when the parent tags itself and a child failed, warns when the node is the only tagged expression, and warns with a placeholder when no reason was recorded. The tests capture WARN output through a log4j2 appender attached to the QueryPlanSerde logger.

On noise, which was the main thing I wanted to check before proposing this: CometArrayExpressionSuite (57 tests) produced exactly one warning, from the test that disables the dispatcher config, and it is a true positive:

CometScalaUDF reported Compatible for UDF(knownnotnull(_4#1136)) but convert() returned None
(scalaudf: spark.comet.exec.scalaUDF.codegen.enabled=false; expression has no native path so
the plan falls back to Spark), so the enclosing operator falls back to Spark.

Under the default configuration it is silent.

[sunchao]

The counter adds thread-local work even with WARN disabled, while the PR base has neither this counter nor the new warning. Could you add a focused HEAD-versus-base planning comparison for one wide successful projection and one nested-fallback shape? Please match the hardware, build, Spark/JDK profile, warmup and inputs, and report planning time, allocations and warning counts with WARN on and off. Outside the timed work, please verify unchanged results and intended native/dispatcher/fallback routing, including that the successful projection reaches the native path. A synthesized child can be included in the nested case.