[GLUTEN-13003][MINOR] Drop dead Spark 3.3 and 3.2 residue from tests, helpers and comments

[LuciferYang]

What changes are proposed in this pull request?

Residue from #12902 that guards on Spark 3.3 or 3.2, in code that no supported version reaches. Most of it is in backends-bolt, because #12454 branched before #12954 and merged between #12954 and #12981, so it carried the old patterns back in after the first cleanup and before the second one, which touched no bolt file.

Never registered on any supported version: BoltParquetWriteForHiveSuite.scala:409 used testWithMaxSparkVersion(..., "3.3"), so the case ran nowhere and was not reported as skipped either, since the helper just declines to call test(). #12981 deleted the same case from the velox copy of that suite, and for a reason that applies here too: what made the codec stick was Gluten's own HiveFileFormat override, which lived only in shims/spark32 and shims/spark33, and no copy of that file is left in the tree.

Branches that cannot be taken: five isSparkVersionGE("3.4") guards in that same file, ArithmeticAnsiValidateSuite.scala:64, MiscOperatorSuite.scala:962 in bolt on startsWith("3.2"), and the dead startsWith("3.3.") half of a disjunction in VeloxHashJoinSuite.scala:120, where the startsWith("3.4.") half stays because it still varies.

Helpers and settings with no callers: ignoreWithSpecifiedSparkVersion and testWithRangeSparkVersion in GlutenQueryTest.scala, and BackendSettingsApi.rescaleDecimalArithmetic with its two overrides. That last one is named after the decimal rescale path #12991 removes, and it had no reader before that either.

An obsolete workaround: CudfNodeValidationRule.scala:35 walked the plan with find { ... }.isDefined under a comment saying Spark3.2 does not have exists. TreeNode.exists is on every supported version and already used in Validators.scala and ImplementJoinAggregate.scala.

Files nothing compiles: gluten-ut/spark33/ holds two bolt test-settings files and no pom.xml, and backends-bolt/src/test/resources/tpch-approved-plan/*/spark33/ holds 84 approved plans, with another 84 under */spark32/. gluten-ut/pom.xml lists only spark34/35/40/41, so none of the Scala is built, though scala_code_format.yml triggers on gluten-ut/** and still formats it. That is the bulk of the diffstat.

Worth flagging separately, because it is a bolt question rather than something to settle here: the only reader of tpch-approved-plan anywhere is VeloxTPCHSuite.scala:105, and it resolves the path relative to the running module. There is no bolt TPC-H suite, so the spark34 and spark35 copies under backends-bolt have no reader either. They stay, since a bolt suite may be the intent. Also BoltParquetWriteForHiveSuite.scala:285 evaluates "zstd".equalsIgnoreCase(column.getCodec.toString) and discards it: the missing assert is pre-existing, but with the 3.3-only case gone nothing in that file checks the written codec any more.

Four comments that had become wrong rather than merely old: SparkDataSourceRDDPartition said it exists to stay compatible with 3.3 and earlier, when what actually distinguishes it from Spark's class is that it holds the several input partitions Gluten groups into one split; the four copies of WritableColumnVectorShim named a 3.3 abstract-method addition as the reason the file is per-version, when the copies now split {3.4, 3.5} against {4.0, 4.1} over something else. Also // Since spark 3.3.0 in two backends, // For spark 3.2. in one, a commented-out enableSuite under // following UT is removed in spark3.3.1, a duplicated "3.5", "3.5" argument in two cache suites left over from a merge fix in #9297, and All the spark version is supported in VeloxIceberg.md.

Deliberately left alone: the // new added in spark-3.3 and need fix later, random failure may caused by memory free notes in the gluten-ut settings and the For Spark 3.3 + Delta 2.3 notes in the clickhouse tests. Only the version prefix has aged; the rest of each sentence is still the reason the exclusion or the expected file count is there, and rewording someone else's note across sixteen files buys nothing.

How was this patch tested?

clean test-compile on Spark 3.4, 3.5, 4.0 and 4.1 with -Pbackends-velox -Pspark-ut -Piceberg -Pdelta, Scala 2.13, and the 3.5 run adds -Pbackends-clickhouse, that being the only profile clickhouse builds on. spotless:apply produced no changes beyond the edits.

The bolt files needed a detour, since bolt does not compile on main right now: I stacked #12999 on top of this branch, ran test-compile -Pbackends-bolt -Pspark-ut on 3.4 and 3.5, then dropped it, so what is pushed here is this change alone.

No suite was run. Every collapsed branch keeps the arm a supported version already took, and the deleted files are not on any compile or test path. The bolt paths cannot be run here in any case: bolt's native library needs Linux plus GCC 10-12 or Clang 16 per docs/bolt-quick-start.md.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude claude-opus-5

Related issue: #13003

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

🔵 Needs a closer look

The broad cleanup spans source, tests, documentation, and many deleted fixtures, warranting final human verification.

Pull request overview

This pull request removes obsolete Spark 3.2/3.3 residue from tests, helpers, comments, documentation, and Bolt fixtures.

Changes:

• Removes unreachable version branches and unused helpers/settings.
• Updates stale compatibility comments and documentation.
• Deletes uncompiled Spark 3.2/3.3 test settings and approved plans.

File summaries

File	Reviewed change / note
shims/spark41/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVectorShim.java	Updates version rationale comments.
shims/spark40/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVectorShim.java	Updates version rationale comments. Nit (1 vote): make the subject explicit; “Their set” is grammatically incorrect.
shims/spark35/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVectorShim.java	Updates version rationale comments. Nit (1 vote): make the subject explicit; “Their set” is grammatically incorrect.
shims/spark34/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVectorShim.java	Updates version rationale comments. Nit (1 vote): make the subject explicit; “Their set” is grammatically incorrect.
gluten-ut/spark34/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Removes obsolete commented test configuration.
gluten-ut/spark34/src/test/scala/org/apache/gluten/utils/bolt/BoltTestSettings.scala	Removes obsolete commented test configuration.
gluten-ut/spark33/src/test/scala/org/apache/gluten/utils/bolt/BoltSQLQueryTestSettings.scala	Deletes uncompiled Spark 3.3 settings.
gluten-substrait/src/test/scala/org/apache/spark/sql/GlutenQueryTest.scala	Removes unused test helpers.
gluten-substrait/src/main/scala/org/apache/gluten/execution/SparkDataSourceRDDPartition.scala	Corrects class documentation.
gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/BackendSettingsApi.scala	Removes the unused decimal-rescaling setting.
docs/get-started/VeloxIceberg.md	Corrects supported Spark version documentation.
backends-velox/src/test/scala/org/apache/spark/sql/execution/VeloxLocalCacheSuite.scala	Removes a duplicate version argument.
backends-velox/src/test/scala/org/apache/gluten/execution/VeloxHashJoinSuite.scala	Removes a dead Spark 3.3 branch.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala	Removes a stale Spark comment.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala	Removes an obsolete override.
backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHSparkPlanExecApi.scala	Removes a stale Spark comment.
backends-bolt/src/test/scala/org/apache/spark/sql/execution/BoltParquetWriteForHiveSuite.scala	Removes dead tests and unreachable version branches.
backends-bolt/src/test/scala/org/apache/spark/sql/execution/BoltLocalCacheSuite.scala	Removes a duplicate version argument.
backends-bolt/src/test/scala/org/apache/gluten/functions/ArithmeticAnsiValidateSuite.scala	Removes an unreachable version branch.
backends-bolt/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala	Removes Spark 3.2-only behavior.
backends-bolt/src/test/resources/tpch-approved-plan/v1/spark33/6.txt	Deletes the obsolete Spark 3.3 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1/spark32/6.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1/spark32/4.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-ras/spark33/6.txt	Deletes the obsolete Spark 3.3 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-ras/spark32/6.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-ras/spark32/4.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj/spark33/6.txt	Deletes the obsolete Spark 3.3 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj/spark33/4.txt	Deletes the obsolete Spark 3.3 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj/spark33/17.txt	Deletes the obsolete Spark 3.3 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj/spark33/13.txt	Deletes the obsolete Spark 3.3 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj/spark32/6.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj/spark32/4.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj/spark32/22.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj/spark32/17.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj/spark32/13.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/6.txt	Deletes the obsolete Spark 3.3 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/4.txt	Deletes the obsolete Spark 3.3 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/17.txt	Deletes the obsolete Spark 3.3 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark33/13.txt	Deletes the obsolete Spark 3.3 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark32/6.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark32/4.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark32/22.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark32/17.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/test/resources/tpch-approved-plan/v1-bhj-ras/spark32/13.txt	Deletes the obsolete Spark 3.2 approved plan.
backends-bolt/src/main/scala/org/apache/gluten/extension/CudfNodeValidationRule.scala	Uses the supported TreeNode.exists API.
backends-bolt/src/main/scala/org/apache/gluten/execution/RowToBoltColumnarExec.scala	Removes a stale comment.
backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltBackend.scala	Removes an obsolete override.

Review details

• Files reviewed: 61/192 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.