Spark: Remove deprecated and unused methods in SparkRead/WriteConf and SparkSchemaUtil - #17626
Open
dramaticlly wants to merge 2 commits into
Open
Spark: Remove deprecated and unused methods in SparkRead/WriteConf and SparkSchemaUtil#17626dramaticlly wants to merge 2 commits into
dramaticlly wants to merge 2 commits into
Conversation
- SparkSchemaUtil.convertWithFreshIds(Schema, StructType) and prune(Schema, StructType, List<Expression>) in Spark 4.0 and 4.1 - SparkReadConf/SparkWriteConf constructors taking an unused branch argument in Spark 4.1 Spark 3.5 is unaffected: these overloads were never deprecated there.
apache#14308 deprecated SparkSchemaUtil.convertWithFreshIds(Schema, StructType) and prune(Schema, StructType, List<Expression>) for 1.12.0 removal, but only touched spark/v4.0; Spark 4.1 inherited the annotations via the later 4.0<->4.1 directory move. Spark 3.5 was never updated, so the same two methods remain there without @deprecated. Removing them alongside 4.0 and 4.1 rather than leaving 3.5 out of sync for the release. Both are unused: every caller across all three versions uses convertWithFreshIds(schema, sparkType, caseSensitive) and prune(schema, requestedType, filter, caseSensitive) or prune(schema, requestedType). There are no references in main, test or jmh sources. Note this removes public methods that carried no @deprecated in 3.5.
Contributor
Author
|
@ebyhr @huaxingao do you want to take a look? |
Contributor
|
4.0/4.1 look good since those were deprecated in 1.11 for 1.12 removal. But I'm a bit concerned about 3.5 because removing them now is a breaking change for anyone still calling them on 3.5. Could we deprecate them in 3.5 this release and remove them in a later one? |
uros-b
reviewed
Aug 13, 2026
| * @return a Schema corresponding to the Spark projection | ||
| * @throws IllegalArgumentException if the Spark type does not match the Schema | ||
| */ | ||
| public static Schema prune(Schema schema, StructType requestedType, List<Expression> filters) { |
Member
There was a problem hiding this comment.
These are not deprecated in 3.5?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deprecated method in SparkTableUtil will follow up together with position-delete follow up
Note, #14308 only include the deprecation for latest spark 4.1 and it was later copied over to spark 4.0. I also included the same unused class in spark 3.5 (although not marked as deprecated in iceberg 1.11 release) considering to keep them in sync.
AI Disclosure
Model: Claude Opus 5 (1M context)
Platform/Tool: Claude Code
Human Oversight: reviewed
Prompt Summary: split #16449 into smaller self-contained PRs; verify each group compiles and tests green standalone