Repository navigation
[FLINK-34368][filesystems] Bump gcs-connector to 3.1.22 - #30039
Open
MartijnVisser wants to merge 7 commits into
Open
MartijnVisser wants to merge 7 commits into
MartijnVisser wants to merge 7 commits into
Conversation
… integration tests for GCS.
…ntials - Gate the remaining ITCases on IT_CASE_GCS_BUCKET, drop RequireGCSConfiguration and the duplicate behavior suite - Remove the second flink-core test-jar declaration left by the rebase - Add the missing @test on the two overrides in HadoopGSRecoverableWriterITCase so they are reported as skipped - Document that getLocalTmpDir() may return null, and refreeze the ArchUnit store Generated-by: Claude Code (Claude Opus 5.5)
…stCredentials Delete the writer's temporary objects after both recoverable writer suites. They live under .inprogress/<bucket>/ at the bucket root, which the inherited teardown does not reach, and only a commit removes them. Generated-by: Claude Code (Claude Opus 5.5)
…stCredentials Set gs.writer.chunk.size to 256kb in HadoopGSRecoverableWriterITCase. With the 16 MiB default, each 7 MiB component of the *WithMultiPart tests went up in a single request. The default chunk size stays covered by GSRecoverableWriterITCase. Generated-by: Claude Code (Claude Opus 5.5)
…stCredentials Merge the two @AfterEach methods of AbstractHadoopRecoverableWriterITCase back into one. JUnit does not order @AfterEach methods within a class; a finally block keeps the object store cleanup running when the local tmp dir check fails. Make the two overridable tests protected instead of public. Package-private methods cannot be overridden from org.apache.flink.fs.gs. Generated-by: Claude Code (Claude Opus 5.5)
3.1.22 is the only 3.x release that doesn't compose an empty tail on close() after hsync() (GoogleCloudDataproc/hadoop-connectors#1828), which can fail checkpoints with 412 Precondition Failed after a retried request. Align google-cloud-storage, google-cloud-nio and gRPC with the versions 3.1.22 is built against. Exclude grpc-testing, which gcsio declares in compile scope and which brings junit and hamcrest into the plugin jar, and the artifacts already excluded for google-cloud-storage. gcs-connector 3.x ignores GOOGLE_APPLICATION_CREDENTIALS and google.cloud.auth.service.account.json.keyfile, which the RecoverableWriter still uses, so document gs.auth.type and set it in the GCS ITCases. Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
…r credentials differ If GOOGLE_APPLICATION_CREDENTIALS or google.cloud.auth.service.account.json.keyfile is set but fs.gs.auth.type isn't, the RecoverableWriter uses that file and gcs-connector uses the COMPUTE_ENGINE credentials, so a job can read and write as two different identities. The warning names the gs.auth.type value that matches the file the RecoverableWriter uses. Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This builds on #29819. Only the last two commits belong to this PR; I'll rebase once it is merged.
What is the purpose of the change
Bump gcs-connector from 2.2.18 to 3.1.22. 3.1.22 is the only 3.x release with GoogleCloudDataproc/hadoop-connectors#1828, which stops close() after hsync() from failing with a 412.
Brief change log
GOOGLE_APPLICATION_CREDENTIALSand the keyfile option that the RecoverableWriter still uses. The docs describegs.auth.type, and Flink logs a warning if it isn't set.gs.auth.type: APPLICATION_DEFAULT.Verifying this change
ConfigUtilsAuthTypeTest.Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)