Skip to content

[FLINK-34368][filesystems] Bump gcs-connector to 3.1.22 - #30039

Open
MartijnVisser wants to merge 7 commits into
apache:masterfrom
MartijnVisser:FLINK-34368
Open

MartijnVisser wants to merge 7 commits into
apache:masterfrom
MartijnVisser:FLINK-34368

Conversation

@MartijnVisser

Copy link
Copy Markdown
Contributor

This builds on #29819. Only the last two commits belong to this PR; I'll rebase once it is merged.

What is the purpose of the change

Bump gcs-connector from 2.2.18 to 3.1.22. 3.1.22 is the only 3.x release with GoogleCloudDataproc/hadoop-connectors#1828, which stops close() after hsync() from failing with a 412.

Brief change log

  • Bump gcs-connector and align google-cloud-storage, google-cloud-nio and gRPC. Exclude grpc-testing. Update the NOTICE.
  • 3.x ignores GOOGLE_APPLICATION_CREDENTIALS and the keyfile option that the RecoverableWriter still uses. The docs describe gs.auth.type, and Flink logs a warning if it isn't set.
  • The GCS ITCases set gs.auth.type: APPLICATION_DEFAULT.

Verifying this change

  • Added ConfigUtilsAuthTypeTest.
  • Ran the GCS ITCases against a real bucket: 39 run, 4 skipped, 0 failures.
  • The median checkpoint of a RocksDB incremental job (about 15 MiB per checkpoint) went from 4.1 s to 4.9 s, and from 126 to 197 GCS requests.
  • With file merging it went from 1.1 s to 6.9 s, because 2.2.18 doesn't upload anything on sync().

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): yes
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes, checkpoints written to GCS
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)

cnauroth and others added 7 commits November 8, 2026 11:34
…ntials

- Gate the remaining ITCases on IT_CASE_GCS_BUCKET, drop RequireGCSConfiguration and the duplicate behavior suite
- Remove the second flink-core test-jar declaration left by the rebase
- Add the missing @test on the two overrides in HadoopGSRecoverableWriterITCase so they are reported as skipped
- Document that getLocalTmpDir() may return null, and refreeze the ArchUnit store

Generated-by: Claude Code (Claude Opus 5.5)
…stCredentials

Delete the writer's temporary objects after both recoverable writer suites. They live under
.inprogress/<bucket>/ at the bucket root, which the inherited teardown does not reach, and
only a commit removes them.

Generated-by: Claude Code (Claude Opus 5.5)
…stCredentials

Set gs.writer.chunk.size to 256kb in HadoopGSRecoverableWriterITCase. With the 16 MiB default,
each 7 MiB component of the *WithMultiPart tests went up in a single request. The default chunk
size stays covered by GSRecoverableWriterITCase.

Generated-by: Claude Code (Claude Opus 5.5)
…stCredentials

Merge the two @AfterEach methods of AbstractHadoopRecoverableWriterITCase back into one. JUnit
does not order @AfterEach methods within a class; a finally block keeps the object store cleanup
running when the local tmp dir check fails.

Make the two overridable tests protected instead of public. Package-private methods cannot be
overridden from org.apache.flink.fs.gs.

Generated-by: Claude Code (Claude Opus 5.5)
3.1.22 is the only 3.x release that doesn't compose an empty tail on
close() after hsync() (GoogleCloudDataproc/hadoop-connectors#1828), which
can fail checkpoints with 412 Precondition Failed after a retried request.

Align google-cloud-storage, google-cloud-nio and gRPC with the versions
3.1.22 is built against. Exclude grpc-testing, which gcsio declares in
compile scope and which brings junit and hamcrest into the plugin jar,
and the artifacts already excluded for google-cloud-storage.

gcs-connector 3.x ignores GOOGLE_APPLICATION_CREDENTIALS and
google.cloud.auth.service.account.json.keyfile, which the RecoverableWriter
still uses, so document gs.auth.type and set it in the GCS ITCases.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
…r credentials differ

If GOOGLE_APPLICATION_CREDENTIALS or
google.cloud.auth.service.account.json.keyfile is set but fs.gs.auth.type
isn't, the RecoverableWriter uses that file and gcs-connector uses the
COMPUTE_ENGINE credentials, so a job can read and write as two different
identities. The warning names the gs.auth.type value that matches the
file the RecoverableWriter uses.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
@flinkbot

flinkbot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants