diff --git a/docs/content.zh/docs/deployment/filesystems/gcs.md b/docs/content.zh/docs/deployment/filesystems/gcs.md
index 0b971b24747bab..667a7245dd97ac 100644
--- a/docs/content.zh/docs/deployment/filesystems/gcs.md
+++ b/docs/content.zh/docs/deployment/filesystems/gcs.md
@@ -71,7 +71,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place
Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it.
-`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.68.0) library to provide `RecoverableWriter` support.
+`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/3.1.22) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.79.0) library to provide `RecoverableWriter` support.
This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}).
@@ -84,7 +84,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/
### Configuration
-The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}).
+The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}).
For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}). Flink will internally translate this back to `fs.gs.http.connect-timeout`.
@@ -107,41 +107,38 @@ You can also set `gcs-connector` options directly in the Hadoop `core-site.xml`
### Authentication to access GCS
-Most operations on GCS require authentication. To provide authentication credentials, either:
+Most operations on GCS require authentication. `flink-gs-fs-hadoop` uses two clients, which select their credentials separately:
-* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the JSON credentials file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run. This is the **recommended** method.
+* `gcs-connector` handles checkpoints, savepoints and all other file system access. It uses the credentials that `gs.auth.type` selects, and the credentials of the Compute Engine or GKE service account (`COMPUTE_ENGINE`) if `gs.auth.type` isn't set. See the [gcs-connector configuration](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md#authentication) for all values.
+* The `google-cloud-storage` library handles `RecoverableWriter`, which the `FileSink` uses. It uses the JSON credentials file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, then the one in the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml`, and otherwise the [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials).
+To use a JSON credentials file with both clients, either:
-* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path to the JSON credentials file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)):
+* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run, and set `gs.auth.type` in the [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}). This is the **recommended** method.
-```
-
-
- google.cloud.auth.service.account.json.keyfile
- PATH TO GOOGLE AUTHENTICATION JSON FILE
-
-
+```yaml
+gs.auth.type: APPLICATION_DEFAULT
```
-For `flink-gs-fs-hadoop` to use credentials via either of these two methods, the use of service accounts for authentication must be enabled. This is enabled by default; however, it can be disabled in `core-site.xml` by setting:
+* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path of a service account key file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)), and set `gs.auth.type: SERVICE_ACCOUNT_JSON_KEYFILE` in the Flink configuration file:
```
- google.cloud.auth.service.account.enable
- false
+ google.cloud.auth.service.account.json.keyfile
+ PATH TO GOOGLE AUTHENTICATION JSON FILE
```
-{{< hint warning >}}
-`gcs-connector` supports additional options to provide authentication credentials besides the `google.cloud.auth.service.account.json.keyfile` option described above.
-
-However, if you use any of those other options, the provided credentials will not be used by the `google-cloud-storage` library, which provides `RecoverableWriter` support, so Flink recoverable-write operations would be expected to fail.
+Flink logs a warning at startup if `RecoverableWriter` uses one of these files and `gs.auth.type` isn't set.
-For this reason, use of the `gcs-connector` authentication-credentials options other than `google.cloud.auth.service.account.json.keyfile` is **not recommended.**
+Setting `google.cloud.auth.service.account.enable` to `false` in `core-site.xml` makes `RecoverableWriter` ignore `google.cloud.auth.service.account.json.keyfile` and use the application default credentials. `gcs-connector` doesn't read this property.
+{{< hint warning >}}
+The other values of `gs.auth.type`, such as `USER_CREDENTIALS` or `ACCESS_TOKEN_PROVIDER`, only apply to `gcs-connector`. `RecoverableWriter` doesn't use those credentials, so it can run as a different identity than `gcs-connector` and its operations may fail.
+If you use the `FileSink`, set `gs.auth.type` as described above, or keep the default `COMPUTE_ENGINE` without configuring either file.
{{< /hint >}}
diff --git a/docs/content/docs/deployment/filesystems/gcs.md b/docs/content/docs/deployment/filesystems/gcs.md
index 164dbcc26156aa..ecf486d8991a89 100644
--- a/docs/content/docs/deployment/filesystems/gcs.md
+++ b/docs/content/docs/deployment/filesystems/gcs.md
@@ -71,7 +71,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place
Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it.
-`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.68.0) library to provide `RecoverableWriter` support.
+`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/3.1.22) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.79.0) library to provide `RecoverableWriter` support.
This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}).
@@ -84,7 +84,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/
### Configuration
-The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}).
+The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}).
For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). Flink will internally translate this back to `fs.gs.http.connect-timeout`.
@@ -107,41 +107,38 @@ You can also set `gcs-connector` options directly in the Hadoop `core-site.xml`
### Authentication to access GCS
-Most operations on GCS require authentication. To provide authentication credentials, either:
+Most operations on GCS require authentication. `flink-gs-fs-hadoop` uses two clients, which select their credentials separately:
-* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the JSON credentials file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run. This is the **recommended** method.
+* `gcs-connector` handles checkpoints, savepoints and all other file system access. It uses the credentials that `gs.auth.type` selects, and the credentials of the Compute Engine or GKE service account (`COMPUTE_ENGINE`) if `gs.auth.type` isn't set. See the [gcs-connector configuration](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md#authentication) for all values.
+* The `google-cloud-storage` library handles `RecoverableWriter`, which the `FileSink` uses. It uses the JSON credentials file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, then the one in the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml`, and otherwise the [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials).
+To use a JSON credentials file with both clients, either:
-* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path to the JSON credentials file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)):
-
-```
-
-
- google.cloud.auth.service.account.json.keyfile
- PATH TO GOOGLE AUTHENTICATION JSON FILE
-
-
+* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run, and set `gs.auth.type` in the [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). This is the **recommended** method.
+
+```yaml
+gs.auth.type: APPLICATION_DEFAULT
```
-For `flink-gs-fs-hadoop` to use credentials via either of these two methods, the use of service accounts for authentication must be enabled. This is enabled by default; however, it can be disabled in `core-site.xml` by setting:
+* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path of a service account key file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)), and set `gs.auth.type: SERVICE_ACCOUNT_JSON_KEYFILE` in the Flink configuration file:
```
- google.cloud.auth.service.account.enable
- false
+ google.cloud.auth.service.account.json.keyfile
+ PATH TO GOOGLE AUTHENTICATION JSON FILE
```
-{{< hint warning >}}
-`gcs-connector` supports additional options to provide authentication credentials besides the `google.cloud.auth.service.account.json.keyfile` option described above.
-
-However, if you use any of those other options, the provided credentials will not be used by the `google-cloud-storage` library, which provides `RecoverableWriter` support, so Flink recoverable-write operations would be expected to fail.
+Flink logs a warning at startup if `RecoverableWriter` uses one of these files and `gs.auth.type` isn't set.
-For this reason, use of the `gcs-connector` authentication-credentials options other than `google.cloud.auth.service.account.json.keyfile` is **not recommended.**
+Setting `google.cloud.auth.service.account.enable` to `false` in `core-site.xml` makes `RecoverableWriter` ignore `google.cloud.auth.service.account.json.keyfile` and use the application default credentials. `gcs-connector` doesn't read this property.
+{{< hint warning >}}
+The other values of `gs.auth.type`, such as `USER_CREDENTIALS` or `ACCESS_TOKEN_PROVIDER`, only apply to `gcs-connector`. `RecoverableWriter` doesn't use those credentials, so it can run as a different identity than `gcs-connector` and its operations may fail.
+If you use the `FileSink`, set `gs.auth.type` as described above, or keep the default `COMPUTE_ENGINE` without configuring either file.
{{< /hint >}}
diff --git a/flink-filesystems/flink-gs-fs-hadoop/archunit-violations/cce59259-d3cb-4048-a1bd-31197f847189 b/flink-filesystems/flink-gs-fs-hadoop/archunit-violations/cce59259-d3cb-4048-a1bd-31197f847189
index f3c98676934cb7..8b8372de55a65e 100644
--- a/flink-filesystems/flink-gs-fs-hadoop/archunit-violations/cce59259-d3cb-4048-a1bd-31197f847189
+++ b/flink-filesystems/flink-gs-fs-hadoop/archunit-violations/cce59259-d3cb-4048-a1bd-31197f847189
@@ -10,3 +10,21 @@ org.apache.flink.fs.gs.GSRecoverableWriterITCase does not satisfy: only one of t
* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\
* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\
or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule
+org.apache.flink.fs.gs.HadoopGSFileSystemITCase does not satisfy: only one of the following predicates match:\
+* reside in a package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type InternalMiniClusterExtension and annotated with @RegisterExtension\
+* reside outside of package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type MiniClusterExtension and annotated with @RegisterExtension or are , and of type MiniClusterTestEnvironment and annotated with @TestEnv\
+* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\
+* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\
+ or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule
+org.apache.flink.fs.gs.HadoopGSRecoverableWriterExceptionITCase does not satisfy: only one of the following predicates match:\
+* reside in a package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type InternalMiniClusterExtension and annotated with @RegisterExtension\
+* reside outside of package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type MiniClusterExtension and annotated with @RegisterExtension or are , and of type MiniClusterTestEnvironment and annotated with @TestEnv\
+* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\
+* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\
+ or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule
+org.apache.flink.fs.gs.HadoopGSRecoverableWriterITCase does not satisfy: only one of the following predicates match:\
+* reside in a package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type InternalMiniClusterExtension and annotated with @RegisterExtension\
+* reside outside of package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type MiniClusterExtension and annotated with @RegisterExtension or are , and of type MiniClusterTestEnvironment and annotated with @TestEnv\
+* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\
+* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\
+ or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule
diff --git a/flink-filesystems/flink-gs-fs-hadoop/pom.xml b/flink-filesystems/flink-gs-fs-hadoop/pom.xml
index 48c7442cb99879..21503676f2955c 100644
--- a/flink-filesystems/flink-gs-fs-hadoop/pom.xml
+++ b/flink-filesystems/flink-gs-fs-hadoop/pom.xml
@@ -34,11 +34,11 @@ under the License.
- 2.68.0
- hadoop3-2.2.18
- 0.128.7
+ 2.79.0
+ 3.1.22
+ 0.143.0
- 1.81.0
+ 1.85.1
@@ -67,6 +67,14 @@ under the License.
test
+
+ org.apache.flink
+ flink-hadoop-fs
+ ${project.version}
+ test
+ test-jar
+
+
org.apache.flink
flink-hadoop-fs
@@ -156,6 +164,23 @@ under the License.
org.codehaus.mojo
animal-sniffer-annotations
+
+ org.checkerframework
+ checker-qual
+
+
+ com.google.j2objc
+ j2objc-annotations
+
+
+ com.google.guava
+ failureaccess
+
+
+
+ io.grpc
+ grpc-testing
+
javax.annotation
diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/utils/ConfigUtils.java b/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/utils/ConfigUtils.java
index cdb9bfffbdd02c..b1bac70ca5230d 100644
--- a/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/utils/ConfigUtils.java
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/utils/ConfigUtils.java
@@ -24,6 +24,9 @@
import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemConfiguration;
+import com.google.cloud.hadoop.util.HadoopConfigurationProperty;
+import com.google.cloud.hadoop.util.HadoopCredentialsConfiguration;
+import com.google.cloud.hadoop.util.HadoopCredentialsConfiguration.AuthenticationType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -53,6 +56,9 @@ public class ConfigUtils {
private static final String HADOOP_OPTION_SERVICE_ACCOUNT_JSON_KEYFILE =
"google.cloud.auth.service.account.json.keyfile";
+ private static final HadoopConfigurationProperty CONNECTOR_AUTH_TYPE =
+ HadoopCredentialsConfiguration.AUTHENTICATION_TYPE_SUFFIX;
+
/**
* Loads the Hadoop configuration, by loading from a Hadoop conf dir (if one exists) and then
* overlaying properties derived from the Flink config.
@@ -109,7 +115,7 @@ public static org.apache.hadoop.conf.Configuration getHadoopConfiguration(
public static Optional getStorageCredentials(
org.apache.hadoop.conf.Configuration hadoopConfig, ConfigContext configContext) {
- // follow the same rules as for the Hadoop connector, i.e.
+ // RecoverableWriter credential order:
// 1) only use service credentials at all if Hadoop
// "google.cloud.auth.service.account.enable" is true (default: true)
// 2) use GOOGLE_APPLICATION_CREDENTIALS as location of credentials, if supplied
@@ -119,6 +125,8 @@ public static Optional getStorageCredentials(
// store any credentials we are to use, here
Optional credentialsPath = Optional.empty();
+ // the gcs-connector authentication type that uses the same credentials
+ AuthenticationType matchingAuthType = null;
// only look for credentials if service account support is enabled
boolean enableServiceAccount =
@@ -129,10 +137,12 @@ public static Optional getStorageCredentials(
// "google.cloud.auth.service.account.json.keyfile" from Hadoop
credentialsPath = configContext.getenv("GOOGLE_APPLICATION_CREDENTIALS");
if (credentialsPath.isPresent()) {
+ matchingAuthType = AuthenticationType.APPLICATION_DEFAULT;
LOGGER.info(
"GSRecoverableWriter is using GOOGLE_APPLICATION_CREDENTIALS at {}",
credentialsPath.get());
} else {
+ matchingAuthType = AuthenticationType.SERVICE_ACCOUNT_JSON_KEYFILE;
credentialsPath =
Optional.ofNullable(
hadoopConfig.get(HADOOP_OPTION_SERVICE_ACCOUNT_JSON_KEYFILE));
@@ -149,6 +159,15 @@ public static Optional getStorageCredentials(
LOGGER.info(
"Creating GSRecoverableWriter using credentials from {}",
credentialsPath.get());
+ if (!isConnectorAuthTypeSet(hadoopConfig)) {
+ LOGGER.warn(
+ "GSRecoverableWriter uses the credentials from {}, but gcs-connector uses"
+ + " {} credentials because 'gs.auth.type' is not set. Set"
+ + " 'gs.auth.type' to {} to use the same credentials for both.",
+ credentialsPath.get(),
+ CONNECTOR_AUTH_TYPE.getDefault(),
+ matchingAuthType);
+ }
GoogleCredentials credentials =
configContext.loadStorageCredentialsFromFile(credentialsPath.get());
return Optional.of(credentials);
@@ -158,6 +177,17 @@ public static Optional getStorageCredentials(
}
}
+ private static boolean isConnectorAuthTypeSet(
+ org.apache.hadoop.conf.Configuration hadoopConfig) {
+ // gcs-connector reads the authentication type under each of its key prefixes
+ for (String prefix : GoogleHadoopFileSystemConfiguration.CONFIG_KEY_PREFIXES) {
+ if (hadoopConfig.get(prefix + CONNECTOR_AUTH_TYPE.getKey()) != null) {
+ return true;
+ }
+ }
+ return false;
+ }
+
public static Optional getGcsRootUrl(
org.apache.hadoop.conf.Configuration hadoopConfig) {
// Ignore the default value, only returning a value if actually included in the config
diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE
index cc18aa1c1a0b7f..f4b86d396846c5 100644
--- a/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE
@@ -11,104 +11,113 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-databind:2.22.2
- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.22.2
- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.22.2
-- com.fasterxml.woodstox:woodstox-core:7.0.0
+- com.fasterxml.woodstox:woodstox-core:7.2.0
- com.google.android:annotations:4.1.1.4
-- com.google.api-client:google-api-client-jackson2:2.0.1
-- com.google.api-client:google-api-client:2.7.2
-- com.google.api.grpc:gapic-google-cloud-storage-v2:2.68.0
-- com.google.api.grpc:grpc-google-cloud-storage-v2:2.68.0
+- com.google.api-client:google-api-client-gson:2.9.2
+- com.google.api-client:google-api-client:2.9.2
+- com.google.api.grpc:gapic-google-cloud-storage-v2:2.79.0
+- com.google.api.grpc:grpc-google-cloud-storage-control-v2:2.49.0
+- com.google.api.grpc:grpc-google-cloud-storage-v2:2.79.0
+- com.google.api.grpc:proto-google-cloud-logging-v2:0.110.4
- com.google.api.grpc:proto-google-cloud-monitoring-v3:3.52.0
-- com.google.api.grpc:proto-google-cloud-storage-v2:2.68.0
-- com.google.api.grpc:proto-google-common-protos:2.71.0
-- com.google.api.grpc:proto-google-iam-v1:1.66.0
+- com.google.api.grpc:proto-google-cloud-storage-control-v2:2.49.0
+- com.google.api.grpc:proto-google-cloud-storage-v2:2.79.0
+- com.google.api.grpc:proto-google-common-protos:2.82.0
+- com.google.api.grpc:proto-google-iam-v1:1.77.0
- com.google.apis:google-api-services-iamcredentials:v1-rev20211203-2.0.0
-- com.google.apis:google-api-services-storage:v1-rev20260204-2.0.0
+- com.google.apis:google-api-services-storage:v1-rev20260524-2.0.0
- com.google.auto.value:auto-value-annotations:1.11.0
-- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.18
-- com.google.cloud.bigdataoss:gcsio:2.2.18
-- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.18
-- com.google.cloud.bigdataoss:util:2.2.18
-- com.google.cloud.opentelemetry:detector-resources-support:0.33.0
+- com.google.cloud.bigdataoss:gcs-connector:3.1.22
+- com.google.cloud.bigdataoss:gcsio:3.1.22
+- com.google.cloud.bigdataoss:util-hadoop:3.1.22
+- com.google.cloud.bigdataoss:util:3.1.22
+- com.google.cloud.opentelemetry:detector-resources-support:0.31.0
- com.google.cloud.opentelemetry:exporter-metrics:0.33.0
- com.google.cloud.opentelemetry:shared-resourcemapping:0.33.0
-- com.google.cloud:google-cloud-core-grpc:2.70.0
-- com.google.cloud:google-cloud-core-http:2.70.0
-- com.google.cloud:google-cloud-core:2.70.0
+- com.google.cloud:google-cloud-core-grpc:2.81.0
+- com.google.cloud:google-cloud-core-http:2.81.0
+- com.google.cloud:google-cloud-core:2.81.0
+- com.google.cloud:google-cloud-logging:3.21.4
- com.google.cloud:google-cloud-monitoring:3.52.0
-- com.google.cloud:google-cloud-storage:2.68.0
-- com.google.code.gson:gson:2.13.2
-- com.google.flogger:flogger-system-backend:0.7.1
-- com.google.flogger:flogger:0.7.1
-- com.google.flogger:google-extensions:0.7.1
+- com.google.cloud:google-cloud-storage-control:2.49.0
+- com.google.cloud:google-cloud-storage:2.79.0
+- com.google.code.gson:gson:2.14.0
+- com.google.flogger:flogger-system-backend:0.7.4
+- com.google.flogger:flogger:0.7.4
+- com.google.flogger:google-extensions:0.7.4
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
-- com.google.http-client:google-http-client-apache-v2:2.1.0
-- com.google.http-client:google-http-client-appengine:2.1.0
-- com.google.http-client:google-http-client-gson:2.1.0
-- com.google.http-client:google-http-client-jackson2:2.1.0
-- com.google.http-client:google-http-client:2.1.0
+- com.google.http-client:google-http-client-apache-v2:2.2.0
+- com.google.http-client:google-http-client-appengine:2.2.0
+- com.google.http-client:google-http-client-gson:2.2.0
+- com.google.http-client:google-http-client-jackson2:2.2.0
+- com.google.http-client:google-http-client:2.2.0
- com.google.oauth-client:google-oauth-client:1.39.0
- com.lmax:disruptor:3.4.2
- commons-codec:commons-codec:1.15
-- io.grpc:grpc-alts:1.81.0
-- io.grpc:grpc-api:1.81.0
-- io.grpc:grpc-auth:1.81.0
-- io.grpc:grpc-census:1.81.0
-- io.grpc:grpc-context:1.81.0
-- io.grpc:grpc-core:1.81.0
-- io.grpc:grpc-googleapis:1.81.0
-- io.grpc:grpc-grpclb:1.81.0
-- io.grpc:grpc-inprocess:1.81.0
-- io.grpc:grpc-netty-shaded:1.81.0
-- io.grpc:grpc-opentelemetry:1.81.0
-- io.grpc:grpc-protobuf-lite:1.81.0
-- io.grpc:grpc-protobuf:1.81.0
-- io.grpc:grpc-rls:1.81.0
-- io.grpc:grpc-services:1.81.0
-- io.grpc:grpc-stub:1.81.0
-- io.grpc:grpc-util:1.81.0
-- io.grpc:grpc-xds:1.81.0
+- dev.cel:common:0.13.1
+- dev.cel:protobuf:0.13.1
+- dev.cel:runtime:0.13.1
+- io.grpc:grpc-alts:1.85.1
+- io.grpc:grpc-api:1.85.1
+- io.grpc:grpc-auth:1.85.1
+- io.grpc:grpc-census:1.85.1
+- io.grpc:grpc-context:1.85.1
+- io.grpc:grpc-core:1.85.1
+- io.grpc:grpc-googleapis:1.85.1
+- io.grpc:grpc-grpclb:1.85.1
+- io.grpc:grpc-inprocess:1.85.1
+- io.grpc:grpc-netty-shaded:1.85.1
+- io.grpc:grpc-opentelemetry:1.85.1
+- io.grpc:grpc-protobuf-lite:1.85.1
+- io.grpc:grpc-protobuf:1.85.1
+- io.grpc:grpc-rls:1.85.1
+- io.grpc:grpc-services:1.85.1
+- io.grpc:grpc-stub:1.85.1
+- io.grpc:grpc-util:1.85.1
+- io.grpc:grpc-xds:1.85.1
- io.opencensus:opencensus-api:0.31.1
-- io.opencensus:opencensus-contrib-exemplar-util:0.31.0
-- io.opencensus:opencensus-contrib-grpc-metrics:0.31.0
+- io.opencensus:opencensus-contrib-exemplar-util:0.31.1
+- io.opencensus:opencensus-contrib-grpc-metrics:0.31.1
- io.opencensus:opencensus-contrib-http-util:0.31.1
-- io.opencensus:opencensus-contrib-resource-util:0.31.0
-- io.opencensus:opencensus-exporter-metrics-util:0.31.0
-- io.opencensus:opencensus-exporter-stats-stackdriver:0.31.0
-- io.opencensus:opencensus-impl-core:0.31.0
-- io.opencensus:opencensus-impl:0.31.0
+- io.opencensus:opencensus-contrib-resource-util:0.31.1
+- io.opencensus:opencensus-exporter-metrics-util:0.31.1
+- io.opencensus:opencensus-exporter-stats-stackdriver:0.31.1
+- io.opencensus:opencensus-impl-core:0.31.1
+- io.opencensus:opencensus-impl:0.31.1
- io.opentelemetry.contrib:opentelemetry-gcp-resources:1.37.0-alpha
- io.opentelemetry.semconv:opentelemetry-semconv:1.29.0-alpha
-- io.opentelemetry:opentelemetry-api:1.57.0
-- io.opentelemetry:opentelemetry-common:1.57.0
-- io.opentelemetry:opentelemetry-context:1.57.0
-- io.opentelemetry:opentelemetry-sdk-common:1.57.0
-- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.57.0
-- io.opentelemetry:opentelemetry-sdk-logs:1.57.0
-- io.opentelemetry:opentelemetry-sdk-metrics:1.57.0
-- io.opentelemetry:opentelemetry-sdk-trace:1.57.0
-- io.opentelemetry:opentelemetry-sdk:1.57.0
+- io.opentelemetry:opentelemetry-api:1.67.0
+- io.opentelemetry:opentelemetry-common:1.67.0
+- io.opentelemetry:opentelemetry-context:1.67.0
+- io.opentelemetry:opentelemetry-sdk-common:1.67.0
+- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.67.0
+- io.opentelemetry:opentelemetry-sdk-logs:1.67.0
+- io.opentelemetry:opentelemetry-sdk-metrics:1.67.0
+- io.opentelemetry:opentelemetry-sdk-trace:1.67.0
+- io.opentelemetry:opentelemetry-sdk:1.67.0
- io.perfmark:perfmark-api:0.27.0
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.14
-- org.conscrypt:conscrypt-openjdk-uber:2.5.2
+- org.conscrypt:conscrypt-openjdk-uber:2.8.0
This project bundles the following dependencies under BSD-3 License (https://opensource.org/licenses/BSD-3-Clause).
See bundled license files for details.
-- com.google.api:api-common:2.63.0
-- com.google.api:gax-grpc:2.80.0
-- com.google.api:gax-httpjson:2.80.0
-- com.google.api:gax:2.80.0
-- com.google.auth:google-auth-library-credentials:1.47.0
-- com.google.auth:google-auth-library-oauth2-http:1.47.0
-- com.google.protobuf:protobuf-java-util:4.33.2
-- com.google.protobuf:protobuf-java:4.33.2
+- com.google.api:api-common:2.74.0
+- com.google.api:gax-grpc:2.91.0
+- com.google.api:gax-httpjson:2.91.0
+- com.google.api:gax:2.91.0
+- com.google.auth:google-auth-library-credentials:1.58.0
+- com.google.auth:google-auth-library-oauth2-http:1.58.0
+- com.google.protobuf:protobuf-java-util:4.33.6
+- com.google.protobuf:protobuf-java:4.33.6
+- org.threeten:threeten-extra:1.8.0
- org.threeten:threetenbp:1.7.0
This project bundles the following dependencies under BSD License (https://opensource.org/licenses/bsd-license.php).
See bundled license files for details.
-- org.codehaus.woodstox:stax2-api:4.2.2 (https://github.com/FasterXML/stax2-api/tree/stax2-api-4.2.2)
+- org.codehaus.woodstox:stax2-api:4.3.0 (https://github.com/FasterXML/stax2-api/tree/stax2-api-4.3.0)
This project bundles the following dependencies under the Go License (https://golang.org/LICENSE).
See bundled license files for details.
diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/licenses/LICENSE.threeten-extra b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/licenses/LICENSE.threeten-extra
new file mode 100644
index 00000000000000..6615eadc8c8a58
--- /dev/null
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/licenses/LICENSE.threeten-extra
@@ -0,0 +1,29 @@
+Copyright (c) 2007-present, Stephen Colebourne & Michael Nascimento Santos.
+
+All rights reserved.
+
+* Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+* Neither the name of JSR-310 nor the names of its contributors
+ may be used to endorse or promote products derived from this software
+ without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSFileSystemBehaviorITCase.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSFileSystemBehaviorITCase.java
index 68cb21c759e16e..565199e0f8d4e8 100644
--- a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSFileSystemBehaviorITCase.java
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSFileSystemBehaviorITCase.java
@@ -44,7 +44,7 @@ class GSFileSystemBehaviorITCase extends FileSystemBehaviorTestSuite {
@BeforeAll
static void checkCredentialsAndSetup() {
GSTestCredentials.assumeCredentialsAvailable();
- FileSystem.initialize(new Configuration(), null);
+ FileSystem.initialize(GSTestCredentials.createConfiguration(), null);
}
@AfterAll
diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSRecoverableWriterITCase.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSRecoverableWriterITCase.java
index fd25ee48b568d0..2058352d569732 100644
--- a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSRecoverableWriterITCase.java
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSRecoverableWriterITCase.java
@@ -56,7 +56,7 @@ class GSRecoverableWriterITCase {
@BeforeAll
static void checkCredentialsAndSetup() throws Exception {
GSTestCredentials.assumeCredentialsAvailable();
- FileSystem.initialize(new Configuration(), null);
+ FileSystem.initialize(GSTestCredentials.createConfiguration(), null);
basePath = new Path(GSTestCredentials.getTestBucketUri() + TEST_DATA_DIR);
fileSystem = basePath.getFileSystem();
}
diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSTestCredentials.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSTestCredentials.java
index 0b26f2ccef957f..a0bf9c83593d58 100644
--- a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSTestCredentials.java
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSTestCredentials.java
@@ -18,8 +18,17 @@
package org.apache.flink.fs.gs;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.fs.gs.storage.GSBlobIdentifier;
+import org.apache.flink.fs.gs.storage.GSBlobStorage;
+import org.apache.flink.fs.gs.storage.GSBlobStorageImpl;
+import org.apache.flink.fs.gs.utils.BlobUtils;
+
import javax.annotation.Nullable;
+import java.util.List;
+
import static org.assertj.core.api.Assumptions.assumeThat;
/**
@@ -42,6 +51,16 @@ static void assumeCredentialsAvailable() {
.isNotBlank();
}
+ /**
+ * Returns a configuration that makes gcs-connector use application default credentials, like
+ * the Google Cloud Storage client of the recoverable writer.
+ */
+ static Configuration createConfiguration() {
+ final Configuration config = new Configuration();
+ config.setString("gs.auth.type", "APPLICATION_DEFAULT");
+ return config;
+ }
+
/** Returns the URI of the test bucket, e.g. {@code gs://my-bucket/temp/}. */
static String getTestBucketUri() {
if (GCS_TEST_BUCKET == null) {
@@ -50,4 +69,21 @@ static String getTestBucketUri() {
}
return "gs://" + GCS_TEST_BUCKET + "/temp/";
}
+
+ /**
+ * Deletes the temporary objects the recoverable writer created for files under {@code
+ * basePath}. They live under the in-progress prefix at the bucket root, outside {@code
+ * basePath}, and only a commit removes them, so uncommitted streams leave them behind.
+ */
+ static void deleteTemporaryObjects(Path basePath) {
+ final GSFileSystemFactory factory = new GSFileSystemFactory();
+ factory.configure(createConfiguration());
+ final GSBlobStorage storage = new GSBlobStorageImpl(factory.getStorage());
+ final GSBlobIdentifier base = BlobUtils.parseUri(basePath.toUri());
+ final List temporaryObjects =
+ storage.list(base.bucketName, BlobUtils.getTemporaryObjectPartialName(base));
+ if (!temporaryObjects.isEmpty()) {
+ storage.delete(temporaryObjects);
+ }
+ }
}
diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSFileSystemITCase.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSFileSystemITCase.java
new file mode 100644
index 00000000000000..84e7cafe84953c
--- /dev/null
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSFileSystemITCase.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.fs.gs;
+
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.fs.hdfs.AbstractHadoopFileSystemITTest;
+
+import org.junit.jupiter.api.BeforeAll;
+
+import java.io.IOException;
+import java.util.UUID;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Integration tests for GCS file system support. */
+class HadoopGSFileSystemITCase extends AbstractHadoopFileSystemITTest {
+
+ @BeforeAll
+ static void setup() throws IOException {
+ GSTestCredentials.assumeCredentialsAvailable();
+ FileSystem.initialize(GSTestCredentials.createConfiguration(), null);
+ basePath = new Path(GSTestCredentials.getTestBucketUri() + "tests-" + UUID.randomUUID());
+ fs = basePath.getFileSystem();
+ assertThat(fs.exists(basePath)).isFalse();
+ }
+}
diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterExceptionITCase.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterExceptionITCase.java
new file mode 100644
index 00000000000000..58b9e9621b7937
--- /dev/null
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterExceptionITCase.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.fs.gs;
+
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.fs.hdfs.AbstractHadoopRecoverableWriterExceptionITCase;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+
+import java.io.IOException;
+import java.util.UUID;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Integration tests for exception handling in {@link
+ * org.apache.flink.fs.gs.writer.GSRecoverableWriter}.
+ */
+class HadoopGSRecoverableWriterExceptionITCase
+ extends AbstractHadoopRecoverableWriterExceptionITCase {
+
+ @BeforeAll
+ static void setup() throws IOException {
+ GSTestCredentials.assumeCredentialsAvailable();
+ FileSystem.initialize(GSTestCredentials.createConfiguration(), null);
+ basePath = new Path(GSTestCredentials.getTestBucketUri() + "tests-" + UUID.randomUUID());
+ assertThat(basePath.getFileSystem().exists(basePath)).isFalse();
+ skipped = false;
+ }
+
+ @AfterAll
+ static void cleanUpTemporaryObjects() {
+ if (!skipped) {
+ GSTestCredentials.deleteTemporaryObjects(basePath);
+ }
+ }
+
+ @Override
+ protected String getLocalTmpDir() throws Exception {
+ return null;
+ }
+}
diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterITCase.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterITCase.java
new file mode 100644
index 00000000000000..4557b8f7f9e833
--- /dev/null
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterITCase.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.fs.gs;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.fs.RecoverableWriter;
+import org.apache.flink.runtime.fs.hdfs.AbstractHadoopRecoverableWriterITCase;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.UUID;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.fail;
+
+/** Integration tests for {@link org.apache.flink.fs.gs.writer.GSRecoverableWriter}. */
+class HadoopGSRecoverableWriterITCase extends AbstractHadoopRecoverableWriterITCase {
+
+ @BeforeAll
+ static void setup() throws IOException {
+ GSTestCredentials.assumeCredentialsAvailable();
+ // The smallest chunk size GCS accepts, so each 7 MiB component of the *WithMultiPart
+ // tests is uploaded in several chunks instead of one (the default is 16 MiB).
+ final Configuration config = GSTestCredentials.createConfiguration();
+ config.set(GSFileSystemOptions.WRITER_CHUNK_SIZE, MemorySize.parse("256kb"));
+ FileSystem.initialize(config, null);
+ basePath = new Path(GSTestCredentials.getTestBucketUri() + "tests-" + UUID.randomUUID());
+ assertThat(basePath.getFileSystem().exists(basePath)).isFalse();
+ bigDataChunk = createBigDataChunk(BIG_CHUNK_DATA_PATTERN, 7L << 20);
+ skipped = false;
+ }
+
+ @AfterAll
+ static void cleanUpTemporaryObjects() {
+ if (!skipped) {
+ GSTestCredentials.deleteTemporaryObjects(basePath);
+ }
+ }
+
+ @Test
+ @Override
+ @Disabled("This test is not meaningful. GSRecoverableWriter#cleanupRecoverableState is no-op.")
+ protected void testCleanupRecoverableState() {}
+
+ @Test
+ @Override
+ @Disabled("This test assumes one file for state, but GSRecoverableWriter uses multiple files.")
+ protected void testCallingDeleteObjectTwiceDoesNotThroughException() {}
+
+ @Override
+ protected String getLocalTmpDir() throws Exception {
+ return null;
+ }
+
+ @Override
+ protected String getIncompleteObjectName(RecoverableWriter.ResumeRecoverable recoverable) {
+ fail(
+ "GCS integration tests should not call this method, because incomplete state may be "
+ + "stored at multiple paths.");
+ return null;
+ }
+}
diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/utils/ConfigUtilsAuthTypeTest.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/utils/ConfigUtilsAuthTypeTest.java
new file mode 100644
index 00000000000000..0e9ea6e69942c2
--- /dev/null
+++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/utils/ConfigUtilsAuthTypeTest.java
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.fs.gs.utils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.fs.gs.TestUtils;
+import org.apache.flink.testutils.logging.LoggerAuditingExtension;
+
+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemConfiguration;
+import com.google.cloud.hadoop.util.HadoopCredentialsConfiguration;
+import com.google.cloud.hadoop.util.HadoopCredentialsConfiguration.AuthenticationType;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.slf4j.event.Level;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Tests that the two documented credential setups give gcs-connector and the RecoverableWriter the
+ * same credentials, and that Flink warns when they don't.
+ */
+class ConfigUtilsAuthTypeTest {
+
+ private static final String CONF_DIR = "/hadoop/conf";
+
+ private static final String APPLICATION_CREDENTIALS = "/opt/application.json";
+
+ private static final String KEYFILE = "/opt/keyfile.json";
+
+ @RegisterExtension
+ private final LoggerAuditingExtension warnings =
+ new LoggerAuditingExtension(ConfigUtils.class, Level.WARN);
+
+ @Test
+ void warnsForApplicationCredentialsWithoutAuthType() {
+ final org.apache.hadoop.conf.Configuration hadoopConfig =
+ loadCredentials(
+ Collections.emptyMap(),
+ Collections.emptyMap(),
+ APPLICATION_CREDENTIALS,
+ APPLICATION_CREDENTIALS);
+
+ assertThat(hadoopConfig.get("fs.gs.auth.type")).isNull();
+ assertThat(connectorAuthType(hadoopConfig)).isEqualTo(AuthenticationType.COMPUTE_ENGINE);
+ assertThat(warnings.getMessages())
+ .singleElement()
+ .asString()
+ .contains(APPLICATION_CREDENTIALS, "uses COMPUTE_ENGINE", "to APPLICATION_DEFAULT");
+ }
+
+ @Test
+ void warnsForKeyfileWithoutAuthType() {
+ final org.apache.hadoop.conf.Configuration hadoopConfig =
+ loadCredentials(
+ Collections.emptyMap(),
+ Collections.singletonMap(
+ "google.cloud.auth.service.account.json.keyfile", KEYFILE),
+ null,
+ KEYFILE);
+
+ assertThat(hadoopConfig.get("fs.gs.auth.type")).isNull();
+ assertThat(warnings.getMessages())
+ .singleElement()
+ .asString()
+ .contains(KEYFILE, "to SERVICE_ACCOUNT_JSON_KEYFILE");
+ }
+
+ @Test
+ void warnsForApplicationCredentialsWhenKeyfileIsAlsoSet() {
+ loadCredentials(
+ Collections.emptyMap(),
+ Collections.singletonMap("google.cloud.auth.service.account.json.keyfile", KEYFILE),
+ APPLICATION_CREDENTIALS,
+ APPLICATION_CREDENTIALS);
+
+ assertThat(warnings.getMessages())
+ .singleElement()
+ .asString()
+ .contains(APPLICATION_CREDENTIALS, "to APPLICATION_DEFAULT")
+ .doesNotContain(KEYFILE);
+ }
+
+ @Test
+ void applicationDefaultSetupUsesApplicationCredentials() {
+ final org.apache.hadoop.conf.Configuration hadoopConfig =
+ loadCredentials(
+ Collections.singletonMap("gs.auth.type", "APPLICATION_DEFAULT"),
+ Collections.emptyMap(),
+ APPLICATION_CREDENTIALS,
+ APPLICATION_CREDENTIALS);
+
+ assertThat(connectorAuthType(hadoopConfig))
+ .isEqualTo(AuthenticationType.APPLICATION_DEFAULT);
+ assertThat(warnings.getMessages()).isEmpty();
+ }
+
+ @Test
+ void keyfileSetupUsesKeyfile() {
+ final org.apache.hadoop.conf.Configuration hadoopConfig =
+ loadCredentials(
+ Collections.singletonMap("gs.auth.type", "SERVICE_ACCOUNT_JSON_KEYFILE"),
+ Collections.singletonMap(
+ "google.cloud.auth.service.account.json.keyfile", KEYFILE),
+ null,
+ KEYFILE);
+
+ assertThat(connectorAuthType(hadoopConfig))
+ .isEqualTo(AuthenticationType.SERVICE_ACCOUNT_JSON_KEYFILE);
+ assertThat(
+ HadoopCredentialsConfiguration.SERVICE_ACCOUNT_JSON_KEYFILE_SUFFIX
+ .withPrefixes(
+ GoogleHadoopFileSystemConfiguration.CONFIG_KEY_PREFIXES)
+ .get(hadoopConfig, hadoopConfig::get))
+ .isEqualTo(KEYFILE);
+ assertThat(warnings.getMessages()).isEmpty();
+ }
+
+ @Test
+ void doesNotWarnForAuthTypeWithHadoopPrefix() {
+ final org.apache.hadoop.conf.Configuration hadoopConfig =
+ loadCredentials(
+ Collections.emptyMap(),
+ Collections.singletonMap("google.cloud.auth.type", "APPLICATION_DEFAULT"),
+ APPLICATION_CREDENTIALS,
+ APPLICATION_CREDENTIALS);
+
+ assertThat(connectorAuthType(hadoopConfig))
+ .isEqualTo(AuthenticationType.APPLICATION_DEFAULT);
+ assertThat(warnings.getMessages()).isEmpty();
+ }
+
+ @Test
+ void doesNotWarnWithoutCredentialsFile() {
+ loadCredentials(Collections.emptyMap(), Collections.emptyMap(), null, null);
+
+ assertThat(warnings.getMessages()).isEmpty();
+ }
+
+ @Test
+ void doesNotWarnWhenServiceAccountIsDisabled() {
+ loadCredentials(
+ Collections.emptyMap(),
+ Collections.singletonMap("google.cloud.auth.service.account.enable", "false"),
+ APPLICATION_CREDENTIALS,
+ null);
+
+ assertThat(warnings.getMessages()).isEmpty();
+ }
+
+ /**
+ * Builds the Hadoop config the way GSFileSystemFactory does, loads the RecoverableWriter
+ * credentials and checks which file they came from.
+ */
+ private static org.apache.hadoop.conf.Configuration loadCredentials(
+ Map flinkOptions,
+ Map coreSite,
+ String applicationCredentials,
+ String expectedCredentialsFile) {
+ final Map envs = new HashMap<>();
+ envs.put("HADOOP_CONF_DIR", CONF_DIR);
+ if (applicationCredentials != null) {
+ envs.put("GOOGLE_APPLICATION_CREDENTIALS", applicationCredentials);
+ }
+ final GoogleCredentials credentials = GoogleCredentials.newBuilder().build();
+ final Map credentialsFiles = new HashMap<>();
+ if (expectedCredentialsFile != null) {
+ credentialsFiles.put(expectedCredentialsFile, credentials);
+ }
+ final TestingConfigContext configContext =
+ new TestingConfigContext(
+ envs,
+ Collections.singletonMap(CONF_DIR, TestUtils.hadoopConfigFromMap(coreSite)),
+ credentialsFiles);
+
+ final org.apache.hadoop.conf.Configuration hadoopConfig =
+ ConfigUtils.getHadoopConfiguration(
+ Configuration.fromMap(flinkOptions), configContext);
+ if (expectedCredentialsFile == null) {
+ assertThat(ConfigUtils.getStorageCredentials(hadoopConfig, configContext)).isEmpty();
+ } else {
+ assertThat(ConfigUtils.getStorageCredentials(hadoopConfig, configContext))
+ .containsSame(credentials);
+ }
+ return hadoopConfig;
+ }
+
+ private static AuthenticationType connectorAuthType(
+ org.apache.hadoop.conf.Configuration hadoopConfig) {
+ return HadoopCredentialsConfiguration.AUTHENTICATION_TYPE_SUFFIX
+ .withPrefixes(GoogleHadoopFileSystemConfiguration.CONFIG_KEY_PREFIXES)
+ .get(hadoopConfig, hadoopConfig::getEnum);
+ }
+}
diff --git a/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterExceptionITCase.java b/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterExceptionITCase.java
index c82f2e2d4dd5f2..d6f37500ee6966 100644
--- a/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterExceptionITCase.java
+++ b/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterExceptionITCase.java
@@ -31,6 +31,8 @@
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
+import javax.annotation.Nullable;
+
import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
@@ -75,6 +77,8 @@ static void cleanUp() throws Exception {
FileSystem.initialize(new Configuration());
}
+ /** Returns the local directory the writer stages data in, or null if it uses none. */
+ @Nullable
protected abstract String getLocalTmpDir() throws Exception;
@BeforeEach
@@ -82,6 +86,10 @@ void prepare() throws Exception {
basePathForTest = new Path(basePath, StringUtils.getRandomString(RND, 16, 16, 'a', 'z'));
final String defaultTmpDir = getLocalTmpDir();
+ if (defaultTmpDir == null) {
+ // The suite does not use local storage. No need to create directory.
+ return;
+ }
final java.nio.file.Path path = Paths.get(defaultTmpDir);
if (!Files.exists(path)) {
diff --git a/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterITCase.java b/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterITCase.java
index 647670f3f28be8..594added161357 100644
--- a/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterITCase.java
+++ b/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterITCase.java
@@ -34,6 +34,8 @@
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
+import javax.annotation.Nullable;
+
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
@@ -93,6 +95,8 @@ void prepare() throws Exception {
cleanupLocalDir();
}
+ /** Returns the local directory the writer stages data in, or null if it uses none. */
+ @Nullable
protected abstract String getLocalTmpDir() throws Exception;
protected abstract String getIncompleteObjectName(
@@ -100,6 +104,10 @@ protected abstract String getIncompleteObjectName(
private void cleanupLocalDir() throws Exception {
final String defaultTmpDir = getLocalTmpDir();
+ if (defaultTmpDir == null) {
+ // The suite did not use local storage. No cleanup needed.
+ return;
+ }
final java.nio.file.Path defaultTmpPath = Paths.get(defaultTmpDir);
if (Files.exists(defaultTmpPath)) {
@@ -120,7 +128,20 @@ private void cleanupLocalDir() throws Exception {
@AfterEach
void cleanupAndCheckTmpCleanup() throws Exception {
+ try {
+ checkAndDeleteLocalTmpDir();
+ } finally {
+ // delete also object store dir, even if the local tmp dir check failed.
+ getFileSystem().delete(basePathForTest, true);
+ }
+ }
+
+ private void checkAndDeleteLocalTmpDir() throws Exception {
final String defaultTmpDir = getLocalTmpDir();
+ if (defaultTmpDir == null) {
+ // The suite did not use local storage. No cleanup needed.
+ return;
+ }
final java.nio.file.Path localTmpDir = Paths.get(defaultTmpDir);
// delete local tmp dir.
@@ -129,9 +150,6 @@ void cleanupAndCheckTmpCleanup() throws Exception {
assertThat(files).isEmpty();
}
Files.delete(localTmpDir);
-
- // delete also object store dir.
- getFileSystem().delete(basePathForTest, true);
}
protected static FileSystem getFileSystem() throws Exception {
@@ -181,7 +199,7 @@ void testCommitAfterPersist() throws Exception {
}
@Test
- void testCleanupRecoverableState() throws Exception {
+ protected void testCleanupRecoverableState() throws Exception {
final RecoverableWriter writer = getRecoverableWriter();
final Path path = new Path(basePathForTest, "part-0");
@@ -220,7 +238,7 @@ void testCleanupRecoverableState() throws Exception {
}
@Test
- void testCallingDeleteObjectTwiceDoesNotThroughException() throws Exception {
+ protected void testCallingDeleteObjectTwiceDoesNotThroughException() throws Exception {
final RecoverableWriter writer = getRecoverableWriter();
final Path path = new Path(basePathForTest, "part-0");