diff --git a/docs/content.zh/docs/deployment/filesystems/gcs.md b/docs/content.zh/docs/deployment/filesystems/gcs.md index 0b971b24747bab..667a7245dd97ac 100644 --- a/docs/content.zh/docs/deployment/filesystems/gcs.md +++ b/docs/content.zh/docs/deployment/filesystems/gcs.md @@ -71,7 +71,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place Flink provides the `flink-gs-fs-hadoop` file system to write to GCS. This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it. -`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.68.0) library to provide `RecoverableWriter` support. +`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/3.1.22) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.79.0) library to provide `RecoverableWriter` support. This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}). @@ -84,7 +84,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/ ### Configuration -The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}). +The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}). For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}). Flink will internally translate this back to `fs.gs.http.connect-timeout`. @@ -107,41 +107,38 @@ You can also set `gcs-connector` options directly in the Hadoop `core-site.xml` ### Authentication to access GCS -Most operations on GCS require authentication. To provide authentication credentials, either: +Most operations on GCS require authentication. `flink-gs-fs-hadoop` uses two clients, which select their credentials separately: -* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the JSON credentials file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run. This is the **recommended** method. +* `gcs-connector` handles checkpoints, savepoints and all other file system access. It uses the credentials that `gs.auth.type` selects, and the credentials of the Compute Engine or GKE service account (`COMPUTE_ENGINE`) if `gs.auth.type` isn't set. See the [gcs-connector configuration](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md#authentication) for all values. +* The `google-cloud-storage` library handles `RecoverableWriter`, which the `FileSink` uses. It uses the JSON credentials file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, then the one in the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml`, and otherwise the [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials). +To use a JSON credentials file with both clients, either: -* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path to the JSON credentials file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)): +* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run, and set `gs.auth.type` in the [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}). This is the **recommended** method. -``` - - - google.cloud.auth.service.account.json.keyfile - PATH TO GOOGLE AUTHENTICATION JSON FILE - - +```yaml +gs.auth.type: APPLICATION_DEFAULT ``` -For `flink-gs-fs-hadoop` to use credentials via either of these two methods, the use of service accounts for authentication must be enabled. This is enabled by default; however, it can be disabled in `core-site.xml` by setting: +* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path of a service account key file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)), and set `gs.auth.type: SERVICE_ACCOUNT_JSON_KEYFILE` in the Flink configuration file: ``` - google.cloud.auth.service.account.enable - false + google.cloud.auth.service.account.json.keyfile + PATH TO GOOGLE AUTHENTICATION JSON FILE ``` -{{< hint warning >}} -`gcs-connector` supports additional options to provide authentication credentials besides the `google.cloud.auth.service.account.json.keyfile` option described above. - -However, if you use any of those other options, the provided credentials will not be used by the `google-cloud-storage` library, which provides `RecoverableWriter` support, so Flink recoverable-write operations would be expected to fail. +Flink logs a warning at startup if `RecoverableWriter` uses one of these files and `gs.auth.type` isn't set. -For this reason, use of the `gcs-connector` authentication-credentials options other than `google.cloud.auth.service.account.json.keyfile` is **not recommended.** +Setting `google.cloud.auth.service.account.enable` to `false` in `core-site.xml` makes `RecoverableWriter` ignore `google.cloud.auth.service.account.json.keyfile` and use the application default credentials. `gcs-connector` doesn't read this property. +{{< hint warning >}} +The other values of `gs.auth.type`, such as `USER_CREDENTIALS` or `ACCESS_TOKEN_PROVIDER`, only apply to `gcs-connector`. `RecoverableWriter` doesn't use those credentials, so it can run as a different identity than `gcs-connector` and its operations may fail. +If you use the `FileSink`, set `gs.auth.type` as described above, or keep the default `COMPUTE_ENGINE` without configuring either file. {{< /hint >}} diff --git a/docs/content/docs/deployment/filesystems/gcs.md b/docs/content/docs/deployment/filesystems/gcs.md index 164dbcc26156aa..ecf486d8991a89 100644 --- a/docs/content/docs/deployment/filesystems/gcs.md +++ b/docs/content/docs/deployment/filesystems/gcs.md @@ -71,7 +71,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place Flink provides the `flink-gs-fs-hadoop` file system to write to GCS. This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it. -`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.68.0) library to provide `RecoverableWriter` support. +`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/3.1.22) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.79.0) library to provide `RecoverableWriter` support. This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}). @@ -84,7 +84,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/ ### Configuration -The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). +The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). Flink will internally translate this back to `fs.gs.http.connect-timeout`. @@ -107,41 +107,38 @@ You can also set `gcs-connector` options directly in the Hadoop `core-site.xml` ### Authentication to access GCS -Most operations on GCS require authentication. To provide authentication credentials, either: +Most operations on GCS require authentication. `flink-gs-fs-hadoop` uses two clients, which select their credentials separately: -* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the JSON credentials file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run. This is the **recommended** method. +* `gcs-connector` handles checkpoints, savepoints and all other file system access. It uses the credentials that `gs.auth.type` selects, and the credentials of the Compute Engine or GKE service account (`COMPUTE_ENGINE`) if `gs.auth.type` isn't set. See the [gcs-connector configuration](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md#authentication) for all values. +* The `google-cloud-storage` library handles `RecoverableWriter`, which the `FileSink` uses. It uses the JSON credentials file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, then the one in the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml`, and otherwise the [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials). +To use a JSON credentials file with both clients, either: -* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path to the JSON credentials file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)): - -``` - - - google.cloud.auth.service.account.json.keyfile - PATH TO GOOGLE AUTHENTICATION JSON FILE - - +* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run, and set `gs.auth.type` in the [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). This is the **recommended** method. + +```yaml +gs.auth.type: APPLICATION_DEFAULT ``` -For `flink-gs-fs-hadoop` to use credentials via either of these two methods, the use of service accounts for authentication must be enabled. This is enabled by default; however, it can be disabled in `core-site.xml` by setting: +* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path of a service account key file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)), and set `gs.auth.type: SERVICE_ACCOUNT_JSON_KEYFILE` in the Flink configuration file: ``` - google.cloud.auth.service.account.enable - false + google.cloud.auth.service.account.json.keyfile + PATH TO GOOGLE AUTHENTICATION JSON FILE ``` -{{< hint warning >}} -`gcs-connector` supports additional options to provide authentication credentials besides the `google.cloud.auth.service.account.json.keyfile` option described above. - -However, if you use any of those other options, the provided credentials will not be used by the `google-cloud-storage` library, which provides `RecoverableWriter` support, so Flink recoverable-write operations would be expected to fail. +Flink logs a warning at startup if `RecoverableWriter` uses one of these files and `gs.auth.type` isn't set. -For this reason, use of the `gcs-connector` authentication-credentials options other than `google.cloud.auth.service.account.json.keyfile` is **not recommended.** +Setting `google.cloud.auth.service.account.enable` to `false` in `core-site.xml` makes `RecoverableWriter` ignore `google.cloud.auth.service.account.json.keyfile` and use the application default credentials. `gcs-connector` doesn't read this property. +{{< hint warning >}} +The other values of `gs.auth.type`, such as `USER_CREDENTIALS` or `ACCESS_TOKEN_PROVIDER`, only apply to `gcs-connector`. `RecoverableWriter` doesn't use those credentials, so it can run as a different identity than `gcs-connector` and its operations may fail. +If you use the `FileSink`, set `gs.auth.type` as described above, or keep the default `COMPUTE_ENGINE` without configuring either file. {{< /hint >}} diff --git a/flink-filesystems/flink-gs-fs-hadoop/archunit-violations/cce59259-d3cb-4048-a1bd-31197f847189 b/flink-filesystems/flink-gs-fs-hadoop/archunit-violations/cce59259-d3cb-4048-a1bd-31197f847189 index f3c98676934cb7..8b8372de55a65e 100644 --- a/flink-filesystems/flink-gs-fs-hadoop/archunit-violations/cce59259-d3cb-4048-a1bd-31197f847189 +++ b/flink-filesystems/flink-gs-fs-hadoop/archunit-violations/cce59259-d3cb-4048-a1bd-31197f847189 @@ -10,3 +10,21 @@ org.apache.flink.fs.gs.GSRecoverableWriterITCase does not satisfy: only one of t * reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\ * reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\ or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule +org.apache.flink.fs.gs.HadoopGSFileSystemITCase does not satisfy: only one of the following predicates match:\ +* reside in a package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type InternalMiniClusterExtension and annotated with @RegisterExtension\ +* reside outside of package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type MiniClusterExtension and annotated with @RegisterExtension or are , and of type MiniClusterTestEnvironment and annotated with @TestEnv\ +* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\ +* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\ + or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule +org.apache.flink.fs.gs.HadoopGSRecoverableWriterExceptionITCase does not satisfy: only one of the following predicates match:\ +* reside in a package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type InternalMiniClusterExtension and annotated with @RegisterExtension\ +* reside outside of package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type MiniClusterExtension and annotated with @RegisterExtension or are , and of type MiniClusterTestEnvironment and annotated with @TestEnv\ +* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\ +* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\ + or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule +org.apache.flink.fs.gs.HadoopGSRecoverableWriterITCase does not satisfy: only one of the following predicates match:\ +* reside in a package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type InternalMiniClusterExtension and annotated with @RegisterExtension\ +* reside outside of package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type MiniClusterExtension and annotated with @RegisterExtension or are , and of type MiniClusterTestEnvironment and annotated with @TestEnv\ +* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\ +* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\ + or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule diff --git a/flink-filesystems/flink-gs-fs-hadoop/pom.xml b/flink-filesystems/flink-gs-fs-hadoop/pom.xml index 48c7442cb99879..21503676f2955c 100644 --- a/flink-filesystems/flink-gs-fs-hadoop/pom.xml +++ b/flink-filesystems/flink-gs-fs-hadoop/pom.xml @@ -34,11 +34,11 @@ under the License. - 2.68.0 - hadoop3-2.2.18 - 0.128.7 + 2.79.0 + 3.1.22 + 0.143.0 - 1.81.0 + 1.85.1 @@ -67,6 +67,14 @@ under the License. test + + org.apache.flink + flink-hadoop-fs + ${project.version} + test + test-jar + + org.apache.flink flink-hadoop-fs @@ -156,6 +164,23 @@ under the License. org.codehaus.mojo animal-sniffer-annotations + + org.checkerframework + checker-qual + + + com.google.j2objc + j2objc-annotations + + + com.google.guava + failureaccess + + + + io.grpc + grpc-testing + javax.annotation diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/utils/ConfigUtils.java b/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/utils/ConfigUtils.java index cdb9bfffbdd02c..b1bac70ca5230d 100644 --- a/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/utils/ConfigUtils.java +++ b/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/utils/ConfigUtils.java @@ -24,6 +24,9 @@ import com.google.auth.oauth2.GoogleCredentials; import com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemConfiguration; +import com.google.cloud.hadoop.util.HadoopConfigurationProperty; +import com.google.cloud.hadoop.util.HadoopCredentialsConfiguration; +import com.google.cloud.hadoop.util.HadoopCredentialsConfiguration.AuthenticationType; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -53,6 +56,9 @@ public class ConfigUtils { private static final String HADOOP_OPTION_SERVICE_ACCOUNT_JSON_KEYFILE = "google.cloud.auth.service.account.json.keyfile"; + private static final HadoopConfigurationProperty CONNECTOR_AUTH_TYPE = + HadoopCredentialsConfiguration.AUTHENTICATION_TYPE_SUFFIX; + /** * Loads the Hadoop configuration, by loading from a Hadoop conf dir (if one exists) and then * overlaying properties derived from the Flink config. @@ -109,7 +115,7 @@ public static org.apache.hadoop.conf.Configuration getHadoopConfiguration( public static Optional getStorageCredentials( org.apache.hadoop.conf.Configuration hadoopConfig, ConfigContext configContext) { - // follow the same rules as for the Hadoop connector, i.e. + // RecoverableWriter credential order: // 1) only use service credentials at all if Hadoop // "google.cloud.auth.service.account.enable" is true (default: true) // 2) use GOOGLE_APPLICATION_CREDENTIALS as location of credentials, if supplied @@ -119,6 +125,8 @@ public static Optional getStorageCredentials( // store any credentials we are to use, here Optional credentialsPath = Optional.empty(); + // the gcs-connector authentication type that uses the same credentials + AuthenticationType matchingAuthType = null; // only look for credentials if service account support is enabled boolean enableServiceAccount = @@ -129,10 +137,12 @@ public static Optional getStorageCredentials( // "google.cloud.auth.service.account.json.keyfile" from Hadoop credentialsPath = configContext.getenv("GOOGLE_APPLICATION_CREDENTIALS"); if (credentialsPath.isPresent()) { + matchingAuthType = AuthenticationType.APPLICATION_DEFAULT; LOGGER.info( "GSRecoverableWriter is using GOOGLE_APPLICATION_CREDENTIALS at {}", credentialsPath.get()); } else { + matchingAuthType = AuthenticationType.SERVICE_ACCOUNT_JSON_KEYFILE; credentialsPath = Optional.ofNullable( hadoopConfig.get(HADOOP_OPTION_SERVICE_ACCOUNT_JSON_KEYFILE)); @@ -149,6 +159,15 @@ public static Optional getStorageCredentials( LOGGER.info( "Creating GSRecoverableWriter using credentials from {}", credentialsPath.get()); + if (!isConnectorAuthTypeSet(hadoopConfig)) { + LOGGER.warn( + "GSRecoverableWriter uses the credentials from {}, but gcs-connector uses" + + " {} credentials because 'gs.auth.type' is not set. Set" + + " 'gs.auth.type' to {} to use the same credentials for both.", + credentialsPath.get(), + CONNECTOR_AUTH_TYPE.getDefault(), + matchingAuthType); + } GoogleCredentials credentials = configContext.loadStorageCredentialsFromFile(credentialsPath.get()); return Optional.of(credentials); @@ -158,6 +177,17 @@ public static Optional getStorageCredentials( } } + private static boolean isConnectorAuthTypeSet( + org.apache.hadoop.conf.Configuration hadoopConfig) { + // gcs-connector reads the authentication type under each of its key prefixes + for (String prefix : GoogleHadoopFileSystemConfiguration.CONFIG_KEY_PREFIXES) { + if (hadoopConfig.get(prefix + CONNECTOR_AUTH_TYPE.getKey()) != null) { + return true; + } + } + return false; + } + public static Optional getGcsRootUrl( org.apache.hadoop.conf.Configuration hadoopConfig) { // Ignore the default value, only returning a value if actually included in the config diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE index cc18aa1c1a0b7f..f4b86d396846c5 100644 --- a/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE +++ b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE @@ -11,104 +11,113 @@ This project bundles the following dependencies under the Apache Software Licens - com.fasterxml.jackson.core:jackson-databind:2.22.2 - com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.22.2 - com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.22.2 -- com.fasterxml.woodstox:woodstox-core:7.0.0 +- com.fasterxml.woodstox:woodstox-core:7.2.0 - com.google.android:annotations:4.1.1.4 -- com.google.api-client:google-api-client-jackson2:2.0.1 -- com.google.api-client:google-api-client:2.7.2 -- com.google.api.grpc:gapic-google-cloud-storage-v2:2.68.0 -- com.google.api.grpc:grpc-google-cloud-storage-v2:2.68.0 +- com.google.api-client:google-api-client-gson:2.9.2 +- com.google.api-client:google-api-client:2.9.2 +- com.google.api.grpc:gapic-google-cloud-storage-v2:2.79.0 +- com.google.api.grpc:grpc-google-cloud-storage-control-v2:2.49.0 +- com.google.api.grpc:grpc-google-cloud-storage-v2:2.79.0 +- com.google.api.grpc:proto-google-cloud-logging-v2:0.110.4 - com.google.api.grpc:proto-google-cloud-monitoring-v3:3.52.0 -- com.google.api.grpc:proto-google-cloud-storage-v2:2.68.0 -- com.google.api.grpc:proto-google-common-protos:2.71.0 -- com.google.api.grpc:proto-google-iam-v1:1.66.0 +- com.google.api.grpc:proto-google-cloud-storage-control-v2:2.49.0 +- com.google.api.grpc:proto-google-cloud-storage-v2:2.79.0 +- com.google.api.grpc:proto-google-common-protos:2.82.0 +- com.google.api.grpc:proto-google-iam-v1:1.77.0 - com.google.apis:google-api-services-iamcredentials:v1-rev20211203-2.0.0 -- com.google.apis:google-api-services-storage:v1-rev20260204-2.0.0 +- com.google.apis:google-api-services-storage:v1-rev20260524-2.0.0 - com.google.auto.value:auto-value-annotations:1.11.0 -- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.18 -- com.google.cloud.bigdataoss:gcsio:2.2.18 -- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.18 -- com.google.cloud.bigdataoss:util:2.2.18 -- com.google.cloud.opentelemetry:detector-resources-support:0.33.0 +- com.google.cloud.bigdataoss:gcs-connector:3.1.22 +- com.google.cloud.bigdataoss:gcsio:3.1.22 +- com.google.cloud.bigdataoss:util-hadoop:3.1.22 +- com.google.cloud.bigdataoss:util:3.1.22 +- com.google.cloud.opentelemetry:detector-resources-support:0.31.0 - com.google.cloud.opentelemetry:exporter-metrics:0.33.0 - com.google.cloud.opentelemetry:shared-resourcemapping:0.33.0 -- com.google.cloud:google-cloud-core-grpc:2.70.0 -- com.google.cloud:google-cloud-core-http:2.70.0 -- com.google.cloud:google-cloud-core:2.70.0 +- com.google.cloud:google-cloud-core-grpc:2.81.0 +- com.google.cloud:google-cloud-core-http:2.81.0 +- com.google.cloud:google-cloud-core:2.81.0 +- com.google.cloud:google-cloud-logging:3.21.4 - com.google.cloud:google-cloud-monitoring:3.52.0 -- com.google.cloud:google-cloud-storage:2.68.0 -- com.google.code.gson:gson:2.13.2 -- com.google.flogger:flogger-system-backend:0.7.1 -- com.google.flogger:flogger:0.7.1 -- com.google.flogger:google-extensions:0.7.1 +- com.google.cloud:google-cloud-storage-control:2.49.0 +- com.google.cloud:google-cloud-storage:2.79.0 +- com.google.code.gson:gson:2.14.0 +- com.google.flogger:flogger-system-backend:0.7.4 +- com.google.flogger:flogger:0.7.4 +- com.google.flogger:google-extensions:0.7.4 - com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava -- com.google.http-client:google-http-client-apache-v2:2.1.0 -- com.google.http-client:google-http-client-appengine:2.1.0 -- com.google.http-client:google-http-client-gson:2.1.0 -- com.google.http-client:google-http-client-jackson2:2.1.0 -- com.google.http-client:google-http-client:2.1.0 +- com.google.http-client:google-http-client-apache-v2:2.2.0 +- com.google.http-client:google-http-client-appengine:2.2.0 +- com.google.http-client:google-http-client-gson:2.2.0 +- com.google.http-client:google-http-client-jackson2:2.2.0 +- com.google.http-client:google-http-client:2.2.0 - com.google.oauth-client:google-oauth-client:1.39.0 - com.lmax:disruptor:3.4.2 - commons-codec:commons-codec:1.15 -- io.grpc:grpc-alts:1.81.0 -- io.grpc:grpc-api:1.81.0 -- io.grpc:grpc-auth:1.81.0 -- io.grpc:grpc-census:1.81.0 -- io.grpc:grpc-context:1.81.0 -- io.grpc:grpc-core:1.81.0 -- io.grpc:grpc-googleapis:1.81.0 -- io.grpc:grpc-grpclb:1.81.0 -- io.grpc:grpc-inprocess:1.81.0 -- io.grpc:grpc-netty-shaded:1.81.0 -- io.grpc:grpc-opentelemetry:1.81.0 -- io.grpc:grpc-protobuf-lite:1.81.0 -- io.grpc:grpc-protobuf:1.81.0 -- io.grpc:grpc-rls:1.81.0 -- io.grpc:grpc-services:1.81.0 -- io.grpc:grpc-stub:1.81.0 -- io.grpc:grpc-util:1.81.0 -- io.grpc:grpc-xds:1.81.0 +- dev.cel:common:0.13.1 +- dev.cel:protobuf:0.13.1 +- dev.cel:runtime:0.13.1 +- io.grpc:grpc-alts:1.85.1 +- io.grpc:grpc-api:1.85.1 +- io.grpc:grpc-auth:1.85.1 +- io.grpc:grpc-census:1.85.1 +- io.grpc:grpc-context:1.85.1 +- io.grpc:grpc-core:1.85.1 +- io.grpc:grpc-googleapis:1.85.1 +- io.grpc:grpc-grpclb:1.85.1 +- io.grpc:grpc-inprocess:1.85.1 +- io.grpc:grpc-netty-shaded:1.85.1 +- io.grpc:grpc-opentelemetry:1.85.1 +- io.grpc:grpc-protobuf-lite:1.85.1 +- io.grpc:grpc-protobuf:1.85.1 +- io.grpc:grpc-rls:1.85.1 +- io.grpc:grpc-services:1.85.1 +- io.grpc:grpc-stub:1.85.1 +- io.grpc:grpc-util:1.85.1 +- io.grpc:grpc-xds:1.85.1 - io.opencensus:opencensus-api:0.31.1 -- io.opencensus:opencensus-contrib-exemplar-util:0.31.0 -- io.opencensus:opencensus-contrib-grpc-metrics:0.31.0 +- io.opencensus:opencensus-contrib-exemplar-util:0.31.1 +- io.opencensus:opencensus-contrib-grpc-metrics:0.31.1 - io.opencensus:opencensus-contrib-http-util:0.31.1 -- io.opencensus:opencensus-contrib-resource-util:0.31.0 -- io.opencensus:opencensus-exporter-metrics-util:0.31.0 -- io.opencensus:opencensus-exporter-stats-stackdriver:0.31.0 -- io.opencensus:opencensus-impl-core:0.31.0 -- io.opencensus:opencensus-impl:0.31.0 +- io.opencensus:opencensus-contrib-resource-util:0.31.1 +- io.opencensus:opencensus-exporter-metrics-util:0.31.1 +- io.opencensus:opencensus-exporter-stats-stackdriver:0.31.1 +- io.opencensus:opencensus-impl-core:0.31.1 +- io.opencensus:opencensus-impl:0.31.1 - io.opentelemetry.contrib:opentelemetry-gcp-resources:1.37.0-alpha - io.opentelemetry.semconv:opentelemetry-semconv:1.29.0-alpha -- io.opentelemetry:opentelemetry-api:1.57.0 -- io.opentelemetry:opentelemetry-common:1.57.0 -- io.opentelemetry:opentelemetry-context:1.57.0 -- io.opentelemetry:opentelemetry-sdk-common:1.57.0 -- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.57.0 -- io.opentelemetry:opentelemetry-sdk-logs:1.57.0 -- io.opentelemetry:opentelemetry-sdk-metrics:1.57.0 -- io.opentelemetry:opentelemetry-sdk-trace:1.57.0 -- io.opentelemetry:opentelemetry-sdk:1.57.0 +- io.opentelemetry:opentelemetry-api:1.67.0 +- io.opentelemetry:opentelemetry-common:1.67.0 +- io.opentelemetry:opentelemetry-context:1.67.0 +- io.opentelemetry:opentelemetry-sdk-common:1.67.0 +- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.67.0 +- io.opentelemetry:opentelemetry-sdk-logs:1.67.0 +- io.opentelemetry:opentelemetry-sdk-metrics:1.67.0 +- io.opentelemetry:opentelemetry-sdk-trace:1.67.0 +- io.opentelemetry:opentelemetry-sdk:1.67.0 - io.perfmark:perfmark-api:0.27.0 - org.apache.httpcomponents:httpclient:4.5.13 - org.apache.httpcomponents:httpcore:4.4.14 -- org.conscrypt:conscrypt-openjdk-uber:2.5.2 +- org.conscrypt:conscrypt-openjdk-uber:2.8.0 This project bundles the following dependencies under BSD-3 License (https://opensource.org/licenses/BSD-3-Clause). See bundled license files for details. -- com.google.api:api-common:2.63.0 -- com.google.api:gax-grpc:2.80.0 -- com.google.api:gax-httpjson:2.80.0 -- com.google.api:gax:2.80.0 -- com.google.auth:google-auth-library-credentials:1.47.0 -- com.google.auth:google-auth-library-oauth2-http:1.47.0 -- com.google.protobuf:protobuf-java-util:4.33.2 -- com.google.protobuf:protobuf-java:4.33.2 +- com.google.api:api-common:2.74.0 +- com.google.api:gax-grpc:2.91.0 +- com.google.api:gax-httpjson:2.91.0 +- com.google.api:gax:2.91.0 +- com.google.auth:google-auth-library-credentials:1.58.0 +- com.google.auth:google-auth-library-oauth2-http:1.58.0 +- com.google.protobuf:protobuf-java-util:4.33.6 +- com.google.protobuf:protobuf-java:4.33.6 +- org.threeten:threeten-extra:1.8.0 - org.threeten:threetenbp:1.7.0 This project bundles the following dependencies under BSD License (https://opensource.org/licenses/bsd-license.php). See bundled license files for details. -- org.codehaus.woodstox:stax2-api:4.2.2 (https://github.com/FasterXML/stax2-api/tree/stax2-api-4.2.2) +- org.codehaus.woodstox:stax2-api:4.3.0 (https://github.com/FasterXML/stax2-api/tree/stax2-api-4.3.0) This project bundles the following dependencies under the Go License (https://golang.org/LICENSE). See bundled license files for details. diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/licenses/LICENSE.threeten-extra b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/licenses/LICENSE.threeten-extra new file mode 100644 index 00000000000000..6615eadc8c8a58 --- /dev/null +++ b/flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/licenses/LICENSE.threeten-extra @@ -0,0 +1,29 @@ +Copyright (c) 2007-present, Stephen Colebourne & Michael Nascimento Santos. + +All rights reserved. + +* Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of JSR-310 nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSFileSystemBehaviorITCase.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSFileSystemBehaviorITCase.java index 68cb21c759e16e..565199e0f8d4e8 100644 --- a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSFileSystemBehaviorITCase.java +++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSFileSystemBehaviorITCase.java @@ -44,7 +44,7 @@ class GSFileSystemBehaviorITCase extends FileSystemBehaviorTestSuite { @BeforeAll static void checkCredentialsAndSetup() { GSTestCredentials.assumeCredentialsAvailable(); - FileSystem.initialize(new Configuration(), null); + FileSystem.initialize(GSTestCredentials.createConfiguration(), null); } @AfterAll diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSRecoverableWriterITCase.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSRecoverableWriterITCase.java index fd25ee48b568d0..2058352d569732 100644 --- a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSRecoverableWriterITCase.java +++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSRecoverableWriterITCase.java @@ -56,7 +56,7 @@ class GSRecoverableWriterITCase { @BeforeAll static void checkCredentialsAndSetup() throws Exception { GSTestCredentials.assumeCredentialsAvailable(); - FileSystem.initialize(new Configuration(), null); + FileSystem.initialize(GSTestCredentials.createConfiguration(), null); basePath = new Path(GSTestCredentials.getTestBucketUri() + TEST_DATA_DIR); fileSystem = basePath.getFileSystem(); } diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSTestCredentials.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSTestCredentials.java index 0b26f2ccef957f..a0bf9c83593d58 100644 --- a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSTestCredentials.java +++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/GSTestCredentials.java @@ -18,8 +18,17 @@ package org.apache.flink.fs.gs; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.core.fs.Path; +import org.apache.flink.fs.gs.storage.GSBlobIdentifier; +import org.apache.flink.fs.gs.storage.GSBlobStorage; +import org.apache.flink.fs.gs.storage.GSBlobStorageImpl; +import org.apache.flink.fs.gs.utils.BlobUtils; + import javax.annotation.Nullable; +import java.util.List; + import static org.assertj.core.api.Assumptions.assumeThat; /** @@ -42,6 +51,16 @@ static void assumeCredentialsAvailable() { .isNotBlank(); } + /** + * Returns a configuration that makes gcs-connector use application default credentials, like + * the Google Cloud Storage client of the recoverable writer. + */ + static Configuration createConfiguration() { + final Configuration config = new Configuration(); + config.setString("gs.auth.type", "APPLICATION_DEFAULT"); + return config; + } + /** Returns the URI of the test bucket, e.g. {@code gs://my-bucket/temp/}. */ static String getTestBucketUri() { if (GCS_TEST_BUCKET == null) { @@ -50,4 +69,21 @@ static String getTestBucketUri() { } return "gs://" + GCS_TEST_BUCKET + "/temp/"; } + + /** + * Deletes the temporary objects the recoverable writer created for files under {@code + * basePath}. They live under the in-progress prefix at the bucket root, outside {@code + * basePath}, and only a commit removes them, so uncommitted streams leave them behind. + */ + static void deleteTemporaryObjects(Path basePath) { + final GSFileSystemFactory factory = new GSFileSystemFactory(); + factory.configure(createConfiguration()); + final GSBlobStorage storage = new GSBlobStorageImpl(factory.getStorage()); + final GSBlobIdentifier base = BlobUtils.parseUri(basePath.toUri()); + final List temporaryObjects = + storage.list(base.bucketName, BlobUtils.getTemporaryObjectPartialName(base)); + if (!temporaryObjects.isEmpty()) { + storage.delete(temporaryObjects); + } + } } diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSFileSystemITCase.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSFileSystemITCase.java new file mode 100644 index 00000000000000..84e7cafe84953c --- /dev/null +++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSFileSystemITCase.java @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.fs.gs; + +import org.apache.flink.core.fs.FileSystem; +import org.apache.flink.core.fs.Path; +import org.apache.flink.runtime.fs.hdfs.AbstractHadoopFileSystemITTest; + +import org.junit.jupiter.api.BeforeAll; + +import java.io.IOException; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; + +/** Integration tests for GCS file system support. */ +class HadoopGSFileSystemITCase extends AbstractHadoopFileSystemITTest { + + @BeforeAll + static void setup() throws IOException { + GSTestCredentials.assumeCredentialsAvailable(); + FileSystem.initialize(GSTestCredentials.createConfiguration(), null); + basePath = new Path(GSTestCredentials.getTestBucketUri() + "tests-" + UUID.randomUUID()); + fs = basePath.getFileSystem(); + assertThat(fs.exists(basePath)).isFalse(); + } +} diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterExceptionITCase.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterExceptionITCase.java new file mode 100644 index 00000000000000..58b9e9621b7937 --- /dev/null +++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterExceptionITCase.java @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.fs.gs; + +import org.apache.flink.core.fs.FileSystem; +import org.apache.flink.core.fs.Path; +import org.apache.flink.runtime.fs.hdfs.AbstractHadoopRecoverableWriterExceptionITCase; + +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; + +import java.io.IOException; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Integration tests for exception handling in {@link + * org.apache.flink.fs.gs.writer.GSRecoverableWriter}. + */ +class HadoopGSRecoverableWriterExceptionITCase + extends AbstractHadoopRecoverableWriterExceptionITCase { + + @BeforeAll + static void setup() throws IOException { + GSTestCredentials.assumeCredentialsAvailable(); + FileSystem.initialize(GSTestCredentials.createConfiguration(), null); + basePath = new Path(GSTestCredentials.getTestBucketUri() + "tests-" + UUID.randomUUID()); + assertThat(basePath.getFileSystem().exists(basePath)).isFalse(); + skipped = false; + } + + @AfterAll + static void cleanUpTemporaryObjects() { + if (!skipped) { + GSTestCredentials.deleteTemporaryObjects(basePath); + } + } + + @Override + protected String getLocalTmpDir() throws Exception { + return null; + } +} diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterITCase.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterITCase.java new file mode 100644 index 00000000000000..4557b8f7f9e833 --- /dev/null +++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/HadoopGSRecoverableWriterITCase.java @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.fs.gs; + +import org.apache.flink.configuration.Configuration; +import org.apache.flink.configuration.MemorySize; +import org.apache.flink.core.fs.FileSystem; +import org.apache.flink.core.fs.Path; +import org.apache.flink.core.fs.RecoverableWriter; +import org.apache.flink.runtime.fs.hdfs.AbstractHadoopRecoverableWriterITCase; + +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Disabled; +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.fail; + +/** Integration tests for {@link org.apache.flink.fs.gs.writer.GSRecoverableWriter}. */ +class HadoopGSRecoverableWriterITCase extends AbstractHadoopRecoverableWriterITCase { + + @BeforeAll + static void setup() throws IOException { + GSTestCredentials.assumeCredentialsAvailable(); + // The smallest chunk size GCS accepts, so each 7 MiB component of the *WithMultiPart + // tests is uploaded in several chunks instead of one (the default is 16 MiB). + final Configuration config = GSTestCredentials.createConfiguration(); + config.set(GSFileSystemOptions.WRITER_CHUNK_SIZE, MemorySize.parse("256kb")); + FileSystem.initialize(config, null); + basePath = new Path(GSTestCredentials.getTestBucketUri() + "tests-" + UUID.randomUUID()); + assertThat(basePath.getFileSystem().exists(basePath)).isFalse(); + bigDataChunk = createBigDataChunk(BIG_CHUNK_DATA_PATTERN, 7L << 20); + skipped = false; + } + + @AfterAll + static void cleanUpTemporaryObjects() { + if (!skipped) { + GSTestCredentials.deleteTemporaryObjects(basePath); + } + } + + @Test + @Override + @Disabled("This test is not meaningful. GSRecoverableWriter#cleanupRecoverableState is no-op.") + protected void testCleanupRecoverableState() {} + + @Test + @Override + @Disabled("This test assumes one file for state, but GSRecoverableWriter uses multiple files.") + protected void testCallingDeleteObjectTwiceDoesNotThroughException() {} + + @Override + protected String getLocalTmpDir() throws Exception { + return null; + } + + @Override + protected String getIncompleteObjectName(RecoverableWriter.ResumeRecoverable recoverable) { + fail( + "GCS integration tests should not call this method, because incomplete state may be " + + "stored at multiple paths."); + return null; + } +} diff --git a/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/utils/ConfigUtilsAuthTypeTest.java b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/utils/ConfigUtilsAuthTypeTest.java new file mode 100644 index 00000000000000..0e9ea6e69942c2 --- /dev/null +++ b/flink-filesystems/flink-gs-fs-hadoop/src/test/java/org/apache/flink/fs/gs/utils/ConfigUtilsAuthTypeTest.java @@ -0,0 +1,214 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.fs.gs.utils; + +import org.apache.flink.configuration.Configuration; +import org.apache.flink.fs.gs.TestUtils; +import org.apache.flink.testutils.logging.LoggerAuditingExtension; + +import com.google.auth.oauth2.GoogleCredentials; +import com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemConfiguration; +import com.google.cloud.hadoop.util.HadoopCredentialsConfiguration; +import com.google.cloud.hadoop.util.HadoopCredentialsConfiguration.AuthenticationType; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.RegisterExtension; +import org.slf4j.event.Level; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Tests that the two documented credential setups give gcs-connector and the RecoverableWriter the + * same credentials, and that Flink warns when they don't. + */ +class ConfigUtilsAuthTypeTest { + + private static final String CONF_DIR = "/hadoop/conf"; + + private static final String APPLICATION_CREDENTIALS = "/opt/application.json"; + + private static final String KEYFILE = "/opt/keyfile.json"; + + @RegisterExtension + private final LoggerAuditingExtension warnings = + new LoggerAuditingExtension(ConfigUtils.class, Level.WARN); + + @Test + void warnsForApplicationCredentialsWithoutAuthType() { + final org.apache.hadoop.conf.Configuration hadoopConfig = + loadCredentials( + Collections.emptyMap(), + Collections.emptyMap(), + APPLICATION_CREDENTIALS, + APPLICATION_CREDENTIALS); + + assertThat(hadoopConfig.get("fs.gs.auth.type")).isNull(); + assertThat(connectorAuthType(hadoopConfig)).isEqualTo(AuthenticationType.COMPUTE_ENGINE); + assertThat(warnings.getMessages()) + .singleElement() + .asString() + .contains(APPLICATION_CREDENTIALS, "uses COMPUTE_ENGINE", "to APPLICATION_DEFAULT"); + } + + @Test + void warnsForKeyfileWithoutAuthType() { + final org.apache.hadoop.conf.Configuration hadoopConfig = + loadCredentials( + Collections.emptyMap(), + Collections.singletonMap( + "google.cloud.auth.service.account.json.keyfile", KEYFILE), + null, + KEYFILE); + + assertThat(hadoopConfig.get("fs.gs.auth.type")).isNull(); + assertThat(warnings.getMessages()) + .singleElement() + .asString() + .contains(KEYFILE, "to SERVICE_ACCOUNT_JSON_KEYFILE"); + } + + @Test + void warnsForApplicationCredentialsWhenKeyfileIsAlsoSet() { + loadCredentials( + Collections.emptyMap(), + Collections.singletonMap("google.cloud.auth.service.account.json.keyfile", KEYFILE), + APPLICATION_CREDENTIALS, + APPLICATION_CREDENTIALS); + + assertThat(warnings.getMessages()) + .singleElement() + .asString() + .contains(APPLICATION_CREDENTIALS, "to APPLICATION_DEFAULT") + .doesNotContain(KEYFILE); + } + + @Test + void applicationDefaultSetupUsesApplicationCredentials() { + final org.apache.hadoop.conf.Configuration hadoopConfig = + loadCredentials( + Collections.singletonMap("gs.auth.type", "APPLICATION_DEFAULT"), + Collections.emptyMap(), + APPLICATION_CREDENTIALS, + APPLICATION_CREDENTIALS); + + assertThat(connectorAuthType(hadoopConfig)) + .isEqualTo(AuthenticationType.APPLICATION_DEFAULT); + assertThat(warnings.getMessages()).isEmpty(); + } + + @Test + void keyfileSetupUsesKeyfile() { + final org.apache.hadoop.conf.Configuration hadoopConfig = + loadCredentials( + Collections.singletonMap("gs.auth.type", "SERVICE_ACCOUNT_JSON_KEYFILE"), + Collections.singletonMap( + "google.cloud.auth.service.account.json.keyfile", KEYFILE), + null, + KEYFILE); + + assertThat(connectorAuthType(hadoopConfig)) + .isEqualTo(AuthenticationType.SERVICE_ACCOUNT_JSON_KEYFILE); + assertThat( + HadoopCredentialsConfiguration.SERVICE_ACCOUNT_JSON_KEYFILE_SUFFIX + .withPrefixes( + GoogleHadoopFileSystemConfiguration.CONFIG_KEY_PREFIXES) + .get(hadoopConfig, hadoopConfig::get)) + .isEqualTo(KEYFILE); + assertThat(warnings.getMessages()).isEmpty(); + } + + @Test + void doesNotWarnForAuthTypeWithHadoopPrefix() { + final org.apache.hadoop.conf.Configuration hadoopConfig = + loadCredentials( + Collections.emptyMap(), + Collections.singletonMap("google.cloud.auth.type", "APPLICATION_DEFAULT"), + APPLICATION_CREDENTIALS, + APPLICATION_CREDENTIALS); + + assertThat(connectorAuthType(hadoopConfig)) + .isEqualTo(AuthenticationType.APPLICATION_DEFAULT); + assertThat(warnings.getMessages()).isEmpty(); + } + + @Test + void doesNotWarnWithoutCredentialsFile() { + loadCredentials(Collections.emptyMap(), Collections.emptyMap(), null, null); + + assertThat(warnings.getMessages()).isEmpty(); + } + + @Test + void doesNotWarnWhenServiceAccountIsDisabled() { + loadCredentials( + Collections.emptyMap(), + Collections.singletonMap("google.cloud.auth.service.account.enable", "false"), + APPLICATION_CREDENTIALS, + null); + + assertThat(warnings.getMessages()).isEmpty(); + } + + /** + * Builds the Hadoop config the way GSFileSystemFactory does, loads the RecoverableWriter + * credentials and checks which file they came from. + */ + private static org.apache.hadoop.conf.Configuration loadCredentials( + Map flinkOptions, + Map coreSite, + String applicationCredentials, + String expectedCredentialsFile) { + final Map envs = new HashMap<>(); + envs.put("HADOOP_CONF_DIR", CONF_DIR); + if (applicationCredentials != null) { + envs.put("GOOGLE_APPLICATION_CREDENTIALS", applicationCredentials); + } + final GoogleCredentials credentials = GoogleCredentials.newBuilder().build(); + final Map credentialsFiles = new HashMap<>(); + if (expectedCredentialsFile != null) { + credentialsFiles.put(expectedCredentialsFile, credentials); + } + final TestingConfigContext configContext = + new TestingConfigContext( + envs, + Collections.singletonMap(CONF_DIR, TestUtils.hadoopConfigFromMap(coreSite)), + credentialsFiles); + + final org.apache.hadoop.conf.Configuration hadoopConfig = + ConfigUtils.getHadoopConfiguration( + Configuration.fromMap(flinkOptions), configContext); + if (expectedCredentialsFile == null) { + assertThat(ConfigUtils.getStorageCredentials(hadoopConfig, configContext)).isEmpty(); + } else { + assertThat(ConfigUtils.getStorageCredentials(hadoopConfig, configContext)) + .containsSame(credentials); + } + return hadoopConfig; + } + + private static AuthenticationType connectorAuthType( + org.apache.hadoop.conf.Configuration hadoopConfig) { + return HadoopCredentialsConfiguration.AUTHENTICATION_TYPE_SUFFIX + .withPrefixes(GoogleHadoopFileSystemConfiguration.CONFIG_KEY_PREFIXES) + .get(hadoopConfig, hadoopConfig::getEnum); + } +} diff --git a/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterExceptionITCase.java b/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterExceptionITCase.java index c82f2e2d4dd5f2..d6f37500ee6966 100644 --- a/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterExceptionITCase.java +++ b/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterExceptionITCase.java @@ -31,6 +31,8 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; +import javax.annotation.Nullable; + import java.io.File; import java.io.IOException; import java.nio.charset.StandardCharsets; @@ -75,6 +77,8 @@ static void cleanUp() throws Exception { FileSystem.initialize(new Configuration()); } + /** Returns the local directory the writer stages data in, or null if it uses none. */ + @Nullable protected abstract String getLocalTmpDir() throws Exception; @BeforeEach @@ -82,6 +86,10 @@ void prepare() throws Exception { basePathForTest = new Path(basePath, StringUtils.getRandomString(RND, 16, 16, 'a', 'z')); final String defaultTmpDir = getLocalTmpDir(); + if (defaultTmpDir == null) { + // The suite does not use local storage. No need to create directory. + return; + } final java.nio.file.Path path = Paths.get(defaultTmpDir); if (!Files.exists(path)) { diff --git a/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterITCase.java b/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterITCase.java index 647670f3f28be8..594added161357 100644 --- a/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterITCase.java +++ b/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/AbstractHadoopRecoverableWriterITCase.java @@ -34,6 +34,8 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; +import javax.annotation.Nullable; + import java.io.BufferedReader; import java.io.File; import java.io.FileNotFoundException; @@ -93,6 +95,8 @@ void prepare() throws Exception { cleanupLocalDir(); } + /** Returns the local directory the writer stages data in, or null if it uses none. */ + @Nullable protected abstract String getLocalTmpDir() throws Exception; protected abstract String getIncompleteObjectName( @@ -100,6 +104,10 @@ protected abstract String getIncompleteObjectName( private void cleanupLocalDir() throws Exception { final String defaultTmpDir = getLocalTmpDir(); + if (defaultTmpDir == null) { + // The suite did not use local storage. No cleanup needed. + return; + } final java.nio.file.Path defaultTmpPath = Paths.get(defaultTmpDir); if (Files.exists(defaultTmpPath)) { @@ -120,7 +128,20 @@ private void cleanupLocalDir() throws Exception { @AfterEach void cleanupAndCheckTmpCleanup() throws Exception { + try { + checkAndDeleteLocalTmpDir(); + } finally { + // delete also object store dir, even if the local tmp dir check failed. + getFileSystem().delete(basePathForTest, true); + } + } + + private void checkAndDeleteLocalTmpDir() throws Exception { final String defaultTmpDir = getLocalTmpDir(); + if (defaultTmpDir == null) { + // The suite did not use local storage. No cleanup needed. + return; + } final java.nio.file.Path localTmpDir = Paths.get(defaultTmpDir); // delete local tmp dir. @@ -129,9 +150,6 @@ void cleanupAndCheckTmpCleanup() throws Exception { assertThat(files).isEmpty(); } Files.delete(localTmpDir); - - // delete also object store dir. - getFileSystem().delete(basePathForTest, true); } protected static FileSystem getFileSystem() throws Exception { @@ -181,7 +199,7 @@ void testCommitAfterPersist() throws Exception { } @Test - void testCleanupRecoverableState() throws Exception { + protected void testCleanupRecoverableState() throws Exception { final RecoverableWriter writer = getRecoverableWriter(); final Path path = new Path(basePathForTest, "part-0"); @@ -220,7 +238,7 @@ void testCleanupRecoverableState() throws Exception { } @Test - void testCallingDeleteObjectTwiceDoesNotThroughException() throws Exception { + protected void testCallingDeleteObjectTwiceDoesNotThroughException() throws Exception { final RecoverableWriter writer = getRecoverableWriter(); final Path path = new Path(basePathForTest, "part-0");