Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 17 additions & 20 deletions docs/content.zh/docs/deployment/filesystems/gcs.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place
Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it.

`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.68.0) library to provide `RecoverableWriter` support.
`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/3.1.22) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.79.0) library to provide `RecoverableWriter` support.

This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}).

Expand All @@ -84,7 +84,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/

### Configuration

The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}).
The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}).

For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}). Flink will internally translate this back to `fs.gs.http.connect-timeout`.

Expand All @@ -107,41 +107,38 @@ You can also set `gcs-connector` options directly in the Hadoop `core-site.xml`

### Authentication to access GCS

Most operations on GCS require authentication. To provide authentication credentials, either:
Most operations on GCS require authentication. `flink-gs-fs-hadoop` uses two clients, which select their credentials separately:

* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the JSON credentials file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run. This is the **recommended** method.
* `gcs-connector` handles checkpoints, savepoints and all other file system access. It uses the credentials that `gs.auth.type` selects, and the credentials of the Compute Engine or GKE service account (`COMPUTE_ENGINE`) if `gs.auth.type` isn't set. See the [gcs-connector configuration](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md#authentication) for all values.
* The `google-cloud-storage` library handles `RecoverableWriter`, which the `FileSink` uses. It uses the JSON credentials file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, then the one in the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml`, and otherwise the [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials).

To use a JSON credentials file with both clients, either:

* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path to the JSON credentials file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)):
* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run, and set `gs.auth.type` in the [Flink configuration file]({{< ref "docs/deployment/config#flink-配置文件" >}}). This is the **recommended** method.

```
<configuration>
<property>
<name>google.cloud.auth.service.account.json.keyfile</name>
<value>PATH TO GOOGLE AUTHENTICATION JSON FILE</value>
</property>
</configuration>
```yaml
gs.auth.type: APPLICATION_DEFAULT
```

For `flink-gs-fs-hadoop` to use credentials via either of these two methods, the use of service accounts for authentication must be enabled. This is enabled by default; however, it can be disabled in `core-site.xml` by setting:
* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path of a service account key file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)), and set `gs.auth.type: SERVICE_ACCOUNT_JSON_KEYFILE` in the Flink configuration file:

```
<configuration>
<property>
<name>google.cloud.auth.service.account.enable</name>
<value>false</value>
<name>google.cloud.auth.service.account.json.keyfile</name>
<value>PATH TO GOOGLE AUTHENTICATION JSON FILE</value>
</property>
</configuration>
```

{{< hint warning >}}
`gcs-connector` supports additional options to provide authentication credentials besides the `google.cloud.auth.service.account.json.keyfile` option described above.

However, if you use any of those other options, the provided credentials will not be used by the `google-cloud-storage` library, which provides `RecoverableWriter` support, so Flink recoverable-write operations would be expected to fail.
Flink logs a warning at startup if `RecoverableWriter` uses one of these files and `gs.auth.type` isn't set.

For this reason, use of the `gcs-connector` authentication-credentials options other than `google.cloud.auth.service.account.json.keyfile` is **not recommended.**
Setting `google.cloud.auth.service.account.enable` to `false` in `core-site.xml` makes `RecoverableWriter` ignore `google.cloud.auth.service.account.json.keyfile` and use the application default credentials. `gcs-connector` doesn't read this property.

{{< hint warning >}}
The other values of `gs.auth.type`, such as `USER_CREDENTIALS` or `ACCESS_TOKEN_PROVIDER`, only apply to `gcs-connector`. `RecoverableWriter` doesn't use those credentials, so it can run as a different identity than `gcs-connector` and its operations may fail.

If you use the `FileSink`, set `gs.auth.type` as described above, or keep the default `COMPUTE_ENGINE` without configuring either file.
{{< /hint >}}


Expand Down
39 changes: 18 additions & 21 deletions docs/content/docs/deployment/filesystems/gcs.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place
Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it.

`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.68.0) library to provide `RecoverableWriter` support.
`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/3.1.22) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.79.0) library to provide `RecoverableWriter` support.

This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}).

Expand All @@ -84,7 +84,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/

### Configuration

The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}).
The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}).

For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). Flink will internally translate this back to `fs.gs.http.connect-timeout`.

Expand All @@ -107,41 +107,38 @@ You can also set `gcs-connector` options directly in the Hadoop `core-site.xml`

### Authentication to access GCS

Most operations on GCS require authentication. To provide authentication credentials, either:
Most operations on GCS require authentication. `flink-gs-fs-hadoop` uses two clients, which select their credentials separately:

* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the JSON credentials file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run. This is the **recommended** method.
* `gcs-connector` handles checkpoints, savepoints and all other file system access. It uses the credentials that `gs.auth.type` selects, and the credentials of the Compute Engine or GKE service account (`COMPUTE_ENGINE`) if `gs.auth.type` isn't set. See the [gcs-connector configuration](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.22/gcs/CONFIGURATION.md#authentication) for all values.
* The `google-cloud-storage` library handles `RecoverableWriter`, which the `FileSink` uses. It uses the JSON credentials file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, then the one in the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml`, and otherwise the [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials).

To use a JSON credentials file with both clients, either:

* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path to the JSON credentials file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)):

```
<configuration>
<property>
<name>google.cloud.auth.service.account.json.keyfile</name>
<value>PATH TO GOOGLE AUTHENTICATION JSON FILE</value>
</property>
</configuration>
* Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of the file, as described [here](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable), where JobManagers and TaskManagers run, and set `gs.auth.type` in the [Flink configuration file]({{< ref "docs/deployment/config#flink-configuration-file" >}}). This is the **recommended** method.

```yaml
gs.auth.type: APPLICATION_DEFAULT
```

For `flink-gs-fs-hadoop` to use credentials via either of these two methods, the use of service accounts for authentication must be enabled. This is enabled by default; however, it can be disabled in `core-site.xml` by setting:
* Set the `google.cloud.auth.service.account.json.keyfile` property in `core-site.xml` to the path of a service account key file (and make sure that the Hadoop configuration directory is specified to Flink as described [above](#configuration)), and set `gs.auth.type: SERVICE_ACCOUNT_JSON_KEYFILE` in the Flink configuration file:

```
<configuration>
<property>
<name>google.cloud.auth.service.account.enable</name>
<value>false</value>
<name>google.cloud.auth.service.account.json.keyfile</name>
<value>PATH TO GOOGLE AUTHENTICATION JSON FILE</value>
</property>
</configuration>
```

{{< hint warning >}}
`gcs-connector` supports additional options to provide authentication credentials besides the `google.cloud.auth.service.account.json.keyfile` option described above.

However, if you use any of those other options, the provided credentials will not be used by the `google-cloud-storage` library, which provides `RecoverableWriter` support, so Flink recoverable-write operations would be expected to fail.
Flink logs a warning at startup if `RecoverableWriter` uses one of these files and `gs.auth.type` isn't set.

For this reason, use of the `gcs-connector` authentication-credentials options other than `google.cloud.auth.service.account.json.keyfile` is **not recommended.**
Setting `google.cloud.auth.service.account.enable` to `false` in `core-site.xml` makes `RecoverableWriter` ignore `google.cloud.auth.service.account.json.keyfile` and use the application default credentials. `gcs-connector` doesn't read this property.

{{< hint warning >}}
The other values of `gs.auth.type`, such as `USER_CREDENTIALS` or `ACCESS_TOKEN_PROVIDER`, only apply to `gcs-connector`. `RecoverableWriter` doesn't use those credentials, so it can run as a different identity than `gcs-connector` and its operations may fail.

If you use the `FileSink`, set `gs.auth.type` as described above, or keep the default `COMPUTE_ENGINE` without configuring either file.
{{< /hint >}}


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,21 @@ org.apache.flink.fs.gs.GSRecoverableWriterITCase does not satisfy: only one of t
* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\
* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\
or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule
org.apache.flink.fs.gs.HadoopGSFileSystemITCase does not satisfy: only one of the following predicates match:\
* reside in a package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type InternalMiniClusterExtension and annotated with @RegisterExtension\
* reside outside of package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type MiniClusterExtension and annotated with @RegisterExtension or are , and of type MiniClusterTestEnvironment and annotated with @TestEnv\
* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\
* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\
or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule
org.apache.flink.fs.gs.HadoopGSRecoverableWriterExceptionITCase does not satisfy: only one of the following predicates match:\
* reside in a package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type InternalMiniClusterExtension and annotated with @RegisterExtension\
* reside outside of package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type MiniClusterExtension and annotated with @RegisterExtension or are , and of type MiniClusterTestEnvironment and annotated with @TestEnv\
* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\
* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\
or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule
org.apache.flink.fs.gs.HadoopGSRecoverableWriterITCase does not satisfy: only one of the following predicates match:\
* reside in a package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type InternalMiniClusterExtension and annotated with @RegisterExtension\
* reside outside of package 'org.apache.flink.runtime.*' and contain any fields that are static, final, and of type MiniClusterExtension and annotated with @RegisterExtension or are , and of type MiniClusterTestEnvironment and annotated with @TestEnv\
* reside in a package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class InternalMiniClusterExtension\
* reside outside of package 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class MiniClusterExtension\
or contain any fields that are public, static, and of type MiniClusterWithClientResource and final and annotated with @ClassRule or contain any fields that is of type MiniClusterWithClientResource and public and final and not static and annotated with @Rule
33 changes: 29 additions & 4 deletions flink-filesystems/flink-gs-fs-hadoop/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@ under the License.
<properties>
<!-- If updating these dependency versions, please also update the corresponding links -->
<!-- in the GCS file system documentation. -->
<fs.gs.sdk.version>2.68.0</fs.gs.sdk.version>
<fs.gs.connector.version>hadoop3-2.2.18</fs.gs.connector.version>
<fs.gs.cloud.nio.version>0.128.7</fs.gs.cloud.nio.version>
<fs.gs.sdk.version>2.79.0</fs.gs.sdk.version>
<fs.gs.connector.version>3.1.22</fs.gs.connector.version>
<fs.gs.cloud.nio.version>0.143.0</fs.gs.cloud.nio.version>
<!-- Set this to the highest version of grpc artifacts from gcs-connector and google-cloud-storage -->
<fs.gs.grpc.version>1.81.0</fs.gs.grpc.version>
<fs.gs.grpc.version>1.85.1</fs.gs.grpc.version>
</properties>

<dependencies>
Expand Down Expand Up @@ -67,6 +67,14 @@ under the License.
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-hadoop-fs</artifactId>
<version>${project.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-hadoop-fs</artifactId>
Expand Down Expand Up @@ -156,6 +164,23 @@ under the License.
<groupId>org.codehaus.mojo</groupId>
<artifactId>animal-sniffer-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>org.checkerframework</groupId>
<artifactId>checker-qual</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.j2objc</groupId>
<artifactId>j2objc-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>failureaccess</artifactId>
</exclusion>
<!-- gcsio declares grpc-testing (and through it junit and hamcrest) in compile scope -->
<exclusion>
<groupId>io.grpc</groupId>
<artifactId>grpc-testing</artifactId>
</exclusion>
<!-- exclude dependency because of its GPLv2 license, see https://github.com/apache/flink/pull/15599#issuecomment-850241316 -->
<exclusion>
<groupId>javax.annotation</groupId>
Expand Down
Loading