diff --git a/docs/getting-started/create-account.md b/docs/getting-started/create-account.md index 52b6ccef..0484090a 100644 --- a/docs/getting-started/create-account.md +++ b/docs/getting-started/create-account.md @@ -9,7 +9,7 @@ sidebar_position: 1 Your Datacoves account is configured during a setup call with the Datacoves team. We will guide you through the account setup wizard step by step. The wizard tests your data warehouse and git connections live, and will not let you continue until they succeed, so having the access and permissions below in place **before** the call is what keeps it short. :::caution -A common blocker is git repository permissions. You need to be able to add a deploy key to the repository (see [Git](#git)). If you can't, make sure the person who can is available during the call. +A common blocker is git repository permissions. Datacoves reaches the repository through a deploy key added to it or a GitHub App installed on it (see [Git](#git)). If you can't do either, make sure the person who can is available during the call. ::: ## Before the Call @@ -40,7 +40,7 @@ To set up your Datacoves account, you will need to know your data warehouse prov | Data Warehouse Provider | Information Needed | | --- | --- | -| Snowflake | Account, Warehouse, Database, Role, Schema, User, Password | +| Snowflake | Account, Warehouse, Database, Role, Schema, and [an authentication method](/docs/how-tos/snowflake/snowflake-sign-in): User and Password, User and a key pair, or a Snowflake sign-in | | Redshift | Host, Database, Schema, User, Password | | Databricks | Host, HTTP Path, Schema, Catalog (optional), Token | | BigQuery | Dataset, Keyfile JSON | @@ -52,7 +52,7 @@ For the Snowflake `Account` field, use the `orgname-accountname` format (e.g. `m Check out the Snowflake documentation on [finding the organization and account name for an account](https://docs.snowflake.com/en/user-guide/admin-account-identifier#finding-the-organization-and-account-name-for-an-account). ::: -**Snowflake authentication:** The setup wizard only supports username and password authentication. Make sure your Snowflake user can log in with a password. The Airflow service account can also use key-pair authentication. +**Snowflake authentication:** The setup wizard supports password, key pair and Sign in with Snowflake. For key pair, add a key in the wizard and register its public key on your Snowflake user before testing the connection. The Airflow service account can also use key-pair authentication. - Your personal Snowflake user needs MFA (e.g. Duo) enabled. This does not apply to the Airflow service account. Have your phone handy, since you'll need to approve the push notification when the connection is tested. - We recommend that your Snowflake administrator enables [MFA token caching](https://docs.snowflake.com/en/user-guide/security-mfa#using-mfa-token-caching-to-minimize-the-number-of-prompts-during-authentication-optional) to avoid an MFA prompt on every connection. @@ -60,7 +60,13 @@ Check out the Snowflake documentation on [finding the organization and account n ### Git -Datacoves works with any git provider that supports SSH, such as GitHub, GitLab, Bitbucket, and Azure DevOps. During the wizard, Datacoves generates two SSH keys, and both are tested before you can continue: +Datacoves works with any git provider that supports SSH, such as GitHub, GitLab, Bitbucket, and Azure DevOps. For a repository on github.com, the wizard's Git Repository step authenticates with a GitHub App, and the Authentication select starts on GitHub App. + +**GitHub App** +Register a private GitHub App in your organization with read-only Contents access, install it on the repository, and have its client ID and private key ready. See [How to configure a GitHub App](/docs/how-tos/git/github-app) for the steps. The wizard also generates a Developer Key, and you add it to your own git account as described below. + +**SSH keys** +Use SSH keys for any other git provider, such as GitLab, Bitbucket, or Azure DevOps, and for a github.com organization that cannot use an App. With SSH keys, the wizard generates a Deploy Key and a Developer Key, tests both before you can continue, and the SSH key type is selectable for both. **Deploy Key** Added to the repository and used by Datacoves services. You need the right role in your git provider to add it: diff --git a/docs/how-tos/datacoves/how_to_connection_template.md b/docs/how-tos/datacoves/how_to_connection_template.md index ffc63865..77880e21 100644 --- a/docs/how-tos/datacoves/how_to_connection_template.md +++ b/docs/how-tos/datacoves/how_to_connection_template.md @@ -51,6 +51,9 @@ To create a new connection template click the `Create Connection Template` in th - `Database` - The default connection template database - `Role`- The default connection template role ![Snowflake Connection Type](./assets/connections_editnew_snowflake.png) +- `Client id` and `Client secret` (under **Sign in with Snowflake**) - Optional. Leave them empty to use Snowflake's + built-in sign-in, or enter the client of your own security integration. See + [Sign in with Snowflake](/docs/how-tos/snowflake/snowflake-sign-in). ## For Redshift, the available fields are: diff --git a/docs/how-tos/datacoves/how_to_projects/README.md b/docs/how-tos/datacoves/how_to_projects/README.md index 26ea67d2..d4e67c44 100644 --- a/docs/how-tos/datacoves/how_to_projects/README.md +++ b/docs/how-tos/datacoves/how_to_projects/README.md @@ -20,12 +20,27 @@ A Project configuration consists of the following fields: - **Name:** This is what will be displayed in the Datacoves landing page. - **Git Repo:** This is the git repository associated with this project. - - **Clone strategy:** Determines how Datacoves will communicate with your git repository (SSH, HTTPS, or Azure DevOps Secret/Certificate). Select your desired cloning strategy to see configuration instructions: + - **Clone strategy:** Determines how Datacoves will communicate with your git repository (GitHub App, SSH, HTTPS, or Azure DevOps Secret/Certificate). New projects start on GitHub App. Select your desired cloning strategy to see configuration instructions: - + -When SSH is selected, an SSH public key will be automatically generated for you to configure in your git provider as a deployment key. +When GitHub App is selected, Datacoves authenticates with a private GitHub App installed on the repository. The +following fields must be filled in: + +- **Git SSH url:** SSH clone URL of the repository, which code-server uses +- **Git HTTPS url:** HTTPS clone URL of the repository +- **Client ID:** Client ID of the App +- **Private key:** Private key generated on the App, empty on later edits to keep the stored one + +See this [how-to guide on configuring a GitHub App](/docs/how-tos/git/github-app) for the App registration, +the installation and the private key. + + + + + +When SSH is selected, an SSH public key will be automatically generated for you to configure in your git provider as a deployment key. You can choose the SSH key type for the generated key. ![Repo SSH Key](../assets/projects_ssh_key.png) diff --git a/docs/how-tos/datacoves/how_to_service_connections.md b/docs/how-tos/datacoves/how_to_service_connections.md index b0b5d1ab..43eb76df 100644 --- a/docs/how-tos/datacoves/how_to_service_connections.md +++ b/docs/how-tos/datacoves/how_to_service_connections.md @@ -27,6 +27,8 @@ A Service Connection consists of the following fields: ![Delivery Mode](assets/service_connection_delivery.jpg). - **Connection Template** The connection template to base this service connection on(i.e. the defaults) Depending on the template selected, additional fields will be displayed with the default values entered in the connection template. These default values can be overridden by toggling the indicator next to the given value. Enter the appropriate user, schema, and password. This is commonly a service account created specifically for Airflow and may differ between the development and production environment. + For Snowflake with the Airflow Connection delivery mode, the authentication mechanism can also be a key pair, + Sign in with Snowflake, or Workload identity. See [Sign in with Snowflake](/docs/how-tos/snowflake/snowflake-sign-in). ![Service Connection Connection Details](./assets/serviceconnection_editnew_details.png) diff --git a/docs/how-tos/git/github-app.md b/docs/how-tos/git/github-app.md new file mode 100644 index 00000000..d06671e5 --- /dev/null +++ b/docs/how-tos/git/github-app.md @@ -0,0 +1,80 @@ +--- +title: "Set Up a GitHub App for Repository Access in Datacoves" +sidebar_label: "GitHub App" +description: "Register a private GitHub App, install it on your repository, and give Datacoves its client ID and private key instead of a deploy key." +sidebar_position: 72 +--- +# How to set up a GitHub App for repository access + +For a repository on github.com, Datacoves authenticates with a GitHub App. The setup wizard and new projects +start on it, and it also works in organizations that disable deploy keys. The App gives Datacoves services +read-only access to the repository contents. To run the setup wizard, you still add a developer SSH key to +your own GitHub account. + +## What you need + +- Admin rights on the GitHub organization, to register the App. +- Admin rights on the repository, to install the App on it. + +## 1. Register the App + +1. Open the organization settings, Developer settings, GitHub Apps, and select **New GitHub App**. +2. Set a name such as `datacoves-` and a homepage URL. The webhook can be switched off. +3. Under **Repository permissions**, set **Contents** to **Read-only** and leave **Metadata** at its default + **Read-only**. No other permission is needed. +4. Under **Where can this GitHub App be installed?**, select **Only on this account**. +5. Select **Create GitHub App**. + +## 2. Generate a private key and copy the client ID + +1. The App page shows its **Client ID**. Copy it. +2. Scroll to **Private keys** and select **Generate a private key**. GitHub downloads a `.pem` file. Keep it, + the private key is shown only once. + +## 3. Install the App on the repository + +1. On the App page select **Install App** and pick the organization. +2. Choose **Only select repositories**, select the repository Datacoves will use, and select **Install**. +3. If you add another repository later, come back to the installation and add it there. + +Datacoves finds the installation ID by itself from the repository, so there is nothing else to copy. + +## 4. Add the App to a Datacoves project + +### In the setup wizard + +1. Enter the repository SSH URL, for example `git@github.com:organization/repo.git`, and the release branch. + The URL must be on `github.com`. +2. Leave **Authentication** on **GitHub App**, where it starts. +3. Paste the **Client ID** and the **Private key**. +4. Select an **SSH key type**. The wizard shows your **SSH Developer Key**. Copy it. +5. In GitHub, open Settings, SSH and GPG keys, and select **New SSH key**. Paste the key and save it. +6. If your organization enforces SSO, authorize the key for the organization on the same page. +7. Select **Next**. + +### In a new project + +A new project starts with **Clone strategy** on **GitHub App**. Fill in the **Git SSH url**, the **Git HTTPS +url**, the **Client ID** and the **Private key**, then select **Save Changes**. + +### In an existing project + +1. Open the project, set **Clone strategy** to **GitHub App**. +2. The **Git SSH url** stays as it is, and it is locked once the project is validated. The **Git HTTPS url** + is locked too once it holds a value; fill it in when the project has none. Enter the **Client ID** and the + **Private key**. Leave the private key empty on later edits to keep the stored one. +3. Select **Save Changes**. Datacoves saves the installation ID it looks up from the repository and tests + the connection. + +## What Datacoves stores + +- The client ID, the private key and the installation ID, encrypted at rest with the project. +- The API never returns the private key. + +## What a GitHub App does not cover + +- **Personal development clones.** The App doesn't clone the repository for developers. Each developer adds + their own SSH key under User Settings, Git SSH Keys. Then code-server clones the repository for them. The + person who ran the setup wizard already added their key in that step. +- **Repositories outside the installation.** A repository the App is not installed on fails the connection + test with GitHub's answer. Add it to the installation. diff --git a/docs/how-tos/snowflake/assets/snowflake-auth-settings-key-pair.png b/docs/how-tos/snowflake/assets/snowflake-auth-settings-key-pair.png new file mode 100644 index 00000000..0b5cd13b Binary files /dev/null and b/docs/how-tos/snowflake/assets/snowflake-auth-settings-key-pair.png differ diff --git a/docs/how-tos/snowflake/assets/snowflake-auth-settings-keys.png b/docs/how-tos/snowflake/assets/snowflake-auth-settings-keys.png new file mode 100644 index 00000000..6f4d3fef Binary files /dev/null and b/docs/how-tos/snowflake/assets/snowflake-auth-settings-keys.png differ diff --git a/docs/how-tos/snowflake/assets/snowflake-auth-settings-password.png b/docs/how-tos/snowflake/assets/snowflake-auth-settings-password.png new file mode 100644 index 00000000..c2329aa0 Binary files /dev/null and b/docs/how-tos/snowflake/assets/snowflake-auth-settings-password.png differ diff --git a/docs/how-tos/snowflake/assets/snowflake-auth-template-key-pair.png b/docs/how-tos/snowflake/assets/snowflake-auth-template-key-pair.png new file mode 100644 index 00000000..c7ae4810 Binary files /dev/null and b/docs/how-tos/snowflake/assets/snowflake-auth-template-key-pair.png differ diff --git a/docs/how-tos/snowflake/assets/snowflake-auth-template-mfa.png b/docs/how-tos/snowflake/assets/snowflake-auth-template-mfa.png new file mode 100644 index 00000000..c41792ae Binary files /dev/null and b/docs/how-tos/snowflake/assets/snowflake-auth-template-mfa.png differ diff --git a/docs/how-tos/snowflake/assets/snowflake-auth-template-sign-in.png b/docs/how-tos/snowflake/assets/snowflake-auth-template-sign-in.png new file mode 100644 index 00000000..236cb547 Binary files /dev/null and b/docs/how-tos/snowflake/assets/snowflake-auth-template-sign-in.png differ diff --git a/docs/how-tos/snowflake/assets/snowflake-sign-in-dialog.png b/docs/how-tos/snowflake/assets/snowflake-sign-in-dialog.png new file mode 100644 index 00000000..a124bd96 Binary files /dev/null and b/docs/how-tos/snowflake/assets/snowflake-sign-in-dialog.png differ diff --git a/docs/how-tos/snowflake/assets/snowflake-sign-in-settings.png b/docs/how-tos/snowflake/assets/snowflake-sign-in-settings.png new file mode 100644 index 00000000..9164a56f Binary files /dev/null and b/docs/how-tos/snowflake/assets/snowflake-sign-in-settings.png differ diff --git a/docs/how-tos/snowflake/snowflake-sign-in.md b/docs/how-tos/snowflake/snowflake-sign-in.md new file mode 100644 index 00000000..00ccf8e8 --- /dev/null +++ b/docs/how-tos/snowflake/snowflake-sign-in.md @@ -0,0 +1,163 @@ +--- +title: "Snowflake authentication: password with MFA, key pair and Sign in with Snowflake" +sidebar_label: "Snowflake Authentication" +description: "Connect Datacoves to Snowflake with a password and MFA, a key pair, or Sign in with Snowflake (OAuth), set up on the connection template and in each user's settings." +sidebar_position: 81 +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# How to connect Datacoves to Snowflake + +Datacoves connects to Snowflake in one of three ways: + +- **Username and password with MFA.** Snowflake is + [making MFA mandatory](https://docs.snowflake.com/en/user-guide/security-mfa-rollout) for every password + sign-in. +- **Key pair.** The user signs in with a private key; Snowflake holds the public key. +- **Sign in with Snowflake (OAuth).** The user signs in the way they sign in to Snowflake, with their Snowflake + user or your company's single sign-on. It is the option for accounts where users have no password. + +An admin sets up the connection template first. Each user then adds their own connection in **Settings > +Database Connections**. + +## Connection template setup + +Admins create the template in **Connection Templates** (see +[Configure Connection Templates](/docs/how-tos/datacoves/how_to_connection_template)). Turn on **Enabled for +users** so users can pick it. + + + + +Turn on **Is MFA protected**. dbt then signs in with `authenticator: username_password_mfa`, and users approve +each connection test in their MFA app. + +![Is MFA protected on the connection template](assets/snowflake-auth-template-mfa.png) + +To cut down on MFA prompts, a Snowflake admin turns on MFA token caching for the account. A cached token is valid +for up to four hours ([Snowflake MFA token caching](https://docs.snowflake.com/en/user-guide/security-mfa#using-mfa-token-caching-to-minimize-the-number-of-prompts-during-authentication-optional)): + +```sql +ALTER ACCOUNT SET ALLOW_CLIENT_MFA_CACHING = TRUE; +``` + + + + +Set **User field configuration** to any value except **Provided by user**. Users are offered the key pair only +when Datacoves fills in their Snowflake username from the template. + +![User field configuration on the connection template](assets/snowflake-auth-template-key-pair.png) + + + + +Under **Sign in with Snowflake**, leave **Client id** and **Client secret** empty to use Snowflake's built-in +sign-in, or enter the client of your own security integration. Every connection on the template then signs in +through it. + +![Sign in with Snowflake on the connection template](assets/snowflake-auth-template-sign-in.png) + +Snowflake's built-in sign-in needs no Snowflake admin setup. Datacoves signs in through Snowflake's +`SNOWFLAKE$LOCAL_APPLICATION` integration. Check that your account has it: + +```sql +SHOW SECURITY INTEGRATIONS LIKE 'SNOWFLAKE$LOCAL_APPLICATION'; +``` + +Use your own security integration when your admins want their own OAuth integration, with its own refresh token +validity and role rules. A Snowflake admin with `ACCOUNTADMIN`, or a role with the `CREATE INTEGRATION` +privilege, runs this once. The template form shows the same statements, with your cluster's redirect URI filled +in and a copy button. + +```sql +CREATE SECURITY INTEGRATION DATACOVES_OAUTH + TYPE = OAUTH + ENABLED = TRUE + OAUTH_CLIENT = CUSTOM + OAUTH_CLIENT_TYPE = 'CONFIDENTIAL' + OAUTH_REDIRECT_URI = 'https://api./api/setup/snowflake-oauth/callback' + OAUTH_ISSUE_REFRESH_TOKENS = TRUE + OAUTH_REFRESH_TOKEN_VALIDITY = 7776000 + OAUTH_ENFORCE_PKCE = TRUE + OAUTH_ANY_ROLE_MODE = 'ENABLE' + OAUTH_USE_SECONDARY_ROLES = IMPLICIT; + +SELECT SYSTEM$SHOW_OAUTH_CLIENT_SECRETS('DATACOVES_OAUTH'); +``` + +- `OAUTH_CLIENT = CUSTOM` and `OAUTH_CLIENT_TYPE = 'CONFIDENTIAL'` make an integration that Datacoves + authenticates to with a client id and secret. +- `OAUTH_REDIRECT_URI` is where Snowflake sends the browser after sign-in. It is `https://api.` plus the + domain of your Datacoves cluster, the domain in your Datacoves address, and the path above. +- `OAUTH_ISSUE_REFRESH_TOKENS` and `OAUTH_REFRESH_TOKEN_VALIDITY` let dbt keep working without signing in + again. The validity is in seconds: 86400 (1 day) to 7776000 (90 days). Snowflake Support raises the maximum + on request. +- `OAUTH_ENFORCE_PKCE` requires PKCE on every sign-in. +- `OAUTH_ANY_ROLE_MODE` and `OAUTH_USE_SECONDARY_ROLES` let dbt switch roles inside a session, as it does with + a password or a key pair. +- `SYSTEM$SHOW_OAUTH_CLIENT_SECRETS` returns the client id and secret. Enter them on the connection template + under **Sign in with Snowflake**. + +Snowflake blocks `ACCOUNTADMIN`, `SECURITYADMIN`, `GLOBALORGADMIN` and `ORGADMIN` in OAuth sessions by +default. Use another role for dbt. + +Single-use refresh tokens (`OAUTH_SINGLE_USE_REFRESH_TOKENS_REQUIRED = TRUE`) are not supported. Datacoves +shows the `ALTER SECURITY INTEGRATION` statement that turns them off when it finds them. + +Token requests come from Datacoves' addresses, so Snowflake network policies have to allow them. The setup SQL +panel lists the addresses when your cluster has them configured. + + + + +## User settings + +Each user opens **Settings > Database Connections**, clicks **Add** on the environment, picks the connection +template and chooses the **Auth type**. + + + + +Enter your Snowflake password and save. When the template is MFA protected, approve the connection test in your +MFA app. + +![Password auth type in the Database Connections settings](assets/snowflake-auth-settings-password.png) + + + + +First add a key in **Settings > Database Auth Keys**: **Add**, then **Auto-generate RSA key pairs +(Compatibility)** or **Provide private key**. + +![Adding a key in Database Auth Keys](assets/snowflake-auth-settings-keys.png) + +Then choose **Key pair** as the auth type and pick the key under **Public key**. + +![Key pair auth type in the Database Connections settings](assets/snowflake-auth-settings-key-pair.png) + +Your Snowflake user needs the public key, without the `BEGIN` and `END` lines: + +```sql +ALTER USER SET RSA_PUBLIC_KEY = ''; +``` + + + + +Choose **Sign in with Snowflake** and click the sign-in button. + +![Sign in with Snowflake in the Database Connections settings](assets/snowflake-sign-in-settings.png) + +The sign-in opens in a dialog in Datacoves. Sign in the way you sign in to Snowflake. Passkeys aren't available +in the dialog; pick another method, such as an authenticator app. + +![The Snowflake sign-in page in the Datacoves dialog](assets/snowflake-sign-in-dialog.png) + +A sign-in lasts as long as the integration's `OAUTH_REFRESH_TOKEN_VALIDITY`. When one expires, is about to +expire, or Snowflake revokes it, the Launchpad shows a banner with **Sign in again**, and the connection list +shows a badge. Signing in again restarts your workspace with the new sign-in. + + + diff --git a/docs/how-tos/vs-code/initial.mdx b/docs/how-tos/vs-code/initial.mdx index 78c01803..66af5066 100644 --- a/docs/how-tos/vs-code/initial.mdx +++ b/docs/how-tos/vs-code/initial.mdx @@ -201,7 +201,7 @@ If using key based authentication, you will need to provide or generate a key wh Provide or automatically generate your keys. Then add the public key to Snowflake. :::warning -Elliptic curve keys don't work with the [Snowflake VS Code extension](/docs/getting-started/developer/snowflake-extension) or the [Snowflake MCP server](/docs/how-tos/vs-code/mcp/snowflake). The default **Auto-generate key pairs** creates an elliptic curve key. To use either, pick **Auto-generate RSA key pairs (Compatibility)** or provide an RSA private key. Datacoves shows this warning when you generate, paste, or select an elliptic curve key. +Elliptic curve keys don't work with the [Snowflake VS Code extension](/docs/getting-started/developer/snowflake-extension) or the [Snowflake MCP server](/docs/how-tos/vs-code/mcp/snowflake). Use **Auto-generate RSA key pairs (Compatibility)** or an RSA private key for those. Datacoves shows this warning when you paste or select an elliptic curve key. :::
diff --git a/docs/reference/admin-menu/environments.md b/docs/reference/admin-menu/environments.md index 8a4222e2..bacd4595 100644 --- a/docs/reference/admin-menu/environments.md +++ b/docs/reference/admin-menu/environments.md @@ -31,7 +31,8 @@ Each row contains 2 action buttons, Edit and Delete. ## AI Tools When editing an environment that has VS Code enabled and runs release 6.0 or later, the **AI -Tools** tab controls the AI assistants and the data systems they can reach. It has three sections: +Tools** tab controls the AI assistants and the data systems they can reach. The setup wizard's +**AI Tools and Extensions** step sets these for the first environment. It has three sections: - **AI Extensions** - enable or disable the built-in AI extensions, such as Datacoves Copilot and GitHub Copilot.