Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions docs/getting-started/create-account.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ sidebar_position: 1
Your Datacoves account is configured during a setup call with the Datacoves team. We will guide you through the account setup wizard step by step. The wizard tests your data warehouse and git connections live, and will not let you continue until they succeed, so having the access and permissions below in place **before** the call is what keeps it short.

:::caution
A common blocker is git repository permissions. You need to be able to add a deploy key to the repository (see [Git](#git)). If you can't, make sure the person who can is available during the call.
A common blocker is git repository permissions. Datacoves reaches the repository through a deploy key added to it or a GitHub App installed on it (see [Git](#git)). If you can't do either, make sure the person who can is available during the call.
:::

## Before the Call
Expand Down Expand Up @@ -40,7 +40,7 @@ To set up your Datacoves account, you will need to know your data warehouse prov

| Data Warehouse Provider | Information Needed |
| --- | --- |
| Snowflake | Account, Warehouse, Database, Role, Schema, User, Password |
| Snowflake | Account, Warehouse, Database, Role, Schema, and [an authentication method](/docs/how-tos/snowflake/snowflake-sign-in): User and Password, User and a key pair, or a Snowflake sign-in |
| Redshift | Host, Database, Schema, User, Password |
| Databricks | Host, HTTP Path, Schema, Catalog (optional), Token |
| BigQuery | Dataset, Keyfile JSON |
Expand All @@ -52,15 +52,21 @@ For the Snowflake `Account` field, use the `orgname-accountname` format (e.g. `m
Check out the Snowflake documentation on [finding the organization and account name for an account](https://docs.snowflake.com/en/user-guide/admin-account-identifier#finding-the-organization-and-account-name-for-an-account).
:::

**Snowflake authentication:** The setup wizard only supports username and password authentication. Make sure your Snowflake user can log in with a password. The Airflow service account can also use key-pair authentication.
**Snowflake authentication:** The setup wizard supports password, key pair and Sign in with Snowflake. For key pair, add a key in the wizard and register its public key on your Snowflake user before testing the connection. The Airflow service account can also use key-pair authentication.
- Your personal Snowflake user needs MFA (e.g. Duo) enabled. This does not apply to the Airflow service account. Have your phone handy, since you'll need to approve the push notification when the connection is tested.
- We recommend that your Snowflake administrator enables [MFA token caching](https://docs.snowflake.com/en/user-guide/security-mfa#using-mfa-token-caching-to-minimize-the-number-of-prompts-during-authentication-optional) to avoid an MFA prompt on every connection.

**Network Access:** Verify that your Data Warehouse is accessible from outside your network. You'll need to whitelist the Datacoves IP: `74.179.200.137`.

### Git

Datacoves works with any git provider that supports SSH, such as GitHub, GitLab, Bitbucket, and Azure DevOps. During the wizard, Datacoves generates two SSH keys, and both are tested before you can continue:
Datacoves works with any git provider that supports SSH, such as GitHub, GitLab, Bitbucket, and Azure DevOps. For a repository on github.com, the wizard's Git Repository step authenticates with a GitHub App, and the Authentication select starts on GitHub App.

**GitHub App**
Register a private GitHub App in your organization with read-only Contents access, install it on the repository, and have its client ID and private key ready. See [How to configure a GitHub App](/docs/how-tos/git/github-app) for the steps. The wizard also generates a Developer Key, and you add it to your own git account as described below.

**SSH keys**
Use SSH keys for any other git provider, such as GitLab, Bitbucket, or Azure DevOps, and for a github.com organization that cannot use an App. With SSH keys, the wizard generates a Deploy Key and a Developer Key, tests both before you can continue, and the SSH key type is selectable for both.

**Deploy Key**
Added to the repository and used by Datacoves services. You need the right role in your git provider to add it:
Expand Down
3 changes: 3 additions & 0 deletions docs/how-tos/datacoves/how_to_connection_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ To create a new connection template click the `Create Connection Template` in th
- `Database` - The default connection template database
- `Role`- The default connection template role
![Snowflake Connection Type](./assets/connections_editnew_snowflake.png)
- `Client id` and `Client secret` (under **Sign in with Snowflake**) - Optional. Leave them empty to use Snowflake's
built-in sign-in, or enter the client of your own security integration. See
[Sign in with Snowflake](/docs/how-tos/snowflake/snowflake-sign-in).

## For Redshift, the available fields are:

Expand Down
21 changes: 18 additions & 3 deletions docs/how-tos/datacoves/how_to_projects/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,27 @@ A Project configuration consists of the following fields:

- **Name:** This is what will be displayed in the Datacoves landing page.
- **Git Repo:** This is the git repository associated with this project.
- **Clone strategy:** Determines how Datacoves will communicate with your git repository (SSH, HTTPS, or Azure DevOps Secret/Certificate). Select your desired cloning strategy to see configuration instructions:
- **Clone strategy:** Determines how Datacoves will communicate with your git repository (GitHub App, SSH, HTTPS, or Azure DevOps Secret/Certificate). New projects start on GitHub App. Select your desired cloning strategy to see configuration instructions:

<Tabs>
<TabItem value="ssh" label="SSH" default>
<TabItem value="github-app" label="GitHub App" default>

When SSH is selected, an SSH public key will be automatically generated for you to configure in your git provider as a deployment key.
When GitHub App is selected, Datacoves authenticates with a private GitHub App installed on the repository. The
following fields must be filled in:

- **Git SSH url:** SSH clone URL of the repository, which code-server uses
- **Git HTTPS url:** HTTPS clone URL of the repository
- **Client ID:** Client ID of the App
- **Private key:** Private key generated on the App, empty on later edits to keep the stored one

See this [how-to guide on configuring a GitHub App](/docs/how-tos/git/github-app) for the App registration,
the installation and the private key.

</TabItem>

<TabItem value="ssh" label="SSH">

When SSH is selected, an SSH public key will be automatically generated for you to configure in your git provider as a deployment key. You can choose the SSH key type for the generated key.

![Repo SSH Key](../assets/projects_ssh_key.png)

Expand Down
2 changes: 2 additions & 0 deletions docs/how-tos/datacoves/how_to_service_connections.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ A Service Connection consists of the following fields:
![Delivery Mode](assets/service_connection_delivery.jpg).
- **Connection Template** The connection template to base this service connection on(i.e. the defaults)
Depending on the template selected, additional fields will be displayed with the default values entered in the connection template. These default values can be overridden by toggling the indicator next to the given value. Enter the appropriate user, schema, and password. This is commonly a service account created specifically for Airflow and may differ between the development and production environment.
For Snowflake with the Airflow Connection delivery mode, the authentication mechanism can also be a key pair,
Sign in with Snowflake, or Workload identity. See [Sign in with Snowflake](/docs/how-tos/snowflake/snowflake-sign-in).

![Service Connection Connection Details](./assets/serviceconnection_editnew_details.png)

Expand Down
80 changes: 80 additions & 0 deletions docs/how-tos/git/github-app.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
---
title: "Set Up a GitHub App for Repository Access in Datacoves"
sidebar_label: "GitHub App"
description: "Register a private GitHub App, install it on your repository, and give Datacoves its client ID and private key instead of a deploy key."
sidebar_position: 72
---
# How to set up a GitHub App for repository access

For a repository on github.com, Datacoves authenticates with a GitHub App. The setup wizard and new projects
start on it, and it also works in organizations that disable deploy keys. The App gives Datacoves services
read-only access to the repository contents. To run the setup wizard, you still add a developer SSH key to
your own GitHub account.

## What you need

- Admin rights on the GitHub organization, to register the App.
- Admin rights on the repository, to install the App on it.

## 1. Register the App

1. Open the organization settings, Developer settings, GitHub Apps, and select **New GitHub App**.
2. Set a name such as `datacoves-<company>` and a homepage URL. The webhook can be switched off.
3. Under **Repository permissions**, set **Contents** to **Read-only** and leave **Metadata** at its default
**Read-only**. No other permission is needed.
4. Under **Where can this GitHub App be installed?**, select **Only on this account**.
5. Select **Create GitHub App**.

## 2. Generate a private key and copy the client ID

1. The App page shows its **Client ID**. Copy it.
2. Scroll to **Private keys** and select **Generate a private key**. GitHub downloads a `.pem` file. Keep it,
the private key is shown only once.

## 3. Install the App on the repository

1. On the App page select **Install App** and pick the organization.
2. Choose **Only select repositories**, select the repository Datacoves will use, and select **Install**.
3. If you add another repository later, come back to the installation and add it there.

Datacoves finds the installation ID by itself from the repository, so there is nothing else to copy.

## 4. Add the App to a Datacoves project

### In the setup wizard

1. Enter the repository SSH URL, for example `git@github.com:organization/repo.git`, and the release branch.
The URL must be on `github.com`.
2. Leave **Authentication** on **GitHub App**, where it starts.
3. Paste the **Client ID** and the **Private key**.
4. Select an **SSH key type**. The wizard shows your **SSH Developer Key**. Copy it.
5. In GitHub, open Settings, SSH and GPG keys, and select **New SSH key**. Paste the key and save it.
6. If your organization enforces SSO, authorize the key for the organization on the same page.
7. Select **Next**.

### In a new project

A new project starts with **Clone strategy** on **GitHub App**. Fill in the **Git SSH url**, the **Git HTTPS
url**, the **Client ID** and the **Private key**, then select **Save Changes**.

### In an existing project

1. Open the project, set **Clone strategy** to **GitHub App**.
2. The **Git SSH url** stays as it is, and it is locked once the project is validated. The **Git HTTPS url**
is locked too once it holds a value; fill it in when the project has none. Enter the **Client ID** and the
**Private key**. Leave the private key empty on later edits to keep the stored one.
3. Select **Save Changes**. Datacoves saves the installation ID it looks up from the repository and tests
the connection.

## What Datacoves stores

- The client ID, the private key and the installation ID, encrypted at rest with the project.
- The API never returns the private key.

## What a GitHub App does not cover

- **Personal development clones.** The App doesn't clone the repository for developers. Each developer adds
their own SSH key under User Settings, Git SSH Keys. Then code-server clones the repository for them. The
person who ran the setup wizard already added their key in that step.
- **Repositories outside the installation.** A repository the App is not installed on fails the connection
test with GitHub's answer. Add it to the installation.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
163 changes: 163 additions & 0 deletions docs/how-tos/snowflake/snowflake-sign-in.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
---
title: "Snowflake authentication: password with MFA, key pair and Sign in with Snowflake"
sidebar_label: "Snowflake Authentication"
description: "Connect Datacoves to Snowflake with a password and MFA, a key pair, or Sign in with Snowflake (OAuth), set up on the connection template and in each user's settings."
sidebar_position: 81
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# How to connect Datacoves to Snowflake

Datacoves connects to Snowflake in one of three ways:

- **Username and password with MFA.** Snowflake is
[making MFA mandatory](https://docs.snowflake.com/en/user-guide/security-mfa-rollout) for every password
sign-in.
- **Key pair.** The user signs in with a private key; Snowflake holds the public key.
- **Sign in with Snowflake (OAuth).** The user signs in the way they sign in to Snowflake, with their Snowflake
user or your company's single sign-on. It is the option for accounts where users have no password.

An admin sets up the connection template first. Each user then adds their own connection in **Settings >
Database Connections**.

## Connection template setup

Admins create the template in **Connection Templates** (see
[Configure Connection Templates](/docs/how-tos/datacoves/how_to_connection_template)). Turn on **Enabled for
users** so users can pick it.

<Tabs groupId="snowflake-auth">
<TabItem value="password" label="Password with MFA" default>

Turn on **Is MFA protected**. dbt then signs in with `authenticator: username_password_mfa`, and users approve
each connection test in their MFA app.

![Is MFA protected on the connection template](assets/snowflake-auth-template-mfa.png)

To cut down on MFA prompts, a Snowflake admin turns on MFA token caching for the account. A cached token is valid
for up to four hours ([Snowflake MFA token caching](https://docs.snowflake.com/en/user-guide/security-mfa#using-mfa-token-caching-to-minimize-the-number-of-prompts-during-authentication-optional)):

```sql
ALTER ACCOUNT SET ALLOW_CLIENT_MFA_CACHING = TRUE;
```

</TabItem>
<TabItem value="key-pair" label="Key pair">

Set **User field configuration** to any value except **Provided by user**. Users are offered the key pair only
when Datacoves fills in their Snowflake username from the template.

![User field configuration on the connection template](assets/snowflake-auth-template-key-pair.png)

</TabItem>
<TabItem value="sign-in" label="Sign in with Snowflake">

Under **Sign in with Snowflake**, leave **Client id** and **Client secret** empty to use Snowflake's built-in
sign-in, or enter the client of your own security integration. Every connection on the template then signs in
through it.

![Sign in with Snowflake on the connection template](assets/snowflake-auth-template-sign-in.png)

Snowflake's built-in sign-in needs no Snowflake admin setup. Datacoves signs in through Snowflake's
`SNOWFLAKE$LOCAL_APPLICATION` integration. Check that your account has it:

```sql
SHOW SECURITY INTEGRATIONS LIKE 'SNOWFLAKE$LOCAL_APPLICATION';
```

Use your own security integration when your admins want their own OAuth integration, with its own refresh token
validity and role rules. A Snowflake admin with `ACCOUNTADMIN`, or a role with the `CREATE INTEGRATION`
privilege, runs this once. The template form shows the same statements, with your cluster's redirect URI filled
in and a copy button.

```sql
CREATE SECURITY INTEGRATION DATACOVES_OAUTH
TYPE = OAUTH
ENABLED = TRUE
OAUTH_CLIENT = CUSTOM
OAUTH_CLIENT_TYPE = 'CONFIDENTIAL'
OAUTH_REDIRECT_URI = 'https://api.<your Datacoves domain>/api/setup/snowflake-oauth/callback'
OAUTH_ISSUE_REFRESH_TOKENS = TRUE
OAUTH_REFRESH_TOKEN_VALIDITY = 7776000
OAUTH_ENFORCE_PKCE = TRUE
OAUTH_ANY_ROLE_MODE = 'ENABLE'
OAUTH_USE_SECONDARY_ROLES = IMPLICIT;

SELECT SYSTEM$SHOW_OAUTH_CLIENT_SECRETS('DATACOVES_OAUTH');
```

- `OAUTH_CLIENT = CUSTOM` and `OAUTH_CLIENT_TYPE = 'CONFIDENTIAL'` make an integration that Datacoves
authenticates to with a client id and secret.
- `OAUTH_REDIRECT_URI` is where Snowflake sends the browser after sign-in. It is `https://api.` plus the
domain of your Datacoves cluster, the domain in your Datacoves address, and the path above.
- `OAUTH_ISSUE_REFRESH_TOKENS` and `OAUTH_REFRESH_TOKEN_VALIDITY` let dbt keep working without signing in
again. The validity is in seconds: 86400 (1 day) to 7776000 (90 days). Snowflake Support raises the maximum
on request.
- `OAUTH_ENFORCE_PKCE` requires PKCE on every sign-in.
- `OAUTH_ANY_ROLE_MODE` and `OAUTH_USE_SECONDARY_ROLES` let dbt switch roles inside a session, as it does with
a password or a key pair.
- `SYSTEM$SHOW_OAUTH_CLIENT_SECRETS` returns the client id and secret. Enter them on the connection template
under **Sign in with Snowflake**.

Snowflake blocks `ACCOUNTADMIN`, `SECURITYADMIN`, `GLOBALORGADMIN` and `ORGADMIN` in OAuth sessions by
default. Use another role for dbt.

Single-use refresh tokens (`OAUTH_SINGLE_USE_REFRESH_TOKENS_REQUIRED = TRUE`) are not supported. Datacoves
shows the `ALTER SECURITY INTEGRATION` statement that turns them off when it finds them.

Token requests come from Datacoves' addresses, so Snowflake network policies have to allow them. The setup SQL
panel lists the addresses when your cluster has them configured.

</TabItem>
</Tabs>

## User settings

Each user opens **Settings > Database Connections**, clicks **Add** on the environment, picks the connection
template and chooses the **Auth type**.

<Tabs groupId="snowflake-auth">
<TabItem value="password" label="Password with MFA" default>

Enter your Snowflake password and save. When the template is MFA protected, approve the connection test in your
MFA app.

![Password auth type in the Database Connections settings](assets/snowflake-auth-settings-password.png)

</TabItem>
<TabItem value="key-pair" label="Key pair">

First add a key in **Settings > Database Auth Keys**: **Add**, then **Auto-generate RSA key pairs
(Compatibility)** or **Provide private key**.

![Adding a key in Database Auth Keys](assets/snowflake-auth-settings-keys.png)

Then choose **Key pair** as the auth type and pick the key under **Public key**.

![Key pair auth type in the Database Connections settings](assets/snowflake-auth-settings-key-pair.png)

Your Snowflake user needs the public key, without the `BEGIN` and `END` lines:

```sql
ALTER USER <your Snowflake user> SET RSA_PUBLIC_KEY = '<your public key>';
```

</TabItem>
<TabItem value="sign-in" label="Sign in with Snowflake">

Choose **Sign in with Snowflake** and click the sign-in button.

![Sign in with Snowflake in the Database Connections settings](assets/snowflake-sign-in-settings.png)

The sign-in opens in a dialog in Datacoves. Sign in the way you sign in to Snowflake. Passkeys aren't available
in the dialog; pick another method, such as an authenticator app.

![The Snowflake sign-in page in the Datacoves dialog](assets/snowflake-sign-in-dialog.png)

A sign-in lasts as long as the integration's `OAUTH_REFRESH_TOKEN_VALIDITY`. When one expires, is about to
expire, or Snowflake revokes it, the Launchpad shows a banner with **Sign in again**, and the connection list
shows a badge. Signing in again restarts your workspace with the new sign-in.

</TabItem>
</Tabs>
2 changes: 1 addition & 1 deletion docs/how-tos/vs-code/initial.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ If using key based authentication, you will need to provide or generate a key wh
Provide or automatically generate your keys. Then add the public key to Snowflake.

:::warning
Elliptic curve keys don't work with the [Snowflake VS Code extension](/docs/getting-started/developer/snowflake-extension) or the [Snowflake MCP server](/docs/how-tos/vs-code/mcp/snowflake). The default **Auto-generate key pairs** creates an elliptic curve key. To use either, pick **Auto-generate RSA key pairs (Compatibility)** or provide an RSA private key. Datacoves shows this warning when you generate, paste, or select an elliptic curve key.
Elliptic curve keys don't work with the [Snowflake VS Code extension](/docs/getting-started/developer/snowflake-extension) or the [Snowflake MCP server](/docs/how-tos/vs-code/mcp/snowflake). Use **Auto-generate RSA key pairs (Compatibility)** or an RSA private key for those. Datacoves shows this warning when you paste or select an elliptic curve key.
:::

<div style={{ maxWidth: "700px", margin: "0 auto" }}>
Expand Down
Loading
Loading