diff --git a/docs/how-tos/airflow/api-triggered-dag.md b/docs/how-tos/airflow/api-triggered-dag.md index e4c89488..a414c4d9 100644 --- a/docs/how-tos/airflow/api-triggered-dag.md +++ b/docs/how-tos/airflow/api-triggered-dag.md @@ -1,17 +1,23 @@ --- title: How to Trigger an Airflow DAG Using Datasets sidebar_label: Trigger DAG via Datasets -description: "Step-by-step guide to triggering Airflow DAGs via dataset events in Datacoves, enabling data-driven pipeline orchestration without time-based schedules." +description: "Step-by-step guide to triggering Airflow DAGs via dataset (asset) events in Datacoves, from another DAG or from an external process with one API call, enabling data-driven pipeline orchestration without time-based schedules." sidebar_position: 6 --- # How to Trigger a DAG using Datasets ## Overview -This guide explains how to trigger Airflow DAGs with Datasets. DAGs can be triggered by another DAG using datasets or by an external process that sends a dataset event using the Airflow API. +This guide explains how to trigger Airflow DAGs with Datasets, called **Assets** in Airflow 3. A dataset is a pointer to a piece of data, identified by a URI such as `s3://my_bucket/my_folder/my_file.csv`. A DAG scheduled on a dataset runs every time an event for that dataset is recorded. The event can come from another DAG (a producer) or from an external process (a script, a Lambda function, a CI job) that tells Airflow the data is ready. + +The external process does not need to know which DAGs depend on the data: it only says "this data is ready", and every DAG scheduled on it runs. + +:::note +The examples import `Asset` from `airflow.sdk`, available in Airflow 3. On Airflow 2 use `from airflow.datasets import Dataset` instead; the rest of the code is the same. +::: ## Producer DAG -Airflow enables DAGs to be triggered dynamically based on dataset updates. A producer DAG updates a dataset, automatically triggering any consumer DAGs subscribed to it. +A producer DAG updates a dataset, automatically triggering any consumer DAGs subscribed to it. To implement this, start by creating a DAG and defining the dataset it will update. @@ -20,13 +26,12 @@ To implement this, start by creating a DAG and defining the dataset it will upda import datetime from airflow.decorators import dag, task -from airflow.datasets import Dataset - +from airflow.sdk import Asset -# A dataset can be anything, it will be a poiner in the Airflow db. +# A dataset can be anything, it will be a pointer in the Airflow db. # If you need to access url like s3://my_bucket/my_file.txt then you can set # it with the proper path for reuse. -DAG_UPDATED_DATASET = Dataset("upstream_data") +DAG_UPDATED_DATASET = Asset("upstream_data") @dag( default_args={ @@ -51,11 +56,65 @@ def data_aware_producer_dag(): dag = data_aware_producer_dag() ``` -Thats it, now you are ready to create your [Consumer DAG](#setting-up-the-airflow-dag) +That's it, now you are ready to create your [Consumer DAG](#setting-up-the-airflow-dag). + +## Trigger from outside Airflow with Datacoves + +On Airflow 3 environments, an external process publishes a dataset event with a single request to Datacoves, authenticated with an [API key of the environment](/docs/how-tos/airflow/use-airflow-api#step-2-generate-an-api-key). Datacoves calls the environment's Airflow for it, so the process never handles Airflow credentials. The URL is on the environment's `Environment Keys` page, under `How to use them` > `Trigger asset`. + +```bash +curl -X POST "https://api./api/v1/trigger-asset" \ + -H "Authorization: Bearer $DATACOVES_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"asset_uri": "s3://my_bucket/my_folder/my_file.csv", "extra": {"rows": 1200}}' +``` + +```python +import os + +import requests + +response = requests.post( + os.environ["DATACOVES_TRIGGER_ASSET_URL"], + headers={"Authorization": f"Bearer {os.environ['DATACOVES_API_KEY']}"}, + json={ + "asset_uri": "s3://my_bucket/my_folder/my_file.csv", + "extra": {"rows": 1200}, + }, + timeout=60, +) +response.raise_for_status() +print(response.json()) +``` + +The request body: + +| Field | Required | Description | +| --- | --- | --- | +| `asset_uri` | Yes | The URI of the dataset, exactly as the DAG declares it. | +| `extra` | No | A JSON object stored with the event; DAGs can read it from the triggering event. | +| `partition_key` | No | For partitioned assets. | +| `environment` | No | The environment slug. Environment API keys can only publish to their own environment, so it is only needed with a project key. | + +The response is Airflow's: the created event on success. Errors: + +| Status | Meaning | +| --- | --- | +| `400` | The request is malformed, or the environment runs Airflow 2 (see below). | +| `401` | The API key is missing, wrong or was deleted. | +| `403` | The key belongs to another environment, or its owner cannot run DAGs in this Airflow. | +| `404` | No dataset with that URI exists in Airflow yet (see the note at the end). | +| `409` | Airflow is not enabled in the environment. | +| `429` | Too many requests with this key; retry after a while. | +| `503` | Airflow could not be reached, for example while it restarts; retry later. | + +:::note +On Airflow 2 environments the trigger-asset endpoint is not available. Send the event to Airflow's own API instead: `POST /datasets/events` with `{"dataset_uri": "..."}` and the header `Authorization: Bearer `. +::: ## Lambda Function -Alternatively, you can trigger a DAG externally using the [Airflow API](/docs/how-tos/airflow/use-airflow-api). In this example we will be using an AWS Lambda Function to trigger your DAG once data lands in an S3 Bucket. +In this example we will be using an AWS Lambda Function to trigger your DAG once data lands in an S3 Bucket. ### Creating your zip files @@ -67,55 +126,43 @@ along with your `lambda_function.py` file. **Example Lambda function:** ```python -import requests -import os import json +import os + +import requests # In Lambda, environment variables are set in the Lambda configuration # rather than using dotenv -API_URL = os.environ.get("AIRFLOW_API_URL") +TRIGGER_ASSET_URL = os.environ.get("DATACOVES_TRIGGER_ASSET_URL") API_KEY = os.environ.get("DATACOVES_API_KEY") -def update_dataset(dataset_name): - url = f"{API_URL}/datasets/events" +def trigger_asset(asset_uri, extra): response = requests.post( - url=url, - headers={ - "Authorization": f"Token {API_KEY}", - }, - json={"dataset_uri": dataset_name,} + url=TRIGGER_ASSET_URL, + headers={"Authorization": f"Bearer {API_KEY}"}, + json={"asset_uri": asset_uri, "extra": extra}, + timeout=60, ) + response.raise_for_status() + return response.json() - try: - return response.json() - except ValueError: - return response.text - -def print_response(response): - if response: - msg = json.dumps(response, indent=2) - print(f"Event posted successfully:\n{'='*30}\n\n {msg}") def lambda_handler(event, context): print("Lambda execution started") try: - print(f"Environment variables: API_URL={API_URL is not None}, API_KEY={API_KEY is not None}") - # Extract S3 information bucket = event['Records'][0]['s3']['bucket']['name'] key = event['Records'][0]['s3']['object']['key'] - print(f"S3 event details: bucket={bucket}, key={key}") - print(f"File uploaded: {bucket}/{key}") - # Airflow Dataset name must be static so if filename changes, that would have to - # be addressed above - dataset_name = f"s3://{bucket}/{key}" + # The dataset URI must match the one in the consumer DAG, so if the + # file name changes, map it to the static URI here. + asset_uri = f"s3://{bucket}/{key}" - response = update_dataset(dataset_name) - print_response(response) + response = trigger_asset(asset_uri, extra={"bucket": bucket, "key": key}) + print(f"Event posted successfully:\n{json.dumps(response, indent=2)}") return { 'statusCode': 200, @@ -151,10 +198,10 @@ zip -g deployment-package.zip lambda_function.py ### Set Environment Variables -- Gather your [API credentials](/docs/how-tos/airflow/use-airflow-api#step-1-navigate-to-your-target-environment) Configure the following environment variables in the Lambda Function's Configuration: +- Gather your [API key](/docs/how-tos/airflow/use-airflow-api#step-2-generate-an-api-key) and the `Trigger Asset URL` from the environment's `Environment Keys` page. Configure the following environment variables in the Lambda Function's Configuration: - - `AIRFLOW_API_URL` (the API URL for Airflow) - - `AIRFLOW_API_KEY` (the API key for authentication) + - `DATACOVES_TRIGGER_ASSET_URL` (the Trigger Asset URL) + - `DATACOVES_API_KEY` (the API key for authentication) ## Configuring the S3 Event Notification @@ -170,7 +217,7 @@ Now you are ready to set up your Consumer DAG. ## Setting Up the Airflow DAG -Whether you decide to use a producer DAG or the Airflow API, the last step is to create an Airflow DAG that is triggered by a dataset event rather than a schedule. This particular example can be triggered with either a `LAMBDA_UPDATED_DATASET` or `DAG_UPDATED_DATASET`. +Whether you decide to use a producer DAG or an external event, the last step is to create an Airflow DAG that is triggered by a dataset event rather than a schedule. This particular example can be triggered with either a `LAMBDA_UPDATED_DATASET` or `DAG_UPDATED_DATASET`. ![datasets graph](assets/datasets_graph.png) @@ -180,10 +227,10 @@ Whether you decide to use a producer DAG or the Airflow API, the last step is to import datetime from airflow.decorators import dag, task -from airflow.datasets import Dataset +from airflow.sdk import Asset -LAMBDA_UPDATED_DATASET = Dataset("s3://my_bucket/my_folder/my_file.csv") -DAG_UPDATED_DATASET = Dataset("upstream_data") +LAMBDA_UPDATED_DATASET = Asset("s3://my_bucket/my_folder/my_file.csv") +DAG_UPDATED_DATASET = Asset("upstream_data") @dag( default_args={ @@ -192,7 +239,7 @@ DAG_UPDATED_DATASET = Dataset("upstream_data") "email": "gomezn@example.com", "retries": 1 }, - description="Sample Producer DAG", + description="Sample Consumer DAG", schedule=(LAMBDA_UPDATED_DATASET | DAG_UPDATED_DATASET), tags=["transform"], catchup=False, @@ -210,6 +257,5 @@ dag = data_aware_consumer_dag() :::note -Ensure the Dataset you are sending an event to exists in Airflow. It will be created automatically when a DAG is created. If a dataset does not exist when the API event is sent, the API call will fail. +Ensure the Dataset you are sending an event to exists in Airflow. It will be created automatically when a DAG that uses it is parsed. If a dataset does not exist when the event is sent, the call fails (`404` from the trigger-asset endpoint). ::: - diff --git a/docs/how-tos/airflow/assets/environment_api_key.jpg b/docs/how-tos/airflow/assets/environment_api_key.jpg index 40b86a62..30d51ec1 100644 Binary files a/docs/how-tos/airflow/assets/environment_api_key.jpg and b/docs/how-tos/airflow/assets/environment_api_key.jpg differ diff --git a/docs/how-tos/airflow/assets/environment_api_url.jpg b/docs/how-tos/airflow/assets/environment_api_url.jpg deleted file mode 100644 index 2f430bb0..00000000 Binary files a/docs/how-tos/airflow/assets/environment_api_url.jpg and /dev/null differ diff --git a/docs/how-tos/airflow/dags/dag-parse-times.md b/docs/how-tos/airflow/dags/dag-parse-times.md index a0a8b280..b2fffd25 100644 --- a/docs/how-tos/airflow/dags/dag-parse-times.md +++ b/docs/how-tos/airflow/dags/dag-parse-times.md @@ -18,13 +18,13 @@ Airflow 2 does not expose parse durations through its API; see [the note below]( ## What you need -1. An Airflow API token for the target environment: in the launchpad, go to **Environments**, open your environment's **Keys**, and click **Generate New Airflow Token**. Copy the token immediately -- it is only shown once. +1. An API key for the target environment: in the launchpad, go to **Environments**, open your environment's **Keys**, and click **Generate New API Key**. Copy the key immediately -- it is only shown once. It does not expire; delete it to revoke it. See [How to use the Airflow API](/docs/how-tos/airflow/use-airflow-api). 2. The **Airflow API URL** shown on that same page (`https://airflow-./api/v2/`). 3. Store both in a `.env` file (and add it to `.gitignore`): ```env AIRFLOW_API_URL = "https://airflow-your-env.your-domain.com/api/v2/" -AIRFLOW_API_TOKEN = "your-token-here" +DATACOVES_API_KEY = "your-key-here" ``` The script works the same from CI, from Datacoves VSCode, or from any environment you use to reach Datacoves -- the Airflow API endpoints are reachable from all of them. @@ -42,7 +42,7 @@ from dotenv import load_dotenv load_dotenv() API_URL = os.getenv("AIRFLOW_API_URL").rstrip("/") -HEADERS = {"Authorization": f"Bearer {os.getenv('AIRFLOW_API_TOKEN')}"} +HEADERS = {"Authorization": f"Bearer {os.getenv('DATACOVES_API_KEY')}"} def fetch_all_dags(): @@ -92,20 +92,7 @@ A parse time consistently above ~1 second is worth investigating: the usual caus ## Alternative: authenticate as your own user -The Airflow API token above belongs to the environment's service account and is a JWT with an expiration date, so long-lived automation would need re-generating it. As an alternative, you can authenticate with your personal Datacoves API key (generated under [User Settings > My Airflow API](/docs/how-tos/my_airflow/use-my-airflow-api) -- despite the name, it is a general Datacoves key): exchange it for a short-lived Airflow JWT at `POST /auth/token` using the special `__datacoves_token__` username. This authenticates you as **your own user** with your permissions, the key does not expire, and the same key works against **any environment you have access to**: - -```python -def get_jwt(base_url, datacoves_api_key): - """Exchange a Datacoves API key for a short-lived Airflow JWT.""" - response = requests.post( - f"{base_url}/auth/token", - json={"username": "__datacoves_token__", "password": datacoves_api_key}, - ) - response.raise_for_status() - return response.json()["access_token"] -``` - -Use the returned token in the same `Authorization: Bearer` header as above (`base_url` is the Airflow URL without `/api/v2`). +The API key above belongs to the environment's service account. You can also use your personal Datacoves API key (generated under [User Settings > My Airflow API](/docs/how-tos/my_airflow/use-my-airflow-api) -- despite the name, it is a general Datacoves key) in the same `Authorization: Bearer` header. Airflow then runs the requests as **your own user**, with your permissions, and the same key works against **any environment you have access to**. ## Airflow 2 environments diff --git a/docs/how-tos/airflow/use-airflow-api.md b/docs/how-tos/airflow/use-airflow-api.md index 03a30194..99ea1dd8 100644 --- a/docs/how-tos/airflow/use-airflow-api.md +++ b/docs/how-tos/airflow/use-airflow-api.md @@ -1,7 +1,7 @@ --- title: Access the Airflow REST API in Datacoves sidebar_label: Airflow REST API -description: "Authenticate with and call the Airflow REST API in Datacoves: generate API tokens, trigger DAGs, check run status, and query task instances programmatically." +description: "Authenticate with and call the Airflow REST API in Datacoves: generate API keys, trigger DAGs, check run status, and query task instances programmatically." sidebar_position: 2 --- # How to use the Airflow API @@ -9,7 +9,7 @@ sidebar_position: 2 :::warning Users must have Project Level Admin Group to use the Airflow API. The API will allow you to view secrets values in plain text. Always exercise the principle of least privilege. ::: -This how to will walk you through configuring the Airflow API and using it in a DAG. +This how to will walk you through getting an API key for an environment and using it with the Airflow API. ## Step 1: Navigate to your target environment @@ -17,30 +17,52 @@ This how to will walk you through configuring the Airflow API and using it in a ![Environments](../datacoves/assets/menu_environments.gif) -- Then Select the Pad lock icon for the Airflow environment you wish to access. +- Then Select the Pad lock icon for the Airflow environment you wish to access. It opens the environment's `Environment Keys` page. ![Pad lock](assets/environments_api_edit.jpg) -## Step 2: Copy the API URL +## Step 2: Generate an API key -Copy the `Airflow API URL` +Generate your API key and copy it somewhere secure. Once you click away it will not be shown again. -![API URL](assets/environment_api_url.jpg) +![Environment Keys page](assets/environment_api_key.jpg) -## Step 3:Generate the API KEY +API keys do not expire. To revoke one, delete it from the list: Datacoves rejects it right away, and Airflow 3 stops accepting it within a minute. Create one key per integration (for example one per CI pipeline) so you can revoke them independently. -Generate your API key and copy it somewhere secure. Once you click away it will not be shown again. +The same key works with the Airflow API and, on Airflow 3, with the [trigger-asset endpoint](/docs/how-tos/airflow/api-triggered-dag#trigger-from-outside-airflow-with-datacoves) that publishes asset events. + +:::note +Airflow 3 environments used to issue Airflow tokens on this page. They are listed under `Legacy Airflow Tokens` until they expire, and Airflow keeps accepting each one until then, even after you remove it from the list. Move your integrations to an API key. +::: + +## Step 3: Copy the API URL + +On the right of the same page, under `How to use them`, copy the `Airflow API URL`. It depends on the Airflow version: + +| Airflow 2 | Airflow 3 | +| --- | --- | +| `https://api-airflow-./api/v1/` | `https://airflow-./api/v2/` | -![Generate API](assets/environment_api_key.jpg) +Send the API key in the `Authorization` header as a bearer token, on both versions: + +```bash +curl "$AIRFLOW_API_URL/dags" -H "Authorization: Bearer $DATACOVES_API_KEY" +``` + +:::note +Airflow 2 environments also accept the older `Authorization: Token ` header, so existing integrations keep working. +::: ## Step 4: Add your credentials to a .env file Create a `.env` file inside your `orchestrate/` directory and be sure to add the file to your `.gitignore`. Add your credentials there. -```env +```ini AIRFLOW_API_URL = "https://..." DATACOVES_API_KEY = "...." +# 3 for Airflow 3, 2 for Airflow 2 (only changes how a DAG run is triggered) +AIRFLOW_VERSION = "3" ``` ## Step 5: Use it in a python script @@ -56,86 +78,58 @@ Below is a sample script that makes use of the Airflow API. - Triggers a new DAG run for a specified DAG using the API. -- Updates an Airflow dataset using the API. - -- Handles API errors and retries requests if necessary. +To publish asset (dataset) events from outside Airflow, see [How to trigger a DAG using Datasets](/docs/how-tos/airflow/api-triggered-dag). ```python # airflow_api_call.py -import requests -import os import json +import os + +import requests from dotenv import load_dotenv load_dotenv() -API_URL = os.getenv("AIRFLOW_API_URL") -API_KEY = os.getenv("DATACOVES_API_KEY") +API_URL = os.getenv("AIRFLOW_API_URL").rstrip("/") +API_KEY = os.getenv("DATACOVES_API_KEY") +AIRFLOW_VERSION = os.getenv("AIRFLOW_VERSION", "3") -def update_dataset(name): - url=f"{API_URL}/datasets/events" - - response = requests.post( - url=url, - headers = { - "Authorization": f"Token {API_KEY}", - }, - json={"dataset_uri": "upstream_data",} - ) - return response.json() +HEADERS = {"Authorization": f"Bearer {API_KEY}"} def trigger_dag(dag_id): - url=f"{API_URL}/dags/{dag_id}/dagRuns" + body = {"note": "Trigger from API"} + if AIRFLOW_VERSION == "3": + # Airflow 3 requires the logical date; None runs it now. + body["logical_date"] = None response = requests.post( - url=url, - headers = { - "Authorization": f"Token {API_KEY}", - }, - json={"note": "Trigger from API",} + url=f"{API_URL}/dags/{dag_id}/dagRuns", + headers=HEADERS, + json=body, ) + response.raise_for_status() return response.json() def list_dags(): - url=f"{API_URL}/dags" - - response = requests.get( - url=url, - headers = { - "Authorization": f"Token {API_KEY}", - "Content-Type": "application/json", - }, - ) - - # Extract just the DAG names from the response - dags_data = response.json() - dag_names = [dag['dag_id'] for dag in dags_data['dags']] + response = requests.get(url=f"{API_URL}/dags", headers=HEADERS) + response.raise_for_status() - # Sort the names alphabetically for better readability - dag_names.sort() - return dag_names + # Extract just the DAG names from the response, sorted for readability + return sorted(dag["dag_id"] for dag in response.json()["dags"]) def print_response(response): if response: msg = json.dumps(response, indent=2) - print(f"Event posted successfully:\n{'='*30}\n\n {msg}") + print(f"Response:\n{'='*30}\n\n {msg}") if __name__ == "__main__": - - # Update an Airflow Dataset - # dataset_name = "upstream_data" - # response = update_dataset(dataset_name) - # print_response(response) - # Trigger a DAG - # dag_id = "bad_variable_usage" - # response = trigger_dag(dag_id) + # response = trigger_dag("bad_variable_usage") # print_response(response) # List DAGs - response = list_dags() - print_response(response) + print_response(list_dags()) ``` diff --git a/docs/how-tos/airflow/use-key-pair-authentication.md b/docs/how-tos/airflow/use-key-pair-authentication.md index 8f17a3a9..4fd9fa41 100644 --- a/docs/how-tos/airflow/use-key-pair-authentication.md +++ b/docs/how-tos/airflow/use-key-pair-authentication.md @@ -18,6 +18,8 @@ Also, make sure to select `Authentication mechanism: RSA Key-pair` and [assign t ![authentication_mechanism_copy](./assets/authentication_mechanism_copy.png) +To keep the private key in your own secrets backend instead of in Datacoves, and rotate it there, see [Use warehouse credentials from your secrets backend](/docs/how-tos/datacoves/how_to_credentials_from_secrets_backend). + Once you have properly configured your Service Connection, your Airflow instance will restart. This process may take up to 10 minutes. To confirm the above worked correctly, you can access your Airflow `Admin -> Connections` menu, and make sure your Service Connection is now an Airflow Connection. diff --git a/docs/how-tos/datacoves/how_to_credentials_from_secrets_backend.md b/docs/how-tos/datacoves/how_to_credentials_from_secrets_backend.md new file mode 100644 index 00000000..e4142f36 --- /dev/null +++ b/docs/how-tos/datacoves/how_to_credentials_from_secrets_backend.md @@ -0,0 +1,158 @@ +--- +title: "Use Credentials and Git Keys From Your Secrets Backend" +sidebar_label: "Credentials From Your Secrets Backend" +description: "Reference Snowflake key pairs, Redshift passwords, Databricks tokens, BigQuery keyfiles and git SSH keys from your own secrets backend so Datacoves never stores them and you can rotate them in place." +sidebar_position: 50 +--- +# How to use credentials and git keys from your secrets backend + +Instead of pasting a private key, password, token, keyfile or git SSH key into Datacoves, you can keep it in your own secrets backend (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager or HashiCorp Vault) and give Datacoves only a reference to it. Datacoves stores the reference, never the value, and reads the secret when Airflow or VS Code use it. To rotate the credential you update the secret in your backend; nothing has to change in Datacoves. + +What can be referenced: + +| Warehouse | Credential | Available from | +| --- | --- | --- | +| Snowflake | Key-pair private key | Release 6.1 | +| Redshift | Password | Release 6.2 | +| Databricks | Token | Release 6.2 | +| BigQuery | Service account keyfile (JSON) | Release 6.2 | +| Git | Project deploy key and your own SSH keys | Release 6.2 | + +Snowflake passwords can not be referenced: Snowflake is retiring password-only sign-in, so use a key pair instead. + +## Requirements + +- **The feature is enabled for your cluster.** It needs the External Secrets Operator installed in the cluster and the `external_secrets` feature turned on. Both are done by Datacoves; contact us to have it enabled. Until then, the options described below do not appear. +- **The environment has an Airflow secrets backend configured**, at the project or the environment level. Datacoves reads the credentials from that same backend, with the same credentials or identity, so there is nothing new to configure. See [configure your project's secrets backend](/docs/how-tos/datacoves/how_to_projects) and, for example, [AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) or [Azure Key Vault](/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault). +- **That identity can read the secrets you reference.** Grant it read access to them in your backend, the same way you grant it access to the Airflow variables and connections it reads. + +### Supported backends and authentication + +| Backend | Supported authentication | +| --- | --- | +| AWS Secrets Manager | Static access keys, IRSA | +| Azure Key Vault | Service principal, workload identity, managed identity | +| GCP Secret Manager | Service account keyfile | +| HashiCorp Vault | Token, AppRole, Kubernetes | + +Not supported: EKS Pod Identity and GCP Workload Identity. An environment whose secrets backend uses one of them can still use it for Airflow, but credentials can not be referenced from it. + +For HashiCorp Vault with Kubernetes authentication, bind the Vault role to the Airflow worker service account of the environment: `-airflow-worker`, or `datacoves-airflow` when the environment uses a workload identity. Airflow must be enabled in the environment. + +## Store the secret + +Create the secret in your backend before you reference it: + +- **Snowflake key pair:** the unencrypted private key in PEM format (`-----BEGIN PRIVATE KEY-----` ...). Register the matching public key on the Snowflake user yourself; see [Snowflake key-pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth). +- **Redshift password** or **Databricks token:** the value itself. +- **BigQuery keyfile:** the whole service account keyfile JSON. + +The secret can hold the value on its own, or as one property of a secret that stores several values (a Vault KV secret, or a JSON secret in AWS Secrets Manager). + +The reference is the name or path of the secret exactly as your backend calls it. The naming rules of the Airflow secrets backend (the `datacoves-` prefix, `airflow-connections-` and so on) do not apply here. For HashiCorp Vault, the path is relative to the mount point configured in the secrets backend, for example `snowflake/svc_loader` for `secret/snowflake/svc_loader`. + +## Reference it from Datacoves + +Every reference has two fields: + +- **Secret reference:** the name or path of the secret. +- **Property:** the field that holds the value, when the secret stores several values. Leave it empty when the whole secret is the value. + +### Service connections (Airflow) + +In the [service connection](/docs/how-tos/datacoves/how_to_service_connections) form: + +- **Snowflake:** set the authentication mechanism to key pair and choose `From your secrets backend` as the key-pair type. Fill in the reference and the property, optionally a key name, and click `Save reference`. Both delivery modes work: + - **Airflow Connection:** the connection's private key file points to the key, so [`@task.datacoves_dbt`](/docs/reference/airflow/datacoves-decorators) works as usual. + - **Environment Variables:** the path to the key is delivered as `DATACOVES____PRIVATE_KEY_PATH`. +- **Redshift, Databricks and BigQuery:** set the password, token or keyfile source to `From your secrets backend` and fill in the reference and the property. These credentials are delivered as environment variables only, so the delivery mode switches to `Environment Variables`. For BigQuery, also enter the `BigQuery project ID`: Datacoves does not hold the keyfile, so it can not take the project from it for the dbt profiles. + +With environment variables, the password and the token keep the names Datacoves already uses (`DATACOVES____PASSWORD`, `DATACOVES____TOKEN`), so the profile in your repository does not change. The Snowflake key and the BigQuery keyfile are files, and the profile points to them: + +```yaml +# Snowflake +private_key_path: "{{ env_var('DATACOVES__MAIN__PRIVATE_KEY_PATH') }}" + +# BigQuery +method: service-account +keyfile: "{{ env_var('DATACOVES__MAIN__KEYFILE_PATH') }}" +project: "{{ env_var('DATACOVES__MAIN__PROJECT') }}" +``` + +Replace `MAIN` with the name of your service connection. + +### Your own database connections (VS Code) + +- **Snowflake:** in your user _Settings_, under _Database Authorization Keys_, click `Add` and choose `Reference from your secrets backend`. Give the key a name and fill in the reference and the property. Then, in your database connection, choose `Key pair` as the auth type and select that key. +- **Redshift, Databricks and BigQuery:** in your database connection, set the password, token or keyfile source to `From your secrets backend` and fill in the reference and the property (and the BigQuery project ID). + +Datacoves writes your dbt profiles for VS Code, so there is nothing to change in them. + +The reference is read from the secrets backend of the environment the connection belongs to. The same reference in two environments reads from each environment's own backend. + +## Git SSH keys + +The project deploy key, which Airflow and dbt Docs use to clone your repository, and the SSH keys you use from VS Code can also come from your secrets backend. Store the OpenSSH private key (`-----BEGIN OPENSSH PRIVATE KEY-----` ...) in the backend, and register its public key in your git provider yourself: Datacoves does not generate these keys. + +- **Deploy key:** in _Admin > Projects_, edit the project, set the clone strategy to `SSH` and, under _Deploy key_, choose `From your secrets backend` instead of `Generated by Datacoves`. Fill in the secret reference and the property, optionally paste the public key so you can tell which key it is, and click `Use this reference`. Then save the project. +- **Your own SSH key:** in your user _Settings_, under _Git SSH Keys_, click `Add` and choose `Reference from your secrets backend`. Fill in the reference, the property and optionally the public key, and save. Then test the connection of your repository as usual. + +Each environment reads the key from its own Airflow secrets backend (the project's, unless the environment overrides it), so the secret must exist in the backend of every environment that uses it. The connection test reads the key through one of the project's environments, so a deploy key can only be tested once the project has an environment. + +To rotate a git key, add the new public key to your git provider next to the old one and update the secret. Airflow and dbt Docs use the new key within about two minutes, with no restart. VS Code reads its keys when the workspace starts, so restart it. Then remove the old public key from your git provider. + +Git keys show no status badge and send no rotation or failure notices; if the key can not be read, the connection test of the project or of your repository shows the error. + +To stop using the reference, switch back to `Generated by Datacoves` (deploy key) or delete the key from your settings; Airflow, dbt Docs and VS Code go back to the key Datacoves stores. + +## Rotation + +This section covers warehouse credentials; git keys are covered [above](#git-ssh-keys). Update the secret in your backend. Datacoves reads every referenced secret again every minute, and the new value reaches your workloads without any change in Datacoves: + +| Credential | Airflow | VS Code | +| --- | --- | --- | +| Snowflake key, BigQuery keyfile (files) | Within about two minutes, no restart | Within about two minutes, no restart | +| Redshift password, Databricks token (environment variables) | Tasks that start after the change | After your VS Code workspace restarts | + +To rotate a Snowflake key pair without failures, keep the old key valid until Datacoves uses the new one. Snowflake accepts two public keys per user for this purpose: + +1. Generate the new key pair and register the new public key in the free slot, for example `ALTER USER svc_loader SET RSA_PUBLIC_KEY_2 = '...';`. +2. Update the private key in your secrets backend. +3. Wait for the rotation notice (see below), or check that the fingerprint shown in Datacoves matches `RSA_PUBLIC_KEY_2_FP` in `DESC USER svc_loader`. +4. Once your jobs run with the new key, remove the old one: `ALTER USER svc_loader UNSET RSA_PUBLIC_KEY;`. + +For passwords and tokens, keep the old value valid until running Airflow tasks finish and VS Code workspaces have restarted, where your warehouse allows it. + +## Status and notifications + +Credentials whose value comes from your secrets backend show a status badge in the service connections list, in the connection forms and next to your keys: + +| Badge | Meaning | +| --- | --- | +| `Key pending` / `Secret pending` | Not read yet, for example right after the connection was saved. | +| `Key in sync` / `Secret in sync` | Read from your secrets backend. | +| `Key out of sync` / `Secret out of sync` | The last read failed. Datacoves keeps using the value it read before, so nothing fails yet. | +| `Key unreadable` / `Secret unreadable` | The value has never been read. Connections that use it fail until it can be read. | + +Hover the badge for the error and for when the value in use last changed. For keys, the badge also shows the fingerprint of the public key in use, in the same `SHA256:...` format as Snowflake's `RSA_PUBLIC_KEY_FP`. Click it to copy it. The fingerprint appears after Datacoves next checks your secrets, within about two minutes of saving the connection. + +Datacoves also tells you, by email and as a notification in the workbench, when: + +- the value in use changed (a rotation reached the environment), including the new and the previous fingerprint for keys; +- reading the secret started failing, with the error. The previous value is still in use, so do not remove it from the warehouse until this is fixed; +- reading works again. + +The notices for a database connection go to its owner. The notices for a service connection go to the users who can edit service connections in the account. A rotation is noticed within about three minutes, and a secret used in several environments sends one notice per environment. + +## Troubleshooting + +Testing a connection (saving its form, or the `Test connection` button next to it) reads the secret again right away, without waiting for the next read, and shows the error your backend returned, for example `Secret does not exist`. After fixing the problem in your backend, test the connection again to retry. + +Common errors: + +- **`Secrets referenced from your secrets backend need the External Secrets Operator, which is not installed in this cluster`:** the feature is not set up in your cluster; contact Datacoves. +- **`The environment's secrets backend can't be used for keys` (or `secrets`) followed by a reason:** the environment has no secrets backend, or its authentication is not supported (see the tables above). +- **`Secret does not exist`, or an access denied error:** check the reference and that the backend identity can read it. +- **`The secret was read but it holds no ...`:** the secret exists but the property is wrong or empty. Check the property name, or leave it empty if the whole secret is the value. + +If the feature is turned off for a cluster later, credentials that already reference your secrets backend keep working; only new references can not be created. diff --git a/docs/how-tos/datacoves/how_to_projects/README.md b/docs/how-tos/datacoves/how_to_projects/README.md index 26ea67d2..0311a766 100644 --- a/docs/how-tos/datacoves/how_to_projects/README.md +++ b/docs/how-tos/datacoves/how_to_projects/README.md @@ -27,6 +27,8 @@ A Project configuration consists of the following fields: When SSH is selected, an SSH public key will be automatically generated for you to configure in your git provider as a deployment key. +If your cluster has it enabled, you can instead keep the deploy key in your own secrets backend: choose `From your secrets backend` under _Deploy key_. See [git SSH keys from your secrets backend](/docs/how-tos/datacoves/how_to_credentials_from_secrets_backend#git-ssh-keys). + ![Repo SSH Key](../assets/projects_ssh_key.png) @@ -71,6 +73,6 @@ See this [how-to guide on configuring Azure DevOps](/docs/how-tos/datacoves/how_ - **CI/CD Provider:** When provided, this will display a link to your CI/CD jobs on the Observe tab of a Datacoves environment. Once you choose your provider, you will be able to specify your `CI jobs home URL`. -- **Secrets Backend:** Datacoves provides a Secrets Backend out of the box; you can also configure additional Secrets Backends at the project level, at the environment level, or both. See [AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) for details. +- **Secrets Backend:** Datacoves provides a Secrets Backend out of the box; you can also configure additional Secrets Backends at the project level, at the environment level, or both. See [AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) for details. The same backend can also hold the warehouse credentials of your connections; see [Use warehouse credentials from your secrets backend](/docs/how-tos/datacoves/how_to_credentials_from_secrets_backend). ![Project Secrets Backend](../assets/edit_project_secrets_backend.jpg) diff --git a/docs/how-tos/datacoves/how_to_service_connections.md b/docs/how-tos/datacoves/how_to_service_connections.md index b0b5d1ab..6f25c66d 100644 --- a/docs/how-tos/datacoves/how_to_service_connections.md +++ b/docs/how-tos/datacoves/how_to_service_connections.md @@ -30,6 +30,8 @@ A Service Connection consists of the following fields: ![Service Connection Connection Details](./assets/serviceconnection_editnew_details.png) +Instead of entering the password, token, keyfile or Snowflake private key, you can reference it from your own secrets backend. See [Use warehouse credentials from your secrets backend](/docs/how-tos/datacoves/how_to_credentials_from_secrets_backend). + ## Getting Started Next Steps In the following step, you will update your repository by incorporating the necessary folders and files for Airflow. Specifically, you will add the `orchestrate/dags` directories along with `automate/dbt/profiles.yml`. diff --git a/docs/how-tos/git/ssh-keys.md b/docs/how-tos/git/ssh-keys.md index 0b6e4e29..2aa29848 100644 --- a/docs/how-tos/git/ssh-keys.md +++ b/docs/how-tos/git/ssh-keys.md @@ -6,6 +6,8 @@ sidebar_position: 71 --- # How to configure a SSH Key to clone a git repository +Datacoves can generate the key for you, take a private key you provide, or read it from your own secrets backend; see [git SSH keys from your secrets backend](/docs/how-tos/datacoves/how_to_credentials_from_secrets_backend#git-ssh-keys). + ## Clone a Git repository for development purposes (write access required) [Github SSH user keys](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account) diff --git a/docs/reference/airflow/airflow-variables.md b/docs/reference/airflow/airflow-variables.md index ed246299..0df8dfc6 100644 --- a/docs/reference/airflow/airflow-variables.md +++ b/docs/reference/airflow/airflow-variables.md @@ -31,3 +31,8 @@ Variables containing versions: - `DATACOVES__VERSION_MAJOR_MINOR`: Datacoves version excluding patch level (e.g. 5.0) - `DATACOVES__VERSION__ENV`: Complete version of Datacoves including patch level for this environment - `DATACOVES__VERSION_MAJOR_MINOR__ENV`: Datacoves version excluding patch level for this environment + +## Datacoves API token +- `DATACOVES__API_TOKEN`: token Datacoves uses from inside Airflow, for example to read secrets from the Datacoves secrets manager and to upload the dbt manifest. From release 6.2 it belongs to a service user created for each environment with only those permissions: it can not manage Airflow users, connections or variables, and calls to the Airflow REST API with it are rejected. + +The Airflow tasks do not receive the credentials of the Datacoves admin service account. To call the Airflow API from your own code, generate an API key as described in [how to use the Airflow API](/docs/how-tos/airflow/use-airflow-api) and store it in your secrets backend.