diff --git a/docs/how-tos/airflow/api-triggered-dag.md b/docs/how-tos/airflow/api-triggered-dag.md index e4c89488..a414c4d9 100644 --- a/docs/how-tos/airflow/api-triggered-dag.md +++ b/docs/how-tos/airflow/api-triggered-dag.md @@ -1,17 +1,23 @@ --- title: How to Trigger an Airflow DAG Using Datasets sidebar_label: Trigger DAG via Datasets -description: "Step-by-step guide to triggering Airflow DAGs via dataset events in Datacoves, enabling data-driven pipeline orchestration without time-based schedules." +description: "Step-by-step guide to triggering Airflow DAGs via dataset (asset) events in Datacoves, from another DAG or from an external process with one API call, enabling data-driven pipeline orchestration without time-based schedules." sidebar_position: 6 --- # How to Trigger a DAG using Datasets ## Overview -This guide explains how to trigger Airflow DAGs with Datasets. DAGs can be triggered by another DAG using datasets or by an external process that sends a dataset event using the Airflow API. +This guide explains how to trigger Airflow DAGs with Datasets, called **Assets** in Airflow 3. A dataset is a pointer to a piece of data, identified by a URI such as `s3://my_bucket/my_folder/my_file.csv`. A DAG scheduled on a dataset runs every time an event for that dataset is recorded. The event can come from another DAG (a producer) or from an external process (a script, a Lambda function, a CI job) that tells Airflow the data is ready. + +The external process does not need to know which DAGs depend on the data: it only says "this data is ready", and every DAG scheduled on it runs. + +:::note +The examples import `Asset` from `airflow.sdk`, available in Airflow 3. On Airflow 2 use `from airflow.datasets import Dataset` instead; the rest of the code is the same. +::: ## Producer DAG -Airflow enables DAGs to be triggered dynamically based on dataset updates. A producer DAG updates a dataset, automatically triggering any consumer DAGs subscribed to it. +A producer DAG updates a dataset, automatically triggering any consumer DAGs subscribed to it. To implement this, start by creating a DAG and defining the dataset it will update. @@ -20,13 +26,12 @@ To implement this, start by creating a DAG and defining the dataset it will upda import datetime from airflow.decorators import dag, task -from airflow.datasets import Dataset - +from airflow.sdk import Asset -# A dataset can be anything, it will be a poiner in the Airflow db. +# A dataset can be anything, it will be a pointer in the Airflow db. # If you need to access url like s3://my_bucket/my_file.txt then you can set # it with the proper path for reuse. -DAG_UPDATED_DATASET = Dataset("upstream_data") +DAG_UPDATED_DATASET = Asset("upstream_data") @dag( default_args={ @@ -51,11 +56,65 @@ def data_aware_producer_dag(): dag = data_aware_producer_dag() ``` -Thats it, now you are ready to create your [Consumer DAG](#setting-up-the-airflow-dag) +That's it, now you are ready to create your [Consumer DAG](#setting-up-the-airflow-dag). + +## Trigger from outside Airflow with Datacoves + +On Airflow 3 environments, an external process publishes a dataset event with a single request to Datacoves, authenticated with an [API key of the environment](/docs/how-tos/airflow/use-airflow-api#step-2-generate-an-api-key). Datacoves calls the environment's Airflow for it, so the process never handles Airflow credentials. The URL is on the environment's `Environment Keys` page, under `How to use them` > `Trigger asset`. + +```bash +curl -X POST "https://api./api/v1/trigger-asset" \ + -H "Authorization: Bearer $DATACOVES_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"asset_uri": "s3://my_bucket/my_folder/my_file.csv", "extra": {"rows": 1200}}' +``` + +```python +import os + +import requests + +response = requests.post( + os.environ["DATACOVES_TRIGGER_ASSET_URL"], + headers={"Authorization": f"Bearer {os.environ['DATACOVES_API_KEY']}"}, + json={ + "asset_uri": "s3://my_bucket/my_folder/my_file.csv", + "extra": {"rows": 1200}, + }, + timeout=60, +) +response.raise_for_status() +print(response.json()) +``` + +The request body: + +| Field | Required | Description | +| --- | --- | --- | +| `asset_uri` | Yes | The URI of the dataset, exactly as the DAG declares it. | +| `extra` | No | A JSON object stored with the event; DAGs can read it from the triggering event. | +| `partition_key` | No | For partitioned assets. | +| `environment` | No | The environment slug. Environment API keys can only publish to their own environment, so it is only needed with a project key. | + +The response is Airflow's: the created event on success. Errors: + +| Status | Meaning | +| --- | --- | +| `400` | The request is malformed, or the environment runs Airflow 2 (see below). | +| `401` | The API key is missing, wrong or was deleted. | +| `403` | The key belongs to another environment, or its owner cannot run DAGs in this Airflow. | +| `404` | No dataset with that URI exists in Airflow yet (see the note at the end). | +| `409` | Airflow is not enabled in the environment. | +| `429` | Too many requests with this key; retry after a while. | +| `503` | Airflow could not be reached, for example while it restarts; retry later. | + +:::note +On Airflow 2 environments the trigger-asset endpoint is not available. Send the event to Airflow's own API instead: `POST /datasets/events` with `{"dataset_uri": "..."}` and the header `Authorization: Bearer `. +::: ## Lambda Function -Alternatively, you can trigger a DAG externally using the [Airflow API](/docs/how-tos/airflow/use-airflow-api). In this example we will be using an AWS Lambda Function to trigger your DAG once data lands in an S3 Bucket. +In this example we will be using an AWS Lambda Function to trigger your DAG once data lands in an S3 Bucket. ### Creating your zip files @@ -67,55 +126,43 @@ along with your `lambda_function.py` file. **Example Lambda function:** ```python -import requests -import os import json +import os + +import requests # In Lambda, environment variables are set in the Lambda configuration # rather than using dotenv -API_URL = os.environ.get("AIRFLOW_API_URL") +TRIGGER_ASSET_URL = os.environ.get("DATACOVES_TRIGGER_ASSET_URL") API_KEY = os.environ.get("DATACOVES_API_KEY") -def update_dataset(dataset_name): - url = f"{API_URL}/datasets/events" +def trigger_asset(asset_uri, extra): response = requests.post( - url=url, - headers={ - "Authorization": f"Token {API_KEY}", - }, - json={"dataset_uri": dataset_name,} + url=TRIGGER_ASSET_URL, + headers={"Authorization": f"Bearer {API_KEY}"}, + json={"asset_uri": asset_uri, "extra": extra}, + timeout=60, ) + response.raise_for_status() + return response.json() - try: - return response.json() - except ValueError: - return response.text - -def print_response(response): - if response: - msg = json.dumps(response, indent=2) - print(f"Event posted successfully:\n{'='*30}\n\n {msg}") def lambda_handler(event, context): print("Lambda execution started") try: - print(f"Environment variables: API_URL={API_URL is not None}, API_KEY={API_KEY is not None}") - # Extract S3 information bucket = event['Records'][0]['s3']['bucket']['name'] key = event['Records'][0]['s3']['object']['key'] - print(f"S3 event details: bucket={bucket}, key={key}") - print(f"File uploaded: {bucket}/{key}") - # Airflow Dataset name must be static so if filename changes, that would have to - # be addressed above - dataset_name = f"s3://{bucket}/{key}" + # The dataset URI must match the one in the consumer DAG, so if the + # file name changes, map it to the static URI here. + asset_uri = f"s3://{bucket}/{key}" - response = update_dataset(dataset_name) - print_response(response) + response = trigger_asset(asset_uri, extra={"bucket": bucket, "key": key}) + print(f"Event posted successfully:\n{json.dumps(response, indent=2)}") return { 'statusCode': 200, @@ -151,10 +198,10 @@ zip -g deployment-package.zip lambda_function.py ### Set Environment Variables -- Gather your [API credentials](/docs/how-tos/airflow/use-airflow-api#step-1-navigate-to-your-target-environment) Configure the following environment variables in the Lambda Function's Configuration: +- Gather your [API key](/docs/how-tos/airflow/use-airflow-api#step-2-generate-an-api-key) and the `Trigger Asset URL` from the environment's `Environment Keys` page. Configure the following environment variables in the Lambda Function's Configuration: - - `AIRFLOW_API_URL` (the API URL for Airflow) - - `AIRFLOW_API_KEY` (the API key for authentication) + - `DATACOVES_TRIGGER_ASSET_URL` (the Trigger Asset URL) + - `DATACOVES_API_KEY` (the API key for authentication) ## Configuring the S3 Event Notification @@ -170,7 +217,7 @@ Now you are ready to set up your Consumer DAG. ## Setting Up the Airflow DAG -Whether you decide to use a producer DAG or the Airflow API, the last step is to create an Airflow DAG that is triggered by a dataset event rather than a schedule. This particular example can be triggered with either a `LAMBDA_UPDATED_DATASET` or `DAG_UPDATED_DATASET`. +Whether you decide to use a producer DAG or an external event, the last step is to create an Airflow DAG that is triggered by a dataset event rather than a schedule. This particular example can be triggered with either a `LAMBDA_UPDATED_DATASET` or `DAG_UPDATED_DATASET`. ![datasets graph](assets/datasets_graph.png) @@ -180,10 +227,10 @@ Whether you decide to use a producer DAG or the Airflow API, the last step is to import datetime from airflow.decorators import dag, task -from airflow.datasets import Dataset +from airflow.sdk import Asset -LAMBDA_UPDATED_DATASET = Dataset("s3://my_bucket/my_folder/my_file.csv") -DAG_UPDATED_DATASET = Dataset("upstream_data") +LAMBDA_UPDATED_DATASET = Asset("s3://my_bucket/my_folder/my_file.csv") +DAG_UPDATED_DATASET = Asset("upstream_data") @dag( default_args={ @@ -192,7 +239,7 @@ DAG_UPDATED_DATASET = Dataset("upstream_data") "email": "gomezn@example.com", "retries": 1 }, - description="Sample Producer DAG", + description="Sample Consumer DAG", schedule=(LAMBDA_UPDATED_DATASET | DAG_UPDATED_DATASET), tags=["transform"], catchup=False, @@ -210,6 +257,5 @@ dag = data_aware_consumer_dag() :::note -Ensure the Dataset you are sending an event to exists in Airflow. It will be created automatically when a DAG is created. If a dataset does not exist when the API event is sent, the API call will fail. +Ensure the Dataset you are sending an event to exists in Airflow. It will be created automatically when a DAG that uses it is parsed. If a dataset does not exist when the event is sent, the call fails (`404` from the trigger-asset endpoint). ::: - diff --git a/docs/how-tos/airflow/assets/environment_api_key.jpg b/docs/how-tos/airflow/assets/environment_api_key.jpg index 40b86a62..30d51ec1 100644 Binary files a/docs/how-tos/airflow/assets/environment_api_key.jpg and b/docs/how-tos/airflow/assets/environment_api_key.jpg differ diff --git a/docs/how-tos/airflow/assets/environment_api_url.jpg b/docs/how-tos/airflow/assets/environment_api_url.jpg deleted file mode 100644 index 2f430bb0..00000000 Binary files a/docs/how-tos/airflow/assets/environment_api_url.jpg and /dev/null differ diff --git a/docs/how-tos/airflow/dags/dag-parse-times.md b/docs/how-tos/airflow/dags/dag-parse-times.md index a0a8b280..b2fffd25 100644 --- a/docs/how-tos/airflow/dags/dag-parse-times.md +++ b/docs/how-tos/airflow/dags/dag-parse-times.md @@ -18,13 +18,13 @@ Airflow 2 does not expose parse durations through its API; see [the note below]( ## What you need -1. An Airflow API token for the target environment: in the launchpad, go to **Environments**, open your environment's **Keys**, and click **Generate New Airflow Token**. Copy the token immediately -- it is only shown once. +1. An API key for the target environment: in the launchpad, go to **Environments**, open your environment's **Keys**, and click **Generate New API Key**. Copy the key immediately -- it is only shown once. It does not expire; delete it to revoke it. See [How to use the Airflow API](/docs/how-tos/airflow/use-airflow-api). 2. The **Airflow API URL** shown on that same page (`https://airflow-./api/v2/`). 3. Store both in a `.env` file (and add it to `.gitignore`): ```env AIRFLOW_API_URL = "https://airflow-your-env.your-domain.com/api/v2/" -AIRFLOW_API_TOKEN = "your-token-here" +DATACOVES_API_KEY = "your-key-here" ``` The script works the same from CI, from Datacoves VSCode, or from any environment you use to reach Datacoves -- the Airflow API endpoints are reachable from all of them. @@ -42,7 +42,7 @@ from dotenv import load_dotenv load_dotenv() API_URL = os.getenv("AIRFLOW_API_URL").rstrip("/") -HEADERS = {"Authorization": f"Bearer {os.getenv('AIRFLOW_API_TOKEN')}"} +HEADERS = {"Authorization": f"Bearer {os.getenv('DATACOVES_API_KEY')}"} def fetch_all_dags(): @@ -92,20 +92,7 @@ A parse time consistently above ~1 second is worth investigating: the usual caus ## Alternative: authenticate as your own user -The Airflow API token above belongs to the environment's service account and is a JWT with an expiration date, so long-lived automation would need re-generating it. As an alternative, you can authenticate with your personal Datacoves API key (generated under [User Settings > My Airflow API](/docs/how-tos/my_airflow/use-my-airflow-api) -- despite the name, it is a general Datacoves key): exchange it for a short-lived Airflow JWT at `POST /auth/token` using the special `__datacoves_token__` username. This authenticates you as **your own user** with your permissions, the key does not expire, and the same key works against **any environment you have access to**: - -```python -def get_jwt(base_url, datacoves_api_key): - """Exchange a Datacoves API key for a short-lived Airflow JWT.""" - response = requests.post( - f"{base_url}/auth/token", - json={"username": "__datacoves_token__", "password": datacoves_api_key}, - ) - response.raise_for_status() - return response.json()["access_token"] -``` - -Use the returned token in the same `Authorization: Bearer` header as above (`base_url` is the Airflow URL without `/api/v2`). +The API key above belongs to the environment's service account. You can also use your personal Datacoves API key (generated under [User Settings > My Airflow API](/docs/how-tos/my_airflow/use-my-airflow-api) -- despite the name, it is a general Datacoves key) in the same `Authorization: Bearer` header. Airflow then runs the requests as **your own user**, with your permissions, and the same key works against **any environment you have access to**. ## Airflow 2 environments diff --git a/docs/how-tos/airflow/use-airflow-api.md b/docs/how-tos/airflow/use-airflow-api.md index 03a30194..99ea1dd8 100644 --- a/docs/how-tos/airflow/use-airflow-api.md +++ b/docs/how-tos/airflow/use-airflow-api.md @@ -1,7 +1,7 @@ --- title: Access the Airflow REST API in Datacoves sidebar_label: Airflow REST API -description: "Authenticate with and call the Airflow REST API in Datacoves: generate API tokens, trigger DAGs, check run status, and query task instances programmatically." +description: "Authenticate with and call the Airflow REST API in Datacoves: generate API keys, trigger DAGs, check run status, and query task instances programmatically." sidebar_position: 2 --- # How to use the Airflow API @@ -9,7 +9,7 @@ sidebar_position: 2 :::warning Users must have Project Level Admin Group to use the Airflow API. The API will allow you to view secrets values in plain text. Always exercise the principle of least privilege. ::: -This how to will walk you through configuring the Airflow API and using it in a DAG. +This how to will walk you through getting an API key for an environment and using it with the Airflow API. ## Step 1: Navigate to your target environment @@ -17,30 +17,52 @@ This how to will walk you through configuring the Airflow API and using it in a ![Environments](../datacoves/assets/menu_environments.gif) -- Then Select the Pad lock icon for the Airflow environment you wish to access. +- Then Select the Pad lock icon for the Airflow environment you wish to access. It opens the environment's `Environment Keys` page. ![Pad lock](assets/environments_api_edit.jpg) -## Step 2: Copy the API URL +## Step 2: Generate an API key -Copy the `Airflow API URL` +Generate your API key and copy it somewhere secure. Once you click away it will not be shown again. -![API URL](assets/environment_api_url.jpg) +![Environment Keys page](assets/environment_api_key.jpg) -## Step 3:Generate the API KEY +API keys do not expire. To revoke one, delete it from the list: Datacoves rejects it right away, and Airflow 3 stops accepting it within a minute. Create one key per integration (for example one per CI pipeline) so you can revoke them independently. -Generate your API key and copy it somewhere secure. Once you click away it will not be shown again. +The same key works with the Airflow API and, on Airflow 3, with the [trigger-asset endpoint](/docs/how-tos/airflow/api-triggered-dag#trigger-from-outside-airflow-with-datacoves) that publishes asset events. + +:::note +Airflow 3 environments used to issue Airflow tokens on this page. They are listed under `Legacy Airflow Tokens` until they expire, and Airflow keeps accepting each one until then, even after you remove it from the list. Move your integrations to an API key. +::: + +## Step 3: Copy the API URL + +On the right of the same page, under `How to use them`, copy the `Airflow API URL`. It depends on the Airflow version: + +| Airflow 2 | Airflow 3 | +| --- | --- | +| `https://api-airflow-./api/v1/` | `https://airflow-./api/v2/` | -![Generate API](assets/environment_api_key.jpg) +Send the API key in the `Authorization` header as a bearer token, on both versions: + +```bash +curl "$AIRFLOW_API_URL/dags" -H "Authorization: Bearer $DATACOVES_API_KEY" +``` + +:::note +Airflow 2 environments also accept the older `Authorization: Token ` header, so existing integrations keep working. +::: ## Step 4: Add your credentials to a .env file Create a `.env` file inside your `orchestrate/` directory and be sure to add the file to your `.gitignore`. Add your credentials there. -```env +```ini AIRFLOW_API_URL = "https://..." DATACOVES_API_KEY = "...." +# 3 for Airflow 3, 2 for Airflow 2 (only changes how a DAG run is triggered) +AIRFLOW_VERSION = "3" ``` ## Step 5: Use it in a python script @@ -56,86 +78,58 @@ Below is a sample script that makes use of the Airflow API. - Triggers a new DAG run for a specified DAG using the API. -- Updates an Airflow dataset using the API. - -- Handles API errors and retries requests if necessary. +To publish asset (dataset) events from outside Airflow, see [How to trigger a DAG using Datasets](/docs/how-tos/airflow/api-triggered-dag). ```python # airflow_api_call.py -import requests -import os import json +import os + +import requests from dotenv import load_dotenv load_dotenv() -API_URL = os.getenv("AIRFLOW_API_URL") -API_KEY = os.getenv("DATACOVES_API_KEY") +API_URL = os.getenv("AIRFLOW_API_URL").rstrip("/") +API_KEY = os.getenv("DATACOVES_API_KEY") +AIRFLOW_VERSION = os.getenv("AIRFLOW_VERSION", "3") -def update_dataset(name): - url=f"{API_URL}/datasets/events" - - response = requests.post( - url=url, - headers = { - "Authorization": f"Token {API_KEY}", - }, - json={"dataset_uri": "upstream_data",} - ) - return response.json() +HEADERS = {"Authorization": f"Bearer {API_KEY}"} def trigger_dag(dag_id): - url=f"{API_URL}/dags/{dag_id}/dagRuns" + body = {"note": "Trigger from API"} + if AIRFLOW_VERSION == "3": + # Airflow 3 requires the logical date; None runs it now. + body["logical_date"] = None response = requests.post( - url=url, - headers = { - "Authorization": f"Token {API_KEY}", - }, - json={"note": "Trigger from API",} + url=f"{API_URL}/dags/{dag_id}/dagRuns", + headers=HEADERS, + json=body, ) + response.raise_for_status() return response.json() def list_dags(): - url=f"{API_URL}/dags" - - response = requests.get( - url=url, - headers = { - "Authorization": f"Token {API_KEY}", - "Content-Type": "application/json", - }, - ) - - # Extract just the DAG names from the response - dags_data = response.json() - dag_names = [dag['dag_id'] for dag in dags_data['dags']] + response = requests.get(url=f"{API_URL}/dags", headers=HEADERS) + response.raise_for_status() - # Sort the names alphabetically for better readability - dag_names.sort() - return dag_names + # Extract just the DAG names from the response, sorted for readability + return sorted(dag["dag_id"] for dag in response.json()["dags"]) def print_response(response): if response: msg = json.dumps(response, indent=2) - print(f"Event posted successfully:\n{'='*30}\n\n {msg}") + print(f"Response:\n{'='*30}\n\n {msg}") if __name__ == "__main__": - - # Update an Airflow Dataset - # dataset_name = "upstream_data" - # response = update_dataset(dataset_name) - # print_response(response) - # Trigger a DAG - # dag_id = "bad_variable_usage" - # response = trigger_dag(dag_id) + # response = trigger_dag("bad_variable_usage") # print_response(response) # List DAGs - response = list_dags() - print_response(response) + print_response(list_dags()) ```