Skip to content

[DCV-3973][6.1] Document the datacoves airflow3-check command - #59

Open
gams87 wants to merge 7 commits into
mainfrom
DCV-3973-airflow3-check-docs
Open

gams87 wants to merge 7 commits into
mainfrom
DCV-3973-airflow3-check-docs

Conversation

@gams87

@gams87 gams87 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Adds a "Check your DAGs for Airflow 3 migration issues" how-to under How-tos > Airflow, documenting the new datacoves airflow3-check command (datacoves/datacoves#1591, DCV-3973).

Covers what the check finds (a table of Airflow 2 patterns and their Airflow 3 replacements), how to run it against the environment's configured DAGs folder or an explicit path, exit codes for scripting, --fix and its safe-fixes-only behavior, and a recommended migration workflow including the portable try/except import pattern - which the check recognizes and does not flag (verified against a real portable DAG).

Publish together with the release that ships the command in the code-server image.

🤖 Generated with Claude Code

@gams87
gams87 marked this pull request as draft August 18, 2026 19:47
@gams87
gams87 marked this pull request as ready for review August 18, 2026 21:08
@gams87
gams87 requested a review from noel August 18, 2026 21:09
@gams87 gams87 changed the title [DCV-3973] Document the datacoves airflow3-check command [DCV-3973][6.1] Document the datacoves airflow3-check command Oct 9, 2026
@gams87
gams87 requested review from fmercadop and removed request for noel November 19, 2026 14:13
gams87 and others added 2 commits November 19, 2026 08:17
…ck-docs

# Conflicts:
#	docs/how-tos/airflow/use-azure-key-vault.mdx
…tes, its exit codes and ruff version

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pcV36f8rrHS7tBAXLWWGZ
@gams87

gams87 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I re-checked each point with ruff 0.16.0 (the 6.1 code-server version) on sample DAGs and against airflow3_check.py on releases/6.1; all confirmed, and the guide is updated in 4983239:

  1. email_on_failure / email_on_retry: removed from the "detected" table. A new "check by hand" table lists them (in default_args or passed to an operator), and the text no longer says the command looks for "all of these".
  2. --fix: the guide now says safe fixes include DummyOperator -> EmptyOperator and Dataset -> Asset (besides schedule_interval -> schedule), while the ones ruff marks unsafe (airflow.decorators, BashOperator) are reported but not applied. Added the tip that the old imports are left behind unused and ruff check --select F401 --fix <path> removes them.
  3. Not detected: the "check by hand" table also has sla inside default_args (noting that sla on an operator and sla_miss_callback are flagged) and context["execution_date"] -> context["logical_date"].
  4. Missing details: exit codes table (0 no issues, 1 findings, 2 could not run, e.g. no DAGs folder); with Airflow not enabled there is no default DAGs folder, so pass a path; full rule coverage needs ruff 0.15+ (6.1 ships 0.16), older versions get a warning.

Also merged main to clear the conflict (use-azure-key-vault.mdx front matter: kept main's new description and the sidebar_position: 110 from the reorder). npm run build passes.

@fmercadop fmercadop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update, the four points from the last review are addressed and the page matches airflow3_check.py on releases/6.1 (defaults, the Airflow-not-enabled message, ruff 0.15 minimum, exit codes, safe vs unsafe fixes, leftover imports, portable try/except not flagged, and the URL the command prints matches this page's route).

I dug into --fix with ruff 0.16.0 against real installs of Airflow 2.11.2 and 3.3.1 (the versions in the airflow-v2 / airflow images on releases/6.1), and there's one blocker plus two corrections.

1. --fix breaks DAGs on Airflow 2.11.2 (blocker)

--fix applies three fixes that ruff marks safe:

Fix What it writes Airflow 2.11.2 Airflow 3.3.1
schedule_interval -> schedule (both DAG(...) and @dag(...)) schedule= ✅ ✅
DummyOperator -> EmptyOperator from airflow.providers.standard.operators.empty import EmptyOperator ❌ No module named 'airflow.providers.standard' ✅
Dataset -> Asset from airflow.sdk import Asset ❌ No module named 'airflow.sdk' ✅

The standard provider isn't in the official 2.11.2 image extras nor in src/common/providers-v2/providers.txt, so it's not available on our Airflow 2 environments.

The "Recommended workflow" tells users to run the check while still on Airflow 2. Anyone who runs --fix there and pushes will break their DAGs. Ruff's "safe" means "doesn't change behavior on Airflow 3", not "compatible with Airflow 2". Only schedule_interval -> schedule is safe on Airflow 2.

2. Dataset and DummyOperator are not optional

Ruff reports airflow.datasets.Dataset as airflow3-suggested-update ("It still works in Airflow 3.0..."), but on 3.3.1 airflow.datasets no longer exists (ModuleNotFoundError). Same for airflow.operators.dummy. Both must be migrated before the upgrade.

Portable alternatives I verified on both 2.11.2 and 3.3.1 (and the check doesn't flag them):

# DummyOperator -> works on 2.11 and 3.3.1 (deprecation warning on 3)
from airflow.operators.empty import EmptyOperator

# Dataset -> Asset
try:
    from airflow.sdk import Asset
except ImportError:
    from airflow.datasets import Dataset as Asset

3. context["execution_date"] is detected in most cases

The "check by hand" table says it isn't, but ruff 0.16 flags execution_date (plus next_ds, prev_execution_date, etc.) when it comes from **context / **kwargs in a @task (both ["..."] and .get(...)) or as a task parameter (def t(execution_date=None)).

It is not detected when it comes from get_current_context()["execution_date"] or from a callback (def on_failure(context): context["execution_date"]). I'd narrow the row to those two cases.

Suggested text for "Apply automatic fixes"

:::warning
Only schedule_interval -> schedule is safe to apply while the environment runs Airflow 2. The other automatic fixes write Airflow 3-only imports (airflow.sdk, airflow.providers.standard) that fail on Airflow 2. Run --fix on a branch you merge together with the upgrade, or use the portable patterns in the workflow below.
:::

Dataset and DummyOperator must be migrated before upgrading: airflow.datasets and airflow.operators.dummy no longer exist in the Airflow 3 version Datacoves runs, even though the check reports Dataset as a suggested update.

And add the two portable patterns above next to the airflow.sdk one in the workflow.

Nits (non-blocking)

  • Mention that [*] in the output marks the findings --fix will apply; it's the easiest way to know beforehand. The real output also includes a code snippet and a [*] N fixable ... (M hidden fixes ...) summary line.
  • ruff check --select F401 --fix removes every unused import in the file, not just the migration leftovers; worth a "review the diff" note.
  • from airflow import DAG is flagged too (moved to airflow.sdk, no auto-fix) and will probably be the most common finding; consider adding it to the first table.

…rating Dataset and DummyOperator, and add portable patterns

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pcV36f8rrHS7tBAXLWWGZ
@gams87

gams87 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, all confirmed and addressed in 9a85026. I re-checked against our images: on Airflow 2.11.2 (airflow-airflow-v2 6.0) airflow.providers.standard and airflow.sdk don't import; on 3.3.1 airflow.datasets and airflow.operators.dummy don't; your portable patterns import on both. With ruff 0.16.0, execution_date / next_ds are flagged from **context, **kwargs and as a task parameter, not from get_current_context() or a callback.

  1. Blocker: added your warning to "Apply automatic fixes" (only schedule_interval -> schedule is safe on Airflow 2; the others write Airflow 3-only imports; run --fix on a branch merged with the upgrade or use the portable patterns).
  2. Dataset / DummyOperator: the page says they must be migrated before upgrading, both are in the first table, and the two portable patterns are in the workflow next to the airflow.sdk one.
  3. execution_date: the "check by hand" row is narrowed to get_current_context() and callbacks, and says the **context / **kwargs / parameter cases are detected.

Nits: [*] and the fixable-count line are explained, the F401 tip now says it removes every unused import (review the diff), and from airflow import DAG -> from airflow.sdk import DAG is in the first table. npm run build passes.

@fmercadop

Copy link
Copy Markdown
Contributor

Thanks, 9a85026 addresses everything from my review. One correction on my side, though: the EmptyOperator portable pattern I suggested is wrong.

from airflow.operators.empty import EmptyOperator is flagged

I only tested the import, not usage. Once the operator is used, ruff 0.16 flags it:

airflow3-suggested-to-move-to-provider: `airflow.operators.empty.EmptyOperator` is deprecated and moved into `standard` provider in Airflow 3.0

That also means step 2 of the workflow ("Re-run it until it reports no issues") can't be reached on Airflow 2: EmptyOperator and BashOperator stay flagged, and their new location (airflow.providers.standard) doesn't exist on 2.11.2.

Fix: try/except for every Airflow 3-only location

try:
    from airflow.sdk import DAG, Asset
    from airflow.providers.standard.operators.empty import EmptyOperator
    from airflow.providers.standard.operators.bash import BashOperator
except ImportError:
    from airflow import DAG
    from airflow.datasets import Dataset as Asset
    from airflow.operators.empty import EmptyOperator
    from airflow.operators.bash import BashOperator

Verified with a DAG that actually uses each import:

  • ruff 0.16.0 (--select AIR --preview): All checks passed!, exit 0
  • Airflow 2.11.2: loads, resolves to the old modules
  • Airflow 3.3.1: loads, resolves to airflow.providers.standard and airflow.sdk.Asset

Suggested changes:

  • Replace the # DummyOperator: EmptyOperator from this module... block with this pattern.
  • Generalize the sentence to something like: "For any import whose new location exists only in Airflow 3 (airflow.sdk, airflow.providers.standard), use a try/except import."

Nits

  • First table, DummyOperator row: spell out from airflow.providers.standard.operators.empty import EmptyOperator instead of just EmptyOperator.
  • Consider moving the :::warning above the datacoves airflow3-check --fix block so it's read before the command is copied.

Sorry for the extra round; with the pattern fixed this is good to go from my side.

…and move the --fix warning above the command

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pcV36f8rrHS7tBAXLWWGZ
@gams87

gams87 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, and good catch. Confirmed with ruff 0.16.0: airflow.operators.empty.EmptyOperator is flagged once it's used, and your try/except pattern passes (All checks passed!, exit 0). Addressed in 0260e6c:

  • The workflow now has a single try/except block for every Airflow 3-only location (DAG, Asset, dag, task, EmptyOperator, BashOperator), with the generalized sentence ("for any import whose new location exists only in Airflow 3 (airflow.sdk, airflow.providers.standard), use a try/except import"). The incorrect airflow.operators.empty block is gone, and the page notes that importing it without the try/except is flagged. I added dag, task to your pattern so there's one block instead of two; re-verified with a DAG using all six: ruff exit 0, loads on 2.11.2 (old modules) and 3.3.1 (airflow.sdk, airflow.providers.standard).
  • Nits: the DummyOperator row spells out from airflow.providers.standard.operators.empty import EmptyOperator, and the :::warning is now above the --fix command.

npm run build passes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants