Repository navigation
Conversation
…ck-docs # Conflicts: # docs/how-tos/airflow/use-azure-key-vault.mdx
…tes, its exit codes and ruff version Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015pcV36f8rrHS7tBAXLWWGZ
|
Thanks for the review. I re-checked each point with ruff 0.16.0 (the 6.1 code-server version) on sample DAGs and against
Also merged |
fmercadop
left a comment
There was a problem hiding this comment.
Thanks for the update, the four points from the last review are addressed and the page matches airflow3_check.py on releases/6.1 (defaults, the Airflow-not-enabled message, ruff 0.15 minimum, exit codes, safe vs unsafe fixes, leftover imports, portable try/except not flagged, and the URL the command prints matches this page's route).
I dug into --fix with ruff 0.16.0 against real installs of Airflow 2.11.2 and 3.3.1 (the versions in the airflow-v2 / airflow images on releases/6.1), and there's one blocker plus two corrections.
1. --fix breaks DAGs on Airflow 2.11.2 (blocker)
--fix applies three fixes that ruff marks safe:
| Fix | What it writes | Airflow 2.11.2 | Airflow 3.3.1 |
|---|---|---|---|
schedule_interval -> schedule (both DAG(...) and @dag(...)) |
schedule= |
✅ | ✅ |
DummyOperator -> EmptyOperator |
from airflow.providers.standard.operators.empty import EmptyOperator |
❌ No module named 'airflow.providers.standard' |
✅ |
Dataset -> Asset |
from airflow.sdk import Asset |
❌ No module named 'airflow.sdk' |
✅ |
The standard provider isn't in the official 2.11.2 image extras nor in src/common/providers-v2/providers.txt, so it's not available on our Airflow 2 environments.
The "Recommended workflow" tells users to run the check while still on Airflow 2. Anyone who runs --fix there and pushes will break their DAGs. Ruff's "safe" means "doesn't change behavior on Airflow 3", not "compatible with Airflow 2". Only schedule_interval -> schedule is safe on Airflow 2.
2. Dataset and DummyOperator are not optional
Ruff reports airflow.datasets.Dataset as airflow3-suggested-update ("It still works in Airflow 3.0..."), but on 3.3.1 airflow.datasets no longer exists (ModuleNotFoundError). Same for airflow.operators.dummy. Both must be migrated before the upgrade.
Portable alternatives I verified on both 2.11.2 and 3.3.1 (and the check doesn't flag them):
# DummyOperator -> works on 2.11 and 3.3.1 (deprecation warning on 3)
from airflow.operators.empty import EmptyOperator
# Dataset -> Asset
try:
from airflow.sdk import Asset
except ImportError:
from airflow.datasets import Dataset as Asset3. context["execution_date"] is detected in most cases
The "check by hand" table says it isn't, but ruff 0.16 flags execution_date (plus next_ds, prev_execution_date, etc.) when it comes from **context / **kwargs in a @task (both ["..."] and .get(...)) or as a task parameter (def t(execution_date=None)).
It is not detected when it comes from get_current_context()["execution_date"] or from a callback (def on_failure(context): context["execution_date"]). I'd narrow the row to those two cases.
Suggested text for "Apply automatic fixes"
:::warning
Onlyschedule_interval->scheduleis safe to apply while the environment runs Airflow 2. The other automatic fixes write Airflow 3-only imports (airflow.sdk,airflow.providers.standard) that fail on Airflow 2. Run--fixon a branch you merge together with the upgrade, or use the portable patterns in the workflow below.
:::
DatasetandDummyOperatormust be migrated before upgrading:airflow.datasetsandairflow.operators.dummyno longer exist in the Airflow 3 version Datacoves runs, even though the check reportsDatasetas a suggested update.
And add the two portable patterns above next to the airflow.sdk one in the workflow.
Nits (non-blocking)
- Mention that
[*]in the output marks the findings--fixwill apply; it's the easiest way to know beforehand. The real output also includes a code snippet and a[*] N fixable ... (M hidden fixes ...)summary line. ruff check --select F401 --fixremoves every unused import in the file, not just the migration leftovers; worth a "review the diff" note.from airflow import DAGis flagged too (moved toairflow.sdk, no auto-fix) and will probably be the most common finding; consider adding it to the first table.
…rating Dataset and DummyOperator, and add portable patterns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015pcV36f8rrHS7tBAXLWWGZ
|
Thanks, all confirmed and addressed in 9a85026. I re-checked against our images: on Airflow 2.11.2 (
Nits: |
|
Thanks, 9a85026 addresses everything from my review. One correction on my side, though: the
|
…and move the --fix warning above the command Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015pcV36f8rrHS7tBAXLWWGZ
|
Thanks, and good catch. Confirmed with ruff 0.16.0:
|
Adds a "Check your DAGs for Airflow 3 migration issues" how-to under How-tos > Airflow, documenting the new
datacoves airflow3-checkcommand (datacoves/datacoves#1591, DCV-3973).Covers what the check finds (a table of Airflow 2 patterns and their Airflow 3 replacements), how to run it against the environment's configured DAGs folder or an explicit path, exit codes for scripting,
--fixand its safe-fixes-only behavior, and a recommended migration workflow including the portable try/except import pattern - which the check recognizes and does not flag (verified against a real portable DAG).Publish together with the release that ships the command in the code-server image.
🤖 Generated with Claude Code