Airflow DAG patterns that keep pipelines maintainable as they grow

Most Airflow pipelines start simple and become unmaintainable in the same predictable way: task granularity gets finer over time without anyone deciding it should, sensors pile up waiting on things that rarely fail, and nobody can safely rerun a single day’s DAG without side effects. Here’s how to avoid all three.

Pick task granularity once, and write it down

A task should represent one unit of work you’d want to retry independently. “Extract,” “transform,” “load” as three tasks is usually right. Splitting “transform” into ten tasks because each one is a different SQL statement usually isn’t — it just means ten places a DAG can go yellow in the UI without adding any real retry value.

The test: if two tasks always succeed or fail together and you’d never want to retry one without the other, they should be one task.

Idempotency is not optional

Every task should be safe to rerun for the same execution date without creating duplicate data or double-charging a downstream system. In practice this means:

• Loads should DELETE + INSERT or MERGE on the partition/date key, never blind INSERT.
• API calls that aren’t naturally idempotent need an idempotency key or a “has this run for this date” check before firing.
• Avoid tasks that depend on wall-clock time (NOW()) instead of the DAG’s logical execution date — this is the single most common cause of “it worked in backfill testing but broke in production.”

Sensors: use them sparingly and always set a timeout

A sensor with no timeout set will happily poll forever, holding a worker slot and hiding the fact that an upstream dependency never showed up. Always set timeout and mode='reschedule' (not mode='poke') for anything that might wait more than a few minutes — reschedule releases the worker slot between checks instead of blocking it.

wait_for_upstream = ExternalTaskSensor(
    task_id="wait_for_upstream",
    external_dag_id="upstream_dag",
    timeout=60 * 60 * 2,
    mode="reschedule",
    poke_interval=300,
)

Group related tasks with TaskGroups, not naming conventions

If you’re naming tasks extract_orders_1, extract_orders_2 to signal they’re related, use a TaskGroup instead. It keeps the DAG graph readable as the pipeline grows and makes it obvious in the UI which tasks belong together, instead of relying on everyone reading task IDs carefully.

What to check in your current DAGs

Open your busiest DAG and ask three questions: does every load task handle reruns safely, does every sensor have a timeout, and would a new team member understand task grouping from the graph view alone. Any “no” is your next fix.