Building Reliable Data Pipelines With Apache Airflow

Apache Airflow is one of the most widely used tools for orchestrating data pipelines. It lets you define workflows as Python code, schedule them, manage dependencies, retry failures and see everything in a web UI. But Airflow is an orchestrator, not a processing engine, and many painful Airflow deployments come from forgetting that.

This guide covers the core concepts and the practices that keep pipelines reliable.

#Key takeaways

• Airflow orchestrates workflows defined as DAGs (directed acyclic graphs) of tasks.
• Airflow should coordinate work. Heavy processing belongs in a warehouse, Spark, containers or external services.
• Make every task idempotent and parameterized by its data interval, so re-runs and backfills are safe.
• Keep DAG files lightweight at parse time. The scheduler parses them constantly.
• Add retries, timeouts, alerts, data quality checks and tests.

#Core concepts

• DAG: a workflow with tasks and dependencies, plus a schedule.
• Task: a unit of work, created from an operator (Python, SQL, Bash, Kubernetes pod, cloud service) or the TaskFlow @task decorator.
• Data interval: the time window a run is responsible for. A daily run for 2026-09-27 processes data from that day.
• Scheduler: decides what should run and when.
• Executor and workers: actually run tasks (locally, on a Celery worker pool, or as Kubernetes pods).
• Connections and variables: centrally managed credentials and configuration.

#A minimal DAG

pythonfrom datetime import datetime, timedelta
from airflow.decorators import dag, task

default_args = {
    "retries": 3,
    "retry_delay": timedelta(minutes=5),
    "retry_exponential_backoff": True,
    "execution_timeout": timedelta(minutes=30),
}

@dag(
    schedule="@daily",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    default_args=default_args,
    tags=["orders"],
)
def orders_daily():

    @task
    def extract(data_interval_start=None, data_interval_end=None):
        # Pull only this run's window, then write to object storage
        path = f"s3://raw/orders/{data_interval_start:%Y-%m-%d}.parquet"
        export_orders(since=data_interval_start, until=data_interval_end, to=path)
        return path

    @task
    def load(path: str):
        load_into_warehouse(path, table="raw.orders", mode="replace_partition")

    @task
    def transform():
        run_sql_models(select="orders+")   # e.g., trigger dbt models

    load(extract()) >> transform()

orders_daily()

Notice: the task processes its data interval, writes to a partition that gets replaced on re-run, and passes a reference (a file path), not the data itself, between tasks.

#Principle 1: Idempotency

Running a task twice for the same interval must produce the same result. Techniques:

• Overwrite partitions instead of appending (DELETE WHERE date = X then INSERT, or MERGE).
• Use deterministic output paths keyed by the interval.
• Avoid datetime.now() in logic. Use the run's data interval.

Idempotency makes retries, manual re-runs and backfills safe. It's the single most important property of a pipeline.

#Principle 2: Airflow orchestrates; it doesn't crunch

Anti-pattern: loading a 20 GB DataFrame into memory inside a Python task on a shared worker. Instead:

• Push SQL down to the warehouse.
• Run heavy Python in containers (Kubernetes pods) or managed processing services.
• Use Spark or similar for large distributed processing.

Airflow triggers, monitors and sequences this work.

#Principle 3: Pass references, not data

XComs let tasks share small values: IDs, file paths, row counts. Don't push large datasets through XCom; write data to storage and pass its location.

#Principle 4: Keep DAG parsing fast

The scheduler re-parses DAG files regularly. Top-level code in DAG files runs on every parse. Avoid:

• Database or API calls at module level.
• Heavy imports that aren't needed to define the DAG.
• Generating hundreds of DAGs from slow external lookups.

Put work inside tasks, where it runs only at execution time.

#Scheduling, catchup and backfills

• catchup=False: don't automatically run all missed intervals since start_date. Usually what you want for new DAGs.
• Backfills: deliberately re-run historical intervals, for example after fixing a bug. Idempotent tasks make this safe.
• Data-aware scheduling: trigger DAGs when upstream datasets are updated, instead of guessing with time offsets.

#Sensors and waiting

Sensors wait for a condition: a file landing, a partition existing, an upstream DAG finishing. Use deferrable sensors or reschedule mode, so waiting doesn't occupy a worker slot for hours.

#Failure handling

• Retries with exponential backoff for transient issues.
• Execution timeouts so hung tasks don't block pipelines forever.
• SLAs and alerts (Slack, email, PagerDuty) routed to owners, with context: DAG, task, interval and log link.
• Failure callbacks to create tickets or post to incident channels for critical pipelines.

#Data quality inside the pipeline

A pipeline that runs successfully but loads bad data is worse than one that fails. Add checks as tasks:

python@task
def check_orders(data_interval_start=None):
    rows = count_rows("raw.orders", date=data_interval_start)
    if rows == 0:
        raise ValueError("No orders loaded — upstream export likely failed")
    nulls = count_nulls("raw.orders", "customer_id", date=data_interval_start)
    if nulls / rows > 0.01:
        raise ValueError(f"customer_id null rate too high: {nulls}/{rows}")

Or integrate a dedicated framework. See the data quality guide.

#Testing DAGs

• DAG integrity tests: every DAG imports without errors, has no cycles, has owners and tags, and uses allowed operators.
• Unit tests for task functions (pure Python logic).
• Integration tests against a staging environment with sample data.
• Run tests in CI before deploying DAG changes (CI/CD pipeline guide).

#Common anti-patterns

#When Airflow might not be the right fit

Airflow is best for batch, schedule-based orchestration with complex dependencies. Consider alternatives when:

• You need low-latency event processing → a streaming stack (batch vs stream processing).
• You have a handful of simple scheduled SQL models → your transformation tool's scheduler may be enough.
• You need application workflows with long-running state and human steps → durable workflow engines designed for application logic.

#FAQ

#Is Airflow an ETL tool?

Airflow is an orchestrator. It schedules and coordinates ETL/ELT tasks, but the actual extraction, loading and transformation are usually done by operators calling databases, warehouses, APIs or processing engines.

#How do I pass data between Airflow tasks?

Pass small metadata (IDs, file paths, counts) via XCom, and store actual datasets in object storage or a database. Downstream tasks read from those locations.

#Should I use Airflow or cron?

Cron works for a few independent scripts. Airflow adds dependency management, retries, backfills, monitoring, alerting and a UI. Those become valuable once you have multiple interdependent pipelines.

#Can Airflow run on Kubernetes?

Yes. It's a common deployment model, and each task can run in its own pod for isolation and resource control. Managed Airflow services are also available if you'd rather not operate it yourself.