At some point, every data team hits the same roadblock. Each night, the script that used to run smoothly starts to go wrong at 2 a.m. No one knows what step has gone wrong, and reruns are double-counting the numbers that the finance team has already published. Apache Airflow is great because it's been rebuilt from the ground up to suit the way work teams actually do things today.
This cookbook provides you hundreds of solutions that are independent and will take you from the first installation to a platform that supports ETL, ELT, MLOps, AIOps and business operations all at the same time. Each recipe starts with a real problem, has short and easy-to-read code, and ends by showing how the fix works with real terminal output. We can learn to author DAGs with the Task SDK, schedule pipelines on data rather than on the clock, bind extractions so reruns repair instead of duplicates, and run any task inside its own container. We will be practising to write custom operators, packaging them for other teams, extending Airflow through plugins and secrets backends, orchestrating model training and promotion, provisioning infrastructure that tears itself down, and diagnosing stalls from their symptoms.
To me, this book is best suited for every software engineer, backend developer and every such platform teams who keep on building, running and troubleshooting the workflows every day.
Key Learnings
Convert legacy operator DAGs into TaskFlow functions that infer dependencies from ordinary calls
Bound every extraction to its data interval so reruns repair rather than duplicate
Schedule pipelines on asset updates instead of guessing when upstream work finishes
Fan tasks out dynamically over lists discovered at runtime using expand and partial
Run any task inside a Kubernetes pod with its own image and resources
Write custom operators, hooks and sensors, then package them as installable providers
Extend Airflow through plugins, macros, secrets backends and custom XCom storage
Orchestrate model training, registry logging and promotion gates without a serving endpoint
Provision ephemeral infrastructure with setup and teardown pairs that always release resources
Diagnose stuck queues, starved pools and zombie tasks from their distinct symptoms
Table of Content
Getting Airflow 3 Running
Authoring DAGs with Task SDK
Scheduling, Assets and Event-Driven Pipelines
Building ETL Pipelines
ELT and Warehouse Orchestration
Extracting Insights from Batch Processes
Containers and Kubernetes
Custom Operators, Hooks and Sensors
Plugin Interface and Extending Airflow
Managing ML Pipelines
AIOps, Infrastructure and Business Operations
Testing, Monitoring and Troubleshooting
