airflow_aws

Apache Airflow: The Operators Guide

Apache Airflow is an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.

In the Apache Airflow: The Operators Guide, you are going to learn how to create reliable, efficient and powerful tasks in your Airflow data pipelines. Learn how to create incredible data pipelines and discover the most untold secrets of Airflow Operators!

Materials (required for the course)

You will find the materials directly in a video of the course

Curriculum

Section 1: Introduction

  1. Important Prerequisites
  2. Course Objectives
  3. Who am I?
  4. Development Environment

Section 2: The BaseOperator Exposed

  1. Introduction
  2. What you need to know
  3. Important details
  4. The task id exposed
  5. DAG versioning
  6. Who owns the task?
  7. The start_date is more than you think
  8. Be PROPERLY notified by emails
  9. Make your tasks dependent across DAG Runs
  10. Wait for downstream tasks
  11. Pool Party
  12. Task priority
  13. Define the priority
  14. A callback to react
  15. Catch a timeout with the failure callback
  16. Share data with the famous XCOMs
  17. Limit the concurrency
  18. Chain and Cross dependency helpers
  19. Let’s play with chain and cross helpers
  20. Customize your task according to the executor

Section 3: The Most Common Operators

  1. Introduction to Providers
  2. The PythonOperator
  3. The PythonOperator with the Taskflow API
  4. The BashOperator
  5. Quick note about templated fields
  6. The PostgresOperator
  7. Passing Dynamic Parameters to the PostgresOperator

Section 4: Choose your path

  1. Introduction
  2. The BranchPythonOperator
  3. DAG scheduling based on a Calendar!
  4. The BranchSQLOperator
  5. The BranchDatetimeOperator
  6. The BranchDayOfWeekOperator

Section 5: DAG dependencies

  1. Introduction
  2. The SubDAGOperator
  3. Deep dive into the SubDAGOperator
  4. Discover the TaskGroups
  5. TriggerDagRunOperator
  6. The famous ExternalTaskSensor
  7. DAG dependencies view

Section 6: The Exotic ones

  1. One DAG different schedules with the ShortCircuitOperator
  2. Prevent from running past tasks with the LatestOnlyOperator
  3. Side Note with the LatestOnlyOperator
  4. You think you know the DummyOperator?

Section 7: Last words

  1. Choose the next operator