Blog

timezones in apache airflow

How to use timezones in Apache Airflow

Dealing with timezones in general can become a real nightmare if they are not correctly used. Understanding how timezones in Apache Airflow work is important since you may want to schedule your DAGs according to your local time zone, which can lead to surprises when DST (Daylight Saving Time) happens.

Read More »
monitoring_apache_airflow

Apache Airflow monitoring with TIG: Part 1

Monitoring Apache Airflow should be your top priority when you are in production. Without monitoring you have no way to know if anything goes wrong. Imagine that Airflow stops working for any reason, since it is your orchestrator,  your data pipelines won’t be scheduled anymore and so your data won’t

Read More »
templates_and_macros_in_apache_airflow

How to use templates and macros in Apache Airflow

Templates and Macros in Apache Airflow are the way to pass dynamic data to your DAGs at runtime. Let’s imagine that you would like to execute a SQL request using the execution date of your DAG? How can you do that? How could you use the DAG id of your

Read More »
kubernetes-executor-airflow

Apache Airflow with Kubernetes Executor and MiniKube

In this tutorial, we are going to see how to use Apache Airflow with Kubernetes Executor. If you are using Airflow in production, there is a big chance that your workload fluctuates over time. Sometimes you have many tasks to execute and sometimes not at all. The problem is in both cases,

Read More »
dockeroperator_with_spark

How to use the DockerOperator in Apache Airflow

Wondering how to use the DockerOperator in Apache Airflow to kick off a docker and run commands? Let’s discover this operator through a practical example. In a more and more containerized world, it can be very useful to know how to interact with your Docker containers through Apache Airflow. In this

Read More »
python-operator-basics-in-apache-airflow

Apache Airflow | How to use the PythonOperator

Wondering how can we run python code through Airflow ? The Airflow PythonOperator does exactly what you are looking for. It is a very simple but powerful operator, allowing you to execute a Python callable function from your DAG. You may have seen in my course “The Complete Hands-On Course

Read More »
Scroll to Top