Since I started creating courses a year ago, I got so many messages asking me what are the best practices in Apache Airflow. As engineer, we always seek for the best ways to apply what we learn while being constantly improving ourselves. In this series of tutorial, I would like to share with you everything …
Wouldn’t be convenient to be able to run Apache Airflow locally with the Kubernetes Executor on a multi-node Kubernetes cluster? That’s could be a great way to test your DAGs and understand how Apache Airflow works in a Kubernetes environment isn’t it? Well that’s exactly what we are going to do here. I will show …
Wondering how to deal with variables in Apache Airflow? Well you are at the right place. In this tutorial, you are going to learn everything you need about the variables in Airflow. What are they, how they work, how can you define them, how to get them and more. If you followed my course “Apache Airflow: …
Wondering how can you execute bash commands through Airflow ? The Airflow BashOperator does exactly what you are looking for. It is a very simple but powerful operator, allowing you to execute either a bash script, a command or a set of commands from your DAGs. You may have seen in my course “The Complete …
When you start using Apache Airflow in production, one of your top priority is to prevent its access to everyone. Indeed, since Airflow orchestrates data pipelines, it is a master piece of your data platform and can potentially deal with sensitive data. When we are working in a company, different teams may want to access …
Dealing with timezones in general can become a real nightmare if they are not correctly used. Understanding how timezones in Apache Airflow work is important since you may want to schedule your DAGs according to your local time zone, which can lead to surprises when DST (Daylight Saving Time) happens. There are some subtle notions …
Monitoring Apache Airflow should be your top priority when you are in production. Without monitoring you have no way to know if anything goes wrong. Imagine that Airflow stops working for any reason, since it is your orchestrator, your data pipelines won’t be scheduled anymore and so your data won’t be processed as expected. If …
Templates and Macros in Apache Airflow are the way to pass dynamic data to your DAGs at runtime. Let’s imagine that you would like to execute a SQL request using the execution date of your DAG? How can you do that? How could you use the DAG id of your DAG in your bash script …
How to use templates and macros in Apache Airflow Read More »
In this tutorial, we are going to see how to use Apache Airflow with Kubernetes Executor. If you are using Airflow in production, there is a big chance that your workload fluctuates over time. Sometimes you have many tasks to execute and sometimes not at all. The problem is in both cases, your resources stay allocated to …
Apache Airflow with Kubernetes Executor and MiniKube Read More »
Wondering how to use the DockerOperator in Apache Airflow to kick off a docker and run commands? Let’s discover this operator through a practical example. In a more and more containerized world, it can be very useful to know how to interact with your Docker containers through Apache Airflow. In this article, we are going to …
