running apache airflow locally

Running Apache Airflow locally on Kubernetes

Wouldn’t be convenient to be able to run Apache Airflow locally with the Kubernetes Executor on a multi-node Kubernetes cluster? That’s could be a great way to test your DAGs and understand how Airflow works in a Kubernetes environment isn’t it? Well that’s exactly what we are going to do

Read More »

Apache Airflow 2.0: What to expect?

Apache Airflow 2.0 is coming with a lot of big improvements. In this post, I’m gonna explain to you what are the most important features to expect what they solve. If you are totally new to Airflow, check my introduction course right there, or if you want an in-depth approach

Read More »

Variables in Apache Airflow: The Guide

How to deal with variables in Apache Airflow? In this tutorial, you are going to learn everything you need about variables in Airflow. What are they, how do they work, how can you define them, how to get them, and more. If you followed my course “Apache Airflow: The Hands-On

Read More »

Apache Airflow | How to use the BashOperator

Wondering how can you execute bash commands through Airflow ? The Airflow BashOperator does exactly what you are looking for. It is a very simple but powerful operator, allowing you to execute either a bash script, a command or a set of commands from your DAGs. You may have seen

Read More »
password authentication in apache airflow

Password Authentication in Apache Airflow

When you start using Apache Airflow in production, one of your top priority is to prevent its access to everyone. Indeed, since Airflow orchestrates data pipelines, it is a master piece of your data platform and can potentially deal with sensitive data. When we are working in a company, different

Read More »
timezones in apache airflow

How to use timezones in Apache Airflow

Dealing with timezones in general can become a real nightmare if they are not correctly used. Understanding how timezones in Apache Airflow work is important since you may want to schedule your DAGs according to your local time zone, which can lead to surprises when DST (Daylight Saving Time) happens.

Read More »