The ShortCircuitOperator in Apache Airflow is simple but powerful. It allows skipping tasks based on the result of a condition. There are many reasons why you may want to stop running tasks. Let’s see how to use the ShortCircuitOperator and what you should be aware of. By the way, if you are new to Airflow, …
ShortCircuitOperator in Apache Airflow: The guide Read More »
Airflow dynamic DAGs can save you a ton of time. As you know, Apache Airflow is written in Python, and DAGs are created via Python scripts. That makes it very flexible and powerful (even complex sometimes). By leveraging Python, you can create DAGs dynamically based on variables, connections, a typical pattern, etc. This very nice …
Dynamic DAGs in Apache Airflow: The Ultimate Guide Read More »
Airflow TaskGroups have been introduced to make your DAG visually cleaner and easier to read. They are meant to replace SubDAGs which was the historic way of grouping your tasks. The problem with SubDAGs is that they are much more than that. They bring a lot of complexity as you need to create a DAG …
Looking for creating your first Airflow DAG? Wondering how to process your data in Airflow? What are the steps to code your own data pipelines? You’ve come to the right place! At the end of this short tutorial, you will be able to code your first Airflow DAG! You might think it’s hard to start …
Airflow DAG: Creating your first DAG in 5 minutes Read More »
If you take a look at some DAG examples in my course “The Complete Hands-On Course to Master Apache Airflow”, you may notice the use of the “with” statement when a dag object is created. The question is why? Why it is considered as a best practice to use this Python keyword when you instantiate a …
