DAG Parsing Performance with Large Number of Dynamically Generated DAGs

[pedro-cf]

Current Setup & Challenge

I have a dynamic DAG generation system where DAGs are created from configurations stored in a database table. The main issue I'm facing is DAG parsing performance as I scale to 1000+ dynamic DAGs.

The Problem

• Each time the DAG file processor runs, it needs to parse all DAGs
• With 1000+ DAGs being generated dynamically, the parsing time is becoming a bottleneck
• This affects scheduler performance and DAG file processing interval

Current Process

1. Query database for all enabled DAG configurations
2. For each configuration, generate a DAG object
3. Add each DAG to the global namespace
4. This happens every time the DAG file is processed

Questions

1. What's the recommended approach to handle DAG parsing for large numbers of dynamically generated DAGs?

2. Are there built-in mechanisms in Airflow to partition DAG file processing to improve parsing performance?

3. How do other organizations handle similar scale with dynamic DAG generation?

[potiuk]

1. What's the recommended approach to handle DAG parsing for large numbers of dynamically generated DAGs?

Make them as fast and as little overhead as you can https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code

2. Are there built-in mechanisms in Airflow to partition DAG file processing to improve parsing performance?

This is coming in Airflow 3 in the form of DAG bundles: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816356

3. How do other organizations handle similar scale with dynamic DAG generation?

• Optimizing your parsing time is the best idea
• Have several standalone dag processors and splitting the processing by --subdir flags - you can split your DAGs by subdirectories and parse each subdirectory by one or more separate dag file processors.
• Split your airflow installation into several independent ones

[pedro-cf]

@potiuk My issue is that I have 1 dag file dynamic_dags.py that queries the database and based on the configuration it creates specific dags. When the number of dag configs exceeds 2000 the parsing takes too long. How can I split this workload through multiple dag processors without manually creating multiple dag files with a query skip/limit?

[tanelk]

Perhaps this is helpfull: https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html#optimizing-dag-parsing-delays-during-execution

It will not break the "main" dag parsing loop into pieces, but reduces the load when executing.