Why three schedulers are all sending UPDATE dag SQL to backend meta db?

[trikker]

HI, I am deploying a production airflow (2.10.3) environment on CentOS7 and I want to make sure every component has HA(highly available), including:

• webserver
• scheduler
• celery worker

The webserver and celery worker are OK because they are not stateful, but after I started three schedulers on different machines, I found that three schedulers were all sending UPDATE dag SQL to backend meta db(which is MySQL 8.0):

I am sure all of the three schedulers sent the update SQL to db:

UPDATE dag SET next_dagrun='2024-11-18 13:35:00', next_dagrun_data_interval_start='2024-11-18 13:35:00', next_dagrun_data_interval_end='2024-11-18 13:36:00', next_dagrun_create_after='2024-11-18 13:36:00' WHERE dag.dag_id = 'mytest_dag'

Does it means that the three schedulers are working at the same time and will possibly make data in meta db wrong?

I was following this part of document in airflow, which says we can run more than one scheduler at the same time to get HA of scheduler by using the feature SELECT * FROM ... FOR UPDATE NOWAIT of DB as long as the meta DB supports the SQL.

https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html#running-more-than-one-scheduler

Thanks for help!

[potiuk]

No. they might do the same "work" - i.e. parse the same files and update their serialized and code. This is built-in in airflow scheduler design - DagFileProcessor is not supposed to "avoid" parsing the same file by multiple processors. There are various ways you can control it:

• you can have standalone dag file processor
• those processors can work only on a specified folders (but two dag file processors will still be able to parse and update info about the same DAG files)
• https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#file-parsing-sort-mode "random_seeded_by_host" sorting order is implemented so that your multiple schedulers (or multiple dag file processors) are not processing the DAG files in the same order - but randomly - to avoid the possibility that your dag file processors constantly process the same files at the same time. That allows you to decrease the parsing frequency as effectively your multiple processors will be processing the same files more frequently - but in different order.

[trikker]

Thanks, I think parsing the files for multiple times by multiple schedulers is ok because it has nothing to do with scheduling. I just want to avoid that one dag is scheduled by multiple schedulers by multiple times at the same time. So how can I confirm it?

[potiuk]

What confirmation do you need? You can watch this talk that explains how scheduler works for example https://www.youtube.com/watch?v=DYC4-xElccE