airflow dags list shows multiple copies of the same dag

Apache Airflow version

3.1.6

If "Other Airflow 3 version" selected, which one?

No response

What happened?

After running airflow dags reserialize, airflow dags list starts showing multiple copies of the DAG.

What you think should happen instead?

airflow dags list should only show one copy of each DAG.

How to reproduce

Use docker compose setup.
I adjusted the docker-compose.yaml by setting AIRFLOW__CORE__LOAD_EXAMPLES: 'false'.

I created this file dags/test1.py:

from airflow import sdk
from airflow.providers.standard.operators.python import PythonOperator
from time import sleep
import datetime

with sdk.DAG(
    dag_id="test1",
    schedule="* * * * *",
    start_date=datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(minutes=1),
    tags=["example"],
) as dag:
    PythonOperator(
        task_id="test1_task",
        python_callable=lambda: sleep(600),
    )

Started Airflow with docker compose up and waited for everything to start.

Ran the commands below. You can see that multiple copies of the dag are listed. No changes were made to test1.py.

2026-01-15 16:26:37 --> docker compose exec airflow-worker airflow dags list
dag_id | fileloc                    | owners  | is_paused | bundle_name | bundle_version
=======+============================+=========+===========+=============+===============
test1  | /opt/airflow/dags/test1.py | airflow | True      | dags-folder | None

2026-01-15 16:26:47 --> docker compose exec airflow-worker airflow dags unpause test1
dag_id | is_paused
=======+==========
test1  | True

2026-01-15 16:26:54 --> docker compose exec airflow-worker airflow dags list
dag_id | fileloc                    | owners  | is_paused | bundle_name | bundle_version
=======+============================+=========+===========+=============+===============
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None

2026-01-15 16:27:06 --> docker compose exec airflow-worker airflow dags reserialize
2026-01-15T15:27:12.526384Z [info     ] DAG bundles loaded: dags-folder [airflow.dag_processing.bundles.manager.DagBundlesManager] loc=manager.py:179
2026-01-15T15:27:12.581200Z [info     ] Filling up the DagBag from /opt/airflow/dags [airflow.models.dagbag.DagBag] loc=dagbag.py:593
2026-01-15T15:27:12.609688Z [info     ] Sync 1 DAGs                    [airflow.serialization.serialized_objects] loc=serialized_objects.py:2942
2026-01-15T15:27:12.623842Z [info     ] Setting next_dagrun for test1 to 2026-01-15 15:28:00+00:00, run_after=2026-01-15 15:28:00+00:00 [airflow.models.dag] loc=dag.py:693
2026-01-15 16:27:12 --> docker compose exec airflow-worker airflow dags list
dag_id | fileloc                    | owners  | is_paused | bundle_name | bundle_version
=======+============================+=========+===========+=============+===============
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None

2026-01-15 16:27:27 --> docker compose exec airflow-worker airflow dags list
dag_id | fileloc                    | owners  | is_paused | bundle_name | bundle_version
=======+============================+=========+===========+=============+===============
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None

2026-01-15 16:28:26 -->
2026-01-15 16:35:21 --> docker compose exec airflow-worker airflow dags list
dag_id | fileloc                    | owners  | is_paused | bundle_name | bundle_version
=======+============================+=========+===========+=============+===============
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None
test1  | /opt/airflow/dags/test1.py | airflow | False     | dags-folder | None

Operating System

MacOS (using docker images of Airflow)

Versions of Apache Airflow Providers

Using unmodified 3.1.6-python3.12 docker image

Deployment

Official Apache Airflow Helm Chart

Deployment details

I noticed this on our productive helm-chart deployed instance. I've reproduced it with the documented docker compose setup.

Anything else?

This has been happening since at least Airflow 3.1.5

It is repeatable with the provided example.

The dag_version and dag_code tables start filling up with multiple records for the test1 DAG. The dag_code.source_code_hash value is identical for all records where dag_id = 'test1'.

To trigger this, it seems to be necessary to

have a dag with task_instances (e.g. a DAG that has never run won't see this problem)
have run airflow dags reserialize at least once

Are you willing to submit PR?

Yes I am willing to submit a PR!
[ x ] No, I looked at the code and am unsure exactly why this is happening. It might be best if someone familiar with the DAG serialization code takes a look.

Code of Conduct

I agree to follow this project's Code of Conduct

The query needs to be updated to take into account dag versioning since in Airflow 3 multiple serialized entries can be present for a dag.

airflow/airflow-core/src/airflow/cli/commands/dag_command.py

Line 393 in 7524107

    
           dags_list.extend(cast("DAG", sm.dag) for sm in session.scalars(select(SerializedDagModel)))

@tirkarthi

@tirkarthi yes, certainly a good idea. However, I think it's also important that the dag_version and dag_code tables should not be filling up with new records, when there is no change to the DAG.

@brki

@brki That would be a different issue. Can you please share a simple reproducible demo? Please also note dynamic content in description, start_date etc. could also create a new version. Some reports in the past.

#58310
#59465
#54285

@hourly

Ok, I see dag start date being dynamic start_date=datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(minutes=1). Please use static start dates.

airflow.apache.org/docs/apache-airflow/stable/faq.html#what-s-the-deal-with-start-date

We recommend against using dynamic values as start_date, especially datetime.now() as it can be quite confusing. The task is triggered once the period closes, and in theory an @hourly Dag would never get to an hour after now as now() moves along.

https://stackoverflow.com/questions/41134524/why-is-it-recommended-against-using-a-dynamic-start-date-in-airflow

@tirkarthi

@tirkarthi ah, you're right, making it a static start date prevents continuous reserialization of an unchanged DAG file. Thanks for pointing that out.

Now I just need to figure out why my production DAG, that does use a static start date, is reserializing constantly (only while it is running). It is a long-running DAG, so dozens of serialized_dag and dag_code are being added every day.

It's worth checking out the diff of data column in serialised dag and dag code table entries which should give an idea.

That's the thing. dag_code.source_code and dag_code.source_code_hash are identical for all records for that DAG.

Please also check the serialised dag table data column for diff. In the above case datetime.now() remains the same in source but it gives different value every time when dag processor parses it.

I've been able to reproduce some strange behaviour with dag reserialization. I'll open a new issue for that.

@tirkarthi

@tirkarthi In the serialized_dag table, the data does sometimes differ. If I copy them into a python file and compare them, the unserialized dicts compare as equal (e.g. unserialized_dag_dict1 == unserialized_dag_dict2). The serialized strings are different though. Inspecting that, one can see that the order of some keys in the dict differs.

Navigation Menu

Repository navigation

airflow dags list shows multiple copies of the same dag #60594

Apache Airflow version

If "Other Airflow 3 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

airflow dags list shows multiple copies of the same dag #60594

Description

Apache Airflow version

If "Other Airflow 3 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Activity

tirkarthi commented on Jan 16, 2026

brki commented on Jan 16, 2026

tirkarthi commented on Jan 16, 2026

tirkarthi commented on Jan 16, 2026

brki commented on Jan 16, 2026

tirkarthi commented on Jan 16, 2026

brki commented on Jan 16, 2026

tirkarthi commented on Jan 16, 2026

brki commented on Jan 21, 2026

brki commented on Jan 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions