Repository navigation
Airflow 3.2.0 scheduler/triggerer deadlock on task_instance due to concurrent updates of deferrable tasks #65818
Description
Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.2.0
What happened and how to reproduce it?
Description
After upgrading from Airflow 3.1.7 → 3.2.0, we are consistently observing MySQL deadlocks between the scheduler and triggerer when processing deferrable tasks.
This did not occur in 3.1.7 under the same workload.
Environment
- Airflow version: 3.2.0
- Previous version (no issue): 3.1.7
- Executor: CeleryExecutor
- DB: MySQL
- Scheduler replicas: 3
- Triggerer: 1 instance
- Workload: heavy use of deferrable operators (sensors / async tasks)
Symptoms
- Scheduler/Trigger crashes or restarts due to DB deadlocks
- Deadlocks consistently involve task_instance table
- System becomes unstable under load
Example deadlock pattern:
UPDATE task_instance
SET updated_at=..., trigger_id=NULL
WHERE task_instance.state != 'deferred'
AND task_instance.trigger_id IS NOT NULL
conflicting with:
UPDATE task_instance
SET state='scheduled', trigger_id=NULL,
next_method='__fail__', next_kwargs=...
WHERE task_instance.state = 'deferred'
AND task_instance.trigger_timeout < now()
Root Cause Analysis
Key observation
In Airflow 3.2.0, both scheduler and triggerer mutate task_instance rows for deferrable tasks:
Triggerer (set-based update)
- Performs bulk UPDATE on deferred tasks that timeout
- Updates:
- state
- trigger_id
- next_method
- next_kwargs
Scheduler (callback-driven updates)
- Processes executor callbacks via:
callback = session.get(Callback, callback_id)
callback.run(session=session)
- Inside callback.run():
- Loads TaskInstance
- Mutates:
- state
- trigger_id
- other fields
Result
Two independent writers:
- Triggerer → bulk UPDATE (set-based)
- Scheduler → row-by-row ORM UPDATE
Both target overlapping task_instance rows.
Why this causes deadlocks
- Both queries scan overlapping row sets (even if predicates are logically disjoint)
- Lock acquisition order differs:
- Triggerer: index scan order
- Scheduler: callback / primary key order
- With multiple scheduler replicas, contention increases significantly
Typical pattern:
Scheduler: locks row A → waits for row B
Triggerer: locks row B → waits for row A
→ DEADLOCK
What you think should happen instead?
- Avoid concurrent writes:
- Scheduler should not mutate task_instance fields owned by triggerer
- Enforce consistent ordering:
- Ensure both components lock rows in deterministic order
- Batch updates:
- Avoid large scans or uncontrolled ORM flushes
- Ownership separation:
- Triggerer handles deferred lifecycle exclusively
- Scheduler only consumes results
Operating System
Ubuntu 22.04.5 LTS
Deployment
None
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
No response
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Reacted by Mat, Brian Dolhansky, Shoma Okamoto and SeonghwanLee
Activity
boring-cyborg commentedon Apr 24, 2026
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
Fix scheduler/triggerer deadlock on task_instance for deferrable tasks
shaealh commentedon Apr 27, 2026
Opened this PR
uplsh580 commentedon May 23, 2026
We are seeing the same class of deadlock on Airflow 3.1.8 / MySQL with 2 triggerer replicas.
In our case the deadlock happens on a single-row UPDATE through the
Trigger.submit_event→handle_event_submitpath (not the bulk UPDATE paths inSchedulerJobRunner.check_trigger_timeouts/Trigger.clean_unusedthat #65920 / #65836 target). The exception is not retried anywhere in the call chain (triggerer_job_runner.handle_events→Trigger.submit_event→handle_event_submit), so it propagates up toTriggerRunnerSupervisor.runand the triggerer process exits, after which Kubernetes restarts the pod. No task failure — deferred tasks are picked up by the other triggerer — but the restart shows up as an alert.Environment
Failing statement
UPDATE task_instance
SET state='scheduled', scheduled_dttm=..., updated_at=..., trigger_id=NULL, next_kwargs=...
WHERE task_instance.id = '019e511c-c507-74d8-ace0-f4c396cc8ef1'
→
MySQLdb.OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction')Stack
airflow/jobs/triggerer_job_runner.py:534 run
airflow/jobs/triggerer_job_runner.py:561 handle_events
airflow/models/trigger.py:252 submit_event
airflow/models/trigger.py:422 handle_event_submit
→ session.flush()
Looking at
airflow-core/src/airflow/models/trigger.pyon the 3.1.8 tag, neithersubmit_event(L239) norhandle_event_submit(L394 / L426) carries@retry_db_transactionor any try/except, while several deadlock-sensitive paths on the scheduler side already do. Wrapping these entry points (or the per-event loop intriggerer_job_runner.handle_events) with@retry_db_transactionwould also cover this single-row UPDATE case that the existing PRs do not address.fedemgp commentedon Jun 29, 2026
is it possible that this issue will be solved after 3.2.2? due to this changes: #66412
uplsh580 commentedon Aug 7, 2026
Following up on my earlier comment: we hit the same deadlock again on 2026-08-06, now on Airflow 3.2.2 — same single-row UPDATE through the
Trigger.submit_event→handle_event_submitpath, and the triggerer process died, causing a container restart (the pod itself stayed up).@fedemgp to answer your question: no — I think 3.2.2 does not fix this. #66412 addresses the comms-channel race inside the triggerer (lock-based serialization → response multiplexing), which is unrelated to this MySQL row-lock deadlock on
task_instance. We can confirm it still reproduces on 3.2.2.Environment
Failing statement
UPDATE task_instance
SET state='scheduled', scheduled_dttm=..., updated_at=..., trigger_id=NULL, next_kwargs=...
WHERE task_instance.id = '.....'
→ MySQLdb.OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction')
Stack (3.2.2)
airflow/jobs/triggerer_job_runner.py:218 _execute
airflow/jobs/triggerer_job_runner.py:612 run
airflow/jobs/triggerer_job_runner.py:655 handle_events
airflow/models/trigger.py:274 submit_event
airflow/models/trigger.py:505 handle_event_submit
→ session.flush()
One additional detail compared to our previous occuer process did not terminate on SIGINT/SIGTERM afterthe exception and had to be SIGKILLed by the supervisor (
Process exited exit_code=-9 signal_sent=SIGKILL), so triggers thstill running at that point were interrupted mid-fle other replica picked them up), but the restart isnoisy at the alerting level.On 3.2.2 the call chain (handle_events→submit_mit) still has no@retry_db_transactionorequivalent, so a single transient 1213 kills the whix PRs for the bulk-UPDATE paths (#65836, #65920)have gone stale, I'd propose wrapping this per-evennsaction` (matching the existing scheduler-sidepattern) as an incremental fix — it wouldn't removef, but it would stop a single transient deadlock from taking down the triggerer. I'll put up a PR along tRetry transient deadlocks in Trigger.submit_event/submit_failure
Retry transient deadlocks in Trigger.submit_event/submit_failure
Vamsi-klu commentedon Aug 27, 2026