Fix db migrate failure to Airflow 2.9.2 under PyMySQL driver by Aaryan123456679 · Pull Request #70235 · apache/airflow

Aaryan123456679 · 2026-07-22T12:35:41Z

Migrating the metadata database to Airflow 2.9.2 (revision 686269002441) raised pymysql.err.ProgrammingError on MySQL when using the PyMySQL driver, because the migration used MySQL's PREPARE/EXECUTE/DEALLOCATE PREPARE dynamic SQL to conditionally drop unique constraints on the connection and dag_run tables. This syntax is not supported by PyMySQL (see PyMySQL#202), so any deployment using PyMySQL as its MySQL driver could not migrate past this revision.

Replaced the dynamic SQL with a plain lookup of existing constraints from information_schema.TABLE_CONSTRAINTS, followed by conditional ALTER TABLE ... DROP INDEX statements — this works with any MySQL driver.

Verified against a real MySQL backend forced to use mysql+pymysql://: downgrading to bff083ad727d and re-running airflow db migrate now completes without error, and the expected unique constraints (connection_conn_id_uq, dag_run_dag_id_*_key) are present afterward.

closes: #43690

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Sonnet 5)

Generated-by: Claude Code (Sonnet 5) following the guidelines

Aaryan123456679 · 2026-07-22T12:36:09Z

Added a newsfragment for the release notes.

1fanwang

LGTM overall, thanks for addressing this issue.
Verified on MySQL 8.4 the original dynamic SQL works under mysqlclient and fails under PyMySQL.

Upgrading through revision 686269002441 raised pymysql.err.ProgrammingError because the migration used MySQL's PREPARE/EXECUTE/DEALLOCATE PREPARE dynamic SQL to conditionally drop unique constraints, which PyMySQL's driver does not support (see PyMySQL/PyMySQL#202). Replaced with a plain lookup of existing constraints from information_schema.TABLE_CONSTRAINTS followed by conditional ALTER TABLE ... DROP INDEX statements, which works with any MySQL driver. Verified against a real MySQL/PyMySQL backend: downgrading to bff083ad727d and re-running db migrate now completes without error. closes: apache#43690

The PyMySQL online-mode fix in this migration queries information_schema.TABLE_CONSTRAINTS via conn.execute() to decide which legacy unique constraints to drop. In Alembic's offline (--sql) mode there is no live connection, so conn.execute() returns None and iterating over it raises TypeError, breaking the CI "Test offline migration" step. Guard the query-then-conditionally-drop logic behind context.is_offline_mode(), falling back to the original dynamic prepare/execute/deallocate SQL text for the offline branch, since that text is only ever emitted as script output there rather than executed directly.

Bind the table name as a query parameter in the information_schema lookup instead of interpolating it into the SQL string, since it's a value comparison rather than an identifier — matches Airflow's usual parameter-binding convention even though every call site currently passes a hardcoded literal. Also correct the docstring: PyMySQL can run a bare PREPARE statement fine, what it actually can't do is send more than one statement per cursor.execute() call, because it never enables CLIENT.MULTI_STATEMENTS.

uranusjr

I think we can call SELECT ... FROM information_schema.TABLE_CONSTRAINTS less in both upgrade in downgrade. But this looks good to me in general. (Five-ish extra calls in a migration won’t destroy the world.)

1fanwang

thank you! I hit the same bug while doing airflow 2->3 upgrade in our infra and this is the same approach I took
stamping my approval again but you'd need committer/PMC who has approval/merge access

github-actions · 2026-09-07T08:31:00Z

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-3-test

…ver (#70235) (#72613) * Fix db migrate failure to Airflow 2.9.2 under PyMySQL driver Upgrading through revision 686269002441 raised pymysql.err.ProgrammingError because the migration used MySQL's PREPARE/EXECUTE/DEALLOCATE PREPARE dynamic SQL to conditionally drop unique constraints, which PyMySQL's driver does not support (see PyMySQL/PyMySQL#202). Replaced with a plain lookup of existing constraints from information_schema.TABLE_CONSTRAINTS followed by conditional ALTER TABLE ... DROP INDEX statements, which works with any MySQL driver. Verified against a real MySQL/PyMySQL backend: downgrading to bff083ad727d and re-running db migrate now completes without error. closes: #43690 * Add newsfragment for MySQL PyMySQL migration fix * Fix offline migration for 2.9.2 ORM-consistency revision The PyMySQL online-mode fix in this migration queries information_schema.TABLE_CONSTRAINTS via conn.execute() to decide which legacy unique constraints to drop. In Alembic's offline (--sql) mode there is no live connection, so conn.execute() returns None and iterating over it raises TypeError, breaking the CI "Test offline migration" step. Guard the query-then-conditionally-drop logic behind context.is_offline_mode(), falling back to the original dynamic prepare/execute/deallocate SQL text for the offline branch, since that text is only ever emitted as script output there rather than executed directly. * Address review feedback on MySQL PyMySQL migration fix Bind the table name as a query parameter in the information_schema lookup instead of interpolating it into the SQL string, since it's a value comparison rather than an identifier — matches Airflow's usual parameter-binding convention even though every call site currently passes a hardcoded literal. Also correct the docstring: PyMySQL can run a bare PREPARE statement fine, what it actually can't do is send more than one statement per cursor.execute() call, because it never enables CLIENT.MULTI_STATEMENTS. * Fixed the PR comments (cherry picked from commit 08e428c) Co-authored-by: Aaryan Mahajan <aaryanmhjn@gmail.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

…70235) * Fix db migrate failure to Airflow 2.9.2 under PyMySQL driver Upgrading through revision 686269002441 raised pymysql.err.ProgrammingError because the migration used MySQL's PREPARE/EXECUTE/DEALLOCATE PREPARE dynamic SQL to conditionally drop unique constraints, which PyMySQL's driver does not support (see PyMySQL/PyMySQL#202). Replaced with a plain lookup of existing constraints from information_schema.TABLE_CONSTRAINTS followed by conditional ALTER TABLE ... DROP INDEX statements, which works with any MySQL driver. Verified against a real MySQL/PyMySQL backend: downgrading to bff083ad727d and re-running db migrate now completes without error. closes: apache#43690 * Add newsfragment for MySQL PyMySQL migration fix * Fix offline migration for 2.9.2 ORM-consistency revision The PyMySQL online-mode fix in this migration queries information_schema.TABLE_CONSTRAINTS via conn.execute() to decide which legacy unique constraints to drop. In Alembic's offline (--sql) mode there is no live connection, so conn.execute() returns None and iterating over it raises TypeError, breaking the CI "Test offline migration" step. Guard the query-then-conditionally-drop logic behind context.is_offline_mode(), falling back to the original dynamic prepare/execute/deallocate SQL text for the offline branch, since that text is only ever emitted as script output there rather than executed directly. * Address review feedback on MySQL PyMySQL migration fix Bind the table name as a query parameter in the information_schema lookup instead of interpolating it into the SQL string, since it's a value comparison rather than an identifier — matches Airflow's usual parameter-binding convention even though every call site currently passes a hardcoded literal. Also correct the docstring: PyMySQL can run a bare PREPARE statement fine, what it actually can't do is send more than one statement per cursor.execute() call, because it never enables CLIENT.MULTI_STATEMENTS. * Fixed the PR comments

Aaryan123456679 requested a review from ephraimbuddy as a code owner July 22, 2026 12:35

boring-cyborg Bot added the area:db-migrations PRs with DB migration label Jul 22, 2026

Aaryan123456679 force-pushed the fix/mysql-pymysql-migration-43690 branch from 977c6d0 to 3caee08 Compare July 22, 2026 18:47

1fanwang mentioned this pull request Aug 6, 2026

Fix MySQL metadata database failing when only PyMySQL is installed #71251

Merged

1fanwang approved these changes Aug 6, 2026

View reviewed changes

Comment thread .../airflow/migrations/versions/0017_2_9_2_fix_inconsistency_between_ORM_and_migration_files.py Resolved

Comment thread .../airflow/migrations/versions/0017_2_9_2_fix_inconsistency_between_ORM_and_migration_files.py Resolved

1fanwang mentioned this pull request Aug 6, 2026

Fix task_instance_history downgrade failing under the PyMySQL driver #71258

Closed

Aaryan123456679 added 4 commits August 9, 2026 18:53

Add newsfragment for MySQL PyMySQL migration fix

6560d15

Aaryan123456679 force-pushed the fix/mysql-pymysql-migration-43690 branch from 3caee08 to e792ca7 Compare August 9, 2026 13:24

uranusjr approved these changes Aug 9, 2026

View reviewed changes

eladkal reviewed Aug 10, 2026

View reviewed changes

Comment thread airflow-core/newsfragments/70235.bugfix.rst Resolved

eladkal added type:bug-fix Changelog: Bug Fixes backport-to-v3-3-test labels Aug 10, 2026

eladkal added this to the Airflow 3.3.2 milestone Aug 10, 2026

Fixed the PR comments

2227105

Aaryan123456679 requested review from 1fanwang, eladkal and uranusjr August 20, 2026 14:05

1fanwang approved these changes Aug 21, 2026

View reviewed changes

vatsrahul1001 approved these changes Sep 7, 2026

View reviewed changes

vatsrahul1001 merged commit 08e428c into apache:main Sep 7, 2026
78 checks passed

Navigation Menu

Repository navigation

Fix db migrate failure to Airflow 2.9.2 under PyMySQL driver - #70235

Fix db migrate failure to Airflow 2.9.2 under PyMySQL driver#70235
vatsrahul1001 merged 5 commits into
apache:mainfrom
Aaryan123456679:fix/mysql-pymysql-migration-43690

Aaryan123456679 commented Jul 22, 2026

Uh oh!

Aaryan123456679 commented Jul 22, 2026

Uh oh!

1fanwang left a comment

Uh oh!

Uh oh!

Uh oh!

uranusjr left a comment

Uh oh!

Uh oh!

1fanwang left a comment

Uh oh!

Uh oh!

github-actions Bot commented Sep 7, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Aaryan123456679 commented Jul 22, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

Aaryan123456679 commented Jul 22, 2026

Uh oh!

1fanwang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

uranusjr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

1fanwang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Sep 7, 2026

Backport successfully created: v3-3-test

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants