Repository navigation
fix(worker): tolerate transient DB disconnects during task routing lookups - #2576
sentry[bot] wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2576 +/- ##
==========================================
- Coverage 91.64% 91.62% -0.03%
==========================================
Files 1341 1341
Lines 53339 53365 +26
Branches 1649 1649
==========================================
+ Hits 48885 48898 +13
- Misses 4133 4146 +13
Partials 321 321
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
|
The Sentry GitHub App installation for this repository is missing permissions it needs to keep iterating on this pull request to get CI passing. Review and accept the updated permissions to let Seer continue: https://github.com/organizations/codecov/settings/installations/86101127/permissions/update |
Problem
When a Celery chord completes (for example at the end of an upload processing batch), Celery's Redis backend runs
on_chord_part_returnand executescallback.delay(...). In worker tasks, this callsBaseCodecovTask.apply_async(...), which runs synchronous SQLAlchemy queries against Postgres to look up the owner plan (_get_user_plan_from_task) and owner ID (_get_ownerid_from_task) for task queue routing.During a transient Postgres connection drop or server restart, these routing lookups raised
OperationalError: server closed the connection unexpectedly. Because the exception occurred inside Celery's chord callback handler, the chord was marked failed and the callback task (e.g.UploadFinisher) was never enqueued, leaving the upload stranded.Fix
_safe_routing_lookupinapps/worker/celery_task_router.pyto wrap routing queries called by_get_user_plan_from_taskand_get_ownerid_from_task.DBAPIErrorindicatesconnection_invalidated, the session is rolled back to discard the dead pooled connection and the query is retried once on a fresh connection.DEFAULT_FREE_PLANandNone), allowing the task to still be enqueued rather than dropped entirely.Legal Boilerplate
Look, I get it. The entity doing business as "Codecov" is owned by Harness, Inc. In 2026 Harness acquired Codecov and as a result Harness is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Harness can use, modify, copy, and redistribute my contributions, under Harness's choice of terms.
Fixes WORKER-ZAY
@sentry <feedback>: Autofix iterates on these changes@sentry stop iterating: Autofix stops iterating on this runThis PR was automatically generated by Sentry. You can adjust this setting at any time.
Note
Medium Risk
Changes enqueue-time routing for many worker tasks; repeated DB failures can enqueue work with default plan/owner routing instead of failing the chord callback.
Overview
Adds
_safe_routing_lookuparound the Postgres queries used when Celery routes tasks inapply_async(including chord callbacks such asUploadFinisher).On
DBAPIErrorwithconnection_invalidated, the session is rolled back, the lookup is retried once, and if it still fails the router logs a warning and returns safe defaults instead of raising—DEFAULT_FREE_PLANfor plan lookups andNonefor owner ID lookups. Other DB errors are unchanged._get_ownerid_from_taskand_get_user_plan_from_tasknow route through this helper instead of calling lookup functions directly (with explicit early returns when no lookup is registered).Reviewed by Cursor Bugbot for commit ccc85b0. Bugbot is set up for automated code reviews on this repo. Configure here.