Skip to content

fix(worker): tolerate transient DB disconnects during task routing lookups - #2576

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-routing-db-disconnect-tolerance
Open

sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-routing-db-disconnect-tolerance

Conversation

@sentry

@sentry sentry Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

When a Celery chord completes (for example at the end of an upload processing batch), Celery's Redis backend runs on_chord_part_return and executes callback.delay(...). In worker tasks, this calls BaseCodecovTask.apply_async(...), which runs synchronous SQLAlchemy queries against Postgres to look up the owner plan (_get_user_plan_from_task) and owner ID (_get_ownerid_from_task) for task queue routing.

During a transient Postgres connection drop or server restart, these routing lookups raised OperationalError: server closed the connection unexpectedly. Because the exception occurred inside Celery's chord callback handler, the chord was marked failed and the callback task (e.g. UploadFinisher) was never enqueued, leaving the upload stranded.

Fix

  • Added _safe_routing_lookup in apps/worker/celery_task_router.py to wrap routing queries called by _get_user_plan_from_task and _get_ownerid_from_task.
  • When a DBAPIError indicates connection_invalidated, the session is rolled back to discard the dead pooled connection and the query is retried once on a fresh connection.
  • If the second attempt also fails, a warning is logged with task context and fallback defaults are returned (DEFAULT_FREE_PLAN and None), allowing the task to still be enqueued rather than dropped entirely.
  • Non-disconnect exceptions are re-raised unchanged.

Legal Boilerplate

Look, I get it. The entity doing business as "Codecov" is owned by Harness, Inc. In 2026 Harness acquired Codecov and as a result Harness is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Harness can use, modify, copy, and redistribute my contributions, under Harness's choice of terms.

Fixes WORKER-ZAY

@sentry <feedback>: Autofix iterates on these changes
@sentry stop iterating: Autofix stops iterating on this run

This PR was automatically generated by Sentry. You can adjust this setting at any time.


Note

Medium Risk
Changes enqueue-time routing for many worker tasks; repeated DB failures can enqueue work with default plan/owner routing instead of failing the chord callback.

Overview
Adds _safe_routing_lookup around the Postgres queries used when Celery routes tasks in apply_async (including chord callbacks such as UploadFinisher).

On DBAPIError with connection_invalidated, the session is rolled back, the lookup is retried once, and if it still fails the router logs a warning and returns safe defaults instead of raising—DEFAULT_FREE_PLAN for plan lookups and None for owner ID lookups. Other DB errors are unchanged.

_get_ownerid_from_task and _get_user_plan_from_task now route through this helper instead of calling lookup functions directly (with explicit early returns when no lookup is registered).

Reviewed by Cursor Bugbot for commit ccc85b0. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 56.66667% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.62%. Comparing base (585ddae) to head (ccc85b0).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
apps/worker/celery_task_router.py 56.66% 13 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2576      +/-   ##
==========================================
- Coverage   91.64%   91.62%   -0.03%     
==========================================
  Files        1341     1341              
  Lines       53339    53365      +26     
  Branches     1649     1649              
==========================================
+ Hits        48885    48898      +13     
- Misses       4133     4146      +13     
  Partials      321      321              
Flag Coverage Δ
workerintegration 58.47% <56.66%> (-0.02%) ⬇️
workerunit 90.48% <56.66%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@sentry

sentry Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

⚠️ Sentry needs additional GitHub App permissions

The Sentry GitHub App installation for this repository is missing permissions it needs to keep iterating on this pull request to get CI passing.

Review and accept the updated permissions to let Seer continue: https://github.com/organizations/codecov/settings/installations/86101127/permissions/update

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants