Skip to content

fix(worker): rollback DB session on failed log context population - #2574

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-log-context-pending-rollback
Open

sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-log-context-pending-rollback

Conversation

@sentry

@sentry sentry Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

In BaseCodecovTask.run, before executing run_impl, LogContext.populate_from_sqlalchemy queries the database for repository and owner metadata to enrich task logs. If the database connection drops during this query (e.g. psycopg2.OperationalError: server closed the connection unexpectedly), the exception was caught and logged with log.exception("Failed to populate log context"), but the db_session was left with an invalidated transaction.

Because the thread-scoped session is reused across the task, the task's first database query in run_impl (such as CommitUpdate's commits.first()) would subsequently fail with sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.

Changes

  • In apps/worker/helpers/log_context.py, add a guarded dbsession.rollback() within the except block of populate_from_sqlalchemy so any invalidated transaction state is cleared before run_impl runs.
  • Safely handle query.first() returning None before tuple unpacking when a repository or owner record is missing, avoiding unnecessary TypeErrors during log population.

Legal Boilerplate

Look, I get it. The entity doing business as "Codecov" is owned by Harness, Inc. In 2026 Harness acquired Codecov and as a result Harness is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Harness can use, modify, copy, and redistribute my contributions, under Harness's choice of terms.

Fixes WORKER-ZAW

@sentry <feedback>: Autofix iterates on these changes
@sentry stop iterating: Autofix stops iterating on this run

This PR was automatically generated by Sentry. You can adjust this setting at any time.


Note

Low Risk
Narrow change to error handling in optional log enrichment; rollback on failure is the intended fix for poisoned sessions.

Overview
Fixes worker tasks failing with PendingRollbackError when optional log-context DB lookups error out but the shared SQLAlchemy session is still used for real work.

In populate_from_sqlalchemy, failed queries (e.g. dropped connections) now trigger a guarded dbsession.rollback() so the thread-scoped session is usable again before run_impl. Repository/owner lookups also skip tuple unpacking when query.first() returns None, avoiding extra TypeErrors during enrichment.

Reviewed by Cursor Bugbot for commit ae7d646. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.64%. Comparing base (585ddae) to head (ae7d646).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
apps/worker/helpers/log_context.py 80.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2574      +/-   ##
==========================================
- Coverage   91.64%   91.64%   -0.01%     
==========================================
  Files        1341     1341              
  Lines       53339    53347       +8     
  Branches     1649     1649              
==========================================
+ Hits        48885    48891       +6     
- Misses       4133     4135       +2     
  Partials      321      321              
Flag Coverage Δ
workerintegration 58.47% <30.00%> (-0.02%) ⬇️
workerunit 90.54% <80.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@sentry

sentry Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

⚠️ Sentry needs additional GitHub App permissions

The Sentry GitHub App installation for this repository is missing permissions it needs to keep iterating on this pull request to get CI passing.

Review and accept the updated permissions to let Seer continue: https://github.com/organizations/codecov/settings/installations/86101127/permissions/update

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants