Repository navigation
[Bug]: A start report handled after the agent's completion can set the completed agent job back to running #1376
Description
Activity
Now observed in the field, on two instances (PostgreSQL, main at 5e15222, agent 0.1.20). The issue text says "Not observed in the field so far"; that no longer holds.
Between 10-08 00:00Z and 10-09 12:00Z the two instances had 22 failed agent jobs, all with the same signature:
status = failed,error_message = "Reaped by server: no agent activity for over 15 minutes (agent session lost; job orphaned)."- but
resultholds the agent's real outcome withreturn_code: 0(stdout included) - the server log has
POST /api/agents/jobs/<id>/completereceived before the job'sstarted_at, e.g./completereceived at 05:45:10.49, 200 at 05:45:10.90,started_at05:45:10.80; the job's log lines arrived at 05:45:10.81 to .86 completed_atis the reaper's time, about 15.5 min afterstarted_at
Affected job kinds:
repository.storage_usage: about 4 % of runs (on each instance 2 to 8 of ~175 per day) since that job kind first ran; read-only and usually done in well under a secondscript.runfor a plan pre-backup hook that finishes in ~0.1 s: once per instance. The plan run waited for the hook until the reaper ended it, then went on (hook has continue on error) and closedcompleted_with_warningsabout 16 min late. Without continue on error, the plan would have stopped before the backup.
So the window is hit regularly by any job that is shorter than the session's delivery lag, not only at unit level.
- added a commit that references this issue
on Oct 9, 2026
Summary
An agent sends its start report over the session and its outcome over REST (
/complete), so the two can reach the server in either order._mark_agent_job_started(app/api/agents.py) checksjob.status in FINAL_AGENT_JOB_STATUSESon the job as its session loaded it, then writesjob.status = "running"as a plain attribute. If the completion (recorded in a worker thread with its own session since #1324) commits after that load, the start report's commit sets thecompletedjob back torunning.On PostgreSQL the window is wide: the report's guarded
start_notified_atUPDATE waits for the completion's lock on the job row, so the report has always loaded the job before the completion committed.Found while fixing #1366, which covers the same race for the linked operation: with that fix the operation keeps its verdict, but the agent job's own status is not covered there.
Steps to reproduce (unit level)
claimed,started_atandstart_notified_atunset._record_job_completion(its own session, commitscompleted)._mark_agent_job_startedwith the job loaded in step 2 and commit.The job ends
running; the linked operation stayscompleted.Actual behavior
The agent job is
runningafter a successful completion. A waiter (wait_for_agent_repository_operation_job) keeps polling until its timeout, and admission holds the repository for the job until the stale-job reaper ends it.Expected behavior
A start report never moves a job out of a final status: the transition to
runningis a guarded write (WHERE status NOT IN FINAL_AGENT_JOB_STATUSES), like the terminal transition in_claim_terminal_transition.Borg UI version
main at 196a74a
Additional context
It needs the first start report to reach the server after the REST completion, i.e. a job shorter than the session's delivery lag, so it is rarer than #1366. Not observed in the field so far.