Skip to content

Fix failed jobs polling forever: check for 'error' status, not 'failed' - #159

Open
matthewberry wants to merge 5 commits into
mainfrom
mjberry/158-error-status
Open

matthewberry wants to merge 5 commits into
mainfrom
mjberry/158-error-status

Conversation

@matthewberry

@matthewberry matthewberry commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #158.

Problem

The backend reports a failed job with phase 'error' (see JobStatus in the generated client), but the results page checked for 'failed'. When a job failed, the page kept polling GET /clean/jobs/{id} every 10 seconds and never showed the failure message.

Change

  • In results.component.ts, the two status checks (stop polling, set failedJob) now compare against JobStatus.Error instead of 'failed'. The 'completed' check now uses JobStatus.Completed. Completed jobs are handled the same as before.
  • PollingResponseStatus.status is now typed JobStatus instead of string. The results component is the only code that uses this type. With this type, a comparison against 'failed' doesn't compile (TS2367).

Testing

Added tests in results.component.spec.ts. They use a stubbed ResultService whose polling stream ends when gotEndResult() is called, like the real service:

  • error job: polling stops, failedJob is set, results aren't fetched, and .failJob renders
  • completed job: polling stops, results are fetched, and there's no failure message
  • processing job: polling continues

npx ng test --watch=false --browsers=ChromeHeadless: 9 of 9 pass on this branch, which is rebased onto main after #160. With the old 'failed' checks restored, the error-job test fails. ng build succeeds. I didn't test against a real failed job in a browser.

The backend reports failed jobs with phase 'error', but the results page
checked for 'failed', so failed jobs polled forever and never showed the
failure message.
@matthewberry
matthewberry force-pushed the mjberry/158-error-status branch from 945b2fd to cece705 Compare October 28, 2026 03:38
- Check for a terminal job status before parsing job_info, so a malformed
  or missing job_info on an errored job can't keep polling alive.
- Wait 10s before retrying a failed status request instead of retrying
  immediately in a tight loop.
- Type PollingResponseResult.status as JobStatus and replace the remaining
  hand-typed status strings.
- Drop the unneeded 'as Job' cast in the results spec.
- ResultService: cap status retries at 5 consecutive failures so errors
  reach the subscriber instead of retrying forever.
- ResultsComponent: show the failure message when status polling errors,
  stop polling and show it when the job id matches no job, and skip the
  sequence count when job_info is missing instead of throwing every poll.
- ResultsComponent: read the job id from ActivatedRoute.paramMap instead of
  window.location, reloading on each id change since the router reuses the
  component between /results/:id URLs. Specs no longer depend on the Karma
  page URL.
@matthewberry

Copy link
Copy Markdown
Collaborator Author

Review follow-up, pushed as 0554a39. Each fix below has a spec that failed before the change and passes after it. Full suite: 17/17 passing.

Reproduced and fixed

  • Status errors never reached the UI. retry({ delay: 10000 }) had no count limit, so a 404/403/500 from the status endpoint kept the loader spinning forever. Retries are now capped at 5 consecutive failures (resetOnSuccess: true, so occasional errors during a long job don't add up). The component now shows the failure message when polling errors.
  • Missing job_info threw on every poll. For a queued or processing job, JSON.parse(job_info!) threw inside the next handler every 10s. It is now skipped when job_info is absent. The existing spec "stops polling for an errored job even when job_info is missing" now expects no unhandled error.
  • Unknown job id polled forever. When the job lookup returns [], polling now stops and the failure message is shown.
  • Job id read from window.location. The component now reads it from ActivatedRoute.paramMap and reloads when the id changes. The router reuses the component between /results/A and /results/B, and before this change ngOnInit never ran again for the second job. That navigation can't happen in the UI today, because the only links to /results/:id come from the configuration page. The new spec guards it anyway. This change also means the specs no longer depend on the Karma page URL.

Checked, not a bug

  • The concern that polling continues after leaving the page: ngOnDestroy already unsubscribes. A spec now guards that.

Not addressed

  • status: firstMatch?.phase! still hides that Job.phase is optional. A job with no phase keeps polling.
  • For an errored job with no job_info, the template still shows "Valid No. of Sequences: undefined Sequence(s)".

Open questions

  • Can the backend briefly return [] for a job that was just submitted? If so, treating [] as "not found" will wrongly fail real jobs, and it should allow a few empty responses first.
  • Is a cap of 5 retries (about 50s) right?
  • /results with no id now passes undefined as the job id. Previously it polled for a job named results. Either way the page ends up showing the failure message.

Job.phase is optional in the generated client and the `!` hid that. The
backend also has a 'canceled' status the generated enum lacks, so a
canceled job polled forever. Treat every phase other than queued and
processing as final, and show the failure message unless it completed.
@matthewberry

Copy link
Copy Markdown
Collaborator Author

Follow-up on status: firstMatch?.phase!, pushed as e31ed41.

The backend (mmli-backend origin/main at 091482a) shows the ! was hiding a different problem than a missing phase:

  • job.phase is NOT NULL with a default of queued (app/models/sqlmodel/models.py), so the job list endpoint should never return a job without a phase. The client marks it optional only because the field has a default.
  • The backend's JobStatus (app/models/enums.py) also has canceled, which the generated client enum (queued | processing | completed | error) lacks. A canceled job matched neither terminal check, so the page polled every 10s forever.

Fix: any phase other than queued or processing now ends polling. The page shows the failure message unless the job completed. That covers canceled, any status added later, and a missing phase. PollingResponseStatus.status is now optional to match Job.phase, so the ! is gone. The completion handler no longer re-derives failedJob; the next handler sets it before stopping polling.

New specs cover a job with no phase and a canceled job. Both failed before the fix. Full suite: 19/19 passing, and ng build succeeds.

Two related notes:

  • A canceled job now shows the generic "Job failed possibly due to incorrect input…" message. A dedicated "canceled" message would need a product decision.
  • Regenerating the client from the current backend OpenAPI spec would add Canceled to the enum.

From the same backend read: for an unknown id, the list endpoint returns [] rather than a 404 (app/routers/job.py:232), which confirms the not-found path fixed in 0554a39 is reachable. I didn't check whether the backend commits the job row before the create request returns, so the earlier question about a brief [] right after submission is still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failed jobs poll forever and never show the failure message: results page checks for 'failed', backend says 'error'

1 participant