Repository navigation
feat: /v1 job flow with /result/:id and a queued state - #25
matthewberry wants to merge 12 commits into
Conversation
matthewberry
left a comment
There was a problem hiding this comment.
Review of the /v1 job flow. Nothing was run. Each finding comes from reading the diff, mmli-backend origin/main's /v1 code, and the installed ngx-matomo-client@8.0.0 source. Treat them as hypotheses.
What checks out:
- The OpenAPI snapshot matches
origin/main's/v1routes, operations, parameters (includingX-Notify-Email) and status enum. - The multipart body (an
inputspart plusfilesparts) matches what the backend parses. - The tests call the handlers the templates actually bind:
(ngSubmit)callsonSubmit, and(submitted)callsonResubmitted. - The PR body still says #148 isn't merged; it is.
The async entry points: there is no generation token or abort signal anywhere. Cancellation rests on switchMap and shareReplay, and that's where findings 1, 2, 3 and 5 come from.
src/app/models/job-result.ts:40: polling outlives the page.jobResultResponse$usesshareReplay(1)withoutrefCount. So once something subscribes,statusResponse$never drops to zero subscribers, and the 10s timer keeps callingGET /v1/jobs/:idafter the component is destroyed. That contradicts the docstring. Every unfinished job a user visits adds another poll that runs until the job ends.test-result.polling.spec.ts:38callsdiscardPeriodicTasks()afterfixture.destroy(), which hides this. Asserting thatgetJobis not called after destroy would fail.src/app/models/job-result.ts:23: anygetJoberror freezes the page. That includes a 404 "Unknown job" for a mistyped or deleted id from an email link, and a 502/504 during a deploy. There's nocatchErroror retry, so polling stops for good,isLoading$stays true, and the page shows "Processing. Estimated processing time…" forever.src/app/models/job-result.ts:35:getResultserrors aren't handled either. The backend returns 409JOB_FAILEDwhen a job completes with no results, and 502 when results fail the output schema. Either one leaves a full progress bar over "Processing…", and the error only reaches the console.src/app/components/test/test.component.ts:111: double-submit. There's no in-flight guard and noIdempotency-Key(which /v1 supports), and Submit is disabled only while the form is invalid. Two clicks during a slow multipart upload create two jobs and send two emails.src/app/components/test/test.component.ts:111: the submit subscription isn't tied to the component's lifetime. A response that arrives after the user has navigated away still callsrouter.navigateand pulls them to/result/<id>.src/main.ts:46:MatomoRouterModule.forRoot()becameMatomoRouterModule. In 8.0.0, the bare module only providesMATOMO_ROUTER_ENABLED=true. It never registersMatomoRouteror itsENVIRONMENT_INITIALIZER. And because core setstrackAppInitialLoad = !MATOMO_ROUTER_ENABLED, even the initial page load isn't tracked. Result: no page views at all (ngx-matomo-client-router.mjs:364-435,core.mjs:96). This change looks unrelated to the PR.src/app/components/test-result/test-result.component.ts:47: no check that the job belongs to this tool.job.toolisn't compared withAPP_IDENTITY.jobType, andjob.inputsis cast straight toToolInput. /v1 job ids are global, so/result/<another tool's id>throws intoFastawhen the user opens the Input tab.src/app/components/loading/loading.component.html:9: queued jobs briefly show as processing.statusis undefined until the first poll returns, so the template falls through to the processing branch. A queued job shows the processing estimate before switching to "Queued".src/app/components/test/test.component.ts:114: the consent checkbox is ignored.X-Notify-Emailis sent whenever the email field is filled.agreeToSubscriptionis never read..github/workflows/e2e.yml:32: CI only checks the schema types. Nothing checkssrc/app/api/v1against the OpenAPI snapshot. Separately, the hand-writtenJobinv1-job.tsleaves outprovenanceandlinks, which_job_documentreturns.
- src/app/api/v1: typescript-angular client generated from mmli-backend's /v1 OpenAPI (snapshot from PR #148, unmerged). openapi-generator 7.24.0 via openapitools.v1.json: 6.6.0 types FastAPI's OpenAPI 3.1 parameters as any. - npm run schemas / schemas:sync: ToolInput and ToolResults generated from the tool's input and output JSON Schemas (json-schema-to-typescript 16.0.0); CI checks they are current. The template's schemas are job-template's sample.
- The form posts the input document and any files as multipart, with the notification address in X-Notify-Email, and shows the problem document when the backend refuses the input. - Loading tells queued (waiting for a node) from processing, and shows error and canceled states. - The results page moves from test/result/:id to result/:id, the route the backend's default notification email links to. - app.yaml's job_type reaches the app as APP_IDENTITY.jobType. The template's is job-template-sample, which the backend has not registered yet. - e2e: submit -> queued -> processing -> results against a mock /v1 API. nightwatch/tsconfig targets es2017: compiled to es5, async tests are not detected as async and their awaited commands do not wait. Part of #17
- jobResultResponse$'s shareReplay had no refCount, so it kept the status poll subscribed after the page was gone (rxjs 7.8 defaults refCount to false). The new test destroys the page and asserts no further getJob calls, without discardPeriodicTasks. - A getJob or getResults error (404 unknown job, 409 JOB_FAILED, 502 invalid results) left the page on "Processing" forever. Transient errors (network, 502/503/504) are retried twice; anything else shows the problem document. - A job from another tool (/v1 ids are global) shows a problem instead of rendering inputs this page can't read.
…ter leaving Submit is disabled and shows a spinner until the backend answers, and is enabled again after a refusal. The submit subscription ends with the component, so a late response no longer pulls the user to /result/<id>.
…ce and links to Job FILES is the generator's own listing after a rerun, which is stable; the first generation also listed .openapi-generator-ignore.
5658da1 to
764b906
Compare
|
Responses to the review. The branch is rebased onto
Verification:
|
matthewberry
left a comment
There was a problem hiding this comment.
Re-review at head 764b906 (rebased on af0989e). I read the dispositions comment and the code; nothing was run.
The prior findings, checked against the code:
- 1 (fixed): both streams now use
shareReplay({bufferSize: 1, refCount: true}). The new destroy test has nodiscardPeriodicTasks(), and a leftover poll would fail its call-count assertion. - 4 (fixed): the double-click guard and the
[disabled]binding are in place. The test dispatchessubmiton the<form>, so it goes through(ngSubmit), the binding the template uses. - 5 (fixed):
takeUntilDestroyed. See new finding 2 for a side effect. - 6 (fixed):
src/main.ts:46is back toMatomoRouterModule.forRoot(). - 7 (fixed): the tool check runs before the
loadedJobcast. - 8 (fixed): the queued and processing messages are each gated on their own status.
- 10 (fixed): CI diffs
src/app/api/v1after regenerating it.provenanceandlinksmatch_job_documenton mmli-backendorigin/main. - 2 and 3 (fixed except for retries): a 404, a tool mismatch, and
getResults409/502 now reachloadProblem$and the error UI. The retry gaps are new findings 1 and 3. - 9 (declined as a product question): noted.
- Async entry points: each one (the poll tick,
getResults, submit, the unchanged-request navigate) has a cancellation path. The only readers ofloadProblem$andsubmittingare the result template,onSubmitand the button, and each is consistent with every write site.
New findings:
src/app/models/job-result.ts:44: the transient retry can never finish. The retry (count 2, 5 s apart) sits insideswitchMapon the 10 s timer. Retry 2 is due at about 10 s plus twice the request latency, but the tick at 10 s switches it out and starts over with a fresh count. So a lasting 0/502/503/504 never reachesfail(), and the page shows "Processing" forever, which is the original finding 2. The only test fails once and then succeeds, so it can't catch this. Moving the retry outsideswitchMap, or counting consecutive failures across ticks, would fix it.src/app/components/test/test.component.ts:126:takeUntilDestroyedaborts a POST the server may already have accepted. On/result/A, a user edits the inputs, submits, then clicks the Results tab before the upload finishes. The@ifdestroys<app-test>and the XHR is cancelled. If the backend already got the body, job B exists and its email goes out, but the user is never sent to it. With no Idempotency-Key, resubmitting creates a third job.src/app/models/job-result.ts:67: any 502 fromgetResultsis treated as final. 502 is both the backend's "Job produced invalid results" and a gateway error during a deploy. A job that completes mid-redeploy shows a permanent error, though the same outage ongetJobwould be retried. The problemtype(JOB_FAILED / invalid-results) is what tells the two apart, not the status code.src/app/components/test/test.component.ts:124: the double-submit guard is incomplete.finalizeclearssubmittingwhen the POST completes, beforerouter.navigateresolves, so the button re-enables in that window. With no Idempotency-Key, retries and refreshes during a slow upload still create duplicates. This is the second half of the original finding 4.src/app/services/test.examples.ts:10:EXAMPLE_JOB.toolis hard-coded tojob-template-sample. A team that setsjob_type: my-toolinapp.yamlthen sees "This job belongs to another tool" on every/resultpage in frontend-only mode. UsingAPP_IDENTITY.jobTypethere would fix it.src/app/components/loading/loading.component.ts:73: the error and canceled screens appear only after the firstinterval(1000)emission. They're also read via.valuerather thanasync. For a job that already failed, the page shows an empty bar with no message for a second. The specs tick 1000 ms before checking, which hides this.src/app/components/test-result/test-result.polling.spec.ts:38: the first polling test still callsdiscardPeriodicTasks()afterfixture.destroy(). WithrefCountin place the call does nothing. IfrefCountwere removed, this test would still pass, so delete the call.
- The transient retry sat inside switchMap on the 10 s tick, so every tick reset its count and a lasting outage never reached the error. Consecutive transient failures are now counted across ticks (3, about 30 s). - Transient means 0/502/503/504 WITHOUT a problem document. The backend's own 502 (invalid results.json, upstream-failure) carries one and is final; a gateway's doesn't. getResults now retries the latter twice. - The first polling test no longer calls discardPeriodicTasks(). It did nothing: fakeAsync in zone.js 0.15 defaults to flush: true, which flushes leftover periodic timers instead of failing, so the call count is the leak check.
…through navigation takeUntilDestroyed cancelled an upload the backend may already have accepted, hiding the job it created. The request now completes, and only the navigation is skipped once the form is gone. Submit stays disabled until the navigation to the new job resolves, not just until the POST does.
…s job_type The error and canceled screens were set as a side effect of the 1 s progress interval and read with .value, so they appeared a second late. They now follow the status directly. EXAMPLE_JOB.tool comes from APP_IDENTITY, so frontend-only mode isn't rejected as another tool's job once job_type changes.
|
Responses to the second pass, fixed in 9cd5ea8, c220cd7 and 77d05e6. Each fixed finding has a test that failed before its fix, except 5 (see below).
Verification:
|
matthewberry
left a comment
There was a problem hiding this comment.
Re-review at head 77d05e6. Fixes 2, 4, 5, 6 and 7 hold. The new code for 1 and 3 has a regression (finding 1 below). I read the code, plus the installed node_modules in a checkout; nothing was run.
The fixes:
- 1 (consecutive failures): the counter rises per failed tick and resets on success. The 50 s test would fail on the old code: I traced zone.js's scheduler, and the interval tick at t=10000 was queued ahead of the retry timer due at the same time. There are three gaps (findings 1–3).
- 2: the POST is no longer cancelled, and
submitted.emitand navigation are skipped after destroy. The test would fail on the old code. - 3: the backend half checks out. On mmli-backend
origin/main, every /v1 502 goes throughproblem()and always carriestype, including the legacyHTTPException(502)converted byhttp_exception_handler. /v1 sends no 503 or 504 problem documents. The client half has finding 1. - 4–6: fixed as described.
- 7: you're right, and my premise was wrong. zone.js 0.15.1 is installed, matching the lockfile. Its
fakeAsyncdefaultsflush = trueand callsflush(20, true). The "periodic timer(s) still in the queue" error is only raised whenflushis false.
New findings:
src/app/models/job-result.ts:13: status-0 failures are no longer retried. The new!error.error?.typecheck is the problem.- The app uses Angular's XHR backend (
withInterceptorsFromDi(), nowithFetch). In @angular/common 20.3.0, a network error buildsnew HttpErrorResponse({error, status: xhr.status || 0}), whereerroris theProgressEvent. Its.typeis'error', so the check treats it as final. - Status 0 is what the app sees for a Wi-Fi blip, a DNS failure, or a cross-origin gateway 502/503 that has no
Access-Control-Allow-Originheader. Now the poll ends at once with "The job could not be loaded", andgetResultsstops retrying. Before this round, status 0 was always retried. - The specs build errors with
error: null, so no test sees this. - Fix: check for a problem object with a
title, asfail()does, or skip the type check when status is 0.
- The app uses Angular's XHR backend (
src/app/models/job-result.ts:53: failures slower than the 10 s tick never reach the counter.switchMapcancels the in-flightgetJobat every tick. A hung pod, nginx's default 60 sproxy_read_timeout504, or a blackholed connection (no XHR timeout is set) never errors before it's cancelled. The count stays 0, and the page shows "Processing" forever. Atimeout(...)shorter than the tick inside theswitchMap, orexhaustMap, would make these count.src/app/models/job-result.ts:48:transientFailuresis an instance field and isn't reset whenrefCountrestarts the poll. After two failures, opening the Input tab removes every subscriber. Coming back starts a freshtimer(0, …)with the count still at 2, so the next failure shows the error page. Fix: reset it per subscription, e.g.defer(() => { this.transientFailures = 0; return timer(0, 10000); }).src/app/models/job-result.ts:16: off by one tick. TheTRANSIENT_POLLSdoc and the disposition say the page gives up after 30 s. With++count < 3, failures at t=0, 10 and 20 s throw at 20 s.src/app/models/job-result.ts:48: the shared-poll JSDoc now documents the wrong member.private transientFailures = 0;sits betweenstatusResponse$'s JSDoc andstatusResponse$, so the doc now attaches to the counter. Move the field above the JSDoc.src/app/components/test/test.component.ts:63: the doc onsubmittingis stale. It says "True from submit until the backend answers", butsubmittingnow stays true untilrouter.navigatesettles. Someone trusting the doc could move the reset back to the response, which would bring back round-2's finding 4.src/app/components/test/test.component.ts:66: the hand-rolleddestroyedflag duplicatesDestroyRef.destroyed. Angular 20.3.0 already provides it (abstract get destroyed(): boolean), so checkthis.destroyRef.destroyedinstead.src/app/components/loading/loading.component.spec.ts:59: stale test comment and helper. "The terminal states are set by the progress stream, which ticks once a second" and theshowAfterATickhelper describe the mechanism this round removed. Thetick(1000)in the older error and canceled tests now does nothing, and those tests duplicate the new "as soon as the status arrives" test.
… with the poll - isTransient treated status 0 as final: Angular 20.3's XHR backend puts the ProgressEvent in error.error, and its .type is 'error'. A problem document is now an object with a string title, as fail() already assumed. - switchMap cancelled a request slower than the 10 s tick before its failure was counted, so a hung request kept the page on Processing. exhaustMap lets it finish, and timeout(10 s) bounds one that never answers. - The count is reset when refCount restarts the poll (defer), not only on success. - The give-up doc now matches measured behavior: 20 s after the first failure when polls fail fast, 30 s when each times out.
…op stale Loading tests The tick-based error and canceled tests and their helper described the mechanism the previous commit removed, and duplicated the immediate test.
|
Responses to the third pass, fixed in 4710035 and 888549d. Findings 1–3 each have a test that failed before its fix.
Verification:
|
matthewberry
left a comment
There was a problem hiding this comment.
Round-4 re-review at head 888549d. All four round-3 fixes are correct. Nothing was run. I read the code, plus the rxjs 7.8.2 source (timeout.ts, exhaustMap.ts, AsyncAction.ts), the version in the lockfile.
Your question: is the getResults download bound by the 10 s timeout? No. timeout(POLL_MS) wraps only this.service.getJob(this.jobId) inside the exhaustMap callback (job-result.ts:77-78). jobResultResponse$ waits for 'completed', then calls getResults with only retry attached (:107-108). So large downloads aren't cut off, but a hung one is never bounded (finding 2).
The fixes:
- 1:
isProblemDocumentrequires a stringtitle, so a status-0ProgressEventis now transient. The test uses a realnew ProgressEvent('error')and would have failed before the fix. The backend'sproblem()(app/routers/v1/problems.py:40,origin/main) takestitleas a required argument, so every /v1 problem document has one. - 2:
exhaustMapdrops ticks while a request is in flight.timeout(number)is{each}in rxjs 7, andTimeoutErroris checked first inisTransient. The inclusivetakeWhilestill ends the poll on a terminal status, andrefCountteardown unsubscribes the timer and the in-flight request. - 3:
deferresets the counter on every resubscribe. The restart test goes through therefCountreset. - 4–8: done.
Findings:
src/app/models/job-result.ts:78: the per-request timeout equals the poll period, so tie-breaking with the timer decides the give-up time. Reasoned from the rxjs source; not reproduced in a browser.- In
AsyncAction,timer(0, POLL_MS)creates itssetIntervalafter the first tick runs, then reuses it with deadlines at 10, 20, 30 s. - Each
timeout(POLL_MS)gets its own interval, created just after the tick that started its request. Request 1's timeout was created before the timer's interval, so it fires first at 10 s. But request 2's timeout is due at 20 s plus a sliver, after the 20 s tick, soexhaustMapdrops that tick. - Request 3 then starts at 30 s, and the page gives up at about 40 s, not the documented 30 s. While the backend is slow, every other poll is skipped.
fakeAsyncre-queues an interval after its callback runs, which reverses the tie. That's why the spec measures 30 s.- Fix: make the timeout strictly shorter than
POLL_MS, or schedule each poll after the previous request finishes, e.g.repeat({delay: POLL_MS}).
- In
src/app/models/job-result.ts:107:getResultshas no bound at all. If the connection stalls after'completed',retrynever sees an error, soisLoading$stays true and the spinner runs forever. TheTimeoutErrorbranch ofisTransientcan never trigger on this path either. If you want a bound, give the download its own, much longer timeout.src/app/models/job-result.spec.ts:127: the timeout test doesn't pin the documented 30 s. "Gives up on a request that never answers" advances 60 s and only checks that a problem was set, so a 40 s or 50 s give-up still passes. AssertingloadProblem$is null at 29.9 s and set at 30 s would pin it, thoughfakeAsync's tie-breaking still differs from the browser's (finding 1).src/app/components/test/test.component.ts:138: the submit handler has its own problem check. It testserror.error?.titlefor truthiness instead of usingisProblemDocument. Export the helper and share it, so the two definitions can't drift apart.src/app/models/job-result.ts:54: the counter could be local to each subscription. Declaringlet failures = 0inside thedefercallback, with theexhaustMappipe built there, ties the count to each subscription. That removes the field and the reset line, so correctness no longer depends onshareReplayholding exactly one source subscription.src/app/models/job-result.ts:48: a timeout and a network failure look the same.fail()reports a poll timeout as status 0. An overloaded backend that answers in 12 s then shows the same "could not be loaded" with status 0 as being offline. A distinct title or detail forTimeoutErrorwould separate them.
…ownload - The request timeout equalled POLL_MS, so in a browser the next tick could arrive while the request was still open and exhaustMap dropped it, pushing the give-up past the documented time. Requests now time out after 8 s, and the test pins the give-up at exactly 28 s. - getResults gets its own 2-minute timeout, so a stalled download ends. - isProblemDocument moves to models/v1-job.ts, and the submit form uses it too.
|
Round 4, fixed in 18d568d:
Verification:
|
matthewberry
left a comment
There was a problem hiding this comment.
Round-5 re-review of 18d568d: the round-4 findings are handled, and there are no new findings. It's good from my side. I read the diff; nothing was run.
- 1:
REQUEST_TIMEOUT_MS = 8000is shorter thanPOLL_MS. That removes the tie between the timer's tick and the request's timeout, soexhaustMapno longer drops a tick while a request is open. The test now pins the exact boundary: no error at 27,999 ms, an error at 28,000 ms. The arithmetic matches the doc: polls at 0, 10 and 20 s, each timing out 8 s later. - 2:
isProblemDocumentnow lives inmodels/v1-job.tsas a type guard, and both the result page and the submit form use it. - 3:
timeout(RESULTS_TIMEOUT_MS)sits beforeretry, so each attempt gets its own 2 minutes.isTransientcounts aTimeoutError, so a stalled download is retried twice and then reachesfail(). The worst case is about 6 minutes, not forever. - 4: the declines are fine; both were nits.
Part of #17. Built against mmli-backend #148, which has since merged and is deployed to staging and prod. The
/v1OpenAPI snapshot here comes from its final commit (aac06e6), and nothing in/v1changed after it.What changes
The template's input → loading → results flow now uses mmli-backend's
/v1API instead of the legacy per-type routes.inputspart plus anyfiles), with the notification address inX-Notify-Email. When the backend rejects the input, the form shows the problem document, including each field's JSON Pointer and reason./result/:id, which is what the backend's default notification email (mmli-backend #151) links to. Nothing linked to the oldtest/result/:id(I grepped the repo; no template app is deployed), so there's no redirect.src/app/api/v1, plusnpm run openapi:v1andopenapi:v1:sync. It uses openapi-generator 7.24.0 (openapitools.v1.json), because 6.6.0, which the legacy client uses, types FastAPI's OpenAPI 3.1 parameters asany. The legacy client is unchanged.npm run schemasandschemas:syncgenerateToolInputandToolResultsfromschemas/{input,output}.schema.json, andschemas:syncfetches those through the tool descriptor's links. CI checks that the generated types are current. The template's copies are job-template's sample tool.app.yamljob_typenow reaches the app asAPP_IDENTITY.jobType. The template's isjob-template-sample, a name I chose for the sample tool. The backend hasn't registered it; when it does, it should use that name, or this value should change.Findings for mmli-backend #148
POST /tools/{tool}/jobsdeclares bothapplication/jsonandmultipart/form-data. typescript-angular (6.6.0 and 7.24.0 alike) generates a method that always setsContent-Type: application/json, so uploads go throughHttpClientdirectly (TestService.submit)./v1response is a plain dict, so the generated client returnsanyfor jobs and results. The job document, results envelope and problem document are typed by hand insrc/app/models/v1-job.ts, mirroring_job_document. Response models on the routes would let the client generate these.started_atis always null today because the watcher never setstime_start. The progress estimate falls back tosubmitted_at.How it was verified
ng test --watch=false --browsers=ChromeHeadless. New tests cover:nightwatch/src/job_flow_spec.tsstarts its own mock/v1API and points the app at it by answering theenvvars.jsonrequest over CDP. It checks submit (multipart, the tool name, the input document, the email header), then/result/e2e-job, queued, processing, and the results table. I ran it locally with Chrome 155 againstnpm start, and both suites passed.nightwatch/tsconfig.jsonnow targets es2017. At es5, async tests compile to generators, so Nightwatch doesn't recognize them as async (it checksfn.constructor.name === 'AsyncFunction',lib/utils/index.js:77), and awaited commands don't wait.ng build,npm run app:sync -- --check,npm run schemas -- --check, andnpm run test:scripts(20 pass) all succeed.ng lintisn't configured in this repo.Deferred acceptance criteria
X-Notify-EmailorIdempotency-Keyyet; the fix is in mmli2 MR !7./result/:id: the route exists, but the link needs mmli-backend #151 deployed and the tool'sfrontend_urlset.Not in this PR
@storybook/angularis missing), so.storybook/tsconfig.jsondoesn't compile, before this change and after. The mock was ported anyway.