Repository navigation
flaky: recovery-rehearsal "effect 9 after" blocks in audit-dispatch with "exact value read unavailable" under load #874
Description
What happened
On 2026-09-29, one case failed during a local pnpm test:release-controller run (3950/3952 passed; the 2 failures were this one case and its parent):
test at scripts/release/test/recovery-rehearsal.test.mjs:89:11
✖ effect 9 after (23346.5775ms)
Error: HTTP recovery rehearsal failed in audit-dispatch; service errors: []; Recovery writer blocked: Recovery observation blocked: exact value read unavailable
at Object.drive (scripts/release/test/support/recovery-rehearsal.mjs:632:13)
[cause]: Error: Recovery writer blocked: Recovery observation blocked: exact value read unavailable
at requireThat (scripts/release/recovery/writer.mjs:41:21)
at observe (scripts/release/recovery/writer.mjs:237:5)
at async scripts/release/recovery/writer.mjs:948:25
at async transaction (scripts/release/recovery/writer.mjs:401:14)
at async withRecoveryPayloadReuse (scripts/release/recovery/payload-reuse.mjs:225:12)
at async resume (scripts/release/test/support/recovery-rehearsal.mjs:423:14)
at async Object.drive (scripts/release/test/support/recovery-rehearsal.mjs:465:28)
- Branch: the branch of fix(release): let detect resume correlate-audit from an attached successful attempt #867, whose only change is to
scripts/release/observe.mjs. That file isn't imported by the recovery path, which uses onlyscripts/release/recovery/observe.mjs. - Machine: macOS, Node 24.20.0, during a full-suite run that took about 17 minutes.
- Rerun on its own,
node --test scripts/release/test/recovery-rehearsal.test.mjspassed 70/70. Two later full-suite runs (3959/3959 and 3975/3975) didn't reproduce it, and it hasn't been seen in CI.
Likely cause (not confirmed)
The message comes from readerContext in scripts/release/recovery/observe.mjs (around line 107-130): envelope() requires runRecoveryRead(...) to return PRESENT. That reader uses a real wall-clock phaseDeadline = Date.now() + RECOVERY_RETRY.phaseDeadlineMs and a real setTimeout sleep, not the rehearsal's controlled clock. The case took 23s. With the test's concurrency: 2 and the whole suite competing for the machine, a read against the local HTTP rehearsal server probably ran past the deadline or used up its retries, so it came back non-PRESENT. resume() in the rehearsal only retries errors matching /uncertain|stopped/, so "read unavailable" failed the case.
This fits the load-sensitive pattern in #535 / #715 (flakes that reproduce under process-spawn contention, not CPU spin).
Suggested next steps
- Try to reproduce by running
recovery-rehearsal.test.mjsnext to process-spawn-heavy load (as in test(k8s-compat): size the docker smoke watchdog for a wedge, not a stopwatch #715), and log the underlyingrunRecoveryReadresult (status/code), which the current error message drops. - If it's the deadline, either inject the rehearsal's clock into
readerContext, or give the rehearsal a longerphaseDeadlineMs. Don't loosen what production accepts. - Keep the diagnostic regardless: include the envelope's status/code in the
exact ${key} read unavailablemessage, so the next occurrence says why.
🤖 Generated with Claude Code
Activity
blove commentedon Sep 30, 2026
Findings from #877. I'm leaving this issue open because the natural trigger is still unconfirmed.
phaseDeadlineMsis 20 minutes. The limits a single observe read can hit in real time are the 15s per-attempt race timer and the 15s HTTP timeout.listReleaseAssetsGET inaudit-dispatchfor 16s duringeffect 9 afterreproduces the reported message exactly. The underlying envelope wasERROR READ_TIMEOUT_UNSETTLED. The effect-9 fault had already been resumed infive-lanes, soresume()could not retry again.READ_TIMEOUT_UNSETTLED;Date.nowdeadlines count sleep time but the test runner's monotonic duration does not, which could explain a 23s case. That would giveRECOVERY_DEADLINEorTIMEOUT.runRecoveryReadcalls still use a real 15s race timer.🤖 Generated with Claude Code
test(release): run recovery rehearsal observation reads on an injecte…