Skip to content

iOS CI: pair-and-push Try Again, relay reply wait: a proxy "down" that answered 200, and a stale thread read that took the pick away - #1821

Merged
backnotprop merged 1 commit into
mainfrom
fix/ios-ci-flakes
Oct 10, 2026
Merged

backnotprop merged 1 commit into
mainfrom
fix/ios-ci-flakes

Conversation

@backnotprop

@backnotprop backnotprop commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Two older iPhone CI proof flakes, fixed forward. No timeout was raised.

(a) pair-and-push: ProofTests could not find "Try Again"

Run 38031955782 (head 3c0a4d8), job 114155099825, the same failure as runs 37987495244 and 38013707345 attempt 3:

t = 174.36s Failed: Failed to compute hit point for Button, {{inf, inf}, {0.0, 0.0}}, identifier: 'thread-unreachable', label: 'Try Again'
ProofTests testPairPickSendResolveDeleteRemove : Failed to no matches found ... '"Try Again" IN identifiers'

The recording shows the thread opening by itself about one second after /proxy pass, before the tap landed.

Cause: a test race, in proof.ts. The proxy's down mode answered every call with a Response whose body stream errors. Bun ends such a response as a clean, empty 200 (checked with curl: 200, 0 bytes). So the app's health probe read the path as reachable. The app then read the list, the thread and the event stream every 2 s through the "outage" (a local run's proxy log shows the cycle at 07:23:19.75, 07:23:21.83 and 07:23:23.43). The first cycle after pass opened the thread under the test's tap. A real outage does not do this, so the "Can't reach" screen in that step existed only because of the harness.

Fix (proof code only):

  • down stops the proxy's listener. A call is refused and the open connections close. pass listens again on the same port.
  • down also cuts the Inbox's relay socket. The phone is paired with the relay on, so with only its direct path refused it correctly reads the thread through the relay (seen locally once the listener was really gone). The computer out of reach now means out of reach on every path. The screen is "Can't reach", with the relay variant of its words ("It opens once your computer is back online"), and stays there until Try Again.
  • ProofTests waits for Try Again to be hittable, and each wait in the step stops the test (require, as in PR 1820).

(b) relay-transport: /reply: no reply within 90 s

Run 38017849289 attempt 2, job 114120836855, and run 37958404153 before the split. Both times it was the first Send on the relay path:

t = 141.07s Tap "send" ... Synthesize event
RelayTransportTests testTheRelayAsTheFallbackPath : failed: caught error: "/reply: ["error": no reply within 90 s]"

The 90 s is proof.ts's own wait on the agent's wait_for_reply, not the relay's. The relay's log (artifact relay-wrangler.log) shows slow command posts on that runner: POST .../commands 202 Accepted (2622ms), (2618ms), (1256ms). After the row opened, only nine commands came in, and then nothing for 90 s. A local run of the same step posts the Send and then five or more reads after it. So the Send was never posted.

Cause: a real race, in the app (SourceSession.loadThread). Opening the thread starts a thread read through the relay. The pick goes out while that read is still being polled for. The two commands' results are read back by separate polls on their own backoff (0.25 s doubling to 4 s). So the read's answer, taken by the Inbox before the pick, can land after the pick's own answer. threads[id] = answer.thread then put the question back to open, at its old revision. With no pick and no words, send() returns without posting anything.

Fix: the one app change, below.

Not reproduced locally: three runs with a 2.6 s delay on each relay command post, and seven without, all passed. The cause comes from the relay log and the polling code.

The app change (reviewed by the coordinator before the push)

apps/ios/Plannotator/App/SourceSession.swift, loadThread. It closes this race: a thread read that started before this phone changed the thread's questions overwrote that change when its answer landed later. On the relay path the change is a pick drawn, or the Inbox's answer to a pick or a Send. The read's and the pick's results are read back by separate polls, so the read's answer can land after the pick's. A per-session counter, bumped as each read starts and at each question change (threadClock, and the per-thread mark questionsChanged), lets loadThread drop a read that started before the thread's last change. The next read brings both. A dropped read never leaves threads[id] empty, because a question change needs a loaded thread. The proof that exercised it: the relay-transport shard, 5 of 5 locally with the change (plus 3 earlier runs with a 2.6 s delay on each relay command post), and the two CI runs below. Every other shard was run once locally with it too.

Also

  • On a failed run the artifact now carries the Inbox's own inbox.log (each relay command it applied and its status). A /reply that times out takes a no-reply screenshot. Neither was available for (b).
  • The dropped() comment said what it does not do. It now says it. drop-reply is unchanged.

Runs

Local, final code, simulators made and deleted by proof.ts: pair-and-push 5/5, relay-transport 5/5, decisions-guides 1/1, attachments 1/1.

CI, workflow "iPhone app" run 38039142175 on head 18fb170:

Shard Run 1 (attempt 1) Run 2 (attempt 2, then attempt 3)
build success success
pair-and-push success success
relay-transport success failure, then success on the one rerun
attachments success success
decisions-guides success success

Run 2's relay-transport failure was outside both faults. wrangler dev (the local relay) crashed in esbuild (fatal error: all goroutines are asleep - deadlock! in relay-wrangler.log), after which the relay proxy could not reach it (error: Unable to connect) and the next /ask reported no push reached Apple. That was late in the class, after every relay Send step had passed. The failed job was rerun once (gh run rerun --failed, attempt 3), and it passed.

…t answered 200, and a stale thread read that took the pick away

pair-and-push (runs 37987495244, 38013707345 attempt 3, 38031955782):
the proxy's down mode answered with a Response whose body errors, which
Bun ends as a clean empty 200. The app's health probe read the path as
reachable, so the app read again every 2 s through the outage and opened
the thread itself right after /proxy pass, under the Try Again tap. Down
now stops the proxy's listener (refused, open connections closed) and
cuts the Inbox's relay socket, the computer out of reach on every path;
pass listens again on the same port. ProofTests waits for Try Again to be
hittable and its waits stop the test.

relay-transport (runs 37958404153, 38017849289 attempt 2): on a slow
relay the thread read started when the row opened could land after the
pick's own answer and put the question back to open, so the Send had
nothing to post. SourceSession.loadThread no longer applies a read that
started before this phone last changed the thread's questions.

A failed proof keeps the Inbox's inbox.log in the artifact, and a /reply
that times out takes a no-reply screenshot.
@backnotprop
backnotprop merged commit 61f65b1 into main Oct 10, 2026
44 of 46 checks passed
@backnotprop
backnotprop deleted the fix/ios-ci-flakes branch October 10, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant