Repository navigation
iOS CI: pair-and-push Try Again, relay reply wait: a proxy "down" that answered 200, and a stale thread read that took the pick away - #1821
Merged
Conversation
…t answered 200, and a stale thread read that took the pick away pair-and-push (runs 37987495244, 38013707345 attempt 3, 38031955782): the proxy's down mode answered with a Response whose body errors, which Bun ends as a clean empty 200. The app's health probe read the path as reachable, so the app read again every 2 s through the outage and opened the thread itself right after /proxy pass, under the Try Again tap. Down now stops the proxy's listener (refused, open connections closed) and cuts the Inbox's relay socket, the computer out of reach on every path; pass listens again on the same port. ProofTests waits for Try Again to be hittable and its waits stop the test. relay-transport (runs 37958404153, 38017849289 attempt 2): on a slow relay the thread read started when the row opened could land after the pick's own answer and put the question back to open, so the Send had nothing to post. SourceSession.loadThread no longer applies a read that started before this phone last changed the thread's questions. A failed proof keeps the Inbox's inbox.log in the artifact, and a /reply that times out takes a no-reply screenshot.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two older iPhone CI proof flakes, fixed forward. No timeout was raised.
(a) pair-and-push: ProofTests could not find "Try Again"
Run 38031955782 (head 3c0a4d8), job 114155099825, the same failure as runs 37987495244 and 38013707345 attempt 3:
The recording shows the thread opening by itself about one second after
/proxy pass, before the tap landed.Cause: a test race, in proof.ts. The proxy's
downmode answered every call with aResponsewhose body stream errors. Bun ends such a response as a clean, empty200(checked with curl:200, 0 bytes). So the app's health probe read the path as reachable. The app then read the list, the thread and the event stream every 2 s through the "outage" (a local run's proxy log shows the cycle at 07:23:19.75, 07:23:21.83 and 07:23:23.43). The first cycle afterpassopened the thread under the test's tap. A real outage does not do this, so the "Can't reach" screen in that step existed only because of the harness.Fix (proof code only):
downstops the proxy's listener. A call is refused and the open connections close.passlistens again on the same port.downalso cuts the Inbox's relay socket. The phone is paired with the relay on, so with only its direct path refused it correctly reads the thread through the relay (seen locally once the listener was really gone). The computer out of reach now means out of reach on every path. The screen is "Can't reach", with the relay variant of its words ("It opens once your computer is back online"), and stays there until Try Again.require, as in PR 1820).(b) relay-transport:
/reply: no reply within 90 sRun 38017849289 attempt 2, job 114120836855, and run 37958404153 before the split. Both times it was the first Send on the relay path:
The 90 s is proof.ts's own wait on the agent's
wait_for_reply, not the relay's. The relay's log (artifactrelay-wrangler.log) shows slow command posts on that runner:POST .../commands 202 Accepted (2622ms),(2618ms),(1256ms). After the row opened, only nine commands came in, and then nothing for 90 s. A local run of the same step posts the Send and then five or more reads after it. So the Send was never posted.Cause: a real race, in the app (
SourceSession.loadThread). Opening the thread starts a thread read through the relay. The pick goes out while that read is still being polled for. The two commands' results are read back by separate polls on their own backoff (0.25 s doubling to 4 s). So the read's answer, taken by the Inbox before the pick, can land after the pick's own answer.threads[id] = answer.threadthen put the question back to open, at its old revision. With no pick and no words,send()returns without posting anything.Fix: the one app change, below.
Not reproduced locally: three runs with a 2.6 s delay on each relay command post, and seven without, all passed. The cause comes from the relay log and the polling code.
The app change (reviewed by the coordinator before the push)
apps/ios/Plannotator/App/SourceSession.swift,loadThread. It closes this race: a thread read that started before this phone changed the thread's questions overwrote that change when its answer landed later. On the relay path the change is a pick drawn, or the Inbox's answer to a pick or a Send. The read's and the pick's results are read back by separate polls, so the read's answer can land after the pick's. A per-session counter, bumped as each read starts and at each question change (threadClock, and the per-thread markquestionsChanged), letsloadThreaddrop a read that started before the thread's last change. The next read brings both. A dropped read never leavesthreads[id]empty, because a question change needs a loaded thread. The proof that exercised it: the relay-transport shard, 5 of 5 locally with the change (plus 3 earlier runs with a 2.6 s delay on each relay command post), and the two CI runs below. Every other shard was run once locally with it too.Also
inbox.log(each relay command it applied and its status). A/replythat times out takes ano-replyscreenshot. Neither was available for (b).dropped()comment said what it does not do. It now says it.drop-replyis unchanged.Runs
Local, final code, simulators made and deleted by proof.ts: pair-and-push 5/5, relay-transport 5/5, decisions-guides 1/1, attachments 1/1.
CI, workflow "iPhone app" run 38039142175 on head 18fb170:
Run 2's relay-transport failure was outside both faults.
wrangler dev(the local relay) crashed in esbuild (fatal error: all goroutines are asleep - deadlock!inrelay-wrangler.log), after which the relay proxy could not reach it (error: Unable to connect) and the next/askreportedno push reached Apple. That was late in the class, after every relay Send step had passed. The failed job was rerun once (gh run rerun --failed, attempt 3), and it passed.