Skip to content

iOS CI: decisions-guides shard: races on a slow runner, the first web view, and failure cascades - #1820

Merged
backnotprop merged 4 commits into
mainfrom
fix/ios-ci-decisions-guides
Oct 10, 2026
Merged

backnotprop merged 4 commits into
mainfrom
fix/ios-ci-decisions-guides

Conversation

@backnotprop

@backnotprop backnotprop commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Cause

Since the split, the decisions-guides shard runs M3 first, straight after the warm-up, on its own runner. That runner is slower than the old single job: M3 took 386 to 524 s there, against 338 s on 046a3a5. Main's code passes locally 3 of 3 times. The failures are races that only a slow runner hits, plus one real ordering dependency. Each failure then spread to the next class, and in three runs to a hung job.

  1. Back tapped while the decision sheet was still closing. Run 38010716662 (PR 1819):
    t = 226.06s Tap Button (Element at index 0)
    t = 226.18s     Check for interrupting elements affecting "decision-cancel" Button
    Failed to tap "Decisions" Button: No matches found for Descendants matching type TabBar
    
  2. The New message sheet was read as closed 3 s after Send. Run 37973911538 (main, b0f60ec): DecisionsProofTests.swift:132: XCTAssertFalse failed - the sheet closes once delivered. The text field still existed at t=275, 277 and 279 s after the Send tap at t=272 s.
  3. M4 opens the shard's first web view. Run 38017849289 attempt 2: the guide cover showed only its spinner for 30 s (the guide's title not found, with the screen printed). Before the split, AttachmentProofTests paid WebKit's start-up cost on the simulator first.
  4. A failed class kept its screen recording running. The next class's simctl io recordVideo exits at once with Host recording is already in progress (exit 16, reproduced locally). /video/stop then waited for an exit that had already happened, so the next class failed on NSURLErrorDomain Code=-1001 "The request timed out.". This happened to M4 in 37973911538 and 38010716662, and to RelayPushTests in pair-and-push in 37958261402 and 37987495244.
  5. The job hung until the 45 min timeout after a ProofCase failure. This happened on d5d5f54 (37958261402), 38017849289 attempt 2 and 38023662529. An async XCTest runs on past a failure. ProofCase's failure tearDown terminated and reinstalled the app while the test body was still tapping. The log shows Tear Down, then Terminate, then the body's Tap Button (Element at index 0), then a second Tear Down, then nothing for 30 min.
  6. Also on a slow runner: Reply instead was read as not hittable before the swipe had settled (38023662529), and the 8.1 menu did not close on the first tap outside (38027139806 attempt 2).

Fix (proof code only; every class still runs)

  • DecisionsProofTests
    • Is now a ProofCase: cold install at setUp, shared helpers, duplicates removed.
    • Waits for each sheet to close (after Cancel, after Done and before Back, after both Sends).
    • Waits for Reply instead to be hittable.
    • Closes the 8.1 menu and the 8.3 words by tapping outside, up to three times.
  • GuideProofTests: the first wait for the surface uses ProofWait.opening, like AttachmentProofTests' surface waits. This is the one bound that changed. It is where WebKit's first start is paid in this shard.
  • Waits now throw (require): the pairing waits, the guide's waits and M3's sheet waits end the test body when they fail. The pairing waits print the screen when they fail. Two older pairing failures had nothing to read: 38013707345 (the cover did not open) and d5d5f54 (no list after the code).
  • ProofCase no longer resets in a tearDown. The next class's setUp resets. RelayPushTests, the one class after a ProofCase that did not reset itself, now resets at setUp.
  • proof.ts: stopVideo() returns when the recorder has already exited, and /video/start stops one that is still running.

Runs

  • Local, iOS 26.5, proof.ts --prebuilt --only DecisionsProofTests,GuideProofTests:
    • main's code: 3 of 3 pass.
    • fixed code: 5 of 5 pass across the heads, one of them on 3c0a4d8.
    • pair-and-push passes once on 1f2e9a6.
  • CI, decisions-guides on each head:
    • f56fd30: green, then red (Guide's first web view, the tearDown hang).
    • 03de7dc: red (Reply instead, the tearDown hang).
    • 1f2e9a6: green, then red (the 8.1 menu; no hang, and M4 passed after it).
    • 3c0a4d8: green (run 38031955782).
  • Other shards on 3c0a4d8: relay-transport and attachments green. pair-and-push red on a different, older fault: ProofTests could not find "Try Again", the same failure as dependabot run 37987495244 before this PR. RelayPushTests passed after it, so the reset at setUp works. Not addressed here.

…class's recording no longer hangs the next

The shard runs M3 first on a fresh runner, slower than the old single job.
DecisionsProofTests tapped Back while the decision sheet was still sliding
away (its Cancel is then the first bar button, run 38010716662) and read the
New message sheet as closed 3 s after Send (run 37973911538). It now waits
for each sheet to close, and is a ProofCase (cold install at setUp, reset on
failure) like the other classes.

A class that failed before /video/stop left simctl recording; the next
class's recorder exited at once (Host recording is already in progress) and
/video/stop waited on that exit forever, so GuideProofTests (and RelayPushTests
in pair-and-push) failed on a 120 s request timeout. stopVideo() returns when
the recorder has exited, /video/start stops a running one, and a failed
ProofCase stops its own recording. The pairing waits print the screen when
they fail.
…e's first surface waits as an opening screen

Run 38017849289 attempt 2: M3 passed; M4's guide cover showed only its
spinner for 30 s. M4 is the first class in this shard to open a web view, so
WebKit's first start on the simulator is paid there (AttachmentProofTests
paid it before the split); that one wait uses ProofWait.opening, as
Attachment's surface waits do.

After that failure the test body kept tapping while XCTest ran its tearDown,
and the job hung until the 45 min timeout (as on d5d5f54). An async test
does not stop at an XCTFail. The pairing waits, the guide's waits and M3's
sheet waits now throw (require), so the body ends before tearDown.
…ill-running test

Run 38023662529: M3 failed at the largest text size (Reply instead read as
not hittable before the swipe settled) and the job hung until the timeout,
the third hang after a ProofCase failure. An async test runs on past a
failure, so ProofCase's failure tearDown terminated and reinstalled the app
under the test body. The reset at the next class's setUp already gives the
cold install; RelayPushTests, the one class after a ProofCase that did not
reset itself, now resets at setUp too. A leftover recording is stopped by
the next /video/start. Reply instead is read once hittable.
@backnotprop backnotprop changed the title iOS CI: decisions-guides shard: sheets awaited before Back, a failed class's recording no longer hangs the next iOS CI: decisions-guides shard: races on a slow runner, the first web view, and failure cascades Oct 10, 2026
… they close

Run 38027139806 attempt 2: the 8.1 menu did not close on the first tap
outside within 10 s (DecisionsProofTests.swift:212). M4 passed after it and
the job ended without a hang. Up to three taps, 5 s apart, for the 8.1
menu and the 8.3 words.
@backnotprop
backnotprop merged commit 604ed42 into main Oct 10, 2026
32 of 34 checks passed
@backnotprop
backnotprop deleted the fix/ios-ci-decisions-guides branch October 10, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant