Repository navigation
iOS CI: build once, proof classes in shards - #1809
Merged
Merged
Conversation
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The iPhone CI job ran every proof class (the warm-up, M1 to M6, the Keychain and PlannotatorKit tests) on one macOS runner. It no longer fit its 75-minute limit: PR 1808 ran 1h16m and was cancelled. This PR builds once, runs the proof classes in four parallel shards (21 to 34 minutes on this PR, 45 at most), and keeps the old job name as the single verdict. No limit is raised and every class still runs.
What changed
.github/workflows/ios.ymlbuild("iPhone app build (binary and build-for-testing)", 30 min): the colour check, a check that every proof class is in a shard, the compiledplannotatorbinary, and onexcodebuild build-for-testingagainst a simulator like the one proof.ts makes (newest iOS runtime, iPhone 17). It keeps the SwiftPM and DerivedData cache.Build/Productsand.local/plannotatorare packed into one tar (an artifact upload drops executable bits) and uploaded asios-build(kept 1 day).proof("iPhone proof ()", matrix,fail-fast: false, 45 min each): checkout,bun install, Xcode 26, unpack the artifact, thenproof.ts --prebuilt --only <the shard's classes>. proof.ts already makes its own simulator, runsWarmUpLaunchfirst (two tries, the cold-runner rule from M3), runs the shard's classes withtest-without-building, and deletes the simulator. The shots and the recording are uploaded per shard asios-proof-<shard>.ios-app("iPhone app (XCUITest against the compiled binary)", the old job's name):needs: [build, proof],if: always(). It is green only when the build and every shard passed.apps/ios/scripts/proof.ts: one additive flag,--prebuilt, which skipsbuild-for-testingwhen--derivedalready holds the products. Without the flag, local runs behave exactly as before.Shard map and durations
Wall time per job. "3 shards" is run 37942225127 (head 7ce453e, all green). "4 shards" is run 37948454365 (head 11299f3, the layout in this PR).
Each shard spends about 7 to 12 minutes before its first class: bun install, proof.ts's Inbox, agents and relay, the simulator boot, and a normal warm-up (72 to 177 s). The first layout had three shards. Its largest shard (34m30s) was too close to 45, so it was split once more. RelayTransportTests is one test, so its shard cannot get smaller without splitting the test.
The one red, and its cause. Attempt 1 of
relay-transporton run 37948454365 hit the 45-minute limit inside the warm-up, not in a proof class. Try 1 failed withWarmUpLaunch testFirstLaunch : Failed to get matching snapshots: Unable to perform work on main run loop, process main thread busy for 30.0sand then spent 577 s in 2-secondremove-sourcewaits. Try 2 failed atWarmUpLaunch.swift:61after 789 s. RelayTransportTests started at minute 32. The failed job was re-run once and passed in 25m45s. The single job carries the same risk, because the warm-up's worst case (two long tries, about 23 minutes) is part of every run. In a shard, that worst case plus RelayTransport's 16 minutes and the setup can pass 45. This PR does not change the warm-up. The follow-up is to bound it: fail a try early when the first screen does not come, and cap theremove-sourceloop.Coverage matches the single job's run 37888395408: the same 21 PlannotatorKit tests, the Keychain test, and every UI class. The
buildjob fails if aPlannotatorUITestsclass (other thanWarmUpLaunchand the baseProofCase) or either unit-test target is missing from the matrix.The check name
Neither a ruleset nor branch protection on
mainrequires any check:gh api repos/backnotprop/plannotator/rulesetslists only "Protect release tags", andrules/branches/mainandbranches/main/protectionare empty or 404. The old name "iPhone app (XCUITest against the compiled binary)" still exists as the summary job, so anyone who reads that check, or a ruleset added later, still gets one verdict.On failure
fail-fast: false: a failing shard does not cancel the others, so one run reports every failing class. The failing shard keeps its screenshots, the recording,relay-wrangler.logand any crash report inios-proof-<shard>. The summary job goes red if the build or any shard failed or was cancelled. Ifbuildfails, no shard starts and the summary is red.Deviations from the brief
--prebuiltflag. The shard must run through proof.ts, because the test calls its control server for the Inbox, the agents and the relay. Without the flag, proof.ts runsbuild-for-testingagain in every shard.always(), as the single job did, not only on failure. The passing recordings stay available for review.Gates
.semgrep.yml) onproof.ts: one finding,request-to-filesystemat thesimctl pushbody write. It is the same finding on main's copy of the file, so nothing new. The workflow file is not a Semgrep target.