Skip to content

iOS CI: build once, proof classes in shards - #1809

Merged
backnotprop merged 2 commits into
mainfrom
ci/ios-shards
Oct 9, 2026
Merged

backnotprop merged 2 commits into
mainfrom
ci/ios-shards

Conversation

@backnotprop

@backnotprop backnotprop commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

The iPhone CI job ran every proof class (the warm-up, M1 to M6, the Keychain and PlannotatorKit tests) on one macOS runner. It no longer fit its 75-minute limit: PR 1808 ran 1h16m and was cancelled. This PR builds once, runs the proof classes in four parallel shards (21 to 34 minutes on this PR, 45 at most), and keeps the old job name as the single verdict. No limit is raised and every class still runs.

What changed

  • .github/workflows/ios.yml
    • build ("iPhone app build (binary and build-for-testing)", 30 min): the colour check, a check that every proof class is in a shard, the compiled plannotator binary, and one xcodebuild build-for-testing against a simulator like the one proof.ts makes (newest iOS runtime, iPhone 17). It keeps the SwiftPM and DerivedData cache. Build/Products and .local/plannotator are packed into one tar (an artifact upload drops executable bits) and uploaded as ios-build (kept 1 day).
    • proof ("iPhone proof ()", matrix, fail-fast: false, 45 min each): checkout, bun install, Xcode 26, unpack the artifact, then proof.ts --prebuilt --only <the shard's classes>. proof.ts already makes its own simulator, runs WarmUpLaunch first (two tries, the cold-runner rule from M3), runs the shard's classes with test-without-building, and deletes the simulator. The shots and the recording are uploaded per shard as ios-proof-<shard>.
    • ios-app ("iPhone app (XCUITest against the compiled binary)", the old job's name): needs: [build, proof], if: always(). It is green only when the build and every shard passed.
  • apps/ios/scripts/proof.ts: one additive flag, --prebuilt, which skips build-for-testing when --derived already holds the products. Without the flag, local runs behave exactly as before.

Shard map and durations

Wall time per job. "3 shards" is run 37942225127 (head 7ce453e, all green). "4 shards" is run 37948454365 (head 11299f3, the layout in this PR).

Job Classes 3 shards 4 shards
build binary + build-for-testing 5m31s 4m12s
relay-transport RelayTransportTests (M6) 30m09s (test 966 s) attempt 1: cancelled at 45m (see below); attempt 2: 25m45s (test 923 s, warm-up 72 s)
pair-and-push ProofTests (M1), RelayPushTests (M5) 28m23s (685 s + 424 s) 34m03s (567 s + 392 s; the warm-up needed its second try)
attachments PlannotatorKitTests (21 Swift Testing tests), PlannotatorTests (Keychain), AttachmentProofTests (M2), WorkspacesProofTests (skips without its account, as before) one shard with decisions-guides: 34m30s 21m49s (Attachment 734 s)
decisions-guides DecisionsProofTests (M3), GuideProofTests (M4) (in the shard above: 348 s + 239 s) 23m52s (524 s + 263 s)

Each shard spends about 7 to 12 minutes before its first class: bun install, proof.ts's Inbox, agents and relay, the simulator boot, and a normal warm-up (72 to 177 s). The first layout had three shards. Its largest shard (34m30s) was too close to 45, so it was split once more. RelayTransportTests is one test, so its shard cannot get smaller without splitting the test.

The one red, and its cause. Attempt 1 of relay-transport on run 37948454365 hit the 45-minute limit inside the warm-up, not in a proof class. Try 1 failed with WarmUpLaunch testFirstLaunch : Failed to get matching snapshots: Unable to perform work on main run loop, process main thread busy for 30.0s and then spent 577 s in 2-second remove-source waits. Try 2 failed at WarmUpLaunch.swift:61 after 789 s. RelayTransportTests started at minute 32. The failed job was re-run once and passed in 25m45s. The single job carries the same risk, because the warm-up's worst case (two long tries, about 23 minutes) is part of every run. In a shard, that worst case plus RelayTransport's 16 minutes and the setup can pass 45. This PR does not change the warm-up. The follow-up is to bound it: fail a try early when the first screen does not come, and cap the remove-source loop.

Coverage matches the single job's run 37888395408: the same 21 PlannotatorKit tests, the Keychain test, and every UI class. The build job fails if a PlannotatorUITests class (other than WarmUpLaunch and the base ProofCase) or either unit-test target is missing from the matrix.

The check name

Neither a ruleset nor branch protection on main requires any check: gh api repos/backnotprop/plannotator/rulesets lists only "Protect release tags", and rules/branches/main and branches/main/protection are empty or 404. The old name "iPhone app (XCUITest against the compiled binary)" still exists as the summary job, so anyone who reads that check, or a ruleset added later, still gets one verdict.

On failure

fail-fast: false: a failing shard does not cancel the others, so one run reports every failing class. The failing shard keeps its screenshots, the recording, relay-wrangler.log and any crash report in ios-proof-<shard>. The summary job goes red if the build or any shard failed or was cancelled. If build fails, no shard starts and the summary is red.

Deviations from the brief

  • proof.ts gets the one --prebuilt flag. The shard must run through proof.ts, because the test calls its control server for the Inbox, the agents and the relay. Without the flag, proof.ts runs build-for-testing again in every shard.
  • The shots upload runs on always(), as the single job did, not only on failure. The passing recordings stay available for review.
  • The required-check premise does not hold (see above). The name is kept anyway.

Gates

  • gitleaks over merge-base..HEAD: no leaks.
  • Semgrep (the pinned CE image, .semgrep.yml) on proof.ts: one finding, request-to-filesystem at the simctl push body write. It is the same finding on main's copy of the file, so nothing new. The workflow file is not a Semgrep target.
  • actionlint: info-level shellcheck notes only.

@backnotprop
backnotprop merged commit d5d5f54 into main Oct 9, 2026
38 of 40 checks passed
@backnotprop
backnotprop deleted the ci/ios-shards branch October 9, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant