Repository navigation
ci: require exact test identities in shard evidence (#83, #85) - #110
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
LIghtJUNction
left a comment
LIghtJUNction
left a comment
There was a problem hiding this comment.
Reviewed exact final diff: only scripts/ci_shards.py and tests/test_ci_shards.py differ from main b1ac767. Downloaded all four artifacts from exact-head run 36352580956 and independently matched each ZIP SHA-256 against the provider's digest. The new gate verifies all 1,098 exact test identities, disjoint and complete. JUnit cases by shard: 263/286/295/254; all zero failures, errors and skips. Conformance: 8 passed, zero failures/errors/skips. Wheel and sdist are present. The real subprocess regression validates pytest's actual JUnit identity output, not just synthetic reports. The final GitHub contracts job is still queued, so this is independent evidence review, not a claim that the overall workflow status is success yet. No application/authority change, skipped tests or production action.
Refs #82, #84, #85. Share the transaction-local deadline/attempt fence across notification, tool and bundle-push completion. Commit maintenance results atomically with the fenced terminal state. No automatic resend of expired work and no authority expansion. Exact-head CI 36353193024 succeeded: 1,109 tests plus 8 conformance checks, zero failures/errors/skips, wheel/sdist and contracts gate. All 11 new PostgreSQL regressions passed. Four downloaded artifacts matched provider SHA-256 digests and independently passed the exact test-identity gate. Current main's #110 tree is identical to the tested foundation. No production deployment, restore or umbrella issue closure.
Refs #83 and #85. Strengthen the existing full-suite evidence gate; no acceptance umbrella closure.
Reproduced problem
The gate verified the selected-test count but not which testcases JUnit contained. A report with the correct number of cases could replace or duplicate an executed test identity while still passing.
Fix
Record each selected pytest node ID as a JUnit user property, then require exactly one identity per testcase and exact equality with the shard's selected IDs. Keep the existing complete collection, disjoint partition, count, failure/error and successful-runner gates. The manifest version is 2 so old reports cannot be silently mixed with identity-bearing reports. No filename allowlist, test exclusions, timeout increase or workflow/service changes.
Verification
Four new same-count corruption cases failed before the fix and pass after it. All 22 local tests pass, including an actual subprocess pytest run with parameter punctuation, Unicode and class-based tests, followed by verification of its real JUnit XML. Local Python 3.13.5; full exact-head CI still required. Published script and test Git blob hashes exactly match the locally verified files.
This does not assert that previous CI reports contained wrong identities, only that the previous gate could not detect that class of error. No application, database, authority or production deployment change.