Repository navigation
No CI gate on main: the regression suites never run before a merge, and a suite that skips itself reports success #417
Description
main has no automated gate. The three regression suites run when somebody remembers
to run them, which is not often enough to be relied on.
What CI has to get right here
The suites are written for a developer machine, where a missing dependency is a good
reason to step aside: they print SKIP and exit 77. On CI that is the one outcome that
must never be tolerated, because a gate that opens when its input is missing is worse
than no gate, the green is read as evidence by whoever comes next. Two instances, both
of which would otherwise pass unnoticed:
- the parser is a separate repository; without it all three suites exit 77
test/java/torturereads class files withjava.lang.classfile, which is not final
before JDK 24. On an older JDK it exits 77 and the ten-family oracle does not run,
while the suite still reports success
Every shape of "did not actually run" therefore has to become a failure: exit 77, a
sub-harness that swallowed its own 77, and a suite reporting passed 0.
Scope
- three required tiers: build and typecheck; the parser-free repo invariants; and the
java / python / typescript suites in parallel - ground truth, not only goldens. A golden says "the same as last time", which a
wrong answer satisfies as long as it was wrong last time too. Java and TypeScript run
with--oracle, scored against javac/javap and the TypeScript compiler respectively - the parser pinned by commit, so a change in the other repository cannot turn this one
red with no commit here to point at, and a parser regression cannot quietly become the
new expectation - a nightly run against the parser's
mainthat opens an issue when the pin falls behind - contribution path: issue templates, a pull request template,
CODEOWNERS - build-status badges on the README
Branch protection
The ruleset for main, pull request required, approving review, CI green, linear
history, no force-push, no deletion, no bypass actors, is written as
.github/scripts/protect-main.sh and applied separately from this change.
Activity
swapnilpaliwal-sd commentedon Sep 12, 2026
Two findings from getting the suites onto a clean machine, both fixed in #418
1. The java torture families cannot be scored without the platform IR
The harness stages the JVM platform IR as a library on purpose: half the families call
java.util.List,Mapand the functional interfaces, and with the platform absent thosereceivers cannot be typed by any rule. Measured on a run with it removed:
The harness already warns that in this state "this score measures the staging, not the
rules". That IR is 1.8 GB and is built from a JDK source checkout, so it cannot live in
a repository, an Actions cache, or a runner.
Running the families anyway would paint a red that reads as a regression in whatever
change happened to meet it — the most expensive kind of false signal, because the next
person debugs their own diff. CI therefore passes
--no-torture, and the suite printsthat the families were excluded: an excluded family must never be mistakeable for one
that passed. They remain a local gate.
2. Client->library coverage could vanish without anything noticing
Worth stating plainly, because the coverage is better than it looks:
lib-src/stub library, passed as--librarySo client->library is already exercised broadly, and all of it runs in CI. Restricting to
the torture scripts would be a reduction — and for java specifically, torture is the one
part that cannot run in CI at all, per the finding above.
The real risk was different: the client->library half rests on fewer fixtures, and
deleting one of its goldens breaks nothing. The case still runs, still passes, and quietly
stops making the claim, because a suite can only check the assertions it still has.
test/tools/lib-coverage.shnow pins the shape — both goldens per TypeScript lib case,a golden per java stub case, and floors that cannot fall without being lowered in the same
commit. No parser, no solver, runs in seconds.
Both torture harnesses also already assert library monotonicity — solving the same
client against an empty library must never produce MORE answers than solving it against
the real one. That is the property most worth having and it is easy to lose silently.
Still open, not blocking this PR
that is not in version control anywhere, so the python leg is goldens-only. Weaker than
the other two legs, and the workflow says so where it matters.
01-inheritance-overrideflaked once — failed inside a full run, then passed alone,on re-run, and again under
--oracle. A cross-language solver-cache collision wasprobed and disproved. A flake erodes a merge gate faster than a missing test does, so
this deserves its own issue.
and on a cache hit
run-souffle.shnever invokes the solver at all — so a prebuiltengine would free consumers from needing either Souffle or a C++ toolchain. It does not
help this repo's CI, which must compile the rules as changed in the PR; a prebuilt
binary there would score rules that predate the change under review. Two notes if it is
taken up:
-march=native(run-souffle.sh:285, :352) would have to go, and upstream shipsx86_64 Linux only, so macOS, Windows and arm64 builds would be ours to produce and
maintain.
ci: gate main on the three regression suites, and make a skipped suit…