[potiuk]

What

A full eval run over the 53 skills touched by #1532 and #1533 turned up cases that pass and fail on identical code.
This PR removes that variance.
Each case is fixed at the layer that caused it, and every case passes all of its repeat runs afterwards.

Case	Cause	Fix	Failures before	After
audit-finding-fix step 7 case-3	"how many findings" read as satisfied by describing the edits	skill + output spec: an explicit number	1/15	0/18
issue-deduplicate step 2 case-3	grader reply with no parseable JSON failed every field	runner: one retry on an unparseable reply; real JSON decoder as a last-resort extractor	1/5	0/18
issue-reproducer step 3 case-1	the project's own DB session read as an "external fixture" (shape D)	output spec: A covers the project's runtime	1/15	0/18
pairing-self-review step 2 case-10	no rule for a diff that already carries the release marker	skill: marker present + compatible graph means no finding	1/5	0/18
pr-management-mentor tone case-13	exact-quote match although the skill asks for "the phrase"	regex property for the offending sentence	1/15	0/18
release-keys-sync step 0 case-3	the append-or-replace hand-off lived only in troubleshooting	Step 0 states it	1/5	0/18
security-issue-triage step 3 case-4	the fixture report contradicted itself (2.9 code path vs 3.x API)	coherent fixture	2/5	0/8
security-issue-triage step 3 case-2	the fixture claimed connections are DAG-scoped	rewritten as a cross-DAG XCom read	4/5	0/8
security-issue-sync step 5b case-5	the expected reason restated fields checked elsewhere	expected reason states only the deciding rule	1 in 4	0/8
setup-isolated-setup-verify step 1 case-3	"sourced" ambiguous when the sourced script is missing	skill: the rc source line decides check 4; a missing script is check 3's finding	3/3 after #1533, 1/3 on main	0/5
skill-reconciler step 2 case-5	an embedded injection classified as a safety-baseline failure	skill: an embedded instruction is drift plus an injection flag when the clause wording matches	3/3	0/5

pairing-self-review case-1-clean-diff failed every run, also before #1532.
Its diff was not actually clean: it had a behaviour-changing refactor and a no-op branch.
It becomes case-1-test-coverage-gap-only: a correct two-line guard whose only acceptable finding is the missing test.

The whole affected steps pass when run once in full.
The safety-baseline cases still fail where a clause really is missing; for example, skill-reconciler step 2 passes 8/8, including cases 4, 6, 7 and 8.

Harness

_call_grader is the single entry point for the three grader paths.
It retries once, and only when the reply holds no parseable JSON.
A parsed NO is never re-asked, so a NO cannot become a YES; timeouts and non-zero exits still fail at once.
7 new tests cover it, and the 6 that exercise new behaviour fail on the old runner.

After this merges, the user-scope runner copy in ~/.claude/scripts/skill_evals/ needs refreshing (/magpie-setup-isolated-setup-update reports the drift), or wrapper runs will not get the retry.

Not fixed here

These fail the same way before #1532, so they are not caused by the config-home series.
They are left for follow-ups:

• pairing-self-review case-13: its blocking correctness finding sometimes omits the fix;
• preflight-floor cases 1, 6, 7 and 8 (case-6 references an undefined assertion);
• setup-status step 3 cases 3 and 4, which expect setup adopt where the skill says setup install.

Testing

• prek run --all-files is green.
• tools/skill-evals pytest is green; mypy reports no new errors.
• Repeat runs as in the table, plus one full run of every affected step.

---

Gen-AI disclosure: prepared with the assistance of Claude Code (Claude Opus 5); reviewed by the author.

🤖 Generated with Claude Code