Repository navigation
Shape: refuse duplicate provisional commands and narrow drafting guidance - #470
Conversation
…'s (#469) The drafting prompt now says a provisional's optional cmd selects only its missing regression by name, in a runner form that fails when nothing matches, or is left out. A new draft whose provisional proposes a command another check also uses is refused with both check numbers; stored seeds read unchanged. The recorded #296 call 8 reply (three provisionals all `python3 -m unittest discover`) now hears this refusal alongside its review fault. Refs #469 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj
jon-devlapaz
left a comment
There was a problem hiding this comment.
Independent Codex review of 536157b found one regression to fix before merge: command identity ignores cwd. The 16 existing reply-contract tests pass. An additional fake-provider/real-ledger probe using two targeted provisional checks with the same command in package_a and package_b refuses on this commit (exit 101), but drafts successfully with the base commit's validation (exit 0). The isolated probe changes were restored byte-for-byte. Stored-seed reads appear unaffected because the added rule is limited to draft validation. Full verification and the before/after model evaluation remain pending; this is not acceptance of those results.
| else { | ||
| continue; | ||
| }; | ||
| if let Some(other) = (0..commands.len()).find(|&i| i != index && commands[i] == Some(cmd)) { |
There was a problem hiding this comment.
[P2] Include cwd when comparing proposed commands. Two packages may both propose python -m pytest tests/test_lists.py::test_custom_conjunction, one with cwd package_a and the other with cwd package_b. These invoke different tests, but this comparison rejects the draft and tells Shape to drop a valid proposal. I reproduced the refusal through the fake-provider → Shape step → real ledger path; the same probe drafts with the base validator. Compare the (cmd, cwd) pair and add a regression showing distinct directories can draft while the same command in the same directory is still refused.
There was a problem hiding this comment.
Fixed in 5a3fb93. The rule now compares the (cmd, cwd) pair, and the refusal names the cwd. A new case in a_provisional_proposing_another_checks_command_is_refused_with_a_plain_reason checks that the same command with cwd TEST-package-b drafts, and the same command in the same cwd is still refused. Reversion probe: comparing cmd alone makes the new case fail (elsewhere refused). The docs follow in b548853.
Generated by Claude Code
…view) The same targeted command run from two package directories selects different tests, so it now drafts; the same command in the same cwd is still refused, and the reason names that cwd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj
The refusal text is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj
jon-devlapaz
left a comment
There was a problem hiding this comment.
The command/cwd finding is resolved, including the prompt wording in f3e990f. I reran the 16 reply-contract tests plus my independent two-package probe on 93d9e3a: 17 passed. That probe failed on 536157b and passed with the base validator. The only subsequent change through f3e990f is the matching prompt qualification, which I reviewed. No remaining code-review blocker found in this bounded change. This is not a merge clearance yet: the completed full check, CI, and real before/after evaluation on the final prompt revision still need review. The heuristic remains intentionally narrow and does not itself prove that a unique command tests the requested behavior.
|
CI on f3e990f has one failure, and I don't think it's from this PR. The failing check is Why I think it's unrelated:
No fix exists yet, because this looks like a readiness-timing wait in the fixture. I'm re-running the failed job once. If it fails again, I'll treat it as real and dig in. Generated by Claude Code |
jon-devlapaz
left a comment
There was a problem hiding this comment.
The separate authorized end-to-end humanize trial found a remaining failure of #469's intended behavior on f3e990f. After one malformed completion-review response and an explicit retry, Shape drafted one provisional with cmd: "python -m pytest tests/test_lists.py -k natural_list", cwd: ".", and missing: "A behavior-specific custom-conjunction test and its invocation do not yet exist in the inspected tests; add the regression before this check can run." I ran that exact command on untouched humanize baseline 6b817b963dbb091b9ad20a746cb6a26653a93b95: exit 0, 14 existing tests passed; no custom-conjunction implementation or regression was present. This selects the existing function's tests rather than a named missing regression. The seed is still unconfirmed. Please count this as a failed targeting sample and address it before declaring #469 complete. A command looking narrower than the whole suite is not sufficient; the evaluation should execute it against the pre-feature checkout and check the actual failure reason. This observation used the separate trial budget, not the #469 evaluation allowance.
|
The operator has asked Codex to take over because Claude is unavailable. I am continuing #469 on this PR, with independent review, and accounting for the existing evaluation budget before further live calls. Please do not push or resume the earlier evaluation concurrently. The outstanding live finding is unchanged: the proposed |
jon-devlapaz
left a comment
There was a problem hiding this comment.
Independent release review of this exact head found no blocking code defect in the limited guard-and-guidance change. The duplicate-command guard has public-interface and reversion evidence; same-command/different-directory behavior is preserved. Full local verification and CI passed.
The live evaluation does not complete #469: humanize 0/2 drafts, pi-skimmer one draft with 3/3 provisionals omitting commands, zero targeted regressions observed. Completion-review refusals occurred in unchanged validation; prompt effects on completion remain uncertain. Recommendation: merge this narrower change, preserve the live limitations, and leave #469 open. This is agent technical review, not human acceptance.
Refs #469. This ships the duplicate-command guard and narrower drafting guidance; the runnable-regression outcome remains open on the issue.
Shape sometimes proposed a whole suite, or
pytest -k natural_list, for a regression that did not exist. The latter passed all 14 existing humanize tests before the requested feature existed.The drafting prompt now asks for a proposed new regression identifier and its assertions after inspecting existing tests. It selects that exact identifier using the repository's runner, or omits the command when selection or missing-test behavior is unknown. Suite checks remain separate. The ledger also refuses a new provisional that repeats another check's exact command in the same working directory; the same command in a different directory remains valid. Stored seeds are unchanged.
The rule stays at the existing ledger boundary (architecture diagnostics 3 and 6); no schema or UI is added. The pinned Seed Me contract is unchanged.
Validation on
0c30f39c3d289e9b7947879b6cf86a9877af3c5b:Limits: the ledger compares command and directory bytes; it does not interpret arbitrary commands or prove semantic coverage. Prompt instructions are not a semantic validator. The observed live samples do not establish general model reliability or reduced operator effort.
Claude began this change; Codex took over at the operator's request when Claude became unavailable.