Skip to content

Shape: refuse duplicate provisional commands and narrow drafting guidance - #470

Merged
jon-devlapaz merged 7 commits into
mainfrom
claude/project-thread-8nx7v9
Oct 28, 2026
Merged

jon-devlapaz merged 7 commits into
mainfrom
claude/project-thread-8nx7v9

Conversation

@jon-devlapaz

@jon-devlapaz jon-devlapaz commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Refs #469. This ships the duplicate-command guard and narrower drafting guidance; the runnable-regression outcome remains open on the issue.

Shape sometimes proposed a whole suite, or pytest -k natural_list, for a regression that did not exist. The latter passed all 14 existing humanize tests before the requested feature existed.

The drafting prompt now asks for a proposed new regression identifier and its assertions after inspecting existing tests. It selects that exact identifier using the repository's runner, or omits the command when selection or missing-test behavior is unknown. Suite checks remain separate. The ledger also refuses a new provisional that repeats another check's exact command in the same working directory; the same command in a different directory remains valid. Stored seeds are unchanged.

The rule stays at the existing ledger boundary (architecture diagnostics 3 and 6); no schema or UI is added. The pinned Seed Me contract is unchanged.

Validation on 0c30f39c3d289e9b7947879b6cf86a9877af3c5b:

  • Full zero-call check passed: Tinkery Rust library 818, Node 39, all 52 Python files (including terminal journeys), formatting and Clippy. Workspace and integration Rust tests also passed.
  • The duplicate-command regression turns red with the rule removed. Independent review reproduced and caught the original different-directory false rejection; the revised public-interface case passes.
  • Independent review of the final prompt and contract found no blocker.
  • Live counterexample retained: the former proposal passes 14 baseline humanize tests, without the feature.
  • Final CI passed on this exact head.
  • Corrected-prompt evaluation used 10 of 12 actual provider requests. Humanize: 0 of 2 attempts produced a draft (both refused malformed completion review). Pi-skimmer: one draft; all 3 provisionals truthfully omitted commands for missing tests. Zero targeted regression commands were observed. Its separate baseline typecheck and formatting commands passed, which proves existing health only.
  • The remaining 2 requests were preserved. The earlier Claude evaluation did not retain a count of actual provider dispatches; its apparent remaining allowance was not reused.
  • Independent release review recommends this limited guard-and-guidance change, while keeping Shape: propose a command that targets the missing regression, not the whole suite #469 open. The two-task redraft criterion and improved runnable regression generation remain unproven.

Limits: the ledger compares command and directory bytes; it does not interpret arbitrary commands or prove semantic coverage. Prompt instructions are not a semantic validator. The observed live samples do not establish general model reliability or reduced operator effort.

Claude began this change; Codex took over at the operator's request when Claude became unavailable.

jon-devlapaz and others added 2 commits October 28, 2026 03:29
…'s (#469)

The drafting prompt now says a provisional's optional cmd selects only
its missing regression by name, in a runner form that fails when
nothing matches, or is left out. A new draft whose provisional proposes
a command another check also uses is refused with both check numbers;
stored seeds read unchanged. The recorded #296 call 8 reply (three
provisionals all `python3 -m unittest discover`) now hears this refusal
alongside its review fault.

Refs #469

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj

@jon-devlapaz jon-devlapaz left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent Codex review of 536157b found one regression to fix before merge: command identity ignores cwd. The 16 existing reply-contract tests pass. An additional fake-provider/real-ledger probe using two targeted provisional checks with the same command in package_a and package_b refuses on this commit (exit 101), but drafts successfully with the base commit's validation (exit 0). The isolated probe changes were restored byte-for-byte. Stored-seed reads appear unaffected because the added rule is limited to draft validation. Full verification and the before/after model evaluation remain pending; this is not acceptance of those results.

Comment thread src/seed_me/ledger/checks.rs Outdated
else {
continue;
};
if let Some(other) = (0..commands.len()).find(|&i| i != index && commands[i] == Some(cmd)) {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Include cwd when comparing proposed commands. Two packages may both propose python -m pytest tests/test_lists.py::test_custom_conjunction, one with cwd package_a and the other with cwd package_b. These invoke different tests, but this comparison rejects the draft and tells Shape to drop a valid proposal. I reproduced the refusal through the fake-provider → Shape step → real ledger path; the same probe drafts with the base validator. Compare the (cmd, cwd) pair and add a regression showing distinct directories can draft while the same command in the same directory is still refused.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5a3fb93. The rule now compares the (cmd, cwd) pair, and the refusal names the cwd. A new case in a_provisional_proposing_another_checks_command_is_refused_with_a_plain_reason checks that the same command with cwd TEST-package-b drafts, and the same command in the same cwd is still refused. Reversion probe: comparing cmd alone makes the new case fail (elsewhere refused). The docs follow in b548853.


Generated by Claude Code

jon-devlapaz and others added 4 commits October 28, 2026 03:37
…view)

The same targeted command run from two package directories selects
different tests, so it now drafts; the same command in the same cwd is
still refused, and the reason names that cwd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj
The refusal text is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KwScoLKgvmHqpyHGAqVCZj

@jon-devlapaz jon-devlapaz left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The command/cwd finding is resolved, including the prompt wording in f3e990f. I reran the 16 reply-contract tests plus my independent two-package probe on 93d9e3a: 17 passed. That probe failed on 536157b and passed with the base validator. The only subsequent change through f3e990f is the matching prompt qualification, which I reviewed. No remaining code-review blocker found in this bounded change. This is not a merge clearance yet: the completed full check, CI, and real before/after evaluation on the final prompt revision still need review. The heuristic remains intentionally narrow and does not itself prove that a unique command tests the requested behavior.

Copy link
Copy Markdown
Owner Author

CI on f3e990f has one failure, and I don't think it's from this PR.

The failing check is check (ubuntu-latest). The failing test is sdlc::tests::failed_commands_stop_descendants_and_preserve_results, which panics at tests/support/command_descendant.rs:72 with "the fake descendant signalled readiness", run under tests/test_rust_test_homes.py.

Why I think it's unrelated:

  • The diff touches only src/seed_me/ledger/checks.rs, src/shape/prompt.txt, src/shape/step/reply_contract_tests.rs and the design doc. None of those reach the sdlc command runner or its descendant fixture.
  • The same commit passed the full check locally: Rust lib 818, Node 39, Python 52/52, fmt and clippy.

No fix exists yet, because this looks like a readiness-timing wait in the fixture. I'm re-running the failed job once. If it fails again, I'll treat it as real and dig in.


Generated by Claude Code

@jon-devlapaz jon-devlapaz left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The separate authorized end-to-end humanize trial found a remaining failure of #469's intended behavior on f3e990f. After one malformed completion-review response and an explicit retry, Shape drafted one provisional with cmd: "python -m pytest tests/test_lists.py -k natural_list", cwd: ".", and missing: "A behavior-specific custom-conjunction test and its invocation do not yet exist in the inspected tests; add the regression before this check can run." I ran that exact command on untouched humanize baseline 6b817b963dbb091b9ad20a746cb6a26653a93b95: exit 0, 14 existing tests passed; no custom-conjunction implementation or regression was present. This selects the existing function's tests rather than a named missing regression. The seed is still unconfirmed. Please count this as a failed targeting sample and address it before declaring #469 complete. A command looking narrower than the whole suite is not sufficient; the evaluation should execute it against the pre-feature checkout and check the actual failure reason. This observation used the separate trial budget, not the #469 evaluation allowance.

Copy link
Copy Markdown
Owner Author

The operator has asked Codex to take over because Claude is unavailable. I am continuing #469 on this PR, with independent review, and accounting for the existing evaluation budget before further live calls. Please do not push or resume the earlier evaluation concurrently.

The outstanding live finding is unchanged: the proposed python -m pytest tests/test_lists.py -k natural_list passes all 14 baseline tests without the conjunction feature. I am making a narrow prompt correction and will report the actual baseline result of the replacement check, final verification, and budget use before merge.

@jon-devlapaz jon-devlapaz changed the title Shape: a provisional proposes a targeted command, never another check's (#469) Shape: target missing regressions in provisional checks (#469) Oct 28, 2026
@jon-devlapaz jon-devlapaz changed the title Shape: target missing regressions in provisional checks (#469) Shape: refuse duplicate provisional commands and narrow drafting guidance Oct 28, 2026

@jon-devlapaz jon-devlapaz left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent release review of this exact head found no blocking code defect in the limited guard-and-guidance change. The duplicate-command guard has public-interface and reversion evidence; same-command/different-directory behavior is preserved. Full local verification and CI passed.

The live evaluation does not complete #469: humanize 0/2 drafts, pi-skimmer one draft with 3/3 provisionals omitting commands, zero targeted regressions observed. Completion-review refusals occurred in unchanged validation; prompt effects on completion remain uncertain. Recommendation: merge this narrower change, preserve the live limitations, and leave #469 open. This is agent technical review, not human acceptance.

@jon-devlapaz
jon-devlapaz marked this pull request as ready for review October 28, 2026 04:27
@jon-devlapaz
jon-devlapaz merged commit 276472c into main Oct 28, 2026
5 checks passed
@jon-devlapaz
jon-devlapaz deleted the claude/project-thread-8nx7v9 branch October 28, 2026 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant