Skip to content

fix(workflows): skip the right side of and/or when the left side decides - #4923

Open
kartsan03 wants to merge 1 commit into
github:mainfrom
kartsan03:fix/workflow-boolean-short-circuit
Open

kartsan03 wants to merge 1 commit into
github:mainfrom
kartsan03:fix/workflow-boolean-short-circuit

Conversation

@kartsan03

Copy link
Copy Markdown
Contributor

Description

Workflow expressions evaluate both operands of and and or before combining them (_evaluate_simple_expression, expressions.py:884-894 on main). So a guard can't protect the operand after it. The usual way to read an optional JSON input fails when the input is left empty:

inputs:
  payload:
    type: string
    default: ""
steps:
  - id: check
    type: if
    condition: "{{ inputs.payload != '' and (inputs.payload | from_json | length) }}"
    then:
      - id: provided
        type: shell
        run: echo provided
    else:
      - id: absent
        type: shell
        run: echo absent

inputs.payload != '' is false, but from_json still runs on the empty string and the run fails before either branch. inputs.payload == '' or (...) fails the same way.

This PR returns as soon as the left operand decides the result: and skips its right side when the left is false, and or when the left is true. Results are still booleans, precedence and parentheses are unchanged, and an operand that does get evaluated still fails the run on an error.

The validation probes behind condition corrections (_evaluator_rejects, _collect_leaves) rely on visiting both operands. test_both_sides_of_a_boolean_are_reported exists to catch a short-circuit that would blind them. The probes now pass a _ProbeNamespace as the outer namespace, and the evaluator keeps visiting both operands for it, so an unknown name or a misused filter in a skipped operand is still reported. Root lookups use dict.get(), so the wrapper doesn't change what resolves. With a plain short-circuit and no probe exception, 10 probe tests fail, including the existing test_both_sides_of_a_boolean_are_reported.

docs/reference/workflows.md gets a paragraph on how and/or evaluate, with this guard as the example.

Repro with the workflow above:

Command main (0443760) This PR
specify workflow run optional.yml Error: from_json: invalid JSON: Expecting value: line 1 column 1 (char 0), exit 1 [absent] runs, Status: completed, exit 0
specify workflow run optional.yml --input payload=[1] [provided] runs, exit 0 same
specify workflow run optional.yml --input payload=broken from_json: invalid JSON, exit 1 same (the right side is needed, so it still fails)

The bundled workflows don't use and/or today, so this affects workflows that users write.

Testing

  • Tested locally with uv run specify --help
  • Ran existing tests with uv sync && uv run pytest
  • Tested with a sample project (if applicable)

New tests:

  • tests/test_workflows.py::TestExpressions: guards with and, or, mixed precedence and nesting short-circuit on an empty payload; a needed operand, on either side, still raises.
  • tests/unit/test_condition_expression_block.py: the probes still collect leaves and reject unknown or misused filters in operands that runtime skips (true or (...), false and (...), nested).
  • tests/specify_cli/workflows/test_command_run.py: specify workflow run --json in process, with real shell steps and persisted state, for both guards with an empty, a valid and an invalid payload.

With only src/ reverted to main, 5 of the 24 new or changed tests fail: 3 expression cases and the 2 empty-payload CLI cases. With the fix, all 24 pass.

Full suite on Linux, Python 3.13: 11435 passed, 264 skipped (uv sync --extra test, then pytest in that venv, 12m36s). Ruff 0.15.0 (ruff check src tests) passes, and markdownlint passes on docs/reference/workflows.md.

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (fill in the disclosure below)

AI disclosure: Two agents, both autonomous. Codex CLI (model: gpt-6-astra, reasoning effort xhigh, autonomous, sandboxed and offline) found this while scouting the workflow expression evaluator, wrote the reproduction, and generated the fix, tests and docs paragraph. Claude Code (model: claude-opus-5-5, autonomous) checked that the probe marker doesn't change name resolution, moved and rewrote the docs paragraph and the _ProbeNamespace docstring, reran the tests against main and the fix (plus the plain short-circuit variant above), reran the repro with the real specify binary, ran the full suite and lint, and drafted this description. Commits carry Assisted-by: trailers.

`and` and `or` evaluated both operands before combining them, so a guard
could not protect the operand after it. With an empty optional input,
`inputs.payload != '' and (inputs.payload | from_json | length)` still ran
`from_json` on the empty string and failed the run with "invalid JSON".

Return as soon as the left operand decides the result. The validation
probes behind condition corrections (`_evaluator_rejects`,
`_collect_leaves`) now pass a `_ProbeNamespace` as the outer namespace and
keep evaluating both operands, so an unknown name or misused filter in a
skipped operand is still reported.

Assisted-by: Codex CLI (model: gpt-6-astra, autonomous)
Assisted-by: Claude Code (model: claude-opus-5-5, autonomous)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

馃煝 Approval recommended

The implementation is consistent, documented, and covered by positive, negative, probe, and end-to-end regression tests.

0 open findings

What changed in this PR

Implements lazy boolean evaluation in workflow expressions while preserving exhaustive validation probes.

Changes:

  • Short-circuits and/or at runtime.
  • Preserves full operand traversal during validation.
  • Adds unit, CLI regression, and documentation coverage.

The submitted regression evidence was reviewed; tests were not rerun in this environment.

File Description
src/鈥媠pecify_cli/鈥媤orkflows/鈥媏xpressions.py Implements short-circuiting and probe behavior.
tests/鈥媡est_workflows.py Covers expression semantics and failures.
tests/鈥媢nit/鈥媡est_condition_expression_block.py Verifies exhaustive validation probes.
tests/鈥媠pecify_cli/鈥媤orkflows/鈥媡est_command_run.py Adds end-to-end workflow regression coverage.
docs/鈥媟eference/鈥媤orkflows.md Documents boolean evaluation behavior.

馃 Review effort: Balanced


馃挕 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants