Skip to content

Evals: a volume case for the communication rule, many done items and one buried problem #40

Open

Description

@sabind

Before you file

  • I searched the existing issues.
  • I read evals/README.md.

Component

communication (rule)

Behaviour

The four cases under evals/rules/communication/ score 1.00 with and without the plugin (PR #31's table, Δ 0.00 on each), so they guard the rule's wording against regression but cannot show that the rule changes behaviour. The transcripts that motivated the rule showed the burying under volume: a hand-back with many done items and one problem in the middle, where a fresh session with one small task leads with the failure either way. The missing case is that volume case: a scaffold whose state gives the model a long list of things that went right and one thing that went wrong, graded on the problem being the first thing written.

Grading

A regex grader on the final message: the problem, named by what it is, appears before any done item. A second grader that the done items are present, so leaving them out is not a pass. Run with EVAL_ARGS="--ablation with-without" and expect a Δ above zero; if the baseline also leads with the problem at this volume, the case says so and the rule's effect stays unproved.

Prompt

A task on the calc fixture that produces, say, eight passing changes and one failing check or one carried finding, then asks for the hand-back. The exact shape is for the case's author; the point is volume with one buried problem.

Activity

  1. added
    qualityThe behaviour of the skills, agents and rules, and the evals and checks that guard it
    triageNot yet classified by a maintainer
    on Oct 16, 2026
  2. added theissue type on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    qualityThe behaviour of the skills, agents and rules, and the evals and checks that guard ittriageNot yet classified by a maintainer

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions