[Feature]: Triage agent that separates real regressions from stale tests, using the spec as source of truth #43675

🚀 Feature Request

Add a Playwright Test agent that runs before the healer and decides why a test is failing. It would return one of four verdicts:

• Regression: the app no longer does what the spec says. Report it and leave the test alone.
• Stale test: the app still matches the spec, but the test is out of date. Hand off to the healer.
• Stale spec: the app changed on purpose and the spec needs updating. Flag it for a human.
• Environment or flaky: network, credentials, backend errors, or timing.

It would use the same evidence tools the healer already has (snapshot, console messages, network requests, test debug), plus the spec the test was generated from.

Example

1. The planner writes specs/checkout.md, and the generator produces checkout.spec.ts with a header comment linking to that spec section.
2. A change breaks the order confirmation, and the test fails.
3. Today, the healer can end up adjusting the assertion until the test passes, because it has no independent reference for the correct behavior.
4. With triage, the agent reads the spec ("after payment, the user sees an order confirmation"), inspects the page and network, and reports: "Regression: the confirmation page returns a 500 after payment. The test is correct." The test is not edited.
5. If instead a button was only renamed, the verdict is "Stale test" and the healer fixes the locator.

The verdict could also be structured output, such as JSON, so CI can post it as a PR comment.

Motivation

The healer is built to get failing tests passing. Its prompt says to act non-interactively and do the most reasonable thing to pass, and its remediation includes fixing assertions and expected values. That works for stale locators and timing issues, but when the application really regressed, the likely result is a test that has been quietly adjusted to match the bug. Guides to the healer already warn that auto-healing a test that exposes a real bug hides the bug.

Planner and generator already produce human-readable specs linked to tests, so the information needed to judge intent already exists. Using it would let Playwright's agents answer the question teams ask after every red build: did the app break, or does the test need updating?

I'm happy to prototype this as a standalone agent file first and share results on seeded regressions if there's interest.