[Copilot]

Summary

Adds outcome-backed experiment metrics so gh aw experiments analyze <workflow> --outcome can compare real GitHub acceptance and human-intervention outcomes without declaring a grader or eval, or adding a scheduled collector.

• Supports outcome:accepted and outcome:zero_touch as primary, secondary, and guardrail metrics, with compiler/schema validation.
• Makes live outcome collection explicitly opt-in with --outcome because it downloads artifacts and makes additional GitHub API requests. Without the flag, an outcome-backed metric produces an actionable error before collecting metric evidence; workflows without outcome metrics retain their existing behavior.
• Adds outcome_window: auto (the default), learning a pooled 95th-percentile decision time across variants using Kaplan–Meier censoring for still-open outputs. The learned window is bounded to 24 hours–30 days, with a 72h cold-start fallback until 20 outputs have known decision times. A positive Go duration provides an explicit override.
• Joins on-demand outcome evidence to persisted experiment assignments and reports per-variant observations, coverage, secondary comparisons, and effective-window metadata in text and JSON.
• Automatically selects a two-proportion test for binary per-run values or Welch’s t-test for fractional values. Explicit proportion_test and bayesian_ab require binary observations.
• Updates experiment documentation, the draft ADR, proposer guidance, and generated workflow output. Adds focused compiler, schema, outcome-observation, opt-in, serialization, and host-handling tests.

Measurement semantics

Only runs at least the effective maturity window contribute observations. Each run contributes one value: accepted or zero-touch outputs divided by resolved outputs. Pending outputs beyond the window score zero; unknown/errored outcomes are excluded. Lifecycle, ignored, and skipped outputs do not contribute, and runs without outcome-bearing outputs are no-ops rather than successes. Only usable run observations count toward min_samples.

Zero-touch requires a merged PR with no non-bot comments, reviews, or commits after creation. Accepted PRs with incomplete zero-touch evidence are excluded from that metric; other resolved output types remain in its denominator and score zero. Evidence completeness is propagated through outcome reports and JSON schemas, and outcome evaluation rejects repository hosts that do not match the workflow run host.

Coverage distinguishes immature, stale_pending, unavailable, and no_op runs. These are per-run flags: a partially observed run may contribute a measurement and also count as stale pending or unavailable.

Validation status

Merged the refreshed origin/main (33 incoming commits), resolving the sole conflict by regenerating the proposer workflow lock file. The merge and opt-in changes are committed and pushed.

• Build, formatting, standard change-scoped Go lint, schema freshness, and impacted sampled Go tests passed.
• Experiment/outcome tests across pkg/cli, pkg/parser, and pkg/workflow passed using a nonsymlink temporary directory. The initial run hit an unrelated macOS /var symlink restriction in a Harbor export test.
• All 359 workflows compiled successfully with validation.
• The full progress gate remains blocked by 13 pre-existing custom-linter findings in the experiment/compiler code. Linter findings introduced by the opt-in change were fixed; the remaining findings predate it.

• Fixes experiments: support outcome-backed metrics (outcome:accepted, outcome:zero_touch) #74988

[pelikhan]

@copilot apply #74988 (comment)

[github-actions]

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: gpt56 max · ~1 AIC · ⌖ ~1 AIC · ⊞ 6.3K · ◷

🧪 Test quality analysis by Test Quality Sentinel

[github-actions]

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

[github-actions]

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

[github-actions]

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt56 max · ~7 AIC · ⌖ ~1 AIC · ⊞ 118 · ◷

Generated by Ponytail Reviewer for #74992

[github-actions]

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)

reviewed_at: 2026-11-05T15:36:02Z
review_event: COMMENT
top_themes: [secondary outcome metrics silently omitted, historical schema compatibility]
files_reviewed: [pkg/cli/experiments_outcome_observations.go, schemas/audit.schema.json, schemas/logs-jsonl.schema.json, schemas/logs.schema.json, pkg/workflow/compiler_experiments.go]
comment_count: 2

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~4 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷
_{Comment /review to run again}

[github-actions]

Review completed with two actionable correctness/compatibility findings.

Highlights

• Outcome-backed secondary_metrics are accepted by validation but omitted from outcome collection/analysis.
• Requiring zero_touch_evidence_complete in persisted log schemas breaks validation of historical records that predate this field.

The specialist pass was unavailable because its environment could not read the pre-fetched diff; findings above are from the independent source review. No tests were run per the static-review constraints.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~4 AIC · ⌖ ~1 AIC · ⊞ 1.6K
_{Comment /review to run again}

[github-actions]

🏗️ Design Decision Gate — ADR Required

This PR makes significant changes to core business logic (1,029 new lines in pkg/) but does not have a linked Architecture Decision Record (ADR). The PR body and linked issue #74988 contain a proposal, not an ADR with Context, Decision, Alternatives Considered, and Consequences sections.

📄 Draft ADR committed: docs/adr/74992-outcome-backed-experiment-metrics-with-fixed-maturity-window.md — review and complete it before merging.

🔒 This PR cannot merge until an ADR is linked in the PR body.

Key decision captured: add outcome:accepted and outcome:zero_touch experiment metrics. The experiments analyze CLI computes them on demand with a fixed, configurable outcome_window (default 72h). Issue #74988 proposed a window learned per workflow with Kaplan–Meier (outcome_window: auto), but this PR doesn't implement that. Please confirm in the ADR whether learned windows are deferred to a follow-up or rejected (marked [TODO: verify]).

📋 What to do next

1. Review the draft ADR committed to your branch. It was generated from the PR diff and issue experiments: support outcome-backed metrics (outcome:accepted, outcome:zero_touch) #74988.

2. Complete the missing sections: add context the AI couldn't infer, refine the decision rationale (especially fixed vs. learned window), and list real alternatives you considered.

3. Commit the finalized ADR to docs/adr/ on your branch.

4. Reference the ADR in this PR body by adding a line such as:

   ADR: ADR-74992: Outcome-Backed Experiment Metrics with a Fixed, Configurable Maturity Window (link unavailable: invalid HTTPS URL)

Once an ADR is linked in the PR body, this gate will re-run and check that the implementation matches the decision.

❓ Why ADRs Matter

"AI made me procrastinate on key design decisions. Because refactoring was cheap, I could always say 'I'll deal with this later.' Deferring decisions corroded my ability to think clearly."

ADRs keep a searchable, permanent record of why the codebase looks the way it does. Future contributors (and your future self) will thank you.

📋 Michael Nygard ADR Format Reference

An ADR must contain these four sections to be considered complete:

• Context: What is the problem? What forces are at play?
• Decision: What did you decide? Why?
• Alternatives Considered: What else could have been done?
• Consequences: What are the trade-offs (positive and negative)?

All ADRs are stored in docs/adr/ as Markdown files numbered by PR number (e.g., 0042-use-postgresql.md for PR #42).

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · pi · routed: opus55 xhigh · ~43 AIC · ⌖ ~2 AIC · ⊞ 7.7K · ◷
_{Comment /review to run again}

[Copilot]

🟡 Changes recommended

Unvalidated artifact-derived hosts can expose enterprise tokens, a contradictory test expectation fails deterministically, and automatic censor-aware maturity windows are missing.

3 open issues

• Validate artifact-derived API hosts · New
• Fix duplicate create_issue expectation · New
• Support learned auto maturity windows · New

What changed in this PR

Adds live GitHub outcome metrics to gh aw experiments analyze, extending experiment measurements beyond grader and eval results toward the goals of #74988.

Changes:

• Adds outcome metric validation and configurable maturity durations.
• Collects per-run outcomes, tracks evidence completeness and coverage, and reuses existing statistical analysis.
• Updates schemas, focused tests, documentation, and experiment-proposer guidance.

File	Description
schemas/​logs.schema.json	Adds zero-touch evidence completeness to log reports.
schemas/​logs-jsonl.schema.json	Adds evidence completeness to JSONL records.
schemas/​audit.schema.json	Adds evidence completeness to audit outcomes.
pkg/​workflow/​frontmatter_types.go	Adds experiment maturity-window configuration.
pkg/​workflow/​compiler_experiments.go	Validates outcome references and durations.
pkg/​workflow/​compiler_experiments_outcome_test.go	Tests outcome compiler validation.
pkg/​parser/​schemas/​main_workflow_schema.json	Defines outcome-window schema and metric descriptions.
pkg/​parser/​schema_experiments_outcome_test.go	Tests outcome frontmatter schema validation.
pkg/​cli/​outcomes_episodes.go	Includes evidence completeness in observation equality.
pkg/​cli/​outcomes_episodes_test.go	Tests conflicting evidence-completeness values.
pkg/​cli/​outcome_eval.go	Adds the evidence-completeness report field.
pkg/​cli/​outcome_eval_pr.go	Separates incomplete evidence from human intervention.
pkg/​cli/​outcome_eval_jsonl.go	Serializes evidence completeness.
pkg/​cli/​outcome_eval_jsonl_test.go	Extends round-trip fixtures with completeness values.
pkg/​cli/​experiments_outcome_observations.go	Collects, scores, and reports outcome observations.
pkg/​cli/​experiments_outcome_observations_test.go	Tests scoring, maturity, collection, and coverage.
pkg/​cli/​experiments_grader_statistics.go	Handles outcome sets without observations.
pkg/​cli/​experiments_grader_observations.go	Adds outcome coverage to shared observation sets.
pkg/​cli/​experiments_command.go	Merges outcome observations into experiment analysis.
pkg/​cli/​experiments_analyze_statistics.go	Exposes primary and guardrail outcome coverage.
docs/​src/​content/​docs/​experimental/​experiments.md	Documents outcome metric usage and interpretation.
docs/​src/​content/​docs/​experimental/​experiments-specification.md	Specifies outcome validation and analysis semantics.
.github/​workflows/​daily-harness-experiment-proposer.md	Allows outcome metrics in experiment proposals.
.github/​workflows/​daily-harness-experiment-proposer.lock.yml	Refreshes compiled workflow body hashes.

_{Review effort: Balanced (auto)}

[github-actions]

Skills-Based Review 🧠

Applied /tdd, /codebase-design and /diagnosing-bugs. Requesting changes because the focused outcome tests fail on the PR head.

📋 Key Themes & Highlights

Key Themes

• Failing test on head: TestOutcomeMetricRunValue/non_PR_acceptance_has_unknown_zero_touch_evidence expects Unavailable: 1, but the code correctly keeps resolved non-PR outputs in the denominator at zero. The spec agrees with the code. Fix the expectation, see the inline comment.
• Implicit per-run coverage flags: Unavailable/StalePending are set with =, which is correct but undocumented. A one-line comment would help.
• Empty-set metric type: graderObservationMetricType returns binary for coverage-only sets with no observations. Unverified impact, needs a test.

Positive Highlights

• ✅ Zero-touch excludes incomplete PR evidence rather than scoring zero, matching the spec
• ✅ Outcome reference parsing and validation have focused tests, and pkg/workflow and pkg/parser pass
• ✅ Shared API cache and per-run download pool keep primary and guardrail collection to one pass

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~8 AIC · ⌖ ~1 AIC · ⊞ 5.4K
_{Comment /matt to run again}

[gh-aw-bot]

@copilot

Copilot assignment requested for the following outstanding work:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Review (pkg/cli/experiments_outcome_observations.go:96): Outcome-backed entries in secondary_metrics are validated but never added to refs, so they are silently omitted from collection and analysis. - feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992 (comment)
3. Review (schemas/audit.schema.json:3873): Making zero_touch_evidence_complete required for every outcome record breaks validation of existing audit artifacts produced before this field existed. - feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992 (comment)
4. Review (pkg/cli/experiments_outcome_observations.go:81): An assigned run's artifact can supply an attacker-controlled host in a manifest repository or URL, which this call passes through resolveItemRepo to gh api --hostname without checking it against the run's host. If the analyzer inherits GH_ENTERPRISE_TOKEN or GITHUB_ENTERPRISE_TOKEN, requests to that host can expose the token. Reject artifact-derived hosts outside the trusted run host or an explicitly authorized host list before making outcome API requests. - feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992 (comment)
5. Review (pkg/cli/experiments_outcome_observations_test.go:62): This fixture is identical to issue in the case at line 68, but the two cases expect opposite results. For an accepted create_issue, outcomeMetricRunValue returns measured=true with zero unavailable coverage, so this assertion fails deterministically. Update this expectation to match the documented non-PR denominator behavior, or remove the duplicate case. - feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992 (comment)
6. Review (pkg/cli/experiments_outcome_observations.go:123): #74988 requires omitted or auto windows to learn a pooled, censor-aware maturity window, but this code always defaults to 72h and rejects auto; compiler validation does the same at pkg/workflow/compiler_experiments.go:522–525. Slow-review workflows can therefore score pending outputs as failures before their learned window expires. Accept auto and implement the requested Kaplan–Meier estimate, preserving explicit overrides and using 72h only for the cold-start fallback. - feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992 (comment)
7. Review (pkg/cli/experiments_grader_statistics.go:49): [/tdd] set.Coverage != nil now forces binary even when there are zero observations. An experiment whose outcome runs are all immature or unavailable gets MetricType: binary, and the comparison then runs a proportion test on empty samples. I did not run this case, so the impact is unconfirmed. - feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992 (comment)
8. Review (pkg/cli/experiments_outcome_observations_test.go:62): [/tdd] This expectation fails on the PR head: TestOutcomeMetricRunValue/non_PR_acceptance_has_unknown_zero_touch_evidence gets Unavailable: 0, not 1. The spec says resolved non-PR outputs must stay in the denominator and contribute zero, so outcomeMetricRunValue is correct and this case is wrong. - feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992 (comment)
9. Review (pkg/cli/experiments_outcome_observations.go:411): [/codebase-design] Setting coverage.Unavailable = 1 (and StalePending = 1 on line 285) instead of += is intentional: these are per-run flags, so a run with several bad outputs counts once. Nothing in the code says so, and a future reader may replace it with += and break that. - feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992 (comment)

Once you start work, post this work summary as a PR comment without the leading Copilot mention, including the Sous-chef head, work, and state lines below, to record that the assignment is running. Apply the necessary fixes on this PR's existing branch, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved; stop when only human review or CI remains. Do not merge or trigger CI.

Sous-chef requested-head: c5dba44
Sous-chef requested-work: 0130cea0257270fb0ac98f7472cb5441b1daa87b22d4afc292c3ddd5d9032133 01fbe7fcfa74ed2103941b322d5a99822c8a498447db56f5ae20764e0cc94d71 0937051472c3ff409c2bb502a4b0c12803d162c8f2167f44a9810965c625fca0 2647a5b381c23678582e0937ad441399e5688582710474897284fc7b622669a5 3cb68474a93cc22943fed1bab5abf636a972ac5870530c3ef801a6fb14ec7281 59d384807963e34612da3662af4a108676c140d471f6da3845ed95b182c39b4c 6361381073793c88abe2b3c24c703442c57f66a4d3632635d7824789a8e10d45 d87657c8e55b531758d3c5f55614fd94c8d1181dadad578016114e6897e674f7
Sous-chef requested-state: 6df56448556897ac8d2b441e1da4fb5fda121a635b57775e075e188cb9c17424

Generated by 👨🍳 PR Sous Chef · pi · haiku45 · ~10 AIC · ⌖ ~14 AIC · ⊞ 1.2K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot

Copilot assignment requested for the following outstanding work:...

Assignment in progress: merge the latest main, address the listed outcome-metrics review findings, and validate the changes. No CI will be triggered.

Sous-chef requested-head: c5dba44
Sous-chef requested-work: 0130cea0257270fb0ac98f7472cb5441b1daa87b22d4afc292c3ddd5d9032133 01fbe7fcfa74ed2103941b322d5a99822c8a498447db56f5ae20764e0cc94d71 0937051472c3ff409c2bb502a4b0c12803d162c8f2167f44a9810965c625fca0 2647a5b381c23678582e0937ad441399e5688582710474897284fc7b622669a5 3cb68474a93cc22943fed1bab5abf636a972ac5870530c3ef801a6fb14ec7281 59d384807963e34612da3662af4a108676c140d471f6da3845ed95b182c39b4c 6361381073793c88abe2b3c24c703442c57f66a4d3632635d7824789a8e10d45 d87657c8e55b531758d3c5f55614fd94c8d1181dadad578016114e6897e674f7
Sous-chef requested-state: 6df56448556897ac8d2b441e1da4fb5fda121a635b57775e075e188cb9c17424

done

[github-actions]

🎉 This pull request is included in a new release.

Release: v1.2.1