You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support report-only criteria without discarding their declared weight #101
A criterion can currently be excluded from task pass/fail and weighted scoring only by setting weight to 0. Some evaluation contracts need to preserve a non-zero declared weight for denominator/provenance parity while making a criterion genuinely report-only.
Using pass_threshold: 0.0 keeps a binary run_command miss from failing final_status, but calculate_weighted_score still includes its score and weight. The persisted CriterionResult also reports gating: true because gating is derived solely from weight > 0.
Reproduction
With two run_command criteria:
criterion A: score 1, weight 8, pass_threshold 1
criterion B: score 0, weight 1, pass_threshold 0
The task finishes SUCCESS, but weighted_score is 8/9 and criterion B is persisted as gating: true.
Requested contract
Add an explicit scoring/gating control, or document and implement pass_threshold: 0.0 as report-only, so a checker can still run and record its raw outcome while being excluded from both final_status and weighted_score without rewriting its declared weight.
The HTML/JUnit pass counters and persisted gating field should follow the same source of truth. A regression test should cover a failed report-only criterion with a non-zero declared weight.
Activity