[Decisions] Make confidence mean the same thing on every backend (Laya reports normalized entropy, others max p or Jev's margin) #89

Why

Every backend returns a confidence next to its choice, and System1-Agents passes it straight through to step records (tool/loop.py, browser/decision_model.py), s1a probe, s1a decide and the MCP decide tool. Right now that number means different things depending on the backend:

Two recent write-ups show why this matters:

• Fast Models, Slow Evidence (arXiv:2610.02267), a paired Laya vs Jev evaluation with released raw outputs, found that Laya's confidence field differs from max_k p_k by 0.299 on average (0.010 for Jev) and warns that "using that field would silently break any threshold". It recommends computing confidence as max_k p_k and choosing thresholds on held-out data.
• Laya itself now says the same thing. In laya 0.4.11 (current PyPI), laya/confidence.py documents that confidence is normalized entropy, "does not transfer across option counts", and that "a threshold carried over from Jev selects about twice as much on Laya". It adds a separate answer_confidence (max p) field for gating. This repo locks laya 0.3.20.

Nothing in Agents thresholds on confidence today, so this is not a live bug yet. But the conf= column in run summaries and replays, the MCP decide output, and any future confidence-gated step (for example the System-1 to System-2 delegation in SearchJev, arXiv:2610.05107) would compare numbers that are on different scales for different backends.

Proposed scope (one PR)

1. Decide on one documented definition for Choice.confidence / Noul.confidence across backends. The suggestion is max_k p_k for choice and max(p, 1-p) for noul, both computed in validation.py from the validated probabilities, so every backend gets the same number.
2. Keep each backend's own number for provenance, e.g. raw["backend_confidence"] (Laya's entropy value, Jev's normalized margin), so nothing is lost and old records can still be read.
3. Update docs/decision-models.md (the Decision contract section) and the s1a probe / decide / MCP field descriptions to say exactly what confidence is, and that it is not calibrated unless a backend says so.
4. Add the definition to DecisionModelContract so a new backend can't drift.

Acceptance criteria

• Unit tests (offline, using fakes and recorded Laya/Jev/OmniJev answer dicts) show the same probabilities yield the same confidence on every backend.
• Backend-native confidence is retained in raw and the step records, under a name that does not collide with confidence.
• The CHANGELOG notes the change in meaning for laya / laya-served / jev / omnijev records, since historical conf= values are not comparable with new ones.
• No change to which option is chosen; only the reported confidence changes.

Out of scope

• Calibrating any backend (temperature fitting, per-domain recalibration).
• Adding a confidence gate or System-2 fallback. That can be a follow-up once the number means one thing.
• Bumping laya beyond the current lock (track separately if wanted).

References

• Fast Models, Slow Evidence: https://arxiv.org/abs/2610.02267 (code and raw outputs: https://github.com/David-DL-Space/sys1-eval)
• SearchJev (confidence-gated System-2 delegation): https://arxiv.org/abs/2610.05107
• Laya package (see laya/confidence.py, laya/common.py::confidence_from_probs): https://pypi.org/project/laya/
• Jev confidence formula as referenced by Laya: TypeSafe system-one-adapter-python, _utils/confidence_metrics.py; Introducing System One Models & Jev
• Related: [Fix] Fit the browser front's state and questions into Laya's window #9 / [Deps] Require laya 0.3.9, lock 0.3.20, and drop the local weight-init skip #37 (Laya shaping and pin), [Feat] OmniJev as a vision decision model for the browser agents #33 (OmniJev confidence recompute)