codex agent: error notifications (incl. willRetry) are dropped by _CodexTurnState.dispatch — an 18-minute provider retry loop left no trace in task.json/task.log #151

What happened

adhoc run adhoc-2026-09-02_18-30-39, v2 arm, skill-flow-datafabric-smoke-update-existing-flow (gpt-5.6-luna via the Azure custom provider, coder_eval 0.11.5, codex-cli 0.137.0-alpha.4): between a tool result at 18:32:55 and the next assistant generation at 18:51:27 the turn sat for 1 113 s. The next generation carried 159 055 uncached input tokens (a 497 KB shell result the model had let through — see openai/codex issue filed alongside). Nothing in task.json or task.log says what happened in those 18 minutes: the run was SUCCESS 1.000, duration 1285.3s, and looked like "the model was slow".

Why nothing was recorded

coder_eval/agents/codex_agent.py, _CodexTurnState.dispatch routes item/started, item/completed, item/agentMessage/delta, thread/tokenUsage/updated, turn/completed; every other method — including the app-server's error notification, whose payload is { error: TurnError, threadId, turnId, willRetry: bool } (openai_codex/generated/v2_all.py ErrorNotification) — falls through to return False. The only other sink is log_raw_sdk_event, gated behind CODER_EVAL_RAW_SDK_LOG (off in every nightly/adhoc run). Codex's client retries a stalled/failed stream silently (stream_idle_timeout_ms 300 000 × stream_max_retries 5, request_max_retries 4 — provider defaults, none set by _build_thread_options), so a request that times out or is 429'd shows up in our record as pure model latency.

Ask

1. Handle error notifications in dispatch: log at WARNING (willRetry, error type/message, elapsed since the last item) and persist them per iteration (e.g. iterations[].provider_errors[]) so a stall is attributable in the run record without turning on the raw dump.
2. Optionally record per-generation request timing (started_at of the request vs first token) — today generation_duration_ms only covers streaming, so an 18-minute wait before the first token is invisible.

Not asking to change provider retry/idle knobs or run_limits.

Evidence and method: workspaces/reports/2026-09-02-maestro-flow-datafabric-smoke-update-existing-flow/rca-datafabric-smoke-update-existing-flow.md (UiPath-internal).

🤖 Generated with Claude Code

https://claude.ai/code/session_01WrUStgpnX7bSPe35r3nWBF