Investigate RNGD to ATOM HTTP connection reset under sixteen queued requests #267

Problem

The four-model HTTPc16 validation stops on RNGD→ATOM whole-KV mode with ConnectionResetError(104) before all correctness cases finish. Native ATOM for the same Qwen3-0.6B checkpoint completed all conditions. Root cause is not yet established.

Evidence

Runtime33d6cb0, isolated streaming package, one RNGD/one ATOM, B1/context9216/kv_blocks3, sourcewidth8192, cacheoff. Inputs128/512/2048/8192, output32/128, concurrency16. Eight warmups and five nonstreamed correctness batches (128/32,128/128,512/32,512/128,2048/32) finish. Failure occurs during2048/128; no timed batches recorded. Client traceback ends while reading HTTP status with connection reset by peer. Server log contains only device-binding diagnostics, no Python traceback. Harness currently kills its servers in finally without recording pre-cleanup exit status/cgroup memory events. Owned Pod cleanup verified separately; no other workloads changed.

Scope / hypothesis

Determine whether the server exited (including OOM/native failure) or a connection failed while the server stayed alive (including listener backlog). Add diagnostic-only capture of process state/return codes, cgroup memory events and per-request exceptions before cleanup; reproduce the same condition before changing production behavior. Preserve the failed run and keep native/whole/stream comparisons distinct.

Done when

Failure is reproduced or bounded with sufficient evidence; an identified software cause has a regression test and targeted fix, or an external limit is recorded with reproducible evidence. Rerun the original c16 case and all four models' affected HTTP paths. Never turn an incomplete run into a pass by ignoring failed requests.

Owner

Codex investigation; maintainer review for changes. Related #201 and #265 (PR266). Raw local record: experiments/2026-10-24-rngd-layer-stream265.md; stream265-qwen06b-http-atom-c16-whole.json/log/server0.log.