Validate long-context ATOM B4 serving against the B1 c16 baseline #270

Problem

The four-model long-context ATOM HTTP matrix currently compiles batch_size=1 while submitting16 concurrent requests. This does not establish multi-lane serving behavior or its effect on queue latency.

Evidence

PR #266 evidence f1d9117: KONI native ATOM at8192 input/128 output/c16 reaches7.7847tok/s, max first SSE253.93s and max completion263.08s. Six timed requests exceed180s. The separate600s diagnostic passes264 responses with no OOM/listener drops; the original180s failure remains failed. This does not prove batch size alone is the cause.
The experimental atom_server265_http.py hardcodes engine batch_size=1 and producer slots=1. AtomRblnEngine already accepts batch_size and creates distinct compiled-cache paths. AtomBatcher own block tables require at least lanes+1 blocks for PD/scratch ownership.

Hypothesis

A matched B4/context9216/k5 artifact on the same assigned ATOM can reduce c16 queueing and increase throughput without breaking lane ownership, cancellation or model-specific attention behavior. Compiler/runtime memory and numerical effects must be measured, not assumed.

Scope

• Four models: Qwen3-0.6B, Qwen3-4B, KONI-8B, Gemma3-1B.
• Reuse existing batch support; isolated benchmark configuration and new cache paths first, no default change.
• Native ATOM and RNGD-to-ATOM whole/layer-stream paths, inputs128/512/2048/8192 and outputs32/128 at matched concurrency.
• Verify actual occupied lanes, full sequence repeat/within-configuration parity, through-EOS native comparisons, partial cancellation/slot reuse, and any numerical differences.
• Retain B1 references and180s timeout failure, record compile/load times and memory, never reuse incompatible B1 graphs or affect other workloads.

Done when

Distinct B4 artifacts and full commands/SDK/source hashes are recorded; all requested model/routes have supported pass evidence or explicit diagnosed failures; matched B1/B4 latency and throughput are reported without attributing socket-timeout changes to performance; necessary fixes have regression coverage and reviewable PRs.

Owner

Codex implementation and experiment follow-up; maintainer review. Related #201, #265, #267. Current owned B1 job completes before allocating devices for this hypothesis.

Configuration preflight follow-up

The retained KONI native/whole/stream configs use B1/k1, unlike the three-block baseline used by other models. Moving directly to B4/k5 changes both batch size and block-table ownership mode. Add B1/k5 controls (same context/model/native-or-PD path) to separate these effects; retain original B1/k1 and B1/k3 artifacts. No B4 hardware run has started. The isolated harness now validates B4/k5 and passes parser controls, but device compilation, memory fit and actual lane use remain unverified.

Device-capacity finding

Read-only monitoring of the assigned ATOM shows16,357,785,600 bytes used out of16,877,879,296 during KONI B1/k1 serving. Its compiled config has64 BF16 KV inputs of shape[1,8,9216,128], totaling1,207,959,552 bytes. k5 would require6,039,797,760 KV bytes; if other allocations stay unchanged, total would be21,189,623,808 bytes, over the observed device capacity. This is a conditional estimate, not an actual compiler failure. Do not blindly launch the planned KONI TP1/k5 controls; assess a matched two-ATOM B1/B4 configuration or a separately labelled shorter-context experiment while retaining the8K objective. Other models still need device-fit verification. Existing owned B1 job remains unchanged.