Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions opt_prime/examples/pd/http_matrix/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,17 @@ Use `--backend atom` in an isolated allocation exposing exactly one ATOM. Existi
Inputs128/2048/8192 and clients4/8 are queued over one ATOM execution lane, not hardware batching. All four cache-off runs pass lifecycle checks and48/48 concurrent/serial comparisons. Both Qwen device/host cache cases pass12/12 each. Gemma requires the causal-prefill SDK graph correction and #210 hybrid local-table/mask fix; its prefix/host/PD restoration remains unsupported. KONI's one-block configuration has no spare prefix block.

See experiments/2026-10-23-atom-http211.md and evidence/atom-http-*.json. Files ending `fixed` include #214's HTTP MSG_NOSIGNAL fix; the Qwen0.6 matched rerun preserves all output sequences and eliminates the observed native disconnect backtraces. This does not establish all RNGD/heterogeneous routes or real transfer overlap.

### Torch PD producer reproducibility controls

`serve_batched.py --pd-prefill-device cuda:0` keeps producer prefix caching enabled and uses a 2048-token producer chunk by default. These are independent of decoder `--prefix-cache` and local `--chunk`.

To compare BF16 native and PD under uncached, matching chunk execution:

```bash
python examples/pd/serve_batched.py --model "$MODEL" --device cuda:1 --slots 4 \
--max-context 9216 --chunk 256 --pd-prefill-device cuda:0 \
--pd-prefill-no-prefix-cache --pd-prefill-chunk 256
```

The new flags affect the in-process Torch producer only. Nonpositive producer chunks are rejected. Disabling reuse or using smaller chunks can increase prefill work/overhead; these options do not promise a speedup or universal bitwise equality. CPU factory and four-model GPU controls are recorded under issue #241.
4 changes: 4 additions & 0 deletions opt_prime/examples/pd/http_matrix/STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,7 @@ Static B1 with one KV block now supports cache-off full-prefix PD. KONI live RNG
## Subprocess failure lifecycle (#238)

Structured SDK logs and cancelled/malformed replies previously killed the receiver and could strand later requests. Terminal errors now settle all queued Futures and reject subsequent submits. Failed startup and bounded close release the owned process and slab; fresh work succeeds on an explicitly created replacement worker. Nine CPU fault scenarios plus the existing tiny Torch subprocess path pass in the 64-test focused set. Full suite: 460 passed, 15 skipped, 135 subtests passed. Accelerator fault recovery and service-level automatic restart remain separate acceptance items. See prefill-worker-failure238 record.

## Explicit Torch producer policy (#241)

New opt-in controls disable producer prefix reuse and select producer chunk size; defaults remain cache-on/chunk2048. Public cache-off/chunk256 configuration passes all four real GPU PD HTTP paths:48/48 concurrent equality,12/12 native-PD serial equality at128/2048/8192tokens,84remote prefills and0producer hits. Lifecycle checks pass.20focused and464full tests pass (15skipped,135subtests). This is tested numerical reproducibility, not a speedup or arbitrary-input equality guarantee; original default-policy discrepancies remain in#240.
2 changes: 2 additions & 0 deletions opt_prime/examples/pd/http_matrix/evidence241/events.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"id": "bb5828e86f364b2f8bc860f5f50d6e4a", "ts": "2026-10-23T23:41:12+09:00", "task": "Torch PD producer cache control (aicomp #241)", "result": "Added explicit producer cache opt-out; actual tiny-model factory red1/3 -> green3/3,19focused tests pass. Qwen0.6B/KONI GPU controls recover concurrent12/12 with cache off; public-flag GPU validation pending.", "next": "Complete full suite and run new flag on GPU; retain enabled default and independent decoder policy.", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-23-pd-producer-cache241.md"], "labels": ["bug"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "8f4709fcaa1146a3b7e60d4e4e12bbf4", "ts": "2026-10-23T23:47:49+09:00", "task": "Torch PD producer cache control (aicomp #241)", "result": "Public cache-off/chunk256 flags: four-model GPU PD48/48 concurrent and12/12 native-PD serial equality;84remote prefills,0producer cache hits; all lifecycle checks pass.20focused tests pass.", "next": "Publish configuration controls and full comparison evidence; measure performance cost separately.", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"dtype": "bfloat16", "producer_prefix_cache": "false", "producer_chunk": "256"}, "artifacts": [{"path": "analysis/serving-matrix-201/gpu241-summary.json", "size": 1185, "sha256": "d63548c8bc387463"}], "experiments": ["experiments/2026-10-23-pd-producer-cache241.md"], "labels": ["experiment"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
Loading