Skip to content

feat: expose Torch PD producer cache and chunk settings - #242

Draft
gspark-etri wants to merge 1 commit into
fix/238-prefill-worker-failurefrom
feat/241-pd-producer-cache
Draft

gspark-etri wants to merge 1 commit into
fix/238-prefill-worker-failurefrom
feat/241-pd-producer-cache

Conversation

@gspark-etri

Copy link
Copy Markdown
Collaborator

Background

Torch PD serving always enabled producer prefix reuse and used chunk2048, independently of the decoder's cache/chunk settings. This prevented an uncached, matching-chunk comparison from the CLI. BF16 GPU PD showed concurrent differences on Qwen3-0.6B, KONI and Gemma; precision/cache/chunk controls under #240 isolate the tested conditions.

Changes

Expose --pd-prefill-no-prefix-cache and --pd-prefill-chunk for the in-process Torch producer. Preserve cache-on/chunk2048 defaults and existing programmatic callers. Reject nonpositive chunks before loading the model. Document the independent producer/decoder policy.

Verification

  • Tiny real-model factory regressions: cache opt-out initially failed1/3, chunk control initially failed1/4; final focused set20passed.
  • Full project suite:464passed,15skipped,135subtests passed; one existing flex_attention warning (142.43s).
  • Two A10080GB PCIe, Torch2.13.0+cu130/Transformers5.15.0, BF16: all four models pass actual GPU0-prefill/GPU1-decode HTTP with the new cache-off/chunk256 flags. Inputs128/2048/8192,32outputs ignoringEOS,decode slots4,concurrency4/8. Concurrent equality48/48; native/PD serial equality12/12;84remote prefills,0producer cache hits; cancel/disconnect/reuse and invalid-request checks pass.
  • Runtime: core369c8ff plus this factory patch; pinned HTTP9e442cd; Validate four-model live GPU PD HTTP over long inputs and lifecycle events #240 validator forwards the new flags. Raw selected JSON and exact setup are in opt_prime/examples/pd/http_matrix/evidence241 and the linked experiment record. No KV transport changes.
  • Ruff for the new tests and diff checks pass. Scoped sensitive scan clean; repository-wide scan retains four unchanged baseline findings.

Compatibility and limits

Stacked on #239. Explicit policy controls only; no default numerical or speedup promise. Disabling reuse and smaller chunks may cost performance. Vendor/subprocess producer configuration is unchanged. Original mismatches are retained under #240; broader numerical quality, performance and RNGD/GPU overlap remain separate work.

Related Issue

Closes #241; supports #240.

gspark-etri added a commit that referenced this pull request Oct 30, 2026
…214, #229, #241) (#281)

- llmd: HTTP handler writes via MSG_NOSIGNAL so a client disconnect raises
  BrokenPipeError instead of hitting a vendor SIGPIPE handler (#215).
- serve_batched: add the tokenizer EOS to generation_config EOS ids (#230);
  --pd-prefill-chunk / --pd-prefill-no-prefix-cache for the Torch producer (#242).
- test that serve_stage's handler actually writes through the no-signal writer.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant