[100-JM]

Problem

SUT.process_queries (Offline) in language/llama3.1-8b/SUT_VLLM.py calls

self.model.generate(prompt_token_ids=input_ids_tensor, sampling_params=...)

prompt_token_ids= was a deprecated keyword that newer vLLM releases no longer accept. On vLLM 0.10.2 (NGC vllm:25.10-py3):

TypeError: LLM.generate() got an unexpected keyword argument 'prompt_token_ids'

SUTServer already passes pre-tokenized input as TokensPrompt objects, which works on both the pinned vllm==0.6.3 and current releases; only the Offline path was left on the old form.

The exception also exposed a second problem: it only killed the worker thread. LoadGen kept waiting for responses that would never arrive and the run sat idle until the job's wall-clock limit — four hours in our case — instead of failing.

Changes

• Offline path builds [TokensPrompt(prompt_token_ids=...) for q in qitem] and passes it positionally, exactly as SUTServer does. Same tokens reach the model; no behavioural change on 0.6.3.

• process_queries in both SUTs now wraps the worker loop: on an unhandled exception it logs the traceback and os._exit(1)s, so a broken SUT fails the run immediately instead of hanging LoadGen. If maintainers would rather keep the harness hands-off here, the first change stands alone.

• Both Offline and Server explicitly disable prefix caching so cross-query reuse does not depend on the installed vLLM default.

Verification

• Prefix-caching follow-up: two CPU-only tests exercise the real load_model methods with mocked engine constructors and fail before the fix/pass afterward. Python compilation and autopep8 checks pass. No GPU benchmark was rerun for this follow-up; the GPU results below predate the explicit cache setting.
• Ran the Offline scenario on vLLM 0.10.2 (B300, TP1, nvidia/Llama-3.1-8B-Instruct-NVFP4) with the equivalent patch — generate([{"prompt_token_ids": ids}, …]), i.e. the same dict TokensPrompt constructs: performance VALID, accuracy run completes and passes all five evaluation metrics (ROUGE1/2/L/Lsum and gen_len above the 99% thresholds). The fail-fast wrapper was exercised only by the original TypeError run, where the thread died and LoadGen waited four hours.
• python -m py_compile and autopep8 -a --max-line-length 79 clean. I could not exercise vllm==0.6.3 itself, but TokensPrompt has existed since 0.4.x and is what SUTServer already uses on that version.

Related: #2341 (Server/SingleStream EngineDeadError) is a different failure in the same file and is not addressed here.

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[100-JM]

recheck2

[100milliongold]

Adjacent finding from running this SUT with a recent vLLM, in case it belongs in this PR.

SUT_VLLM.py constructs the engine without touching prefix caching:

self.model = LLM(
    self.model_path,
    dtype=self.dtype,
    tensor_parallel_size=self.tensor_parallel_size,
)

so it inherits the vLLM default, which is now on:

CacheConfig().enable_prefix_caching  ->  True      # vLLM 0.28.1rc1.dev580+g385dce36b

This is not only nominal. A full Offline run on this SUT logged real cross-query reuse:

prefix_caching=True
Prefix cache hit rate: 1.7%
Prefix cache hit rate: 1.8%

inference_rules.adoc seems to draw the line at the batch boundary:

Q: Is it allowed to store continuous keys and values in non-contiguous memory space for the KV-cache, i.e. PagedAttention?
A: Yes, it is allowed as long as the KV-cache block is reused only within the batch of queries.

Prefix caching reuses blocks across queries by design, so a nonzero hit rate means blocks are being reused outside the batch that produced them. Line 459 ("caching of any other queries, query parameters, or intermediate results is prohibited") and line 542 point the same way.

Older vLLM defaulted this off, so the reference implementation was compliant when it was written. The default flipped with the V1 engine, and the SUT inherits whatever the installed version does. That makes compliance depend on the vLLM version rather than on the reference code.

If it fits the scope here, one line makes it explicit:

self.model = LLM(
    self.model_path,
    dtype=self.dtype,
    tensor_parallel_size=self.tensor_parallel_size,
    enable_prefix_caching=False,
)

SUTServer builds AsyncEngineArgs separately and would need the same treatment.

I have not confirmed how the working group reads this for prefix caching specifically; the rule text above is the closest thing I found. Happy to open a separate issue instead if that is the better place.

[100-JM]

@100milliongold Addressed the prefix-caching concern in 4ebfdaf. Both the Offline LLM constructor and the Server AsyncEngineArgs now explicitly set enable_prefix_caching=False, keeping cross-query reuse independent of the installed vLLM default. The option is also supported by the pinned vLLM 0.6.3 API.

Added two CPU-only tests that exercise the actual load_model methods with mocked engine constructors. Both fail without the fix and pass with it; Python compilation and autopep8 checks also pass. No GPU performance or accuracy run was repeated for this follow-up, so the earlier GPU results in the description predate this configuration change.

[100-JM]

recheck

[100-JM]

recheck