Repository navigation
Run matched HTTP measurements on assigned ATOM and RNGD adapters - #252
Draft
gspark-etri wants to merge 4 commits into
Draft
gspark-etri wants to merge 4 commits into
gspark-etri wants to merge 4 commits into
Conversation
This was referenced Oct 23, 2026
gspark-etri
added a commit
that referenced
this pull request
Oct 30, 2026
…248, #263) (#286) Add opt_prime/examples/pd/http_matrix with the reusable parts of the HTTP work from #243, #252 and #264: - validate.py: HTTP lifecycle validation through serve_batched.build - benchmark.py, benchmark_server.py, atom_server.py: warm native/dp2/pd HTTP benchmark with token parity checks and external-server configs - heldout_quality.py and frozen fixtures for four models (fixtures/) - tests moved to opt_prime/tests so CI collects them Adjustments: atom_server.py module overrides are optional (default to the in-tree modules); heldout_quality.py reads .json.gz fixtures and accepts --model for the request name; the Qwen3-0.6B fixture records the Hub name instead of a local cache path (manifest hash updated; input IDs and per-case hashes unchanged). Run outputs are git-ignored. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The matched HTTP benchmark assumed GPU servers, so it could not describe or test the actual RNGD-to-ATOM path. This change accepts trusted external server argv/configuration and records the real device allocation while retaining identical prompts, load and parity checks.
Changes
--server-configJSON with command arrays and validated resource metadata; no shell interpolation.--startup-timeout, default180s, to accommodate cold research-producer initialization.Verification
Three harness tests pass, including red/green configuration and startup-bound regressions. Full setup, exact commands, runtime SHAs, SDKs and artifacts are in
opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heterogeneous-performance248.mdandevidence248-heterogeneous/.All eight model/mode configurations completed12timed batches each (96total), inputs128/8192, output32, concurrency1/4, three repeats. Each PD route reported42remote/0local prefills. Native uses1ATOM; PD uses1ATOM+1RNGD, physical decode batch1. Producer width is fixed8192; cold startup is excluded. These are research-producer results, not native furiosa-llm performance.
At8192input/c4, median tok/s (all these long-input outputs match):
Gemma fails forced-output parity at input128. All five correctness checks diverge at output index5, after EOS106 at index2; six timed batches also differ. Offline comparison through the checkpoint EOS[1,106] matches, but is not a new normal-EOS HTTP validation. The benchmark deliberately ignoresEOS for fixed work. Failed raw data and exit1 are retained; no equivalent-output short-Gemma comparison or all-model pass claim. This does not resolve the earlier pre-EOS arithmetic divergence in#233.
Qwen0.6B's first PD attempt hit the180s startup bound with zero measured batches; its successful retry and that failure are both retained. Successful PD cold starts:369.73/810.53/432.86/852.22s for Qwen0.6/Qwen4/KONI/Gemma. All model PD paths are slower at short input. The owned Pod was deleted after confirming no live benchmark processes; subsequent lookup returnedNotFound.
Diff and scoped sensitive checks clean. No new full-core-suite claim for this experiment tooling.
Compatibility and limits
External commands are trusted code, not request-controlled. Resource metadata must describe the supplied server. Concurrent clients queue on physical batch1. Prefix caches are disabled; output32 ignoresEOS. Model answer quality, normal-EOS behavior, cold-start stage attribution and request-pipeline improvements are separate pending work.
Related Issue
Refs#248. Stacked on#251. Draft for heterogeneous evidence and failure review; maintainer merges.