Skip to content

docs: record native RNGD coverage and KONI parity limits - #275

Open
gspark-etri wants to merge 8 commits into
mainfrom
docs/274-native-rngd-evidence
Open

gspark-etri wants to merge 8 commits into
mainfrom
docs/274-native-rngd-evidence

Conversation

@gspark-etri

@gspark-etri gspark-etri commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Background

The four-model serving task needs native RNGD evidence separate from research-engine PD results. On the tested SDK2026.4 registry configurations, exact Qwen3-0.6B, Qwen3-4B and Gemma3-1B are rejected; KONI builds and serves but changes greedy output under concurrency. This PR records those limits, reproduction clients and controls. It does not implement a production engine fix.

Changes

Publish one experiment record, portable HTTP clients and fixture checks, a SHA256 source manifest and small evidence summaries. Retain failed measurements and distinguish execution, exact-token parity, strict answer quality, cache reuse and lifecycle behavior.

Evidence Observed result
Native KONI build 105/105 kernel entries; FXB hash and frozen model revision recorded
Cache-off HTTP matrix 24 conditions, 344 responses; 63/168 token and 63/168 SSE-text mismatches against single-request references
Default/off/off/default profile-guided batching 344 additional responses; each arm has 29/42 token mismatches; disabling the flag does not fix drift
Log-probability controls 168 responses, 84 instrumented; output multisets unchanged by instrumentation; all 58 divergent instrumented responses expose competing probabilities at first shared-history divergence
Cache-off frozen quality Sequential 31/36 answers, 34/36 formats; concurrent 32/36 answers, 35/36 formats; two repeats each
Cache-on matrix 336 responses, 24 successful resets; positive reuse hits in 24/24 conditions; cold-start/reuse output multisets equal in only 19/24
Cache-on frozen quality 144 responses; sequential 32/36 answers, 35/36 formats; concurrent 31/36 answers, 34/36 formats. First/reuse tokens equal 36/36 within each mode; cache-off comparison changes an 8K case's JSON compliance
Context boundary and recovery Cache off/on, 30 requests: 6 valid edge probes accepted, 8 invalid probes HTTP400, 14/14 normal recoveries token-exact
Active stream disconnect Cache off/on, 12 engine abort counter increments; 72/72 concurrent survivors and 12/12 recoveries token-exact; final running/waiting gauges zero

Verification

Evidence head: 052f84f; base main fd89265. One RNGD, TP8, context9216, furiosa-llm2026.4.0 b1baffa75, compiler2026.4.0 ae27d44cd8. Model/FXB hashes, exact commands, settings, timings, raw artifact hashes and cleanup records are documented in experiments/2026-10-25-native-rngd-coverage.md and experiments/evidence274/.

Final review checks rerun from the published source copies:

python3 experiments/evidence274/selfcheck_native201_http_profile.py
python3 experiments/evidence274/selfcheck_native201_benchmark.py
python3 experiments/evidence274/selfcheck_native201_quality.py
python3 experiments/evidence274/selfcheck_native274_logprobs.py
python3 experiments/evidence274/selfcheck_native274_quality_concurrent.py
python3 experiments/evidence274/selfcheck_native276_cache.py
python3 experiments/evidence274/selfcheck_native277_boundary.py
python3 experiments/evidence274/selfcheck_native278_disconnect.py

All eight checks pass. These are measurement-client fixtures, not substitutes for the hardware outcomes above. All27 source-manifest hashes and git diff --check verified. Changed-file sensitive scan clean; the full-repository scan retains seven findings in six unchanged baseline files and is not clean. Owned hardware experiments are stopped and Pod deletion verified.

Compatibility and limits

Evidence-only change; no engine behavior or default changes. Exact registry rejections are version/configuration scoped. Single timed batches do not establish stable or equivalent-output speedups. Frozen36-case synthetic quality is not a broad quality benchmark. Reported top-k probabilities do not identify a compiler/kernel root cause.

Lifecycle inputs repeat valid token1000. Streaming victims close at first nonempty content, observed at44/80generated token IDs, so this is mid-decode cancellation rather than first-token or prefill cancellation. Abort counters and idle gauges do not prove physical KV allocation reclamation. Earlier native batch/shape drift remains unresolved.

Related Issue

Related to #201, #274, #276, #277, #278, #261 and #263. Does not close the overall four-model task. Production export/cache fixes and their acceptance remain separately tracked in #271 and #273; transport implementation is in PR266.

AI-assisted work: prepared and verified with Codex; maintainer review and merge required.

@gspark-etri
gspark-etri marked this pull request as ready for review October 24, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant