Skip to content

exp: characterize KONI numerical drift and output-head precision limits - #262

Draft
gspark-etri wants to merge 1 commit into
fix/258-dense-hybrid-kvfrom
exp/261-koni-numerics
Draft

gspark-etri wants to merge 1 commit into
fix/258-dense-hybrid-kvfrom
exp/261-koni-numerics

Conversation

@gspark-etri

Copy link
Copy Markdown
Collaborator

Background

KONI-8B live RNGD→GPU completes requests but differs from nativeGPU beforeEOS on128-token outputs. Lifecycle success does not explain numerical drift. This PR retains failed comparisons and controlled probes without changing production sampling.

Changes

Add original live matrix/audit, matched-prefill and padded-prefill controls, shortKV repeat/replay metadata, identical-prefix BF16/FP32/head-onlyFP32 probes, and unforced output-head precision controls. Include portable forced-prefix entry point and exact local research scripts. No production code changes.

Verification

Runtime32ce5d5, checkpoint0172bf7e73977ae213613529f3d2d5400a33f678; A10080GB+RNGD, SDK2026.4, GPU Torch2.13. Original inputs128/8192,outputs32/128,c1/4,3repeats,4paged lanes/context9216/cacheoff, RNGDwidth8192.

  • Original32-output matches;128-output differs atindex64(short) or98(long), beforeEOS.
  • Long nativeGPU full8192prefill and retainedRNGDKV both reproduce live128tokens; chunk256 reproduces originalreference.
  • Short GPU128prefill/import matches native; padding to8192 matches neither native nor live. Padding-only explanation fails.
  • Short liveKV exported twice: firsttoken andBF16K/V bit-identical; replay reproduces original live128tokens. Native original128tokens also reproduced.
  • Common64-token prefix: BF16native logits3940/8712 tie14.875/14.875; retainedKV14.8125/14.9375. FP32native/import both select8712. Output-head-onlyFP32 also resolves that inspected tie.
  • Unforced128-token controls reject head-onlyFP32 as a generalfix: KONIshort firstdifference14 (original64), Qwen4long13 (matchedBF16original107); KONIlong matchedprefill remains equal. No fixture-specific production tie-breaking added.
    Hardware command/results are in experiments/2026-10-24-koni-numerics261.md. All raw complete flags/counts verified; portable CLI syntax/help and scoped sensitive/diff scans passed. CLI adaptation itself was not hardware rerun; equivalent original script was.

Compatibility and limits

Diagnostic evidence, not a universal numerical/quality fix. Firstdifference measures equality, not semantic accuracy. FP32native changes prefill/decode arithmetic; importedKV remainsBF16.17MBshort and1.1GBlong KV dumps remain local; capture script/dependencies documented. Local research scripts need their environment paths adapted; matching Qwen control inputs are in #257. No equivalent-output speedup claim.

Related Issue

Refs #261,#201,#256. Does not close numerical investigation. Stacked on #259; draft for review and broader validation. Maintainer merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant