Skip to content

Align Raw DXCGC Run timing with Runtime API perf - #1478

Open
Qiong Wu (qiowu) (DingmaomaoBJTU) wants to merge 5 commits into
mainfrom
codex/align-perf-timing
Open

Qiong Wu (qiowu) (DingmaomaoBJTU) wants to merge 5 commits into
mainfrom
codex/align-perf-timing

Conversation

@DingmaomaoBJTU

@DingmaomaoBJTU Qiong Wu (qiowu) (DingmaomaoBJTU) commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Raw DXCGC benchmarks previously counted Python buffer allocation, input argument preparation, and copying results into NumPy, while WinML Runtime benchmarks counted only Pipeline.run(). This PR removes those extra steps from Raw's measured interval so both paths measure their logical Run call.

What changes

Raw DXCGC step Before After
Allocate NumPy outputs and prepare input arguments Timed Before timing
Call native Run: upload inputs, record/replay commands, wait for GPU completion, and materialize native CPU outputs Timed Timed
Copy native outputs into NumPy arrays Timed Separate, untimed Fetch
  • Split the native bridge into Run and Fetch operations. The reported host time comes from the Python stopwatch around Run, including ctypes call overhead.
  • Add timing_scope metadata: python-native-run for Raw DXCGC, pipeline-run for WinML Runtime, and session-run for other paths using the shared report. Runtime execution and timing are unchanged.
  • Upgrade the bridge to ABI 5 and reject older DLLs. Rebuild both the native EXE and DLL.
  • Remove compare_sequence.py; paired inputs, alternating run order, and cross-path reports are delegated to the benchmark agent.

This changes the measurement protocol, not model execution speed. Historical results keep their original timing scope. Run still includes CPU work and GPU transfers; it is not a pure GPU metric, and the two paths still differ in their bindings, resource management, and scheduling.

Validation

  • Native Release build and CTest passed.
  • 85 focused Python tests passed; 2 GPU-fixture tests skipped.
  • Ruff passed.
  • A deterministic clock test verifies that a 2 ms Run is reported as 2 ms even when Fetch takes 10 seconds, and that native-only telemetry does not replace the Python call measurement.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant