Profile test performance and improve it #1363

Activity

1. vee1e commented on Aug 17, 2026

   

   vee1e

   on Aug 17, 2026

   Last edited by vee1e

   CollaboratorAuthor

   More actions

   the premise is not supported by the CI data, in fact the opposite is true. Quantumstrand runs 114 more tests but finishes the same test matrix faster than master.

   CI comparison (same 7-job matrix on both branches)

   pytest suite duration on windows-3.10, the slowest/most expensive job:

   Branch	Tests	pytest duration
   master	159	746s, 733s, 764s, 557s → ~700s avg
   quantumstrand	273	628s, 609s, 573s, 576s → ~597s avg

   Same story on ubuntu-3.10: master 675s/159 tests vs QS 544-600s/273 tests. Per-test, quantumstrand is ~2x faster (2.0s/test vs 4.2s/test). Total CI wall-clock is unchanged (~11-14 min, dominated by the windows-3.10 job, which is actually faster on QS).

   reasons:

   • the recent QS PRs (unified-pipeline iterations) shipped performance improvements
   • the ~114 added tests are mostly cheap pure-logic tests (layout, render, JSON roundtrip, ranges) not full-FLOSS runs.

   But local profiling of a single full main() run (7.8s) do show bottlenecks:

   • make_emulator — 3.1s (~40%) is called 79× per run, once per decoding routine/context. Each call constructs a fresh emulator which which re-instantiates every architecture module including expensive register-definition parsing without any caching.
   • flirt.flirt.compile — 0.97s: FLIRT signatures recompiled on every main call (not cached).
   • load_vw — 2.0s: (workspace + FLIRT registration).

   Recommended improvements

   • Reuse emulators across decoding routines instead of building a fresh one per context, or cache envi.getArchModules.
     Biggest win: ~40% off full-pipeline tests (test_cli_args.py 48s, test_language_rust_known_binary.py 31s, YAML decode cases ~200s).
   • Cache compiled FLIRT signatures across invocations.