Repository navigation
fix(eval): resolve VitPose image processor metadata - #1310
ssss141414 wants to merge 2 commits into
Conversation
## Summary Adds CPU fp32/fp16 recipes for `nielsr/vitpose-base-simple` and fixes recipe-free VitPose input generation. Optimum's `VitPoseDummyInputGenerator` inherits a static Vision generator that reloads preprocessors and indexes the result with `[-1]`. Config-only resolution has no loaded processor, so that path raises `IndexError`. A WinML `VitPoseIOConfig` now preserves Optimum's VitPose I/O declarations and model patcher while replacing only its dummy generator with the config-driven `DummyVisionInputGenerator`. The workaround is contained in the existing VitPose model adapter. Generic export I/O behavior is unchanged. The generic Perf range-consumption change was split into #1309. The VitPose evaluator processor fallback was split into #1310. ## Baseline and outcome Without a recipe, current-main baseline export resolves `VitPoseOnnxConfig` but fails while its static dummy generator loads an empty processor list: ```text Error: Build failed: list index out of range ``` With this change, recipe-free input resolution produces `pixel_values` `[1, 3, 256, 192]`. The checked-in recipes represent verified CPU fp32/fp16 coverage. ## Validation - Final head: `5635a06aee12a2dfe3d2f0a4e7c92dc8ac00f8f1` - Recipe-free CPU fp32 `winml build`: passed; `pixel_values` `[1, 3, 256, 192]` float32 -> `heatmaps`; final `model.onnx` generated - Checked-in CPU fp32 recipe `winml build`: exit 0; completed in 73.5s; final `model.onnx` generated - Checked-in CPU fp16 recipe `winml build`: exit 0; completed in 82.7s (Export 38.0s, Optimize 35.3s, FP16 8.8s); final `model.onnx` generated - `tests/unit/export/test_io.py` + `tests/unit/models/test_vitpose_mapping.py`: 101 passed - Ruff check: passed - Ruff format check: passed - GitHub checks: 9/9 passed - Recipe README remains unchanged ## Files - `examples/recipes/nielsr_vitpose-base-simple/cpu/cpu/keypoint-detection_fp32_config.json` - `examples/recipes/nielsr_vitpose-base-simple/cpu/cpu/keypoint-detection_fp16_config.json` - `src/winml/modelkit/models/hf/vitpose.py` - `tests/unit/models/test_vitpose_mapping.py`
9eb7dd8 to
b9f924e
Compare
REQUEST_CHANGESReviewed exact PR head Blocking items
Independent verification
After both blocking items are corrected, re-run review from the new exact head; any push invalidates the current check/thread snapshot. |
a49e317 to
4e12415
Compare
REQUEST_CHANGESReviewed exact PR head Blocking item
Independent verification
After Tester supplies a new immutable sealed root, re-enter review without changing the PR branch unless a product fix is independently required. Any head or main movement restarts the current-main gate. |
APPROVEReviewed exact PR head Independent verification
BlockersNone. The prior moved-main/body and evidence-seal blockers are cleared. |
4e12415 to
1207965
Compare
REQUEST_CHANGESReviewed exact PR head Blocking item
This is body-only and does not require a product-branch push. After the Explainer updates the body and refreshes its shipment artifact, invoke a new independent reviewer iteration. Independent verification
BlockerExplainer must make the two exact source-fidelity replacements above and refresh the shipment artifact. All code, test, evidence-seal, reuse, coverage, knowledge, check, and thread gates otherwise pass. |
APPROVEReviewed exact PR head Iteration 1 correctionThe only iteration-1 blocker is resolved. After newline normalization, the live body exactly equals the corrected explainer-r2 artifact. The r1-to-r2 diff contains exactly two paragraph replacements and no other semantic changes: Baseline now explicitly says no baseline Independent verification
BlockerNone. Keep the PR in DRAFT; this comment does not create a GitHub Review state. |
Summary
This L2 contribution enables
nielsr/vitpose-base-simplekeypoint-detection evaluation by resolving the VitPose image-processor metadata alias through a class-wide, metadata-gated fallback. It ships evaluator code and regression tests while the existing CPU fp32/fp16 recipes remain byte-for-byte unchanged. Full CPU fp32/fp16 coverage reached L2, and the committed Goal reached L3 PASS through one bounded fp32 CPU COCO functional smoke.Model metadata
What the model does
nielsr/vitpose-base-simpleis a top-down human pose model that accepts an RGB image plus person bounding boxes, transforms each person crop into a 256 x 192 tensor, and emits 17 heatmaps whose peaks represent body-keypoint locations.2d440ccdd65125f8fdf61110d5b9e2d3ded4a964declarespipeline_tag=keypoint-detection,model_type=vitpose,architectures=[ViTPoseForPoseEstimation],image_size=[256,192], and 17 labels; Transformers v5.14.1VitPoseImageProcessoruses COCO-format boxes for affine person crops, andViTPoseForPoseEstimationreturns per-keypoint heatmaps. Confidence:verified.Primary user stories
VitPoseImageProcessorpreprocessing/postprocessing contracts and the WinML keypoint evaluator's image, bbox, keypoints, and area fields. Confidence:verified.inferred.Supported tasks
keypoint-detectionacross checkpoint, Transformers, Optimum ONNX, and WinML surfaces. Evidence: the pinned checkpoint task and architecture; Transformers v5.14.1ViTPoseForPoseEstimationandVitPoseImageProcessor; Optimum'svitpose/keypoint-detectionregistration toVitPoseOnnxConfig; and WinML'sVitPoseIOConfig,VitPoseForPoseEstimation, and merged CPU fp32/fp16 recipes. Confidence:verified.Model architecture
ViTPoseForPoseEstimation/VitPoseSimpleDecodersource (verified).Validation and support evidence
Baseline
The current pinned
mainis2b9ec0e9e57bba8003d25eaff85f8c7c9e30d75awith WinML 0.3.0; the prior accepted base was48bfd0f91478a8e281b205b0449222024729f3ea. Baseline provisioning remains the historicalFAIL-UPSTREAMmeasured at24cccfb23d6e3c3b93ae9bb94399c326cee8ecea: canonical-lock synchronization failed on TLS transport, offline sync lacked locked torch, and product execution never began. Therefore no fresh current-main baseline config/build/perf/Eval result, build time, node count, latency, or metric is claimed.No baseline
winml configcommand executed and no baseline recipe was generated. Static installed-source evidence identifies thekeypoint-detectiontask,ViTPoseForPoseEstimation, Optimum'svitpose/keypoint-detectionregistration, and WinML's override withDummyVisionInputGenerator; these are source-inspection facts, not an executed auto-config selection. No registration was found forvit-poseorvit_pose. A fresh dynamic probe was blocked by an internally incompatible current lock and is not represented as fresh evidence.This is a
PARTIAL-RERUN. Between the prior and current bases, only PR #1312 changedwinml syssource, documentation, and tests; dependency impact analysis found no reachability into config, build, perf, Eval, analyze, loader/export, model, quantization, or optimization paths. Quality gates reran at shipped head1207965854b5eeb3dec055a7cfa1f08db30fa6f9; config/build/perf/parity/Eval/analyze remain reused from original execution SHAb9f924e3b23bd8caf85339faa3853eeb20df2f17under no-impact attestation and exact range-diff, stable-patch, owned-blob, and two-file diff equivalence. The current runtime patch commit88bd424bbf0e415fa05c25c3cf9ad369277a06d5is exact-equivalent to the originally measured runtime patch.Goal
Outcome
Outcome L2 shipped at head
1207965854b5eeb3dec055a7cfa1f08db30fa6f9; the highest Goal verdict is L3 PASS. Coverage is full for both required CPU tuples, with no deferred tuples. The product diff contains onlysrc/winml/modelkit/eval/keypoint_detection_evaluator.pyandtests/unit/eval/test_keypoint_detection_evaluator.py; the existing merged fp32/fp16 recipes remain byte-for-byte unchanged, and the production recipe README is untouched.Fresh exact-head quality evidence passed insert-license, Ruff, and Mypy (436 source files); focused evaluator tests 16 passed; direct
winml sysunit tests 11 passed; the direct sys-e2e probe reported 20 skipped because-m e2eis required and therefore is not executed e2e coverage; the commands partition reported 3618 passed/9 skipped; the remaining partition reported 871 passed/2 skipped/1 deselected. All nine exact-head GitHub checks completed successfully.No new model finding was produced in this rerun. Prior Lane A entries
vitpose-017throughvitpose-021are pushed at commita8d9a406bca1efaaf4dd4bace8d990334b5edc97on remote refrefs/heads/ssss141414/pr1310-component-mapping-skill; this run was deduplicated and found no methodology friction. These entries are not claimed to be onmain, and no Lane A PR URL is asserted.Per-EP/device/precision results and Functional smoke Eval
All model-stage rows below are reused from original execution SHA
b9f924e3b23bd8caf85339faa3853eeb20df2f17; they were not rerun at the shipped head.L0 fp32 structure: IR 8, opset 17, input
pixel_values [1,3,256,192], outputheatmaps [1,17,64,48], 359 nodes, 202 FLOAT initializers, and 343649280 external bytes. L0 fp16 structure: IR 8, opset 17, the same input/output shapes, 361 nodes, 201 FLOAT16 initializers, and 171824640 external bytes; realized fp16 precision was verified.Parity used named input
pixel_values [1,3,256,192]and output[1,17,64,48].Functional smoke Eval (L3 PASS): CPUExecutionProvider / cpu / fp32 on the official COCO 2017 person-keypoints validation release (
person_keypoints_val2017, validation), reduced to fixed image 29397 and non-crowd annotation 449985. One image was selected and processed with one person, one affine crop, one model invocation, one prediction, 17 keypoints, and one metric accumulation; schema, label semantics, and prediction semantics were verified. COCO keypoint OKS metrics were AP0.9999999999999998, AP500.9999999999999999, AP750.9999999999999999, AP_large0.9999999999999998, AP_medium-1.0, AR1.0, AR501.0, and AR751.0.The evaluator capability removed the former
AutoImageProcessorVitPose alias blocker by using the explicit metadata-gated fallback. This smoke was executed atb9f924e3b23bd8caf85339faa3853eeb20df2f17with runtime-equivalent commite7c8bd2041ac1aa40329b866362ffa72933f6252; it was not rerun at shipped head1207965854b5eeb3dec055a7cfa1f08db30fa6f9and is reused through no-impact and exact-equivalence attestation. It proves end-to-end functional operability only, not representative accuracy or benchmark quality; no fp16 Eval or other-EP accuracy is inferred. PR #1299's historicalmAP=0.8citation lacks the required command/artifact/revision provenance and is not this accepted smoke result.Delta
No baseline recipe was generated because baseline provisioning failed before product execution, so no baseline-generated-recipe or JSON-pointer comparison exists. The existing merged
examples/recipes/nielsr_vitpose-base-simple/cpu/cpu/keypoint-detection_fp32_config.jsonandkeypoint-detection_fp16_config.jsonfiles remain byte-for-byte unchanged, and this PR has no recipe delta. Original-execution recipe-free build/Eval evidence atb9f924e3b23bd8caf85339faa3853eeb20df2f17verifies the code path independently under the stated no-impact and exact-equivalence reuse attestation. The productionexamples/recipes/README.mdremains untouched.The code delta is
WinMLKeypointDetectionEvaluator.prepare_pipelinebehavior plus its focused unit tests. No product content changed during the rebase: the two commits, resulting evaluator/test blobs, and exact two-file diff are equivalent; only commit and parent identities changed.Bug fix explanation:
AutoImageProcessorraised its exact unrecognized-image-processorValueErrorfor a VitPose-family checkpoint.WinMLKeypointDetectionEvaluator.prepare_pipelineretainsAutoImageProcessoras primary, inspects model config only after the known error, and retries throughVitPoseImageProcessorwhenmodel_typeorarchitecturesidentifies VitPose;trust_remote_codeis forwarded to both paths.ValueErrors, non-ValueErrorexceptions, missing model IDs, input-size overrides, crop fanout, box conversion, custom keypoint metadata, dataset indexing, prediction flattening, COCO metrics, and MIGraphX layout behavior remain unchanged. The intentional change is that metadata-identified VitPose checkpoints with this alias mismatch can proceed.trust_remote_codeforwarding. Recipe-free fp32 build and the bounded COCO smoke establish reducibility and evaluator operability; existing recipes remain unchanged.Analyze summary — component level and op level
Status is
ANALYZE-PARTIAL-SUCCESS(exit code 1): a complete seven-record static rule roll-up was retained while unavailable plugin/rule groups remained explicit. This is static rule analysis, not runtime execution.Component-level summary
Resizeunknown on NvTensorRTRTX, OpenVINO, and QNN; no partial or unsupported operators reported.Op-level summary
Resizeunknown; the other 358 nodes are covered.CUDA, MIGraphX, TensorRT, and DML have no shipped rules, so all operators are unknown for those groups. Every static-analysis row has
runtime_support=false; no runtime support or compatibility is inferred.Reproduce commands