[richiemcilroy]

• Implement D3D11 zero-copy texture sharing with pipeline caching for hardware-accelerated decoding
  • Add native NV12 video frame support to WebGPU renderer, avoiding CPU-side YUV conversion
  • Optimize frame WebSocket streaming with watch channels and NV12 downscaling
  • Refactor FFmpeg decoder frame caching and seeking logic
  • Add Windows 11 caption controls to editor skeleton
  • Fix system audio loading errors and improve recording recovery checks

Greptile Summary

This PR implements comprehensive Windows hardware acceleration and NV12 rendering optimizations across the video pipeline. The changes enable zero-copy D3D11→D3D12 texture sharing via shared handles, eliminating expensive CPU-side YUV conversions during hardware-accelerated decoding. Native NV12 support is added to the WebGPU renderer with a dedicated shader pipeline for YUV→RGB conversion on the GPU. The WebSocket frame streaming now uses watch channels with intelligent NV12 downscaling (50%) for efficient preview delivery. FFmpeg decoder refactoring separates CPU/GPU frame processing paths with improved frame caching and seeking logic. Additional improvements include WebGPU pipeline caching for faster shader compilation, Windows 11 caption controls in the editor UI, and enhanced recording recovery with minimum audio file size validation (500 bytes).

Key architectural improvements:

• Zero-copy texture sharing between D3D11 (decoder) and D3D12 (wgpu) eliminates data copies
• NV12 format preserved end-to-end from decoder through WebSocket to WebGPU renderer
• Watch-based frame streaming replaces broadcast for better backpressure handling
• Pipeline cache persistence reduces shader compilation overhead on subsequent runs

Issue found:

• renderFrameCount variable in socket.ts:533 is used but never declared or incremented in the direct canvas rendering path

Confidence Score: 4/5

• This PR is safe to merge with one minor undefined variable issue that needs fixing
• The implementation is architecturally sound with proper Windows API usage, error handling, and platform-specific code guards. The D3D11/D3D12 interop follows correct patterns with shared handle management. However, there is one logic error where renderFrameCount is referenced but never declared in the direct canvas rendering path in socket.ts, which will cause a runtime reference error. The rest of the changes are well-structured, follow existing patterns, and add substantial performance improvements through hardware acceleration.
• apps/desktop/src/utils/socket.ts requires fixing the undefined renderFrameCount variable before merging

Important Files Changed

Filename	Overview
crates/rendering/src/d3d_texture.rs	Implements D3D11→D3D12 zero-copy texture sharing via shared handles with proper Windows API usage and caching for NV12 planes
apps/desktop/src/utils/webgpu-renderer.ts	Adds native NV12 WebGPU rendering with YUV→RGB conversion shader and dual pipeline support for RGBA and NV12 formats
apps/desktop/src-tauri/src/frame_ws.rs	Implements watch-based frame streaming with NV12 downscaling (50%) and CPU-side YUV conversion for efficient WebSocket delivery
apps/desktop/src/utils/socket.ts	Handles NV12 frame detection and rendering with main-thread WebGPU fallback, includes detailed performance metrics logging
crates/rendering/src/decoder/ffmpeg.rs	Refactors frame caching with separate CPU/GPU paths, adds NV12/YUV420p support, and improves seeking logic with frame tolerance
crates/video-decode/src/media_foundation.rs	Adds comprehensive Media Foundation hardware acceleration capabilities query and D3D11 texture creation with shared handles
crates/rendering/src/composite_frame.rs	Implements WebGPU pipeline caching with persistent storage to dirs::data_local_dir() for faster shader compilation on subsequent runs

Sequence Diagram

sequenceDiagram
    participant FFmpeg as FFmpeg Decoder<br/>(Rust)
    participant D3D11 as D3D11 Device<br/>(Windows)
    participant Shared as Shared Handle<br/>(NT Handle)
    participant D3D12 as D3D12 Device<br/>(wgpu)
    participant WebSocket as WebSocket Server<br/>(Tauri)
    participant Frontend as Frontend<br/>(TypeScript)
    participant WebGPU as WebGPU Renderer<br/>(Browser)

    Note over FFmpeg,D3D11: Hardware Decoding Path
    FFmpeg->>D3D11: Decode video frame to NV12
    D3D11->>D3D11: Create shared texture<br/>(D3D11_RESOURCE_MISC_SHARED_NTHANDLE)
    D3D11->>Shared: Export shared handle
    
    Note over Shared,D3D12: Zero-Copy Texture Sharing
    Shared->>D3D12: OpenSharedHandle()
    D3D12->>D3D12: Create HAL texture from D3D12Resource
    D3D12->>WebGPU: create_texture_from_hal()
    
    Note over FFmpeg,WebSocket: Frame Streaming (Camera Preview)
    FFmpeg->>WebSocket: Send NV12 frame data
    WebSocket->>WebSocket: Downscale 50%<br/>RGB→YUV conversion
    WebSocket->>Frontend: Send over ws:// with metadata
    Frontend->>Frontend: Detect NV12 magic (0x4e563132)
    Frontend->>WebGPU: renderNv12FrameWebGPU()
    WebGPU->>WebGPU: Upload Y plane (r8unorm)
    WebGPU->>WebGPU: Upload UV plane (rg8unorm)
    WebGPU->>WebGPU: YUV→RGB shader conversion
    WebGPU->>Frontend: Display frame on canvas

Loading

[greptile-apps]

_{30 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

logic: renderFrameCount is referenced here but never declared or incremented in this code path. The variable is only reset to 0 on line 540, but never defined or incremented when frames are actually rendered through directCtx.putImageData.

Prompt To Fix With AI

This is a comment left during a code review.
Path: apps/desktop/src/utils/socket.ts
Line: 533:533

Comment:
**logic:** `renderFrameCount` is referenced here but never declared or incremented in this code path. The variable is only reset to 0 on line 540, but never defined or incremented when frames are actually rendered through `directCtx.putImageData`.

How can I resolve this? If you propose a fix, please make it concise.