Skip to content

Release SenseVoice GPU memory after UI transcription - #381

Open
LauraGPT wants to merge 1 commit into
OpenBMB:mainfrom
LauraGPT:codex/fix-ui-asr-vram-20260813
Open

LauraGPT wants to merge 1 commit into
OpenBMB:mainfrom
LauraGPT:codex/fix-ui-asr-vram-20260813

Conversation

@LauraGPT

@LauraGPT LauraGPT commented Aug 12, 2026 •

Copy link
Copy Markdown

Summary

  • Clear the demo's stored SenseVoice model reference after each Ultimate Cloning reference transcription.
  • Run that cleanup in a finally block on both successful and failed ASR requests.
  • Collect Python garbage and ask the CUDA allocator to return unused cache when the selected ASR device is CUDA.
  • Add regression coverage for clearing the stored reference and attempting cache cleanup on the success and exception paths.

Root Cause And Scope

The Gradio process keeps VoxCPM2 and the optional enhancer on the GPU. Previously, after Ultimate Cloning was used once, self.asr_model also retained SenseVoice for the rest of the process lifetime. This change keeps ASR lazy and removes that long-lived reference after transcription.

Clearing a reference is not a guarantee of immediate physical GPU-memory release on every error path. A propagating exception's traceback can still hold the model through its generate() frame while the finally block calls gc.collect() and torch.cuda.empty_cache(). An object-lifetime probe confirmed this distinction: the controlled model was already unreachable at cache cleanup on success, but was still alive there on failure, becoming unreachable after the caller finished handling the exception. The probe used Python weak references and a fake model; it did not measure GPU memory or establish Gradio's exception-retention lifetime. External references or retained exceptions can also extend that lifetime.

This PR therefore removes the demo's persistent ASR ownership and attempts cleanup; it does not promise that every failure immediately returns all SenseVoice memory to the GPU driver. The existing tests likewise verify reference clearing and a cache-cleanup call, not allocator behavior with a real model.

Fixes #300

Fresh Verification (2026-09-29)

Unchanged published head: ed2fb51fa3b56a9627cf0df010779f7a125acf7c.

Fetched actual upstream main@f772e498a45fbb5fb8e13fbf9b9c48be9fe33e69. Its intervening changes do not alter app.py, pyproject.toml, or these tests; no history refresh or source changes were needed for this check.

  • PYTHONPATH="src:$DEPS" python3 -m pytest -q tests -ra: 58 passed, no skips/deselections. A separate collection run found all 58 tests; no shard or PYTEST_ADDOPTS override was supplied.
  • python3 -m black --check tests/test_app_asr_lifecycle.py: passed.
  • git diff --check origin/main: passed.
  • CPU-only object-lifetime probe described above: both calls cleared demo.asr_model; the exception traceback retained the fake model until exception handling completed.

This reused the existing Python 3.12.3 unit-test environment and test dependency directory. It is not a clean installation of the project's full dependency set: Gradio is absent there, and the repository tests substitute Gradio, FunASR, VoxCPM, and selected model dependencies with test doubles. The fake CUDA availability/cache functions exercise control flow without CUDA execution. The test run emitted a pkg_resources deprecation warning. No model was downloaded or loaded, no inference or GPU experiment ran, and no end-to-end UI result is claimed by this fresh pass.

Historical GPU Evidence

The earlier validation recorded a successful real iic/SenseVoiceSmall transcription on an NVIDIA H100 80GB with PyTorch 2.11.0+cu128:

  • Model loaded: 906.64 MiB allocated / 964 MiB reserved.
  • After successful transcription and release: 33 MiB allocated / 34 MiB reserved.
  • demo.asr_model is None, with a completed transcript.

Those measurements are historical, were not rerun on September 29, and do not validate the exception path or establish a universal memory reduction. The earlier compile checks are also historical. Maintainer review is still pending; no merge, release, workflow approval, or manual CI dispatch was performed in this follow-up.

Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Randomly Crashing without Error

1 participant