Skip to content

feat(benchmark): report first nonempty transcript latency - #3734

Merged
LauraGPT merged 1 commit into
mainfrom
codex/benchmark-first-text-latency-20260928
Sep 28, 2026
Merged

LauraGPT merged 1 commit into
mainfrom
codex/benchmark-first-text-latency-20260928

Conversation

@LauraGPT

Copy link
Copy Markdown
Collaborator

Summary

  • Preserve first_update_ms, which measures the first result message even when its transcript is empty.
  • Add first_text_ms for the first non-blank partial or sentence text, including final-only transcripts, in each client JSONL row.
  • Report first-text p50/p95 and clients_with_text alongside total clients and errors. Missing text remains null rather than becoming zero latency.
  • Document the distinction and run the focused CPU-only benchmark tests in a path-filtered GitHub Actions workflow.

Related to #3732. This makes first-result and first-visible-transcript measurements distinguishable; it does not resolve the reported high-concurrency tail latency.

Type of change

  • Bug fix
  • Documentation
  • Example or demo
  • Runtime or deployment
  • Benchmark or evaluation
  • Model/training change

Validation

  • Tests first: 8 failed and 5 passed on the unchanged benchmark, including missing exported fields and failure to record the first text.

  • After implementation: 13 focused tests passed, including deterministic empty/whitespace/final-only/zero-latency/missing-text cases and a real localhost WebSocket round trip with two concurrent clients and synthetic PCM16 WAV input.

  • Test formatting, Python compilation, CLI --help, and git diff --check passed.

  • Independent source review found no P1/P2 issue in the four-file scope.

  • python -m compileall funasr examples tests (targeted compilation passed for the changed client and test)

  • Docs or links checked

  • Runtime/deployment command tested (no server/deployment change; benchmark CLI help and localhost round trip tested)

User impact

Realtime service operators can distinguish an early empty result from the first visible transcript and report how many clients actually received text when investigating latency.

Notes for reviewers

No server, decoder, model, dependency manifest, or existing metric semantics changed. No models were loaded and no GPU benchmark was run. Client-visible text is not model token-generation latency, a transcript-quality assertion, or a throughput claim.

Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com>
@LauraGPT
LauraGPT merged commit 55203a9 into main Sep 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant