Repository navigation
16-second hang while establishing a transcribe streaming connection #1401
Copy link
Copy link
Open
Labels
bugThis issue is a bug.This issue is a bug.
Description
Describe the bug
On a particular machine, Transcribe Streaming.StartStreamTranscription hangs for 16 seconds before finally succeeding (then AWS transcribe immediately times out because it hasn't received audio for more than 15 seconds)
Regression Issue
- Select this option if this issue appears to be a regression.
Expected Behavior
I expect connection to be established in a timely fashion
Current Behavior
Connection establishment hangs for 16 seconds on running read_after_execution interceptors:
2026-01-27T06:40:35.945921Z DEBUG Transcribe Streaming.StartStreamTranscription{rpc.service="Transcribe Streaming" rpc.method="StartStreamTranscription" sdk_invocation_id=5764325 rpc.system="aws-api"}:try_op: aws_smithy_runtime::client::retries::strategy::standard: attempt #1 classified as NoActionIndicated, not retrying
2026-01-27T06:40:35.945927Z DEBUG Transcribe Streaming.StartStreamTranscription{rpc.service="Transcribe Streaming" rpc.method="StartStreamTranscription" sdk_invocation_id=5764325 rpc.system="aws-api"}:try_op: aws_smithy_runtime::client::orchestrator: a retry is either unnecessary or not possible, exiting attempt loop
2026-01-27T06:40:35.945939Z TRACE Transcribe Streaming.StartStreamTranscription{rpc.service="Transcribe Streaming" rpc.method="StartStreamTranscription" sdk_invocation_id=5764325 rpc.system="aws-api"}:finally_op: aws_smithy_runtime::client::interceptors: running `modify_before_completion` interceptors
2026-01-27T06:40:35.945948Z TRACE Transcribe Streaming.StartStreamTranscription{rpc.service="Transcribe Streaming" rpc.method="StartStreamTranscription" sdk_invocation_id=5764325 rpc.system="aws-api"}:finally_op: aws_smithy_runtime::client::interceptors: running `read_after_execution` interceptors
2026-01-27T06:40:51.956883Z TRACE Transcribe Streaming.StartStreamTranscription{rpc.service="Transcribe Streaming" rpc.method="StartStreamTranscription" sdk_invocation_id=5764325 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}:Connection{peer=Client}:poll: h2::proto::connection: connection.state=Open
Reproduction Steps
This only reproduces on a single machine, and occurs using GStreamer's AWS transcriber when the connection builder is built:
I can probably turn this into a sample, but the issue most likely won't be reproducable everywhere
Possible Solution
I don't have an idea what might be causing this
Additional Information/Context
No response
Version
├── aws-config v1.8.12
│ ├── aws-credential-types v1.2.11
│ │ ├── aws-smithy-async v1.2.7
│ │ ├── aws-smithy-runtime-api v1.10.0
│ │ │ ├── aws-smithy-async v1.2.7 (*)
│ │ │ ├── aws-smithy-types v1.3.6
│ │ ├── aws-smithy-types v1.3.6 (*)
│ ├── aws-runtime v1.5.18
│ │ ├── aws-credential-types v1.2.11 (*)
│ │ ├── aws-sigv4 v1.3.7
│ │ │ ├── aws-credential-types v1.2.11 (*)
│ │ │ ├── aws-smithy-eventstream v0.60.14
│ │ │ │ ├── aws-smithy-types v1.3.6 (*)
│ │ │ ├── aws-smithy-http v0.62.6
│ │ │ │ ├── aws-smithy-eventstream v0.60.14 (*)
│ │ │ │ ├── aws-smithy-runtime-api v1.10.0 (*)
│ │ │ │ ├── aws-smithy-types v1.3.6 (*)
│ │ │ ├── aws-smithy-runtime-api v1.10.0 (*)
│ │ │ ├── aws-smithy-types v1.3.6 (*)
│ │ ├── aws-smithy-async v1.2.7 (*)
│ │ ├── aws-smithy-eventstream v0.60.14 (*)
│ │ ├── aws-smithy-http v0.62.6 (*)
│ │ ├── aws-smithy-runtime v1.9.8
│ │ │ ├── aws-smithy-async v1.2.7 (*)
│ │ │ ├── aws-smithy-http v0.62.6 (*)
│ │ │ ├── aws-smithy-http-client v1.1.5
│ │ │ │ ├── aws-smithy-async v1.2.7 (*)
│ │ │ │ ├── aws-smithy-runtime-api v1.10.0 (*)
│ │ │ │ ├── aws-smithy-types v1.3.6 (*)
│ │ │ ├── aws-smithy-observability v0.2.0
│ │ │ │ └── aws-smithy-runtime-api v1.10.0 (*)
│ │ │ ├── aws-smithy-runtime-api v1.10.0 (*)
│ │ │ ├── aws-smithy-types v1.3.6 (*)
│ │ ├── aws-smithy-runtime-api v1.10.0 (*)
│ │ ├── aws-smithy-types v1.3.6 (*)
│ │ ├── aws-types v1.3.11
│ │ │ ├── aws-credential-types v1.2.11 (*)
│ │ │ ├── aws-smithy-async v1.2.7 (*)
│ │ │ ├── aws-smithy-runtime-api v1.10.0 (*)
│ │ │ ├── aws-smithy-types v1.3.6 (*)
│ ├── aws-sdk-sts v1.96.0
│ │ ├── aws-credential-types v1.2.11 (*)
│ │ ├── aws-runtime v1.5.18 (*)
│ │ ├── aws-smithy-async v1.2.7 (*)
│ │ ├── aws-smithy-http v0.62.6 (*)
│ │ ├── aws-smithy-json v0.61.9
│ │ │ └── aws-smithy-types v1.3.6 (*)
│ │ ├── aws-smithy-observability v0.2.0 (*)
│ │ ├── aws-smithy-query v0.60.9
│ │ │ ├── aws-smithy-types v1.3.6 (*)
│ │ ├── aws-smithy-runtime v1.9.8 (*)
│ │ ├── aws-smithy-runtime-api v1.10.0 (*)
│ │ ├── aws-smithy-types v1.3.6 (*)
│ │ ├── aws-smithy-xml v0.60.13
│ │ ├── aws-types v1.3.11 (*)
│ ├── aws-smithy-async v1.2.7 (*)
│ ├── aws-smithy-http v0.62.6 (*)
│ ├── aws-smithy-json v0.61.9 (*)
│ ├── aws-smithy-runtime v1.9.8 (*)
│ ├── aws-smithy-runtime-api v1.10.0 (*)
│ ├── aws-smithy-types v1.3.6 (*)
│ ├── aws-types v1.3.11 (*)
├── aws-credential-types v1.2.11 (*)
├── aws-sdk-polly v1.97.0
│ ├── aws-credential-types v1.2.11 (*)
│ ├── aws-runtime v1.5.18 (*)
│ ├── aws-sigv4 v1.3.7 (*)
│ ├── aws-smithy-async v1.2.7 (*)
│ ├── aws-smithy-http v0.62.6 (*)
│ ├── aws-smithy-json v0.61.9 (*)
│ ├── aws-smithy-observability v0.2.0 (*)
│ ├── aws-smithy-runtime v1.9.8 (*)
│ ├── aws-smithy-runtime-api v1.10.0 (*)
│ ├── aws-smithy-types v1.3.6 (*)
│ ├── aws-types v1.3.11 (*)
├── aws-sdk-s3 v1.120.0
│ ├── aws-credential-types v1.2.11 (*)
│ ├── aws-runtime v1.5.18 (*)
│ ├── aws-sigv4 v1.3.7 (*)
│ ├── aws-smithy-async v1.2.7 (*)
│ ├── aws-smithy-checksums v0.63.13
│ │ ├── aws-smithy-http v0.62.6 (*)
│ │ ├── aws-smithy-types v1.3.6 (*)
│ ├── aws-smithy-eventstream v0.60.14 (*)
│ ├── aws-smithy-http v0.62.6 (*)
│ ├── aws-smithy-json v0.61.9 (*)
│ ├── aws-smithy-observability v0.2.0 (*)
│ ├── aws-smithy-runtime v1.9.8 (*)
│ ├── aws-smithy-runtime-api v1.10.0 (*)
│ ├── aws-smithy-types v1.3.6 (*)
│ ├── aws-smithy-xml v0.60.13 (*)
│ ├── aws-types v1.3.11 (*)
├── aws-sdk-sso v1.92.0
│ ├── aws-credential-types v1.2.11 (*)
│ ├── aws-runtime v1.5.18 (*)
│ ├── aws-smithy-async v1.2.7 (*)
│ ├── aws-smithy-http v0.62.6 (*)
│ ├── aws-smithy-json v0.61.9 (*)
│ ├── aws-smithy-observability v0.2.0 (*)
│ ├── aws-smithy-runtime v1.9.8 (*)
│ ├── aws-smithy-runtime-api v1.10.0 (*)
│ ├── aws-smithy-types v1.3.6 (*)
│ ├── aws-types v1.3.11 (*)
├── aws-sdk-sts v1.96.0 (*)
├── aws-sdk-transcribestreaming v1.96.0
│ ├── aws-credential-types v1.2.11 (*)
│ ├── aws-runtime v1.5.18 (*)
│ ├── aws-sigv4 v1.3.7 (*)
│ ├── aws-smithy-async v1.2.7 (*)
│ ├── aws-smithy-eventstream v0.60.14 (*)
│ ├── aws-smithy-http v0.62.6 (*)
│ ├── aws-smithy-json v0.61.9 (*)
│ ├── aws-smithy-observability v0.2.0 (*)
│ ├── aws-smithy-runtime v1.9.8 (*)
│ ├── aws-smithy-runtime-api v1.10.0 (*)
│ ├── aws-smithy-types v1.3.6 (*)
│ ├── aws-types v1.3.11 (*)
├── aws-sdk-translate v1.92.0
│ ├── aws-credential-types v1.2.11 (*)
│ ├── aws-runtime v1.5.18 (*)
│ ├── aws-smithy-async v1.2.7 (*)
│ ├── aws-smithy-http v0.62.6 (*)
│ ├── aws-smithy-json v0.61.9 (*)
│ ├── aws-smithy-observability v0.2.0 (*)
│ ├── aws-smithy-runtime v1.9.8 (*)
│ ├── aws-smithy-runtime-api v1.10.0 (*)
│ ├── aws-smithy-types v1.3.6 (*)
│ ├── aws-types v1.3.11 (*)
├── aws-types v1.3.11 (*)
Environment details (OS name and version, etc.)
Fedora 43
Logs
I pasted some relevant logs above :)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugThis issue is a bug.This issue is a bug.
Activity
vcjana commentedon Jan 27, 2026
Isn't this issue about how the GStreamer plugin is calling the async code? Looking at the transcriber code it uses
futures::executor::block_on()The SDK uses tokio for managing the HTTP/2 connections. When you use
futures::executor::block_on, it creates a separate executor and blocks the thread which prevents it from polling the h2 connections. You are seeing a 16s hang because it is waiting for a timeout.I think you could use the tokio's block_on instead:
I noticed the GStreamer crate already has a RUNTIME defined, so this should just work. Can you try that and let me know if it fixes the hang?
MathieuDuponchelle commentedon Jan 27, 2026
@sdroege has a MR to do precisely that (https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/2835), but as far as I know that wasn't sufficient. I'll check with him what the status is, if that's also your guess then that's probably right, it's annoying that I can't reproduce this on my end :( Thanks a lot for looking, I know I didn't attach a standalone reproducer :)
I'll keep this updated as soon as I have more info
vcjana commentedon Jan 27, 2026
Interesting that the MR didn't fully fix it. If the issue persists after that change, could you grab logs with
RUST_LOG=tokio=trace,h2=tracewhen you can reproduce it ? That would help us see exactly where the time is being spent.MathieuDuponchelle commentedon Jan 27, 2026
Interestingly enough I can now reproduce both with his branch and the original one, this is very strange, trying to make a minimal reproducer now
vcjana commentedon Jan 27, 2026
Great that would be helpful.
MathieuDuponchelle commentedon Jan 27, 2026
OK so here's the de-gstreamerized minimal reproducer (not using
futures::executor::block_on):And here are the logs (all of them, just
RUST_LOG=trace):Still that hang right after the
read_after_executioninterceptors are mentioned 🤔MathieuDuponchelle commentedon Jan 27, 2026
This part gave me an idea and I tried to add the following before establishing the connection:
Magically this causes the connection to finally get established in a timely fashion.
I'm fairly convinced this is a regression, the element used to work when waiting until
builder.audio_stream()completed before sending any data, I'm a bit puzzled as to why I suddenly started reproducing without runningcargo updatethoughMathieuDuponchelle commentedon Jan 27, 2026
This means I can fix the element by slightly refactoring it of course, but I'm curious what your thoughts are here @vcjana ?
MathieuDuponchelle commentedon Jan 27, 2026
wild guess, but could perhaps be AWS doing a phased rollout of the service side, and not actually an issue in the client side code (SDK)?
vcjana commentedon Jan 27, 2026
From the logs i understand that the SDK returns as soon as response headers arrive, but AWS enforces a 15-second idle timeout on the audio stream. Since no audio was being sent, it timed out.
MathieuDuponchelle commentedon Jan 27, 2026
But the await used to return without audio being sent out, and now it no longer does :)
MathieuDuponchelle commentedon Jan 27, 2026
Note that this 15-second with no audio timeout used to exist before by the way
vcjana commentedon Jan 27, 2026
Ah good point if the timeout always existed but send().await used to return right away, something did change.
MathieuDuponchelle commentedon Jan 27, 2026
It wasn't send().await that used to return right away butnevermind can't read goodbuilder.audio_stream(chunk_stream.into()).send().await, without requiring data being sent :)MathieuDuponchelle commentedon Jan 27, 2026
lol my bad, there's two send() now you're exactly right
vcjana commentedon Jan 27, 2026
In the logs:
The h2 connection isn't being polled during those 16 seconds. In your reproducer:
_audio_txis never used but also never dropped. So when the SDK polls your body stream,audio_rx.next().awaitblocks forever waiting for data that never comes.This blocks the h2 connection from being driven. After 15 seconds, AWS times out and closes the stream server-side, which finally unblocks everything.
Your fix works because spawning audio data unblocks
audio_rx.next(), allowing the h2 connection to be polled properly.MathieuDuponchelle commentedon Jan 27, 2026
Right, but the element does essentially the same thing, and (sorry for repeating myself) that used to not be a problem. Why does the SDK now need to read something from the audio stream before returning from
builder.audio_stream(chunk_stream.into()).send().await;?vcjana commentedon Jan 27, 2026
Which SDK version are you using?
vcjana commentedon Jan 27, 2026
This may be related to a known issue in smithy-rs
smithy-lang/smithy-rs#4435
There's a PR (#4440) that may address this. We're still investigating if this is the exact cause of your issue.
MathieuDuponchelle commentedon Jan 27, 2026
Ah great, I never know where to look for issues with the myriad of repos :)
Here's my cargo tree:
vcjana commentedon Jan 27, 2026
The PR is open. I'll follow up when the fix is released. Your workaround should be the correct approach until we release the fix.
MathieuDuponchelle commentedon Jan 27, 2026
@vcjana thanks again for the diligence and very fast response times, we'll refactor things on our end, might make code simpler anyways :)
vcjana commentedon Jan 27, 2026
Happy to help. Thanks for the detailed reproducer and logs made it much easier to track down
MathieuDuponchelle commentedon Jan 30, 2026
for reference: https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/2849
awstranscriber2: workaround suspected rust SDK regression
awstranscriber2: workaround suspected rust SDK regression