Repository navigation
[android] System.Net.Http.Functional.Tests failure with TimeoutException #118118
Description
Build Information
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1106900&view=results
Build error leg or test failing: System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http2.SslOptions_InvalidName_Throws
07-19 11:06:15.806 25025 1142 I DOTNET : 1) [FAIL] System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http2.SslOptions_InvalidName_Throws Test name: System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http2.SslOptions_InvalidName_Throws
07-19 11:06:15.806 25025 1142 I DOTNET : Assembly: [System.Net.Http.Functional.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
07-19 11:06:15.806 25025 1142 I DOTNET : Exception messages: System.TimeoutException : The operation has timed out. Exception stack traces: at System.Net.Http.Functional.Tests.SocketsHttpHandler_SecurityTest.SslOptions_InvalidName_Throws()
07-19 11:06:15.806 25025 1142 I DOTNET : --- End of stack trace from previous location ---
07-19 11:06:15.806 25025 1142 I DOTNET : Execution time: 72.3227827
Error Message
Fill the error message using step by step known issues guidance.
{
"ErrorMessage": ["[FAIL] System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http2.SslOptions_InvalidName_Throws", "TimeoutException"],
"BuildRetry": false,
"ExcludeConsoleLog": false
}Known issue validation
Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1106900
Error message validated: [TimeoutException.*System\.Net]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/28/2025 9:41:25 AM UTC
Known issue validation
Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1106900
Error message validated: [[FAIL] System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http2.SslOptions_InvalidName_Throws TimeoutException]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/29/2025 12:04:51 PM UTC
Report
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 53 | 210 | 251 |
Activity
dotnet-policy-service commentedon Jul 28, 2025
Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.
dotnet-policy-service commentedon Jul 28, 2025
Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.
BrzVlad commentedon Jul 29, 2025
This seems to have started happening around 19th of July. Potential commits in the area that I'm seeing #117540 or #117667. @antonfirsov @rzikm Any ideas about what could have started causing this ?
rzikm commentedon Jul 29, 2025
@BrzVlad I see the failures also on a PR from 18th, on #117748, maybe that change is relevant? cc @davidnguyen-tech
e.g. https://github.com/dotnet/runtime/runs/46277027211
edit: but that PR was merged on 21st, so it is possible that there is a different root cause altogether
rzikm commentedon Jul 29, 2025
Also, don't use too broad patterns for matching the exception,
TimeoutException.*System\.Netwould match ALL timeouts in the entire networking area.dotnet-policy-service commentedon Jul 29, 2025
Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.
davidnguyen-tech commentedon Jul 29, 2025
I don't think changes in #117748 are relevant - the PR just sets two specific env variables for globalization. Furthermore, the changes from that PR were replaced by a more robust solution in #117980, where we load runtime settings from
runtimeconfig.jsonand apply it using the AppContext, which doesn't seem relevant to me either.BrzVlad commentedon Jul 30, 2025
I haven't been able to reproduce this locally on x64 emulator, arm64 emulator or arm64 device. @simonrozsival Could you give it a try as well, maybe you are luckier ?
simonrozsival commentedon Jul 31, 2025
I was unable to reproduce the timeouts either today (arm64 emulator and arm64 device). I'm also not getting the timeouts in #117963 anymore and I don't think I pulled in new commits from upstream. I don't think I updated my local branch since, which confuses me a little. AFAIK all the tests use a local loopback server so I don't think this could be attributed to flaky network.
EDIT: @BrzVlad apparently I'm getting the timeouts reliably with emulator running API 29 (which we use in CI by default), but I'm not getting the timeouts on newer API levels. What API level emulator did you use?
simonrozsival commentedon Aug 4, 2025
@BrzVlad @rzikm I used
git bisecttoday to identify that the PR that is causing these timeouts is #117428. The problem seems to be here:https://github.com/dotnet/runtime/pull/117428/files#diff-af1f1cbeb01ef6f5fab8d41b75d3f9ac05e31fe12e424c3e07f0740294d292e8L1138-R1150
Previously, the
CreateFatalHandshakeAlertTokenmethod would be called on Android, but now it isn't sinceSslStreamPal.CanGenerateCustomAlertsis set tofalse. When I change this value totrue, the tests are passing again. I wonder if changing this constant is the right fix. It doesn't seem that Android APIs give us a way to generate custom alerts, but apparently it is necessary to run theCreateFatalHandshakeAlertTokenmethod on Android. @rzikm do you have any suggestions how to approach this?rzikm commentedon Aug 5, 2025
@simonrozsival, since
SslStreamPal.ApplyAlertTokenis no-op for Android, then the key is probably calling GenerateAlertToken (which in turn simply calls GenerateToken with empty args). I am curious why would there be any additional data to output that weren't returned as part of the previous call toSslStreamPal.HandshakeInternal, or if there is maybe some additional logic that is being triggered by that call.simonrozsival commentedon Aug 5, 2025
@rzikm when we don't call
CreateFatalHandshakeAlertToken, the serverSslStreamauthentication finishes (throws an exception), but theawait clientSslStream.AuthenticateAsClient(...)doesn't. The client gets stuck waiting for data from the server here:runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs
Line 765 in 0686df9
(
AuthenticateAsClientAsync->ForceAuthenticationAsync->ReceiveHandshakeFrameAsync->EnsureFullTlsFrameAsync)It seems like we are missing some (reasonably short) timeout here. The client can get stuck basically indefinitely waiting for data from the server which won't send any.
Now I believe the server
SslStreamon Android should send and flush all the TLS frames it is supposed to. On the other hand, the client should not wait for the response forever either. How are other platforms handling this?Note: It seems that our tests aren't passing a cancellationToken to the
AuthenticateAsClientmethod. Also ourLoopbackServerTimeoutMillisecondsis 60s which seems too much, although I understand we don't really want to introduce even more flakiness on other platforms by lowering this number.rzikm commentedon Aug 14, 2025
The definition of "reasonable" is difficult for us platforms. Users generally handle the timeouts via cancelling the passed in cancellation token. I believe we do the same in HttpClient.
I believe the right fix here would be investigating why on Android the last call to HandshakeInternal is needed and, if it makes sense, to restore that call on that platform.