Known Test Repository Issue Sometimes Misses Test Failures #6411

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1196220

Build leg reported

Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.BasicTest

Pull Request

dotnet/sdk#51591

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The SSL connection",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

So, I believe there is an issue with the known issue flagger, where it is failing to mark certain failures as known, even if they are a known issue. Please let me know if I am mistaken.

Here's the issue:
Looking at this image, this test was not considered a known issue:
https://github.com/dotnet/sdk/pull/51591/checks?check_run_id=54740778719

I believe that, because it's showing as a test failure, and other similar test failures are counted in the above section:

Furthermore, it appeared to be blocking build-analysis from being green.

Here's such an example of 1 of several 'known issues' that flag this issue:
#5015
dotnet/sdk#51574
#6410

As you can see in, #51574 looks for the string "The SSL connection could not be established" as a ErrorMessage which is supposed to use String.Contains. The test, however, that I think was not flagged as a 'known issue', did indeed have that error message string. It looks like some were flagged, while others were not. Here's the output of that 'unknown' test failure:

Expected command to exit with 0 but it did not.\nFile Name: /Users/runner/work/1/s/artifacts/bin/redist/Release/dotnet/dotnet\nArguments: new console --search --debug:custom-hive /Users/runner/work/1/s/artifacts/tmp/Release/dotnet-new.IntegrationTests/SharedHomeDirectory/20251106211951401\nExit Code: 103\nStdOut:\nWarning: use of 'dotnet new --search' is deprecated. Use 'dotnet new search' instead.\nFor more information, run: \n   dotnet new search -h\n\nSearching for the templates...\nStdErr:\nMatches from template source: NuGet.org\nSearch failed: Failed to update search cache. (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)\n\nFor details on the exit code, refer to https://aka.ms/templating-exit-codes#103\n [[Full log]](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220&view=ms.vss-test-web.build-test-results-tab&runId=33106418&resultId=100166&paneView=debug)

While there may be a confounding variable in that there are now several known issues which are tracking substrings of each other, or may overlap, the whole reason there are multiple in the first place is because the first existing known test repository issue did not flag all of the tests which failed with the known issue.

Could this alternatively be a limitation of:

1. A limit of how many tests can be flagged as a known issue for the same known issue
2. A different 'failing configuration' which causes issues to be missed?
3. A misunderstanding of which strings get searched? Note that I believe the error is also in the 'helix logs'.
4. A failure of my own accord to file the known issue? @dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL Connection C]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 11/6/2025 10:39:11 PM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL connection]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 11/6/2025 10:42:09 PM UTC

Report

Summary

Report

Summary