Skip to content

Fix SOS Helix private-runtime overlay to use the runtime product version - #6143

Merged
steveisok merged 1 commit into
dotnet:mainfrom
steveisok:steveisok-sos-helix-runtime-version-fix
Oct 23, 2026
Merged

steveisok merged 1 commit into
dotnet:mainfrom
steveisok:steveisok-sos-helix-runtime-version-fix

Conversation

@steveisok

Copy link
Copy Markdown
Member

Problem

The SOS Helix legs of dotnet/runtime's runtime-diagnostics pipeline are blocked. Every SOS work item fails before any test runs. This affects all runtime PRs since roughly 2026-10-23 16:00 UTC. Example: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1646832 (dotnet/runtime#136362). Each work item exits 3 with:

The Helix-provisioned runtime was not found at '<HELIX_CORRELATION_PAYLOAD>/dotnet-cli/shared/Microsoft.NETCore.App/11.0.0-rtm.26514.118'.

Root cause

For private runtime testing (PrivateBuildTesting=true + LiveRuntimeDir), src/tests/SOS.Tests/HelixPayload.targets passes --runtime-version "%(RuntimeDownload)" to runtests.sh/runtests.cmd. They use that value as the shared/Microsoft.NETCore.App/<version> folder to overlay the private runtime onto.

RuntimeDownload (MicrosoftNETCorePlatformsVersion) is the build/download version. The installed folder, however, is named after the product version (Runtime = MicrosoftNETCoreAppRefVersion). These two values were identical until #6136 moved .NET 11 to stabilized RTM versioning, so the bug went unnoticed:

before #6136 after #6136
RuntimeDownload 11.0.0-rtm.26480.113 11.0.0-rtm.26514.118
Runtime 11.0.0-rtm.26480.113 11.0.0

How the folder gets its name: the Helix SDK's AddAdditionalRuntimes target (tools/dotnet-cli/DotNetCli.targets) resolves the archive with FindDotNetCliPackage and extracts it into dotnet-cli without changing its layout. The resolved archive, ci.dot.net/public/Runtime/11.0.0-rtm.26514.118/dotnet-runtime-11.0.0-linux-x64.tar.gz (its productVersion.txt is 11.0.0), contains shared/Microsoft.NETCore.App/11.0.0/.

Fix

  • Keep downloading by RuntimeDownload (RequiredRuntimeVersions is unchanged), but pass %(HelixWorkItem.Runtime) as --runtime-version.
  • Fail early with a clear error if a private runtime run has no Runtime metadata.
  • Document the build-version vs. product-version distinction in the SOS.Tests README.

I checked the other places that use these versions and they need no change: the SOS harness metadata, CommonTestRunner and DbgShim already configure by Runtime, the managed Helix projects use the stable TestHostRuntimeVersion, and macOS picks --fx-version from dotnet --list-runtimes.

Validation

  • I ran CreateHelixPayload through a scratch MSBuild project that imports eng/Version.Details.props, eng/Versions.props and SOS.Tests/HelixPayload.targets, with payload staging stubbed out:
    • Private, linux-x64 — before: --runtime-version "11.0.0-rtm.26514.118"; after: --runtime-version "11.0.0". RequiredRuntimeVersions stays 10.0.10;11.0.0-rtm.26514.118.
    • Private, win-x64 (runtests.cmd) — after: --runtime-version "11.0.0".
    • Non-private matrix (Net11/10/9/8) — the generated work items are unchanged.
    • Private run with empty Runtime — the new error is raised.
  • I ran runtests.sh against a fake correlation payload that copies the real archive layout (shared/Microsoft.NETCore.App/11.0.0). With the old value it reproduces the exact CI error. With the new value it overlays the runtime and continues.

The SOS Helix launcher overlays LiveRuntimeDir onto
shared/Microsoft.NETCore.App/<runtime-version>, but HelixPayload.targets
passed RuntimeDownload (build version, 11.0.0-rtm.26514.118). Helix
installs the runtime under its product version (11.0.0, the Runtime
metadata), so since stabilized .NET 11 versions arrived in dotnet#6136 every
private-runtime SOS work item exits 3 before running tests.

Keep downloading by RuntimeDownload but pass Runtime as --runtime-version.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9525fb2-c2dd-4524-8de0-24a2d0b0f6a7
@steveisok
steveisok requested a review from a team as a code owner October 23, 2026 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants