[Experimental] Add MSBuild task-level cache for crossgen2 invocations by sbomer · Pull Request #135446 · dotnet/runtime

sbomer · 2026-10-08T20:25:09Z

Stacked on #135886, which extracts the shared cache lifecycle and storage. Adds an experimental MSBuild task-level crossgen2 cache, enabled only by CROSSGEN2_EXPERIMENTAL_CACHE=true, with optional CROSSGEN2_EXPERIMENTAL_CACHE_PATH. The initial scope is directly executed Linux-x64 and Windows-x64 compilers targeting the corresponding host, with non-composite PE output and supported profiles/symbols.

Crossgen2 uses #135886's shared output contracts and SHA-256 hashing/restore/store pipeline; the task retains eligibility, input discovery, output preparation, and diagnostic policy. It declares owned output files, including absent symbols, without clearing unrelated neighbors. This PR adds optional task-output metadata callbacks so cache hits restore WarningsDetected. Neither cache replays compiler diagnostic text.

Earlier Linux measurements showed CoreLib compilation falling from about 7.9 seconds to a 90 ms hit with byte-identical output, and a 100-assembly workload falling from about 24.4 to 6.7 seconds. Latest analyzer-enabled validation passed 202 crossgen tests, including three real-compiler cases, and 65 maintenance-tool tests; six Windows-only cases were skipped. Net472 was previously built but not executed on Linux.

This remains a draft experiment without automatic enablement or eviction, size limits, or a stable-format guarantee. External managed-runtime/hosting dependencies must remain stable; unsupported invocations visibly bypass caching. The mirrored task changes and common sources still need a coordinated SDK port. See src/tasks/Crossgen2Tasks/README.md for the precise identity inventory and supported contract.

Note

This content was created with assistance from AI.

Adds an opt-in, environment-gated cache for RunReadyToRunCompiler, covering native Linux-x64, non-composite PE crossgen2 compilation. Crossgen2Cache computes a key from full SHA-256 hashes of the task assembly, compiler, JIT, inputs/references/profiles, and effective arguments/options, then restores or stores the declared outputs (image + perf map) with atomic publish via cache-local staging and rename. Cache hits replay captured diagnostics and preserve the original exit code. Unsupported invocations (composites, legacy crossgen, extra arguments, non-Linux-x64, managed-host deployment) visibly bypass caching. Adds Crossgen2Tasks.Tests with unit, negative-control, and real crossgen2 integration coverage, and documents the contract, hashed-input scope, deliberate omissions, and dotnet/sdk coordination needed in README.md. Assisted-by: GitHub Copilot CLI:claude-sonnet-5 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Real crossgen2 invocations driven through Microsoft.NET.CrossGen.targets (including the CoreLib build) always set Crossgen2ContainerFormat="pe" (the file's own default when unset) and pass extra, pathless boolean switches such as --embed-pgo-data. Both unconditionally bypassed the cache, so it never engaged for any real-world caller despite working in synthetic tests. - Treat an explicit "pe" container format the same as unset: both mean plain PE output, the only format this cache supports. - Replace the all-or-nothing extra-argument bypass with a small allowlist of known, pathless boolean switches actually used by in-repo callers. Each is still included verbatim in the cache key, so a different value still produces a different entry; anything unrecognized still bypasses. Verified end-to-end against the real CoreLib crossgen2 invocation: a cold run now populates the cache (cache miss/store), and a subsequent run restores from it (cache hit), with byte-identical output and no change to bypassed/unsupported shapes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

azure-pipelines · 2026-10-08T20:26:03Z

Azure Pipelines: Successfully started running 4 pipeline(s). 12 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-10-08T20:26:18Z

Tagging subscribers to this area: @dotnet/crossgen-contrib
See info in area-owners.md if you want to be subscribed.

Copilot

🟡 Changes recommended

The test suite is not integrated as an automated test project, and the new environment variables violate repository naming conventions.

5 open findings

Use a directory to reliably trigger profile read failure · New
Configure the project as a repository test project · New
Rename cache variables with the required DOTNET_ prefix · New
Register the regression tests in automated CI · New
Clarify bypass diagnostic for unrecognized extra arguments · New

What changed in this PR

Adds an opt-in, content-addressed cache for supported Linux-x64 crossgen2 task invocations.

Changes:

Implements atomic cache storage, validation, restoration, and diagnostic replay.
Adds eligibility checks and documentation for supported invocation shapes.
Adds unit and optional real-compiler tests.

File	Description
`src/tasks/tasks.proj`	Excludes the test project from task traversal.
`src/tasks/Crossgen2Tasks/RunReadyToRunCompiler.cs`	Integrates caching into compiler execution.
`src/tasks/Crossgen2Tasks/README.md`	Documents configuration and cache behavior.
`src/tasks/Crossgen2Tasks/Crossgen2Tasks.csproj`	Exposes internals to tests.
`src/tasks/Crossgen2Tasks/Crossgen2Cache.cs`	Implements cache identity, storage, and restoration.
`src/tasks/Crossgen2Tasks.Tests/Crossgen2Tasks.Tests.csproj`	Defines the new test project.
`src/tasks/Crossgen2Tasks.Tests/Crossgen2CacheTests.cs`	Tests cache behavior and concurrency.

🧠 Review effort: Balanced

Register the non-shipping test project in clr.toolstests, make the unreadable-profile case independent of Unix permissions, and clarify the extra-argument bypass diagnostic. Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Normalize the IL directory separately from the item glob so MSBuild supplies individual reference files. The function-wrapped wildcard was passed literally to crossgen2 and made the invocation cache bypass because it could not fingerprint those references. Uncached compilation and cached restoration produce byte-identical images and perf maps. Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cc6353a6-7def-4dd2-8d2a-2914edbec8ad

Cache response-file path SearchValues and use Convert.ToHexString while preserving uppercase SHA-256 identities. Keep hashing the on-disk task assembly, explicitly bypass caching when its location is unavailable, and narrowly suppress the single-file analyzer warning at the guarded lookup. Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Keep ILLink configuration and input identity in the task while sharing entry validation, publication, directory restoration, and usage metadata. Preserve v1 serialization, mutex names, diagnostics, and purge semantics. Compile shared sources into consumers without adding a runtime assembly. Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Model owned directories and declared file sets in shared source. Keep tool identity and diagnostics in the task adapters, and preserve ILLink's directory format. Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Validate rooted and adjacent CodeView PDB candidates before cleanup or cache restoration in every invocation mode. Share candidate discovery with cache identity and cover enabled, disabled, bypassed, composite and legacy invocations. Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Copilot

🟡 Changes recommended

Windows drive-relative CodeView paths fail unconditional task validation instead of bypassing caching, including when caching is disabled.

1 open issue

Avoid rejecting Windows drive-relative paths when caching is disabled · New

1 resolved since last review

Prevent cleanup from deleting overlapping input PDBs

_{Review effort: Balanced}

Keep drive-relative CodeView candidates available for ordinary per-drive overlap validation while cache input validation continues to reject them. Extend Windows coverage for cache-disabled execution and root/adjacent PDB overlap protection. Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Copilot

🟡 Changes recommended

Wildcard inputs are validated as literal paths, allowing overlap checks to miss compiler inputs that ExecuteNormally can delete before compilation, even when caching is disabled.

1 open issue

Expand wildcard inputs before validation to prevent deleting compiler inputs · New

1 resolved since last review

Avoid rejecting Windows drive-relative paths when caching is disabled

_{Review effort: Balanced}

Expand crossgen2 assembly and reference patterns before overlap and associated-symbol validation while retaining wildcard cache bypass. Cover reference and composite inputs, stale symbols, disabled caching, and non-overlapping patterns. Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Copilot

🔵 Needs a closer look

Human assessment is needed because inconsistent input normalization can allow cleanup to delete an actual compiler input, even when caching is disabled.

No open issues 🎉

1 resolved since last review

Expand wildcard inputs before validation to prevent deleting compiler inputs

_{Review effort: Balanced}

Extracts ILLink's experimental invocation cache into shared sources, compiled into each consumer without adding a shared DLL. Tasks provide invocation descriptions and prepare their outputs before executing; the shared engine handles content hashing, lookup, restoration, storage, and purge. This intentionally goes beyond a mechanical extraction: the common output contract supports owned directories and file sets, records absent outputs, and uses checksummed manifests. That changes the experimental v1 entry format: existing entries are not reused, but remain purgable. #135446 builds on this infrastructure to add crossgen2 caching. Replaces the ILLink-specific maintenance package and command with `dotnet-build-cache`, shared by both caches. Existing users should install that package instead of `dotnet-illink-cache` and replace `dotnet illink-cache purge ...` with `dotnet build-cache purge --cache illink ...`. Commands already selecting an explicit directory can use `dotnet build-cache purge --cache-directory <path> ...` without a cache name. The cache-enablement environment variables are unchanged. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Assisted-by: Copilot:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Protect leading-whitespace primary and composite assembly paths and their associated PDBs before output cleanup, regardless of cache eligibility. Assisted-by: Copilot:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Copilot

🟡 Changes recommended

Validation mistakes the native-PDB output record for an input dependency, blocking Windows .NET 5 PDB-only compilation before /CreatePDB runs even when caching is disabled.

1 open issue

Exclude native PDB output from PDB-only input validation · New

_{Review effort: Balanced}

Preserve PDB-only compilation of native images without weakening protection of original IL symbols or reference inputs. Assisted-by: GitHub Copilot:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

Copilot

🔵 Needs a closer look

Human assessment is needed because inconsistent symbol-path whitespace handling can misdirect cleanup and overlap validation, risking deletion of unrelated files while leaving stale compiler outputs.

No open issues 🎉

1 resolved since last review

Exclude native PDB output from PDB-only input validation

_{Review effort: Balanced}

sbomer and others added 2 commits October 5, 2026 13:12

github-actions Bot added the area-ReadyToRun label Oct 8, 2026

dotnet-policy-service Bot assigned sbomer Oct 8, 2026

github-project-automation Bot added this to AppModel Oct 8, 2026

sbomer requested a balanced review from Copilot October 8, 2026 22:23

Copilot started reviewing on behalf of sbomer October 8, 2026 22:24

View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

build-analysis Bot mentioned this pull request Oct 9, 2026

System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback fails with Assert.Contains() Failure #134230

Closed

sbomer and others added 3 commits October 9, 2026 09:38

Merge main into crossgen-cache

a117962

Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

This was referenced Oct 9, 2026

[browser] Wasm.Build.Tests timeout - Timed out after 10s waiting for 'WASM EXIT' #116697

Open

Crash: access violation in coreclr!PgoManager::getPgoInstrumentationResults during tier-1 background JIT (Windows x86) #134695

Open

sbomer and others added 2 commits October 13, 2026 11:36

Support Windows crossgen2 caching and declared symbol directories

4a88cee

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cc6353a6-7def-4dd2-8d2a-2914edbec8ad

Clarify experimental crossgen2 cache hosting policy

5ba8111

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cc6353a6-7def-4dd2-8d2a-2914edbec8ad

This was referenced Oct 14, 2026

[ci-scan] Test failure: System.Net.Security.Tests.SslStream tests fail with "The remote certificate was rejected ..." #135096

Closed

Build race condition while building helperildll.dll #135717

Closed

sbomer and others added 4 commits October 15, 2026 11:46

Merge remote-tracking branch 'origin/main' into crossgen-cache

b573d70

Merge shared ILLink cache foundation

b726a80

Assisted-by: GitHub Copilot CLI:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

sbomer mentioned this pull request Oct 15, 2026

Extract shared invocation cache storage from ILLink #135886

Merged

dotnet-policy-service Bot added the linkable-framework Issues associated with delivering a linker friendly framework label Oct 15, 2026

sbomer requested a balanced review from Copilot October 26, 2026 20:28

Copilot started reviewing on behalf of sbomer October 26, 2026 20:28

View session

Copilot AI reviewed Oct 26, 2026

View reviewed changes

Comment thread src/tasks/Crossgen2Tasks/Crossgen2Cache.cs Resolved

sbomer requested a balanced review from Copilot October 27, 2026 20:20

Copilot started reviewing on behalf of sbomer October 27, 2026 20:20

View session

Copilot AI reviewed Oct 27, 2026

View reviewed changes

Comment thread src/tasks/Crossgen2Tasks/RunReadyToRunCompiler.cs Resolved

sbomer requested a balanced review from Copilot October 28, 2026 21:11

Copilot started reviewing on behalf of sbomer October 28, 2026 21:11

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

build-analysis Bot mentioned this pull request Oct 29, 2026

System.Net.Http.Functional.Tests.SocketsHttpHandler_DiagnosticsTest_Http3.SendAsync_ReuseRequestInHandler_ResetsHeadersForEachReuse: System.TimeoutException #132085

Open

Base automatically changed from sbomer/shared-invocation-cache to main October 29, 2026 17:34

sbomer and others added 2 commits October 29, 2026 11:13

Merge main into crossgen cache experiment

2b6aac2

Assisted-by: Copilot:gpt-6.1-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: da7cf090-9a92-453d-beb2-f672a9a7b541

sbomer requested a balanced review from Copilot October 29, 2026 18:15

Copilot started reviewing on behalf of sbomer October 29, 2026 18:15

View session

Copilot AI reviewed Oct 29, 2026

View reviewed changes

Comment thread src/tasks/Crossgen2Tasks/RunReadyToRunCompiler.cs Resolved

sbomer requested a balanced review from Copilot October 29, 2026 19:02

Copilot started reviewing on behalf of sbomer October 29, 2026 19:02

View session

Copilot AI reviewed Oct 29, 2026

View reviewed changes

Navigation Menu

Repository navigation

[Experimental] Add MSBuild task-level cache for crossgen2 invocations - #135446

[Experimental] Add MSBuild task-level cache for crossgen2 invocations#135446
sbomer wants to merge 43 commits into
mainfrom
sbomer/crossgen-cache

sbomer commented Oct 8, 2026 •

edited

Loading

Uh oh!

azure-pipelines Bot commented Oct 8, 2026

Uh oh!

dotnet-policy-service Bot commented Oct 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sbomer commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azure-pipelines Bot commented Oct 8, 2026

Uh oh!

dotnet-policy-service Bot commented Oct 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sbomer commented Oct 8, 2026 •

edited

Loading