Repository navigation
Filter leading GC polling frames from EventPipe stacks. - #135219
Merged
lateralusX merged 2 commits intoOct 19, 2026
Merged
lateralusX merged 2 commits into
lateralusX merged 2 commits into
Conversation
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
Tagging subscribers to this area: @agocke |
mdh1418
approved these changes
Oct 15, 2026
mdh1418
approved these changes
Oct 16, 2026
Make the non-inlined PollGCWorker helper binder-visible and bind it before managed execution, allowing stack walks to recognize polling helpers without loading methods during suspension. Omit only leading PollGC and PollGCWorker frames in the shared EventPipe stack walker, preserving caller frames, samples, and classification. Add raw-stack regression coverage using JIT and rundown method ranges, including caller retention and ordinary event stacks. Exclude interpreter configurations, which use a different safepoint path. Fixes dotnet#135050
lateralusX
force-pushed
the
lateralusX/fix-135050-pollgcworker-in-cpu-sample
branch
from
October 16, 2026 16:15
9e51f27 to
8d3638f
Compare
mdh1418
approved these changes
Oct 16, 2026
This was referenced Oct 16, 2026
Closed
Member
Author
|
/ba-g The investigated failures are unrelated to this PR’s EventPipe changes. Mutex scope and directory-permission failures are tracked in #135887 and fixed by merged #135898; all nine failed entries in run 45496548 were checked against the JSON results and Helix logs. The Windows x86 System.Runtime.Tests crash matches #136020, which explicitly cites this build, with a pending fix in #136021. The HalfEstimateR2R path-casing failure is fixed by merged #135976. Android HTTP certificate timeouts are matched by Build Analysis to #123572. |
This was referenced Oct 19, 2026
steveisok
pushed a commit
that referenced
this pull request
Oct 19, 2026
#136077) Fixes Issue #135050 main PR #135219 # Description Manually adapted backport of #135219 to release/11.0. EventPipe sampling suspension can cause threads to enter the managed GC polling helpers, which then appear at the top of captured stacks and obscure the actual workload. Make PollGCWorker binder-visible and bind it during runtime startup, allowing the EventPipe stack walker to omit leading PollGC and PollGCWorker frames without loading methods during suspension. Preserve caller frames, samples, and sample classification. The backport uses Thread.CoreCLR.cs and the feature guards appropriate to .NET 11, preserving its existing prestub and null-control-PC handling. # Customer Impact CPU-sampling profiles can attribute almost all samples to PollGCWorker instead of the application workload, making performance investigations misleading. This is a profiling artifact, not evidence of equivalent CPU usage or GC pressure. # Regression Yes. The issue was reported as a regression from .NET 9 to .NET 10 after GC polling moved into managed helpers. It also reproduces on the unfixed .NET 11 branch; it is not newly introduced in .NET 11. # Testing Validated locally on Windows x64 with Checked CoreCLR and Release libraries. Build commands, all completed successfully: ```cmd build.cmd clr+libs+host -rc Checked -lc Release -hc Release src\tests\build.cmd -Test tracing\eventpipe\eventsvalidation\SampleProfilerSampleType.csproj x64 Checked -Priority 1 -SkipNative /p:LibrariesConfiguration=Release /p:HostConfiguration=Release build.cmd clr -rc Checked -lc Release -hc Release src\tests\build.cmd -GenerateLayoutOnly x64 Checked /p:LibrariesConfiguration=Release /p:HostConfiguration=Release ``` From the SampleProfilerSampleType test output directory: ```cmd "%CORE_ROOT%\corerun.exe" SampleProfilerSampleType.dll ``` - Before the fix: failed with exit code 101, reporting a GC polling helper at the top of a sample. - After the fix: passed with exit code 100, checking 12,606 sample stacks with zero lost events. Also ran the same executable with DOTNET_TieredCompilation=0: both the existing sample-classification test and the new regression passed, with exit code 100 and zero lost events. The regression checks raw sample stacks, retention of workload frames in managed samples, and retention of workload frames in ordinary event stacks. Other platforms and .NET 10 have not been validated. # Risk Low. The change is narrowly scoped to identifying GC polling helpers and omitting them only while they are leading frames in an EventPipe stack. It does not change GC polling or suspension semantics, discard samples, or change sample classification. The worker is bound before managed execution, avoiding method loading during stack walks. Existing handling of other frames is preserved. Local testing confirmed the original failure and verified both the regression and existing sample-classification coverage after the fix. Resolves #135050
steveisok
added a commit
that referenced
this pull request
Oct 23, 2026
#136077 (#136091) Fixes Issue #135050 main PR #135219 # Description Manually adapted backport of #135219 to `release/10.0`, corresponding to the .NET 11 backport in #136077. EventPipe sampling suspension can cause threads to enter the managed GC polling helpers, which then appear at the top of captured stacks and obscure the actual workload. Make `PollGCWorker` binder-visible and bind it during runtime startup, allowing the EventPipe stack walker to omit leading `PollGC` and `PollGCWorker` frames without loading methods during suspension. Preserve caller frames, samples, and sample classification. The backport uses `Thread.CoreCLR.cs` and the feature guards appropriate to .NET 10, preserving its existing null-control-PC handling. Regression coverage is added to the existing `GCEvents` test project because .NET 10 does not have the newer `SampleProfilerSampleType` test. # Customer Impact CPU-sampling profiles can attribute almost all samples to `PollGCWorker` instead of the application workload, making performance investigations misleading. This is a profiling artifact, not evidence of equivalent CPU usage or GC pressure. # Regression Yes. The issue was reported as a regression from .NET 9 to .NET 10 after GC polling moved into managed helpers. The regression was reproduced locally on the unfixed .NET 10 runtime. # Testing Validated locally on Windows x64 with Checked CoreCLR and Release libraries. Build commands, all completed successfully with the local-only build prerequisites described below: ```cmd build.cmd clr+libs+host -rc Checked -lc Release -hc Release src\tests\build.cmd -Test tracing\eventpipe\eventsvalidation\GCEvents.csproj x64 Checked -Priority 1 -SkipNative /p:LibrariesConfiguration=Release /p:HostConfiguration=Release build.cmd clr -rc Checked -lc Release -hc Release src\tests\build.cmd -GenerateLayoutOnly x64 Checked /p:LibrariesConfiguration=Release /p:HostConfiguration=Release ``` Test commands: ```cmd set CORE_ROOT=C:\git\runtime\artifacts\tests\coreclr\windows.x64.Checked\Tests\Core_Root pushd C:\git\runtime\artifacts\tests\coreclr\windows.x64.Checked\tracing\eventpipe\eventsvalidation\GCEvents "%CORE_ROOT%\corerun.exe" GCEvents.dll echo Exit code: %ERRORLEVEL% popd ``` - Before the fix: the existing GC-events test passed, and the new regression failed with exit code **101**, reporting a GC polling helper at the top of a sample. Zero events were lost. - After the fix: both tests passed with exit code **100**, checking **9,430 sample stacks** with zero lost events. Also ran `GCEvents.dll` with `DOTNET_TieredCompilation=0`: both tests passed with exit code **100**, checking **9,943 sample stacks** with zero lost events. The regression checks raw sample stacks, retention of workload frames in managed samples, and retention of workload frames in ordinary event stacks. Other platforms and configurations have not been validated for this backport. # Risk Low. The change is narrowly scoped to identifying GC polling helpers and omitting them only while they are leading frames in an EventPipe stack. It does not change GC polling or suspension semantics, discard samples, or change sample classification. The worker is bound before managed execution, avoiding method loading during stack walks. Existing handling of other frames is preserved. Local testing confirmed the original failure and verified both the regression and existing GC-events coverage after the fix, with default tiering and tiering disabled. Resolves #135050 Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com>
steveisok
pushed a commit
that referenced
this pull request
Oct 25, 2026
> **TL;DR:** `SampleProfilerSampleType` intermittently throws `OverflowException` on linux-arm Checked. This PR changes the test's stack-address conversion to prevent that overflow. The fix still needs validation in linux-arm CI. It also needs backports to release/11.0 and release/10.0. `SampledStacksOmitGCPollHelpers`, added in #135219, fails intermittently on linux-arm Checked. It throws `OverflowException` from `RawEventPipeStack.Read`. In build 1647637, the same test passed on windows-x86, linux-x64 and linux-arm64. Fixes #136197 ## Cause TraceEvent 3.1.28 fills `DataPtr` for a stack attachment with `(ulong)(eventData.StackBytes - 8)`, where `StackBytes` is an `IntPtr`. That calls `IntPtr`'s explicit conversion to `long`, which sign-extends in a 32-bit process. An attachment at or above `0x80000000` reads back as `0xFFFFFFFF_xxxxxxxx`, and the test's `checked((nuint)address)` throws. Whether the buffer lands that high depends on the process's memory layout. That fits the same Helix image both passing and failing, though the failing runs don't log the pointer value. I fetched run details for 33 report rows in #136197; all 33 are linux-arm Checked. Two other rows returned no details, and one row was added after I collected the details. I also scanned the scheduled main outerloop and jitstress builds from October 18 to 21. The first with the overflow is 1640173, queued about 49 minutes after #135219 merged. The linux-arm tracing work items in the six builds before it show no `OverflowException`. ## Change Convert with `unchecked((nint)address)`, which keeps only the native-width bits. On 64-bit platforms the result is unchanged. ## Validation I couldn't run the test on linux-arm. A small program on x64 applies the 32-bit forms of both conversions to a sign-extended value, `0xFFFFFFFFB6A01000`. With the 32-bit forms, the old conversion throws `OverflowException` and the new one returns `0xB6A01000`. The program also converts `0x00007F0012345678` with the 64-bit forms, and both return `0x7F0012345678`. The linux-arm legs of this PR's CI are the real check, and since the failure is intermittent, one green run shows little. ## Backports #135219 was backported to release/11.0 (#136077) and release/10.0 (#136091), so this fix needs the same backports. cc @lateralusX > [!NOTE] > This change was prepared with AI assistance (Anthropic Claude, checked by OpenAI Codex) at my request. I reviewed the change and the evidence before opening.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make the non-inlined PollGCWorker helper binder-visible and bind it before managed execution, allowing stack walks to recognize polling helpers without loading methods during suspension.
Omit only leading PollGC and PollGCWorker frames in the shared EventPipe stack walker, preserving caller frames, samples, and classification.
Add raw-stack regression coverage using JIT and rundown method ranges, including caller retention and ordinary event stacks. Exclude interpreter configurations, which use a different safepoint path.
Fixes #135050