System.Collections.Test: Assert failure '!CREATE_CHECK_STRING(bSmallObjectHeapPtr || bLargeObjectHeapPtr)'

Hit in libraries-pgo. Example run (console log). cc @dotnet/jit-contrib

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["System.Collections.Test","!CREATE_CHECK_STRING(bSmallObjectHeapPtr || bLargeObjectHeapPtr)"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=954772
Error message validated: [!CREATE_CHECK_STRING(bSmallObjectHeapPtr || bLargeObjectHeapPtr)]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/18/2025 8:42:54 AM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Known issue validation

Build: 🔎
Result validation: ⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: 🔎" line.
Validation performed at: 7/29/2025 1:27:05 AM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Report

Build	Definition	Test	Pull Request
1108628	dotnet/runtime	System.Collections.Tests.WorkItemExecution	#116539

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	1

@JulieLeeMSFT

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

@AndyAyersMS

@AndyAyersMS, PTAL.

Seems like this happened before #111937, so likely something else. Unfortunately, the log is gone already. Will close and hope we get a repro later.

https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-112151-merge-4df3d17f7ce144de85/System.Text.RegularExpressions.Tests/1/console.e0976529.log?helixlogtype=result has a more recent repro and hasn't been purged yet, so will look at that.

This is a very generic GC failure message and the various repros above seem to be quite diverse. So could be more than one underlying issue.

Dump from example run -- linux arm64 is not very useful?

Can't get very far with either lldb or dotnet-dump.

Going to see if I can get a local repro.

Adding back "Known Build Error" label so that it gets matched by build analysis.

Hit again in System.Collections.Tests in #117939 on (azurelinux.3.amd64.open.rt:

  Discovering: System.Collections.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Collections.Tests (found 7102 of 9190 test cases)
  Starting:    System.Collections.Tests (parallel test collections = on [2 threads], stop on fail = off)
ASSERT FAILED
	Expression: !CREATE_CHECK_STRING(bSmallObjectHeapPtr || bLargeObjectHeapPtr)
	Location:   line 562 in /__w/1/s/src/coreclr/vm/object.cpp
	Function:   ValidateInner
	Process:    67303

Analysis of crash dump - runfo get-helix-payload -j a25026df-8744-4b99-b80d-1e5e9322b482 -w System.Collections.Tests -o c:\helix_payload:

Stack trace of the crash:

libcoreclr!PROCAbort+0x2d [/__w/1/s/src/coreclr/pal/src/thread/process.cpp @ 2569] 
libcoreclr!invoke_previous_action+0xb4 [/__w/1/s/src/coreclr/pal/src/exception/signal.cpp @ 447] 
libcoreclr!sigtrap_handler+0xfe [/__w/1/s/src/coreclr/pal/src/exception/signal.cpp @ 15732480] 
libc_so+0x3dc90
libcoreclr!DBG_DebugBreak+0x1 [/__w/1/s/src/coreclr/pal/src/arch/amd64/debugbreak.S @ 9] 
libcoreclr!DebugBreak+0x303 [/__w/1/s/src/coreclr/pal/src/debug/debug.cpp @ 414] 
libcoreclr!Object::ValidateInner+0x5ff [/__w/1/s/src/coreclr/vm/object.cpp @ 562] 
libcoreclr!OBJECTREF::OBJECTREF+0x31 [/crossrootfs/x64/usr/include/x86_64-linux-gnu/bits/stdint-intn.h @ 993] 
libcoreclr!VSD_ResolveWorker+0xb7 [/__w/1/s/src/coreclr/vm/virtualcallstub.cpp @ 1698] 
libcoreclr!ResolveWorkerAsmStub+0x71 [/__w/1/s/src/coreclr/vm/amd64/virtualcallstubamd64.S @ 41]

The managed method that calls VSD resolve is System.Linq.Enumerable.ContainsIterate[System.__Canon](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.__Canon, System.Collections.Generic.IEqualityComparer`1<System.__Canon>)

The problem is that "this" points into the method stack frame. Looks like stack allocation did not detect escaping object reference.

....
00007dcb`bf0c1e56 49bba8eb9fbacb7d0000 mov r11,7DCBBA9FEBA8h // System.SZGenericArrayEnumerator`1<System.String>
00007dcb`bf0c1e60 4c895db0        mov     qword ptr [rbp-50h],r11
00007dcb`bf0c1e64 c745b8ffffffff  mov     dword ptr [rbp-48h],0FFFFFFFFh
00007dcb`bf0c1e6b 4533db          xor     r11d,r11d
00007dcb`bf0c1e6e 44895dbc        mov     dword ptr [rbp-44h],r11d
00007dcb`bf0c1e72 48897dc0        mov     qword ptr [rbp-40h],rdi
00007dcb`bf0c1e76 488d7db0        lea     rdi,[rbp-50h]
00007dcb`bf0c1e7a 48897da8        mov     qword ptr [rbp-58h],rdi <- Stack allocated object
....
00007dcb`bf0c1edf 488b7da8        mov     rdi,qword ptr [rbp-58h] <- Calling VSD on stack allocated object
00007dcb`bf0c1ee3 49bb788bc2becb7d0000 mov r11,7DCBBEC28B78h
00007dcb`bf0c1eed 41ff13          call    qword ptr [r11] 
...

I have update the match to be specific to System.Collections.Test so that this does not get matched against every other GC hole.

Thanks Jan. I was able to get a repro after 38 runs of this. Looks like normally ContainsIterate does not reach the optimized tier.

Were you able to get the dac to load for the coredump above? I would like to know what collection type is being enumerated and verify what code version was running so I can try and create a simpler repro case.

Looks like it was Tier1 (not OSR) during LinkedList_Generic_Tests and the collection type was LinkedList<string>.

Consistent and fairly quick repro:

export DOTNET_TieredPGO_InstrumentOnlyHotCode=0
export DOTNET_TC_CallCountingDelayMs=0

on the xunit invocation, pass System.Collections.Tests.dll as an arg 16 times, and add

-maxthreads 2 -class System.Collections.Tests.LinkedList_Generic_Tests_string

This executes the test methods enough that we reliably hit the problem.

The bug is in conditional escape analysis. Say the collection type is only knowable via GDV, and the likely case is a LinkedList<T> for a ref class T. The LinkedList enumerator first checks if the Count == 0, if so it creates an empty enumerator, if not, a linked list enumerator. For the empty case it (at least for a time) looks like it might allocate a new array enumerator or use the "empty static" enumerator. So there are 4 possible enumerators (unknown, linked list, empty static array, array).

CEA is able to prove the linked list enumerator won't escape, but also thinks the array enumerator won't escape. It also (mistakenly) forces control for empty linked list to always take the array enumerator path (which is stack allocated).

When we clone for CEA we split off the linked list case and force the other 3 cases to make interface calls, but we've stack allocated the array case and so things fail then when we try and enumerate an empty list.

We need to realize that the array enumerator allocation happens on the "failed" side of a CEA and so block its stack allocation.

The code below will repro the issue on linux. On windows we are missing PGO data (possibly from R2R) and do not inline SZGenericArrayEnumeratorBase and so the array enumerator allocation escapes.

using System.Runtime.CompilerServices;

LinkedList<string> e = new LinkedList<string>();
LinkedList<string> e1 = new LinkedList<string>();
e1.AddLast("b");
e1.AddLast("a");

int sum = 0;

for (int i = 0; i < 200; i++)
{
    sum += Test(i % 10 == 0 ? e : e1) ? 1 : 0;
    Thread.Sleep(5);
}

Console.WriteLine(sum);

[MethodImpl(MethodImplOptions.NoInlining)]
bool Test(IEnumerable<string> e)
{
    return e.Contains("a", StringComparer.OrdinalIgnoreCase);
}

This pattern of checking for empty and special casing seems common in many collections, so it's a bit of a mystery still why this bug isn't more prominent. Perhaps the missing PGO is an explanation.

JIT: fix issue in conditional escape analysis

JIT: fix issue in conditional escape analysis (#118214)

Navigation Menu

Repository navigation

System.Collections.Test: Assert failure '!CREATE_CHECK_STRING(bSmallObjectHeapPtr || bLargeObjectHeapPtr)' #111922

Known Issue Error Message

Known issue validation

Report

Summary

Known issue validation

Report

Summary

Report

Summary

28 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

System.Collections.Test: Assert failure '!CREATE_CHECK_STRING(bSmallObjectHeapPtr || bLargeObjectHeapPtr)' #111922

Description

Known Issue Error Message

Known issue validation

Report

Summary

Known issue validation

Report

Summary

Report

Summary

Activity

dotnet-policy-service commented on Jan 28, 2025

JulieLeeMSFT commented on Jan 30, 2025

AndyAyersMS commented on Feb 13, 2025

AndyAyersMS commented on Feb 13, 2025

AndyAyersMS commented on Feb 19, 2025

AndyAyersMS commented on Feb 19, 2025

28 remaining items

jkotas commented on Jul 29, 2025

jkotas commented on Jul 29, 2025

jkotas commented on Jul 29, 2025

AndyAyersMS commented on Jul 29, 2025

AndyAyersMS commented on Jul 29, 2025

AndyAyersMS commented on Jul 29, 2025

AndyAyersMS commented on Jul 29, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions