Skip to content

Deadlock in GC #105780

Closed
Closed
Deadlock in GC#105780
Labels
area-GC-coreclr

Description

@jhudsoncedaron
jhudsoncedaron
opened on Jul 31, 2024

Description

Active worker thread is hung up with a diagnostic call stack that looks something like this:

ntdll.dll!syscall_gate
KERNELBASE.dll!WaitForMultipleObjects
???
CORINFO_HELP_NEWARR_1_VC
inlined!StringBuilder.ctor
Cedaron.Common.HL7.DLL!Cedaron.Common.HL7.HL7Segment.HL7Decode

All managed stacks as follows:

Main thread:
 	ntdll.dll!00007ff9df400d24()	Unknown
 	KERNELBASE.dll!00007ff9dc9d464c()	WaitForSingleObject
 	[Managed to Native Transition]	
 	System.Private.CoreLib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout) Line 156	C#
 	System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 561	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 3072	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 3007	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2768	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.Wait(int millisecondsTimeout) Line 2728	C#
 	[Waiting on Async Operation, double-click or press enter to view Async Call Stacks]	
	System.IO.FileSystem.Watcher.dll!System.IO.FileSystemWatcher.WaitForChanged(System.IO.WatcherChangeTypes changeType, int timeout) Line 631	C#
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Engine.Program.Main(string[] args) Line 300	C#

Background task scheduler thread:
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
 	Cedaron.Common.dll!Cedaron.Common.Portable.MultiWaiter.Win32WaitForAnyObject(nint[] waithandle, int tv_sec, int tv_msec)	Unknown
 	Cedaron.Common.dll!Cedaron.Common.Portable.MultiWaiter.WaitForAnyRead(System.Collections.Generic.IList<Cedaron.Common.Either<System.Net.Sockets.Socket, Cedaron.Common.Portable.IMultiWaitable>> waitfor, int tv_sec, int tv_msec)	Unknown
	Cedaron.Interface.Engine.dll!Cedaron.Interface.Engine.InterfaceEngine.BackgroundJobRunner()

Background keepalive thread:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
	System.Private.CoreLib.dll!System.Threading.Thread.Sleep(int millisecondsTimeout) Line 367	C#
 	Cedaron.Interface.FormBuilder.dll!Cedaron.Interface.FormBuilder.APIConfiguration.RefreshToken()	Unknown

.NET TP Gate:
 	ntdll.dll!00007ff9df400d24()	Unknown
 	KERNELBASE.dll!00007ff9dc9d464c()	WaitForSingleObject
 	[Managed to Native Transition]	
>	System.Private.CoreLib.dll!System.Threading.WaitHandle.WaitOneNoCheck(int millisecondsTimeout) Line 128	C#
 	System.Private.CoreLib.dll!System.Threading.PortableThreadPool.GateThread.GateThreadStart() Line 48	C#

.NET ThreadPool IO:
 	ntdll.dll!00007ff9df4030c4()	Unknown
 	KERNELBASE.dll!00007ff9dcab2724()	Unknown
 	[Managed to Native Transition]	
>	System.Private.CoreLib.dll!System.Threading.PortableThreadPool.IOCompletionPoller.Poll() Line 188	C#

.NET Timer:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
>	System.Private.CoreLib.dll!System.Threading.WaitHandle.WaitOneNoCheck(int millisecondsTimeout) Line 128	C#
 	System.Private.CoreLib.dll!System.Threading.TimerQueue.TimerThread() Line 87	C#

Connection Listener:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
 	Cedaron.Common.dll!Cedaron.Common.Portable.MultiWaiter.Win32WaitForAnyObject(nint[] waithandle, int tv_sec, int tv_msec)	Unknown
 	Cedaron.Common.dll!Cedaron.Common.Portable.MultiWaiter.WaitForAnyRead(System.Collections.Generic.IList<Cedaron.Common.Either<System.Net.Sockets.Socket, Cedaron.Common.Portable.IMultiWaitable>> waitfor, int tv_sec, int tv_msec)	Unknown
 	Cedaron.HL7.Client.dll!Cedaron.HL7.Client.SocketTransport.GetConnected(System.Action monitor, int monitorinterval)	Unknown
>	Cedaron.Interface.Engine.dll!Cedaron.Interface.Receivers.TCP.Cedaron.Interface.Declarations.IInterfaceReceiver.ReceiveLoop()
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Engine.Interface.ReceiveLoop()

.NET Finalizer:
>	ntdll.dll!00007ff9df400d24()	Unknown
 	KERNELBASE.dll!00007ff9dc9d464c()	WaitForSingleObject
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

Only Active Worker Thread:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
>	Cedaron.Common.HL7.dll!Cedaron.Common.HL7.HL7Segment.HL7Decode(string data, int offset, int length, char msh1, string msh2, bool translateunknownesc) Line 391	C#
 	Cedaron.Common.HL7.dll!Cedaron.Common.HL7.HL7Segment.HL7Segment(string msgdata, int offset, int length, char msh1, string msh2, bool translateunknownesc) Line 300	C#
 	Cedaron.Common.HL7.dll!Cedaron.Common.HL7.HL7Message.HL7Message(byte[] message, int offset, int length, bool translateunknownesc) Line 523	C#
 	Cedaron.Interface.FormBuilder.dll!Cedaron.Interface.FormBuilder.ADT.UpdateManualPromoteRecords()
 	Cedaron.Interface.FormBuilder.dll!Cedaron.Interface.FormBuilder.ADT.ProcessADT()
 	Cedaron.Interface.FormBuilder.dll!Cedaron.Interface.FormBuilder.ADT.Cedaron.Interface.Declarations.IMessageProcessor.ProcessMessage()
 	Cedaron.Interface.FormBuilder.dll!Cedaron.Interface.FormBuilder.DatabaseArchiver.Cedaron.Interface.Declarations.IMessageProcessor.ProcessMessage()
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Engine.Interface.ProcessMessage()
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Receivers.TCP.Cedaron.Interface.Declarations.IInterfaceReceiver.ReceiveLoop.AnonymousMethod__0(Cedaron.HL7.Client.HL7BlockingStream stream)
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Receivers.TCP.Cedaron.Interface.Declarations.IInterfaceReceiver.ReceiveLoop()
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Engine.Interface.ReceiveLoop()

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Debugger:
>	ntdll.dll!00007ff9df400d24()	Unknown
 	KERNELBASE.dll!00007ff9dc9d464c()	WaitForSingleObject
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Event Pipe:
 	[Waiting on a lock, load symbols for ntdll.dll to show thread lock information]	
>	ntdll.dll!00007ff9df403cc4()	Unknown
 	ntdll.dll!00007ff9df379f09()	Unknown
 	ntdll.dll!00007ff9df3774c2()	Unknown
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET TP Worker:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
>	System.Private.CoreLib.dll!System.Threading.LowLevelLifoSemaphore.WaitForSignal(int timeoutMs) Line 140	C#
 	System.Private.CoreLib.dll!System.Threading.LowLevelLifoSemaphore.Wait(int timeoutMs, bool spinWait) Line 65	C#
 	System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 100	C#

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET TP Worker:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
>	System.Private.CoreLib.dll!System.Threading.LowLevelLifoSemaphore.WaitForSignal(int timeoutMs) Line 140	C#
 	System.Private.CoreLib.dll!System.Threading.LowLevelLifoSemaphore.Wait(int timeoutMs, bool spinWait) Line 65	C#
 	System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 100	C#
 	[Native to Managed Transition]	
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

Reproduction Steps

No controlled reproduction available. We currently get the issue about three times per two weeks.

Expected behavior

new StringBuilder() does not hang forever.

Actual behavior

new StringBuilder() calls WaitForMultipleObjects, which does not return

Regression?

Yes; Issue is less than three months old.

Known Workarounds

No response

Configuration

.NET Runtime 8.0.7 via dotnet publish -r win-x64
OS: Windows Server Core; probably 2022

Other information

We have a full memory dump but can't release it publicly. This is a production instance and PHI is on the stack.

Activity

  1. ghost added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Jul 31, 2024
  2. dotnet-policy-service
    added
    untriagedNew issue has not been triaged by the area owner
    on Jul 31, 2024

  3. jkotas commented on Aug 1, 2024

    @jkotas
    jkotas
    on Aug 1, 2024
    Member

    Is this with regular CoreCLR with a JIT or with native AOT? RhpNewArray is a native AOT specific method, but the stacktraces you have shared look like regular CoreCLR with a JIT.

  4. jhudsoncedaron commented on Aug 1, 2024

    @jhudsoncedaron
    jhudsoncedaron
    on Aug 1, 2024
    Author

    @jkotas : I'm sorry I don't understand the difference. When I did a code search for CORINFO_HELP_NEWARR_1_VC ; it seems to be mapped to RhpNewArray by RyuJIT. The environment is whatever dotnet publish -r win-x64 spits out.

  5. jkotas
    added
    area-VM-coreclr
    and removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Aug 1, 2024

  6. jkotas commented on Aug 1, 2024

    @jkotas
    jkotas
    on Aug 1, 2024
    Member

    cc @VSadov

  7. dotnet-policy-service commented on Aug 1, 2024

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on Aug 1, 2024 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @mangod9
    See info in area-owners.md if you want to be subscribed.

  8. mangod9 commented on Aug 1, 2024

    @mangod9
    mangod9
    on Aug 1, 2024
    Member

    Hi @jhudsoncedaron, you mention that this started occurring recently, so do you observe that this is a regression in 8.0.7?

  9. jhudsoncedaron commented on Aug 1, 2024

    @jhudsoncedaron
    jhudsoncedaron
    on Aug 1, 2024
    Author

    @mangod9 : Barring exotic hypotheses, yes; however we jumped straight from 8.0.5 to 8.0.7 and are not sure of the exact start date of the problem. Support was slow to report the issue to engineering. In addition, we just turned on gcDynamic a couple of months ago.

    However exotic hypotheses are on the table; the data being processed as seen in the dump is trying to allocate about 5,000 StringBuilder objects in 50ms.

  10. jhudsoncedaron
    changed the title [-]Potential Deadlock in GC Allocate via RhpNewArray[/-] [+]Potential Deadlock in GC Allocate via CORINFO_HELP_NEWARR_1_VC[/+] on Aug 1, 2024

  11. mangod9 commented on Aug 1, 2024

    @mangod9
    mangod9
    on Aug 1, 2024
    Member

    Hey @jhudsoncedaron, are you able to privately share a dump to investigate further?

  12. jhudsoncedaron commented on Aug 1, 2024

    @jhudsoncedaron
    jhudsoncedaron
    on Aug 1, 2024
    Author

    @mangod9 : I will ask the guy capable of approving immediately. If that doesn't work, there's other ways of getting necessary information from a dump.

  13. jhudsoncedaron commented on Aug 1, 2024

    @jhudsoncedaron
    jhudsoncedaron
    on Aug 1, 2024
    Author

    @mangod9 : Dump upload is not happening. So it comes down to tell me what to look for and I can look for it.

  14. jhudsoncedaron commented on Aug 2, 2024

    @jhudsoncedaron
    jhudsoncedaron
    on Aug 2, 2024
    Author

    So I feel the need to explain a few things.

    We are not sitting idle on this. At once other things are happening.

    1. We are trying to switch that one instance to workstation GC to see if it clears it (this will hit production first).
    2. We are optimizing the allocation-hot loop (in fact the very one with the blocked thread) to allocate less stuff. This should make it happen less often.
    3. We looked at the wait handle code; and found a very unlikey pathway that could corrupt wait handle references. This requires two billion exceptions to be thrown from the same finally block where the only possible exception is OutOfMemoryException allocating a List enumerator. On triggering the fault it would overflow the reference counter inside SafeWaitHandle; but I don't see how it could cause this particular fault. (Fix: replace that particular foreach loop with a counted loop).
    4. We are trying to find a way to reproduce this in a dev environment. It's not looking promising.
    5. The dump contains Medical Records data. There's no way any production dump is getting uploaded anywhere until all other options are exhausted. I am reasonably certain a paid support case will happen first.

  15. 35 remaining items

  16. mangod9 commented on Apr 24, 2025

    @mangod9
    mangod9
    on Apr 24, 2025
    Member

    No the one which got reverted was this issue: #110589. Note that this one only affects Windows so shouldnt affect Linux. #110174 has not been reverted and should be included in latest servicing release. Have you tried it out?

  17. jhudsoncedaron commented on Apr 24, 2025

    @jhudsoncedaron
    jhudsoncedaron
    on Apr 24, 2025
    Author

    @mangod9 : We have not. We weren't going to push .NET 9 into production and onto servers outside of our reach resulting in DATAS on by default thinking the bug was still there so it crashes beyond our reach to fix immediately.

    So I will pass up the chain there were two GC fixes and the one that got reverted wasn't the one we were waiting for.

  18. jhudsoncedaron commented on Apr 24, 2025

    @jhudsoncedaron
    jhudsoncedaron
    on Apr 24, 2025
    Author

    So that was that; .NET 9 upgrade will most likely happen soon.

    The only reason I can come up with to leave this item open is if somebody else needs the Linux or Mac fixed GC library for .NET 8.

  19. cshung commented on Apr 24, 2025

    @cshung
    cshung
    on Apr 24, 2025
    Contributor

    The only reason I can come up with to leave this item open is if somebody else needs the Linux or Mac fixed GC library for .NET 8.

    Just in case this is needed, the repo has documentation on how to build here, we can always cherry-pick whatever change we need and make a build out of it.

    For the clrgc library in particular, it is built with compatibility in mind. You can install the latest .NET preview, copy and the library and apply it on earlier .NET version, and it should work. The only caveat is that this usage is not regularly tested.

  20. akoeplinger
    modified the milestones: 9.0.0, 9.0.x on May 16, 2025
  21. agocke
    added this to AppModelon Sep 23, 2025
  22. jkotas
    added
    backlog-cleanup-candidateAn inactive issue that has been marked for automated closure.
    on Nov 12, 2025

  23. dotnet-policy-service commented on Nov 12, 2025

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on Nov 12, 2025 – with Dotnet Policy Service
    Contributor

    Due to lack of recent activity, this issue has been marked as a candidate for backlog cleanup. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will undo this process.

    This process is part of our issue cleanup automation.

  24. dotnet-policy-service
    added
    no-recent-activity
    on Nov 12, 2025
  25. dotnet-policy-service
    removed this from the 9.0.x milestone on Nov 26, 2025
  26. dotnet-policy-service
    closed this as completedon Nov 26, 2025
  27. github-actions
    locked and limited conversation to collaborators on Dec 27, 2025
  28. dotnet-policy-service
    removed
    no-recent-activity
    backlog-cleanup-candidateAn inactive issue that has been marked for automated closure.
    on Dec 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-GC-coreclr

    Type

    No type

    Projects

    • AppModel
      Status
      No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions