.Net Core 3.1 - Kestrel Heartbeat.OnHeartbeat error

[philieu]

This is related to an existing reported issue here. The old issue is related to .net core 2.2. I've now updated to the latest version of .net core (e.g. 3.1) and the issue is still there.

Describe the bug

We are seeing the following error from Kestrel heartbeat:
Heartbeat.OnHeartbeat
System.ArgumentNullException: Value cannot be null.
at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Abstractions.Internal.TransportConnection.TickHeartbeat()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.ConnectionManager.Walk(Action`1 callback)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.OnHeartbeat()

When the above error happened, the application crashed.
I was not able to get more information in the docs to understand what that error means.

To Reproduce

We are using the following:

• ASP.NET Core SDK 3.1 (we are using this docker image to build the application: mcr.microsoft.com/dotnet/core/sdk:3.1)
• AWS Linux 2 AMI
• SQL Server backend (not sure that matter)

Exceptions (if any)

When debugging using "lldb", I got the following trace:

thread dotnet/aspnetcore#65, name = '******', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
frame #0: 0x00007fa5538730ba libcoreclr.soObject::TryEnterObjMonitorSpinHelper() + 42 libcoreclr.soObject::TryEnterObjMonitorSpinHelper:
-> 0x7fa5538730ba <+42>: movl -0x4(%rbx), %eax
0x7fa5538730bd <+45>: testl $0x1800ffff, %eax ; imm = 0x1800FFFF
0x7fa5538730c2 <+50>: je 0x7fa5538730f4 ; <+100>
0x7fa5538730c4 <+52>: testl $0x8000000, %eax ; imm = 0x8000000

OS Thread Id: 0x14a7 (65)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libcoreclr.so!Object::TryEnterObjMonitorSpinHelper() + 0x2a
Child-SP RetAddr Caller, Callee
00007F9FFBFFE600 00007fa55386b45a libcoreclr.so!JIT_MonReliableEnter_Portable + 0x1a, calling libcoreclr.so!Object::TryEnterObjMonitorSpinHelper()
00007F9FFBFFE620 00007fa4e2af1d3a (MethodDesc 00007fa4deb6bd20 + 0x4a Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection.TickHeartbeat()), calling libcoreclr.so!JIT_MonReliableEnter_Portable
00007F9FFBFFE6A0 00007fa4e005f8b0 (MethodDesc 00007fa4deb0fad8 + 0xd0 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.ConnectionManager.Walk(System.Action1<Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection>)) 00007F9FFBFFE6B0 00007fa5538668c9 libcoreclr.so!JIT_NewS_MP_FastPortable(CORINFO_CLASS_STRUCT_*) + 0x19, calling libcoreclr.so!__tls_get_addr 00007F9FFBFFE700 00007fa4e005f784 (MethodDesc 00007fa4deb0fdd8 + 0x54 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.HeartbeatManager.OnHeartbeat(System.DateTimeOffset)), calling (MethodDesc 00007fa4deb0fad8 + 0 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.ConnectionManager.Walk(System.Action1<Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection>))
00007F9FFBFFE730 00007fa4e005e941 (MethodDesc 00007fa4deb0eee8 + 0xb1 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.OnHeartbeat())
00007F9FFBFFE7A0 00007fa4e00429bd (MethodDesc 00007fa4da603f30 + 0x8d System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007F9FFBFFE7F0 00007fa4e0042906 (MethodDesc 00007fa4dba127f0 + 0x116 System.Threading.TimerQueueTimer.CallCallback(Boolean)), calling (MethodDesc 00007fa4da603f30 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007F9FFBFFE830 00007fa4e00426d9 (MethodDesc 00007fa4dba127c0 + 0x79 System.Threading.TimerQueueTimer.Fire(Boolean)), calling (MethodDesc 00007fa4dba127f0 + 0 System.Threading.TimerQueueTimer.CallCallback(Boolean))
00007F9FFBFFE850 00007fa4e00413e8 (MethodDesc 00007fa4dba12c40 + 0xc8 System.Threading.TimerQueue.EnsureTimerFiresBy(UInt32)), calling libcoreclr.so!GetTickCount64
00007F9FFBFFE880 00007fa4e0041d23 (MethodDesc 00007fa4dba12c58 + 0x243 System.Threading.TimerQueue.FireNextTimers()), calling (MethodDesc 00007fa4dba127c0 + 0 System.Threading.TimerQueueTimer.Fire(Boolean))
00007F9FFBFFE8F0 00007fa5538f85ff libcoreclr.so!CallDescrWorkerInternal + 0x7c
00007F9FFBFFE910 00007fa553828445 libcoreclr.so!CallDescrWorkerWithHandler(CallDescrData*, int) + 0x75, calling libcoreclr.so!CallDescrWorkerInternal
00007F9FFBFFE950 00007fa553828f20 libcoreclr.so!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 0x4f0, calling libcoreclr.so!CallDescrWorkerWithHandler(CallDescrData*, int)
00007F9FFBFFE960 00007fa553b0611a libcoreclr.so!UnicodeToUTF8 + 0xca, calling libcoreclr.so!UTF8Encoding::GetBytes(char16_t*, int, unsigned char*, int)
00007F9FFBFFE9B0 00007fa553b0d2ff libcoreclr.so!EnvironGetenv + 0xbf, calling libcoreclr.so!CorUnix::InternalLeaveCriticalSection(CorUnix::CPalThread*, _CRITICAL_SECTION*)
00007F9FFBFFE9E0 00007fa553b0df2d libcoreclr.so!SetLastError + 0xd, calling libcoreclr.so!__errno_location
00007F9FFBFFEA40 00007fa553b0d47d libcoreclr.so!GetEnvironmentVariableW + 0x13d, calling libcoreclr.so!PAL_free
00007F9FFBFFEB20 00007fa553973f56 libcoreclr.so!AppDomainTimerCallback_Worker(void*) + 0x46, calling libcoreclr.so!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int)
00007F9FFBFFEC30 00007fa5537fa645 libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 0x155
00007F9FFBFFED50 00007fa5537facad libcoreclr.so!ManagedThreadBase::ThreadPool(void ()(void), void*) + 0x2d, calling libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)
00007F9FFBFFED80 00007fa553974065 libcoreclr.so!AppDomainTimerCallback(void*, unsigned char) + 0xe5, calling libcoreclr.so!ManagedThreadBase::ThreadPool(void ()(void), void*)
00007F9FFBFFEDB0 00007fa55381ce27 libcoreclr.so!ThreadpoolMgr::AsyncTimerCallbackCompletion(void*) + 0xb7
00007F9FFBFFEDC0 00007fa55385d8c1 libcoreclr.so!EEHeapFreeInProcessHeap(unsigned int, void*) + 0x31, calling libcoreclr.so!HeapFree
00007F9FFBFFEE70 00007fa553819999 libcoreclr.so!ThreadpoolMgr::WorkerThreadStart(void*) + 0x4d9
00007F9FFBFFEED0 00007fa553b2c73d libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) + 0x15d
00007F9FFBFFEEF0 00007fa5547ddfa3 libpthread.so.0!start_thread + 0xf3
00007F9FFBFFEFB0 00007fa5543e84cf libc.so.6!clone + 0x3f

I also have the process dump. We can figure out a mechanism to send it through if needed.

Thanks.

[davidfowl]

Looks like a memory corruption issue.

cc @jkotas @kouvel @janvorli

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[danmoseley]

Setting area VM, although there's no evidence where any corruption might be.

[mangod9]

Hey @philieu, are you able to share a dump for this issue? Also has you possibly tried on latest 3.1.6 release? There were some issues fixed as part of that.

[philieu]

Hi @mangod9, sure I'll upgrade to 3.1.6 and will report back.

In the meantime, what's the best way to send the dump over? I can upload to OneDrive and share it with your Microsoft account?

[philieu]

@danmosemsft I've sent you an email with the link to the dump since I can't find the email of @mangod9 .

Thanks.

[mangod9]

ok thanks @philieu . I will get the link for Dan. Just to confirm that you are seeing the crash with the exact same stack, and its repeatable?

[philieu]

@mangod9

We have an integration tests suit and the issue will appear from time to time, I would say 15% of runs. The same happened in stage and prod as well.

The stack frame I posted above is the common one, about 90% of the time. I also include below another stack I saw, although it is not very often.
I included both in the process dump I shared.

Cheers.

---

thread #72, name = '*******', stop reason = signal SIGSEGV: invalid address (fault address: 0x400800080)
frame #0: 0x00007ff8f3ca0271
-> 0x7ff8f3ca0271: cmpl %edi, (%rdi)
0x7ff8f3ca0273: callq 0x7ff8eb3b15b0
0x7ff8f3ca0278: testl %eax, %eax
0x7ff8f3ca027a: jne 0x7ff8f3ca0284

(lldb) dumpstack
OS Thread Id: 0x130b (72)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: (MethodDesc 00007ff8ef9cfac0 + 0x51 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.ConnectionManager.RemoveConnection(Int64))
Child-SP RetAddr Caller, Callee
00007FF405FF9FE0 00007ff8f3c314c3 (MethodDesc 00007ff8f29169e8 + 0x563 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection+d__32.MoveNext()), calling (MethodDesc 00
007ff8ef9cfac0 + 0 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.ConnectionManager.RemoveConnection(Int64))
00007FF405FFA090 00007ff8f250ad0d (MethodDesc 00007ff8eb4c3f30 + 0x8d System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007FF405FFA0E0 00007ff8f3c6abda (MethodDesc 00007ff8f2e7c878 + 0x4a System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1[[System.Threading.Tasks.VoidTaskResult, System.Private.Core
Lib],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection+d__32, Microsoft.AspNetCore.Server.Kestrel.Core]].MoveNext(System.Threading.Thread)), calling (MethodDesc
00007ff8eb4c3f30 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007FF405FFA0F0 00007ff8f32df398 (MethodDesc 00007ff8eb8e1db0 + 0x18 System.Threading.Tasks.TaskScheduler.get_InternalCurrent()), calling libcoreclr.so!JIT_GetSharedGCThreadStaticBase(DomainLocalModule*, unsi
gned int)
00007FF405FFA110 00007ff8f32e1656 (MethodDesc 00007ff8eb8e8410 + 0x106 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
00007FF405FFA150 00007ff8f32dd830 (MethodDesc 00007ff8eb4c0ba0 + 0xc0 System.Threading.Tasks.Task.RunContinuations(System.Object)), calling (MethodDesc 00007ff8eb8e8410 + 0 System.Threading.Tasks.AwaitTaskCont
inuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
00007FF405FFA160 00007ff8f24db39f (MethodDesc 00007ff8eb8ec520 + 0x7f System.Threading.SpinWait.SpinOnceCore(Int32)), calling libcoreclr.so!ThreadNative::Sleep(int)
00007FF405FFA1B0 00007ff9646f3e57 libcoreclr.so!COMInterlocked::ExchangeObject(void**, void*) + 0x17, calling libcoreclr.so!ErectWriteBarrier(Object**, Object*)
00007FF405FFA1F0 00007ff8f3b5e9cf (MethodDesc 00007ff8eb8ea7d0 + 0x8f System.Threading.Tasks.Task1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTask Result)), calling (MethodDesc 00007ff8eb4c0b88 + 0 System.Threading.Tasks.Task.FinishContinuations()) 00007FF405FFA220 00007ff8f3c32f2c (MethodDesc 00007ff8f2918a78 + 0x7bc Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection+<ProcessRequestsAsync>d__121[[System.__Canon, System.Private.CoreLib]].M
oveNext()), calling (MethodDesc 00007ff8eb8ea7d0 + 0 System.Threading.Tasks.Task1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTaskResult)) 00007FF405FFA330 00007ff8f250ad0d (MethodDesc 00007ff8eb4c3f30 + 0x8d System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)) 00007FF405FFA380 00007ff8f3c6b735 (MethodDesc 00007ff8f2e7c1d8 + 0x95 System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1[[System.Threading.Tasks.VoidTaskResult, System.Private.Core Lib],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection+<ProcessRequestsAsync>d__121[[System.__Canon, System.Private.CoreLib]], Microsoft.AspNetCore.Server.Kestrel.Core]].MoveNext(System.Thread
ing.Thread)), calling (MethodDesc 00007ff8eb4c3f30 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007FF405FFA3C0 00007ff8f32e1656 (MethodDesc 00007ff8eb8e8410 + 0x106 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
00007FF405FFA400 00007ff8f32dd830 (MethodDesc 00007ff8eb4c0ba0 + 0xc0 System.Threading.Tasks.Task.RunContinuations(System.Object)), calling (MethodDesc 00007ff8eb8e8410 + 0 System.Threading.Tasks.AwaitTaskCont
inuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
00007FF405FFA460 00007ff9646f3e57 libcoreclr.so!COMInterlocked::ExchangeObject(void**, void*) + 0x17, calling libcoreclr.so!ErectWriteBarrier(Object**, Object*)
00007FF405FFA4A0 00007ff8f3b5e9cf (MethodDesc 00007ff8eb8ea7d0 + 0x8f System.Threading.Tasks.Task1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTask Result)), calling (MethodDesc 00007ff8eb4c0b88 + 0 System.Threading.Tasks.Task.FinishContinuations()) 00007FF405FFA4D0 00007ff8f3c33593 (MethodDesc 00007ff8f2921af8 + 0x333 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequestsAsync>d__2131[[System.__Canon, System.Private.CoreLib
]].MoveNext()), calling (MethodDesc 00007ff8eb8ea7d0 + 0 System.Threading.Tasks.Task1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTaskResult)) 00007FF405FFA570 00007ff8f250ad0d (MethodDesc 00007ff8eb4c3f30 + 0x8d System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)) 00007FF405FFA5C0 00007ff8f3c69b15 (MethodDesc 00007ff8f2e7b8d8 + 0x95 System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1[[System.Threading.Tasks.VoidTaskResult, System.Private.Core Lib],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequestsAsync>d__2131[[System.__Canon, System.Private.CoreLib]], Microsoft.AspNetCore.Server.Kestrel.Core]].MoveNext(System.Th
reading.Thread)), calling (MethodDesc 00007ff8eb4c3f30 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007FF405FFA600 00007ff8f32e1656 (MethodDesc 00007ff8eb8e8410 + 0x106 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
00007FF405FFA640 00007ff8f32dd830 (MethodDesc 00007ff8eb4c0ba0 + 0xc0 System.Threading.Tasks.Task.RunContinuations(System.Object)), calling (MethodDesc 00007ff8eb8e8410 + 0 System.Threading.Tasks.AwaitTaskCont
inuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
00007FF405FFA670 00007ff8f3c30f3d (MethodDesc 00007ff8f28ed300 + 0x2d System.IO.Pipelines.Pipe+DefaultPipeReader.AdvanceTo(System.SequencePosition, System.SequencePosition)), calling (MethodDesc 00007ff8f28eb1
70 + 0 System.IO.Pipelines.Pipe.AdvanceReader(System.SequencePosition ByRef, System.SequencePosition ByRef))
00007FF405FFA6A0 00007ff9646f3e57 libcoreclr.so!COMInterlocked::ExchangeObject(void**, void*) + 0x17, calling libcoreclr.so!ErectWriteBarrier(Object**, Object*)
00007FF405FFA6E0 00007ff8f3b5e9cf (MethodDesc 00007ff8eb8ea7d0 + 0x8f System.Threading.Tasks.Task1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTask Result)), calling (MethodDesc 00007ff8eb4c0b88 + 0 System.Threading.Tasks.Task.FinishContinuations()) 00007FF405FFA710 00007ff8f3be9f2e (MethodDesc 00007ff8f2922c80 + 0x104e Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__2141[[System.__Canon, System.Private.CoreLib]].M
oveNext()), calling (MethodDesc 00007ff8eb8ea7d0 + 0 System.Threading.Tasks.Task1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTaskResult)) 00007FF405FFA890 00007ff8f32f51e1 (MethodDesc 00007ff8eb4c3f70 + 0x41 System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threadin g.ContextCallback, System.Object)) 00007FF405FFA8D0 00007ff8f3bf1453 (MethodDesc 00007ff8f2e7af98 + 0xd3 System.Runtime.CompilerServices.AsyncTaskMethodBuilder1+AsyncStateMachineBox1[[System.Threading.Tasks.VoidTaskResult, System.Private.Core Lib],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__2141[[System.__Canon, System.Private.CoreLib]], Microsoft.AspNetCore.Server.Kestrel.Core]].MoveNext(System.Threadi
ng.Thread)), calling (MethodDesc 00007ff8eb4c3f70 + 0 System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallbac
k, System.Object))
00007FF405FFA910 00007ff8f32dea59 (MethodDesc 00007ff8eb8e2fb8 + 0x1d9 System.Threading.ThreadPoolWorkQueue.Dispatch())
00007FF405FFA980 00007ff9647af5ff libcoreclr.so!CallDescrWorkerInternal + 0x7c
00007FF405FFA9A0 00007ff9646df445 libcoreclr.so!CallDescrWorkerWithHandler(CallDescrData*, int) + 0x75, calling libcoreclr.so!CallDescrWorkerInternal
00007FF405FFA9E0 00007ff9646dff20 libcoreclr.so!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 0x4f0, calling libcoreclr.so!CallDescrWorkerWithHandler(CallDescrData*, int)
00007FF405FFAA10 00007ff9649c7d08 libcoreclr.so!CorUnix::CSharedMemoryObject::ReleaseObjectDestructionLock(CorUnix::CPalThread*, bool) + 0x48, calling libcoreclr.so!CorUnix::InternalLeaveCriticalSection(CorUni
x::CPalThread*, _CRITICAL_SECTION*)
00007FF405FFAA40 00007ff9652acff6 libc.so.6!__clock_gettime + 0x26
00007FF405FFAA60 00007ff96569b207 libpthread.so.0!__pthread_cond_timedwait + 0x107, calling libpthread.so.0!__pthread_mutex_cond_lock
00007FF405FFAAC8 00007ff965698ee2 libpthread.so.0!__pthread_mutex_unlock_usercnt + 0x42, calling libpthread.so.0!__lll_unlock_wake
00007FF405FFAB00 00007ff96469de04 libcoreclr.so!MetaSig::GetReturnTypeNormalized(TypeHandle*) const + 0x54, calling libcoreclr.so!SigPointer::PeekElemTypeClosed(Module*, SigTypeContext const*) const
00007FF405FFABB0 00007ff964829c2a libcoreclr.so!QueueUserWorkItemManagedCallback(void*) + 0x4a, calling libcoreclr.so!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int)
00007FF405FFACD0 00007ff9646b1645 libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 0x155
00007FF405FFADF0 00007ff9646b1cad libcoreclr.so!ManagedThreadBase::ThreadPool(void ()(void), void*) + 0x2d, calling libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)
00007FF405FFAE20 00007ff96481557e libcoreclr.so!ManagedPerAppDomainTPCount::DispatchWorkItem(bool*, bool*) + 0x10e, calling libcoreclr.so!ManagedThreadBase::ThreadPool(void ()(void), void*)
00007FF405FFAE30 00007ff964814e2a libcoreclr.so!PerAppDomainTPCountList::GetAppDomainIndexForThreadpoolDispatch() + 0x7a
00007FF405FFAE70 00007ff9646d0999 libcoreclr.so!ThreadpoolMgr::WorkerThreadStart(void*) + 0x4d9
00007FF405FFAED0 00007ff9649e373d libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) + 0x15d
00007FF405FFAEF0 00007ff965694fa3 libpthread.so.0!start_thread + 0xf3
00007FF405FFAFB0 00007ff96529f4cf libc.so.6!clone + 0x3f

[philieu]

Just a quick update. We've upgraded the .net core version to 3.1.6. However, we are still seeing the issue from time to time.

[mangod9]

Were you able to capture a dump?

[philieu]

@mangod9, I've loaded the dump for 3.1.6 to the OneDrive folder that previously shared with @danmosemsft . The new dump file is the one ended with "1042.b885b4346a10.1597128998-net-core-3_1_6"

Please let me know if you need me to resend the share link.