mono ARM64 llvmfullaot builds are often timing out

[agocke]

Build Information

Build: https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_build/results?buildId=1908792
Build error leg: mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release
Pull request: #72993

Error Message

{
  "ErrorMessage": "ran longer than the maximum time of 200 minutes",
  "BuildRetry": false
}

Report

Build	Definition	Step Name	Console log
1911947	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1913265	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1913428	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1913487	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1921124	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920967	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920881	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920824	dotnet/runtime	coreclr Pri0 Runtime Tests Run Linux arm checked	Log
1920726	dotnet/runtime	coreclr Pri0 Runtime Tests Run Linux arm checked	Log
1920739	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920293	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920262	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1908792	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920159	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920049	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916561	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916483	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916465	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916473	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1915223	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1914847	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1915172	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1914932	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1914294	dotnet/runtime	coreclr Pri0 Runtime Tests Run Linux arm checked	Log
1915025	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1915087	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux x64 release	Log
1915136	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1915149	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1915217	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1915444	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1915309	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1915875	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1915907	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916266	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916331	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916290	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916394	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916454	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916538	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916590	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916718	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916856	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1916825	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1917321	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux x64 release	Log
1917349	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1917403	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1917500	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1917636	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1917730	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1917822	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1918735	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1919972	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920044	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920313	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920364	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920421	dotnet/runtime	coreclr Pri0 Runtime Tests Run Linux arm64 checked	Log
1920417	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920454	dotnet/runtime	coreclr Pri0 Runtime Tests Run Linux arm checked	Log
1920738	dotnet/runtime	mono monointerpreter Pri0 Runtime Tests Run Linux arm64 release	Log
1920773	dotnet/runtime	mono monointerpreter Pri0 Runtime Tests Run Linux arm64 release	Log
1920782	dotnet/runtime	coreclr Pri0 Runtime Tests Run Linux arm64 checked	Log
1920801	dotnet/runtime	coreclr Pri0 Runtime Tests Run Linux arm checked	Log
1920813	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920831	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920920	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1920993	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log
1921175	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux x64 release	Log
1921231	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
28	68	68

Build	Kind	Start Time
1911075	Rolling	2022-29-07
1915087	Rolling	2022-01-08
1916331	Rolling	2022-01-08
1920159	Rolling	2022-03-08

[agocke]

FYI @noahfalk

[noahfalk]

@agocke Yep, it just hit the rolling build from earlier today.

[agocke]

@markwilkie I just created this issue to track timeouts, but it looks like it only has one failure, but the runfo log shows a bunch more (https://runfo.azurewebsites.net/search/timelines/?q=started%3A%7E7+definition%3Aruntime+text%3A%22ran+longer+than+the+maximum+time+of+200+minutes%22). Is there some filtering that I'm doing wrong?

[steveisok]

/cc @SamMonoRT I think the only realistic option to fix this is to bump the timeout. Thoughts?

[SamMonoRT]

This is one of our slowest lanes, but equally important given the focus on ARM64 perf this release. I agree, we need to bump the timeout.