dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827
Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution
Pull request: #39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827
Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1265150	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52575
2886447	dotnet-sdk	Run FullFramework Tests	Log	#56922
1265178	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52632
1264700	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52575
1264171	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52585
1264022	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52648
1263944	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52585
1264059	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52597
1263963	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52575
1263817	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52648
1263065	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52638
2884896	dotnet-sdk	Run TestBuild Tests	Log	#56922
1262246	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52626
1261829	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52581
1261892	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52434
1261237	dotnet/sdk	Run TestBuild Tests	Log	#52606
1260395	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52585
1260133	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52513
1260015	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52227
1259983	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52225
1259728	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52513
2883499	dotnet-sdk	Run TestBuild Tests	Log	#56922
1258904	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52523
1258882	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52513
1258605	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52469
1258252	dotnet/sdk	Run FullFramework Tests	Log	#52549
1257843	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52513
1257723	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52469
1257598	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52469
1257532	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52234
1257494	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52512
1256869	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52512
1256707	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52522
1256396	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52512
1256296	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52469
1256082	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52434
1255585	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52325
1255464	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52227
2880454	dotnet-sdk	Run TestBuild Tests	Log	#56704
1255221	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52325
1255376	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52500
1254873	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52325
1254595	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52485
1254095	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52325
1253955	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52469
1253974	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51895
1253898	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52325
1253785	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52227
2879468	dotnet-sdk	Run FullFramework Tests	Log	#56704
1253350	dotnet/sdk	Run TestBuild Tests	Log	#52461
1253325	dotnet/sdk	Run TestBuild Tests	Log	#52460
1252459	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52325
2878664	dotnet-sdk	Run TestBuild Tests	Log	#56048
1252474	dotnet/sdk	Run TestBuild Tests	Log	#52443
2878600	dotnet-sdk	Run TestBuild Tests	Log	#56048
1251979	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52325
1251614	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52242
1251269	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52242
1250977	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52415
2877677	dotnet-sdk	Run FullFramework Tests	Log	#56048
1250899	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52325
1250898	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52242
1250978	dotnet/sdk	Run TestBuild Tests	Log	#52414
2877349	dotnet-sdk	Run TestBuild Tests	Log	#56048
1250678	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52242
1250544	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52225
1249761	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52242
1249558	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52325
1249553	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52242
1249602	dotnet/sdk	Run TestBuild Tests	Log	#52376
1248894	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52225
1247957	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52325
1247900	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52242
2875266	dotnet-sdk	Run FullFramework Tests	Log	#56048
1247901	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52325
1247845	dotnet/sdk	Run TestBuild Tests	Log	#52098
1247725	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52157
1247713	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52325
1247714	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52242
2875192	dotnet-sdk	Run TestBuild Tests	Log	#56048
1247732	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52372
1247723	dotnet/sdk	Run TestBuild Tests	Log	#52364
1247264	dotnet/sdk	Run TestBuild Tests	Log	#52099
1246634	dotnet/sdk	Run FullFramework Tests	Log	#52313
1246399	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52157
1246526	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52325
1246628	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52053
1246527	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52242
1246485	dotnet/sdk	Run TestBuild Tests	Log	#52313
1246323	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52325
1245831	dotnet/sdk	Run FullFramework Tests	Log	#52345
1245292	dotnet/sdk	Run FullFramework Tests	Log	#52289
1245282	dotnet/sdk	Run FullFramework Tests	Log	#52313
1244967	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52333
2873284	dotnet-sdk	Run TestBuild Tests	Log	#56048
1236411	dotnet/sdk	Run FullFramework Tests	Log	#52251
1244189	dotnet/sdk	Run TestBuild Tests	Log	#52291
1244128	dotnet/sdk	Run TestBuild Tests	Log	#52291
1243965	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52157
1243977	dotnet/sdk	Run TestBuild Tests	Log	#52291
Displaying 100 of 113 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	32	113

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: #41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR:
#39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.