Crash on shutdown in v8impl::(anonymous namespace)::RefBase::Finalize

[deepak1556]

Refs https://dev.azure.com/monacotools/a6d41577-0fa3-498e-af22-257312ff0545/_build/results?buildId=146077

Symbolicated stack:

Operating system: Mac OS X
                  11.6.1 20G224
CPU: amd64
     family 6 model 58 stepping 9
     3 CPUs

GPU: UNKNOWN

Crash reason:  EXC_BAD_ACCESS / EXC_I386_GPFLT
Crash address: 0x10851f6e9
Process uptime: 6 seconds

Thread 0 (crashed)
 0  Electron Framework!v8impl::(anonymous namespace)::RefBase::Finalize(bool) [js_native_api_v8.cc : 248 + 0xf]
    rax = 0x400007f94f44697f   rdx = 0x0000000000116c40
    rcx = 0x0000000000000000   rbx = 0x00007f94f66afee0
    rsi = 0x000007f94f66f55e   rdi = 0x00007f94f66f55e0
    rbp = 0x00007ffeee161870   rsp = 0x00007ffeee161858
     r8 = 0x000000000000f157    r9 = 0x0000000000000004
    r10 = 0x00007f94f6600000   r11 = 0x00007f94f6658440
    r12 = 0x00007f94f586d1f0   r13 = 0x00007f94f496f208
    r14 = 0x00007f94f586ca00   r15 = 0x00007ffeee161888
    rip = 0x000000010851f6e9
    Found by: given as instruction pointer in context
 1  Electron Framework!node::Environment::RunCleanup() [env.cc : 626 + 0x2]
    rbp = 0x00007ffeee161a80   rsp = 0x00007ffeee161880
    rip = 0x00000001085097d8
    Found by: previous frame's frame pointer
 2  Electron Framework!node::FreeEnvironment(node::Environment*) [environment.cc : 389 + 0x8]
    rbp = 0x00007ffeee161ac0   rsp = 0x00007ffeee161a90
    rip = 0x00000001084cf353
    Found by: previous frame's frame pointer
 3  Electron Framework!electron::NodeEnvironment::~NodeEnvironment() [javascript_environment.cc : 374 + 0x5]
    rbp = 0x00007ffeee161ae0   rsp = 0x00007ffeee161ad0
    rip = 0x000000010528b875
    Found by: previous frame's frame pointer
 4  Electron Framework!electron::ElectronBrowserMainParts::PostMainMessageLoopRun() [memory : 1335 + 0x8]
    rbp = 0x00007ffeee161b50   rsp = 0x00007ffeee161af0
    rip = 0x0000000105281fc9
    Found by: previous frame's frame pointer
 5  Electron Framework!content::BrowserMainLoop::ShutdownThreadsAndCleanUp() [browser_main_loop.cc : 1032 + 0x6]
    rbp = 0x00007ffeee161c00   rsp = 0x00007ffeee161b60
    rip = 0x00000001059840d2
    Found by: previous frame's frame pointer
 6  Electron Framework!content::BrowserMainRunnerImpl::Shutdown() [browser_main_runner_impl.cc : 179 + 0x5]
    rbp = 0x00007ffeee161c50   rsp = 0x00007ffeee161c10
    rip = 0x00000001059856fa
    Found by: previous frame's frame pointer
 7  Electron Framework!content::BrowserMain(content::MainFunctionParams const&) [browser_main.cc : 49 + 0x6]
    rbp = 0x00007ffeee161ca0   rsp = 0x00007ffeee161c60
    rip = 0x00000001059815f0
    Found by: previous frame's frame pointer
 8  Electron Framework!content::ContentMainRunnerImpl::RunBrowser(content::MainFunctionParams&, bool) [content_main_runner_impl.cc : 598 + 0x8]
    rbp = 0x00007ffeee161d80   rsp = 0x00007ffeee161cb0
    rip = 0x00000001058c32a7
    Found by: previous frame's frame pointer
 9  Electron Framework!content::ContentMainRunnerImpl::Run(bool) [content_main_runner_impl.cc : 956 + 0x10]
    rbp = 0x00007ffeee161e00   rsp = 0x00007ffeee161d90
    rip = 0x00000001058c2da8
    Found by: previous frame's frame pointer
10  Electron Framework!content::RunContentProcess(content::ContentMainParams const&, content::ContentMainRunner*) [content_main.cc : 376 + 0x9]
    rbp = 0x00007ffeee162020   rsp = 0x00007ffeee161e10
    rip = 0x00000001058c1f56
    Found by: previous frame's frame pointer
11  Electron Framework!content::ContentMain(content::ContentMainParams const&) [content_main.cc : 402 + 0x8]
    rbp = 0x00007ffeee162050   rsp = 0x00007ffeee162030
    rip = 0x00000001058c2042
    Found by: previous frame's frame pointer
12  Electron Framework!ElectronMain [electron_library_main.mm : 32 + 0x5]
    rbp = 0x00007ffeee1620f0   rsp = 0x00007ffeee162060
    rip = 0x00000001051bdb56
    Found by: previous frame's frame pointer
13  Electron!main [electron_main.cc : 362 + 0xa]
    rbp = 0x00007ffeee1621a0   rsp = 0x00007ffeee162100
    rip = 0x0000000101a9c7e6
    Found by: previous frame's frame pointer
14  libdyld.dylib!start + 0x1
    rbp = 0x00007ffeee1621b0   rsp = 0x00007ffeee1621b0
    rip = 0x00007fff20350f3d
    Found by: previous frame's frame pointer

/cc @alexdima

[deepak1556]

We basically need nodejs/node#37802 which is available from node >=14.16.0.

Question as to why we are suddenly hitting this issue, not sure from the stack but we are triggering a gc on main process shutdown. Wondering if this is related to process shutdown changes @bpasero recently made ? @bpasero is there any native module that now runs quite late in the process shutdown phase of the main process ?

[bpasero]

Wondering if this is related to process shutdown changes

Can you clarify which changes you mean? Is there any hint as to what exactly causes the crash on shutdown?

[deepak1556]

IIRC you mentioned that the backup service is now ensured to run last in the shutdown phase, is there a native module involved in this service ?

Is there any hint as to what exactly causes the crash on shutdown?

Yeah that is the problem, the current stack does not give me useful information on whether this is from a native module or the core itself. Basically, a gc is triggered on a weak reference handle (don't know which handle it is atm) once node environment shutdown is triggered.

[bpasero]

Yeah the backup service now runs after any other before shutdown handler that could veto the shutdown just to make sure that no unload handler would result in a file getting dirty and we loose that backup. That shouldn't change anything though for how the window is being closed because it was always:

• before shutdown
• will shutdown
• window close

And now it is

• before shutdown
• before final shutdown (backups)
• will shutdown
• window close

Besides, all the backup service does is file read/write.

[deepak1556]

Oh okay, in that case it is unrelated. I will keep this open if we can get better traces in other runs but will be closed eventually when Electron 16 PR is merged.

[bpasero]

But I do believe that this crash is relatively recent, I have been looking a lot at CI runs over the past days and never saw it until maybe 2-3days ago.