[Python] macOS Segfault on Import, Both arm64 and x86_64

[essandess]

Describe the bug, including details regarding any error messages, version, and platform.

The pyarrow packages create a segfault on import when compiled on macOS boxes. This happens on macOS arm64 and x86_64, using (at least) version 12.0.0 and 13.0.0.

Context: I am trying to update the MacPorts apache-arrow ports at macports/macports-ports#19664 and discovered this issue. MacPorts uses the apache-arrow build instructions and approach in python.rst#build-and-test and python_wheel_macos_build.sh.

Behavior:

$ port installed py310-pyarrow
The following ports are currently installed:
  py310-pyarrow @12.0.0_3 (active)

$ PYTHONFAULTHANDLER=1 python3.10 -c 'import pyarrow'
Fatal Python error: Segmentation fault

Current thread 0x00007ff84d123700 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1176 in create_module
  File "<frozen importlib._bootstrap>", line 571 in module_from_spec
  File "<frozen importlib._bootstrap>", line 674 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow/__init__.py", line 65 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "<string>", line 1 in <module>
Segmentation fault: 11

Component(s)

Python

[kou]

Could you get backtrace by lldb?

[essandess]

Here's the macOS Crash Reporter output. What's the lldb command to capture the trace for python3.10 -c 'import pyarrow'?

Also, I've tried setting a few more cmake flags to be more consistent with the build script at python_wheel_macos_build.sh, but keep hitting the same issue.

Process:               Python [51251]
Path:                  /opt/local/Library/Frameworks/Python.framework/Versions/3.10/Resources/Python.app/Contents/MacOS/Python
Identifier:            org.python.python
Version:               3.10.12 (3.10.12)
Code Type:             X86-64 (Native)
Parent Process:        bash [15799]
Responsible:           Terminal [15797]
User ID:               502

Date/Time:             2023-08-03 16:08:12.4564 -0400
OS Version:            macOS 13.5 (22G74)
Report Version:        12
Bridge OS Version:     7.6 (20P6072)

Time Awake Since Boot: 810000 seconds
Time Since Wake:       607027 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000012340
Exception Codes:       0x0000000000000001, 0x0000000000012340

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [51251]

VM Region Info: 0x12340 is not in any region.  Bytes before following region: 4352785600
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      __TEXT                      103736000-10373a000    [   16K] r-x/r-x SM=COW  .../MacOS/Python

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libjemalloc.2.dylib           	       0x1059ec3df je_free_default + 193
1   libprotobuf.3.21.12.0.dylib   	       0x105688477 google::protobuf::internal::ArenaStringPtr::Destroy() + 39
2   libprotobuf.3.21.12.0.dylib   	       0x10571d5a5 google::protobuf::FileDescriptorProto::SharedDtor() + 149
3   libprotobuf.3.21.12.0.dylib   	       0x10571d6bd google::protobuf::FileDescriptorProto::~FileDescriptorProto() + 45
4   libprotobuf.3.21.12.0.dylib   	       0x105738657 google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) + 167
5   libprotobuf.3.21.12.0.dylib   	       0x1056e34f4 google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) + 36
6   libprotobuf.3.21.12.0.dylib   	       0x1057544d9 google::protobuf::(anonymous namespace)::AddDescriptors(google::protobuf::internal::DescriptorTable const*) + 105
7   dyld                          	    0x7ff81b4f93fb invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 175
8   dyld                          	    0x7ff81b537b7a invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 242
9   dyld                          	    0x7ff81b52bf22 invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 577
10  dyld                          	    0x7ff81b4dc0af dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 245
11  dyld                          	    0x7ff81b52b0bf dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 175
12  dyld                          	    0x7ff81b53773a dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 470
13  dyld                          	    0x7ff81b4f666c dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 220
14  dyld                          	    0x7ff81b4f685a dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 178
…

[essandess]

Here's the lldb trace:

lldb trace

lldb trace

/opt/local/bin/lldb-mp-16
(lldb) command script import pyarrow
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /opt/local/libexec/llvm-16/bin/lldb
1.	HandleCommand(command = "command script import pyarrow")
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  libLLVM.dylib               0x0000000113a91c6c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  libLLVM.dylib               0x0000000113a90c38 llvm::sys::RunSignalHandlers() + 112
2  libLLVM.dylib               0x0000000113a922f8 SignalHandler(int) + 344
3  libsystem_platform.dylib    0x000000018d386a24 _sigtramp + 56
4  libjemalloc.2.dylib         0x000000010f5940e0 je_free_default + 1104
5  libprotobuf.3.21.12.0.dylib 0x000000010f2471f0 google::protobuf::internal::ArenaStringPtr::Destroy() + 56
6  libprotobuf.3.21.12.0.dylib 0x000000010f2d0000 google::protobuf::FileDescriptorProto::SharedDtor() + 144
7  libprotobuf.3.21.12.0.dylib 0x000000010f2d00e4 google::protobuf::FileDescriptorProto::~FileDescriptorProto() + 48
8  libprotobuf.3.21.12.0.dylib 0x000000010f2ea634 google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) + 168
9  libprotobuf.3.21.12.0.dylib 0x000000010f29e878 google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) + 40
10 libprotobuf.3.21.12.0.dylib 0x000000010f3056a8 google::protobuf::(anonymous namespace)::AddDescriptors(google::protobuf::internal::DescriptorTable const*) + 136
11 libprotobuf.3.21.12.0.dylib 0x000000010f3056dc google::protobuf::internal::AddDescriptorsRunner::AddDescriptorsRunner(google::protobuf::internal::DescriptorTable const*) + 24
12 dyld                        0x000000018d01c1d8 invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 168
13 dyld                        0x000000018d05de94 invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 340
14 dyld                        0x000000018d0511a4 invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 528
15 dyld                        0x000000018cffc2d8 dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 296
16 dyld                        0x000000018d0501cc dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192
17 dyld                        0x000000018d05d958 dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 516
18 dyld                        0x000000018d01885c dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 448
19 dyld                        0x000000018d018c10 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 220
20 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
21 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
22 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
23 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
24 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
25 dyld                        0x000000018d01c264 dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_1::operator()() const + 112
26 dyld                        0x000000018d018d90 dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 304
27 dyld                        0x000000018d036d58 dyld4::APIs::dlopen_from(char const*, int, void*) + 1440
28 Python                      0x000000010378ae64 _imp_create_dynamic + 616
29 Python                      0x00000001036ce5d0 cfunction_vectorcall_FASTCALL + 80
30 Python                      0x0000000103760d94 _PyEval_EvalFrameDefault + 50396
31 Python                      0x0000000103763424 _PyEval_Vector + 116
32 Python                      0x000000010368859c object_vacall + 224
33 Python                      0x0000000103688458 PyObject_CallMethodObjArgs + 92
34 Python                      0x00000001037873a8 PyImport_ImportModuleLevelObject + 1224
35 Python                      0x000000010375b8f0 _PyEval_EvalFrameDefault + 28728
36 Python                      0x0000000103753d74 PyEval_EvalCode + 168
37 Python                      0x0000000103750174 builtin_exec + 332
38 Python                      0x00000001036ce6cc cfunction_vectorcall_FASTCALL_KEYWORDS + 76
39 Python                      0x0000000103760d94 _PyEval_EvalFrameDefault + 50396
40 Python                      0x0000000103763424 _PyEval_Vector + 116
41 Python                      0x000000010368859c object_vacall + 224
42 Python                      0x0000000103688458 PyObject_CallMethodObjArgs + 92
43 Python                      0x00000001037873a8 PyImport_ImportModuleLevelObject + 1224
44 Python                      0x000000010375b8f0 _PyEval_EvalFrameDefault + 28728
45 Python                      0x0000000103753d74 PyEval_EvalCode + 168
46 Python                      0x00000001037a5d38 run_eval_code_obj + 84
47 Python                      0x00000001037a5c9c run_mod + 112
48 Python                      0x00000001037a8084 PyRun_StringFlags + 112
49 liblldb.16.0.6.dylib        0x000000010469effc lldb_private::python::runStringMultiLine(llvm::Twine const&, lldb_private::python::PythonDictionary const&, lldb_private::python::PythonDictionary const&) + 120
50 liblldb.16.0.6.dylib        0x00000001046a5838 lldb_private::ScriptInterpreterPythonImpl::ExecuteMultipleLines(char const*, lldb_private::ExecuteScriptOptions const&) + 960
51 liblldb.16.0.6.dylib        0x00000001046ab314 lldb_private::ScriptInterpreterPythonImpl::LoadScriptingModule(char const*, lldb_private::LoadScriptOptions const&, lldb_private::Status&, std::__1::shared_ptr<lldb_private::StructuredData::Object>*, lldb_private::FileSpec) + 2092
52 liblldb.16.0.6.dylib        0x0000000104777158 CommandObjectCommandsScriptImport::DoExecute(lldb_private::Args&, lldb_private::CommandReturnObject&) + 328
53 liblldb.16.0.6.dylib        0x00000001043469c0 lldb_private::CommandObjectParsed::Execute(char const*, lldb_private::CommandReturnObject&) + 484
54 liblldb.16.0.6.dylib        0x000000010433d2e8 lldb_private::CommandInterpreter::HandleCommand(char const*, lldb_private::LazyBool, lldb_private::CommandReturnObject&) + 2136
55 liblldb.16.0.6.dylib        0x00000001043409a0 lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) + 852
56 liblldb.16.0.6.dylib        0x00000001042731a0 lldb_private::IOHandlerEditline::Run() + 304
57 liblldb.16.0.6.dylib        0x0000000104256968 lldb_private::Debugger::RunIOHandlers() + 140
58 liblldb.16.0.6.dylib        0x0000000104341c14 lldb_private::CommandInterpreter::RunCommandInterpreter(lldb_private::CommandInterpreterRunOptions&) + 156
59 liblldb.16.0.6.dylib        0x00000001040a0614 lldb::SBDebugger::RunCommandInterpreter(bool, bool) + 124
60 lldb                        0x0000000102ff48e0 Driver::MainLoop() + 2784
61 lldb                        0x0000000102ff541c main + 2168
62 dyld                        0x000000018cffff28 start + 2236
LLDB diagnostics will be written to /var/folders/2s/r9z6gsps7gn5wd89c0sqs8k80000gq/T/diagnostics-5f4976
Please include the directory content when filing a bug report
Segmentation fault: 11

[kou]

Thanks.
It seems that jemalloc is related.

Could you try disabling jemalloc when you build Apache Arrow C++?
You can disable jemalloc by specifying the -DARROW_JEMALLOC=OFF CMake option.

[essandess]

Thank you, that fixes this issue. I'll bump this upstream to jemalloc.

[kou]

In general, jemalloc is faster than system memory allocator.
So you may want to use jemalloc instead of disabling jemalloc.

Apache Arrow C++ uses bundled jemalloc instead of system jemalloc by default. It means that Apache Arrow C++ and Protobuf use different jemalloc in the same process. It may cause this problem. This problem may be solved by using system jemalloc by specifying -Djemalloc_SOURCE=SYSTEM instead of -DARROW_JEMALLOC=OFF.

[essandess]

Thanks. MacPorts’s apache-arrow already uses these system build flags, so this looks like an issue with jemalloc on macOS.

https://github.com/macports/macports-ports/blob/11afc500de4b609a3ed51465915c8102c8550aac/devel/apache-arrow/Portfile#L156

[essandess]

Several people have pointed out that jemalloc may not be the cause of this issue, so I am reopening the issue here. See:

• macOS Segfault with apache-arrow jemalloc/jemalloc#2511 (comment)
• https://trac.macports.org/ticket/67928#comment:3

[pitrou]

Several people have pointed out that jemalloc may not be the cause of this issue, so I am reopening the issue here.

Well, it depends. Does this issue still happen if you disable Arrow's jemalloc integration with -DARROW_JEMALLOC=OFF?