Shared library paths

[dazza-codes]

If wheels were to use shared library paths, is this project a useful point of consolidation to enable this, with sem-ver support for shared libs and conventions for building/distributing sem-ver shared libs?

In OSGeo/gdal#3060 I raised the problem and question of whether several projects that use a common dependency might all use a shared library in their binary wheel builds. In OSGeo/gdal#3060 (comment) I provided a nasty post-install hack to demonstrate and test the concept. Without repeating a lot of the details from OSGeo/gdal#3060 here, please check it out, including the links to related projects noted in the last comment. Any related pointers or commentary on python practices for shared libs would be great.

A shared-libs path should be able to support installations of many packages for any specific library. Ideally, it might prune a shared-libs path to provide only the latest patch (in major.minor.patch) version of any given library. If libraries were able to truly provide sem-ver releases, it might be possible to prune at the major.minor release level, but that's likely aiming too high. The option to prune shared libs might be enabled by default at the patch level, but could be disabled entirely for debugging.

[dazza-codes]

It may be that any attempt to use a shared-libs python path requires changes in auditwheel or related wheel building, e.g.

• https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh
• Proposal for packaging native libraries as wheels pypa/wheel-builders#2

If so, that's beyond the scope of this project, AFAICT. It seems that this project has a higher level of concern for OS distros than the concerns for building wheels or packaging shared libs into wheels. If the multiwheel builds that are called or used in this multibuild project could patch auditwheel, this project might define one or more conventions for a shared library path (assuming it might use or create new conventions for LD_LIBRARY_PATH).

Any kind of post-install hacks like https://github.com/dazza-codes/aws-lambda-layer-packing/blob/master/layer_create_zip.sh#L108 are risky and might only work for a specific OS target where the LD_LIBRARY_PATH is or can be identified and/or modified and tested. (Not to mention dep-tree resolution issues, which kinda rely on pip-extensions for OS packaging solutions.)

[matthew-brett]

Sorry - was slammed by pays-my-bills work. Yes - this project is for automating wheel building, but I think you're talking about the more general question of building wheels for libraries. That problem does need a solution, but as you see, we've been punting on that, generally, for macOS and Linux, and shipping everything inside the wheels. There has been some work on this, by @njsmith - I think the still-latest word on that is at pypa/wheel-builders#2 - Nathaniel - did I miss / forget something?

[mattip]

Work has been done on this to share openblas wheels between scipy and numpy. However, it needs a tweak due to interactions with sdist distributions. See the conversation starting with this comment and pep 0725. Closing as out-of-scope for this repo.