[Release][Python] PyPI rejects our source distribution due to missing LICENSE.txt

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

When releasing Arrow 22.0.0 and uploading to PyPI, I have been able to release the wheels but PyPI is rejecting our source distribution due to the following error:

Uploading pyarrow-22.0.0.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB • 00:00 • 140.0 MB/s
WARNING  Error during upload. Retry with the --verbose option for more details.                                                              
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/                                                                     
         License-File LICENSE.txt does not exist in distribution file pyarrow-22.0.0.tar.gz at pyarrow-22.0.0/LICENSE.txt        

The LICENSE.txt file is indeed not there but I've checked pyarrow-21.0.0 and is also not there: https://pypi.org/project/pyarrow/21.0.0/#pyarrow-21.0.0.tar.gz

PyPI seems to have implemented some changes to include those checks.

Component(s)

Release, Python

[Schamschula]

This is blocking my update of apache-arrow and py-pyarrow for MacPorts.

[raulcd]

We sign wheels and the source distribution as part of our release process, see the pyarrow-22.0.0.tar.gz asset and pyarrow-22.0.0.tar.gz.asc, pyarrow-22.0.0.tar.gz.sha512 on the assets for the Release: https://github.com/apache/arrow/releases/tag/apache-arrow-22.0.0

I can quickly untar, add a LICENSE.txt, tar and upload to PyPI but this won't be the same asset we verified on the release.
@pitrou @AlenkaF @rok @kou

The R distribution has a similar release process where they apply patches to their release, what are your thoughts?

We definitely should fix this on main but creating a new patch release only for this, is a problem.

[raulcd]

This is blocking my update of apache-arrow and py-pyarrow for MacPorts.

If you really require the source distribution, can't you try installing from the GitHub release:

pip install https://github.com/apache/arrow/releases/download/apache-arrow-22.0.0/pyarrow-22.0.0.tar.gz

[AlenkaF]

Oh, this is unfortunate.
A new patch release only for this, I agree, sounds silly.

Is it possible that we start vote on ML to add a single file (.txt only) and that would make it fine to have it next to a verified release on PyPI?

[rok]

+1 to Alenka's proposal of voting to add the needed file or perhaps just notifying the ML.
As for adding the python/LICENSE.txt, do we also need python/NOTICE.txt? (see #47141)

[raulcd]

I wanted to try with https://test.pypi.org/project/pyarrow/ to validate PyPI is not performing further checks and adding the LICENSE.txt will be enough but I don't have access there, @wesm @xhochy @kszucs can any of you give my user (raulcd) access to the test.pypi pyarrow project?

[pitrou]

Let's simply remove it from https://github.com/apache/arrow/blob/main/python/setup.cfg#L18-L21

[pitrou]

For reference (I don't expect a positive answer there, but wanted to surface the issue):
https://discuss.python.org/t/expressing-project-vs-distribution-licenses-post-pep-639/90314/211

[raulcd]

Let's simply remove it from https://github.com/apache/arrow/blob/main/python/setup.cfg#L18-L21

Are you suggesting me to:

• Untar the https://github.com/apache/arrow/releases/download/apache-arrow-22.0.0/pyarrow-22.0.0.tar.gz
• Update the setup.cfg file to remove metadata section
• Generate the tar.gz again and upload

I am completely fine with that, are we all ok for me to do that?

[pitrou]

I'm certainly fine with it.

[raulcd]

This would be the new tar.gz generated after:

tar zxvf pyarrow-22.0.0.tar.gz
vim pyarrow-22.0.0/setup.cfg #Remove the metadata lines on setup.cfg
mv pyarrow-22.0.0.tar.gz pyarrow-22.0.0.tar.gz.old
tar -czvf pyarrow-22.0.0.tar.gz pyarrow-22.0.0

pyarrow-22.0.0.tar.gz

[rok]

This sounds good to me too. Can you post the diff of setup.cfg change here for the record?