setup_host.sh has packages with unstable transitive dependencies

[RaghavendraSingh]

Context

as part of spark cluster launch on batch, Aztk runs setup_host.sh script and as part of it some python packages are installed

in particular, this line where pipenv is installed - https://github.com/Azure/aztk/blob/master/aztk/node_scripts/setup_host.sh#L153

# ensure these packages are  compatible before upgrading
python3 -m pip install pip=="18.0" pipenv=="2018.7.1"

This step has failed for us twice over the past 2 days

Issue

pipenv=="2018.7.1" has open bounded transitive dependencies and new version releases to these packages causes the pipenv install to be very unstable

for example: On 11th Feb we faced this error:

Collecting pipenv==2018.7.1
  Downloading https://files.pythonhosted.org/packages/eb/64/9b2747d54f2008ac3dfe86c0b1c8ec126042726fd8a540d5208d26732701/pipenv-2018.7.1-py3-none-any.whl (5.0MB)
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.5/dist-packages (from pipenv==2018.7.1) (42.0.2)
Requirement already satisfied: pip>=9.0.1 in /usr/local/lib/python3.5/dist-packages (from pipenv==2018.7.1) (18.0)
Collecting virtualenv-clone>=0.2.5 (from pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/ba/f8/50c2b7dbc99e05fce5e5b9d9a31f37c988c99acd4e8dedd720b7b8d4011d/virtualenv_clone-0.5.3-py2.py3-none-any.whl
Collecting virtualenv (from pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/62/89/72ab7d18e001a23b25eb75ce71bbf9e2a3c142e2dad2e4e2b88012e4ca25/virtualenv-20.0.0-py2.py3-none-any.whl (4.6MB)
Collecting certifi (from pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/b9/63/df50cac98ea0d5b006c55a399c3bf1db9da7b5a24de7890bc9cfd5dd9e99/certifi-2019.11.28-py2.py3-none-any.whl (156kB)
Collecting importlib-resources<2,>=1.0; python_version < "3.7" (from virtualenv->pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/2f/f7/b4aa02cdd3ee7ebba375969d77c00826aa15c5db84247d23c89522dccbfa/importlib_resources-1.0.2-py2.py3-none-any.whl
Collecting filelock<4,>=3.0.0 (from virtualenv->pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl
Requirement already satisfied: six<2,>=1.12.0 in /usr/local/lib/python3.5/dist-packages (from virtualenv->pipenv==2018.7.1) (1.13.0)
Collecting importlib-metadata<2,>=0.12; python_version < "3.8" (from virtualenv->pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/8b/03/a00d504808808912751e64ccf414be53c29cad620e3de2421135fcae3025/importlib_metadata-1.5.0-py2.py3-none-any.whl
Collecting appdirs<2,>=1.4.3 (from virtualenv->pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv->pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/21/21/ff0bfa221d5f5c5dc495977219c4de8f44ba8203f6724fb648113f9902cb/zipp-2.2.0-py2.py3-none-any.whl
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named 'setuptools' 

this happened because zipp had released a new version 2.2.0 on 8th Feb which was not compatible

We hot-fixed it by preinstalling zipp==1.1.0 in setup_hosts.sh

# ensure these packages are  compatibile before upgrading
    python3 -m pip install zipp=="1.1.0"
    python3 -m pip install pip=="18.0" pipenv=="2018.7.1"

But then again on 12th Feb a new version of virtualenv was released and we faced the same issue again - same issue caused by the release of a different package (virtualenv released a new version in Feb 11)

Collecting pip==18.0
  Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
Collecting pipenv==2018.7.1
  Downloading https://files.pythonhosted.org/packages/eb/64/9b2747d54f2008ac3dfe86c0b1c8ec126042726fd8a540d5208d26732701/pipenv-2018.7.1-py3-none-any.whl (5.0MB)
Collecting virtualenv-clone>=0.2.5 (from pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/ba/f8/50c2b7dbc99e05fce5e5b9d9a31f37c988c99acd4e8dedd720b7b8d4011d/virtualenv_clone-0.5.3-py2.py3-none-any.whl
Collecting certifi (from pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/b9/63/df50cac98ea0d5b006c55a399c3bf1db9da7b5a24de7890bc9cfd5dd9e99/certifi-2019.11.28-py2.py3-none-any.whl (156kB)
Collecting virtualenv (from pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/51/89/6f2827025b28eee6236b1905555b0fbe1749f27bcbd3732ee137c31a21a8/virtualenv-20.0.2-py2.py3-none-any.whl (4.6MB)
Collecting setuptools>=36.2.1 (from pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/3d/72/1c1498c1e908e0562b1e1cd30012580baa7d33b5b0ffdbeb5fde2462cc71/setuptools-45.2.0-py3-none-any.whl (584kB)
Collecting six<2,>=1.12.0 (from virtualenv->pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting importlib-resources<2,>=1.0; python_version < "3.7" (from virtualenv->pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/2f/f7/b4aa02cdd3ee7ebba375969d77c00826aa15c5db84247d23c89522dccbfa/importlib_resources-1.0.2-py2.py3-none-any.whl
Collecting importlib-metadata<2,>=0.12; python_version < "3.8" (from virtualenv->pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/8b/03/a00d504808808912751e64ccf414be53c29cad620e3de2421135fcae3025/importlib_metadata-1.5.0-py2.py3-none-any.whl
Collecting filelock<4,>=3.0.0 (from virtualenv->pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl
Collecting distlib<1,>=0.3.0 (from virtualenv->pipenv==2018.7.1)
  Downloading https://files.pythonhosted.org/packages/7d/29/694a3a4d7c0e1aef76092e9167fbe372e0f7da055f5dcf4e1313ec21d96a/distlib-0.3.0.zip (571kB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named 'setuptools'

we have again hot-fixed it in a similar way

# ensure these packages are  compatible before upgrading
    python3 -m pip install setuptools=="42.0.2"
    python3 -m pip install zipp=="1.1.0"
    python3 -m pip install virtualenv=="20.0.0"
    python3 -m pip install pip=="18.0" pipenv=="2018.7.1"

Solution
At the moment it looks like the only solution is to lock the transitive dependency version for all dependent packages as well

Please recommend if there is a better way of mitigating this issue

[jafreck]

You are correct, locking the transitive dependencies is the only (short-term) solution for this issue. The hotfix you put forward will be merged and released.

Since pipenv does not lock its dependency tree, it is probably best to move to a different dependency management system. pipenv, while officially recommended for application package management, does not appear to be well maintained any longer: pypa/pipenv#4058