Bug #18336
closedNew httplib2 version 0.20.2 gets pulled from PySDK and causes issues
Description
The httplib2
project released a new version On Nov 2nd (https://pypi.org/project/httplib2/#history) and it broke our testing pipeline on the install phase:
======= install env Requirement already satisfied: setuptools>=18.5 in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (40.8.0) Requirement already satisfied: pip>=7 in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (18.1) Requirement already satisfied: wheel in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (0.37.0) Requirement already satisfied: PyYAML in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (6.0) Requirement already satisfied: httplib2 in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (0.20.2) Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2; python_version > "3.0" in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (from httplib2) (3.0.4) ... Installed /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages/arvados_python_client-2.3.0.dev20211013202728-py3.7.egg Processing dependencies for arvados-python-client==2.3.0.dev20211013202728 error: pyparsing 3.0.4 is installed but pyparsing<3,>=2.0.2 is required by {'packaging'} Leaving behind temp dirs in /home/lucas/.cache/arvados-build Fatal: installing PyYAML and sdk/python failed (encountered in install_env at /home/lucas/arvados/build/run-tests.sh line 667)
This problem sometimes isn't visible, mainly when the temp directory already exists from previous runs.
The way I was able to work around it is by setting an upper limit version on sdk/python/setup.py
like this:
diff --git a/sdk/python/setup.py b/sdk/python/setup.py
index 8d637303b..f82d44ab6 100644
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -50,7 +50,7 @@ setup(name='arvados-python-client',
'future',
'google-api-python-client >=1.6.2, <2',
'google-auth<2',
- 'httplib2 >=0.9.2',
+ 'httplib2 >=0.9.2, <0.20.2',
'pycurl >=7.19.5.1',
'ruamel.yaml >=0.15.54, <0.17.11',
'setuptools',
But let's dig a little deeper to confirm this is a proper solution.
Updated by Lucas Di Pentima almost 3 years ago
- Assigned To set to Lucas Di Pentima
Updated by Lucas Di Pentima almost 3 years ago
- Status changed from New to In Progress
Updated by Lucas Di Pentima almost 3 years ago
Following the dependency tree, it seems that the issue is caused by the following:
- PySDK requires httplib2 > 0.9.8
- httplib2 0.20.2 requires pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2; python_version > "3.0" => installs pyparsing 3.0.4
- PySDK requires google-api-python-client >=1.6.2, <2
- google-api-python-client 1.12.8 requires google-api-core<2dev,>=1.21.0
- google-api-core 1.31.4 requires packaging>=14.3
- packaging-21.2 requires pyparsing<3,>=2.0.2 => ERROR
- google-api-core 1.31.4 requires packaging>=14.3
- google-api-python-client 1.12.8 requires google-api-core<2dev,>=1.21.0
We've already updated google-api-python-client
's requirement on #18002, and I'm not sure if we can further up the version limit without major changes.
Updated by Lucas Di Pentima almost 3 years ago
Updates at 9d74b115e - branch 18336-httplib2-pysdk-issues
Test run: developer-run-tests: #2803
- Pins
httplib2
upper version limit on PySDK
Updated by Lucas Di Pentima almost 3 years ago
- Status changed from In Progress to Resolved
Applied in changeset arvados-private:commit:arvados|16b8b12040790937b5efa8fbb21522d5f31475b0.