Bug #18336

New httplib2 version 0.20.2 gets pulled from PySDK and causes issues

Added by Lucas Di Pentima 26 days ago. Updated 13 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
SDKs
Target version:
Start date:
11/17/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

The httplib2 project released a new version On Nov 2nd (https://pypi.org/project/httplib2/#history) and it broke our testing pipeline on the install phase:

======= install env
Requirement already satisfied: setuptools>=18.5 in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (40.8.0)
Requirement already satisfied: pip>=7 in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (18.1)
Requirement already satisfied: wheel in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (0.37.0)
Requirement already satisfied: PyYAML in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (6.0)
Requirement already satisfied: httplib2 in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (0.20.2)
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2; python_version > "3.0" in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (from httplib2) (3.0.4)
...
Installed /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages/arvados_python_client-2.3.0.dev20211013202728-py3.7.egg
Processing dependencies for arvados-python-client==2.3.0.dev20211013202728
error: pyparsing 3.0.4 is installed but pyparsing<3,>=2.0.2 is required by {'packaging'}
Leaving behind temp dirs in /home/lucas/.cache/arvados-build
Fatal: installing PyYAML and sdk/python failed (encountered in install_env at /home/lucas/arvados/build/run-tests.sh line 667)

This problem sometimes isn't visible, mainly when the temp directory already exists from previous runs.
The way I was able to work around it is by setting an upper limit version on sdk/python/setup.py like this:

diff --git a/sdk/python/setup.py b/sdk/python/setup.py
index 8d637303b..f82d44ab6 100644
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -50,7 +50,7 @@ setup(name='arvados-python-client',
           'future',
           'google-api-python-client >=1.6.2, <2',
           'google-auth<2',
-          'httplib2 >=0.9.2',
+          'httplib2 >=0.9.2, <0.20.2',
           'pycurl >=7.19.5.1',
           'ruamel.yaml >=0.15.54, <0.17.11',
           'setuptools',

But let's dig a little deeper to confirm this is a proper solution.


Subtasks

Task #18353: Review 18336-httplib2-pysdk-issuesResolvedWard Vandewege

History

#1 Updated by Lucas Di Pentima 20 days ago

  • Assigned To set to Lucas Di Pentima

#2 Updated by Lucas Di Pentima 14 days ago

  • Status changed from New to In Progress

#3 Updated by Lucas Di Pentima 14 days ago

Following the dependency tree, it seems that the issue is caused by the following:

  • PySDK requires httplib2 > 0.9.8
    • httplib2 0.20.2 requires pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2; python_version > "3.0" => installs pyparsing 3.0.4
  • PySDK requires google-api-python-client >=1.6.2, <2
    • google-api-python-client 1.12.8 requires google-api-core<2dev,>=1.21.0
      • google-api-core 1.31.4 requires packaging>=14.3
        • packaging-21.2 requires pyparsing<3,>=2.0.2 => ERROR

We've already updated google-api-python-client's requirement on #18002, and I'm not sure if we can further up the version limit without major changes.

#4 Updated by Lucas Di Pentima 13 days ago

Updates at 9d74b115e - branch 18336-httplib2-pysdk-issues
Test run: https://ci.arvados.org/job/developer-run-tests/2803/

  • Pins httplib2 upper version limit on PySDK

#5 Updated by Lucas Di Pentima 13 days ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados-private:commit:arvados|16b8b12040790937b5efa8fbb21522d5f31475b0.

Also available in: Atom PDF