Bug #18336


New httplib2 version 0.20.2 gets pulled from PySDK and causes issues

Added by Lucas Di Pentima 8 months ago. Updated 7 months ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)
Story points:
Release relationship:


The httplib2 project released a new version On Nov 2nd ( and it broke our testing pipeline on the install phase:

======= install env
Requirement already satisfied: setuptools>=18.5 in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (40.8.0)
Requirement already satisfied: pip>=7 in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (18.1)
Requirement already satisfied: wheel in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (0.37.0)
Requirement already satisfied: PyYAML in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (6.0)
Requirement already satisfied: httplib2 in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (0.20.2)
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2; python_version > "3.0" in /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages (from httplib2) (3.0.4)
Installed /home/lucas/.cache/arvados-build/VENV3DIR/lib/python3.7/site-packages/arvados_python_client-2.3.0.dev20211013202728-py3.7.egg
Processing dependencies for arvados-python-client==2.3.0.dev20211013202728
error: pyparsing 3.0.4 is installed but pyparsing<3,>=2.0.2 is required by {'packaging'}
Leaving behind temp dirs in /home/lucas/.cache/arvados-build
Fatal: installing PyYAML and sdk/python failed (encountered in install_env at /home/lucas/arvados/build/ line 667)

This problem sometimes isn't visible, mainly when the temp directory already exists from previous runs.
The way I was able to work around it is by setting an upper limit version on sdk/python/ like this:

diff --git a/sdk/python/ b/sdk/python/
index 8d637303b..f82d44ab6 100644
--- a/sdk/python/
+++ b/sdk/python/
@@ -50,7 +50,7 @@ setup(name='arvados-python-client',
           'google-api-python-client >=1.6.2, <2',
-          'httplib2 >=0.9.2',
+          'httplib2 >=0.9.2, <0.20.2',
           'pycurl >=',
           'ruamel.yaml >=0.15.54, <0.17.11',

But let's dig a little deeper to confirm this is a proper solution.

Subtasks 1 (0 open1 closed)

Task #18353: Review 18336-httplib2-pysdk-issuesResolvedWard Vandewege11/17/2021

Actions #1

Updated by Lucas Di Pentima 8 months ago

  • Assigned To set to Lucas Di Pentima
Actions #2

Updated by Lucas Di Pentima 8 months ago

  • Status changed from New to In Progress
Actions #3

Updated by Lucas Di Pentima 8 months ago

Following the dependency tree, it seems that the issue is caused by the following:

  • PySDK requires httplib2 > 0.9.8
    • httplib2 0.20.2 requires pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2; python_version > "3.0" => installs pyparsing 3.0.4
  • PySDK requires google-api-python-client >=1.6.2, <2
    • google-api-python-client 1.12.8 requires google-api-core<2dev,>=1.21.0
      • google-api-core 1.31.4 requires packaging>=14.3
        • packaging-21.2 requires pyparsing<3,>=2.0.2 => ERROR

We've already updated google-api-python-client's requirement on #18002, and I'm not sure if we can further up the version limit without major changes.

Actions #4

Updated by Lucas Di Pentima 7 months ago

Updates at 9d74b115e - branch 18336-httplib2-pysdk-issues
Test run: developer-run-tests: #2803

  • Pins httplib2 upper version limit on PySDK
Actions #5

Updated by Lucas Di Pentima 7 months ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados-private:commit:arvados|16b8b12040790937b5efa8fbb21522d5f31475b0.


Also available in: Atom PDF