Bug #4922

[Crunch] google-api-python-client fails to install

Added by Bryan Cosca over 5 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Brett Smith
Category:
Crunch
Target version:
Start date:
01/07/2015
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

From qr1hi-8i9sb-wleo6rt176ublrc, using docker image bcosc/all-tools

1/7/2015 1:20:27 PM compute20 1 task-print 0 File "/tmp/crunch-job/src/crunch_scripts/bwa-index", line 3, in <module>
1/7/2015 1:20:27 PM compute20 1 task-print 0 import arvados
1/7/2015 1:20:27 PM compute20 1 task-print 0 File "/usr/local/lib/python2.7/dist-packages/arvados/__init__.py", line 21, in <module>
1/7/2015 1:20:27 PM compute20 1 task-print 0 from api import *
1/7/2015 1:20:27 PM compute20 1 task-print 0 File "/usr/local/lib/python2.7/dist-packages/arvados/api.py", line 8, in <module>
1/7/2015 1:20:27 PM compute20 1 task-print 0 import apiclient
1/7/2015 1:20:27 PM compute20 1 task-print 0 File "/usr/local/lib/python2.7/dist-packages/apiclient/__init__.py", line 9, in <module>
1/7/2015 1:20:27 PM compute20 1 task-print 0 'Previous version of google-api-python-client detected; due to a '
1/7/2015 1:20:27 PM compute20 1 task-print 0 RuntimeError: Previous version of google-api-python-client detected; due to a packaging issue, we cannot perform an in-place upgrade. To repair, remove and reinstall this package, along with oauth2client and uritemplate. One can do this with pip via
1/7/2015 1:20:27 PM compute20 1 task-print 0 pip install -I google-api-python-client
1/7/2015 1:20:27 PM compute20 1 task-dispatch 0 srun: error: compute20: task 0: Exited with exit code 1


Subtasks

Task #5012: Confirm fixClosedTim Pierce

Associated revisions

Revision 59963e1e
Added by Tim Pierce about 5 years ago

Merge branch '4922-google-api-client-force-install'

Fixes #4922.

History

#1 Updated by Tim Pierce over 5 years ago

  • Status changed from New to In Progress
  • Assigned To set to Tim Pierce
  • Target version set to 2015-01-28 Sprint
  • Story points set to 0.5

#2 Updated by Tom Clegg over 5 years ago

Do we just need to add "-I" here? source:sdk/cli/bin/crunch-job#L1849

shell_or_die("$venv_dir/bin/pip", "--quiet", "install", $python_src);

#3 Updated by Tom Clegg over 5 years ago

(This is probably fallout from changing the version format from worked-for-a-while 0.1.date.commit to PEP440-compliant 0.1.date+commit.)

#4 Updated by Tim Pierce over 5 years ago

Tom Clegg wrote:

(This is probably fallout from changing the version format from worked-for-a-while 0.1.date.commit to PEP440-compliant 0.1.date+commit.)

It's not the arvados-python-client version number that it's whinging about, it's google-api-python-client. (Or do you mean that because we're using a properly formatted version number now for Arvados, it's better able to handle client dependencies and therefore exposing cases like this that would not have otherwise seen daylight? I guess something like that is possible?)

Either way -- I can't think of any reason I wouldn't use pip install -I any time I encounter this by hand, so I can't see any reason not to make that the default.

#5 Updated by Tim Pierce about 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:59963e1ee432a51a3f7ae3d98068a541b2738879.

#6 Updated by Tim Pierce about 5 years ago

  • Subject changed from google-api-python-client fails to install to [Crunch] google-api-python-client fails to install
  • Category set to Crunch

#7 Updated by Tim Pierce about 5 years ago

  • Status changed from Resolved to Feedback

#8 Updated by Tim Pierce about 5 years ago

Moved to state 'feedback' so we can confirm the fix once deployed.

#9 Updated by Tom Clegg about 5 years ago

  • Assigned To changed from Tim Pierce to Brett Smith

#10 Updated by Brett Smith about 5 years ago

  • Status changed from Feedback to New
  • Target version changed from 2015-01-28 Sprint to 2015-02-18 sprint

qr1hi-8i9sb-o9719vud1ro9wzh successfully reproduced the issue. Unfortunately this is not fixed.

#11 Updated by Brett Smith about 5 years ago

  • Status changed from New to Resolved

The original job didn't specify an arvados_sdk_version, so changing crunch-job's install process for that would not change anything. My reproducing job did specify an arvados_sdk_version, but pip -I was still a noop because the Docker image doesn't include virtualenv.

The problem is simply that the Docker image has both the apiclient and googleapiclient modules installed. As long as that's true, this exception will be raised on import. There's nothing we can do about it on the Crunch level.

Bryan, I believe you've effectively worked around this by adding virtualenv to the image and using arvados_sdk_version, but next time you're in there, you might want to clean up by removing the old apiclient Python module (pip uninstall apiclient) in favor of googleapiclient. Let me know if you have any questions about it.

Also available in: Atom PDF