Project

General

Profile

Actions

Bug #5766

closed

[SDK] setup.py install always gets latest arvados-python-sdk from PyPi, even when already installed

Added by Peter Amstutz about 7 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
SDKs
Target version:
Start date:
04/27/2015
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

run-tests.sh installs pip packages this way:

            cd "$WORKSPACE/$1" \
                && python setup.py sdist rotate --keep=1 --match .tar.gz \
                && pip install -q --upgrade dist/*.tar.gz

The problem is that --upgrade recursively upgrades all dependencies, including arvados-python-client which means that if the version of arvados-python-client in PyPi has a newer timestamp than the timestamp of the last commit on the current git branch, tests for nodemanager and fuse will run with the newest arvados-python-client in PyPi rather than the intended version from the current git branch.

--upgrade was added in commit ed9e8d8616e919f81703134a7d5cbac62f151877


Subtasks 1 (0 open1 closed)

Task #5786: Review 5766-pip-sdk-install in arvados-devResolvedPeter Amstutz04/27/2015

Actions
Actions #1

Updated by Peter Amstutz about 7 years ago

  • Description updated (diff)
  • Category set to SDKs
Actions #2

Updated by Peter Amstutz about 7 years ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz about 7 years ago

I'm not sure if the --upgrade flag provides any provides any upside here, if our normal process is to re-create the virtualenv and reinstall dependencies (which will fetch the latest packages by default) on each run of the tests.

Actions #4

Updated by Ward Vandewege about 7 years ago

  • Description updated (diff)
Actions #5

Updated by Tom Clegg about 7 years ago

I suspect if we don't use --upgrade we'll need to manually clear the virtualenv when not using --skip-install. This means re-downloading a lot of python dependencies every time, even when using --leave-temp, which would be sad.

Maybe "if pip freeze | egrep ^foo=; then pip uninstall -y foo; fi" instead of "pip install --upgrade"?

Actions #6

Updated by Peter Amstutz about 7 years ago

So removing --upgrade isn't the whole story. There's a pip bug, too:

http://stackoverflow.com/questions/8579879/installing-a-python-package-source-tarball-with-pip

When you run "pip" in the same source directory that you built the package, it sees an ".egg-info" directory and assumes it is already installed. As a result, the "arvados-python-client" package isn't actually installed into the virtualenv. What is happens instead is that when run-tests installs the arvados-node-manager or arvados_fuse packages, it pulls the latest arvados-python-client package from PyPi (NOT what we want) instead of satisfying it with the locally installed version as intended.

The workaround is to not be in the source directory when you install:

            cd "$WORKSPACE/$1" \
                && python setup.py sdist rotate --keep=1 --match .tar.gz \
                && cd "$WORKSPACE" \
                && pip install "$WORKSPACE/$1/dist"/*.tar.gz

tomclegg: If I'm understanding your suggestion, you're suggesting uninstalling everything except the arvados client package before installing arvados_fuse or arvados-node-manager? E.g.

if pip freeze | egrep ^arvados-python-client= ; then
  pip uninstall -y foo
fi

I don't think that will work, because it will delete dependencies needed by one client package but not the other (depending on the order that they run), and it will re-download everything, which I thought we were trying to avoid. How about this?

pip install --upgrade `pip freeze | grep -v arvados | cut -f1 -d=`
Actions #7

Updated by Peter Amstutz about 7 years ago

  • Status changed from New to In Progress
Actions #8

Updated by Peter Amstutz about 7 years ago

  • Target version changed from Bug Triage to 2015-04-29 sprint
Actions #9

Updated by Peter Amstutz about 7 years ago

  • Assigned To set to Peter Amstutz
Actions #10

Updated by Brett Smith about 7 years ago

Please merge 8b51595. Thanks.

Actions #11

Updated by Peter Amstutz about 7 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados-dev|commit:274c7abf3aee7dee8bfe648f485087a5a0ab59ed.

Actions #12

Updated by Tom Clegg about 7 years ago

Peter Amstutz wrote:

tomclegg: If I'm understanding your suggestion, you're suggesting uninstalling everything except the arvados client package before installing arvados_fuse or arvados-node-manager? E.g.

FTR, no, uninstalling dependencies causes the "would be sad" scenario I was referring to. (This step bumps "run arv-git-httpd tests in existing VENV" from 20s to 65s, for example.)

"foo" was meant to represent "the arvados package we're about to install".

I was just suggesting/guessing that, if "upgrade" doesn't DTRT, perhaps "uninstall and then install" does. I was assuming "uninstall -y arvados-fuse" would only delete arvados-fuse, not its dependencies -- and therefore a subsequent "install arvados-fuse" would install the specific file we're trying to convince pip to install without wasting time trying to upgrade/re-install dependencies that are already present. But it sounds like "cd $WORKSPACE" and repeating each package install with --no-deps --ignore-installed were able to address the issue a different way anyway.

Actions

Also available in: Atom PDF