Arvados

I'm not sure if the --upgrade flag provides any provides any upside here, if our normal process is to re-create the virtualenv and reinstall dependencies (which will fetch the latest packages by default) on each run of the tests.

I suspect if we don't use --upgrade we'll need to manually clear the virtualenv when not using --skip-install. This means re-downloading a lot of python dependencies every time, even when using --leave-temp, which would be sad.

Maybe "if pip freeze | egrep ^foo=; then pip uninstall -y foo; fi" instead of "pip install --upgrade"?

So removing --upgrade isn't the whole story. There's a pip bug, too:

http://stackoverflow.com/questions/8579879/installing-a-python-package-source-tarball-with-pip

When you run "pip" in the same source directory that you built the package, it sees an ".egg-info" directory and assumes it is already installed. As a result, the "arvados-python-client" package isn't actually installed into the virtualenv. What is happens instead is that when run-tests installs the arvados-node-manager or arvados_fuse packages, it pulls the latest arvados-python-client package from PyPi (NOT what we want) instead of satisfying it with the locally installed version as intended.

The workaround is to not be in the source directory when you install:

            cd "$WORKSPACE/$1" \
                && python setup.py sdist rotate --keep=1 --match .tar.gz \
                && cd "$WORKSPACE" \
                && pip install "$WORKSPACE/$1/dist"/*.tar.gz

tomclegg: If I'm understanding your suggestion, you're suggesting uninstalling everything except the arvados client package before installing arvados_fuse or arvados-node-manager? E.g.

if pip freeze | egrep ^arvados-python-client= ; then
  pip uninstall -y foo
fi

I don't think that will work, because it will delete dependencies needed by one client package but not the other (depending on the order that they run), and it will re-download everything, which I thought we were trying to avoid. How about this?

pip install --upgrade `pip freeze | grep -v arvados | cut -f1 -d=`

Please merge 8b51595. Thanks.

Peter Amstutz wrote:

tomclegg: If I'm understanding your suggestion, you're suggesting uninstalling everything except the arvados client package before installing arvados_fuse or arvados-node-manager? E.g.

FTR, no, uninstalling dependencies causes the "would be sad" scenario I was referring to. (This step bumps "run arv-git-httpd tests in existing VENV" from 20s to 65s, for example.)

"foo" was meant to represent "the arvados package we're about to install".

I was just suggesting/guessing that, if "upgrade" doesn't DTRT, perhaps "uninstall and then install" does. I was assuming "uninstall -y arvados-fuse" would only delete arvados-fuse, not its dependencies -- and therefore a subsequent "install arvados-fuse" would install the specific file we're trying to convince pip to install without wasting time trying to upgrade/re-install dependencies that are already present. But it sounds like "cd $WORKSPACE" and repeating each package install with --no-deps --ignore-installed were able to address the issue a different way anyway.