Bug #7322

[Deployment] python-virtualenv backport is missing its dependencies

Added by Brett Smith over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Brett Smith
Category:
Deployment
Target version:
Start date:
09/13/2015
Due date:
% Done:

0%

Estimated time:
Story points:
0.5

Description

The current arvados/jobs image can't run virtualenv:

crunch@5bcd6e461833:/$ virtualenv --system-site-packages --python=python2.7 /tmp/venv
Traceback (most recent call last):
  File "/usr/local/bin/virtualenv", line 5, in <module>
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

The python-virtualenv package in Debian declares dependencies on other packages like python-pkg-resources. Our backport is missing those dependencies.

Fix this one way or another. I'm surprised fpm isn't picking them up; it might be worth spending a little time to investigate why that is. But if it's not clear, or if it's a huge pain to fix, adding /backports/python-virtualenv/fpm-info.sh to our source would be acceptable.

We only backport the virtualenv package for debian7, debian8 and ubuntu1204. Looks like this backport was introduced in the very first checkin of the run-build-packages.sh script. Do we even need this backport?

Associated revisions

Revision 9819eac0 (diff)
Added by Brett Smith over 5 years ago

7322: Don't backport virtualenv for Arvados.

The Arvados code only needs virtualenv 1.7, which is provided with all
of our supported distros.

run-tests.sh requires 1.8.3 to use the --setuptools switch, but that
can be handled without publishing a backported package.

Refs #7322.

Revision 9819eac0 (diff)
Added by Brett Smith over 5 years ago

7322: Don't backport virtualenv for Arvados.

The Arvados code only needs virtualenv 1.7, which is provided with all
of our supported distros.

run-tests.sh requires 1.8.3 to use the --setuptools switch, but that
can be handled without publishing a backported package.

Refs #7322.

History

#1 Updated by Brett Smith over 5 years ago

  • Target version set to Arvados Future Sprints

#2 Updated by Brett Smith over 5 years ago

  • Story points set to 0.5

Believe this can be done in half a point. Please do investigate why fpm isn't picking up the dependencies, but if that starts taking a while, go ahead and use the fpm-info solution, which should be straightforward.

Separate question: why are we even backporting this? It's widely available, and AFAIK we don't have any dependency on a specific version of virtualenv.

#3 Updated by Ward Vandewege over 5 years ago

  • Description updated (diff)

#4 Updated by Ward Vandewege over 5 years ago

  • Description updated (diff)

#5 Updated by Brett Smith over 5 years ago

Answers to outstanding questions:

Brett Smith wrote:

Fix this one way or another. I'm surprised fpm isn't picking them up; it might be worth spending a little time to investigate why that is.

virtualenv's setup.py can work with either setuptools or distutils. It assumes you're running it on the same system you're building it, so when it builds with setuptools, it doesn't declare its own dependency on setuptools. The bug is virtualenv's.

We only backport the virtualenv package for debian7, debian8 and ubuntu1204. Looks like this backport was introduced in the very first checkin of the run-build-packages.sh script. Do we even need this backport?

The Arvados code itself needs virtualenv 1.7. crunch-job uses the --system-site-packages switch, which was added then. At least 1.7 is shipped with all of the distros we support, so a backport doesn't seem to be necessary.

run-tests.sh uses the --setuptools switch, which was added in 1.8.3, which is newer than what's in Debian 7 or Ubuntu 12.04. But this is more of a development dependency than a runtime dependency. We might want to install an fpm-built package on, e.g., our ci server, but I don't see much reason to publish it through our package repositories.

#6 Updated by Brett Smith over 5 years ago

  • Status changed from New to In Progress
  • Assigned To set to Brett Smith
  • Target version changed from Arvados Future Sprints to 2015-09-30 sprint

Brett Smith wrote:

run-tests.sh uses the --setuptools switch, which was added in 1.8.3, which is newer than what's in Debian 7 or Ubuntu 12.04. But this is more of a development dependency than a runtime dependency. We might want to install an fpm-built package on, e.g., our ci server, but I don't see much reason to publish it through our package repositories.

We're agreed that we're going to do this, so we need to remove the virtualenv packages from our repositories.

#7 Updated by Brett Smith over 5 years ago

Brett Smith wrote:

run-tests.sh uses the --setuptools switch, which was added in 1.8.3, which is newer than what's in Debian 7 or Ubuntu 12.04.

What I said about versions is true, but it looks like Debian patched this switch in earlier and then upstreamed it. Both distros have that --setuptools switch, so the versions of virtualenv they include are good enough for our purposes. Since that means virtualenv works for both running and developing Arvados out of the box, we can just pull the packages without, e.g., testing for this feature in run-tests, or configuring a specific version on our ci server.

#8 Updated by Brett Smith over 5 years ago

  • Status changed from In Progress to Resolved

We no longer build virtualenv packages, and I just pulled the old ones, so this is done.

Also available in: Atom PDF