Project

General

Profile

Actions

Feature #6348

open

[Deployment] [Documentation] Minimize system-wide dependencies for compute node setup

Added by Tom Clegg over 9 years ago. Updated 8 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Deployment
Target version:
Story points:
1.0
Release:
Release relationship:
Auto

Description

The instructions at http://doc.arvados.org/install/install-compute-node.html install a bunch of packages from our apt repo. This causes dependency hell when using an older system (or a non-Debian distro) on a compute worker.

The current set of dependencies include:
  • Docker
  • Arvados Python SDK debian package (and its dependencies)
There are three reasons why the Arvados Python SDK is needed on the compute node.
  1. The ping script given on that install page uses the Python SDK.
  2. Crunch jobs invoke arv-mount on the host.
  3. Users can run uncontainerized jobs, and those will work only if the relevant SDKs (typically Python) are installed on the worker host.
For the ping script's purposes, the SDK can be installed like this:
  • sudo apt-get install python-virtualenv python-dev libcurl4-openssl-dev
    virtualenv /root/arvados-venv
    /root/arvados-venv/bin/pip install arvados-python-client
    

It should be possible to install arv-mount in a virtualenv rather than system-wide, using a similar recipe. However, some effort will probably be needed to make "srun arv-mount" (when invoked from the controller node) run arv-mount from the appropriate virtualenv.

Uncontainerized jobs can go away when #6096 is resolved -- enabling default_docker_image_for_jobs will be a realistic option.

Solving and documenting the details of the above three points, and offering an alternative install recipe with Docker1 as the only system-wide dependency that isn't already provided by distro vendors, should make it easy to use a much greater variety of linux distros/versions as worker hosts.

1 Docker has its own install docs, so it won't be necessary to use our debian package repo at all.


Related issues

Related to Arvados - Feature #3454: [API] Assign a system-default default docker image (instead of using the non-docker code path) if none is specified in runtime_constraints.ResolvedRadhika Chippada05/07/2015Actions
Blocked by Arvados - Idea #6096: [OPS] Implement a process to regularly deploy a Docker image for running jobs to Arvados clustersResolvedWard Vandewege05/22/2015Actions
Actions #1

Updated by Tom Clegg over 9 years ago

  • Description updated (diff)
Actions #2

Updated by Ward Vandewege over 3 years ago

  • Target version deleted (Arvados Future Sprints)
Actions #3

Updated by Peter Amstutz over 1 year ago

  • Release set to 60
Actions #4

Updated by Peter Amstutz 8 months ago

  • Target version set to Future
Actions

Also available in: Atom PDF