Development and testing with Docker

Arvados is primarily intended as a system to be run on large server clusters. For development and testing purposes, we have provided a process to build and deploy the Arvados system in Docker containers on a single machine.

http://doc.arvados.org/install/install-docker.html shows how to install and update a development system.

Additional sources of information

The docker/README.md file in the source distribution has more notes.

Installing the client libraries

./install_sdk.sh (in the docker directory) will install the CLI tools like arv and arv-put. If that does not work for you, try installing these by hand:

  • RVM -- if you do not already have RVM installed on your workstation, install it now.
    sudo apt-get install curl libcurl3 libcurl3-gnutls libcurl4-openssl-dev python-pip
    curl -sSL https://get.rvm.io | bash -s stable
    source ~/.rvm/scripts/rvm
    rvm install 2.1.0
    
  • The arvados-cli gem:
    gem install arvados-cli
    sudo pip install --upgrade httplib2
    

Debugging containers

Once it's running, you can get run a shell inside a container using nsenter.

Install nsenter:

docker run --rm jpetazzo/nsenter cat /nsenter > /tmp/nsenter
sudo install -m 0755 /tmp/nsenter /usr/local/bin/

Install this shortcut:

sudo tee /usr/local/bin/nse <<'EOF'
#!/bin/sh
PID=$(docker inspect --format {{.State.Pid}} "$*")
exec nsenter --target $PID --mount --uts --ipc --net --pid
EOF

chmod +x /usr/local/bin/nse

Then you can run a shell in a container like this:

nse keep_server_0

Console

./arvdock starts containers in detached mode. You can attach your terminal to them in order to see stdout/stderr:

docker attach --sig-proxy=false keep_server_0

Press control-C to detach. (If you didn't use --sig-proxy=false, control-C will send SIGINT into the container.)

Tutorials

Once you have installed the servers and started them with arvdock start, and installed the client libraries, you should be able to work through the user tutorials (using your host instead of a shell VM) to upload data into Arvados and run Crunch jobs in local development mode.