• Table of contents
• Hacking prerequisites
  • Host options
    • Starting on your workstation
    • Starting on a VM
    • Starting in a docker container
  • Install dev environment
  • Start Postgres
  • Setup groups
  • Run tests

Hacking prerequisites¶

The Arvados test suite can run in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied.

Host options¶

Starting on your workstation¶

If your workstation is a debian buster system -- and you don't mind installing a bunch of packages on your workstation, some of them without apt -- the easiest way to get running is to run tests on bare metal. Skip to "Dependencies".

Other linux distributions should work too with some modifications, but it's probably easier to use a VM.

Starting on a VM¶

Another option is to create a virtual machine using something like Xen or VirtualBox, and run debian buster on it. The instructions below assume you have just a few basic requirements:

• SSH server
• sudo (apt-get install sudo)
• A user account with sudo privileges

Starting in a docker container¶

Arvbox provides a preinstalled Docker-based dev environment. The following instructions are for creating a dev environment inside Docker from scratch.

This can get you started quickly, but (unlike the above options) you'll need to remember to use something like docker commit to save your state before shutting down your container.

See http://docker.io for more about installing docker. On debian it looks something like this.

sudo apt-get install docker-ce
sudo adduser $USER docker
# {log out & log back in}
groups
# {should include "docker"}

Start up a new container with debian 10 (buster), make a new user and log in as that user:

docker run -it --privileged debian:10 bash
apt-get update
apt-get -y install sudo
adduser me
adduser me sudo
sudo -u me -i

The "--privileged" is required in order for /dev/fuse to be accessible (without it, no tests that require FUSE will work).

Install dev environment¶

If you're on Debian 12+ and working on Arvados 2.x (current as of August 2023), see Build_an_Arvados_development_environment_with_OpenSSL_11.

Start with Debian 10+ and run the following commands as root.

Note that the last command here ("arvados-server install -type test") installs additional debian packages to your system, along with additional software in /var/lib/arvados/ (such as suitable versions of Ruby and Go) that do not interfere with system packages. It also creates a postgresql database user named "arvados" with an insecure password. Don't expose this postgresql server to the internet or to untrusted users!

apt update
apt install wget ca-certificates
wget https://apt.arvados.org/bullseye/pool/main/a/arvados-server/arvados-server_2.4.3-1_amd64.deb
dpkg -i arvados-server_2.4.3-1_amd64.deb
arvados-server install -type test

Alternatively, install Go ≥ 1.20 (see https://golang.org) and git, and run arvados-server from source.

wget -O- https://go.dev/dl/go1.22.1.linux-amd64.tar.gz | tar -C /usr/local -xzf -
ln -s /usr/local/go/bin/* /usr/local/bin/

apt install git build-essential libpam-dev

cd
git clone https://git.arvados.org/arvados.git
cd arvados
go mod download
go run ./cmd/arvados-server install -type test

Start Postgres¶

If you're running in a docker container you'll need to start Postgres manually:

sudo /etc/init.d/postgresql start

(If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.)

Setup groups¶

Make sure the fuse and docker groups exist (create them if necessary) and that the user who will run the tests is a member of them.

Run tests¶

time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados

During development, you'll probably want something more like this. It reuses the given temp directory, which avoids a lot of repetitive downloading of dependencies, and allows you to save time with --skip-install or --only-install sdk/ruby and so on.

mkdir -p ~/.cache/arvados-build
time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build