Arvados

{{>toc}}

h1. Hacking prerequisites

The Arvados test suite can run in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied.

h2. Host options

h3. Starting on your workstation

If your workstation is a debian buster system -- and you don't mind installing a bunch of packages on your workstation, some of them without apt -- the easiest way to get running is to run tests on bare metal. Skip to "Dependencies".

Other linux distributions should work too with some modifications, but it's probably easier to use a VM.

h3. Starting on a VM

Another option is to create a virtual machine using something like Xen or VirtualBox, and run debian buster on it. The instructions below assume you have just a few basic requirements:

* SSH server

* sudo (@apt-get install sudo@)

* A user account with sudo privileges

h3. Starting in a docker container

_[[Arvbox]] provides a preinstalled Docker-based dev environment.  The following instructions are for creating a dev environment inside Docker from scratch._

This can get you started quickly, but (unlike the above options) you'll need to remember to use something like @docker commit@ to save your state before shutting down your container.

See http://docker.io for more about installing docker. On debian it looks something like this.

<pre>

sudo apt-get install docker-ce

sudo adduser $USER docker

# {log out & log back in}

groups

# {should include "docker"}

</pre>

Start up a new container with debian 10 (buster), make a new user and log in as that user:

<pre>

docker run -it --privileged debian:10 bash

apt-get update

apt-get -y install sudo

adduser me

adduser me sudo

sudo -u me -i

</pre>

The "--privileged" is required in order for /dev/fuse to be accessible (without it, no tests that require FUSE will work).

h2. Install dev environment

h3. With Ansible

This is a prototype still in development, but initial development was done in January 2025 and it can automate much of the process for you.

h4. Install Ansible

The simplest thing to do is @pip install@ Ansible inside a virtualenv:

<pre><code class="shell">sudo apt install python3-venv

python3 -m venv ~/arvados-ansible

~/arvados-ansible/bin/pip install "ansible~=8.7"

</code></pre>

If that works, you're done and you can go on to the next section. For background:

* The Arvados Ansible playbooks are tested with Ansible 8, which we use for its Python version compatibility. Older versions are known not to work. You're welcome to try newer versions if you want to for any reason, but they haven't been tested.

* There are "other installation options":https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html if you can't do this for some reason.

h4. Write an Arvados database configuration

Next, write an Arvados configuration file that defines how you want to set up the PostgreSQL database. Write a file @~/arvados-ansible/zzzzz.yml@ like this:

<pre><code class="yaml">Clusters:

  zzzzz:

    PostgreSQL:

      Connection:

        user: arvados

        password: GoodPasswordHere

        dbname: arvados_development

        host: localhost

        port: "5432"

</code></pre>

The cluster ID must be @zzzzz@. You can change the @user@, @password@, and @dbname@ settings freely. Our Ansible playbook will configure PostgreSQL so your settings here work.

The playbook will always install the @postgresql@ server package. If you already have this installed and have configured it to listen somewhere other than the default, you may update @host@ and @port@ to reflect that. Note that @port@ is a string containing digits.

h4. Write an Ansible inventory

An inventory file tells Ansible what host(s) to manage, how to connect to them, and what settings they use. Write an inventory file to @~/arvados-ansible/inventory.ini@ like this:

<pre><code class="ini">[arvados-test]

# This is the list of host(s) where we're installing the test environment.

# The line below installs on the same system running Ansible.

# If you want to manage remote hosts, you can write your own host list:

# <https://docs.ansible.com/ansible/latest/getting_started/get_started_inventory.html>

localhost ansible_connection=local

[arvados-test:vars]

# The path to the Arvados cluster configuration you wrote in the previous section.

arvados_config_file={{ lookup('env', 'HOME') }}/arvados-ansible/zzzzz.yml

# The primary user doing Arvados development and tests.

# This user will be added to the `docker` group.

# It defaults to the name of the user running `ansible-playbook`.

# If you want to configure a different user, set that here:

#arvados_dev_user=USERNAME

</code></pre>

h4. Run the playbook

The basic command to run the playbook is:

<pre><code class="sh">cd arvados/tools/ansible

~/arvados-ansible/bin/ansible-playbook -i ~/arvados-ansible/inventory.ini -K install-test-env.yml</code></pre>

The playbook will install symlinks for Go, Node, Singularity, and Yarn under @/usr/local/bin@. The actual tools are installed under @/opt@. If you need different versions of these tools for other work, you'll need to customize your @PATH@ environment variable so the Arvados versions are found first when you're doing Arvados work.

h3. Manually

Start with Debian 10+ and run the following commands as root.

Note that the last command here ("arvados-server install -type test") installs additional debian packages to your system, along with additional software in /var/lib/arvados/ (such as suitable versions of Ruby and Go) that do not interfere with system packages. It also creates a postgresql database user named "arvados" with an insecure password. Don't expose this postgresql server to the internet or to untrusted users!

<pre>

apt update

apt install wget ca-certificates

wget https://apt.arvados.org/bullseye/pool/main/a/arvados-server/arvados-server_2.4.3-1_amd64.deb

dpkg -i arvados-server_2.4.3-1_amd64.deb

arvados-server install -type test

</pre>

Alternatively, install Go ≥ 1.20 (see https://golang.org) and git, and run arvados-server from source.

<pre>

wget -O- https://go.dev/dl/go1.22.1.linux-amd64.tar.gz | tar -C /usr/local -xzf -

ln -s /usr/local/go/bin/* /usr/local/bin/

apt install git build-essential libpam-dev

cd

git clone https://git.arvados.org/arvados.git

cd arvados

go mod download

go run ./cmd/arvados-server install -type test

</pre>

h2. Start Postgres

_If you're running in a docker container_ you'll need to start Postgres manually:

<pre>

sudo /etc/init.d/postgresql start

</pre>

(If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.)

h2. Setup groups

Make sure the fuse and docker groups exist (create them if necessary) and that the user who will run the tests is a member of them.

h2. Run tests

<pre>

time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados

</pre>

During development, you'll probably want something more like this. It reuses the given temp directory, which avoids a lot of repetitive downloading of dependencies, and allows you to save time with @--skip-install@ or @--only-install sdk/ruby@ and so on.

<pre>

mkdir -p ~/.cache/arvados-build

time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build

</pre>