Arvados

h1. Running tests

{{toc}}

The arvados git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the source tree. Jenkins at https://ci.arvados.org uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC.

h2. Background

You have the arvados source tree at @~/arvados@ and you might have local modifications.

h2. Install prerequisites

Follow instructions at [[Hacking prerequisites]]: "Install dev environment", etc. Don't miss creating the Postgres database, docker groups, etc.

h2. Environment

Your locale must use utf-8. Set environment variable LANG=C.UTF-8 if necessary to ensure the "locale" command reports UTF-8.

h2. If you're Jenkins

Run all tests from a clean slate (slow, but more immune to leaks)

<pre>

~/arvados/build/run-tests.sh WORKSPACE=~/arvados

</pre>

h2. If you're a developer

Cache everything needed to run test suites:

<pre>

mkdir ~/.cache/arvados-build

~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install

</pre>

Start interactive mode:

<pre>

$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --interactive

</pre>

Start interactive mode and enabled debug output:

<pre>

$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados ARVADOS_DEBUG=1 --temp ~/.cache/arvados-build --interactive

</pre>

When prompted, choose a test suite to run:

<pre>

== Interactive commands:

test TARGET

test TARGET:py3        (test with python3)

test TARGET -check.vv  (pass arguments to test)

install TARGET

install env            (go/python libs)

install deps           (go/python libs + arvados components needed for integration tests)

reset                  (...services used by integration tests)

exit

== Test targets:

cmd/arvados-client              lib/dispatchcloud/container     sdk/go/auth                     sdk/pam:py3                     services/fuse                   tools/crunchstat-summary

cmd/arvados-server              lib/dispatchcloud/scheduler     sdk/go/blockdigest              sdk/python                      services/fuse:py3               tools/crunchstat-summary:py3

lib/cli                         lib/dispatchcloud/ssh_executor  sdk/go/crunchrunner             sdk/python:py3                  services/health                 tools/keep-block-check

lib/cloud                       lib/dispatchcloud/worker        sdk/go/dispatch                 services/arv-git-httpd          services/keep-balance           tools/keep-exercise

lib/cloud/azure                 lib/service                     sdk/go/health                   services/crunch-dispatch-local  services/keepproxy              tools/keep-rsync

lib/cloud/ec2                   sdk/cwl                         sdk/go/httpserver               services/crunch-dispatch-slurm  services/keepstore              tools/sync-groups

lib/cmd                         sdk/cwl:py3                     sdk/go/keepclient               services/crunch-run             services/keep-web

lib/controller                  sdk/go/arvados                  sdk/go/manifest                 services/crunchstat             services/nodemanager

lib/crunchstat                  sdk/go/arvadosclient            sdk/go/stats                    services/dockercleaner          services/nodemanager:py3

lib/dispatchcloud               sdk/go/asyncbuf                 sdk/pam                         services/dockercleaner:py3      services/ws

What next? 

</pre>

Example: testing lib/dispatchcloud/container, showing verbose/debug logs:

<pre>

What next? test lib/dispatchcloud/container/ -check.vv

======= test lib/dispatchcloud/container

START: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType

WARN[0000] cancel container with no suitable instance type  ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"

WARN[0000] cancel container with no suitable instance type  ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"

START: queue_test.go:37: IntegrationSuite.TearDownTest

PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.846s

PASS: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType     0.223s

START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel

INFO[0001] adding container to queue                     ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued

START: queue_test.go:37: IntegrationSuite.TearDownTest

PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.901s

PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel        0.177s

OK: 2 passed

PASS

ok      git.arvados.org/arvados.git/lib/dispatchcloud/container       2.150s

======= test lib/dispatchcloud/container -- 3s

Pass: lib/dispatchcloud/container tests (3s)

All test suites passed.

</pre>

h3. Running individual test cases

Most Go packages use gocheck. Use gocheck command line args like -check.f.

<pre>

What next? test lib/dispatchcloud/container -check.vv -check.f=LockUnlock

======= test lib/dispatchcloud/container

START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel

INFO[0000] adding container to queue                     ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued

START: queue_test.go:37: IntegrationSuite.TearDownTest

PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.812s

PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel        0.184s

OK: 1 passed

PASS

ok      git.arvados.org/arvados.git/lib/dispatchcloud/container       1.000s

======= test lib/dispatchcloud/container -- 2s

</pre>

Python

<pre>

What next? test sdk/python --test-suite=tests.test_collections.TextModes.test_read_sailboat_across_block_boundary

======= test sdk/python

running test

running egg_info

writing requirements to arvados_python_client.egg-info/requires.txt

writing arvados_python_client.egg-info/PKG-INFO

writing top-level names to arvados_python_client.egg-info/top_level.txt

writing dependency_links to arvados_python_client.egg-info/dependency_links.txt

writing pbr to arvados_python_client.egg-info/pbr.json

reading manifest file 'arvados_python_client.egg-info/SOURCES.txt'

reading manifest template 'MANIFEST.in'

writing manifest file 'arvados_python_client.egg-info/SOURCES.txt'

running build_ext

test_read_sailboat_across_block_boundary (tests.test_collections.TextModes) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.014s

OK

======= test sdk/python -- 1s

</pre>

RailsAPI

<pre>

What next? test services/api TESTOPTS=--name=/.*signed.locators.*/

[...]

# Running:

....

Finished in 1.080084s, 3.7034 runs/s, 461.0751 assertions/s.

</pre>

Workbench

<pre>

What next? test apps/workbench_integration TESTOPTS="-v -n=/.*non-empty.*/"

======= test apps/workbench_integration

[...]

# Running:

Using port 43855 for selenium

.

Finished in 11.831848s, 0.0845 runs/s, 0.1690 assertions/s.

</pre>

h3. Restarting services for integration tests

If you have changed services/api code, and you want to check whether it breaks the lib/dispatchcloud/container integration tests:

<pre>

What next? reset                                # teardown the integration-testing environment

What next? install services/api                 # (only needed if you've updated dependencies)

What next? test lib/dispatchcloud/container     # bring up the integration-testing environment and run tests

What next? test lib/dispatchcloud/container     # leave the integration-testing environment up and run tests

</pre>

h3. Updating cache after pulling master

Always quit interactive mode and restart after modifying run-tests.sh (via git-pull, git-checkout, editing, etc).

When you start, run "install all" to get the latest gem/python dependencies, install updated versions of Arvados services used by integration tests, etc.

Then you can resume your cycle of "test lib/controller", etc.

h3. Controlling test order (Rails)

Rails tests start off with a line like this

<pre>

Run options: -v -d --seed 57089

</pre>

The seed value determines the order tests are run. To reproduce reproduce an order-dependent test failure, specify the same seed as a previous failed run:

<pre>

What next? test services/api TESTOPTS="-v -d --seed 57089"

</pre>

h3. Other options

For more usage info, try:

<pre>

~/arvados/build/run-tests.sh --help

</pre>

h2. Running workbench diagnostics tests

You can run workbench diagnostics tests against any production server. 

Update your workbench application.yml to add a "diagnostics" section with the login token and pipeline details. The below example configuration is to run the "qr1hi-p5p6p-ftcb0o61u4yd2zr" pipeline in "qr1hi" environment.

<pre>

diagnostics:

  secret_token: useanicelongrandomlygeneratedsecrettokenstring

  arvados_workbench_url: https://workbench.qr1hi.arvadosapi.com

  user_tokens:

    active: yourcurrenttokenintheenvironmenttowhichyouarepointing

  pipelines_to_test:

    pipeline_1:

      template_uuid: qr1hi-p5p6p-ftcb0o61u4yd2zr

      input_paths: []

      max_wait_seconds: 300

</pre>

You can now run the "qr1hi" diagnostics tests using the following command:

<pre>

  cd $ARVADOS_HOME

  RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb

</pre>

h2. Running workbench2 tests

These instructions assume a Debian 10 (buster) host system. It is also possible to run the tests in a docker container, which you should start with these arguments:

<pre>

xhost +local:root

ARVADOS_DIR=/path/to/arvados

docker run -ti -v$PWD:$PWD -v$ARVADOS_DIR:/usr/src/arvados -w$PWD --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" debian:buster bash

</pre>

If you use docker, the rest of the instructions should be executed inside the container.

Install the Arvados test dependencies:

<pre>

echo "deb http://deb.debian.org/debian buster-backports main" > /etc/apt/sources.list.d/backports.list

apt-get update

apt-get install -y --no-install-recommends golang -t buster-backports

apt-get install -y --no-install-recommends build-essential ca-certificates git libpam0g-dev

</pre>

Install a few more dependencies for workbench2:

<pre>

apt-get update

apt-get install -y --no-install-recommends gnupg2 sudo curl

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -

echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

apt-get update

apt-get install -y --no-install-recommends yarn

yarn run cypress install

</pre>

Make sure you have both the arvados and arvados-workbench2 source trees available, and then run a command like this (adjust path for the arvados source tree, if necessary) from your workbench2 source tree:

<pre>

./tools/run-integration-tests.sh -i -a /usr/src/arvados

</pre>