Running tests » History » Revision 54
Revision 53 (Ward Vandewege, 03/12/2021 06:52 PM) → Revision 54/64 (Ward Vandewege, 03/12/2021 06:52 PM)
h1. Running tests
{{toc}}
The arvados git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the source tree. Jenkins at https://ci.arvados.org uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC.
h2. Background
You have the arvados source tree at @~/arvados@ and you might have local modifications.
h2. Install prerequisites
Follow instructions at [[Hacking prerequisites]]: "Install dev environment", etc. Don't miss creating the Postgres database, docker groups, etc.
h2. Environment
Your locale must use utf-8. Set environment variable LANG=C.UTF-8 if necessary to ensure the "locale" command reports UTF-8.
h2. If you're Jenkins
Run all tests from a clean slate (slow, but more immune to leaks)
<pre>
~/arvados/build/run-tests.sh WORKSPACE=~/arvados
</pre>
h2. If you're a developer
Cache everything needed to run test suites:
<pre>
mkdir ~/.cache/arvados-build
~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install
</pre>
Start interactive mode:
<pre>
$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --interactive
</pre>
Start interactive mode and enabled debug output:
<pre>
$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados ARVADOS_DEBUG=1 --temp ~/.cache/arvados-build --interactive
</pre>
When prompted, choose a test suite to run:
<pre>
== Interactive commands:
test TARGET
test TARGET:py3 (test with python3)
test TARGET -check.vv (pass arguments to test)
install TARGET
install env (go/python libs)
install deps (go/python libs + arvados components needed for integration tests)
reset (...services used by integration tests)
exit
== Test targets:
cmd/arvados-client lib/dispatchcloud/container sdk/go/auth sdk/pam:py3 services/fuse tools/crunchstat-summary
cmd/arvados-server lib/dispatchcloud/scheduler sdk/go/blockdigest sdk/python services/fuse:py3 tools/crunchstat-summary:py3
lib/cli lib/dispatchcloud/ssh_executor sdk/go/crunchrunner sdk/python:py3 services/health tools/keep-block-check
lib/cloud lib/dispatchcloud/worker sdk/go/dispatch services/arv-git-httpd services/keep-balance tools/keep-exercise
lib/cloud/azure lib/service sdk/go/health services/crunch-dispatch-local services/keepproxy tools/keep-rsync
lib/cloud/ec2 sdk/cwl sdk/go/httpserver services/crunch-dispatch-slurm services/keepstore tools/sync-groups
lib/cmd sdk/cwl:py3 sdk/go/keepclient services/crunch-run services/keep-web
lib/controller sdk/go/arvados sdk/go/manifest services/crunchstat services/nodemanager
lib/crunchstat sdk/go/arvadosclient sdk/go/stats services/dockercleaner services/nodemanager:py3
lib/dispatchcloud sdk/go/asyncbuf sdk/pam services/dockercleaner:py3 services/ws
What next?
</pre>
Example: testing lib/dispatchcloud/container, showing verbose/debug logs:
<pre>
What next? test lib/dispatchcloud/container/ -check.vv
======= test lib/dispatchcloud/container
START: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType
WARN[0000] cancel container with no suitable instance type ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
WARN[0000] cancel container with no suitable instance type ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
START: queue_test.go:37: IntegrationSuite.TearDownTest
PASS: queue_test.go:37: IntegrationSuite.TearDownTest 0.846s
PASS: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType 0.223s
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
INFO[0001] adding container to queue ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
START: queue_test.go:37: IntegrationSuite.TearDownTest
PASS: queue_test.go:37: IntegrationSuite.TearDownTest 0.901s
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel 0.177s
OK: 2 passed
PASS
ok git.arvados.org/arvados.git/lib/dispatchcloud/container 2.150s
======= test lib/dispatchcloud/container -- 3s
Pass: lib/dispatchcloud/container tests (3s)
All test suites passed.
</pre>
h3. Running individual test cases
Most Go packages use gocheck. Use gocheck command line args like -check.f.
<pre>
What next? test lib/dispatchcloud/container -check.vv -check.f=LockUnlock
======= test lib/dispatchcloud/container
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
INFO[0000] adding container to queue ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
START: queue_test.go:37: IntegrationSuite.TearDownTest
PASS: queue_test.go:37: IntegrationSuite.TearDownTest 0.812s
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel 0.184s
OK: 1 passed
PASS
ok git.arvados.org/arvados.git/lib/dispatchcloud/container 1.000s
======= test lib/dispatchcloud/container -- 2s
</pre>
Python
<pre>
What next? test sdk/python --test-suite=tests.test_collections.TextModes.test_read_sailboat_across_block_boundary
======= test sdk/python
running test
running egg_info
writing requirements to arvados_python_client.egg-info/requires.txt
writing arvados_python_client.egg-info/PKG-INFO
writing top-level names to arvados_python_client.egg-info/top_level.txt
writing dependency_links to arvados_python_client.egg-info/dependency_links.txt
writing pbr to arvados_python_client.egg-info/pbr.json
reading manifest file 'arvados_python_client.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'arvados_python_client.egg-info/SOURCES.txt'
running build_ext
test_read_sailboat_across_block_boundary (tests.test_collections.TextModes) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.014s
OK
======= test sdk/python -- 1s
</pre>
RailsAPI
<pre>
What next? test services/api TESTOPTS=--name=/.*signed.locators.*/
[...]
# Running:
....
Finished in 1.080084s, 3.7034 runs/s, 461.0751 assertions/s.
</pre>
Workbench
<pre>
What next? test apps/workbench_integration TESTOPTS="-v -n=/.*non-empty.*/"
======= test apps/workbench_integration
[...]
# Running:
Using port 43855 for selenium
.
Finished in 11.831848s, 0.0845 runs/s, 0.1690 assertions/s.
</pre>
h3. Restarting services for integration tests
If you have changed services/api code, and you want to check whether it breaks the lib/dispatchcloud/container integration tests:
<pre>
What next? reset # teardown the integration-testing environment
What next? install services/api # (only needed if you've updated dependencies)
What next? test lib/dispatchcloud/container # bring up the integration-testing environment and run tests
What next? test lib/dispatchcloud/container # leave the integration-testing environment up and run tests
</pre>
h3. Updating cache after pulling master
Always quit interactive mode and restart after modifying run-tests.sh (via git-pull, git-checkout, editing, etc).
When you start, run "install all" to get the latest gem/python dependencies, install updated versions of Arvados services used by integration tests, etc.
Then you can resume your cycle of "test lib/controller", etc.
h3. Controlling test order (Rails)
Rails tests start off with a line like this
<pre>
Run options: -v -d --seed 57089
</pre>
The seed value determines the order tests are run. To reproduce reproduce an order-dependent test failure, specify the same seed as a previous failed run:
<pre>
What next? test services/api TESTOPTS="-v -d --seed 57089"
</pre>
h3. Other options
For more usage info, try:
<pre>
~/arvados/build/run-tests.sh --help
</pre>
h2. Running workbench diagnostics tests
You can run workbench diagnostics tests against any production server.
Update your workbench application.yml to add a "diagnostics" section with the login token and pipeline details. The below example configuration is to run the "qr1hi-p5p6p-ftcb0o61u4yd2zr" pipeline in "qr1hi" environment.
<pre>
diagnostics:
secret_token: useanicelongrandomlygeneratedsecrettokenstring
arvados_workbench_url: https://workbench.qr1hi.arvadosapi.com
user_tokens:
active: yourcurrenttokenintheenvironmenttowhichyouarepointing
pipelines_to_test:
pipeline_1:
template_uuid: qr1hi-p5p6p-ftcb0o61u4yd2zr
input_paths: []
max_wait_seconds: 300
</pre>
You can now run the "qr1hi" diagnostics tests using the following command:
<pre>
cd $ARVADOS_HOME
RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb
</pre>
h2. Running workbench2 tests
React uses a lot of filesystem watchers (via inotify). The default number of numbe rof watched files is relatively low at 8192. Increase that with:
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
These instructions assume a Debian 10 (buster) host system. It is also possible to run the tests in a docker container, which you should start with these arguments:
<pre>
xhost +local:root
ARVADOS_DIR=/path/to/arvados
docker run -ti -v$PWD:$PWD -v$ARVADOS_DIR:/usr/src/arvados -w$PWD --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" debian:buster bash
</pre>
If you use docker, the rest of the instructions should be executed inside the container.
Install the Arvados test dependencies:
<pre>
echo "deb http://deb.debian.org/debian buster-backports main" > /etc/apt/sources.list.d/backports.list
apt-get update
apt-get install -y --no-install-recommends golang -t buster-backports
apt-get install -y --no-install-recommends build-essential ca-certificates git libpam0g-dev
</pre>
Install a few more dependencies for workbench2:
<pre>
apt-get update
apt-get install -y --no-install-recommends gnupg2 sudo curl
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
apt-get update
apt-get install -y --no-install-recommends yarn libgbm-dev
# we need the correct version of node to install cypress
# use arvados-server install to get it (and all other dependencies)
# All this will then not need to be repeated by ./tools/run-integration-tests.sh
# so we are not doing double work.
cd /usr/src/arvados
go mod download || exit 1
cd cmd/arvados-server
go install
~/go/bin/arvados-server install -type test
cd
yarn run cypress install
</pre>
Make sure you have both the arvados and arvados-workbench2 source trees available, and then run a command like this (adjust path for the arvados source tree, if necessary) from your workbench2 source tree:
<pre>
./tools/run-integration-tests.sh -i -a /usr/src/arvados
</pre>