Running tests » History » Revision 29
Revision 28 (Eric Biagiotti, 01/09/2019 02:12 PM) → Revision 29/65 (Tom Clegg, 03/27/2019 07:01 PM)
h1. Running tests
{{toc}}
The arvados git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the source tree. Jenkins at https://ci.curoverse.com uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC.
h2. Background
You have the arvados source tree at @~/arvados@ and you might have local modifications.
h2. Install prerequisites
Follow instructions at [[Hacking prerequisites]]: "Install dev environment", etc. Don't miss creating the Postgres database, docker groups, etc.
h2. If you're Jenkins
Run The test script doesn't (yet) check for all tests from a clean slate (slow, but more immune prerequisites up front, so you'll save some time by installing these things before going any attempting to leaks)
<pre> run tests:
~/arvados/build/run-tests.sh WORKSPACE=~/arvados * [[Go]]
</pre>
h2. If you're a developer
Cache everything needed * "API server dependencies": http://doc.arvados.org/install/install-api-server.html
** Install debian packages, ruby, bundler. Create "arvados" postgres user with permission to create databases. You don't need to run test suites: "bundle install", though.) <pre>createuser arvados --createdb --pwprompt</pre>
<pre> * "Workbench dependencies":http://doc.arvados.org/install/install-workbench-app.html (Currently just graphviz. Skip "bundle install".)
mkdir ~/.cache/arvados-build ** make sure that the user who will run the tests can successfully start nginx (write access may be required for /var/log/nginx and /var/lib/nginx)
~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only * doc build dependencies: <pre>sudo apt-get install python-epydoc linkchecker</pre>
</pre>
Start interactive mode:
<pre> * "FUSE driver dependencies":https://arvados.org/projects/arvados/wiki/Hacking_Python_SDK#Prerequisites
$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --interactive
</pre>
When prompted, choose a * Workbench test suite to run:
dependencies (see [[Hacking Workbench]]): <pre>
== Interactive commands:
TARGET (short for 'test DIR')
test TARGET
test TARGET:py3 (test with python3)
test TARGET -check.vv (pass arguments to test)
sudo apt-get install TARGET xvfb iceweasel
install env (go/python libs) (set -e
PJS=phantomjs-1.9.7-linux-x86_64
wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2
sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2
sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/
install deps (go/python libs + arvados components needed for integration tests) )
reset (...services used by integration tests) </pre>
exit
== Test targets:
cmd/arvados-client lib/dispatchcloud/container sdk/go/auth sdk/pam:py3 services/fuse tools/crunchstat-summary
cmd/arvados-server lib/dispatchcloud/scheduler sdk/go/blockdigest sdk/python services/fuse:py3 tools/crunchstat-summary:py3
lib/cli lib/dispatchcloud/ssh_executor sdk/go/crunchrunner sdk/python:py3 services/health tools/keep-block-check
lib/cloud lib/dispatchcloud/worker sdk/go/dispatch services/arv-git-httpd services/keep-balance tools/keep-exercise
lib/cloud/azure lib/service sdk/go/health services/crunch-dispatch-local services/keepproxy tools/keep-rsync
lib/cloud/ec2 sdk/cwl sdk/go/httpserver services/crunch-dispatch-slurm services/keepstore tools/sync-groups
lib/cmd sdk/cwl:py3 sdk/go/keepclient services/crunch-run services/keep-web
lib/controller sdk/go/arvados sdk/go/manifest services/crunchstat services/nodemanager
lib/crunchstat sdk/go/arvadosclient sdk/go/stats services/dockercleaner services/nodemanager:py3
lib/dispatchcloud sdk/go/asyncbuf sdk/pam services/dockercleaner:py3 services/ws
What next? ** version 2.0.x is missdetected see https://github.com/teampoltergeist/poltergeist/issues/595
</pre>
Example: * Dependencies specific to testing lib/dispatchcloud/container, showing verbose/debug logs:
<pre>
What next? ** debian package: lsof (used to configure test lib/dispatchcloud/container/ -check.vv environment such as ARVADOS_API_HOST)
======= test lib/dispatchcloud/container
START: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType
WARN[0000] cancel container with no suitable instance type ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
WARN[0000] cancel container with no suitable instance type ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
START: queue_test.go:37: IntegrationSuite.TearDownTest
PASS: queue_test.go:37: IntegrationSuite.TearDownTest 0.846s ** debian package: gitolite3 (required by the sanity checks before testing can start)
h2. Run all tests
PASS: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType 0.223s Basic usage:
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel <pre>
INFO[0001] adding container to queue ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued ~/arvados/build/run-tests.sh WORKSPACE=~/arvados
START: queue_test.go:37: IntegrationSuite.TearDownTest
PASS: queue_test.go:37: IntegrationSuite.TearDownTest 0.901s </pre>
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel 0.177s h2. Save time by skipping install
OK: 2 passed
PASS
ok git.curoverse.com/arvados.git/lib/dispatchcloud/container 2.150s
======= Normally, even when you're running only one component's test lib/dispatchcloud/container -- 3s
Pass: lib/dispatchcloud/container suite, @run-tests.sh@ still runs the _install_ recipe for all of the components. This addresses dependencies between components: for example, Workbench tests (3s)
All test suites passed.
</pre> expect the Python and CLI SDKs to be installed. It also installs third party dependencies (e.g., ruby gems and python modules).
h3. Running individual When you're not worried about these dependencies going out of sync (e.g., you're only touching Workbench tests) you can save time by preserving your install dir between test cases runs. Install everything once this way:
Most Go packages use gocheck. Use gocheck command line args like -check.f.
<pre>
What next? test lib/dispatchcloud/container -check.vv -check.f=LockUnlock mkdir -p ~/.cache/arvados-build
======= test lib/dispatchcloud/container ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
INFO[0000] adding container to queue ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
START: queue_test.go:37: IntegrationSuite.TearDownTest
PASS: queue_test.go:37: IntegrationSuite.TearDownTest 0.812s
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel 0.184s
OK: 1 passed
PASS
ok git.curoverse.com/arvados.git/lib/dispatchcloud/container 1.000s
======= test lib/dispatchcloud/container -- 2s
</pre>
Python
<pre>
What next? On subsequent tests, you can add @--skip-install@ and choose a single test sdk/python --test-suite=tests.test_collections.TextModes.test_read_sailboat_across_block_boundary
======= test sdk/python
running test
running egg_info
writing requirements suite to arvados_python_client.egg-info/requires.txt
writing arvados_python_client.egg-info/PKG-INFO
writing top-level names to arvados_python_client.egg-info/top_level.txt
writing dependency_links to arvados_python_client.egg-info/dependency_links.txt
writing pbr to arvados_python_client.egg-info/pbr.json
reading manifest file 'arvados_python_client.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'arvados_python_client.egg-info/SOURCES.txt'
running build_ext
test_read_sailboat_across_block_boundary (tests.test_collections.TextModes) ... ok run, like this:
---------------------------------------------------------------------- <pre>
Ran 1 test in 0.014s
OK ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --skip-install --only apps/workbench
======= test sdk/python -- 1s
</pre>
Ruby/Rails Or:
<pre>
What next? test sdk/python TESTOPTS=--name=/.*signed.locators.*/ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only-install apps/workbench --only apps/workbench
</pre>
h3. Restarting services for integration h2. Save time by skipping some tests
If you have changed services/api code, and you Running all tests takes a while. You don't want to check whether it breaks do this after each tiny change. You can use the lib/dispatchcloud/container integration tests:
<pre>
What next? reset # teardown the integration-testing environment
What next? install services/api # (only needed if you've updated dependencies)
What next? test lib/dispatchcloud/container # bring up the integration-testing environment --skip and run tests
What next? --only arguments to select specific components. Some components also accept component-specific arguments to select specific test lib/dispatchcloud/container # leave the integration-testing environment up and run tests
</pre>
h3. Updating cache after pulling master
Always quit interactive mode and restart after modifying classes/cases.
run-tests.sh (via git-pull, git-checkout, editing, etc). [...] --skip-install --only apps/workbench apps/workbench_test="TEST=test/integration/websock*rb"
run-tests.sh [...] --skip-install --only apps/workbench apps/workbench_test="TESTOPTS=--name=/.*#6640.*/"
run-tests.sh [...] --skip-install --only sdk/python sdk/python_test="--test-suite=tests.test_keep_locator"
run-tests.sh [...] --skip-install --only sdk/python sdk/python_test="--test-suite=tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators"
When you start, run "install all" to get the latest gem/python dependencies, install updated versions of Arvados services used by integration tests, etc. Presumably @[...]@ here includes a @--temp@ argument (see above).
Then you can resume your cycle of "test lib/controller", etc.
h3. h2. Other options
For more usage info, try:
<pre>
~/arvados/build/run-tests.sh --help
</pre>
h2. Running workbench diagnostics tests
You can run workbench diagnostics tests against any production server.
Update your workbench application.yml to add a "diagnostics" section with the login token and pipeline details. The below example configuration is to run the "qr1hi-p5p6p-ftcb0o61u4yd2zr" pipeline in "qr1hi" environment.
<pre>
diagnostics:
secret_token: useanicelongrandomlygeneratedsecrettokenstring
arvados_workbench_url: https://workbench.qr1hi.arvadosapi.com
user_tokens:
active: yourcurrenttokenintheenvironmenttowhichyouarepointing
pipelines_to_test:
pipeline_1:
template_uuid: qr1hi-p5p6p-ftcb0o61u4yd2zr
input_paths: []
max_wait_seconds: 300
</pre>
You can now run the "qr1hi" diagnostics tests using the following command:
<pre>
cd $ARVADOS_HOME
RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb
</pre>
h1. Deficiencies
There's currently nowhere to add...
* Unit tests for Workbench JavaScript assets (see #4132)
* Unit/functional tests for crunch-job (see #4156)
* Tests for "arv-run-pipeline-instance"