Running tests¶

• Table of contents
• Running tests
  • Background
  • Download/update the test script
  • Install prerequisites
  • Run all tests
  • Save time by skipping some tests
  • Save more time by skipping install
  • Other options
• Deficiencies

The arvados-dev git repository has a run-tests.sh script which tests (nearly) all of the components in the arvados source tree. Jenkins at https://ci.curoverse.com uses this exact script, so running it before pushing a new master is a good way to predict whether Jenkins will fail your build and start pestering you by IRC.

Background¶

You have the arvados source tree at ~/arvados and you might have local modifications.

Download/update the test script¶

First time:

cd
git clone git://git.curoverse.com/arvados-dev.git

Next times, to get any updates:

cd ~/arvados-dev
git pull

Install prerequisites¶

The test script doesn't (yet) check for all prerequisites up front, so you'll save some time by installing these things before going any attempting to run tests:

• Go
• "API server dependencies": http://doc.arvados.org/install/install-api-server.html
  • Install debian packages, ruby, bundler. Create "arvados" postgres user with permission to create databases. You don't need to run "bundle install", though.)

    createuser arvados --createdb  --pwprompt

• Workbench dependencies (Currently just graphviz. Skip "bundle install".)
  • make sure that the user who will run the tests can successfully start nginx (write access may be required for /var/log/nginx and /var/lib/nginx)
• doc build dependencies:

  sudo apt-get install python-epydoc linkchecker

• FUSE driver dependencies
• Workbench test suite dependencies (see Hacking Workbench):

  sudo apt-get install xvfb iceweasel
  (set -e
   PJS=phantomjs-1.9.7-linux-x86_64
   wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2
   sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2
   sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/
  )
  

  • version 2.0.x is missdetected see https://github.com/teampoltergeist/poltergeist/issues/595
• Dependencies specific to testing
  • debian package: lsof (used to configure test environment such as ARVADOS_API_HOST)
  • debian package: gitolite3 (required by the sanity checks before testing can start)

Run all tests¶

Basic usage:

~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados

Save time by skipping some tests¶

Running all tests takes a while. You don't want to do this after each tiny change. You can use the --skip and --only arguments to select specific components. Some components also accept component-specific arguments to select specific test classes/cases.

run-tests.sh [...] --skip-install --only apps/workbench apps/workbench_test="TEST=test/integration/websock*rb"

run-tests.sh [...] --skip-install --only apps/workbench apps/workbench_test="TESTOPTS=--name=/#6640/"

run-tests.sh [...] --skip-install --only sdk/python sdk/python_test="--test-suite tests.test_keep_locator"

run-tests.sh [...] --skip-install --only sdk/python sdk/python_test="--test-suite tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators"

Presumably [...] here includes --leave-temp and the GOPATH, GEMHOME, VENVDIR vars from a previous test run.

Save more time by skipping install¶

Normally, even when you're running only one component's test suite, run-tests.sh still runs the install recipe for all of the components. This addresses dependencies between components: for example, Workbench tests expect the Python and CLI SDKs to be installed.

When you're not worried about these dependencies going out of sync (e.g., you're only touching Workbench tests) you can install everything once this way:

~/arvados-dev/jenkins/run-tests.sh --only install --leave-temp WORKSPACE=~/arvados

After running everything, you'll see something like this:

Leaving behind temp dirs:  VENVDIR="/tmp/tmp.y3tsTmigio" GOPATH="/tmp/tmp.3r4sSA9F3l" 

Copy these temp dirs to the end of your run-tests.sh command line, add --skip-install, and choose a test suite to run, like this:

~/arvados-dev/jenkins/run-tests.sh --only apps/workbench --skip-install WORKSPACE=~/arvados VENVDIR="/tmp/tmp.y3tsTmigio" GOPATH="/tmp/tmp.3r4sSA9F3l" 

Other options¶

For more usage info, try:

~/arvados-dev/jenkins/run-tests.sh --help

Deficiencies¶

There's currently nowhere to add...

• Unit tests for Workbench JavaScript assets (see #4132)
• Unit/functional tests for crunch-job (see #4156)
• Tests for "arv-run-pipeline-instance"