Project

General

Profile

Running tests » History » Revision 27

Revision 26 (Radhika Chippada, 10/13/2016 05:54 PM) → Revision 27/64 (Tom Clegg, 03/31/2017 05:53 PM)

h1. Running tests 

 {{toc}} 

 The arvados git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the source tree. Jenkins at https://ci.curoverse.com uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC. 

 h2. Background 

 You have the arvados source tree at @~/arvados@ and you might have local modifications. 

 h2. Install prerequisites 

 The test script doesn't (yet) check for all prerequisites up front, so you'll save some time by installing these things before going any attempting to run tests: 
 * [[Go]] 
 * "API server dependencies": http://doc.arvados.org/install/install-api-server.html 
 ** Install debian packages, ruby, bundler. Create "arvados" postgres user with permission to create databases. You don't need to run "bundle install", though.) <pre>createuser arvados --createdb    --pwprompt</pre> 

 * "Workbench dependencies":http://doc.arvados.org/install/install-workbench-app.html (Currently just graphviz. Skip "bundle install".) 
 ** make sure that the user who will run the tests can successfully start nginx (write access may be required for /var/log/nginx and /var/lib/nginx) 
 * doc build dependencies: <pre>sudo apt-get install python-epydoc linkchecker</pre> 
 * "FUSE driver dependencies":https://arvados.org/projects/arvados/wiki/Hacking_Python_SDK#Prerequisites 
 * Workbench test suite dependencies (see [[Hacking Workbench]]): <pre> 
 sudo apt-get install xvfb iceweasel 
 (set -e 
  PJS=phantomjs-1.9.7-linux-x86_64 
  wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2 
  sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2 
  sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/ 
 ) 
 </pre> 
 ** version 2.0.x is missdetected see https://github.com/teampoltergeist/poltergeist/issues/595  
 * Dependencies specific to testing 
 ** debian package: lsof (used to configure test environment such as ARVADOS_API_HOST) 
 ** debian package: gitolite3 (required by the sanity checks before testing can start) 


 h2. Run all tests 

 Basic usage: 

 <pre> 
 ~/arvados/build/run-tests.sh WORKSPACE=~/arvados 
 </pre> 

 h2. Save time by skipping install 

 Normally, even when you're running only one component's test suite, @run-tests.sh@ still runs the _install_ recipe for all of the components. This addresses dependencies between components: for example, Workbench tests expect the Python and CLI SDKs to be installed. It also installs third party dependencies (e.g., ruby gems and python modules). 

 When you're not worried about these dependencies going out of sync (e.g., you're only touching Workbench tests) you can save time by preserving your install dir between test runs. Install everything once this way: 

 <pre> 
 mkdir -p ~/.cache/arvados-build 
 ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install 
 </pre> 

 On subsequent tests, you can add @--skip-install@ and choose a single test suite to run, like this: 

 <pre> 
 ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --skip-install --only apps/workbench 
 </pre> 

 Or: 

 <pre> 
 ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only-install apps/workbench --only apps/workbench 
 </pre> 

 

 h2. Save time by skipping some tests 

 Running all tests takes a while. You don't want to do this after each tiny change. You can use the --skip and --only arguments to select specific components. Some components also accept component-specific arguments to select specific test classes/cases. 

  run-tests.sh [...] --skip-install --only apps/workbench apps/workbench_test="TEST=test/integration/websock*rb" 

  run-tests.sh [...] --skip-install --only apps/workbench apps/workbench_test="TESTOPTS=--name=/.*#6640.*/" apps/workbench_test="TESTOPTS=--name=/#6640/" 

  run-tests.sh [...] --skip-install --only sdk/python sdk/python_test="--test-suite tests.test_keep_locator" 

  run-tests.sh [...] --skip-install --only sdk/python sdk/python_test="--test-suite tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators" 

 Presumably @[...]@ here includes a @--temp@ argument (see above). 

 

 h2. Other options 

 For more usage info, try: 

 <pre> 
 ~/arvados/build/run-tests.sh --help 
 </pre> 

 h2. Running workbench diagnostics tests 

 You can run workbench diagnostics tests against any production server.  

 Update your workbench application.yml to add a "diagnostics" section with the login token and pipeline details. The below example configuration is to run the "qr1hi-p5p6p-ftcb0o61u4yd2zr" pipeline in "qr1hi" environment. 

 <pre> 
 diagnostics: 
   secret_token: useanicelongrandomlygeneratedsecrettokenstring 
   arvados_workbench_url: https://workbench.qr1hi.arvadosapi.com 
   user_tokens: 
     active: yourcurrenttokenintheenvironmenttowhichyouarepointing 
   pipelines_to_test: 
     pipeline_1: 
       template_uuid: qr1hi-p5p6p-ftcb0o61u4yd2zr 
       input_paths: [] 
       max_wait_seconds: 300 
 </pre> 

 You can now run the "qr1hi" diagnostics tests using the following command: 
 <pre> 
   cd $ARVADOS_HOME 
   RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb 
 </pre> 

 h1. Deficiencies 

 There's currently nowhere to add... 
 * Unit tests for Workbench JavaScript assets (see #4132) 
 * Unit/functional tests for crunch-job (see #4156) 
 * Tests for "arv-run-pipeline-instance"