Project

General

Profile

Running tests » History » Revision 44

Revision 43 (Ward Vandewege, 12/14/2020 08:44 PM) → Revision 44/64 (Ward Vandewege, 12/14/2020 08:44 PM)

h1. Running tests 

 {{toc}} 

 The arvados git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the source tree. Jenkins at https://ci.arvados.org uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC. 

 h2. Background 

 You have the arvados source tree at @~/arvados@ and you might have local modifications. 

 h2. Install prerequisites 

 Follow instructions at [[Hacking prerequisites]]: "Install dev environment", etc. Don't miss creating the Postgres database, docker groups, etc. 

 h2. Environment 

 Your locale must use utf-8. Set environment variable LANG=C.UTF-8 if necessary to ensure the "locale" command reports UTF-8. 

 h2. If you're Jenkins 

 Run all tests from a clean slate (slow, but more immune to leaks) 

 <pre> 
 ~/arvados/build/run-tests.sh WORKSPACE=~/arvados 
 </pre> 

 h2. If you're a developer 

 Cache everything needed to run test suites: 

 <pre> 
 mkdir ~/.cache/arvados-build 
 ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install 
 </pre> 

 Start interactive mode: 

 <pre> 
 $ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --interactive 
 </pre> 

 Start interactive mode and enabled debug output: 

 <pre> 
 $ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados ARVADOS_DEBUG=1 --temp ~/.cache/arvados-build --interactive 
 </pre> 


 When prompted, choose a test suite to run: 

 <pre> 
 == Interactive commands: 
 test TARGET 
 test TARGET:py3          (test with python3) 
 test TARGET -check.vv    (pass arguments to test) 
 install TARGET 
 install env              (go/python libs) 
 install deps             (go/python libs + arvados components needed for integration tests) 
 reset                    (...services used by integration tests) 
 exit 
 == Test targets: 
 cmd/arvados-client                lib/dispatchcloud/container       sdk/go/auth                       sdk/pam:py3                       services/fuse                     tools/crunchstat-summary 
 cmd/arvados-server                lib/dispatchcloud/scheduler       sdk/go/blockdigest                sdk/python                        services/fuse:py3                 tools/crunchstat-summary:py3 
 lib/cli                           lib/dispatchcloud/ssh_executor    sdk/go/crunchrunner               sdk/python:py3                    services/health                   tools/keep-block-check 
 lib/cloud                         lib/dispatchcloud/worker          sdk/go/dispatch                   services/arv-git-httpd            services/keep-balance             tools/keep-exercise 
 lib/cloud/azure                   lib/service                       sdk/go/health                     services/crunch-dispatch-local    services/keepproxy                tools/keep-rsync 
 lib/cloud/ec2                     sdk/cwl                           sdk/go/httpserver                 services/crunch-dispatch-slurm    services/keepstore                tools/sync-groups 
 lib/cmd                           sdk/cwl:py3                       sdk/go/keepclient                 services/crunch-run               services/keep-web 
 lib/controller                    sdk/go/arvados                    sdk/go/manifest                   services/crunchstat               services/nodemanager 
 lib/crunchstat                    sdk/go/arvadosclient              sdk/go/stats                      services/dockercleaner            services/nodemanager:py3 
 lib/dispatchcloud                 sdk/go/asyncbuf                   sdk/pam                           services/dockercleaner:py3        services/ws 
 What next?  
 </pre> 

 Example: testing lib/dispatchcloud/container, showing verbose/debug logs: 

 <pre> 
 What next? test lib/dispatchcloud/container/ -check.vv 
 ======= test lib/dispatchcloud/container 
 START: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType 
 WARN[0000] cancel container with no suitable instance type    ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type" 
 WARN[0000] cancel container with no suitable instance type    ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type" 
 START: queue_test.go:37: IntegrationSuite.TearDownTest 
 PASS: queue_test.go:37: IntegrationSuite.TearDownTest     0.846s 

 PASS: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType       0.223s 

 START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel 
 INFO[0001] adding container to queue                       ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued 
 START: queue_test.go:37: IntegrationSuite.TearDownTest 
 PASS: queue_test.go:37: IntegrationSuite.TearDownTest     0.901s 

 PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel          0.177s 

 OK: 2 passed 
 PASS 
 ok        git.arvados.org/arvados.git/lib/dispatchcloud/container         2.150s 
 ======= test lib/dispatchcloud/container -- 3s 
 Pass: lib/dispatchcloud/container tests (3s) 
 All test suites passed. 
 </pre> 

 h3. Running individual test cases 

 Most Go packages use gocheck. Use gocheck command line args like -check.f. 

 <pre> 
 What next? test lib/dispatchcloud/container -check.vv -check.f=LockUnlock 
 ======= test lib/dispatchcloud/container 
 START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel 
 INFO[0000] adding container to queue                       ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued 
 START: queue_test.go:37: IntegrationSuite.TearDownTest 
 PASS: queue_test.go:37: IntegrationSuite.TearDownTest     0.812s 

 PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel          0.184s 

 OK: 1 passed 
 PASS 
 ok        git.arvados.org/arvados.git/lib/dispatchcloud/container         1.000s 
 ======= test lib/dispatchcloud/container -- 2s 
 </pre> 

 Python 

 <pre> 
 What next? test sdk/python --test-suite=tests.test_collections.TextModes.test_read_sailboat_across_block_boundary 
 ======= test sdk/python 
 running test 
 running egg_info 
 writing requirements to arvados_python_client.egg-info/requires.txt 
 writing arvados_python_client.egg-info/PKG-INFO 
 writing top-level names to arvados_python_client.egg-info/top_level.txt 
 writing dependency_links to arvados_python_client.egg-info/dependency_links.txt 
 writing pbr to arvados_python_client.egg-info/pbr.json 
 reading manifest file 'arvados_python_client.egg-info/SOURCES.txt' 
 reading manifest template 'MANIFEST.in' 
 writing manifest file 'arvados_python_client.egg-info/SOURCES.txt' 
 running build_ext 
 test_read_sailboat_across_block_boundary (tests.test_collections.TextModes) ... ok 

 ---------------------------------------------------------------------- 
 Ran 1 test in 0.014s 

 OK 
 ======= test sdk/python -- 1s 
 </pre> 

 RailsAPI 

 <pre> 
 What next? test services/api TESTOPTS=--name=/.*signed.locators.*/ 
 [...] 
 # Running: 

 .... 

 Finished in 1.080084s, 3.7034 runs/s, 461.0751 assertions/s. 
 </pre> 

 Workbench 

 <pre> 
 What next? test apps/workbench_integration TESTOPTS="-v -n=/.*non-empty.*/" 
 ======= test apps/workbench_integration 
 [...] 
 # Running: 

 Using port 43855 for selenium 
 . 

 Finished in 11.831848s, 0.0845 runs/s, 0.1690 assertions/s. 
 </pre> 

 h3. Restarting services for integration tests 

 If you have changed services/api code, and you want to check whether it breaks the lib/dispatchcloud/container integration tests: 

 <pre> 
 What next? reset                                  # teardown the integration-testing environment 
 What next? install services/api                   # (only needed if you've updated dependencies) 
 What next? test lib/dispatchcloud/container       # bring up the integration-testing environment and run tests 
 What next? test lib/dispatchcloud/container       # leave the integration-testing environment up and run tests 
 </pre> 

 h3. Updating cache after pulling master 

 Always quit interactive mode and restart after modifying run-tests.sh (via git-pull, git-checkout, editing, etc). 

 When you start, run "install all" to get the latest gem/python dependencies, install updated versions of Arvados services used by integration tests, etc. 

 Then you can resume your cycle of "test lib/controller", etc. 

 h3. Controlling test order (Rails) 

 Rails tests start off with a line like this 

 <pre> 
 Run options: -v -d --seed 57089 
 </pre> 

 The seed value determines the order tests are run. To reproduce reproduce an order-dependent test failure, specify the same seed as a previous failed run: 

 <pre> 
 What next? test services/api TESTOPTS="-v -d --seed 57089" 
 </pre> 


 h3. Other options 

 For more usage info, try: 

 <pre> 
 ~/arvados/build/run-tests.sh --help 
 </pre> 

 h2. Running workbench diagnostics tests 

 You can run workbench diagnostics tests against any production server.  

 Update your workbench application.yml to add a "diagnostics" section with the login token and pipeline details. The below example configuration is to run the "qr1hi-p5p6p-ftcb0o61u4yd2zr" pipeline in "qr1hi" environment. 

 <pre> 
 diagnostics: 
   secret_token: useanicelongrandomlygeneratedsecrettokenstring 
   arvados_workbench_url: https://workbench.qr1hi.arvadosapi.com 
   user_tokens: 
     active: yourcurrenttokenintheenvironmenttowhichyouarepointing 
   pipelines_to_test: 
     pipeline_1: 
       template_uuid: qr1hi-p5p6p-ftcb0o61u4yd2zr 
       input_paths: [] 
       max_wait_seconds: 300 
 </pre> 

 You can now run the "qr1hi" diagnostics tests using the following command: 
 <pre> 
   cd $ARVADOS_HOME 
   RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb 
 </pre> 

 h2. Running workbench2 tests 

 Make sure to follow the instructions at [[Hacking prerequisites#Install-dev-environment]]: "Install dev environment". 

 Install a few more dependencies: 

 <pre> 
 apt-get update 
 apt-get install -y --no-install-recommends gnupg2 sudo curl 
 curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - 
 echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list 
 apt-get update 
 apt-get install -y --install-recommends yarn 
 yarn run cypress install 
 </pre> 

 Make sure you have both the arvados and arvados-workbench2 source trees available, and then run a command like this (adjust path for the arvados source tree, if necessary) from your workbench2 source tree: 

 <pre> 
 # ./tools/run-integration-tests.sh -i -a ../arvados 
 </pre>