Project

General

Profile

Running tests » History » Revision 60

Revision 59 (Stephen Smith, 03/04/2024 03:27 AM) → Revision 60/64 (Stephen Smith, 03/11/2024 04:51 PM)

h1. Running tests 

 {{toc}} 

 The arvados git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the source tree. Jenkins at https://ci.arvados.org uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC. 

 h2. Background 

 You have the arvados source tree at @~/arvados@ and you might have local modifications. 

 h2. Install prerequisites 

 Follow instructions at [[Hacking prerequisites]]: "Install dev environment", etc. Don't miss creating the Postgres database, docker groups, etc. 

 h2. Environment 

 Your locale must use utf-8. Set environment variable LANG=C.UTF-8 if necessary to ensure the "locale" command reports UTF-8. 

 h2. If you're Jenkins 

 Run all tests from a clean slate (slow, but more immune to leaks) 

 <pre> 
 ~/arvados/build/run-tests.sh WORKSPACE=~/arvados 
 </pre> 

 h2. If you're a developer 

 Cache everything needed to run test suites: 

 <pre> 
 mkdir ~/.cache/arvados-build 
 ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install 
 </pre> 

 Start interactive mode: 

 <pre> 
 $ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --interactive 
 </pre> 

 Start interactive mode and enabled debug output: 

 <pre> 
 $ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados ARVADOS_DEBUG=1 --temp ~/.cache/arvados-build --interactive 
 </pre> 


 When prompted, choose a test suite to run: 

 <pre> 
 == Interactive commands: 
 test TARGET 
 test TARGET:py3          (test with python3) 
 test TARGET -check.vv    (pass arguments to test) 
 install TARGET 
 install env              (go/python libs) 
 install deps             (go/python libs + arvados components needed for integration tests) 
 reset                    (...services used by integration tests) 
 exit 
 == Test targets: 
 cmd/arvados-client                lib/dispatchcloud/container       sdk/go/auth                       sdk/pam:py3                       services/fuse                     tools/crunchstat-summary 
 cmd/arvados-server                lib/dispatchcloud/scheduler       sdk/go/blockdigest                sdk/python                        services/fuse:py3                 tools/crunchstat-summary:py3 
 lib/cli                           lib/dispatchcloud/ssh_executor    sdk/go/crunchrunner               sdk/python:py3                    services/health                   tools/keep-block-check 
 lib/cloud                         lib/dispatchcloud/worker          sdk/go/dispatch                   services/arv-git-httpd            services/keep-balance             tools/keep-exercise 
 lib/cloud/azure                   lib/service                       sdk/go/health                     services/crunch-dispatch-local    services/keepproxy                tools/keep-rsync 
 lib/cloud/ec2                     sdk/cwl                           sdk/go/httpserver                 services/crunch-dispatch-slurm    services/keepstore                tools/sync-groups 
 lib/cmd                           sdk/cwl:py3                       sdk/go/keepclient                 services/crunch-run               services/keep-web 
 lib/controller                    sdk/go/arvados                    sdk/go/manifest                   services/crunchstat               services/nodemanager 
 lib/crunchstat                    sdk/go/arvadosclient              sdk/go/stats                      services/dockercleaner            services/nodemanager:py3 
 lib/dispatchcloud                 sdk/go/asyncbuf                   sdk/pam                           services/dockercleaner:py3        services/ws 
 What next?  
 </pre> 

 Example: testing lib/dispatchcloud/container, showing verbose/debug logs: 

 <pre> 
 What next? test lib/dispatchcloud/container/ -check.vv 
 ======= test lib/dispatchcloud/container 
 START: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType 
 WARN[0000] cancel container with no suitable instance type    ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type" 
 WARN[0000] cancel container with no suitable instance type    ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type" 
 START: queue_test.go:37: IntegrationSuite.TearDownTest 
 PASS: queue_test.go:37: IntegrationSuite.TearDownTest     0.846s 

 PASS: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType       0.223s 

 START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel 
 INFO[0001] adding container to queue                       ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued 
 START: queue_test.go:37: IntegrationSuite.TearDownTest 
 PASS: queue_test.go:37: IntegrationSuite.TearDownTest     0.901s 

 PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel          0.177s 

 OK: 2 passed 
 PASS 
 ok        git.arvados.org/arvados.git/lib/dispatchcloud/container         2.150s 
 ======= test lib/dispatchcloud/container -- 3s 
 Pass: lib/dispatchcloud/container tests (3s) 
 All test suites passed. 
 </pre> 

 h3. Running individual test cases 

 Most Go packages use gocheck. Use gocheck command line args like -check.f. 

 <pre> 
 What next? test lib/dispatchcloud/container -check.vv -check.f=LockUnlock 
 ======= test lib/dispatchcloud/container 
 START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel 
 INFO[0000] adding container to queue                       ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued 
 START: queue_test.go:37: IntegrationSuite.TearDownTest 
 PASS: queue_test.go:37: IntegrationSuite.TearDownTest     0.812s 

 PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel          0.184s 

 OK: 1 passed 
 PASS 
 ok        git.arvados.org/arvados.git/lib/dispatchcloud/container         1.000s 
 ======= test lib/dispatchcloud/container -- 2s 
 </pre> 

 Python 

 <pre> 
 What next? test sdk/python --test-suite=tests.test_collections.TextModes.test_read_sailboat_across_block_boundary 
 ======= test sdk/python 
 running test 
 running egg_info 
 writing requirements to arvados_python_client.egg-info/requires.txt 
 writing arvados_python_client.egg-info/PKG-INFO 
 writing top-level names to arvados_python_client.egg-info/top_level.txt 
 writing dependency_links to arvados_python_client.egg-info/dependency_links.txt 
 writing pbr to arvados_python_client.egg-info/pbr.json 
 reading manifest file 'arvados_python_client.egg-info/SOURCES.txt' 
 reading manifest template 'MANIFEST.in' 
 writing manifest file 'arvados_python_client.egg-info/SOURCES.txt' 
 running build_ext 
 test_read_sailboat_across_block_boundary (tests.test_collections.TextModes) ... ok 

 ---------------------------------------------------------------------- 
 Ran 1 test in 0.014s 

 OK 
 ======= test sdk/python -- 1s 
 </pre> 

 RailsAPI 

 <pre> 
 What next? test services/api TESTOPTS=--name=/.*signed.locators.*/ 
 [...] 
 # Running: 

 .... 

 Finished in 1.080084s, 3.7034 runs/s, 461.0751 assertions/s. 
 </pre> 

 Workbench 

 <pre> 
 What next? test apps/workbench_integration TESTOPTS="-v -n=/.*non-empty.*/" 
 ======= test apps/workbench_integration 
 [...] 
 # Running: 

 Using port 43855 for selenium 
 . 

 Finished in 11.831848s, 0.0845 runs/s, 0.1690 assertions/s. 
 </pre> 

 h3. Restarting services for integration tests 

 If you have changed services/api code, and you want to check whether it breaks the lib/dispatchcloud/container integration tests: 

 <pre> 
 What next? reset                                  # teardown the integration-testing environment 
 What next? install services/api                   # (only needed if you've updated dependencies) 
 What next? test lib/dispatchcloud/container       # bring up the integration-testing environment and run tests 
 What next? test lib/dispatchcloud/container       # leave the integration-testing environment up and run tests 
 </pre> 

 h3. Updating cache after pulling master 

 Always quit interactive mode and restart after modifying run-tests.sh (via git-pull, git-checkout, editing, etc). 

 When you start, run "install all" to get the latest gem/python dependencies, install updated versions of Arvados services used by integration tests, etc. 

 Then you can resume your cycle of "test lib/controller", etc. 

 h3. Controlling test order (Rails) 

 Rails tests start off with a line like this 

 <pre> 
 Run options: -v -d --seed 57089 
 </pre> 

 The seed value determines the order tests are run. To reproduce reproduce an order-dependent test failure, specify the same seed as a previous failed run: 

 <pre> 
 What next? test services/api TESTOPTS="-v -d --seed 57089" 
 </pre> 


 h3. Other options 

 For more usage info, try: 

 <pre> 
 ~/arvados/build/run-tests.sh --help 
 </pre> 

 h2. Running workbench diagnostics tests 

 You can run workbench diagnostics tests against any production server.  

 Update your workbench application.yml to add a "diagnostics" section with the login token and pipeline details. The below example configuration is to run the "qr1hi-p5p6p-ftcb0o61u4yd2zr" pipeline in "qr1hi" environment. 

 <pre> 
 diagnostics: 
   secret_token: useanicelongrandomlygeneratedsecrettokenstring 
   arvados_workbench_url: https://workbench.qr1hi.arvadosapi.com 
   user_tokens: 
     active: yourcurrenttokenintheenvironmenttowhichyouarepointing 
   pipelines_to_test: 
     pipeline_1: 
       template_uuid: qr1hi-p5p6p-ftcb0o61u4yd2zr 
       input_paths: [] 
       max_wait_seconds: 300 
 </pre> 

 You can now run the "qr1hi" diagnostics tests using the following command: 
 <pre> 
   cd $ARVADOS_HOME 
   RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb 
 </pre> 

 h2. Running workbench2 tests 

 React uses a lot of filesystem watchers (via inotify). The default number of watched files is relatively low at 8192. Increase that with: 

   echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p 

 

 h3. Docker 

 The integration tests can be run on non-debian based systems using docker. The workbench2 repo includes a Dockerfile that preinstalls the necessary dependencies to run cypress. 

 Build the docker image using this command from within the @arvados@ repository 

 <pre> 
 docker build . -f services/workbench2/docker/Dockerfile -t workbench2-build 
 </pre> 

 Then, start the container using the following arguments: 

 <pre> 
 xhost +local:root 
 ARVADOS_DIR=/path/to/arvados 
 docker run -ti -v$PWD:$PWD -v$ARVADOS_DIR:/usr/src/arvados -w$PWD --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" workbench2-build:latest bash 
 </pre> 

 h4. Docker Troubleshooting 

 h5. No version of Cypress is installed 

 Solution: Manually install the required cypress version by running @./node_modules/.bin/cypress install@ inside the container 

 h3. Debian Host System 

 These instructions assume a Debian 10 (buster) host system.  

 Install the Arvados test dependencies: 

 <pre> 
 echo "deb http://deb.debian.org/debian buster-backports main" > /etc/apt/sources.list.d/backports.list 
 apt-get update 
 apt-get install -y --no-install-recommends golang -t buster-backports 
 apt-get install -y --no-install-recommends build-essential ca-certificates git libpam0g-dev 
 </pre> 

 Install a few more dependencies for workbench2: 

 <pre> 
 apt-get update 
 apt-get install -y --no-install-recommends gnupg2 sudo curl 
 curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - 
 echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list 
 apt-get update 
 apt-get install -y --no-install-recommends yarn libgbm-dev 
 # we need the correct version of node to install cypress 
 # use arvados-server install to get it (and all other dependencies) 
 # All this will then not need to be repeated by ./tools/run-integration-tests.sh 
 # so we are not doing double work. 
 cd /usr/src/arvados 
 go mod download 
 cd cmd/arvados-server 
 go install 
 ~/go/bin/arvados-server install -type test 
 cd <your-wb2-directory> 
 yarn run cypress install 
 </pre> 

 Make sure you have both the arvados and arvados-workbench2 source trees available, and then use the following commands (adjust path for the arvados source tree, if necessary) from your workbench2 source tree. 

 h3. Running Tests 

 Run the unit tests with: 
 <pre> 
 make unit-tests 
 # or 
 yarn test 
 </pre> 

 Run the cypress integration tests with: 

 <pre> 
 ./tools/run-integration-tests.sh -i -a /usr/src/arvados 
 </pre>