Project

General

Profile

Running tests » History » Version 30

Tom Clegg, 03/27/2019 07:07 PM

1 1 Tom Clegg
h1. Running tests
2
3 3 Tom Clegg
{{toc}}
4
5 23 Ward Vandewege
The arvados git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the source tree. Jenkins at https://ci.curoverse.com uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC.
6 1 Tom Clegg
7
h2. Background
8
9
You have the arvados source tree at @~/arvados@ and you might have local modifications.
10
11
h2. Install prerequisites
12
13 29 Tom Clegg
Follow instructions at [[Hacking prerequisites]]: "Install dev environment", etc. Don't miss creating the Postgres database, docker groups, etc.
14 1 Tom Clegg
15 29 Tom Clegg
h2. If you're Jenkins
16
17
Run all tests from a clean slate (slow, but more immune to leaks)
18
19
<pre>
20
~/arvados/build/run-tests.sh WORKSPACE=~/arvados
21 1 Tom Clegg
</pre>
22
23 29 Tom Clegg
h2. If you're a developer
24 1 Tom Clegg
25 29 Tom Clegg
Cache everything needed to run test suites:
26 1 Tom Clegg
27 29 Tom Clegg
<pre>
28
mkdir ~/.cache/arvados-build
29
~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install
30
</pre>
31 1 Tom Clegg
32 29 Tom Clegg
Start interactive mode:
33
34 1 Tom Clegg
<pre>
35 29 Tom Clegg
$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --interactive
36 1 Tom Clegg
</pre>
37
38 29 Tom Clegg
When prompted, choose a test suite to run:
39 1 Tom Clegg
40 29 Tom Clegg
<pre>
41
== Interactive commands:
42
test TARGET
43
test TARGET:py3        (test with python3)
44
test TARGET -check.vv  (pass arguments to test)
45
install TARGET
46
install env            (go/python libs)
47
install deps           (go/python libs + arvados components needed for integration tests)
48
reset                  (...services used by integration tests)
49
exit
50
== Test targets:
51
cmd/arvados-client              lib/dispatchcloud/container     sdk/go/auth                     sdk/pam:py3                     services/fuse                   tools/crunchstat-summary
52
cmd/arvados-server              lib/dispatchcloud/scheduler     sdk/go/blockdigest              sdk/python                      services/fuse:py3               tools/crunchstat-summary:py3
53
lib/cli                         lib/dispatchcloud/ssh_executor  sdk/go/crunchrunner             sdk/python:py3                  services/health                 tools/keep-block-check
54
lib/cloud                       lib/dispatchcloud/worker        sdk/go/dispatch                 services/arv-git-httpd          services/keep-balance           tools/keep-exercise
55
lib/cloud/azure                 lib/service                     sdk/go/health                   services/crunch-dispatch-local  services/keepproxy              tools/keep-rsync
56
lib/cloud/ec2                   sdk/cwl                         sdk/go/httpserver               services/crunch-dispatch-slurm  services/keepstore              tools/sync-groups
57
lib/cmd                         sdk/cwl:py3                     sdk/go/keepclient               services/crunch-run             services/keep-web
58
lib/controller                  sdk/go/arvados                  sdk/go/manifest                 services/crunchstat             services/nodemanager
59
lib/crunchstat                  sdk/go/arvadosclient            sdk/go/stats                    services/dockercleaner          services/nodemanager:py3
60
lib/dispatchcloud               sdk/go/asyncbuf                 sdk/pam                         services/dockercleaner:py3      services/ws
61
What next? 
62
</pre>
63 1 Tom Clegg
64 29 Tom Clegg
Example: testing lib/dispatchcloud/container, showing verbose/debug logs:
65 1 Tom Clegg
66
<pre>
67 29 Tom Clegg
What next? test lib/dispatchcloud/container/ -check.vv
68
======= test lib/dispatchcloud/container
69
START: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType
70
WARN[0000] cancel container with no suitable instance type  ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
71
WARN[0000] cancel container with no suitable instance type  ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
72
START: queue_test.go:37: IntegrationSuite.TearDownTest
73
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.846s
74
75
PASS: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType     0.223s
76
77
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
78
INFO[0001] adding container to queue                     ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
79
START: queue_test.go:37: IntegrationSuite.TearDownTest
80
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.901s
81
82
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel        0.177s
83
84
OK: 2 passed
85
PASS
86
ok      git.curoverse.com/arvados.git/lib/dispatchcloud/container       2.150s
87
======= test lib/dispatchcloud/container -- 3s
88
Pass: lib/dispatchcloud/container tests (3s)
89
All test suites passed.
90 1 Tom Clegg
</pre>
91
92 29 Tom Clegg
h3. Running individual test cases
93 1 Tom Clegg
94 29 Tom Clegg
Most Go packages use gocheck. Use gocheck command line args like -check.f.
95
96 1 Tom Clegg
<pre>
97 29 Tom Clegg
What next? test lib/dispatchcloud/container -check.vv -check.f=LockUnlock
98
======= test lib/dispatchcloud/container
99
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
100
INFO[0000] adding container to queue                     ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
101
START: queue_test.go:37: IntegrationSuite.TearDownTest
102
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.812s
103
104
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel        0.184s
105
106
OK: 1 passed
107
PASS
108
ok      git.curoverse.com/arvados.git/lib/dispatchcloud/container       1.000s
109
======= test lib/dispatchcloud/container -- 2s
110 1 Tom Clegg
</pre>
111
112 29 Tom Clegg
Python
113 1 Tom Clegg
114
<pre>
115 29 Tom Clegg
What next? test sdk/python --test-suite=tests.test_collections.TextModes.test_read_sailboat_across_block_boundary
116
======= test sdk/python
117
running test
118
running egg_info
119
writing requirements to arvados_python_client.egg-info/requires.txt
120
writing arvados_python_client.egg-info/PKG-INFO
121
writing top-level names to arvados_python_client.egg-info/top_level.txt
122
writing dependency_links to arvados_python_client.egg-info/dependency_links.txt
123
writing pbr to arvados_python_client.egg-info/pbr.json
124
reading manifest file 'arvados_python_client.egg-info/SOURCES.txt'
125
reading manifest template 'MANIFEST.in'
126
writing manifest file 'arvados_python_client.egg-info/SOURCES.txt'
127
running build_ext
128
test_read_sailboat_across_block_boundary (tests.test_collections.TextModes) ... ok
129
130
----------------------------------------------------------------------
131
Ran 1 test in 0.014s
132
133
OK
134
======= test sdk/python -- 1s
135 1 Tom Clegg
</pre>
136
137 29 Tom Clegg
Ruby/Rails
138 23 Ward Vandewege
139 29 Tom Clegg
<pre>
140
What next? test sdk/python TESTOPTS=--name=/.*signed.locators.*/
141
</pre>
142 20 Tom Clegg
143 29 Tom Clegg
h3. Restarting services for integration tests
144 20 Tom Clegg
145 29 Tom Clegg
If you have changed services/api code, and you want to check whether it breaks the lib/dispatchcloud/container integration tests:
146 20 Tom Clegg
147 29 Tom Clegg
<pre>
148
What next? reset                                # teardown the integration-testing environment
149
What next? install services/api                 # (only needed if you've updated dependencies)
150
What next? test lib/dispatchcloud/container     # bring up the integration-testing environment and run tests
151
What next? test lib/dispatchcloud/container     # leave the integration-testing environment up and run tests
152
</pre>
153 20 Tom Clegg
154 29 Tom Clegg
h3. Updating cache after pulling master
155 3 Tom Clegg
156 29 Tom Clegg
Always quit interactive mode and restart after modifying run-tests.sh (via git-pull, git-checkout, editing, etc).
157 3 Tom Clegg
158 29 Tom Clegg
When you start, run "install all" to get the latest gem/python dependencies, install updated versions of Arvados services used by integration tests, etc.
159 3 Tom Clegg
160 29 Tom Clegg
Then you can resume your cycle of "test lib/controller", etc.
161
162
h3. Other options
163
164 24 Radhika Chippada
For more usage info, try:
165
166
<pre>
167 26 Radhika Chippada
~/arvados/build/run-tests.sh --help
168 24 Radhika Chippada
</pre>
169
170
h2. Running workbench diagnostics tests
171
172
You can run workbench diagnostics tests against any production server. 
173
174 26 Radhika Chippada
Update your workbench application.yml to add a "diagnostics" section with the login token and pipeline details. The below example configuration is to run the "qr1hi-p5p6p-ftcb0o61u4yd2zr" pipeline in "qr1hi" environment.
175 24 Radhika Chippada
176
<pre>
177
diagnostics:
178
  secret_token: useanicelongrandomlygeneratedsecrettokenstring
179
  arvados_workbench_url: https://workbench.qr1hi.arvadosapi.com
180
  user_tokens:
181
    active: yourcurrenttokenintheenvironmenttowhichyouarepointing
182 26 Radhika Chippada
  pipelines_to_test:
183 24 Radhika Chippada
    pipeline_1:
184
      template_uuid: qr1hi-p5p6p-ftcb0o61u4yd2zr
185 25 Radhika Chippada
      input_paths: []
186 24 Radhika Chippada
      max_wait_seconds: 300
187
</pre>
188 5 Tom Clegg
189
You can now run the "qr1hi" diagnostics tests using the following command:
190
<pre>
191
  cd $ARVADOS_HOME
192
  RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb
193
</pre>