Project

General

Profile

Running tests » History » Version 41

Ward Vandewege, 12/14/2020 08:40 PM

1 1 Tom Clegg
h1. Running tests
2
3 3 Tom Clegg
{{toc}}
4
5 33 Ward Vandewege
The arvados git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the source tree. Jenkins at https://ci.arvados.org uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC.
6 1 Tom Clegg
7
h2. Background
8
9
You have the arvados source tree at @~/arvados@ and you might have local modifications.
10
11
h2. Install prerequisites
12
13 29 Tom Clegg
Follow instructions at [[Hacking prerequisites]]: "Install dev environment", etc. Don't miss creating the Postgres database, docker groups, etc.
14 1 Tom Clegg
15 32 Tom Clegg
h2. Environment
16
17
Your locale must use utf-8. Set environment variable LANG=C.UTF-8 if necessary to ensure the "locale" command reports UTF-8.
18
19 29 Tom Clegg
h2. If you're Jenkins
20
21
Run all tests from a clean slate (slow, but more immune to leaks)
22
23
<pre>
24
~/arvados/build/run-tests.sh WORKSPACE=~/arvados
25 1 Tom Clegg
</pre>
26
27 29 Tom Clegg
h2. If you're a developer
28 1 Tom Clegg
29 29 Tom Clegg
Cache everything needed to run test suites:
30 1 Tom Clegg
31 29 Tom Clegg
<pre>
32
mkdir ~/.cache/arvados-build
33
~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install
34
</pre>
35 1 Tom Clegg
36 29 Tom Clegg
Start interactive mode:
37
38 1 Tom Clegg
<pre>
39 29 Tom Clegg
$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --interactive
40 1 Tom Clegg
</pre>
41
42 34 Ward Vandewege
Start interactive mode and enabled debug output:
43
44
<pre>
45
$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados ARVADOS_DEBUG=1 --temp ~/.cache/arvados-build --interactive
46
</pre>
47
48
49 29 Tom Clegg
When prompted, choose a test suite to run:
50 1 Tom Clegg
51 29 Tom Clegg
<pre>
52
== Interactive commands:
53
test TARGET
54
test TARGET:py3        (test with python3)
55
test TARGET -check.vv  (pass arguments to test)
56
install TARGET
57
install env            (go/python libs)
58
install deps           (go/python libs + arvados components needed for integration tests)
59
reset                  (...services used by integration tests)
60
exit
61
== Test targets:
62
cmd/arvados-client              lib/dispatchcloud/container     sdk/go/auth                     sdk/pam:py3                     services/fuse                   tools/crunchstat-summary
63
cmd/arvados-server              lib/dispatchcloud/scheduler     sdk/go/blockdigest              sdk/python                      services/fuse:py3               tools/crunchstat-summary:py3
64
lib/cli                         lib/dispatchcloud/ssh_executor  sdk/go/crunchrunner             sdk/python:py3                  services/health                 tools/keep-block-check
65
lib/cloud                       lib/dispatchcloud/worker        sdk/go/dispatch                 services/arv-git-httpd          services/keep-balance           tools/keep-exercise
66
lib/cloud/azure                 lib/service                     sdk/go/health                   services/crunch-dispatch-local  services/keepproxy              tools/keep-rsync
67
lib/cloud/ec2                   sdk/cwl                         sdk/go/httpserver               services/crunch-dispatch-slurm  services/keepstore              tools/sync-groups
68
lib/cmd                         sdk/cwl:py3                     sdk/go/keepclient               services/crunch-run             services/keep-web
69
lib/controller                  sdk/go/arvados                  sdk/go/manifest                 services/crunchstat             services/nodemanager
70
lib/crunchstat                  sdk/go/arvadosclient            sdk/go/stats                    services/dockercleaner          services/nodemanager:py3
71
lib/dispatchcloud               sdk/go/asyncbuf                 sdk/pam                         services/dockercleaner:py3      services/ws
72
What next? 
73
</pre>
74 1 Tom Clegg
75 29 Tom Clegg
Example: testing lib/dispatchcloud/container, showing verbose/debug logs:
76 1 Tom Clegg
77
<pre>
78 29 Tom Clegg
What next? test lib/dispatchcloud/container/ -check.vv
79
======= test lib/dispatchcloud/container
80
START: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType
81
WARN[0000] cancel container with no suitable instance type  ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
82
WARN[0000] cancel container with no suitable instance type  ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
83
START: queue_test.go:37: IntegrationSuite.TearDownTest
84
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.846s
85
86
PASS: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType     0.223s
87
88
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
89
INFO[0001] adding container to queue                     ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
90
START: queue_test.go:37: IntegrationSuite.TearDownTest
91
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.901s
92
93
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel        0.177s
94
95
OK: 2 passed
96
PASS
97 33 Ward Vandewege
ok      git.arvados.org/arvados.git/lib/dispatchcloud/container       2.150s
98 29 Tom Clegg
======= test lib/dispatchcloud/container -- 3s
99
Pass: lib/dispatchcloud/container tests (3s)
100
All test suites passed.
101 1 Tom Clegg
</pre>
102
103 29 Tom Clegg
h3. Running individual test cases
104 1 Tom Clegg
105 29 Tom Clegg
Most Go packages use gocheck. Use gocheck command line args like -check.f.
106
107 1 Tom Clegg
<pre>
108 29 Tom Clegg
What next? test lib/dispatchcloud/container -check.vv -check.f=LockUnlock
109
======= test lib/dispatchcloud/container
110
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
111
INFO[0000] adding container to queue                     ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
112
START: queue_test.go:37: IntegrationSuite.TearDownTest
113
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.812s
114
115
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel        0.184s
116
117
OK: 1 passed
118
PASS
119 33 Ward Vandewege
ok      git.arvados.org/arvados.git/lib/dispatchcloud/container       1.000s
120 29 Tom Clegg
======= test lib/dispatchcloud/container -- 2s
121 1 Tom Clegg
</pre>
122
123 29 Tom Clegg
Python
124 1 Tom Clegg
125
<pre>
126 29 Tom Clegg
What next? test sdk/python --test-suite=tests.test_collections.TextModes.test_read_sailboat_across_block_boundary
127
======= test sdk/python
128
running test
129
running egg_info
130
writing requirements to arvados_python_client.egg-info/requires.txt
131
writing arvados_python_client.egg-info/PKG-INFO
132
writing top-level names to arvados_python_client.egg-info/top_level.txt
133
writing dependency_links to arvados_python_client.egg-info/dependency_links.txt
134
writing pbr to arvados_python_client.egg-info/pbr.json
135
reading manifest file 'arvados_python_client.egg-info/SOURCES.txt'
136
reading manifest template 'MANIFEST.in'
137
writing manifest file 'arvados_python_client.egg-info/SOURCES.txt'
138
running build_ext
139
test_read_sailboat_across_block_boundary (tests.test_collections.TextModes) ... ok
140
141
----------------------------------------------------------------------
142
Ran 1 test in 0.014s
143
144
OK
145 1 Tom Clegg
======= test sdk/python -- 1s
146
</pre>
147
148 32 Tom Clegg
RailsAPI
149 23 Ward Vandewege
150 29 Tom Clegg
<pre>
151 31 Tom Clegg
What next? test services/api TESTOPTS=--name=/.*signed.locators.*/
152
[...]
153
# Running:
154
155
....
156
157
Finished in 1.080084s, 3.7034 runs/s, 461.0751 assertions/s.
158 1 Tom Clegg
</pre>
159 32 Tom Clegg
160
Workbench
161
162
<pre>
163 36 Tom Clegg
What next? test apps/workbench_integration TESTOPTS="-v -n=/.*non-empty.*/"
164 32 Tom Clegg
======= test apps/workbench_integration
165
[...]
166
# Running:
167
168
Using port 43855 for selenium
169
.
170
171
Finished in 11.831848s, 0.0845 runs/s, 0.1690 assertions/s.
172
</pre>
173 20 Tom Clegg
174 29 Tom Clegg
h3. Restarting services for integration tests
175 20 Tom Clegg
176 29 Tom Clegg
If you have changed services/api code, and you want to check whether it breaks the lib/dispatchcloud/container integration tests:
177 20 Tom Clegg
178 29 Tom Clegg
<pre>
179
What next? reset                                # teardown the integration-testing environment
180
What next? install services/api                 # (only needed if you've updated dependencies)
181
What next? test lib/dispatchcloud/container     # bring up the integration-testing environment and run tests
182
What next? test lib/dispatchcloud/container     # leave the integration-testing environment up and run tests
183
</pre>
184 20 Tom Clegg
185 29 Tom Clegg
h3. Updating cache after pulling master
186 3 Tom Clegg
187 29 Tom Clegg
Always quit interactive mode and restart after modifying run-tests.sh (via git-pull, git-checkout, editing, etc).
188 3 Tom Clegg
189 29 Tom Clegg
When you start, run "install all" to get the latest gem/python dependencies, install updated versions of Arvados services used by integration tests, etc.
190 3 Tom Clegg
191 29 Tom Clegg
Then you can resume your cycle of "test lib/controller", etc.
192
193 35 Tom Clegg
h3. Controlling test order (Rails)
194
195
Rails tests start off with a line like this
196
197
<pre>
198
Run options: -v -d --seed 57089
199
</pre>
200
201
The seed value determines the order tests are run. To reproduce reproduce an order-dependent test failure, specify the same seed as a previous failed run:
202
203
<pre>
204
What next? test services/api TESTOPTS="-v -d --seed 57089"
205
</pre>
206
207
208 29 Tom Clegg
h3. Other options
209
210 24 Radhika Chippada
For more usage info, try:
211
212
<pre>
213 26 Radhika Chippada
~/arvados/build/run-tests.sh --help
214 24 Radhika Chippada
</pre>
215
216
h2. Running workbench diagnostics tests
217
218
You can run workbench diagnostics tests against any production server. 
219
220 26 Radhika Chippada
Update your workbench application.yml to add a "diagnostics" section with the login token and pipeline details. The below example configuration is to run the "qr1hi-p5p6p-ftcb0o61u4yd2zr" pipeline in "qr1hi" environment.
221 24 Radhika Chippada
222
<pre>
223
diagnostics:
224
  secret_token: useanicelongrandomlygeneratedsecrettokenstring
225
  arvados_workbench_url: https://workbench.qr1hi.arvadosapi.com
226
  user_tokens:
227
    active: yourcurrenttokenintheenvironmenttowhichyouarepointing
228 26 Radhika Chippada
  pipelines_to_test:
229 24 Radhika Chippada
    pipeline_1:
230
      template_uuid: qr1hi-p5p6p-ftcb0o61u4yd2zr
231 25 Radhika Chippada
      input_paths: []
232 24 Radhika Chippada
      max_wait_seconds: 300
233
</pre>
234 5 Tom Clegg
235
You can now run the "qr1hi" diagnostics tests using the following command:
236
<pre>
237
  cd $ARVADOS_HOME
238
  RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb
239
</pre>
240 37 Ward Vandewege
241
h2. Running workbench2 tests
242
243 41 Ward Vandewege
Make sure to follow the instructions at [[Hacking prerequisites#Install-dev-environment]]: "Install dev environment".
244 37 Ward Vandewege
245
Install a few more dependencies:
246
247
<pre>
248 39 Ward Vandewege
apt-get update
249 40 Ward Vandewege
apt-get install -y --no-install-recommends gnupg2 sudo curl
250 37 Ward Vandewege
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
251
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
252
apt-get update
253
apt-get install -y --no-install-recommends yarn
254 38 Ward Vandewege
yarn run cypress install
255 37 Ward Vandewege
</pre>
256
257
Make sure you have both the arvados and arvados-workbench2 source trees available, and then run a command like this (adjust path for the arvados source tree, if necessary):
258
259
<pre>
260
/arvados-workbench2# ./tools/run-integration-tests.sh -i -a ../arvados
261
</pre>