Project

General

Profile

Running tests » History » Version 53

Ward Vandewege, 03/12/2021 06:52 PM

1 1 Tom Clegg
h1. Running tests
2
3 3 Tom Clegg
{{toc}}
4
5 33 Ward Vandewege
The arvados git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the source tree. Jenkins at https://ci.arvados.org uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC.
6 1 Tom Clegg
7
h2. Background
8
9
You have the arvados source tree at @~/arvados@ and you might have local modifications.
10
11
h2. Install prerequisites
12
13 29 Tom Clegg
Follow instructions at [[Hacking prerequisites]]: "Install dev environment", etc. Don't miss creating the Postgres database, docker groups, etc.
14 1 Tom Clegg
15 32 Tom Clegg
h2. Environment
16
17
Your locale must use utf-8. Set environment variable LANG=C.UTF-8 if necessary to ensure the "locale" command reports UTF-8.
18
19 29 Tom Clegg
h2. If you're Jenkins
20
21
Run all tests from a clean slate (slow, but more immune to leaks)
22
23
<pre>
24
~/arvados/build/run-tests.sh WORKSPACE=~/arvados
25 1 Tom Clegg
</pre>
26
27 29 Tom Clegg
h2. If you're a developer
28 1 Tom Clegg
29 29 Tom Clegg
Cache everything needed to run test suites:
30 1 Tom Clegg
31 29 Tom Clegg
<pre>
32
mkdir ~/.cache/arvados-build
33
~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install
34
</pre>
35 1 Tom Clegg
36 29 Tom Clegg
Start interactive mode:
37
38 1 Tom Clegg
<pre>
39 29 Tom Clegg
$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --interactive
40 1 Tom Clegg
</pre>
41
42 34 Ward Vandewege
Start interactive mode and enabled debug output:
43
44
<pre>
45
$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados ARVADOS_DEBUG=1 --temp ~/.cache/arvados-build --interactive
46
</pre>
47
48
49 29 Tom Clegg
When prompted, choose a test suite to run:
50 1 Tom Clegg
51 29 Tom Clegg
<pre>
52
== Interactive commands:
53
test TARGET
54
test TARGET:py3        (test with python3)
55
test TARGET -check.vv  (pass arguments to test)
56
install TARGET
57
install env            (go/python libs)
58
install deps           (go/python libs + arvados components needed for integration tests)
59
reset                  (...services used by integration tests)
60
exit
61
== Test targets:
62
cmd/arvados-client              lib/dispatchcloud/container     sdk/go/auth                     sdk/pam:py3                     services/fuse                   tools/crunchstat-summary
63
cmd/arvados-server              lib/dispatchcloud/scheduler     sdk/go/blockdigest              sdk/python                      services/fuse:py3               tools/crunchstat-summary:py3
64
lib/cli                         lib/dispatchcloud/ssh_executor  sdk/go/crunchrunner             sdk/python:py3                  services/health                 tools/keep-block-check
65
lib/cloud                       lib/dispatchcloud/worker        sdk/go/dispatch                 services/arv-git-httpd          services/keep-balance           tools/keep-exercise
66
lib/cloud/azure                 lib/service                     sdk/go/health                   services/crunch-dispatch-local  services/keepproxy              tools/keep-rsync
67
lib/cloud/ec2                   sdk/cwl                         sdk/go/httpserver               services/crunch-dispatch-slurm  services/keepstore              tools/sync-groups
68
lib/cmd                         sdk/cwl:py3                     sdk/go/keepclient               services/crunch-run             services/keep-web
69
lib/controller                  sdk/go/arvados                  sdk/go/manifest                 services/crunchstat             services/nodemanager
70
lib/crunchstat                  sdk/go/arvadosclient            sdk/go/stats                    services/dockercleaner          services/nodemanager:py3
71
lib/dispatchcloud               sdk/go/asyncbuf                 sdk/pam                         services/dockercleaner:py3      services/ws
72
What next? 
73
</pre>
74 1 Tom Clegg
75 29 Tom Clegg
Example: testing lib/dispatchcloud/container, showing verbose/debug logs:
76 1 Tom Clegg
77
<pre>
78 29 Tom Clegg
What next? test lib/dispatchcloud/container/ -check.vv
79
======= test lib/dispatchcloud/container
80
START: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType
81
WARN[0000] cancel container with no suitable instance type  ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
82
WARN[0000] cancel container with no suitable instance type  ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
83
START: queue_test.go:37: IntegrationSuite.TearDownTest
84
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.846s
85
86
PASS: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType     0.223s
87
88
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
89
INFO[0001] adding container to queue                     ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
90
START: queue_test.go:37: IntegrationSuite.TearDownTest
91
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.901s
92
93
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel        0.177s
94
95
OK: 2 passed
96
PASS
97 33 Ward Vandewege
ok      git.arvados.org/arvados.git/lib/dispatchcloud/container       2.150s
98 29 Tom Clegg
======= test lib/dispatchcloud/container -- 3s
99
Pass: lib/dispatchcloud/container tests (3s)
100
All test suites passed.
101 1 Tom Clegg
</pre>
102
103 29 Tom Clegg
h3. Running individual test cases
104 1 Tom Clegg
105 29 Tom Clegg
Most Go packages use gocheck. Use gocheck command line args like -check.f.
106
107 1 Tom Clegg
<pre>
108 29 Tom Clegg
What next? test lib/dispatchcloud/container -check.vv -check.f=LockUnlock
109
======= test lib/dispatchcloud/container
110
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
111
INFO[0000] adding container to queue                     ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
112
START: queue_test.go:37: IntegrationSuite.TearDownTest
113
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.812s
114
115
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel        0.184s
116
117
OK: 1 passed
118
PASS
119 33 Ward Vandewege
ok      git.arvados.org/arvados.git/lib/dispatchcloud/container       1.000s
120 29 Tom Clegg
======= test lib/dispatchcloud/container -- 2s
121 1 Tom Clegg
</pre>
122
123 29 Tom Clegg
Python
124 1 Tom Clegg
125
<pre>
126 29 Tom Clegg
What next? test sdk/python --test-suite=tests.test_collections.TextModes.test_read_sailboat_across_block_boundary
127
======= test sdk/python
128
running test
129
running egg_info
130
writing requirements to arvados_python_client.egg-info/requires.txt
131
writing arvados_python_client.egg-info/PKG-INFO
132
writing top-level names to arvados_python_client.egg-info/top_level.txt
133
writing dependency_links to arvados_python_client.egg-info/dependency_links.txt
134
writing pbr to arvados_python_client.egg-info/pbr.json
135
reading manifest file 'arvados_python_client.egg-info/SOURCES.txt'
136
reading manifest template 'MANIFEST.in'
137
writing manifest file 'arvados_python_client.egg-info/SOURCES.txt'
138
running build_ext
139
test_read_sailboat_across_block_boundary (tests.test_collections.TextModes) ... ok
140
141
----------------------------------------------------------------------
142
Ran 1 test in 0.014s
143
144
OK
145 1 Tom Clegg
======= test sdk/python -- 1s
146
</pre>
147
148 32 Tom Clegg
RailsAPI
149 23 Ward Vandewege
150 29 Tom Clegg
<pre>
151 31 Tom Clegg
What next? test services/api TESTOPTS=--name=/.*signed.locators.*/
152
[...]
153
# Running:
154
155
....
156
157
Finished in 1.080084s, 3.7034 runs/s, 461.0751 assertions/s.
158 1 Tom Clegg
</pre>
159 32 Tom Clegg
160
Workbench
161
162
<pre>
163 36 Tom Clegg
What next? test apps/workbench_integration TESTOPTS="-v -n=/.*non-empty.*/"
164 32 Tom Clegg
======= test apps/workbench_integration
165
[...]
166
# Running:
167
168
Using port 43855 for selenium
169
.
170
171
Finished in 11.831848s, 0.0845 runs/s, 0.1690 assertions/s.
172
</pre>
173 20 Tom Clegg
174 29 Tom Clegg
h3. Restarting services for integration tests
175 20 Tom Clegg
176 29 Tom Clegg
If you have changed services/api code, and you want to check whether it breaks the lib/dispatchcloud/container integration tests:
177 20 Tom Clegg
178 29 Tom Clegg
<pre>
179
What next? reset                                # teardown the integration-testing environment
180
What next? install services/api                 # (only needed if you've updated dependencies)
181
What next? test lib/dispatchcloud/container     # bring up the integration-testing environment and run tests
182
What next? test lib/dispatchcloud/container     # leave the integration-testing environment up and run tests
183
</pre>
184 20 Tom Clegg
185 29 Tom Clegg
h3. Updating cache after pulling master
186 3 Tom Clegg
187 29 Tom Clegg
Always quit interactive mode and restart after modifying run-tests.sh (via git-pull, git-checkout, editing, etc).
188 3 Tom Clegg
189 29 Tom Clegg
When you start, run "install all" to get the latest gem/python dependencies, install updated versions of Arvados services used by integration tests, etc.
190 3 Tom Clegg
191 29 Tom Clegg
Then you can resume your cycle of "test lib/controller", etc.
192
193 35 Tom Clegg
h3. Controlling test order (Rails)
194
195
Rails tests start off with a line like this
196
197
<pre>
198
Run options: -v -d --seed 57089
199
</pre>
200
201
The seed value determines the order tests are run. To reproduce reproduce an order-dependent test failure, specify the same seed as a previous failed run:
202
203
<pre>
204
What next? test services/api TESTOPTS="-v -d --seed 57089"
205
</pre>
206
207
208 29 Tom Clegg
h3. Other options
209
210 24 Radhika Chippada
For more usage info, try:
211
212
<pre>
213 26 Radhika Chippada
~/arvados/build/run-tests.sh --help
214 24 Radhika Chippada
</pre>
215
216
h2. Running workbench diagnostics tests
217
218
You can run workbench diagnostics tests against any production server. 
219
220 26 Radhika Chippada
Update your workbench application.yml to add a "diagnostics" section with the login token and pipeline details. The below example configuration is to run the "qr1hi-p5p6p-ftcb0o61u4yd2zr" pipeline in "qr1hi" environment.
221 24 Radhika Chippada
222
<pre>
223
diagnostics:
224
  secret_token: useanicelongrandomlygeneratedsecrettokenstring
225
  arvados_workbench_url: https://workbench.qr1hi.arvadosapi.com
226
  user_tokens:
227
    active: yourcurrenttokenintheenvironmenttowhichyouarepointing
228 26 Radhika Chippada
  pipelines_to_test:
229 24 Radhika Chippada
    pipeline_1:
230
      template_uuid: qr1hi-p5p6p-ftcb0o61u4yd2zr
231 25 Radhika Chippada
      input_paths: []
232 24 Radhika Chippada
      max_wait_seconds: 300
233
</pre>
234 5 Tom Clegg
235
You can now run the "qr1hi" diagnostics tests using the following command:
236
<pre>
237
  cd $ARVADOS_HOME
238
  RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb
239
</pre>
240 37 Ward Vandewege
241
h2. Running workbench2 tests
242
243 53 Ward Vandewege
React uses a lot of filesystem watchers (via inotify). The default numbe rof watched files is relatively low at 8192. Increase that with:
244
245
  echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
246
247 46 Ward Vandewege
These instructions assume a Debian 10 (buster) host system. It is also possible to run the tests in a docker container, which you should start with these arguments:
248
249
<pre>
250
xhost +local:root
251
ARVADOS_DIR=/path/to/arvados
252 49 Ward Vandewege
docker run -ti -v$PWD:$PWD -v$ARVADOS_DIR:/usr/src/arvados -w$PWD --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" debian:buster bash
253 46 Ward Vandewege
</pre>
254 1 Tom Clegg
255 47 Ward Vandewege
If you use docker, the rest of the instructions should be executed inside the container.
256
257 45 Ward Vandewege
Install the Arvados test dependencies:
258
259
<pre>
260
echo "deb http://deb.debian.org/debian buster-backports main" > /etc/apt/sources.list.d/backports.list
261
apt-get update
262
apt-get install -y --no-install-recommends golang -t buster-backports
263
apt-get install -y --no-install-recommends build-essential ca-certificates git libpam0g-dev
264
</pre>
265
266
Install a few more dependencies for workbench2:
267 37 Ward Vandewege
268
<pre>
269 39 Ward Vandewege
apt-get update
270 40 Ward Vandewege
apt-get install -y --no-install-recommends gnupg2 sudo curl
271 37 Ward Vandewege
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
272
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
273
apt-get update
274 52 Ward Vandewege
apt-get install -y --no-install-recommends yarn libgbm-dev
275 51 Ward Vandewege
# we need the correct version of node to install cypress
276
# use arvados-server install to get it (and all other dependencies)
277
# All this will then not need to be repeated by ./tools/run-integration-tests.sh
278
# so we are not doing double work.
279
cd /usr/src/arvados
280
go mod download || exit 1
281
cd cmd/arvados-server
282
go install
283
~/go/bin/arvados-server install -type test
284
cd
285 38 Ward Vandewege
yarn run cypress install
286 37 Ward Vandewege
</pre>
287
288 43 Ward Vandewege
Make sure you have both the arvados and arvados-workbench2 source trees available, and then run a command like this (adjust path for the arvados source tree, if necessary) from your workbench2 source tree:
289 37 Ward Vandewege
290
<pre>
291 48 Ward Vandewege
./tools/run-integration-tests.sh -i -a /usr/src/arvados
292 37 Ward Vandewege
</pre>