Project

General

Profile

Running tests » History » Version 32

Tom Clegg, 03/29/2019 05:27 PM

1 1 Tom Clegg
h1. Running tests
2
3 3 Tom Clegg
{{toc}}
4
5 23 Ward Vandewege
The arvados git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the source tree. Jenkins at https://ci.curoverse.com uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC.
6 1 Tom Clegg
7
h2. Background
8
9
You have the arvados source tree at @~/arvados@ and you might have local modifications.
10
11
h2. Install prerequisites
12
13 29 Tom Clegg
Follow instructions at [[Hacking prerequisites]]: "Install dev environment", etc. Don't miss creating the Postgres database, docker groups, etc.
14 1 Tom Clegg
15 32 Tom Clegg
h2. Environment
16
17
Your locale must use utf-8. Set environment variable LANG=C.UTF-8 if necessary to ensure the "locale" command reports UTF-8.
18
19 29 Tom Clegg
h2. If you're Jenkins
20
21
Run all tests from a clean slate (slow, but more immune to leaks)
22
23
<pre>
24
~/arvados/build/run-tests.sh WORKSPACE=~/arvados
25 1 Tom Clegg
</pre>
26
27 29 Tom Clegg
h2. If you're a developer
28 1 Tom Clegg
29 29 Tom Clegg
Cache everything needed to run test suites:
30 1 Tom Clegg
31 29 Tom Clegg
<pre>
32
mkdir ~/.cache/arvados-build
33
~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install
34
</pre>
35 1 Tom Clegg
36 29 Tom Clegg
Start interactive mode:
37
38 1 Tom Clegg
<pre>
39 29 Tom Clegg
$ ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --interactive
40 1 Tom Clegg
</pre>
41
42 29 Tom Clegg
When prompted, choose a test suite to run:
43 1 Tom Clegg
44 29 Tom Clegg
<pre>
45
== Interactive commands:
46
test TARGET
47
test TARGET:py3        (test with python3)
48
test TARGET -check.vv  (pass arguments to test)
49
install TARGET
50
install env            (go/python libs)
51
install deps           (go/python libs + arvados components needed for integration tests)
52
reset                  (...services used by integration tests)
53
exit
54
== Test targets:
55
cmd/arvados-client              lib/dispatchcloud/container     sdk/go/auth                     sdk/pam:py3                     services/fuse                   tools/crunchstat-summary
56
cmd/arvados-server              lib/dispatchcloud/scheduler     sdk/go/blockdigest              sdk/python                      services/fuse:py3               tools/crunchstat-summary:py3
57
lib/cli                         lib/dispatchcloud/ssh_executor  sdk/go/crunchrunner             sdk/python:py3                  services/health                 tools/keep-block-check
58
lib/cloud                       lib/dispatchcloud/worker        sdk/go/dispatch                 services/arv-git-httpd          services/keep-balance           tools/keep-exercise
59
lib/cloud/azure                 lib/service                     sdk/go/health                   services/crunch-dispatch-local  services/keepproxy              tools/keep-rsync
60
lib/cloud/ec2                   sdk/cwl                         sdk/go/httpserver               services/crunch-dispatch-slurm  services/keepstore              tools/sync-groups
61
lib/cmd                         sdk/cwl:py3                     sdk/go/keepclient               services/crunch-run             services/keep-web
62
lib/controller                  sdk/go/arvados                  sdk/go/manifest                 services/crunchstat             services/nodemanager
63
lib/crunchstat                  sdk/go/arvadosclient            sdk/go/stats                    services/dockercleaner          services/nodemanager:py3
64
lib/dispatchcloud               sdk/go/asyncbuf                 sdk/pam                         services/dockercleaner:py3      services/ws
65
What next? 
66
</pre>
67 1 Tom Clegg
68 29 Tom Clegg
Example: testing lib/dispatchcloud/container, showing verbose/debug logs:
69 1 Tom Clegg
70
<pre>
71 29 Tom Clegg
What next? test lib/dispatchcloud/container/ -check.vv
72
======= test lib/dispatchcloud/container
73
START: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType
74
WARN[0000] cancel container with no suitable instance type  ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
75
WARN[0000] cancel container with no suitable instance type  ContainerUUID=zzzzz-dz642-queuedcontainer error="no suitable instance type"
76
START: queue_test.go:37: IntegrationSuite.TearDownTest
77
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.846s
78
79
PASS: queue_test.go:99: IntegrationSuite.TestCancelIfNoInstanceType     0.223s
80
81
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
82
INFO[0001] adding container to queue                     ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
83
START: queue_test.go:37: IntegrationSuite.TearDownTest
84
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.901s
85
86
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel        0.177s
87
88
OK: 2 passed
89
PASS
90
ok      git.curoverse.com/arvados.git/lib/dispatchcloud/container       2.150s
91
======= test lib/dispatchcloud/container -- 3s
92
Pass: lib/dispatchcloud/container tests (3s)
93
All test suites passed.
94 1 Tom Clegg
</pre>
95
96 29 Tom Clegg
h3. Running individual test cases
97 1 Tom Clegg
98 29 Tom Clegg
Most Go packages use gocheck. Use gocheck command line args like -check.f.
99
100 1 Tom Clegg
<pre>
101 29 Tom Clegg
What next? test lib/dispatchcloud/container -check.vv -check.f=LockUnlock
102
======= test lib/dispatchcloud/container
103
START: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel
104
INFO[0000] adding container to queue                     ContainerUUID=zzzzz-dz642-queuedcontainer InstanceType=testType Priority=1 State=Queued
105
START: queue_test.go:37: IntegrationSuite.TearDownTest
106
PASS: queue_test.go:37: IntegrationSuite.TearDownTest   0.812s
107
108
PASS: queue_test.go:42: IntegrationSuite.TestGetLockUnlockCancel        0.184s
109
110
OK: 1 passed
111
PASS
112
ok      git.curoverse.com/arvados.git/lib/dispatchcloud/container       1.000s
113
======= test lib/dispatchcloud/container -- 2s
114 1 Tom Clegg
</pre>
115
116 29 Tom Clegg
Python
117 1 Tom Clegg
118
<pre>
119 29 Tom Clegg
What next? test sdk/python --test-suite=tests.test_collections.TextModes.test_read_sailboat_across_block_boundary
120
======= test sdk/python
121
running test
122
running egg_info
123
writing requirements to arvados_python_client.egg-info/requires.txt
124
writing arvados_python_client.egg-info/PKG-INFO
125
writing top-level names to arvados_python_client.egg-info/top_level.txt
126
writing dependency_links to arvados_python_client.egg-info/dependency_links.txt
127
writing pbr to arvados_python_client.egg-info/pbr.json
128
reading manifest file 'arvados_python_client.egg-info/SOURCES.txt'
129
reading manifest template 'MANIFEST.in'
130
writing manifest file 'arvados_python_client.egg-info/SOURCES.txt'
131
running build_ext
132
test_read_sailboat_across_block_boundary (tests.test_collections.TextModes) ... ok
133
134
----------------------------------------------------------------------
135
Ran 1 test in 0.014s
136
137
OK
138 1 Tom Clegg
======= test sdk/python -- 1s
139
</pre>
140
141 32 Tom Clegg
RailsAPI
142 23 Ward Vandewege
143 29 Tom Clegg
<pre>
144 31 Tom Clegg
What next? test services/api TESTOPTS=--name=/.*signed.locators.*/
145
[...]
146
# Running:
147
148
....
149
150
Finished in 1.080084s, 3.7034 runs/s, 461.0751 assertions/s.
151 1 Tom Clegg
</pre>
152 32 Tom Clegg
153
Workbench
154
155
<pre>
156
What next? test apps/workbench_integration TESTOPTS=-n=/.*non-empty.*/
157
======= test apps/workbench_integration
158
[...]
159
# Running:
160
161
Using port 43855 for selenium
162
.
163
164
Finished in 11.831848s, 0.0845 runs/s, 0.1690 assertions/s.
165
</pre>
166
167 20 Tom Clegg
168 29 Tom Clegg
h3. Restarting services for integration tests
169 20 Tom Clegg
170 29 Tom Clegg
If you have changed services/api code, and you want to check whether it breaks the lib/dispatchcloud/container integration tests:
171 20 Tom Clegg
172 29 Tom Clegg
<pre>
173
What next? reset                                # teardown the integration-testing environment
174
What next? install services/api                 # (only needed if you've updated dependencies)
175
What next? test lib/dispatchcloud/container     # bring up the integration-testing environment and run tests
176
What next? test lib/dispatchcloud/container     # leave the integration-testing environment up and run tests
177
</pre>
178 20 Tom Clegg
179 29 Tom Clegg
h3. Updating cache after pulling master
180 3 Tom Clegg
181 29 Tom Clegg
Always quit interactive mode and restart after modifying run-tests.sh (via git-pull, git-checkout, editing, etc).
182 3 Tom Clegg
183 29 Tom Clegg
When you start, run "install all" to get the latest gem/python dependencies, install updated versions of Arvados services used by integration tests, etc.
184 3 Tom Clegg
185 29 Tom Clegg
Then you can resume your cycle of "test lib/controller", etc.
186
187
h3. Other options
188
189 24 Radhika Chippada
For more usage info, try:
190
191
<pre>
192 26 Radhika Chippada
~/arvados/build/run-tests.sh --help
193 24 Radhika Chippada
</pre>
194
195
h2. Running workbench diagnostics tests
196
197
You can run workbench diagnostics tests against any production server. 
198
199 26 Radhika Chippada
Update your workbench application.yml to add a "diagnostics" section with the login token and pipeline details. The below example configuration is to run the "qr1hi-p5p6p-ftcb0o61u4yd2zr" pipeline in "qr1hi" environment.
200 24 Radhika Chippada
201
<pre>
202
diagnostics:
203
  secret_token: useanicelongrandomlygeneratedsecrettokenstring
204
  arvados_workbench_url: https://workbench.qr1hi.arvadosapi.com
205
  user_tokens:
206
    active: yourcurrenttokenintheenvironmenttowhichyouarepointing
207 26 Radhika Chippada
  pipelines_to_test:
208 24 Radhika Chippada
    pipeline_1:
209
      template_uuid: qr1hi-p5p6p-ftcb0o61u4yd2zr
210 25 Radhika Chippada
      input_paths: []
211 24 Radhika Chippada
      max_wait_seconds: 300
212
</pre>
213 5 Tom Clegg
214
You can now run the "qr1hi" diagnostics tests using the following command:
215
<pre>
216
  cd $ARVADOS_HOME
217
  RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb
218
</pre>