Project

General

Profile

Running tests » History » Version 21

Tom Clegg, 09/12/2015 02:44 AM

1 1 Tom Clegg
h1. Running tests
2
3 3 Tom Clegg
{{toc}}
4
5 1 Tom Clegg
The arvados-dev git repository has a @run-tests.sh@ script which tests (nearly) all of the components in the arvados source tree. Jenkins at https://ci.curoverse.com uses this exact script, so running it _before pushing a new master_ is a good way to predict whether Jenkins will fail your build and start pestering you by IRC.
6
7
h2. Background
8
9
You have the arvados source tree at @~/arvados@ and you might have local modifications.
10
11
h2. Download/update the test script
12
13
First time:
14
15
<pre>
16
cd
17 11 Tom Clegg
git clone git://git.curoverse.com/arvados-dev.git
18 1 Tom Clegg
</pre>
19
20
Next times, to get any updates:
21
22
<pre>
23
cd ~/arvados-dev
24
git pull
25
</pre>
26
27
h2. Install prerequisites
28
29
The test script doesn't (yet) check for all prerequisites up front, so you'll save some time by installing these things before going any attempting to run tests:
30
* [[Go]]
31 13 Nico César
* "API server dependencies": http://doc.arvados.org/install/install-api-server.html
32
** Install debian packages, ruby, bundler. Create "arvados" postgres user with permission to create databases. You don't need to run "bundle install", though.) <pre>createuser arvados --createdb  --pwprompt</pre>
33
34 1 Tom Clegg
* "Workbench dependencies":http://doc.arvados.org/install/install-workbench-app.html (Currently just graphviz. Skip "bundle install".)
35 17 Joshua Randall
** make sure that the user who will run the tests can successfully start nginx (write access may be required for /var/log/nginx and /var/lib/nginx)
36 4 Tom Clegg
* doc build dependencies: <pre>sudo apt-get install python-epydoc linkchecker</pre>
37 8 Ward Vandewege
* "FUSE driver dependencies":https://arvados.org/projects/arvados/wiki/Hacking_Python_SDK#Prerequisites
38 4 Tom Clegg
* Workbench test suite dependencies (see [[Hacking Workbench]]): <pre>
39
sudo apt-get install xvfb iceweasel
40
(set -e
41
 PJS=phantomjs-1.9.7-linux-x86_64
42
 wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2
43
 sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2
44
 sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/
45
)
46 2 Tom Clegg
</pre>
47 14 Nico César
** version 2.0.x is missdetected see https://github.com/teampoltergeist/poltergeist/issues/595 
48 16 Joshua Randall
* Dependencies specific to testing
49
** debian package: lsof (used to configure test environment such as ARVADOS_API_HOST)
50
** debian package: gitolite3 (required by the sanity checks before testing can start)
51
52 1 Tom Clegg
53
h2. Run all tests
54
55
Basic usage:
56
57
<pre>
58
~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados
59
</pre>
60
61 20 Tom Clegg
h2. Save time by skipping install
62 1 Tom Clegg
63 21 Tom Clegg
Normally, even when you're running only one component's test suite, @run-tests.sh@ still runs the _install_ recipe for all of the components. This addresses dependencies between components: for example, Workbench tests expect the Python and CLI SDKs to be installed. It also installs third party dependencies (e.g., ruby gems and python modules).
64 1 Tom Clegg
65 18 Tom Clegg
When you're not worried about these dependencies going out of sync (e.g., you're only touching Workbench tests) you can save time by preserving your install dir between test runs. Install everything once this way:
66 1 Tom Clegg
67
<pre>
68
mkdir ~/.cache/arvados-build
69
~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only install
70
</pre>
71
72
On subsequent tests, you can add @--skip-install@ and choose a single test suite to run, like this:
73
74
<pre>
75
~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --skip-install --only apps/workbench
76
</pre>
77
78
Or:
79
80
<pre>
81
~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build --only-install apps/workbench --only apps/workbench
82 18 Tom Clegg
</pre>
83 20 Tom Clegg
84
h2. Save time by skipping some tests
85
86
Running all tests takes a while. You don't want to do this after each tiny change. You can use the --skip and --only arguments to select specific components. Some components also accept component-specific arguments to select specific test classes/cases.
87
88
 run-tests.sh [...] --skip-install --only apps/workbench apps/workbench_test="TEST=test/integration/websock*rb"
89
90
 run-tests.sh [...] --skip-install --only apps/workbench apps/workbench_test="TESTOPTS=--name=/#6640/"
91
92
 run-tests.sh [...] --skip-install --only sdk/python sdk/python_test="--test-suite tests.test_keep_locator"
93
94
 run-tests.sh [...] --skip-install --only sdk/python sdk/python_test="--test-suite tests.test_keep_locator.ArvadosKeepLocatorTest.base_locators"
95
96
Presumably @[...]@ here includes a @--temp@ argument (see above).
97 3 Tom Clegg
98
h2. Other options
99
100
For more usage info, try:
101
102
<pre>
103
~/arvados-dev/jenkins/run-tests.sh --help
104 1 Tom Clegg
</pre>
105 5 Tom Clegg
106
h1. Deficiencies
107
108
There's currently nowhere to add...
109
* Unit tests for Workbench JavaScript assets (see #4132)
110
* Unit/functional tests for crunch-job (see #4156)
111
* Tests for "arv-run-pipeline-instance"