Project

General

Profile

Hacking prerequisites » History » Version 19

Tom Clegg, 11/05/2015 10:18 PM

1 1 Tom Clegg
{{>toc}}
2
3
h1. Hacking prerequisites
4
5
The Arvados test suite can run in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied.
6
7
h2. Host options
8
9
h3. Starting on your workstation
10
11
If your workstation is a debian wheezy or jessie system -- and you don't mind installing a bunch of packages on your workstation, some of them without apt -- the easiest way to get running is to run tests on bare metal. Skip to "Dependencies".
12
13
Other linux distributions should work too with some modifications, but it's probably easier to use a VM.
14
15
h3. Starting on a VM
16
17
Another option is to create a virtual machine using something like Xen or VirtualBox, and run debian jessie on it. The instructions below assume you have just a few basic requirements:
18
* SSH server
19
* sudo (@apt-get install sudo@)
20
* A user account with sudo privileges
21
22
h3. Starting in a docker container
23
24
This can get you started quickly, but (unlike the above options) you'll need to remember to use something like @docker commit@ to save your state before shutting down your container.
25
26
See http://docker.io for more about installing docker. On debian it looks something like this.
27
28
<pre>
29
echo 'deb http://http.debian.net/debian jessie-backports main' | sudo tee -a /etc/apt/sources.list.d/jessie-backports.list
30
sudo apt-get install docker.io
31
sudo adduser $USER docker
32
# {log out & log back in}
33
groups
34
# {should include "docker"}
35
</pre>
36
37
Start up a new container with jessie, make a new user and log in as that user:
38
39
<pre>
40 12 Joshua Randall
docker run -it --privileged debian:jessie bash
41 1 Tom Clegg
apt-get update
42
apt-get install sudo
43
adduser me
44
adduser me sudo
45
sudo -u me -i
46
</pre>
47 12 Joshua Randall
48 15 Tom Clegg
The &quot;--privileged&quot; is required in order for /dev/fuse to be accessible (without it, no tests that require FUSE will work).
49 1 Tom Clegg
50
h2. Install dev environment
51
52 3 Tom Clegg
<pre>
53 19 Tom Clegg
sudo apt-get install --no-install-recommends \
54
    bison build-essential fuse gettext git gitolite3 graphviz \
55 7 Bryan Cosca
    iceweasel libattr1-dev libfuse-dev libcrypt-ssleay-perl libjson-perl libcurl3 libcurl3-gnutls \
56 1 Tom Clegg
    libcurl4-openssl-dev libpcre3-dev libpq-dev libpython2.7-dev \
57 8 Joshua Randall
    libreadline-dev libssl-dev libxslt1.1 linkchecker lsof nginx perl-modules \
58 10 Joshua Randall
    postgresql python python-epydoc pkg-config sudo virtualenv \
59 2 Tom Clegg
    wget xvfb zlib1g-dev
60
61 1 Tom Clegg
# ruby 2.1:
62 6 Tom Clegg
sudo apt-get install ruby2.1 ruby2.1-dev || \
63 1 Tom Clegg
(
64
 set -e
65
 mkdir -p ~/src
66
 cd ~/src
67 15 Tom Clegg
 wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.7.tar.gz
68
 tar xzf ruby-2.1.7.tar.gz
69
 cd ruby-2.1.7
70 1 Tom Clegg
 ./configure --disable-install-doc
71
 make
72
 sudo make install
73
 sudo gem install bundler
74
)
75
76
# go >= 1.3
77
sudo apt-get install golang=2:1.3.3-1 || \
78
(
79
 set -e
80
 wget https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz
81
 sudo tar -C /usr/local -xzf go1.4.2.linux-amd64.tar.gz
82
 cd /usr/local/bin
83
 sudo ln -s ../go/bin/* .
84
)
85
86
# phantomjs 1.9.8
87
(
88
 set -e
89
 PJS=phantomjs-1.9.8-linux-x86_64
90
 wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2
91
 sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2
92
 sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/
93
)
94
</pre>
95
96 18 Bryan Cosca
Note: For ubuntu, virtualenv is dh-virtualenv
97
98 1 Tom Clegg
h2. Get the arvados source tree and test scripts
99
100
<pre>
101
cd
102
git clone https://github.com/curoverse/arvados.git
103
git clone https://github.com/curoverse/arvados-dev.git
104
</pre>
105
106
...or, if you're a committer with your public key on our git server:
107
108
<pre>
109
cd
110
git clone git@git.curoverse.com:arvados.git
111
git clone git@git.curoverse.com:arvados-dev.git
112
</pre>
113
114 5 Tom Clegg
h2. Start Postgres
115
116
_If you're running in a docker container_ you'll need to start Postgres manually:
117
118
<pre>
119
sudo /etc/init.d/postgresql start
120
</pre>
121
122 1 Tom Clegg
(If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.)
123 9 Joshua Randall
124 11 Joshua Randall
h2. Start X11
125
126
In order for the apps/workbench tests to function, firefox needs to have an X11 server (or it will fail to start). If you are on a workstation with a "real" X server, that should work. If not, Xvfb is an X server that renders to a virtual framebuffer so that selenium/firefox tests can run in headless mode. Also make sure that the DISPLAY environment variable is set accordingly.
127
128
<pre>
129
Xvfb :0.0
130
export DISPLAY=":0.0"
131
</pre>
132
133 9 Joshua Randall
h2. Setup groups
134
135 17 Tom Clegg
Make sure the fuse and docker groups exist (create them if necessary) and that the user who will run the tests is a member of them.
136 13 Joshua Randall
137 1 Tom Clegg
h2. Create a Postgres user
138
139 5 Tom Clegg
Create an "arvados" user with "create database" privileges. The test suite will create and drop the arvados_test database as needed.
140 1 Tom Clegg
141
<pre>
142
newpw=`tr -cd a-zA-Z </dev/urandom |head -c32`
143
sudo -u postgres psql -c "create user arvados with createdb encrypted password '$newpw'"
144 16 Tom Clegg
cp -i ~/arvados/services/api/config/database.yml{.example,}
145 1 Tom Clegg
newpw="$newpw" perl -pi~ -e 's/xxxxxxxx/$ENV{newpw}/' ~/arvados/services/api/config/database.yml
146
</pre>
147
148
h2. Run tests
149
150
<pre>
151
time ~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados
152
</pre>
153
154 14 Tom Clegg
During development, you'll probably want something more like this. It reuses the given temp directory, which avoids a lot of repetitive downloading of dependencies, and allows you to save time with @--skip-install@ or @--only-install sdk/ruby@ and so on.
155 1 Tom Clegg
156
<pre>
157 14 Tom Clegg
mkdir ~/.cache/arvados-build
158
time ~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build
159 1 Tom Clegg
</pre>