Project

General

Profile

Hacking prerequisites » History » Version 5

Tom Clegg, 06/30/2015 09:32 PM

1 1 Tom Clegg
{{>toc}}
2
3
h1. Hacking prerequisites
4
5
The Arvados test suite can run in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied.
6
7
h2. Host options
8
9
h3. Starting on your workstation
10
11
If your workstation is a debian wheezy or jessie system -- and you don't mind installing a bunch of packages on your workstation, some of them without apt -- the easiest way to get running is to run tests on bare metal. Skip to "Dependencies".
12
13
Other linux distributions should work too with some modifications, but it's probably easier to use a VM.
14
15
h3. Starting on a VM
16
17
Another option is to create a virtual machine using something like Xen or VirtualBox, and run debian jessie on it. The instructions below assume you have just a few basic requirements:
18
* SSH server
19
* sudo (@apt-get install sudo@)
20
* A user account with sudo privileges
21
22
h3. Starting in a docker container
23
24
This can get you started quickly, but (unlike the above options) you'll need to remember to use something like @docker commit@ to save your state before shutting down your container.
25
26
See http://docker.io for more about installing docker. On debian it looks something like this.
27
28
<pre>
29
echo 'deb http://http.debian.net/debian jessie-backports main' | sudo tee -a /etc/apt/sources.list.d/jessie-backports.list
30
sudo apt-get install docker.io
31
sudo adduser $USER docker
32
# {log out & log back in}
33
groups
34
# {should include "docker"}
35
</pre>
36
37
Start up a new container with jessie, make a new user and log in as that user:
38
39
<pre>
40
docker run -it debian:jessie bash
41
apt-get update
42
apt-get install sudo
43
adduser me
44
adduser me sudo
45
sudo -u me -i
46
</pre>
47
48
h2. Install dev environment
49
50 3 Tom Clegg
<pre>
51 4 Tom Clegg
sudo apt-get install bison build-essential fuse gettext git graphviz \
52 1 Tom Clegg
    iceweasel libattr1-dev libfuse-dev libcurl3 libcurl3-gnutls \
53
    libcurl4-openssl-dev libpcre3-dev libpq-dev libpython2.7-dev \
54 2 Tom Clegg
    libreadline-dev libssl-dev libxslt1.1 linkchecker nginx \
55
    postgresql python python-epydoc pkg-config sudo virtualenv \
56
    wget xvfb zlib1g-dev
57
58 1 Tom Clegg
# ruby 2.1:
59
sudo apt-get install ruby2.1 || \
60
(
61
 set -e
62
 mkdir -p ~/src
63
 cd ~/src
64
 wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.6.tar.gz
65
 tar xzf ruby-2.1.6.tar.gz
66
 cd ruby-2.1.6
67
 ./configure --disable-install-doc
68
 make
69
 sudo make install
70
 sudo gem install bundler
71
)
72
73
# go >= 1.3
74
sudo apt-get install golang=2:1.3.3-1 || \
75
(
76
 set -e
77
 wget https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz
78
 sudo tar -C /usr/local -xzf go1.4.2.linux-amd64.tar.gz
79
 cd /usr/local/bin
80
 sudo ln -s ../go/bin/* .
81
)
82
83
# phantomjs 1.9.8
84
(
85
 set -e
86
 PJS=phantomjs-1.9.8-linux-x86_64
87
 wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2
88
 sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2
89
 sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/
90
)
91
</pre>
92
93
h2. Get the arvados source tree and test scripts
94
95
<pre>
96
cd
97
git clone https://github.com/curoverse/arvados.git
98
git clone https://github.com/curoverse/arvados-dev.git
99
</pre>
100
101
...or, if you're a committer with your public key on our git server:
102
103
<pre>
104
cd
105
git clone git@git.curoverse.com:arvados.git
106
git clone git@git.curoverse.com:arvados-dev.git
107
</pre>
108
109 5 Tom Clegg
h2. Start Postgres
110
111
_If you're running in a docker container_ you'll need to start Postgres manually:
112
113
<pre>
114
sudo /etc/init.d/postgresql start
115
</pre>
116
117
(If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.)
118
119 1 Tom Clegg
h2. Create a Postgres user
120
121 5 Tom Clegg
Create an "arvados" user with "create database" privileges. The test suite will create and drop the arvados_test database as needed.
122 1 Tom Clegg
123
<pre>
124
newpw=`tr -cd a-zA-Z </dev/urandom |head -c32`
125
sudo -u postgres psql -c "create user arvados with createdb encrypted password '$newpw'"
126
cp -i ~/arvados/services/api/config/database.yml{.sample,}
127
newpw="$newpw" perl -pi~ -e 's/xxxxxxxx/$ENV{newpw}/' ~/arvados/services/api/config/database.yml
128
</pre>
129
130
h2. Run tests
131
132
<pre>
133
time ~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados
134
</pre>
135
136
During development, you'll probably want something more like this. It leaves its temp dirs in place after running tests, which allows you to save time with @--skip-install@ or @--only-install sdk/ruby@ and so on.
137
138
<pre>
139
time ~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados \
140
 VENVDIR=~/.cache/arvados-venv \
141
 VENV3DIR=~/.cache/arvados-venv3 \
142
 GOPATH=~/.cache/arvados-gopath \
143
 GEMHOME=~/.cache/arvados-gemhome
144
</pre>