Project

General

Profile

Hacking prerequisites » History » Version 52

Tom Clegg, 03/28/2019 06:59 PM

1 1 Tom Clegg
{{>toc}}
2
3
h1. Hacking prerequisites
4
5
The Arvados test suite can run in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied.
6
7
h2. Host options
8
9
h3. Starting on your workstation
10
11 31 Ward Vandewege
If your workstation is a debian jessie system -- and you don't mind installing a bunch of packages on your workstation, some of them without apt -- the easiest way to get running is to run tests on bare metal. Skip to "Dependencies".
12 1 Tom Clegg
13
Other linux distributions should work too with some modifications, but it's probably easier to use a VM.
14
15
h3. Starting on a VM
16
17
Another option is to create a virtual machine using something like Xen or VirtualBox, and run debian jessie on it. The instructions below assume you have just a few basic requirements:
18
* SSH server
19
* sudo (@apt-get install sudo@)
20
* A user account with sudo privileges
21
22
h3. Starting in a docker container
23
24
This can get you started quickly, but (unlike the above options) you'll need to remember to use something like @docker commit@ to save your state before shutting down your container.
25
26
See http://docker.io for more about installing docker. On debian it looks something like this.
27
28
<pre>
29
echo 'deb http://http.debian.net/debian jessie-backports main' | sudo tee -a /etc/apt/sources.list.d/jessie-backports.list
30
sudo apt-get install docker.io
31
sudo adduser $USER docker
32
# {log out & log back in}
33
groups
34
# {should include "docker"}
35
</pre>
36
37
Start up a new container with jessie, make a new user and log in as that user:
38
39
<pre>
40 12 Joshua Randall
docker run -it --privileged debian:jessie bash
41 1 Tom Clegg
apt-get update
42 34 Tom Clegg
apt-get -y install sudo
43 1 Tom Clegg
adduser me
44 33 Tom Clegg
adduser me sudo
45 1 Tom Clegg
sudo -u me -i
46
</pre>
47 12 Joshua Randall
48 15 Tom Clegg
The &quot;--privileged&quot; is required in order for /dev/fuse to be accessible (without it, no tests that require FUSE will work).
49 1 Tom Clegg
50
h2. Install dev environment
51
52 3 Tom Clegg
<pre>
53 37 Tom Clegg
# only on debian 9 (stretch), to permit ruby 2.3 compilation (see https://github.com/rbenv/ruby-build/wiki#openssl-usrincludeopensslasn1_mach102-error-error-this-file-is-obsolete-please-update-your-software):
54 36 Tom Clegg
sudo apt-get install --no-install-recommends libssl1.0-dev
55
56 45 Tom Clegg
# other systems:
57
sudo apt-get install --no-install-recommends libssl-dev
58
59
# all systems:
60
61 38 Tom Clegg
sudo apt-get install --no-install-recommends \
62 22 Tom Clegg
    bison build-essential cadaver fuse gettext git gitolite3 graphviz \
63
    iceweasel libattr1-dev libfuse-dev libcrypt-ssleay-perl libjson-perl \
64 52 Tom Clegg
    libcrypt-ssleay-perl libcurl3 libcurl3-gnutls libcurl4-openssl-dev curl \
65 1 Tom Clegg
    libjson-perl libpcre3-dev libpq-dev libpython2.7-dev libreadline-dev \
66 45 Tom Clegg
    libxslt1.1 libwww-perl linkchecker lsof nginx perl-modules \
67 10 Joshua Randall
    postgresql python python-epydoc pkg-config sudo virtualenv \
68 43 Tom Clegg
    wget xvfb zlib1g-dev libgnutls28-dev python3-dev \
69 51 Tom Clegg
    r-base r-cran-testthat libxml2-dev pandoc cython bsdmainutils
70 2 Tom Clegg
71 32 Tom Clegg
# ruby 2.3
72 1 Tom Clegg
(
73
 set -e
74
 mkdir -p ~/src
75
 cd ~/src
76 50 Tom Clegg
 wget http://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.8.tar.gz
77
 tar xzf ruby-2.3.8.tar.gz
78
 cd ruby-2.3.8
79 1 Tom Clegg
 ./configure --disable-install-doc
80
 make
81
 sudo make install
82
 sudo gem install bundler
83
)
84
85 49 Tom Clegg
# go >= 1.10
86
sudo apt-get install golang-1.10 || \
87 1 Tom Clegg
(
88
 set -e
89 49 Tom Clegg
 wget https://storage.googleapis.com/golang/go1.12.1.linux-amd64.tar.gz
90
 sudo tar -C /usr/local -xzf go1.12.1.linux-amd64.tar.gz
91 1 Tom Clegg
 cd /usr/local/bin
92
 sudo ln -s ../go/bin/* .
93
)
94
95
# phantomjs 1.9.8
96
(
97
 set -e
98
 PJS=phantomjs-1.9.8-linux-x86_64
99
 wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2
100
 sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2
101
 sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/
102
)
103 43 Tom Clegg
104 48 Tom Clegg
# geckodriver
105
(
106
 set -e
107 50 Tom Clegg
 GD=v0.24.0
108 48 Tom Clegg
 wget -P /tmp https://github.com/mozilla/geckodriver/releases/download/$GD/geckodriver-$GD-linux64.tar.gz
109
 sudo tar -C /usr/local/bin -xzf /tmp/geckodriver-$GD-linux64.tar.gz geckodriver
110
)
111
112 43 Tom Clegg
# npm
113
(
114
 set -e
115
 wget -O- https://nodejs.org/dist/v6.11.2/node-v6.11.2-linux-x64.tar.xz | sudo tar -C /usr/local -xJf -
116 1 Tom Clegg
 sudo ln -s ../node-v6.11.2-linux-x64/bin/{node,npm} /usr/local/bin/
117 43 Tom Clegg
)
118 45 Tom Clegg
</pre>
119 1 Tom Clegg
120 20 Tom Clegg
Note: For ubuntu, virtualenv is python-virtualenv
121 18 Bryan Cosca
122 1 Tom Clegg
h2. Get the arvados source tree and test scripts
123
124
<pre>
125
cd
126
git clone https://github.com/curoverse/arvados.git
127
</pre>
128
129
...or, if you're a committer with your public key on our git server:
130
131
<pre>
132
cd
133
git clone git@git.curoverse.com:arvados.git
134
</pre>
135
136 5 Tom Clegg
h2. Start Postgres
137
138
_If you're running in a docker container_ you'll need to start Postgres manually:
139
140
<pre>
141
sudo /etc/init.d/postgresql start
142
</pre>
143
144 1 Tom Clegg
(If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.)
145 9 Joshua Randall
146 47 Tom Clegg
h2. Ensure entropy
147 11 Joshua Randall
148 47 Tom Clegg
If you're running in a VM, you might run out of entropy, which will make some tests run very slowly. The easiest solution is to install haveged.
149 11 Joshua Randall
150 29 Tom Morris
<pre>
151 47 Tom Clegg
sudo apt-get install haveged
152 11 Joshua Randall
</pre>
153
154 9 Joshua Randall
h2. Setup groups
155
156 17 Tom Clegg
Make sure the fuse and docker groups exist (create them if necessary) and that the user who will run the tests is a member of them.
157 13 Joshua Randall
158 1 Tom Clegg
h2. Create a Postgres user
159
160 5 Tom Clegg
Create an "arvados" user with "create database" privileges. The test suite will create and drop the arvados_test database as needed.
161 1 Tom Clegg
162
<pre>
163
newpw=`tr -cd a-zA-Z </dev/urandom |head -c32`
164
sudo -u postgres psql -c "create user arvados with createdb encrypted password '$newpw'"
165 16 Tom Clegg
cp -i ~/arvados/services/api/config/database.yml{.example,}
166 1 Tom Clegg
newpw="$newpw" perl -pi~ -e 's/xxxxxxxx/$ENV{newpw}/' ~/arvados/services/api/config/database.yml
167
</pre>
168
169
h2. Run tests
170
171
<pre>
172 24 Ward Vandewege
time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados
173 1 Tom Clegg
</pre>
174
175 14 Tom Clegg
During development, you'll probably want something more like this. It reuses the given temp directory, which avoids a lot of repetitive downloading of dependencies, and allows you to save time with @--skip-install@ or @--only-install sdk/ruby@ and so on.
176 1 Tom Clegg
177
<pre>
178 23 Tom Clegg
mkdir -p ~/.cache/arvados-build
179 24 Ward Vandewege
time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build
180 1 Tom Clegg
</pre>