Project

General

Profile

Hacking prerequisites » History » Version 65

Tom Clegg, 03/12/2020 08:01 PM

1 1 Tom Clegg
{{>toc}}
2
3
h1. Hacking prerequisites
4
5
The Arvados test suite can run in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied.
6
7
h2. Host options
8
9
h3. Starting on your workstation
10
11 54 Tom Clegg
If your workstation is a debian stretch system -- and you don't mind installing a bunch of packages on your workstation, some of them without apt -- the easiest way to get running is to run tests on bare metal. Skip to "Dependencies".
12 1 Tom Clegg
13
Other linux distributions should work too with some modifications, but it's probably easier to use a VM.
14
15
h3. Starting on a VM
16
17 54 Tom Clegg
Another option is to create a virtual machine using something like Xen or VirtualBox, and run debian stretch on it. The instructions below assume you have just a few basic requirements:
18 1 Tom Clegg
* SSH server
19
* sudo (@apt-get install sudo@)
20
* A user account with sudo privileges
21
22
h3. Starting in a docker container
23
24 55 Peter Amstutz
_[[Arvbox]] provides a preinstalled Docker-based dev environment.  The following instructions are for creating a dev environment inside Docker from scratch._
25
26 1 Tom Clegg
This can get you started quickly, but (unlike the above options) you'll need to remember to use something like @docker commit@ to save your state before shutting down your container.
27
28
See http://docker.io for more about installing docker. On debian it looks something like this.
29
30
<pre>
31 54 Tom Clegg
sudo apt-get install docker-ce
32 1 Tom Clegg
sudo adduser $USER docker
33
# {log out & log back in}
34
groups
35
# {should include "docker"}
36
</pre>
37
38 53 Tom Clegg
Start up a new container with debian 9 (stretch), make a new user and log in as that user:
39 1 Tom Clegg
40
<pre>
41 53 Tom Clegg
docker run -it --privileged debian:9 bash
42 1 Tom Clegg
apt-get update
43 34 Tom Clegg
apt-get -y install sudo
44 1 Tom Clegg
adduser me
45 33 Tom Clegg
adduser me sudo
46 1 Tom Clegg
sudo -u me -i
47
</pre>
48 12 Joshua Randall
49 15 Tom Clegg
The &quot;--privileged&quot; is required in order for /dev/fuse to be accessible (without it, no tests that require FUSE will work).
50 1 Tom Clegg
51
h2. Install dev environment
52
53 3 Tom Clegg
<pre>
54 37 Tom Clegg
# only on debian 9 (stretch), to permit ruby 2.3 compilation (see https://github.com/rbenv/ruby-build/wiki#openssl-usrincludeopensslasn1_mach102-error-error-this-file-is-obsolete-please-update-your-software):
55 36 Tom Clegg
sudo apt-get install --no-install-recommends libssl1.0-dev
56
57 45 Tom Clegg
# other systems:
58
sudo apt-get install --no-install-recommends libssl-dev
59
60
# all systems:
61
62 38 Tom Clegg
sudo apt-get install --no-install-recommends \
63 22 Tom Clegg
    bison build-essential cadaver fuse gettext git gitolite3 graphviz \
64
    iceweasel libattr1-dev libfuse-dev libcrypt-ssleay-perl libjson-perl \
65 52 Tom Clegg
    libcrypt-ssleay-perl libcurl3 libcurl3-gnutls libcurl4-openssl-dev curl \
66 1 Tom Clegg
    libjson-perl libpcre3-dev libpq-dev libpython2.7-dev libreadline-dev \
67 59 Tom Clegg
    libxslt1.1 libwww-perl linkchecker lsof net-tools nginx perl-modules \
68 63 Lucas Di Pentima
    postgresql postgresql-contrib python python-epydoc pkg-config sudo virtualenv \
69 65 Tom Clegg
    wget xvfb zlib1g-dev libgnutls28-dev python3-dev libpam-dev \
70 51 Tom Clegg
    r-base r-cran-testthat libxml2-dev pandoc cython bsdmainutils
71 2 Tom Clegg
72 56 Tom Clegg
# ruby 2.5
73 1 Tom Clegg
(
74
 set -e
75
 mkdir -p ~/src
76
 cd ~/src
77 64 Tom Clegg
 wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.7.tar.gz
78
 tar xzf ruby-2.5.7.tar.gz
79
 cd ruby-2.5.7
80 1 Tom Clegg
 ./configure --disable-install-doc
81
 make
82
 sudo make install
83
 sudo gem install bundler
84
)
85
86 64 Tom Clegg
# go >= 1.13
87
sudo apt-get install golang-1.13 || \
88 1 Tom Clegg
(
89
 set -e
90 64 Tom Clegg
 wget https://storage.googleapis.com/golang/go1.13.6.linux-amd64.tar.gz
91
 sudo tar -C /usr/local -xzf go1.13.6.linux-amd64.tar.gz
92 1 Tom Clegg
 cd /usr/local/bin
93
 sudo ln -s ../go/bin/* .
94
)
95
96
# phantomjs 1.9.8
97
(
98
 set -e
99
 PJS=phantomjs-1.9.8-linux-x86_64
100
 wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2
101
 sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2
102
 sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/
103
)
104 43 Tom Clegg
105 48 Tom Clegg
# geckodriver
106
(
107
 set -e
108 50 Tom Clegg
 GD=v0.24.0
109 48 Tom Clegg
 wget -P /tmp https://github.com/mozilla/geckodriver/releases/download/$GD/geckodriver-$GD-linux64.tar.gz
110
 sudo tar -C /usr/local/bin -xzf /tmp/geckodriver-$GD-linux64.tar.gz geckodriver
111
)
112
113 43 Tom Clegg
# npm
114
(
115
 set -e
116
 wget -O- https://nodejs.org/dist/v6.11.2/node-v6.11.2-linux-x64.tar.xz | sudo tar -C /usr/local -xJf -
117 1 Tom Clegg
 sudo ln -s ../node-v6.11.2-linux-x64/bin/{node,npm} /usr/local/bin/
118 43 Tom Clegg
)
119 45 Tom Clegg
</pre>
120 1 Tom Clegg
121 20 Tom Clegg
Note: For ubuntu, virtualenv is python-virtualenv
122 18 Bryan Cosca
123 1 Tom Clegg
h2. Get the arvados source tree and test scripts
124
125
<pre>
126
cd
127
git clone https://github.com/curoverse/arvados.git
128
</pre>
129
130
...or, if you're a committer with your public key on our git server:
131
132
<pre>
133
cd
134
git clone git@git.curoverse.com:arvados.git
135
</pre>
136
137 5 Tom Clegg
h2. Start Postgres
138
139
_If you're running in a docker container_ you'll need to start Postgres manually:
140
141
<pre>
142
sudo /etc/init.d/postgresql start
143
</pre>
144
145 1 Tom Clegg
(If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.)
146 9 Joshua Randall
147 47 Tom Clegg
h2. Ensure entropy
148 11 Joshua Randall
149 47 Tom Clegg
If you're running in a VM, you might run out of entropy, which will make some tests run very slowly. The easiest solution is to install haveged.
150 11 Joshua Randall
151 29 Tom Morris
<pre>
152 47 Tom Clegg
sudo apt-get install haveged
153 11 Joshua Randall
</pre>
154
155 9 Joshua Randall
h2. Setup groups
156
157 17 Tom Clegg
Make sure the fuse and docker groups exist (create them if necessary) and that the user who will run the tests is a member of them.
158 13 Joshua Randall
159 1 Tom Clegg
h2. Create a Postgres user
160
161 5 Tom Clegg
Create an "arvados" user with "create database" privileges. The test suite will create and drop the arvados_test database as needed.
162 1 Tom Clegg
163
<pre>
164
newpw=`tr -cd a-zA-Z </dev/urandom |head -c32`
165 58 Eric Biagiotti
sudo -u postgres psql -c "create user arvados with superuser encrypted password '$newpw'"
166 60 Peter Amstutz
mkdir ~/arvados-test-config
167
cat > ~/arvados-test-config/config.yml <<EOF
168
Clusters:
169
  zzzzz:
170
    PostgreSQL:
171
      Connection:
172
        client_encoding: utf8
173
        dbname: arvados_test
174
        host: localhost
175
        password: $newpw
176
        user: arvados
177
EOF
178 1 Tom Clegg
</pre>
179
180
h2. Run tests
181
182
<pre>
183 61 Peter Amstutz
time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados CONFIGSRC=~/arvados-test-config
184 1 Tom Clegg
</pre>
185
186 14 Tom Clegg
During development, you'll probably want something more like this. It reuses the given temp directory, which avoids a lot of repetitive downloading of dependencies, and allows you to save time with @--skip-install@ or @--only-install sdk/ruby@ and so on.
187 1 Tom Clegg
188
<pre>
189 23 Tom Clegg
mkdir -p ~/.cache/arvados-build
190 62 Tom Clegg
time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados CONFIGSRC=~/arvados-test-config --temp ~/.cache/arvados-build
191 1 Tom Clegg
</pre>