Project

General

Profile

Hacking prerequisites » History » Version 74

Ward Vandewege, 10/02/2020 06:50 PM

1 1 Tom Clegg
{{>toc}}
2
3
h1. Hacking prerequisites
4
5
The Arvados test suite can run in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied.
6
7
h2. Host options
8
9
h3. Starting on your workstation
10
11 70 Ward Vandewege
If your workstation is a debian buster system -- and you don't mind installing a bunch of packages on your workstation, some of them without apt -- the easiest way to get running is to run tests on bare metal. Skip to "Dependencies".
12 1 Tom Clegg
13
Other linux distributions should work too with some modifications, but it's probably easier to use a VM.
14
15
h3. Starting on a VM
16
17 71 Ward Vandewege
Another option is to create a virtual machine using something like Xen or VirtualBox, and run debian buster on it. The instructions below assume you have just a few basic requirements:
18 1 Tom Clegg
* SSH server
19
* sudo (@apt-get install sudo@)
20
* A user account with sudo privileges
21
22
h3. Starting in a docker container
23
24 55 Peter Amstutz
_[[Arvbox]] provides a preinstalled Docker-based dev environment.  The following instructions are for creating a dev environment inside Docker from scratch._
25
26 1 Tom Clegg
This can get you started quickly, but (unlike the above options) you'll need to remember to use something like @docker commit@ to save your state before shutting down your container.
27
28
See http://docker.io for more about installing docker. On debian it looks something like this.
29
30
<pre>
31 54 Tom Clegg
sudo apt-get install docker-ce
32 1 Tom Clegg
sudo adduser $USER docker
33
# {log out & log back in}
34
groups
35
# {should include "docker"}
36
</pre>
37
38 69 Ward Vandewege
Start up a new container with debian 10 (buster), make a new user and log in as that user:
39 1 Tom Clegg
40
<pre>
41 69 Ward Vandewege
docker run -it --privileged debian:10 bash
42 1 Tom Clegg
apt-get update
43 34 Tom Clegg
apt-get -y install sudo
44 1 Tom Clegg
adduser me
45 33 Tom Clegg
adduser me sudo
46 1 Tom Clegg
sudo -u me -i
47
</pre>
48 12 Joshua Randall
49 15 Tom Clegg
The &quot;--privileged&quot; is required in order for /dev/fuse to be accessible (without it, no tests that require FUSE will work).
50 1 Tom Clegg
51
h2. Install dev environment
52
53 67 Tom Clegg
Run the following commands as root.
54 36 Tom Clegg
55 67 Tom Clegg
Note that the last command here ("arvados-server install -type test") installs additional debian packages to your system, along with additional software in /var/lib/arvados/ (such as suitable versions of Ruby and Go) that do not interfere with system packages. It also creates a postgresql database user named "arvados" with an insecure password. Don't expose this postgresql server to the internet or to untrusted users!
56 66 Peter Amstutz
57 73 Ward Vandewege
NOTE: Go 1.12 is the minimum version required to build arvados-server. On Debian Buster, you will want to add the buster-backports apt repository:
58 72 Ward Vandewege
59 1 Tom Clegg
<pre>
60 73 Ward Vandewege
echo "deb http://deb.debian.org/debian buster-backports main" > /etc/apt/sources.list.d/backports.list
61
apt-get update
62 74 Ward Vandewege
apt-get install -y --no-install-recommends golang -t buster-backports
63
apt-get install -y --no-install-recommends build-essential ca-certificates git libpam0g-dev
64 73 Ward Vandewege
</pre>
65
66 74 Ward Vandewege
Alternatively, if you are providing golang 1.12+ a different way:
67 73 Ward Vandewege
68
<pre>
69 74 Ward Vandewege
apt-get install -y --no-install-recommends build-essential ca-certificates git golang libpam0g-dev
70 73 Ward Vandewege
</pre>
71
72
Then:
73
74
<pre>
75 1 Tom Clegg
cd
76 67 Tom Clegg
git clone https://git.arvados.org/arvados.git
77
cd arvados
78 1 Tom Clegg
go mod download
79 68 Ward Vandewege
cd cmd/arvados-server
80 67 Tom Clegg
go install
81
~/go/bin/arvados-server install -type test
82 1 Tom Clegg
</pre>
83
84
h2. Start Postgres
85
86
_If you're running in a docker container_ you'll need to start Postgres manually:
87 9 Joshua Randall
88 47 Tom Clegg
<pre>
89 11 Joshua Randall
sudo /etc/init.d/postgresql start
90 60 Peter Amstutz
</pre>
91
92
(If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.)
93
94
h2. Setup groups
95
96
Make sure the fuse and docker groups exist (create them if necessary) and that the user who will run the tests is a member of them.
97 1 Tom Clegg
98
h2. Run tests
99
100
<pre>
101 67 Tom Clegg
time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados
102 1 Tom Clegg
</pre>
103
104 14 Tom Clegg
During development, you'll probably want something more like this. It reuses the given temp directory, which avoids a lot of repetitive downloading of dependencies, and allows you to save time with @--skip-install@ or @--only-install sdk/ruby@ and so on.
105 1 Tom Clegg
106
<pre>
107 23 Tom Clegg
mkdir -p ~/.cache/arvados-build
108 67 Tom Clegg
time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build
109 1 Tom Clegg
</pre>