Project

General

Profile

All-in-one package » History » Version 3

Tom Clegg, 08/07/2020 08:20 PM

1 1 Tom Clegg
h1. All-in-one package
2
3
(draft / work in progress, see #16306 and "arvados-server boot (google doc)":https://docs.google.com/document/d/1UvABO9WOg5svK2LIETaEgg9jZQp6Lydb47HrG3NMsFM/edit)
4
5
"arvados-server-easy" is a debian/rpm package that can be used to create a single-node production cluster from scratch with minimal effort:
6
7
<pre>
8
apt-get install arvados-server-easy
9
arvados-server init -cluster-id x1234
10
systemctl start arvados
11
</pre>
12
13
This page is mostly about how we build/maintain the package, rather than how to use it.
14
15
h2. Building
16
17
source:cmd/arvados-dev (similar to arvados-client/server) has:
18
* a "buildpackage" subcommand that creates a deb/rpm package from source (i.e., a checkout of the arvados source tree, possibly with local modifications). It takes care of installing dependencies, fpm, etc.
19 2 Tom Clegg
* "docker-build-install", a bash script (could move to Go) that builds a package for a given OS version by injecting the buildpackage command into a docker container with that OS version, installs the resulting package into a new base OS image, and saves the resulting image for testing purposes. (currently, debian:10 is implemented)
20
* "docker-boot", a bash script (could move to Go) that brings up a cluster using the testing image (the one with arvados-server-easy already installed), optionally injecting local changes to enable a quicker dev cycle. For example, "docker-boot.sh cmd/arvados-server" builds a new arvados-server binary on the host, then starts a container using the test image but with the new arvados-server binary bind-mounted over the packaged one.
21 1 Tom Clegg
22
Package publishing outline (pseudocode):
23
<pre>
24 3 Tom Clegg
./cmd/arvados-dev/docker-build-install.sh --os debian:10 --force-buildimage --force-installimage
25 1 Tom Clegg
</pre>
26
27 2 Tom Clegg
Development cycle, when working on Go programs and other things that docker-boot.sh knows how to rebuild+overlay:
28 1 Tom Clegg
<pre>
29 2 Tom Clegg
./cmd/arvados-dev/docker-build-install.sh -os debian:10 # takes 10+ minutes
30 1 Tom Clegg
./cmd/arvados-dev/docker-boot.sh
31
# (edit some Go code locally)
32 2 Tom Clegg
./cmd/arvados-dev/docker-boot.sh cmd/arvados-server     # takes 15+ seconds to start postgresql, create db, etc., and boot
33 1 Tom Clegg
# (edit some Go code locally)
34 2 Tom Clegg
./cmd/arvados-dev/docker-boot.sh cmd/arvados-server nginx.conf
35 1 Tom Clegg
# ...
36
</pre>
37 2 Tom Clegg
38
Development cycle when working on packaging stuff itself:
39 3 Tom Clegg
* Rebuild the package-building image from base OS image, then use it to build a new package (this is slow because it downloads/installs OS package dependencies, compiles Ruby, etc.): <pre>./cmd/arvados-dev/docker-build-install.sh --os debian:10 --force-buildimage</pre>
40
* Build a new package, then install it onto a base OS image (this checks whether the package still installs & boots correctly after you change the OS dependencies in source:lib/install, or after the base OS itself has changed): <pre>./cmd/arvados-dev/docker-build-install.sh --os debian:10 --force-installimage</pre>
41
* Build a new package, and reinstall it on the existing cached installimage (this builds and tests an actual package, but skips re-downloading/re-installing all of its OS dependencies): <pre>./cmd/arvados-dev/docker-build-install.sh --os debian:10</pre>