Story #16306

[install] Build all-in-one server package using arvados-server install/boot in production mode

Added by Tom Clegg 8 months ago. Updated 3 days ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
09/22/2020
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
1.0


Subtasks

Task #16413: Review 16306-arvados-easy-installIn ProgressWard Vandewege

History

#1 Updated by Tom Clegg 8 months ago

  • Tracker changed from Bug to Story

#2 Updated by Tom Clegg 7 months ago

  • Status changed from New to In Progress

#3 Updated by Peter Amstutz 7 months ago

  • Target version changed from 2020-04-22 to 2020-05-06 Sprint

#4 Updated by Tom Clegg 7 months ago

  • Target version changed from 2020-05-06 Sprint to 2020-05-20 Sprint

#5 Updated by Tom Clegg 6 months ago

  • Target version changed from 2020-05-20 Sprint to 2020-06-03 Sprint

#6 Updated by Tom Clegg 6 months ago

16306-arvados-easy-install @ dcfc58303cac84750cc93e92f3b39629b2e3e792

#7 Updated by Tom Clegg 6 months ago

In the "install via OS packages" scenario, the operator won't use "arvados-server install" directly -- rather, they will install the "arvados-server" debian package and use "arvados-server init" to create a new cluster, which (among other things) will set up a systemd service that invokes "arvados-server install -type production" to do any needed setup tasks, followed by "arvados-server boot" to bring up the server processes.

This way, any applicable time consuming setup/upgrade tasks -- compiling a new version of Ruby, installing bundler, running database migrations -- will run asynchronously instead of rudely blocking the "apt-get install" or "apt-get upgrade" process as they do now.

#8 Updated by Peter Amstutz 6 months ago

  • Target version changed from 2020-06-03 Sprint to 2020-06-17 Sprint

#9 Updated by Tom Clegg 5 months ago

  • Subject changed from [install] Draft anticipated instructions for using "arvados-server install" in production to [install] Build all-in-one server package using arvados-server install/boot in production mode

#10 Updated by Tom Clegg 5 months ago

  • Target version changed from 2020-06-17 Sprint to 2020-07-01 Sprint

#11 Updated by Tom Clegg 5 months ago

  • Description updated (diff)

#12 Updated by Peter Amstutz 5 months ago

  • Target version changed from 2020-07-01 Sprint to 2020-07-15

#13 Updated by Tom Clegg 4 months ago

  • Target version changed from 2020-07-15 to 2020-08-12 Sprint

#14 Updated by Tom Clegg 4 months ago

  • Description updated (diff)

#15 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-08-12 Sprint to 2020-08-26 Sprint

#17 Updated by Tom Clegg 3 months ago

  • Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint

#18 Updated by Tom Clegg 3 months ago

  • Target version changed from 2020-09-09 Sprint to 2020-09-23 Sprint

#19 Updated by Tom Clegg 2 months ago

  • Target version changed from 2020-09-23 Sprint to 2020-10-07 Sprint

#20 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2020-10-07 Sprint to 2020-10-21 Sprint

#21 Updated by Tom Clegg about 1 month ago

  • Target version changed from 2020-10-21 Sprint to 2020-11-04 Sprint

#22 Updated by Peter Amstutz 19 days ago

  • Target version changed from 2020-11-04 Sprint to 2020-11-18

#23 Updated by Ward Vandewege 14 days ago

A few notes:

  • bundle now also wants to be able to write to ~/.bundle, which leads to lots of noise like this:
`/var/www` is not writable.
Bundler will use `/tmp/bundler/home/www-data' as your home directory temporarily.

I pushed a patch to the 16306-arvados-easy-install branch in f8d13408e99839f52260f889a5089126761eecb1 to add /var/www/.bundle to the list of directories owned by the www-data user.

  • The arvados-server-easy package includes all of the go source (/var/lib/arvados/go). The fpm exclude command lists '/var/lib/arvados/go':
/root/.gem/ruby/2.5.0/bin/fpm --name arvados-server-easy --version 2.0.0-801-ge8d1a643c-dirty --input-type dir --output-type deb --depends automake --depends bison --depends ca-certificates --depends curl --depends fuse --depends git --depends gitolite3 --depends graphviz --depends haveged --depends libcurl3-gnutls --depends libxslt1.1 --depends make --depends nginx --depends python --depends sudo --depends python3-distutils --depends g++ --depends libcurl4-openssl-dev --depends libpq-dev --depends libpython2.7 --depends mime-support --depends zlib1g-dev --deb-use-file-permissions --rpm-use-file-permissions --exclude /var/lib/arvados/go /var/lib/arvados /var/www/.gem /var/www/.passenger

Unsurprisingly, the debian package is enormous:

-rw-r--r-- 1 root root 572M Nov  9 14:43 arvados-server-easy_2.0.0-801-ge8d1a643c-dirty_amd64.deb

This seems to be caused by an fpm issue (https://github.com/jordansissel/fpm/issues/900), the path needs to be specified without leading slash. See ab5199f71c7eaf8bf8fe2b4477353cf432faf1a7 for the fix.

That leads to a smaller package:

-rw-r--r-- 1 root root 454M Nov  9 14:54 arvados-server-easy_2.0.0-801-ge8d1a643c-dirty_amd64.deb

Which doesn't contain the var/lib/arvados/go directory:

root@efa8c8c00352:/pkg# dpkg -c arvados-server-easy_2.0.0-801-ge8d1a643c-dirty_amd64.deb |grep var/lib/arvados/go
root@efa8c8c00352:/pkg# 

DONE:

-rw-r--r--  1 root root 436M Nov 10 09:14 arvados-server-easy_2.0.0-805-g21ce99325-dirty_amd64.deb
-rw-r--r--  1 root root 422M Nov 10 10:37 arvados-server-easy_2.0.0-805-gda7a2e35f-dirty_amd64.deb
-rw-r--r--  1 root root 422M Nov 10 10:37 arvados-server-easy_2.0.0-805-gda7a2e35f-dirty_amd64.deb
-rw-r--r--  1 root root 241M Nov 10 17:07 arvados-server-easy_2.0.0-806-g821c72733-dirty_amd64.deb

TODO:

  • Suggest moving the shell scripts in cmd/arvados-dev to the build directory, and giving them more descriptive and distinctive names.
  • Suggest suggest renaming cmd/arvados-dev to cmd/arvados-build (the idea is to migrate some of the stuff that is currently in build/ from bash to go.
  • The package dependency list for arvados-server-easy includes a bunch of dev packages, can those be removed? automake, bison, make, g++, libcurl4-openssl-dev, libpq-dev, zlib1g-dev
  • Merge master
  • to check: can we migrate to ruby 2.7? Cf. the comment about a gem command that is no longer needed on ruby 2.6.3+. Maybe we can make the ruby version a command line parameter, it will be nice to test newer ruby versions that way.
  • finally: as for where do we go from here: let's get this into shape as a replacement for arvbox in demo mode

#24 Updated by Tom Clegg 5 days ago

  • Target version changed from 2020-11-18 to 2020-12-02 Sprint

Also available in: Atom PDF