Arvados

16306-arvados-easy-install @ dcfc58303cac84750cc93e92f3b39629b2e3e792

In the "install via OS packages" scenario, the operator won't use "arvados-server install" directly -- rather, they will install the "arvados-server" debian package and use "arvados-server init" to create a new cluster, which (among other things) will set up a systemd service that invokes "arvados-server install -type production" to do any needed setup tasks, followed by "arvados-server boot" to bring up the server processes.

This way, any applicable time consuming setup/upgrade tasks -- compiling a new version of Ruby, installing bundler, running database migrations -- will run asynchronously instead of rudely blocking the "apt-get install" or "apt-get upgrade" process as they do now.

16306-arvados-easy-install @ e8d1a643cdbc3a5f4c0e5c745da58d9f7e1248d8 -- developer-run-tests: #2023

A few notes:

• bundle now also wants to be able to write to ~/.bundle, which leads to lots of noise like this:

`/var/www` is not writable.
Bundler will use `/tmp/bundler/home/www-data' as your home directory temporarily.

I pushed a patch to the 16306-arvados-easy-install branch in f8d13408e99839f52260f889a5089126761eecb1 to add /var/www/.bundle to the list of directories owned by the www-data user.

• The arvados-server-easy package includes all of the go source (/var/lib/arvados/go). The fpm exclude command lists '/var/lib/arvados/go':

/root/.gem/ruby/2.5.0/bin/fpm --name arvados-server-easy --version 2.0.0-801-ge8d1a643c-dirty --input-type dir --output-type deb --depends automake --depends bison --depends ca-certificates --depends curl --depends fuse --depends git --depends gitolite3 --depends graphviz --depends haveged --depends libcurl3-gnutls --depends libxslt1.1 --depends make --depends nginx --depends python --depends sudo --depends python3-distutils --depends g++ --depends libcurl4-openssl-dev --depends libpq-dev --depends libpython2.7 --depends mime-support --depends zlib1g-dev --deb-use-file-permissions --rpm-use-file-permissions --exclude /var/lib/arvados/go /var/lib/arvados /var/www/.gem /var/www/.passenger

Unsurprisingly, the debian package is enormous:

-rw-r--r-- 1 root root 572M Nov  9 14:43 arvados-server-easy_2.0.0-801-ge8d1a643c-dirty_amd64.deb

This seems to be caused by an fpm issue (https://github.com/jordansissel/fpm/issues/900), the path needs to be specified without leading slash. See ab5199f71c7eaf8bf8fe2b4477353cf432faf1a7 for the fix.

That leads to a smaller package:

-rw-r--r-- 1 root root 454M Nov  9 14:54 arvados-server-easy_2.0.0-801-ge8d1a643c-dirty_amd64.deb

Which doesn't contain the var/lib/arvados/go directory:

root@efa8c8c00352:/pkg# dpkg -c arvados-server-easy_2.0.0-801-ge8d1a643c-dirty_amd64.deb |grep var/lib/arvados/go
root@efa8c8c00352:/pkg# 

DONE:

• 0b6013ac5520b7508474705d654f1384994d10e5 fix the nginx.go installation differently so it uses wwwtmp instead of tmp
• 0b6013ac5520b7508474705d654f1384994d10e5 fix arvados-boot so that in production mode, it uses /etc/arvados/config.yml
• 821c7273388a56ad5ae9b89a0cf2f351388e8b61 get rid of lib/libruby-static.a (see https://bugs.ruby-lang.org/issues/12845) - a 60 MiB file. Old and new package size:

-rw-r--r--  1 root root 436M Nov 10 09:14 arvados-server-easy_2.0.0-805-g21ce99325-dirty_amd64.deb
-rw-r--r--  1 root root 422M Nov 10 10:37 arvados-server-easy_2.0.0-805-gda7a2e35f-dirty_amd64.deb

• 20649f2d57f8e5c8bddfb672c164b7a96f75efab reduce the size of the package: cf. the bundles which take up an enormous amount of space. Old and new package size:

-rw-r--r--  1 root root 422M Nov 10 10:37 arvados-server-easy_2.0.0-805-gda7a2e35f-dirty_amd64.deb
-rw-r--r--  1 root root 241M Nov 10 17:07 arvados-server-easy_2.0.0-806-g821c72733-dirty_amd64.deb

TODO:

• Suggest moving the shell scripts in cmd/arvados-dev to the build directory, and giving them more descriptive and distinctive names.

• Suggest suggest renaming cmd/arvados-dev to cmd/arvados-build (the idea is to migrate some of the stuff that is currently in build/ from bash to go.

• The package dependency list for arvados-server-easy includes a bunch of dev packages, can those be removed? automake, bison, make, g++, libcurl4-openssl-dev, libpq-dev, zlib1g-dev

• Merge master

• to check: can we migrate to ruby 2.7? Cf. the comment about a gem command that is no longer needed on ruby 2.6.3+. Maybe we can make the ruby version a command line parameter, it will be nice to test newer ruby versions that way.

• finally: as for where do we go from here: let's get this into shape as a replacement for arvbox in demo mode

• arvados-package build -package-dir /tmp/pkg -package-version 2.1.2~rc20210123 -source /path/to/arvados
• arvados-package testinstall -package-dir /tmp/pkg

• cd /path/to/arvados
• go run ./cmd/arvados-package build -package-dir /tmp/pkg -package-version 2.1.2~rc20210123 -source .
• go run ./cmd/arvados-package testinstall -package-dir /tmp/pkg

Each of "build" and "testinstall" creates a base docker image for the respective task on first run, and (unless you say -rebuild-image) reuses that image on subsequent runs to save time.

• specify version for testinstall (currently testinstall always uses the highest version# in the package-dir, which isn't necessarily the one you want to test, e.g., when you're building a 2.1.8 pkg, but the same package-dir already has a 2.2.0 pkg)
• add workbench2 to package (and make "arvados-server boot" run it)
• use real TLS certificates (either provided by operator, or obtained automatically via acme library)

(As well as a long list of todo's that aren't blockers for the demo use case, like ruby 2.7, rpm, ubuntu...)

16306-arvados-easy-install @ 18def2a271e02fd64749fe650034f50d1b659e45 -- developer-run-tests: #2275

(Test failures seem to be caused by a race in lib/boot that makes the integration test setup unreliable. Tracking that down now.)

16306-arvados-easy-install @ 01e15db1f4a331508117bc841256acec8ca361de -- developer-run-tests: #2277

Test failure fixed (it wasn't a race, the problem was setting HOME=/var/www when running passenger in non-production case)

16306-arvados-easy-install @ 4865911a605128adb454b3280b1cf9dcd38f499e -- developer-run-tests: #2286

Implements "arvados-package testinstall -package-version x ..." so you can reliably test a package you just built, even if the package dir also contains packages with higher version numbers.

I tried:

• cd /path/to/arvados
• go run ./cmd/arvados-package build -package-dir /tmp/pkg -package-version 2.1.2~rc20210123 -source .
• go run ./cmd/arvados-package testinstall -package-dir /tmp/pkg

Building the package worked, though I was suprised that there was also a 'Packages.gz' file created, why is that?

-rw-r--r--  1 ward ward 223937554 Jan 29 14:39 arvados-server-easy_2.1.2~rc20210123_amd64.deb
-rw-r--r--  1 ward ward       503 Jan 29 14:39 Packages.gz

• Testing the package failed during package installation:

...
CREATE ROLE
CREATE DATABASE
CREATE EXTENSION
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
time="2021-01-29T19:41:40.152741421Z" level=warning msg="Clusters.xxxxx.ManagementToken: secret token is not set (use 32+ random characters from a-z, A-Z, 0-9)" 
time="2021-01-29T19:41:40.152807024Z" level=warning msg="Clusters.xxxxx.SystemRootToken: secret token is not set (use 32+ random characters from a-z, A-Z, 0-9)" 
time="2021-01-29T19:41:40.152814373Z" level=warning msg="Clusters.xxxxx.Collections.BlobSigningKey: secret token is not set (use 32+ random characters from a-z, A-Z, 0-9)" 
rake aborted!
NoMethodError: undefined method `mass_assignment_sanitizer=' for ActiveRecord::Base:Class
/var/www/.gem/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/dynamic_matchers.rb:22:in `method_missing'
/var/www/.gem/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/railtie.rb:124:in `block (3 levels) in <class:Railtie>'
/var/www/.gem/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/railtie.rb:123:in `each'
/var/www/.gem/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/railtie.rb:123:in `block (2 levels) in <class:Railtie>'
/var/www/.gem/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/lazy_load_hooks.rb:71:in `instance_eval'
/var/www/.gem/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/lazy_load_hooks.rb:71:in `block in execute_hook'
/var/www/.gem/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/lazy_load_hooks.rb:62:in `with_execution_control'
/var/www/.gem/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/lazy_load_hooks.rb:67:in `execute_hook'
/var/www/.gem/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/lazy_load_hooks.rb:43:in `block in on_load'
/var/www/.gem/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/lazy_load_hooks.rb:42:in `each'
/var/www/.gem/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/lazy_load_hooks.rb:42:in `on_load'
/var/www/.gem/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/railtie.rb:120:in `block in <class:Railtie>'
/var/www/.gem/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/initializable.rb:32:in `instance_exec'
/var/www/.gem/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/initializable.rb:32:in `run'
/var/www/.gem/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/initializable.rb:61:in `block in run_initializers'
/var/www/.gem/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/initializable.rb:60:in `run_initializers'
/var/www/.gem/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/application.rb:361:in `initialize!'
/var/lib/arvados/railsapi/config/environment.rb:10:in `<top (required)>'
/var/www/.gem/ruby/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
/var/www/.gem/ruby/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
/var/www/.gem/ruby/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
/var/www/.gem/ruby/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
/var/www/.gem/ruby/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
/var/www/.gem/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:291:in `block in require'
/var/www/.gem/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/var/www/.gem/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:291:in `require'
/var/www/.gem/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/application.rb:337:in `require_environment!'
/var/www/.gem/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/application.rb:520:in `block in run_tasks_blocks'
/var/www/.gem/ruby/2.5.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/var/www/.gem/ruby/2.5.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/var/lib/arvados/bin/bundle:23:in `load'
/var/lib/arvados/bin/bundle:23:in `<main>'
Tasks: TOP => db:setup => db:schema:load_if_ruby => db:create => db:load_config => environment
(See full trace by running task with --trace)
time="2021-01-29T19:41:40.357768019Z" level=info msg=exiting error="rake db:setup: exit status 1" 
ERRO[2021-01-29T14:41:42.009851483-05:00] failed                                        error="docker run: exit status 1" 
exit status 1

• ruby 2.5 will EOL on 2021-03-31. The latest release is 2.5.8. Can we use 2.7.2 here?

Ward Vandewege wrote:

Building the package worked, though I was suprised that there was also a 'Packages.gz' file created, why is that?

"build" runs dpkg-scanpackages after making the package, which makes it possible for "testinstall" to put package-dir in /etc/apt/sources* and run "apt-get install arvados-server-easy" to install pkg+deps.

I guess this is a bit surprising -- is it potentially troublesome or is it just a "should mention in the usage/help text" thing?

• Testing the package failed during package installation:

I haven't reproduced this, but I suspect you have a file somewhere in your work tree that isn't in the git tree, and it's being included in the package and breaking the app at runtime. The stack trace has "eval" so we don't actually get to see where the code comes from, but I'm guessing services/api/config/environments/*.rb, which is ignored by .gitignore. I added that to the rsync ignore list -- give it a try?

• ruby 2.5 will EOL on 2021-03-31. The latest release is 2.5.8. Can we use 2.7.2 here?

Sure. I've been resisting making that change as part of this branch since it seems independent, but OTOH maybe it's as good a place as any. Done in e28e5002e.

16306-arvados-easy-install @ eb0dd856470fe91ba4592ee24c05a5fed11af217 -- developer-run-tests: #2295

• Packages.gz file

I figured as much; I think that dpkg-scanpackages call should happen as part of the "testinstall" command, not as part of the "build" command. Can that be done?

• Looks like some of the package information fields need to be populated properly:

$ dpkg -I arvados-server-easy_2.1.2~rc20210123_amd64.deb
 new Debian package, version 2.0.
 size 223937554 bytes: control archive=589306 bytes.
     484 bytes,    12 lines      control              
 2567069 bytes, 22526 lines      md5sums              
 Package: arvados-server-easy
 Version: 2.1.2~rc20210123
 License: unknown
 Vendor: none
 Architecture: amd64
 Maintainer: <@fe73d6c9f987>
 Installed-Size: 560606
 Depends: ca-certificates, curl, fuse, git, gitolite3, graphviz, haveged, libcurl3-gnutls, libxslt1.1, nginx, python, sudo, python3-distutils, g++, libcurl4-openssl-dev, libpq-dev, libpython2.7, mime-support, zlib1g-dev
 Section: default
 Priority: extra
 Homepage: http://example.com/no-uri-given
 Description: no description given

Please update to be consistent with our other packages:

...
Vendor: The Arvados Project
Maintainer: Arvados Package Maintainers <packaging@arvados.org>
Homepage: https://arvados.org
Description: FIXME
License: GNU Affero General Public License, version 3.0

I'm not sure if the License line is ideal, since we also have parts that are Apache-2.0. We did it this way for the arvados-src package, though.

• Testing the package failed during package installation:

I haven't reproduced this, but I suspect you have a file somewhere in your work tree that isn't in the git tree, and it's being included in the package and breaking the app at runtime. The stack trace has "eval" so we don't actually get to see where the code comes from, but I'm guessing services/api/config/environments/*.rb, which is ignored by .gitignore. I added that to the rsync ignore list -- give it a try?

Yeah, that seems to have fixed it, thanks!

This already gave me the opportunity to test the -package-version flag, it works!

The `testinstall` command worked, and exited 0 after spewing a lot of text to the cli. I assume that means everything is OK, but perhaps we should print a line a the end if everything worked?

• ruby 2.5 will EOL on 2021-03-31. The latest release is 2.5.8. Can we use 2.7.2 here?

Sure. I've been resisting making that change as part of this branch since it seems independent, but OTOH maybe it's as good a place as any. Done in e28e5002e.

16306-arvados-easy-install @ d515c8eb95b39065f74ea6d0b368d8ca3705fd4a -- developer-run-tests: #2294

Thanks!

OK, "build" no longer creates Packages.gz in -package-dir -- testinstall creates it as needed instead. Needed a bit of hoop-jumping to install dpkg-dev in the testinstall container without accidentally relying on its dependencies at runtime, or leaving its dependencies in the cached image (where they would make "autoremove" slow, because removing files from an overlayfs is slow).

Added package metadata:

 new Debian package, version 2.0.
 size 213191242 bytes: control archive=548064 bytes.
     643 bytes,    12 lines      control              
 2222672 bytes, 20120 lines      md5sums              
 Package: arvados-server-easy
 Version: 2.1.2~rc20210202
 License: GNU Affero General Public License, version 3.0
 Vendor: The Arvados Project
 Architecture: amd64
 Maintainer: Arvados Package Maintainers <packaging@arvados.org>
 Installed-Size: 531355
 Depends: ca-certificates, curl, fuse, git, gitolite3, graphviz, haveged, libcurl3-gnutls, libxslt1.1, nginx, python, sudo, python3-distutils, g++, libcurl4-openssl-dev, libpq-dev, libpython2.7, mime-support, zlib1g-dev
 Section: default
 Priority: extra
 Homepage: https://arvados.org
 Description: platform for managing, processing, and sharing genomic and other large scientific and biomedical data

Re license, although AGPL3 isn't the whole story, I think it is technically correct to use it here, because Apache2-licensed code can re-licensed as AGPL3 -- is that right?

16306-arvados-easy-install @ 81d57969ee70d6118937b4f4f61f6e1cd235de44 -- developer-run-tests: #2296

(I made command line options to override "maintainer" and "vendor" so it's possible to repackage/update without making a new fork/branch -- even if that's never actually used, at least it was cheap to add.)

Tom Clegg wrote:

Re license, although AGPL3 isn't the whole story, I think it is technically correct to use it here, because Apache2-licensed code can re-licensed as AGPL3 -- is that right?

Yeah, that seems like the most prudent approach. There may be a way to be more precise but I don't know it. Probably good enough for now.

16306-arvados-easy-install @ 81d57969ee70d6118937b4f4f61f6e1cd235de44 -- developer-run-tests: #2296

that LGTM, the only thing I noticed is that my new package was much larger than the one I built a few days ago:

-rw-r--r--  1 ward ward 333782472 Feb  2 16:33 arvados-server-easy_2.1.2~rc20210201_amd64.deb
-rw-r--r--  1 ward ward 223937554 Jan 29 14:39 arvados-server-easy_2.1.2~rc20210123_amd64.deb

That difference turned out to be because of the switch in Ruby version. The new package uses Ruby 2.7 but it still included `/var/lib/arvados/lib/ruby/gems/2.5.0/` as well.

I rebuilt the image with the `-rebuild-image` flag you suggested:

go run ./cmd/arvados-package build -package-dir /tmp/pkg -package-version 2.1.2~rc20210202 -rebuild-image -source .

And that solved the issue:

-rw-r--r--  1 ward ward 222745314 Feb  2 16:51 arvados-server-easy_2.1.2~rc20210202_amd64.deb

Thanks for adding that 'PASSED' line after the testinstall command.

LGTM!