Project

General

Profile

Actions

Idea #16053

closed

[boot] subcommand to install/update dev and runtime dependencies

Added by Tom Clegg about 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
4.0
Release relationship:
Auto

Description

The dev/test mode should cover everything at Hacking prerequisites

The non-dev/test mode should omit things like Go and phantomjs

Dev mode should install symlinks in /usr/local/bin so the developer can run phantomjs, passenger, keepstore, etc. easily. Conflicting contents of /usr/local/bin (which aren't symlinks to /var/lib/arvados) should be reported as an error.

All modes should install dependencies in /var/lib/arvados; arvados-server boot and run-tests.sh should prefer to run binaries from there. For example, presence of an old version of ruby in /usr/local shouldn't break (or be broken by) anything in Arvados.

[Deferred] Before downloading anything (like ruby 2.5.5 source code), check /var/lib/arvados/archives first. This allows a workaround for packages that aren't available from the usual source, and allows testing without repetitive downloading.

References

arvbox dependency install:

https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/tools/arvbox/lib/arvbox/docker/Dockerfile.base

the jenkins worker image install:

:packer.git/jenkins/run_jenkins-slave-image-run-tests.sh


Subtasks 1 (0 open1 closed)

Task #16099: Review 16053-install-depsResolvedTom Clegg03/31/2020Actions

Related issues

Related to Arvados Epics - Idea #15941: arvados-bootNewActions
Related to Arvados - Idea #16264: Handle R SDK dependencies betterResolvedWard VandewegeActions
Blocks Arvados - Bug #16267: arvbox build uses arvados-server installResolvedWard Vandewege09/24/2020Actions
Actions #1

Updated by Tom Clegg about 4 years ago

Actions #2

Updated by Tom Clegg about 4 years ago

  • Description updated (diff)
Actions #3

Updated by Tom Clegg about 4 years ago

  • Story points set to 4.0
  • Assigned To set to Tom Clegg
Actions #4

Updated by Tom Clegg about 4 years ago

  • Target version set to 2020-02-12 Sprint
Actions #5

Updated by Tom Clegg about 4 years ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz about 4 years ago

  • Description updated (diff)
Actions #7

Updated by Peter Amstutz about 4 years ago

  • Description updated (diff)
Actions #8

Updated by Tom Clegg about 4 years ago

  • Target version changed from 2020-02-12 Sprint to 2020-02-26 Sprint
Actions #9

Updated by Peter Amstutz about 4 years ago

  • Target version changed from 2020-02-26 Sprint to 2020-03-11 Sprint
Actions #10

Updated by Peter Amstutz about 4 years ago

  • Target version changed from 2020-03-11 Sprint to 2020-03-25 Sprint
Actions #11

Updated by Tom Clegg almost 4 years ago

  • Status changed from New to In Progress
Actions #12

Updated by Tom Clegg almost 4 years ago

16053-install-deps @ 0344940255604764cb37cf9ea248b538cce04ff4 -- developer-run-tests: #1775

This version should take care of dev/test dependencies mentioned at Hacking prerequisites. For example:
  • Build arvados-server binary ("go build -o /tmp ./cmd/arvados-server")
  • Bind mount it into a docker container ("docker run -it -v /tmp/arvados-server:/bin/arvados-server:ro debian:10")
  • Run the install subcommand ("arvados-server install -type test")
  • Commit the docker container
  • The resulting image can check out (or bind-mount) an arvados source tree, and run ("arvados-server boot -type test -own-temporary-database -source /path/to/arvados -config /path/to/arvados/doc/examples/config/zzzzz.yml")

There's a test along these lines, but it's disabled by default. You can run it with

go test -tags docker ./lib/install

The test downloads a lot, so it's good to use a proxy. If you set http_proxy and https_proxy env vars, they will be propagated to the docker container.

Once merged, this should reduce the "Hacking prerequisites" page to:

# (start with debian buster)
apt-get install --no-install-recommends build-essential ca-certificates git golang
go run git.arvados.org/arvados.git/cmd/arvados-server install -type test

...but for now, you need to insert extra steps to checkout this branch.

apt-get install --no-install-recommends build-essential ca-certificates git golang
go get -d -v git.arvados.org/arvados.git
(
  export GO111MODULE=on
  cd ~/go/src/git.arvados.org/arvados.git/cmd/arvados-server
  git checkout origin/16053-install-deps
  go mod download
  go install
)
~/go/bin/arvados-server install -type test

You can also build the arvados-server binary elsewhere, copy it to the target machine, and run:

/path/to/arvados-server install -type test

There's also a script that nearly makes it practical to use run-tests.sh in a docker container, without having to install stuff on your dev machine at all. Currently it operates on a copy of your working tree instead of a read-only mount, though, so it's not quite conducive to edit-and-test cycles. Next we need to clean up the remaining things in run-tests.sh (and test suites) that write files/dirs in the source tree.

# create a docker image with build deps:
./lib/install/arvadostest_docker_build.sh

# update the docker image with latest deps:
./lib/install/arvadostest_docker_build.sh --update

# run tests:
./lib/install/arvadostest_docker_run.sh --interactive
<pre>
Actions #13

Updated by Peter Amstutz almost 4 years ago

  • Blocks Bug #16267: arvbox build uses arvados-server install added
Actions #14

Updated by Peter Amstutz almost 4 years ago

  • Related to Idea #16264: Handle R SDK dependencies better added
Actions #15

Updated by Peter Amstutz almost 4 years ago

  • Target version changed from 2020-03-25 Sprint to 2020-04-08 Sprint
Actions #16

Updated by Lucas Di Pentima almost 4 years ago

After trying many times, running the test keep failing for me, even if I set a large timeout value. The log messages that I think are giving clues about what's the problem are:

$ time go test -timeout 40m -tags docker ./lib/install

[... snip! ...]

[controller] {"PID":27041,"RequestID":"req-8u1gpz93nqcl13impilx","level":"info","msg":"request","remoteAddr":"127.0.0.1:44016","reqBytes":0,"reqForwardedFor":"127.0.0.1","reqHost":"localhost:40115","reqMethod":"GET","reqPath":"discovery/v1/apis/arvados/v1/rest","reqQuery":"","time":"2020-04-03T13:55:33.346575442Z"}
[apps/workbench: bundle] Fetching gem metadata from https://rubygems.org/..[services/api: passenger] [ N 2020-04-03 13:55:35.5422 8775/T5 age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
.[-l] 2020-04-03 13:55:36.484 UTC [8914] ERROR:  database "arvados_test" already exists
[-l] 2020-04-03 13:55:36.484 UTC [8914] STATEMENT:  CREATE DATABASE "arvados_test" ENCODING = 'utf8' TEMPLATE = "template0" LC_COLLATE = 'en_US.utf8'
[services/api: rake] Database 'arvados_test' already exists
...[services/api: rake] rake aborted!
[services/api: rake] Errno::EROFS: Read-only file system @ apply2files - /arvados/services/api/tmp/cache/A4B/050/arvados_v1_rest_discovery
[services/api: rake] /root/.gem/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/cache/file_store.rb:90:in `delete

[... snip! ...]

----------------------------------------------------------------------
FAIL: deps_test.go:28: Suite.TestInstallDeps

deps_test.go:45:
    c.Check(runBash(script, os.Stdout, os.Stderr), check.IsNil)
... value *exec.ExitError = &exec.ExitError{ProcessState:(*os.ProcessState)(0xc0000ac3c0), Stderr:[]uint8(nil)} ("exit status 1")

OOPS: 0 passed, 1 FAILED
--- FAIL: Test (802.11s)
FAIL
FAIL    git.arvados.org/arvados.git/lib/install    802.129s
FAIL

real    13m22.914s
user    0m5.988s
sys    0m1.641s

I've set up a SQUID webproxy cache to speed up my attempts and I only was able to save ~660MB of downloads on each run (mostly deb packages), because the majority of things (github, gems, etc) are downloaded via HTTPS. I've researched a little about how to set up SQUID to cache tunneled requests, and it's doable, but I don't think is practical for us. It's a shame because making this test faster would allow us to run it on every commit.

In summary: All manual tests described above worked fine except for the manual test run.

Actions #17

Updated by Tom Clegg almost 4 years ago

"Database 'arvados_test' already exists" is normal from rake db:setup

"Errno::EROFS: Read-only file system @ apply2files" is from rails trying to write cache/tmp files in the RO-mounted source tree. I didn't get this error because my services/api/tmp dir is owned by non-root. Fix:

16053-install-deps @ 1dcef97d54da6b5fa88adac30c474b430b746b15 -- developer-run-tests: #1802

Actions #18

Updated by Lucas Di Pentima almost 4 years ago

It now worked OK for me, took about 15 minutes on a ~50 Mbps Internet connection. LGTM, thanks!

Actions #19

Updated by Tom Clegg almost 4 years ago

  • Description updated (diff)
Actions #20

Updated by Tom Clegg almost 4 years ago

  • Description updated (diff)
Actions #21

Updated by Tom Clegg almost 4 years ago

  • Status changed from In Progress to Resolved
Actions #22

Updated by Peter Amstutz over 3 years ago

  • Release set to 25
Actions

Also available in: Atom PDF