Story #16053

[boot] subcommand to install/update dev and runtime dependencies

Added by Tom Clegg 4 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
03/31/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
4.0

Description

The dev/test mode should cover everything at Hacking prerequisites

The non-dev/test mode should omit things like Go and phantomjs

Dev mode should install symlinks in /usr/local/bin so the developer can run phantomjs, passenger, keepstore, etc. easily. Conflicting contents of /usr/local/bin (which aren't symlinks to /var/lib/arvados) should be reported as an error.

All modes should install dependencies in /var/lib/arvados; arvados-server boot and run-tests.sh should prefer to run binaries from there. For example, presence of an old version of ruby in /usr/local shouldn't break (or be broken by) anything in Arvados.

[Deferred] Before downloading anything (like ruby 2.5.5 source code), check /var/lib/arvados/archives first. This allows a workaround for packages that aren't available from the usual source, and allows testing without repetitive downloading.

References

arvbox dependency install:

https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/tools/arvbox/lib/arvbox/docker/Dockerfile.base

the jenkins worker image install:

:packer.git/jenkins/run_jenkins-slave-image-run-tests.sh


Subtasks

Task #16099: Review 16053-install-depsResolvedTom Clegg


Related issues

Related to Arvados Epics - Story #15941: arvados-bootIn Progress01/15/202006/17/2020

Related to Arvados - Story #16264: Handle R SDK dependencies betterNew

Blocks Arvados - Bug #16267: arvbox build uses arvados-server installIn Progress

Associated revisions

Revision c2cba515
Added by Tom Clegg about 2 months ago

Merge branch '16053-install-deps'

refs #16053

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Tom Clegg 4 months ago

#2 Updated by Tom Clegg 4 months ago

  • Description updated (diff)

#3 Updated by Tom Clegg 4 months ago

  • Story points set to 4.0
  • Assigned To set to Tom Clegg

#4 Updated by Tom Clegg 4 months ago

  • Target version set to 2020-02-12 Sprint

#5 Updated by Tom Clegg 4 months ago

  • Description updated (diff)

#6 Updated by Peter Amstutz 4 months ago

  • Description updated (diff)

#7 Updated by Peter Amstutz 4 months ago

  • Description updated (diff)

#8 Updated by Tom Clegg 4 months ago

  • Target version changed from 2020-02-12 Sprint to 2020-02-26 Sprint

#9 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-02-26 Sprint to 2020-03-11 Sprint

#10 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-03-11 Sprint to 2020-03-25 Sprint

#11 Updated by Tom Clegg 3 months ago

  • Status changed from New to In Progress

#12 Updated by Tom Clegg 2 months ago

16053-install-deps @ 0344940255604764cb37cf9ea248b538cce04ff4 -- https://ci.arvados.org/view/Developer/job/developer-run-tests/1775/

This version should take care of dev/test dependencies mentioned at Hacking prerequisites. For example:
  • Build arvados-server binary ("go build -o /tmp ./cmd/arvados-server")
  • Bind mount it into a docker container ("docker run -it -v /tmp/arvados-server:/bin/arvados-server:ro debian:10")
  • Run the install subcommand ("arvados-server install -type test")
  • Commit the docker container
  • The resulting image can check out (or bind-mount) an arvados source tree, and run ("arvados-server boot -type test -own-temporary-database -source /path/to/arvados -config /path/to/arvados/doc/examples/config/zzzzz.yml")

There's a test along these lines, but it's disabled by default. You can run it with

go test -tags docker ./lib/install

The test downloads a lot, so it's good to use a proxy. If you set http_proxy and https_proxy env vars, they will be propagated to the docker container.

Once merged, this should reduce the "Hacking prerequisites" page to:

# (start with debian buster)
apt-get install --no-install-recommends build-essential ca-certificates git golang
go run git.arvados.org/arvados.git/cmd/arvados-server install -type test

...but for now, you need to insert extra steps to checkout this branch.

apt-get install --no-install-recommends build-essential ca-certificates git golang
go get -d -v git.arvados.org/arvados.git
(
  export GO111MODULE=on
  cd ~/go/src/git.arvados.org/arvados.git/cmd/arvados-server
  git checkout origin/16053-install-deps
  go mod download
  go install
)
~/go/bin/arvados-server install -type test

You can also build the arvados-server binary elsewhere, copy it to the target machine, and run:

/path/to/arvados-server install -type test

There's also a script that nearly makes it practical to use run-tests.sh in a docker container, without having to install stuff on your dev machine at all. Currently it operates on a copy of your working tree instead of a read-only mount, though, so it's not quite conducive to edit-and-test cycles. Next we need to clean up the remaining things in run-tests.sh (and test suites) that write files/dirs in the source tree.

# create a docker image with build deps:
./lib/install/arvadostest_docker_build.sh

# update the docker image with latest deps:
./lib/install/arvadostest_docker_build.sh --update

# run tests:
./lib/install/arvadostest_docker_run.sh --interactive
<pre>

#13 Updated by Peter Amstutz 2 months ago

  • Blocks Bug #16267: arvbox build uses arvados-server install added

#14 Updated by Peter Amstutz 2 months ago

  • Related to Story #16264: Handle R SDK dependencies better added

#15 Updated by Peter Amstutz 2 months ago

  • Target version changed from 2020-03-25 Sprint to 2020-04-08 Sprint

#16 Updated by Lucas Di Pentima about 2 months ago

After trying many times, running the test keep failing for me, even if I set a large timeout value. The log messages that I think are giving clues about what's the problem are:

$ time go test -timeout 40m -tags docker ./lib/install

[... snip! ...]

[controller] {"PID":27041,"RequestID":"req-8u1gpz93nqcl13impilx","level":"info","msg":"request","remoteAddr":"127.0.0.1:44016","reqBytes":0,"reqForwardedFor":"127.0.0.1","reqHost":"localhost:40115","reqMethod":"GET","reqPath":"discovery/v1/apis/arvados/v1/rest","reqQuery":"","time":"2020-04-03T13:55:33.346575442Z"}
[apps/workbench: bundle] Fetching gem metadata from https://rubygems.org/..[services/api: passenger] [ N 2020-04-03 13:55:35.5422 8775/T5 age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
.[-l] 2020-04-03 13:55:36.484 UTC [8914] ERROR:  database "arvados_test" already exists
[-l] 2020-04-03 13:55:36.484 UTC [8914] STATEMENT:  CREATE DATABASE "arvados_test" ENCODING = 'utf8' TEMPLATE = "template0" LC_COLLATE = 'en_US.utf8'
[services/api: rake] Database 'arvados_test' already exists
...[services/api: rake] rake aborted!
[services/api: rake] Errno::EROFS: Read-only file system @ apply2files - /arvados/services/api/tmp/cache/A4B/050/arvados_v1_rest_discovery
[services/api: rake] /root/.gem/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/cache/file_store.rb:90:in `delete

[... snip! ...]

----------------------------------------------------------------------
FAIL: deps_test.go:28: Suite.TestInstallDeps

deps_test.go:45:
    c.Check(runBash(script, os.Stdout, os.Stderr), check.IsNil)
... value *exec.ExitError = &exec.ExitError{ProcessState:(*os.ProcessState)(0xc0000ac3c0), Stderr:[]uint8(nil)} ("exit status 1")

OOPS: 0 passed, 1 FAILED
--- FAIL: Test (802.11s)
FAIL
FAIL    git.arvados.org/arvados.git/lib/install    802.129s
FAIL

real    13m22.914s
user    0m5.988s
sys    0m1.641s

I've set up a SQUID webproxy cache to speed up my attempts and I only was able to save ~660MB of downloads on each run (mostly deb packages), because the majority of things (github, gems, etc) are downloaded via HTTPS. I've researched a little about how to set up SQUID to cache tunneled requests, and it's doable, but I don't think is practical for us. It's a shame because making this test faster would allow us to run it on every commit.

In summary: All manual tests described above worked fine except for the manual test run.

#17 Updated by Tom Clegg about 2 months ago

"Database 'arvados_test' already exists" is normal from rake db:setup

"Errno::EROFS: Read-only file system @ apply2files" is from rails trying to write cache/tmp files in the RO-mounted source tree. I didn't get this error because my services/api/tmp dir is owned by non-root. Fix:

16053-install-deps @ 1dcef97d54da6b5fa88adac30c474b430b746b15 -- https://ci.arvados.org/view/Developer/job/developer-run-tests/1802/

#18 Updated by Lucas Di Pentima about 2 months ago

It now worked OK for me, took about 15 minutes on a ~50 Mbps Internet connection. LGTM, thanks!

#19 Updated by Tom Clegg about 2 months ago

  • Description updated (diff)

#20 Updated by Tom Clegg about 2 months ago

  • Description updated (diff)

#21 Updated by Tom Clegg about 2 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF