Arvados

18700-boot-wb2 @ 343cedddf4364c861b268da6e0a0082016ceb79b --

18700-boot-wb2 @ 8257b9e9049a2592c9858941775a11b5a98ec1f7 --

1. Get a public DNS name for your host, and use acmetool or something similar to get a key & certificate for that name in /var/lib/acme/live/$host/privkey and .../cert (also works with "tailscale cert")
2. In arvados source tree, check out the desired commit, then: go run ./cmd/arvados-package build -target-os=debian:11
   • check -help for options like version number and package destination directory
   • this leaves behind an "arvados-package-build-debian" docker image which will make the "build" command much faster next time
   • in a CI situation it's a good idea to run -rebuild-image periodically (otherwise, installing newly introduced dependencies will get slower over time)
3. When package is built, use it to bring up a test cluster: go run ./cmd/arvados-package testinstall -target-os=debian:11 -live=host.example.com
   • Browse https://host.example.com:4442

• this installs the package, runs "arvados-server init" to create a config file, and runs "arvados-server boot -shutdown" to wait for all services to come up healthy and then exit 0
• this leaves behind an "arvados-package-deps-debian" docker image which will make the "testinstall" command much faster next time
• above -rebuild-image comment applies here too

• default to ./build/version-at-commit.sh instead of git-describe if no -version specified when building package
• arvados-server init uses ports 43xx-44xx for external URLs, to make it easier to expose/forward those ports through docker etc
• -live=host.example.com option invokes the existing arvados-server init -domain=host.example.com option (tell clients that the external URLs are host.example.com:4xxx)
• fix iceweasel/firefox issue so arvados-server install works on debian:11 bullseye
• in prod/pkg mode, arvados-server install builds a workbench2 release and puts it in /var/lib/arvados/workbench2 where the arvados-server boot nginx config will find it
• if /var/lib/acme/live/$controllerExternalURLHost/privkey and .../cert exist, arvados-server boot uses those for the external (Nginx) endpoints
• fix nginx -g options that were being ignored (if we pass more than one, all but the last are ignored) and removed the associated workarounds (like repeating "error_log /dev/stderr" in nginx.conf because it was being ignored on the command line)
• const workbench2version in lib/install/deps.go determines the version of workbench2 that gets built for the package; it currently points to an unmerged "update to yarn3" branch
• arvados-server install doesn't install nodejs in production mode
• arvados-server init -login=test enables test login (admin/admin), -login=pam uses pam (Google/OIDC need more config details and are less relevant when testing, so not as well suited to specifying on the command line)
• health aggregator obeys TLS.Insecure setting when pinging other services
• health aggregator checks ExternalURL if a service has an ExternalURL but no InternalURLs (like workbench2)

18700-boot-wb2 @ 78b4cdb54bd58df85d61d72e639a728ec38bb377 -- developer-run-tests: #2966
retry wb1 developer-run-tests-apps-workbench-integration: #3170

developer-run-tests: #2983

• Merge main
• Omit test/development gems from package (--without "development test diagnostics performance") -- this seems like a change we can/should make in source:build/package-build-dockerfiles/debian11/Dockerfile etc. too
• Fix ExternalURL resolution in testinstall docker container
• Fix WebDAV and websocket URLs in "init" config template
• Enable TrustAllContent in testinstall config (so login→create collection→upload files→preview works in -live mode)
• Add systemd unit file

developer-run-tests: #3000

• update `arvados-server install` dev dependencies for debian:11 (added #18942 to remove perl sdk)
• discard local changes to yarn.lock when checking out wb2 commit

I tried this:

if you have /var/lib/acme/live/X where X is a dns name for this machine, and you have docker, then

go run ./cmd/arvados-package build -target-os=debian:11
go run ./cmd/arvados-package testinstall -target-os=debian:11 -live=X

should bring everything up and https://X:4442 should give you a working workbench that lets you upload files etc.

That did indeed work, nice! A few things:

• wb2 at port :4442 by default seems weird. Why not default to 443?
• what exactly is the external port range that is used by default? A comment above mentiones 43xx-44xx which seems very big. Can we trim this to the necessary range (only)? It should be documented. In the single node salt installer, we default to wb2 on 443 and everything else on 8800-8805. Can we do the same here?
• despite having /var/lib/acme/live/X, wb2 came up with a 10 year self-signed cert. The directory was correctly loaded into the docker container, afaict:

ls -aF /var/lib/acme/live/wardtest.arvados.org/
./  ../  account@  cert  chain    fullchain  privkey@  url

But the config file had

    TLS:
      Insecure: true

• looks like wb1 is broken, I got a "We're sorry, but something went wrong." message, and the backtrace indicates that perhaps an asset generation step was not run:

App 1056 output: #<ActionView::Template::Error: couldn't find file 'awesomplete' with type 'text/css'
App 1056 output: Checked in these paths:
App 1056 output:   /var/lib/arvados/workbench1/app/assets/images
...

• wb2 does not realize it's running in insecure mode, the token blurb does not set the right env var:

HISTIGNORE=$HISTIGNORE:'export ARVADOS_API_TOKEN=*'
export ARVADOS_API_TOKEN=v2/x1234-gj3su-redacted/redacted
export ARVADOS_API_HOST=wardtest.arvados.org:4430
unset ARVADOS_API_HOST_INSECURE

Is this a wb2 bug that should be filed separately?

Ward Vandewege wrote:

• wb2 at port :4442 by default seems weird. Why not default to 443?

I was trying to keep everything in one range, and avoid privileged ports and ports that are often in use, so the same config could work without root (like dev mode).

Could change wb2 to 443 though.

• what exactly is the external port range that is used by default? A comment above mentiones 43xx-44xx which seems very big. Can we trim this to the necessary range (only)? It should be documented. In the single node salt installer, we default to wb2 on 443 and everything else on 8800-8805. Can we do the same here?

Oops, that should have said 443x-444x. Yes, I can change to 880x (we currently use 9 ports).

• despite having /var/lib/acme/live/X, wb2 came up with a 10 year self-signed cert. The directory was correctly loaded into the docker container, afaict:

Hm, this sounds like a bug. Need to investigate / improve error reporting.

• looks like wb1 is broken, I got a "We're sorry, but something went wrong." message, and the backtrace indicates that perhaps an asset generation step was not run:

This also sounds like a bug, I think it should work.

• wb2 does not realize it's running in insecure mode, the token blurb does not set the right env var:

Is this a wb2 bug that should be filed separately?

That sounds like a wb2 shortcoming yeah.

developer-run-tests: #3031

• mount the whole /var/lib/acme dir, not just the desired live/$domain dir (because live/$domain/privkey is a symlink to elsewhere in /var/lib/acme) -- should fix the "still using snakeoil" issue
• include TLS.Insecure flag in exported config, so wb2 can use it in the host/token env var script (although wb2 still doesn't actually use it yet)
• fix missing wb1 assets (npm:install + assets:precompile)
• fix max keepproxy request body size in nginx config (was preventing wb1 upload from working)
• renumber default/example ports to 9000-9020 (internal) and 4440-4460 (external)
• fix "arvados-server install" broken on debian:10 (perl-modules-5.32 not found)

developer-run-tests: #3033

developer-run-tests-apps-workbench-integration: #3240

• merge main
• change default wb2 port to 443

Tom Clegg wrote:

18700-boot-wb2 @ 13116cc7dc549e93b0757f1948c2ac18b760c681 -- developer-run-tests: #3031

• mount the whole /var/lib/acme dir, not just the desired live/$domain dir (because live/$domain/privkey is a symlink to elsewhere in /var/lib/acme) -- should fix the "still using snakeoil" issue
• include TLS.Insecure flag in exported config, so wb2 can use it in the host/token env var script (although wb2 still doesn't actually use it yet)
• fix missing wb1 assets (npm:install + assets:precompile)
• fix max keepproxy request body size in nginx config (was preventing wb1 upload from working)
• renumber default/example ports to 9000-9020 (internal) and 4440-4460 (external)
• fix "arvados-server install" broken on debian:10 (perl-modules-5.32 not found)

18700-boot-wb2 @ b53513423ab948804425424278ac554870864997 -- developer-run-tests: #3033 wb2 retry developer-run-tests-apps-workbench-integration: #3240

• merge main
• change default wb2 port to 443

I'm still seeing wb2 at 4443, but otherwise LGTM thanks!

Oops, I didn't actually push that last commit with the change to 443.

18700-boot-wb2 @ 66a90e37d9dbc9a6526a7de2a2d0b286e8e6f87f -- developer-run-tests: #3034