Arvados

Currently the docker client library is incompatible with Go 1.19.11 and 1.20.6 (https://github.com/moby/moby/issues/45935 makes crunch-run tests fail) but seems to work fine with Go 1.20.5.

20735-update-deps @ 2ecf666bae35ad8f49a0e3d66c4cc90413073fdc -- developer-run-tests: #3739

20735-update-deps @ 49c153fed482e6911333d984ce88bdec1a426396 -- developer-run-tests: #3755

I would personally appreciate if this branch could also update the version pins in the Dockerfiles and supporting Makefiles under build. This is another place where version numbers are hardcoded and when things get out of sync it's easy to wonder why after the fact. See, e.g., 71a4ceafa05a66b7f90f582db414b7ad68ff7d54.

These files pin versions for Go, Node, Ruby, RVM, and Bundler at least. Also fpm but that feels less pressing to me since it doesn't get installed on user systems at all.

Brett Smith wrote in #note-5:

I would personally appreciate if this branch could also update the version pins in the Dockerfiles and supporting Makefiles under build. This is another place where version numbers are hardcoded and when things get out of sync it's easy to wonder why after the fact. See, e.g., 71a4ceafa05a66b7f90f582db414b7ad68ff7d54.

The build scripts get the current Go version from source:lib/install/deps.go.

source:tools/arvbox/lib/arvbox/docker/common.sh says "RUBY_VERSION=2.7.0" but afaict this is not actually used any more now that arvbox uses arvados-server install.

The package-build/test dockerfiles say "ruby-2.7" so rvm decides whether we build/test with 2.7.5 or 2.7.7.

So I think this branch already has what you're looking for.

On that note, 71a4ceafa05a66b7f90f582db414b7ad68ff7d54 looks like an error. Before that, all of the (many) places where we specify a bundler version to use/install said 2.2.19. But evidently there weren't enough such places, because at some point someone used bundle 2.3.26 to update Gemfile.lock, and committed the resulting file with "BUNDLED WITH 2.3.26", which tricked you into using 2.3.26 for the rocky8 build/test images while everything else stayed at 2.2.19.

I tried updating bundler to current (2.4.18) and adding a BUNDLERVERSION build arg similar to the GOVERSION build arg so we could change that in one place next time, but it turned into a big diff and jenkins failed, so I'm inclined to step back, review/merge the pending updates on the working branch, and then revisit structural improvements. 20735-bundler-version

20735-update-deps @ 49c153fed482e6911333d984ce88bdec1a426396 -- developer-run-tests: #3760

Tom Clegg wrote in #note-6:

So I think this branch already has what you're looking for.

Thanks.

On that note, 71a4ceafa05a66b7f90f582db414b7ad68ff7d54 looks like an error. Before that, all of the (many) places where we specify a bundler version to use/install said 2.2.19. But evidently there weren't enough such places, because at some point someone used bundle 2.3.26 to update Gemfile.lock, and committed the resulting file with "BUNDLED WITH 2.3.26", which tricked you into using 2.3.26 for the rocky8 build/test images while everything else stayed at 2.2.19.

Bundler specifically issued a two-line warning about "the running version of Bundler is older than Gemfile.lock specifies" every time it ran during the package build process. I figured the other target Dockerfiles also needed updating, but, you know, I also didn't want to roll that in to my already-large branch. This kind of maintenance never seems to be in scope for a ticket.

I tried updating bundler to current (2.4.18) and adding a BUNDLERVERSION build arg similar to the GOVERSION build arg so we could change that in one place next time, but it turned into a big diff and jenkins failed, so I'm inclined to step back, review/merge the pending updates on the working branch, and then revisit structural improvements. 20735-bundler-version

+1, I agree since you're not doing anything with Bundler directly in this branch you shouldn't feel obligated to deal with this now either. Thanks for looking at it.

Ruby 2.7 cannot be built against OpenSSL 3.0, so that creates pressure to upgrade Ruby to 3.0+.

Moved source:services/api/Gemfile.lock and the rocky8 build/test Dockerfiles to say 2.2.19 like everything else.

20735-update-deps @ 3dd5dc39765ef59d2ea6101b80975133ca628af9 -- developer-run-tests: #3762

Some comments:

• There's one non wb1 test failure: developer-run-tests-remainder: #3951
• We also compile singularity on our compute image build scripts at arvados/tools/compute-images/scripts/base.sh:L123
• Singularity 3.9.9 is mentioned at the bottom of this doc page: https://doc.arvados.org/main/architecture/singularity.html

Pretty sure the test failure is the same one addressed in #20705 (9cc06669c). Merged main into branch.

Updated singularity version in compute image build script and doc page. On our set up compute node with singularity doc page, changed the link to singularity docs to link from "3.9" to "latest" (which is already 3.11) instead of "3.10".

That install page says

You can download SingularityCE from one of the releases. To see a full list, visit the GitHub release page. After deciding on a release to install, [...]

I think the version story might make a bit more sense if we edit our install-compute-node-singularity.html something like this, adding the bold bit:

Follow the Singularity installation instructions. Note that while the latest stable version is normally expected to be compatible, Arvados is currently tested with singularity 3.10.4. Make sure singularity and mksquashfs are working:

WDYT -- does that make sense, or should we just tell people to install 3.10.4?

20735-update-deps @ aa6ca6f50268f29c7cce987f5957796826bfeeed -- developer-run-tests: #3764

Tom Clegg wrote in #note-13:

That install page says

You can download SingularityCE from one of the releases. To see a full list, visit the GitHub release page. After deciding on a release to install, [...]

I think the version story might make a bit more sense if we edit our install-compute-node-singularity.html something like this, adding the bold bit:

Follow the Singularity installation instructions. Note that while the latest stable version is normally expected to be compatible, Arvados is currently tested with singularity 3.10.4. Make sure singularity and mksquashfs are working:

WDYT -- does that make sense, or should we just tell people to install 3.10.4?

I think it makes sense to clarify which is the latest version we tested and confirm it works and give them freedom to try another if they want.

In version 2.4.1 we added an upgrade note that mentioned the new 3.9.9 recommendation, do we need something similar this time? Asking because this time we're not fixing any bug by upgrading, so I think we may not mention the change if it doesn't break anything.

The rest LGTM.

For the record, I ran into a couple of missing dev packages required to build the new singularity from scratch, added libseccomp-dev and libglib2.0-dev in a6c53fb8dc1e29f5e386567ad3439e75fb65f2a2 .