Idea #20735
closedUpdate Go, Ruby, singularity, etc.
Updated by Tom Clegg about 1 year ago
Currently the docker client library is incompatible with Go 1.19.11 and 1.20.6 (https://github.com/moby/moby/issues/45935 makes crunch-run tests fail) but seems to work fine with Go 1.20.5.
20735-update-deps @ 2ecf666bae35ad8f49a0e3d66c4cc90413073fdc -- developer-run-tests: #3739
Updated by Peter Amstutz about 1 year ago
- Target version changed from To be scheduled to Development 2023-08-16
Updated by Tom Clegg about 1 year ago
20735-update-deps @ 49c153fed482e6911333d984ce88bdec1a426396 -- developer-run-tests: #3755
Updated by Brett Smith about 1 year ago
I would personally appreciate if this branch could also update the version pins in the Dockerfiles and supporting Makefiles under build
. This is another place where version numbers are hardcoded and when things get out of sync it's easy to wonder why after the fact. See, e.g., 71a4ceafa05a66b7f90f582db414b7ad68ff7d54.
These files pin versions for Go, Node, Ruby, RVM, and Bundler at least. Also fpm but that feels less pressing to me since it doesn't get installed on user systems at all.
Updated by Tom Clegg about 1 year ago
Brett Smith wrote in #note-5:
I would personally appreciate if this branch could also update the version pins in the Dockerfiles and supporting Makefiles under
build
. This is another place where version numbers are hardcoded and when things get out of sync it's easy to wonder why after the fact. See, e.g., 71a4ceafa05a66b7f90f582db414b7ad68ff7d54.
The build scripts get the current Go version from source:lib/install/deps.go.
source:tools/arvbox/lib/arvbox/docker/common.sh says "RUBY_VERSION=2.7.0" but afaict this is not actually used any more now that arvbox uses arvados-server install
.
The package-build/test dockerfiles say "ruby-2.7" so rvm decides whether we build/test with 2.7.5 or 2.7.7.
So I think this branch already has what you're looking for.
On that note, 71a4ceafa05a66b7f90f582db414b7ad68ff7d54 looks like an error. Before that, all of the (many) places where we specify a bundler version to use/install said 2.2.19. But evidently there weren't enough such places, because at some point someone used bundle 2.3.26 to update Gemfile.lock, and committed the resulting file with "BUNDLED WITH 2.3.26", which tricked you into using 2.3.26 for the rocky8 build/test images while everything else stayed at 2.2.19.
I tried updating bundler to current (2.4.18) and adding a BUNDLERVERSION
build arg similar to the GOVERSION
build arg so we could change that in one place next time, but it turned into a big diff and jenkins failed, so I'm inclined to step back, review/merge the pending updates on the working branch, and then revisit structural improvements. 20735-bundler-version
Updated by Tom Clegg about 1 year ago
20735-update-deps @ 49c153fed482e6911333d984ce88bdec1a426396 -- developer-run-tests: #3760
Updated by Brett Smith about 1 year ago
Tom Clegg wrote in #note-6:
So I think this branch already has what you're looking for.
Thanks.
On that note, 71a4ceafa05a66b7f90f582db414b7ad68ff7d54 looks like an error. Before that, all of the (many) places where we specify a bundler version to use/install said 2.2.19. But evidently there weren't enough such places, because at some point someone used bundle 2.3.26 to update Gemfile.lock, and committed the resulting file with "BUNDLED WITH 2.3.26", which tricked you into using 2.3.26 for the rocky8 build/test images while everything else stayed at 2.2.19.
Bundler specifically issued a two-line warning about "the running version of Bundler is older than Gemfile.lock specifies" every time it ran during the package build process. I figured the other target Dockerfiles also needed updating, but, you know, I also didn't want to roll that in to my already-large branch. This kind of maintenance never seems to be in scope for a ticket.
I tried updating bundler to current (2.4.18) and adding a
BUNDLERVERSION
build arg similar to theGOVERSION
build arg so we could change that in one place next time, but it turned into a big diff and jenkins failed, so I'm inclined to step back, review/merge the pending updates on the working branch, and then revisit structural improvements. 20735-bundler-version
+1, I agree since you're not doing anything with Bundler directly in this branch you shouldn't feel obligated to deal with this now either. Thanks for looking at it.
Updated by Brett Smith about 1 year ago
Ruby 2.7 cannot be built against OpenSSL 3.0, so that creates pressure to upgrade Ruby to 3.0+.
Updated by Tom Clegg about 1 year ago
Moved source:services/api/Gemfile.lock and the rocky8 build/test Dockerfiles to say 2.2.19 like everything else.
20735-update-deps @ 3dd5dc39765ef59d2ea6101b80975133ca628af9 -- developer-run-tests: #3762
Updated by Lucas Di Pentima about 1 year ago
Some comments:
- There's one non wb1 test failure: developer-run-tests-remainder: #3951
- We also compile singularity on our compute image build scripts at
arvados/tools/compute-images/scripts/base.sh:L123
- Singularity 3.9.9 is mentioned at the bottom of this doc page: https://doc.arvados.org/main/architecture/singularity.html
Updated by Tom Clegg about 1 year ago
Pretty sure the test failure is the same one addressed in #20705 (9cc06669c). Merged main into branch.
Updated singularity version in compute image build script and doc page. On our set up compute node with singularity doc page, changed the link to singularity docs to link from "3.9" to "latest" (which is already 3.11) instead of "3.10".
That install page says
You can download SingularityCE from one of the releases. To see a full list, visit the GitHub release page. After deciding on a release to install, [...]
I think the version story might make a bit more sense if we edit our install-compute-node-singularity.html something like this, adding the bold bit:
Follow the Singularity installation instructions. Note that while the latest stable version is normally expected to be compatible, Arvados is currently tested with singularity 3.10.4. Make sure singularity and mksquashfs are working:
WDYT -- does that make sense, or should we just tell people to install 3.10.4?
20735-update-deps @ aa6ca6f50268f29c7cce987f5957796826bfeeed -- developer-run-tests: #3764
Updated by Lucas Di Pentima about 1 year ago
Tom Clegg wrote in #note-13:
That install page says
You can download SingularityCE from one of the releases. To see a full list, visit the GitHub release page. After deciding on a release to install, [...]
I think the version story might make a bit more sense if we edit our install-compute-node-singularity.html something like this, adding the bold bit:
Follow the Singularity installation instructions. Note that while the latest stable version is normally expected to be compatible, Arvados is currently tested with singularity 3.10.4. Make sure singularity and mksquashfs are working:
WDYT -- does that make sense, or should we just tell people to install 3.10.4?
I think it makes sense to clarify which is the latest version we tested and confirm it works and give them freedom to try another if they want.
In version 2.4.1 we added an upgrade note that mentioned the new 3.9.9 recommendation, do we need something similar this time? Asking because this time we're not fixing any bug by upgrading, so I think we may not mention the change if it doesn't break anything.
The rest LGTM.
Updated by Peter Amstutz about 1 year ago
For the record, I ran into a couple of missing dev packages required to build the new singularity from scratch, added libseccomp-dev
and libglib2.0-dev
in a6c53fb8dc1e29f5e386567ad3439e75fb65f2a2 .
Updated by Tom Clegg about 1 year ago
- Status changed from In Progress to Resolved