Project

General

Profile

Actions

Idea #20735

closed

Update Go, Ruby, singularity, etc.

Added by Tom Clegg 11 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
08/04/2023
Due date:
Story points:
1.0
Release relationship:
Auto

Subtasks 1 (0 open1 closed)

Task #20813: Review 20735-update-depsResolvedTom Clegg08/04/2023Actions
Actions #1

Updated by Tom Clegg 11 months ago

  • Status changed from New to In Progress
Actions #2

Updated by Tom Clegg 11 months ago

Currently the docker client library is incompatible with Go 1.19.11 and 1.20.6 (https://github.com/moby/moby/issues/45935 makes crunch-run tests fail) but seems to work fine with Go 1.20.5.

20735-update-deps @ 2ecf666bae35ad8f49a0e3d66c4cc90413073fdc -- developer-run-tests: #3739

Actions #3

Updated by Peter Amstutz 10 months ago

  • Target version changed from To be scheduled to Development 2023-08-16
Actions #5

Updated by Brett Smith 10 months ago

I would personally appreciate if this branch could also update the version pins in the Dockerfiles and supporting Makefiles under build. This is another place where version numbers are hardcoded and when things get out of sync it's easy to wonder why after the fact. See, e.g., 71a4ceafa05a66b7f90f582db414b7ad68ff7d54.

These files pin versions for Go, Node, Ruby, RVM, and Bundler at least. Also fpm but that feels less pressing to me since it doesn't get installed on user systems at all.

Actions #6

Updated by Tom Clegg 10 months ago

Brett Smith wrote in #note-5:

I would personally appreciate if this branch could also update the version pins in the Dockerfiles and supporting Makefiles under build. This is another place where version numbers are hardcoded and when things get out of sync it's easy to wonder why after the fact. See, e.g., 71a4ceafa05a66b7f90f582db414b7ad68ff7d54.

The build scripts get the current Go version from source:lib/install/deps.go.

source:tools/arvbox/lib/arvbox/docker/common.sh says "RUBY_VERSION=2.7.0" but afaict this is not actually used any more now that arvbox uses arvados-server install.

The package-build/test dockerfiles say "ruby-2.7" so rvm decides whether we build/test with 2.7.5 or 2.7.7.

So I think this branch already has what you're looking for.

On that note, 71a4ceafa05a66b7f90f582db414b7ad68ff7d54 looks like an error. Before that, all of the (many) places where we specify a bundler version to use/install said 2.2.19. But evidently there weren't enough such places, because at some point someone used bundle 2.3.26 to update Gemfile.lock, and committed the resulting file with "BUNDLED WITH 2.3.26", which tricked you into using 2.3.26 for the rocky8 build/test images while everything else stayed at 2.2.19.

I tried updating bundler to current (2.4.18) and adding a BUNDLERVERSION build arg similar to the GOVERSION build arg so we could change that in one place next time, but it turned into a big diff and jenkins failed, so I'm inclined to step back, review/merge the pending updates on the working branch, and then revisit structural improvements. 20735-bundler-version

Actions #8

Updated by Brett Smith 10 months ago

Tom Clegg wrote in #note-6:

So I think this branch already has what you're looking for.

Thanks.

On that note, 71a4ceafa05a66b7f90f582db414b7ad68ff7d54 looks like an error. Before that, all of the (many) places where we specify a bundler version to use/install said 2.2.19. But evidently there weren't enough such places, because at some point someone used bundle 2.3.26 to update Gemfile.lock, and committed the resulting file with "BUNDLED WITH 2.3.26", which tricked you into using 2.3.26 for the rocky8 build/test images while everything else stayed at 2.2.19.

Bundler specifically issued a two-line warning about "the running version of Bundler is older than Gemfile.lock specifies" every time it ran during the package build process. I figured the other target Dockerfiles also needed updating, but, you know, I also didn't want to roll that in to my already-large branch. This kind of maintenance never seems to be in scope for a ticket.

I tried updating bundler to current (2.4.18) and adding a BUNDLERVERSION build arg similar to the GOVERSION build arg so we could change that in one place next time, but it turned into a big diff and jenkins failed, so I'm inclined to step back, review/merge the pending updates on the working branch, and then revisit structural improvements. 20735-bundler-version

+1, I agree since you're not doing anything with Bundler directly in this branch you shouldn't feel obligated to deal with this now either. Thanks for looking at it.

Actions #9

Updated by Brett Smith 10 months ago

Ruby 2.7 cannot be built against OpenSSL 3.0, so that creates pressure to upgrade Ruby to 3.0+.

Actions #11

Updated by Tom Clegg 10 months ago

Moved source:services/api/Gemfile.lock and the rocky8 build/test Dockerfiles to say 2.2.19 like everything else.

20735-update-deps @ 3dd5dc39765ef59d2ea6101b80975133ca628af9 -- developer-run-tests: #3762

Actions #12

Updated by Lucas Di Pentima 10 months ago

Some comments:

Actions #13

Updated by Tom Clegg 10 months ago

Pretty sure the test failure is the same one addressed in #20705 (9cc06669c). Merged main into branch.

Updated singularity version in compute image build script and doc page. On our set up compute node with singularity doc page, changed the link to singularity docs to link from "3.9" to "latest" (which is already 3.11) instead of "3.10".

That install page says

You can download SingularityCE from one of the releases. To see a full list, visit the GitHub release page. After deciding on a release to install, [...]

I think the version story might make a bit more sense if we edit our install-compute-node-singularity.html something like this, adding the bold bit:

Follow the Singularity installation instructions. Note that while the latest stable version is normally expected to be compatible, Arvados is currently tested with singularity 3.10.4. Make sure singularity and mksquashfs are working:

WDYT -- does that make sense, or should we just tell people to install 3.10.4?

20735-update-deps @ aa6ca6f50268f29c7cce987f5957796826bfeeed -- developer-run-tests: #3764

Actions #14

Updated by Lucas Di Pentima 10 months ago

Tom Clegg wrote in #note-13:

That install page says

You can download SingularityCE from one of the releases. To see a full list, visit the GitHub release page. After deciding on a release to install, [...]

I think the version story might make a bit more sense if we edit our install-compute-node-singularity.html something like this, adding the bold bit:

Follow the Singularity installation instructions. Note that while the latest stable version is normally expected to be compatible, Arvados is currently tested with singularity 3.10.4. Make sure singularity and mksquashfs are working:

WDYT -- does that make sense, or should we just tell people to install 3.10.4?

I think it makes sense to clarify which is the latest version we tested and confirm it works and give them freedom to try another if they want.

In version 2.4.1 we added an upgrade note that mentioned the new 3.9.9 recommendation, do we need something similar this time? Asking because this time we're not fixing any bug by upgrading, so I think we may not mention the change if it doesn't break anything.

The rest LGTM.

Actions #15

Updated by Peter Amstutz 10 months ago

For the record, I ran into a couple of missing dev packages required to build the new singularity from scratch, added libseccomp-dev and libglib2.0-dev in a6c53fb8dc1e29f5e386567ad3439e75fb65f2a2 .

Actions #16

Updated by Tom Clegg 10 months ago

  • Status changed from In Progress to Resolved
Actions #17

Updated by Peter Amstutz 9 months ago

  • Release set to 66
Actions

Also available in: Atom PDF