Arvados

22364-arvados-cli-dep-version @ 88e5c2b57177702f6cd4f0e4ceb95ddb1b36c3cb

This takes all the logic that we previously implemented for Python packages in #21601 and ports it to the Ruby gems as well. There are two key bits:

1. We consider all dependencies in Git when determining the Git commit to use for versioning, not just this single gem's source.
2. When we're building a development gem, we specify interdependencies with other development gems using ~> based on the version we're building.

• All agreed upon points are implemented / addressed.
  • Yes
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • N/A
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • Without any arvados gems installed, manually built and installed both these changes. They had the correct versions and interdependencies.
• Documentation has been updated.
  • N/A
• Behaves appropriately at the intended scale (describe intended scale).
  • No change in scale
• Considered backwards and forwards compatibility issues between client and server.
  • N/A
• Follows our coding standards and GUI style guidelines.
  • I honestly don't know, if I broke any rules sure I'll fix it.

This LGTM, thanks.

One kind-of-related issue that I found was that services/api/Gemfile explicitly ask for gem 'arvados', '~> 2.7.0.rc1'. Given that the 2.7.4 release will eventually be incompatible with the current Ruby, I guess we should update it, but not sure if it'll need a bit more thought so it gets automated, or at least mentioned on the release checklist for next time we release a major version.

The branch will fix things going forward but we still need to push a gem with a fix. We have two options:

• Push an arvados 2.8.0 with documentation that it's identical to arvados 3.0.0. This requires less effort but might be more confusing for people who find it.
• Push an arvados-cli 3.0.0.1 that properly depends on arvados==3.0.0. Since this doesn't follow the normal versioning we'll have to push a branch that just writes this gem metadata manually. Not more difficult but a little more effort, but probably clearer long-term.

22364-arvados-cli-3.0-bugfix @ d3107471cfe5275b7c38d29deac481f96efe1616 - Note this branch is off 3.0-staging

• All agreed upon points are implemented / addressed.
  • Prepares an arvados-cli 3.0.0.1 release as requested.
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • After this passes review the next step is to run this through the Jenkins jobs to actually publish the gem.
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • From this branch, I can gem build arvados-cli, and then gem install it, and it resolves the arvados dependency the way we want:

    % gem install ./arvados-cli-3.0.0.1.gem
    Fetching arvados-3.0.0.gem
    Successfully installed arvados-3.0.0
    Successfully installed arvados-cli-3.0.0.1
    Parsing documentation for arvados-3.0.0
    Installing ri documentation for arvados-3.0.0
    Parsing documentation for arvados-cli-3.0.0.1
    Installing ri documentation for arvados-cli-3.0.0.1
    Done installing documentation for arvados, arvados-cli after 0 seconds
    2 gems installed
    

    I haven't run any Jenkins test because I don't expect any of them would be relevant. They might test that the gem is buildable but I've already done that and they wouldn't test the dependency. They might actually get upset that this gem's dependency isn't resolvable with a similar development version.
• Documentation has been updated.
  • I looked for a place to write a quick "changelog" in the gem metadata to explain the issue but there's nothing like that in the gemspec reference, I'd have to add a whole file which doesn't seem worth it for a one-off.
• Behaves appropriately at the intended scale (describe intended scale).
  • No change in scale
• Considered backwards and forwards compatibility issues between client and server.
  • N/A
• Follows our coding standards and GUI style guidelines.
  • No change in style

Ok, I think the thing left to do is run this:

https://ci.arvados.org/view/Release%20Pipeline/job/build-publish-packages-python-ruby/

With these parameters:

git_hash: d3107471cfe5275b7c38d29deac481f96efe1616
VERNO: 3.0.0
BUILD_RUBY: true
BUILD_PYTHON: false

Peter Amstutz wrote in #note-8:

Ok, I think the thing left to do is run this:

https://ci.arvados.org/view/Release%20Pipeline/job/build-publish-packages-python-ruby/

With these parameters:

git_hash: d3107471cfe5275b7c38d29deac481f96efe1616
VERNO: 3.0.0
BUILD_RUBY: true
BUILD_PYTHON: false

git_hash: 97f20a62f76b721b18be8b5ed47e6f4316a8286e for the arvados-login-sync fix but otherwise, sure. Did that at build-publish-packages-python-ruby: #89 and it says it all worked but nothing has shown up on rubygems.org after an hour-ish. A first pass reviewing the code doesn't reveal any obvious problems. I'll have to dig in in the morning.

Ah, perhaps that needs to be VERNO: 3.0.0.1

The Arvados version in the build environment may override version that is embedded in the Gemspec.

Because we're setting VERNO=3.0.0, which we've already built, the gem build gets skipped thanks to this logic in run-library.sh:handle_ruby_gem:

    local gem_version="$(nohash_version_from_git)"  # Returns "3.0.0" from variable
    …
    if ! [[ -e "${gem_name}-${gem_version}.gem" ]]; then
      # The build is here but we don't enter this block
    fi

I don't think we have any good options with our existing tooling. The build script assumes that the built gem will have the set version. We're breaking that assumption by setting a static version in the gemspec. But if we set VERNO=3.0.0.1, we will build and upload an arvados 3.0.0.1 gem, which we don't really want.

The quickest hack to get around this would be to push a commit, for this branch only, to statically set arvados.gemspec version=3.0.0, and then build with VERNO=3.0.0.1. It will build all the gems, but arvados will come out as 3.0.0. I think the uploader will recognize it's a duplicate upload and not care, but even if it fails, as long as the uploads we care about succeed it doesn't matter.

Brett Smith wrote in #note-11:

Because we're setting VERNO=3.0.0, which we've already built, the gem build gets skipped thanks to this logic in run-library.sh:handle_ruby_gem:

[...]

I don't think we have any good options with our existing tooling. The build script assumes that the built gem will have the set version. We're breaking that assumption by setting a static version in the gemspec. But if we set VERNO=3.0.0.1, we will build and upload an arvados 3.0.0.1 gem, which we don't really want.

The quickest hack to get around this would be to push a commit, for this branch only, to statically set arvados.gemspec version=3.0.0, and then build with VERNO=3.0.0.1. It will build all the gems, but arvados will come out as 3.0.0. I think the uploader will recognize it's a duplicate upload and not care, but even if it fails, as long as the uploads we care about succeed it doesn't matter.

I'm fine with that.

In order to build 3.0.0.1 I had to reconfigure the regexp validation on Jenkins which insisted the version number had to be M.N.O format.

In the process of going four rounds on that by the time I actually got it working I made a mistake and built 3.0.0.1 against main. build-publish-packages-python-ruby: #90

So, I built 3.0.0.2 against the intended commit, and those got published as expected (including skipping arvados). build-publish-packages-python-ruby: #91

Then I yanked all of the 3.0.0.1 gems. They were just a completely bogus build and should be forgotten. Now everything works as intended:

% mkdir ~/Scratch/GEMS
% GEM_HOME="$HOME/Scratch/GEMS" gem install arvados-cli arvados-login-sync
[…]
Done installing documentation for oj, launchy, curb, concurrent-ruby, arvados, arvados-cli after 3 seconds
Fetching arvados-login-sync-3.0.0.2.gem
Successfully installed arvados-login-sync-3.0.0.2
Parsing documentation for arvados-login-sync-3.0.0.2
Installing ri documentation for arvados-login-sync-3.0.0.2
Done installing documentation for arvados-login-sync after 0 seconds
7 gems installed

Lucas Di Pentima wrote in #note-3:

One kind-of-related issue that I found was that services/api/Gemfile explicitly ask for gem 'arvados', '~> 2.7.0.rc1'. Given that the 2.7.4 release will eventually be incompatible with the current Ruby, I guess we should update it, but not sure if it'll need a bit more thought so it gets automated, or at least mentioned on the release checklist for next time we release a major version.

Meant to comment, this needs to be part of some general "update dependencies in the Gemfile" process. We should basically think about it the same way we think about getting security updates in there.