Bug #21384: faraday dependency can break new gem installs on Ruby 2.7 - Arvados

Actions

Copy link

Bug #21384

closed

faraday dependency can break new gem installs on Ruby 2.7

Added by Brett Smith 12 months ago. Updated 11 months ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Brett Smith

Category:

SDKs

Target version:

Development 2024-02-14 sprint

Story points:

Release:

Arvados 2.7.2

Release relationship:

Auto

Description

This is breaking test-provision-ubuntu2004:

  Name: ruby - Function: pkg.installed - Result: Clean - Started: 21:18:08.026795 - Duration: 1170.701 ms
  Name: arvados-api-package-install-gems-deps-pkg-installed - Function: pkg.installed - Result: Changed - Started: 21:18:09.197889 - Duration: 17227.98 ms
----------
          ID: arvados-api-package-install-gem-arvados-cli-installed
    Function: gem.installed
        Name: arvados-cli
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2423, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1293, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/gem.py", line 123, in installed
                  if __salt__["gem.install"](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/gem.py", line 138, in install
                  return _gem(["install"] + gems + options, ruby, gem_bin=gem_bin, runas=runas)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/gem.py", line 57, in _gem
                  raise CommandExecutionError(ret["stderr"])
              salt.exceptions.CommandExecutionError: ERROR:  Error installing arvados-cli:
                  The last version of faraday-net_http (>= 2.0, < 3.2) to support your Ruby & RubyGems was 3.0.2. Try installing it with `gem install faraday-net_http -v 3.0.2` and then running the current command again
                  faraday-net_http requires Ruby version >= 3.0.0. The current ruby version is 2.7.0.0.
     Started: 21:18:26.431321
    Duration: 145526.479 ms
     Changes:   
  Name: nginx - Function: service.mod_watch - Result: Changed - Started: 21:20:51.977148 - Duration: 41.252 ms

Summary for local
--------------
Succeeded: 138 (changed=100)
Failed:      1
--------------

Per discussion with Tom, there are two ways to address the problem: either make sure we explicitly install arvados-google-api-client first, or else restate the acceptable version ranges in the rest of our Gemspecs.

I prefer the latter because I think it's easier to follow, document, and then remove when we don't need it anymore. Tom has no objection.

Files

Download all files

focal-gem.dockerfile (384 Bytes) focal-gem.dockerfile		Brett Smith, 02/05/2024 03:02 PM
bookworm-gem.dockerfile (418 Bytes) bookworm-gem.dockerfile		Brett Smith, 02/05/2024 03:02 PM

Subtasks 2 (0 open — 2 closed)

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Brett Smith 12 months ago

I tried adding what I believe are the correct pins (213575b7c547cb161525e5e46c85baa01746c748), building new development packages, and then rerunning test-provision-ubuntu2004. It failed the same way. I'm not sure whether that means I didn't fix the problem, or it means there's a problem with my testing strategy. Does the installer go get the gems from rubygems.org?

Actions

Copy link

Updated by Tom Clegg 12 months ago

I don't see anything in source:tools/salt-install or arvados-formula.git that hints at installing the arvados-cli gem from source, so I suspect the salt install does indeed only know how to install gems by downloading them from rubygems.org. In that case, it seems like testing this change involves publishing a prerelease version (2.7.2rc1?) to rubygems and specifying that version in the salt installer. Assuming that works, I suppose we would want the salt installer to choose "version 2.7.2rc1 or a newer non-prerelease version" but I don't know if it's even possible to specify that.

Actions

Copy link

Updated by Tom Clegg 12 months ago

(I don't think this is directly related, but) FYI I just now needed to update the arvados-cli / login-sync → arvados dependencies to unbreak the test suite on Ruby 3: 44c2f5790059f3ec0380fbdb659489d8b02831b9

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Target version changed from Development 2024-01-31 sprint to Development 2024-02-14 sprint

Actions

Copy link Download all files

Updated by Brett Smith 11 months ago

File focal-gem.dockerfile focal-gem.dockerfile added
File bookworm-gem.dockerfile bookworm-gem.dockerfile added

21384-ruby27-gemspec @ 25a672f56ab03836ae54ee1d3af4ed9999ff164b

When I implemented Tom's initial suggestion, that just moved the goalposts: I got the same error against the faraday gem itself, rather than the transient dependency. So, I think we can just pin our faraday requirement directly. And, I think we can do that in arvados-google-api-client directly, and put up a new release of that, rather than updating multiple gems.

Here's how I tested. In your dev environment:

$ install -d -m 755 /tmp/gembuild
$ env -C arvados/sdk/ruby-google-api-client gem build arvados-google-api-client.gemspec
$ install -m 644 arvados/sdk/ruby-google-api-client/arvados-google-api-client-0.8.7.6.gem /tmp/gembuild/
$ docker build -f DIST-gem.dockerfile /tmp/gembuild
$ docker run --rm -ti IMAGE_HASH

In the container, paste environment credentials from a cluster you can access, then run arv user current as a quick test (or whatever other CLI SDK commands you want). For me, this test was successful on both focal (Ruby 2.7) and bookworm (Ruby 3.1), so I believe these changes smooth over installs on Ruby 2.7 without breaking Ruby 3.x.

All agreed upon points are implemented / addressed.
- Yes
Anything not implemented (discovered or discussed during work) has a follow-up story.
- N/A
Code is tested and passing, both automated and manual, what manual testing was done is described
- See above
Documentation has been updated.
- I could update CHANGELOG.md in the gem but it's not clear who cares? It doesn't currently have any entries for fourth-digit version changes like this.
Behaves appropriately at the intended scale (describe intended scale).
- No change
Considered backwards and forwards compatibility issues between client and server.
- The arvados and arvados-cli gems have a flexible enough requirement on arvados-google-api-client that existing installs will remain satisfied without change. Thanks to that, this will only smooth over new installs. It does mean new installs on Ruby 3.x will get a slightly older faraday than they strictly "need," but I'm not aware of any concrete problem that would cause.
Follows our coding standards and GUI style guidelines.
- Yes

Actions

Copy link

Updated by Tom Clegg 11 months ago

LGTM.

new installs on Ruby 3.x will get a slightly older faraday than they strictly "need,"

Yeah, I think this is fine.

I notice we have faraday (2.7.10) and faraday-net_http (3.0.2) in source:services/api/Gemfile.lock. Perhaps we should update that to arvados-google-api-client 0.8.7.6 once it's published? That would give us a bit more test coverage with faraday 2.8, and I suppose a bit more reassurance bundler can figure this out.

Actions

Copy link

Updated by Brett Smith 11 months ago

Release set to 69

Actions

Copy link

Updated by Brett Smith 11 months ago

Tom Clegg wrote in #note-6:

I notice we have faraday (2.7.10) and faraday-net_http (3.0.2) in source:services/api/Gemfile.lock. Perhaps we should update that to arvados-google-api-client 0.8.7.6 once it's published? That would give us a bit more test coverage with faraday 2.8, and I suppose a bit more reassurance bundler can figure this out.

Leaving this ticket open to look into this after we have test-provision-ubuntu2004 passing again.

Actions

Copy link

Updated by Brett Smith 11 months ago

Brett Smith wrote in #note-8:

Leaving this ticket open to look into this after we have test-provision-ubuntu2004 passing again.

Which it is: test-provision-ubuntu2004: #764

Actions

Copy link

#10

Updated by Brett Smith 11 months ago

21384-version-pins @ 10a7bcff702d5c9cddccba24a9d3ea4173ada625 - developer-run-tests: #4024

This makes the aforementioned change of updating RailsAPI's dependency on arvados-google-api-client. It also includes #21406 and #21454, which fit the theme of "modernizing the dependency versions we declare" and are all small enough I felt like it made sense to bundle and test them together than to put up a branch for each one.

All agreed upon points are implemented / addressed.
- Yes
Anything not implemented (discovered or discussed during work) has a follow-up story.
- N/A
Code is tested and passing, both automated and manual, what manual testing was done is described
- See above
Documentation has been updated.
- I would argue updating the gemspecs and setup.py files is updating the documentation. Bigger documentation changes are expected to be done as part of #21388.
Behaves appropriately at the intended scale (describe intended scale).
- No scale change
Considered backwards and forwards compatibility issues between client and server.
- This intentionally removes support for older Rubies and Pythons in line with our intended support policy. It's simply formalizing how we already think about and treat these packages.
Follows our coding standards and GUI style guidelines.
- Yes