Arvados

18657-passenger-binstub @ 0b89b2747dc6a2fdbfd84b0cb27690e781c61d07

For API and Workbench1, install binstubs into bin/ and run those directly instead of "bundle exec".

I've built a new dev docker image with this branch and while trying to start an arvbox instance from scratch, I got the same error that when testing main, the sdk logs kept saying:

...
2022-01-24_18:38:11.92651 + BUNDLER=bundler
2022-01-24_18:38:11.92652 + test -x /usr/src/arvados/sdk/cli/bin/bundler
2022-01-24_18:38:11.92658 + flock /var/lib/arvados/lib/ruby/gems/2.5.0/gems.lock bundler install --binstubs --verbose --local --no-deployment --binstubs=/usr/src/arvados/sdk/cli/binstubs
2022-01-24_18:38:11.96719 /var/lib/arvados/lib/ruby/2.7.0/rubygems.rb:277:in `find_spec_for_exe': can't find gem bundler (>= 0.a) with executable bundler (Gem::GemNotFoundException)
2022-01-24_18:38:11.96720       from /var/lib/arvados/lib/ruby/2.7.0/rubygems.rb:296:in `activate_bin_path'
2022-01-24_18:38:11.96721       from /var/lib/arvados/bin/bundler:23:in `<main>'
2022-01-24_18:38:11.96845 + flock /var/lib/arvados/lib/ruby/gems/2.5.0/gems.lock bundler install --binstubs --verbose --no-deployment --binstubs=/usr/src/arvados/sdk/cli/binstubs
2022-01-24_18:38:12.00795 /var/lib/arvados/lib/ruby/2.7.0/rubygems.rb:277:in `find_spec_for_exe': can't find gem bundler (>= 0.a) with executable bundler (Gem::GemNotFoundException)
2022-01-24_18:38:12.00797       from /var/lib/arvados/lib/ruby/2.7.0/rubygems.rb:296:in `activate_bin_path'
2022-01-24_18:38:12.00797       from /var/lib/arvados/bin/bundler:23:in `<main>'
2022-01-24_18:38:13.02256 + . /usr/local/lib/arvbox/common.sh
2022-01-24_18:38:13.02258 ++ export DEBIAN_FRONTEND=noninteractive
2022-01-24_18:38:13.02259 ++ DEBIAN_FRONTEND=noninteractive
2022-01-24_18:38:13.02260 ++ export GEM_HOME=/var/lib/arvados/lib/ruby/gems/2.5.0
...

OTOH, the api errors are slightly different:

...
2022-01-24_19:26:23.72130 + BUNDLER=bundler
2022-01-24_19:26:23.72130 + test -x /usr/src/arvados/services/api/bin/bundler
2022-01-24_19:26:23.72131 + flock /var/lib/arvados/lib/ruby/gems/2.5.0/gems.lock bundler install --binstubs --verbose --local --no-deployment --frozen --without=development
2022-01-24_19:26:23.76577 /var/lib/arvados/lib/ruby/2.7.0/rubygems.rb:277:in `find_spec_for_exe': Could not find 'bundler' (2.2.19) required by your /usr/src/arvados/services/api/Gemfile.lock. (Gem::GemNotFoundException)
2022-01-24_19:26:23.76579 To update to the latest version installed on your system, run `bundle update --bundler`.
2022-01-24_19:26:23.76580 To install the missing version, run `gem install bundler:2.2.19`
2022-01-24_19:26:23.76580       from /var/lib/arvados/lib/ruby/2.7.0/rubygems.rb:296:in `activate_bin_path'
2022-01-24_19:26:23.76580       from /var/lib/arvados/bin/bundler:23:in `<main>'
2022-01-24_19:26:23.76726 + flock /var/lib/arvados/lib/ruby/gems/2.5.0/gems.lock bundler install --binstubs --verbose --no-deployment --frozen --without=development
2022-01-24_19:26:23.83774 /var/lib/arvados/lib/ruby/2.7.0/rubygems.rb:277:in `find_spec_for_exe': Could not find 'bundler' (2.2.19) required by your /usr/src/arvados/services/api/Gemfile.lock. (Gem::GemNotFoundException)
2022-01-24_19:26:23.83776 To update to the latest version installed on your system, run `bundle update --bundler`.
2022-01-24_19:26:23.83776 To install the missing version, run `gem install bundler:2.2.19`
2022-01-24_19:26:23.83777       from /var/lib/arvados/lib/ruby/2.7.0/rubygems.rb:296:in `activate_bin_path'
2022-01-24_19:26:23.83777       from /var/lib/arvados/bin/bundler:23:in `<main>'
...

What it seemed to kind of unstuck this is installing the bundler gem (by running "/var/lib/arvados/bin/gem install bundler -v 2.2.19") inside the container, but then I get other issues that didn't investigate because they might be unrelated to the correct way this should work.

Note that both /var/lib/arvados/lib/ruby/gems/2.5.0 and /var/lib/arvados/lib/ruby/2.7.0 directories get mentioned, I wonder if that's a clue.

Yea, it could be confused about the gem directory.

18657-passenger-binstub @ 6bee9f6a6be594a7bcda9ec4062d56e325b2efc6

18657-passenger-binstub @ f12012fc721b4eeb5a72d3cee471cbd24264ec4d

Remove explicit arvbox GEM_HOME, install into binstubs/ for various Ruby programs, alternately use binstubs or bin/bundle exec depending on what seems to work. Tested with both "arvbox dev' and "arvbox test" starting from scratch.

Couple of minor run-tests.sh changes as well.

developer-run-tests: #2898

Peter Amstutz wrote:

18657-passenger-binstub @ f12012fc721b4eeb5a72d3cee471cbd24264ec4d

Remove explicit arvbox GEM_HOME, install into binstubs/ for various Ruby programs, alternately use binstubs or bin/bundle exec depending on what seems to work. Tested with both "arvbox dev' and "arvbox test" starting from scratch.

Couple of minor run-tests.sh changes as well.

developer-run-tests: #2898

Hmm, this seems to remove a number of the flocks around the bundle calls. Is there any particular reason for that? Because of the way arvbox starts up everything in parallel, I suspect this will cause problems - the flocks were added as part of #16955 to deal with startup race conditions.

I've rebuilt and started the 'dev' and 'localdemo' environments locally, and that seemed to work.