Arvados

We can just cheat and remove the lock by hand, but then we have to remember to keep doing that every time we update a RailsAPI gem for as long as we support Ruby<3.3.

The bundler-override plugin seems to give us a way to drop the base64 dependency directly from the Gemfile.

If we take this approach, it means our Gemfile.lock will not be complete for Ruby 3.4+. However, I think it's a good trade-off to let ourselves defer that problem until we need to support a distribution that ships Ruby 3.4+. By the time we need that support, there might be changes in our dependencies that make it easier to solve this problem. Or we might be able to relax some constraints on our end, like dropping support for Ruby 2.7 or older 3.x.

Both Ubuntu 24.04 and Debian 13 (as I write this) still default to Ruby 3.1, so there is no imminent need for Ruby 3.4 support.

bundler-override plugin

sounds like the least-bad option so far.

Not sure if this is expected, but I tried running bundle install with ruby 2.7.8:

bundle install
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Could not find compatible versions

Because every version of bundler-override depends on Ruby >= 3.0.0
  and Gemfile depends on bundler-override >= 0,
  Ruby >= 3.0.0 is required.
So, because current Ruby version is = 2.7.8,
  version solving has failed

Doing something like this (I think) solved it:

diff --git a/services/api/Gemfile b/services/api/Gemfile
index 1bd3d337bd..fa36c4f18f 100644
--- a/services/api/Gemfile
+++ b/services/api/Gemfile
@@ -58,6 +58,7 @@ gem 'webrick'

 gem 'mini_portile2', '~> 2.8', '>= 2.8.1'

+if RUBY_VERSION.split('.')[0].to_i > 2
 plugin 'bundler-override'
 if bundler_override_paths = Bundler::Plugin.index.load_paths("bundler-override")
   require File.join(bundler_override_paths[0], "bundler-override")
@@ -77,6 +78,7 @@ else
   # just warning about it seems sufficient.
   Warning.warn("bundler-override plugin not available - any changes to Gemfile.lock may be inaccurate\n")
 end
+end

 # Install any plugin gems
 Dir.glob(File.join(File.dirname(__FILE__), 'lib', '**', "Gemfile")) do |f|

WDYT?

Lucas Di Pentima wrote in #note-9:

WDYT?

A change like this would mean anyone doing development with Ruby 2.7 could just reintroduce the problem with Gemfile.lock that caused this bug in the first place.

We should talk about this as a team. IMO we should make the rule for now "RailsAPI must run on Ruby 2.7 (for Ubuntu 20.04), but development must be done using 3.0+."

Brett Smith wrote in #note-10:

A change like this would mean anyone doing development with Ruby 2.7 could just reintroduce the problem with Gemfile.lock that caused this bug in the first place.

We should talk about this as a team. IMO we should make the rule for now "RailsAPI must run on Ruby 2.7 (for Ubuntu 20.04), but development must be done using 3.0+."

At standup we talked about the possibility of detecting whether or not Bundler is going to update Gemfile.lock, and then requiring the plugin accordingly. This isn't possible because it's a chicken-and-egg problem: Bundler needs to read Gemfile to know whether it needs to update Gemfile.lock. I naively wrote the code to do it using Bundler APIs, and I ended up recursing to Ruby's stack limit.

I ended up implementing a refined version of Lucas' suggestion. We only load the plugin if we're running on Ruby 3.0+. Then, if the plugin isn't loaded, we warn the user not to update Gemfile.lock. This should, in principle, flag for folks that they can't do RailsAPI development with Ruby 2.7.

I made the warning text a little sharper and present it with Bundler APIs so it's a little more visible.

21583-railsapi-base64-gem @ b4f40934a6b7cf80895be7936af64a755cfda81b - developer-run-tests: #4108

LGTM.

My only suggestion is to add a test case that greps Gemfile.lock for "base64 (", with an error that links back to the explanation in Gemfile.

After spending all the time diagnosing and working on a fix for this, Bundler told me to go @#$% myself. build-packages-ubuntu2004: #1608 /console

Fetching faraday 2.8.1
Downloading faraday-2.8.1 revealed dependencies not in the API or the lockfile
(base64 (>= 0)).
Either installing with `--full-index` or running `bundle update faraday` should
fix the problem.
The command '/bin/bash -c git clone git://git.arvados.org/arvados.git /tmp/arvados &&     cd /tmp/arvados &&     if [[ -n "${BRANCH}" ]]; then git checkout ${BRANCH}; fi &&     cd /tmp/arvados/services/api &&     /usr/local/rvm/bin/rvm-exec default bundle install &&     cd /tmp/arvados &&     go mod download' returned a non-zero code: 34

Brett Smith wrote in #note-14:

After spending all the time diagnosing and working on a fix for this, Bundler told me to go @#$% myself.

Doing the naive change of adding --full-index to the package build Dockerfile doesn't change the result. I think the error message is referring to running bundle install --full-index in development (e.g., so it fetches all the Gems and updates Gemfile.lock).

The error is 100% correct: Ruby 2.7 does not include base64, so the fact that it's missing from Gemfile.lock is a problem. Earlier when I wrote:

The current version of base64 is 0.2.0. This version is included in Ruby 3.3. Older versions of Ruby we support have 0.1.1.

That was oversimplified. The truth is:

This problem is difficult because we're trying to fight against Bundler. Its whole philosophy is that you should deploy the exact same set of gems every time, down to the version. That's creating a conflict with the way Passenger loads the base64 gem without going through Bundler or checking the version first.

https://www.phusionpassenger.com/docs/references/config_reference/nginx/#passenger_preload_bundler

We want to turn this on. I don't know why it's not the default. The whole crux of this problem is that Passenger loads gems before Bundler. Turning this option on makes it stop doing that.

This does require a semi-recent version of Passenger. But our Salt installer installs at least that version; our install documentation already fobs people off to the Passenger documentation; and it's not packaged by the distros. So this doesn't seem like a huge lift.

If I'm right that this works then I think I should revert the previous branch and prepare a branch that adds this to the installer.

Brett Smith wrote in #note-17:

https://www.phusionpassenger.com/docs/references/config_reference/nginx/#passenger_preload_bundler

We want to turn this on. I don't know why it's not the default. The whole crux of this problem is that Passenger loads gems before Bundler. Turning this option on makes it stop doing that.

This does require a semi-recent version of Passenger. But our Salt installer installs at least that version; our install documentation already fobs people off to the Passenger documentation; and it's not packaged by the distros. So this doesn't seem like a huge lift.

If I'm right that this works then I think I should revert the previous branch and prepare a branch that adds this to the installer.

I was starting to wonder if there was anything we could do to change passenger behavior, so I'm glad you found this, knock on wood.

Brett Smith wrote in #note-14:

After spending all the time diagnosing and working on a fix for this, Bundler told me to go @#$% myself. build-packages-ubuntu2004: #1608 /console

In addition to this problem, on distros where the package built, it was not installable. On Debian 12:

Setting up arvados-api-server (2.8.0~dev20240404141147-1) ...                                                                            

Assumption: nginx is configured to serve Rails from                                                                                      
            /var/www/arvados-api/current                  
Assumption: nginx and passenger run as www-data                                                                                          

Creating symlinks to configuration in /etc/arvados/api ...... done.                                                                      
Installing bundler...Successfully installed bundler-2.2.19                                                                               
1 gem installed                                                                                                                          
 done.                                                                                                                                   
Running bundle config set --local path /var/www/arvados-api/shared/vendor_bundle... done.                                        
Running bundle install...Don't run Bundler as root. Installing your bundle as root will break this                        
application for all non-root users on this machine.                                                                                      

[!] There was an error parsing `Gemfile`: cannot load such file -- /arvados/services/api/.bundle/plugin/gems/bundler-override-0.1.0/lib/bundler-override. Bundler cannot continue.                                                                                                

 #  from /var/www/arvados-api/current/Gemfile:64                                                                                         
 #  -------------------------------------------
 #  if bundler_override_paths = Bundler::Plugin.index.load_paths("bundler-override")                                               
 >    require File.join(bundler_override_paths[0], "bundler-override")                                                               
 #    # Ruby 3.4 drops base64 as a default gem. Because of this, various other gems                                                      
 #  -------------------------------------------
 failed.                                                                                                                                 
dpkg: error processing package arvados-api-server (--configure):
 installed arvados-api-server package post-installation script subprocess returned error exit status 4                                   
Errors were encountered while processing:                                                                                                
 arvados-api-server                                                                                                                      
E: Sub-process /usr/bin/dpkg returned an error code (1)                                                                                  

I have reverted the branch.

Brett Smith wrote in #note-17:

https://www.phusionpassenger.com/docs/references/config_reference/nginx/#passenger_preload_bundler

We want to turn this on. I don't know why it's not the default. The whole crux of this problem is that Passenger loads gems before Bundler. Turning this option on makes it stop doing that.

I have this implemented, and it does get the API server running on Debian 12 where I couldn't before, so I do think we want this. I haven't gotten test-provision-ubuntu2004 passing yet. It's possible there's a different blocking issue, although I admit it would be very surprising given how well the test failures align with the problematic commit. I'm also worried that I'm just not installing with the full stack that I think I am, even though I've double-checked all the versions and refs.