Bug #21583
closedRunning RailsAPI with Passenger implicity requires Ruby 3.3 via base64 0.2.0 lock
Description
Some useful background:
- base64 has been a default Gem for a while, but it will not be included in Ruby 3.4, and Ruby 3.3 warns you about this.
- To make the warning go away, libraries have started declaring a dependency on the base64 gem. The library that's relevant to our story is faraday, which is used by our ruby-google-api-client fork.
- The current version of base64 is 0.2.0. This version is included in Ruby 3.3. Older versions of Ruby we support have 0.1.1.
With this background, when we started using our ruby-google-api-client fork in RailsAPI in cbfdb1b66ab9c1b6e69d1c9cd589633386267177, source:services/api/Gemfile.lock gained a dependency on base64 0.2.0. This works fine in development, because Bundler can load this newer version before any code requires it.
However, Passenger loads the base64 Gem before it starts anything related to your application, including Bundler. Because of this, running RailsAPI behind Passenger with Ruby<3.3 now fails with this error in the Passenger log:
[ E 2024-03-12 15:12:44.8347 907382/Tf age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/arvados-api/current: The application encountered the following error: You have already activated base64 0.1.1, but your Gemfile requires base64 0.2.0. Since base64 is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports base64 as a default gem. (Gem::LoadError)
This is the root cause of #21524. Note how test-provision-ubuntu2004 started failing immediately after cbfdb1b66ab9c1b6e69d1c9cd589633386267177:
What can we do?
There is no single version of base64 that we can lock to that will keep everyone happy. The current lock breaks Ruby<3.3. If we change the lock to base64 0.1.1, we'll break Ruby>=3.3.
We cannot address the problem indirectly by tweaking our faraday dependency. We need version ~>2.8.0 to keep compatibility with the range of Ruby versions we're trying to support, and all those releases declare the base64 dependency.
This random blog post with cool styling says you can upgrade Passenger, but note they upgrade Passenger to the version in bookworm, which is the version I'm testing and have reproduced this problem with:
% apt list --installed '*passenger*' libnginx-mod-http-passenger/bookworm,now 1:6.0.20-1~bookworm1 amd64 [installed] passenger-dev/bookworm,now 1:6.0.20-1~bookworm1 amd64 [installed,automatic] passenger-doc/bookworm,now 1:6.0.20-1~bookworm1 all [installed,automatic] passenger/bookworm,now 1:6.0.20-1~bookworm1 amd64 [installed,automatic]
We could just revert cbfdb1b66ab9c1b6e69d1c9cd589633386267177. That has all the downsides implied by the commit message, but it would work.
We can just cheat and remove the lock by hand, but then we have to remember to keep doing that every time we update a RailsAPI gem for as long as we support Ruby<3.3. That sucks. We could write an extremely stupid test to help us remember this I guess.
🤷
Files
Related issues
Updated by Brett Smith 5 months ago
- Related to Bug #21384: faraday dependency can break new gem installs on Ruby 2.7 added
Updated by Brett Smith 5 months ago
- Blocks Feature #21383: Update Salt installer to support Debian 12 added
Updated by Brett Smith 5 months ago
- Blocks Bug #21524: test-provision-ubuntu2004 intermittently times out waiting for the controller to come up added
Updated by Brett Smith 5 months ago
We can just cheat and remove the lock by hand, but then we have to remember to keep doing that every time we update a RailsAPI gem for as long as we support Ruby<3.3.
The bundler-override plugin seems to give us a way to drop the base64 dependency directly from the Gemfile
.
If we take this approach, it means our Gemfile.lock
will not be complete for Ruby 3.4+. However, I think it's a good trade-off to let ourselves defer that problem until we need to support a distribution that ships Ruby 3.4+. By the time we need that support, there might be changes in our dependencies that make it easier to solve this problem. Or we might be able to relax some constraints on our end, like dropping support for Ruby 2.7 or older 3.x.
Both Ubuntu 24.04 and Debian 13 (as I write this) still default to Ruby 3.1, so there is no imminent need for Ruby 3.4 support.
Updated by Brett Smith 5 months ago
- Status changed from New to In Progress
21583-railsapi-base64-gem @ 7a2175baf2eda32cf53e03fbdf3e0f1f2abb8fc8 - developer-run-tests: #4086
- All agreed upon points are implemented / addressed.
- Yes, implemented removing the lock by using the
bundler-override
plugin, per discussion above.
- Yes, implemented removing the lock by using the
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- N/A
- Code is tested and passing, both automated and manual, what manual testing was done is described
- See above. I also generated the committed update to
Gemfile.lock
by runningbundle update faraday
with the plug-in configured, so that demonstrates updates work as desired.
- See above. I also generated the committed update to
- Documentation has been updated.
- N/A
- Behaves appropriately at the intended scale (describe intended scale).
- N/A
- Considered backwards and forwards compatibility issues between client and server.
- This maintains backwards compatibility for Ruby 2.7-3.2 at the cost of not supporting Ruby 3.4. But every possible solution is a trade-off, see #note-6 for why I think this is the best one.
- Follows our coding standards and GUI style guidelines.
- Yes
Updated by Lucas Di Pentima 5 months ago
Not sure if this is expected, but I tried running bundle install
with ruby 2.7.8:
bundle install Fetching gem metadata from https://rubygems.org/. Resolving dependencies... Could not find compatible versions Because every version of bundler-override depends on Ruby >= 3.0.0 and Gemfile depends on bundler-override >= 0, Ruby >= 3.0.0 is required. So, because current Ruby version is = 2.7.8, version solving has failed
Doing something like this (I think) solved it:
diff --git a/services/api/Gemfile b/services/api/Gemfile
index 1bd3d337bd..fa36c4f18f 100644
--- a/services/api/Gemfile
+++ b/services/api/Gemfile
@@ -58,6 +58,7 @@ gem 'webrick'
gem 'mini_portile2', '~> 2.8', '>= 2.8.1'
+if RUBY_VERSION.split('.')[0].to_i > 2
plugin 'bundler-override'
if bundler_override_paths = Bundler::Plugin.index.load_paths("bundler-override")
require File.join(bundler_override_paths[0], "bundler-override")
@@ -77,6 +78,7 @@ else
# just warning about it seems sufficient.
Warning.warn("bundler-override plugin not available - any changes to Gemfile.lock may be inaccurate\n")
end
+end
# Install any plugin gems
Dir.glob(File.join(File.dirname(__FILE__), 'lib', '**', "Gemfile")) do |f|
WDYT?
Updated by Brett Smith 5 months ago
Lucas Di Pentima wrote in #note-9:
WDYT?
A change like this would mean anyone doing development with Ruby 2.7 could just reintroduce the problem with Gemfile.lock
that caused this bug in the first place.
We should talk about this as a team. IMO we should make the rule for now "RailsAPI must run on Ruby 2.7 (for Ubuntu 20.04), but development must be done using 3.0+."
Updated by Peter Amstutz 5 months ago
- Target version changed from Development 2024-03-27 sprint to Development 2024-04-10 sprint
Updated by Brett Smith 5 months ago
Brett Smith wrote in #note-10:
A change like this would mean anyone doing development with Ruby 2.7 could just reintroduce the problem with
Gemfile.lock
that caused this bug in the first place.We should talk about this as a team. IMO we should make the rule for now "RailsAPI must run on Ruby 2.7 (for Ubuntu 20.04), but development must be done using 3.0+."
At standup we talked about the possibility of detecting whether or not Bundler is going to update Gemfile.lock
, and then requiring the plugin accordingly. This isn't possible because it's a chicken-and-egg problem: Bundler needs to read Gemfile
to know whether it needs to update Gemfile.lock
. I naively wrote the code to do it using Bundler APIs, and I ended up recursing to Ruby's stack limit.
I ended up implementing a refined version of Lucas' suggestion. We only load the plugin if we're running on Ruby 3.0+. Then, if the plugin isn't loaded, we warn the user not to update Gemfile.lock
. This should, in principle, flag for folks that they can't do RailsAPI development with Ruby 2.7.
I made the warning text a little sharper and present it with Bundler APIs so it's a little more visible.
21583-railsapi-base64-gem @ b4f40934a6b7cf80895be7936af64a755cfda81b - developer-run-tests: #4108
Updated by Brett Smith 5 months ago
After spending all the time diagnosing and working on a fix for this, Bundler told me to go @#$% myself. build-packages-ubuntu2004: #1608 /console
Fetching faraday 2.8.1 Downloading faraday-2.8.1 revealed dependencies not in the API or the lockfile (base64 (>= 0)). Either installing with `--full-index` or running `bundle update faraday` should fix the problem. The command '/bin/bash -c git clone git://git.arvados.org/arvados.git /tmp/arvados && cd /tmp/arvados && if [[ -n "${BRANCH}" ]]; then git checkout ${BRANCH}; fi && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle install && cd /tmp/arvados && go mod download' returned a non-zero code: 34
Updated by Brett Smith 5 months ago
Brett Smith wrote in #note-14:
After spending all the time diagnosing and working on a fix for this, Bundler told me to go @#$% myself.
Doing the naive change of adding --full-index
to the package build Dockerfile doesn't change the result. I think the error message is referring to running bundle install --full-index
in development (e.g., so it fetches all the Gems and updates Gemfile.lock
).
Updated by Brett Smith 5 months ago
The error is 100% correct: Ruby 2.7 does not include base64, so the fact that it's missing from Gemfile.lock is a problem. Earlier when I wrote:
The current version of base64 is 0.2.0. This version is included in Ruby 3.3. Older versions of Ruby we support have 0.1.1.
That was oversimplified. The truth is:
Ruby version | base64 version |
2.7 | Not included |
3.0 | 0.1.0 |
3.1 | 0.1.1 |
3.2 | 0.1.1 |
3.3 | 0.2.0, with a warning if you rely on it |
3.4 | Not included |
This problem is difficult because we're trying to fight against Bundler. Its whole philosophy is that you should deploy the exact same set of gems every time, down to the version. That's creating a conflict with the way Passenger loads the base64 gem without going through Bundler or checking the version first.
Updated by Brett Smith 5 months ago
https://www.phusionpassenger.com/docs/references/config_reference/nginx/#passenger_preload_bundler
We want to turn this on. I don't know why it's not the default. The whole crux of this problem is that Passenger loads gems before Bundler. Turning this option on makes it stop doing that.
This does require a semi-recent version of Passenger. But our Salt installer installs at least that version; our install documentation already fobs people off to the Passenger documentation; and it's not packaged by the distros. So this doesn't seem like a huge lift.
If I'm right that this works then I think I should revert the previous branch and prepare a branch that adds this to the installer.
Updated by Peter Amstutz 5 months ago
Brett Smith wrote in #note-17:
https://www.phusionpassenger.com/docs/references/config_reference/nginx/#passenger_preload_bundler
We want to turn this on. I don't know why it's not the default. The whole crux of this problem is that Passenger loads gems before Bundler. Turning this option on makes it stop doing that.
This does require a semi-recent version of Passenger. But our Salt installer installs at least that version; our install documentation already fobs people off to the Passenger documentation; and it's not packaged by the distros. So this doesn't seem like a huge lift.
If I'm right that this works then I think I should revert the previous branch and prepare a branch that adds this to the installer.
I was starting to wonder if there was anything we could do to change passenger behavior, so I'm glad you found this, knock on wood.
Updated by Brett Smith 5 months ago
Brett Smith wrote in #note-14:
After spending all the time diagnosing and working on a fix for this, Bundler told me to go @#$% myself. build-packages-ubuntu2004: #1608
/console
In addition to this problem, on distros where the package built, it was not installable. On Debian 12:
Setting up arvados-api-server (2.8.0~dev20240404141147-1) ... Assumption: nginx is configured to serve Rails from /var/www/arvados-api/current Assumption: nginx and passenger run as www-data Creating symlinks to configuration in /etc/arvados/api ...... done. Installing bundler...Successfully installed bundler-2.2.19 1 gem installed done. Running bundle config set --local path /var/www/arvados-api/shared/vendor_bundle... done. Running bundle install...Don't run Bundler as root. Installing your bundle as root will break this application for all non-root users on this machine. [!] There was an error parsing `Gemfile`: cannot load such file -- /arvados/services/api/.bundle/plugin/gems/bundler-override-0.1.0/lib/bundler-override. Bundler cannot continue. # from /var/www/arvados-api/current/Gemfile:64 # ------------------------------------------- # if bundler_override_paths = Bundler::Plugin.index.load_paths("bundler-override") > require File.join(bundler_override_paths[0], "bundler-override") # # Ruby 3.4 drops base64 as a default gem. Because of this, various other gems # ------------------------------------------- failed. dpkg: error processing package arvados-api-server (--configure): installed arvados-api-server package post-installation script subprocess returned error exit status 4 Errors were encountered while processing: arvados-api-server E: Sub-process /usr/bin/dpkg returned an error code (1)
I have reverted the branch.
Updated by Brett Smith 5 months ago
Brett Smith wrote in #note-17:
https://www.phusionpassenger.com/docs/references/config_reference/nginx/#passenger_preload_bundler
We want to turn this on. I don't know why it's not the default. The whole crux of this problem is that Passenger loads gems before Bundler. Turning this option on makes it stop doing that.
I have this implemented, and it does get the API server running on Debian 12 where I couldn't before, so I do think we want this. I haven't gotten test-provision-ubuntu2004 passing yet. It's possible there's a different blocking issue, although I admit it would be very surprising given how well the test failures align with the problematic commit. I'm also worried that I'm just not installing with the full stack that I think I am, even though I've double-checked all the versions and refs.
Updated by Brett Smith 5 months ago
- Blocks Support #21661: Test provision ubuntu-20.04 passes added
Updated by Peter Amstutz 5 months ago
- Target version changed from Development 2024-04-10 sprint to Development 2024-04-24 sprint
Updated by Brett Smith 4 months ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|eb6f398cb2405deea55a54a9b142d64c841d5f5a.