Bug #21583
Updated by Brett Smith 5 months ago
Some useful background: * base64 has been a default Gem for a while, but it will not be included in Ruby 3.4, and Ruby 3.3 warns you about this. * To make the warning go away, libraries have started declaring a dependency on the base64 gem. The library that's relevant to our story is "faraday":https://github.com/lostisland/faraday/commit/ea30bd0b543882f1cf26e75ac4e46e0705fa7e68, which is used by our ruby-google-api-client fork. * The current version of base64 is 0.2.0. This version is included in Ruby 3.3. Older versions of Ruby we support have 0.1.1. With this background, when we started using our ruby-google-api-client fork in RailsAPI in commit:cbfdb1b66ab9c1b6e69d1c9cd589633386267177, source:services/api/Gemfile.lock file:services/api/Gemfile.lock gained a dependency on base64 0.2.0. This works fine in development, because Bundler can load this newer version before any code requires it. However, Passenger loads the base64 Gem _before_ it starts anything related to your application, including Bundler. Because of this, running RailsAPI behind Passenger with Ruby<3.3 now fails with this error in the Passenger log: <pre>[ E 2024-03-12 15:12:44.8347 907382/Tf age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/arvados-api/current: The application encountered the following error: You have already activated base64 0.1.1, but your Gemfile requires base64 0.2.0. Since base64 is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports base64 as a default gem. (Gem::LoadError) </pre> This is the root cause of #21524. Note how test-provision-ubuntu2004 started failing immediately after commit:cbfdb1b66ab9c1b6e69d1c9cd589633386267177: !clipboard-202403130702-u2cmy.png! What can we do? There is no single version of base64 that we can lock to that will keep everyone happy. The current lock breaks Ruby<3.3. If we change the lock to base64 0.1.1, we'll break Ruby>=3.3. We cannot address the problem indirectly by tweaking our faraday dependency. We need version ~>2.8.0 to keep compatibility with the range of Ruby versions we're trying to support, and all those releases declare the base64 dependency. "This random blog post with cool styling":https://myrtana.sk/articles/my-passenger-was-really-old says you can upgrade Passenger, but note they upgrade Passenger to the version in bookworm, which is the version I'm testing and have reproduced this problem with: <pre>% apt list --installed '*passenger*' libnginx-mod-http-passenger/bookworm,now 1:6.0.20-1~bookworm1 amd64 [installed] passenger-dev/bookworm,now 1:6.0.20-1~bookworm1 amd64 [installed,automatic] passenger-doc/bookworm,now 1:6.0.20-1~bookworm1 all [installed,automatic] passenger/bookworm,now 1:6.0.20-1~bookworm1 amd64 [installed,automatic] </pre> We could just revert commit:cbfdb1b66ab9c1b6e69d1c9cd589633386267177. That has all the downsides implied by the commit message, but it would work. We can just cheat and remove the lock by hand, but then we have to remember to keep doing that every time we update a RailsAPI gem for as long as we support Ruby<3.3. That sucks. We could write an extremely stupid test to help us remember this I guess. 🤷