Bug #16831

arvbox and bundle errors

Added by Nico César about 1 year ago. Updated 12 months ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)
Story points:
Release relationship:


as noted https://dev.arvados.org/issues/15888#note-25
I was forced to do:

@@ -558,7 +555,7 @@ setup_ruby_environment() {
             "$bundle" version | tee /dev/stderr | grep -q 'version 2'
-        ) || fatal 'install bundler'
+        ) #|| fatal 'install bundler' ## This is needed for now.

to make it possible to run arvbox tests (arvados/tools/arvbox/bin/arvbox restart test)

From Peter in the chat today:

Someone needs to dig into run-tests.sh and arvbox and figure out why, after installing a certain version of bundler, it doesn't get back the version it was expecting.

Also, arvbox was being held back on Ruby 2.3 because of SSO, which has problems with bundler 2.0, so the partial answer may be for arvbox to start using a newer Ruby (in other words bundler 2.0 doesn't actually support ruby 2.3 but earlier versions of bundler didn't check for it, and it worked in practice)


Task #16855: review 16267-change-arvbox-depsResolvedPeter Amstutz

Related issues

Related to Arvados - Bug #16814: Remove python2 from arvboxResolved09/11/2020

Related to Arvados - Bug #16853: [arvbox] various bugsNew

Associated revisions

Revision f88624b9
Added by Ward Vandewege about 1 year ago

Merge branch '16831-fix-arvbox-bundler-errors' into master

closes #16831

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>


#1 Updated by Nico César about 1 year ago

  • Related to Bug #16814: Remove python2 from arvbox added

#2 Updated by Ward Vandewege about 1 year ago

  • Target version changed from 2020-10-07 Sprint to 2020-09-23 Sprint
  • Assigned To changed from Nico César to Ward Vandewege

#3 Updated by Ward Vandewege about 1 year ago

  • Status changed from New to In Progress

#4 Updated by Ward Vandewege about 1 year ago

  • Related to Bug #16853: [arvbox] various bugs added

#5 Updated by Ward Vandewege about 1 year ago

Ready for review at c3944344c170a2b1014347cce3ea5388f2b8b79c on branch 16831-fix-arvbox-bundler-errors

This commit also upgrades arvbox to Debian 10. The changes in tools/arvbox/bin/arvbox are whitespace only.

#6 Updated by Nico César about 1 year ago

Ward Vandewege wrote:

Ready for review at c3944344c170a2b1014347cce3ea5388f2b8b79c on branch 16831-fix-arvbox-bundler-errors

This commit also upgrades arvbox to Debian 10. The changes in tools/arvbox/bin/arvbox are whitespace only.

for some reason is not working as expected

I did:

docker rm

docker rmi arvados/arvbox-dev:c3944344c170a2b1014347cce3ea5388f2b8b79c  arvados/arvbox-dev:latest
docker rmi arvados/arvbox-base:c3944344c170a2b1014347cce3ea5388f2b8b79c  arvados/arvbox-base:latest
ARVBOX_CONTAINER=16831 arvbox build dev
ARVBOX_CONTAINER=16831 arvbox publicdev

That never comes up, and this is the trace that I found:

==> /etc/service/postgres/log/main/current <==
2020-09-21_14:44:31.21291 2020-09-21 14:44:31.212 UTC [75584] arvados@arvados_development FATAL:  database "arvados_development" does not exist

#7 Updated by Nico César about 1 year ago

$ arvbox log postgres
2020-09-21_16:43:01.97357 Adding user `arvbox' to group `fuse' ...
2020-09-21_16:43:01.97441 Adding user arvbox to group fuse
2020-09-21_16:43:01.97906 Done.
2020-09-21_16:43:01.99845 Adding user `crunch' to group `fuse' ...
2020-09-21_16:43:01.99937 Adding user crunch to group fuse
2020-09-21_16:43:02.00361 Done.
2020-09-21_16:43:02.26814 + PGVERSION=11
2020-09-21_16:43:02.26817 + test -d /var/lib/postgresql/11/main
2020-09-21_16:43:02.26819 + /usr/lib/postgresql/11/bin/initdb --locale=en_US.UTF-8 -D /var/lib/postgresql/11/main
2020-09-21_16:43:02.28179 The files belonging to this database system will be owned by user "arvbox".
2020-09-21_16:43:02.28180 This user must also own the server process.
2020-09-21_16:43:02.28193 The database cluster will be initialized with locale "en_US.UTF-8".
2020-09-21_16:43:02.28193 The default database encoding has accordingly been set to "UTF8".
2020-09-21_16:43:02.28194 The default text search configuration will be set to "english".
2020-09-21_16:43:02.28194 Data page checksums are disabled.
2020-09-21_16:43:02.28195 creating directory /var/lib/postgresql/11/main ... ok
2020-09-21_16:43:02.28203 creating subdirectories ... ok
2020-09-21_16:43:02.28242 selecting default max_connections ... 100
2020-09-21_16:43:02.29987 selecting default shared_buffers ... 128MB
2020-09-21_16:43:02.31897 selecting default timezone ... Etc/UTC
2020-09-21_16:43:02.33778 selecting dynamic shared memory implementation ... posix
2020-09-21_16:43:02.33781 creating configuration files ... ok
2020-09-21_16:43:02.33836 running bootstrap script ... ok
2020-09-21_16:43:02.42948 performing post-bootstrap initialization ... ok
2020-09-21_16:43:02.82263 syncing data to disk ... ok
2020-09-21_16:43:02.89309 WARNING: enabling "trust" authentication for local connections
2020-09-21_16:43:02.89309 You can change this by editing pg_hba.conf or using the option -A, or
2020-09-21_16:43:02.89310 --auth-local and --auth-host, the next time you run initdb.
2020-09-21_16:43:02.89310 Success. You can now start the database server using:
2020-09-21_16:43:02.89310     /usr/lib/postgresql/11/bin/pg_ctl -D /var/lib/postgresql/11/main -l logfile start
2020-09-21_16:43:02.89368 + mkdir -p /var/run/postgresql/11-main.pg_stat_tmp
2020-09-21_16:43:02.89371 + sh -c 'while ! (psql postgres -c'\''\du'\'' | grep '\''^ arvbox '\'') >/dev/null ; do createuser -s arvbox ; sleep 1 ; done'
2020-09-21_16:43:02.89439 + rm -f /var/lib/postgresql/11/main/postmaster.pid
2020-09-21_16:43:02.89490 + exec /usr/lib/postgresql/11/bin/postgres -D /var/lib/postgresql/11/main -c config_file=/etc/postgresql/11/main/postgresql.conf
2020-09-21_16:43:02.90472 2020-09-21 16:43:02.904 UTC [93] LOG:  listening on IPv4 address "", port 5432
2020-09-21_16:43:02.90474 2020-09-21 16:43:02.904 UTC [93] LOG:  could not bind IPv6 address " 
::1": Cannot assign requested address
2020-09-21_16:43:02.90474 2020-09-21 16:43:02.904 UTC [93] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2020-09-21_16:43:02.90664 2020-09-21 16:43:02.906 UTC [93] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 
2020-09-21_16:43:02.91484 2020-09-21 16:43:02.914 UTC [1576] LOG:  database system was shut down at 2020-09-21 16:43:02 UTC
2020-09-21_16:43:02.91805 2020-09-21 16:43:02.918 UTC [93] LOG:  database system is ready to accept connections
2020-09-21_16:47:52.73587 2020-09-21 16:47:52.735 UTC [40037] arvados@arvados_development FATAL:  database "arvados_development" does not exist
2020-09-21_16:47:53.01654 2020-09-21 16:47:53.016 UTC [40213] arvados@arvados_development FATAL:  database "arvados_development" does not exist
2020-09-21_16:48:23.39905 2020-09-21 16:48:23.399 UTC [46688] arvados@arvados_development FATAL:  database "arvados_development" does not exist

The new directory is: /var/lib/postgresql/11/main

Ad the end this ends up comming up ok... but this came up to my attention:

ARVBOX_CONTAINER=16462 arvbox/bin/arvbox ashell 
arvbox@e6e6faec6ad3:/usr/src/arvados$ git checkout 16462-forecast-wip
M    services/api/db/structure.sql
Branch '16462-forecast-wip' set up to track remote branch '16462-forecast-wip' from 'origin'.
Switched to a new branch '16462-forecast-wip'
arvbox@e6e6faec6ad3:/usr/src/arvados$ git pull 
Already up to date.
arvbox@e6e6faec6ad3:/usr/src/arvados$ git diff services/api/db/structure.sql
diff --git a/services/api/db/structure.sql b/services/api/db/structure.sql
index 83987d051..5a2ebd88f 100644
--- a/services/api/db/structure.sql
+++ b/services/api/db/structure.sql
@@ -10,20 +10,6 @@ SET check_function_bodies = false;
 SET xmloption = content;
 SET client_min_messages = warning;

--- Name: plpgsql; Type: EXTENSION; Schema: -; Owner: -
--- Name: EXTENSION plpgsql; Type: COMMENT; Schema: -; Owner: -
--- COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
 -- Name: pg_trgm; Type: EXTENSION; Schema: -; Owner: -

#8 Updated by Peter Amstutz about 1 year ago

  • Some gems install scripts in $GEMHOME/bin with ruby2.3 baked in
  • Postgres goes from 9.6 to 11 which I think requires a database dump and import

So it won't work with an existing arvbox instance. The easiest solution is "arvbox reset" followed by "rm -rf ~/.arvbox/arvbox/gems".

Longer term, we should start explicitly versioning the arvbox environment in a way that can express what range of arvados versions it should work with, and possibly be able to migrate (or at least provide a useful error) when the arvbox environment itself changes in a non-backwards compatible way.

#9 Updated by Peter Amstutz about 1 year ago

Anyway, aside from those notes, this LGTM.

#10 Updated by Nico César about 1 year ago

I did some research without success about note7 and why did that happened.

LGTM too

#11 Updated by Ward Vandewege about 1 year ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

#12 Updated by Peter Amstutz about 1 year ago

  • Release set to 25

#13 Updated by Nico César about 1 year ago

I don't know if this is related or not. I bumped into the problem that "rails" executable is not available for bundle exec.

I fixed by doing the following:

diff --git a/build/run-tests.sh b/build/run-tests.sh
index 595f72108..97f404a0e 100755

--- a/build/run-tests.sh
+++ b/build/run-tests.sh
@@ -964,7 +964,7 @@ install_services/api() {
         set -ex
         cd "$WORKSPACE/services/api" 
         export RAILS_ENV=test
-        if "$bundle" exec rails db:environment:set ; then
+        if bin/rails db:environment:set ; then
             "$bundle" exec rake db:drop
         "$bundle" exec rake db:setup

because /usr/src/arvados/services/api/bin/rails was the only "rails" executable available.

and according to https://stackoverflow.com/questions/23846493/what-is-the-difference-between-rails-s-and-bundle-exec-rails-s seems like the right thing to do

Similar with passenger inside arvbox:

find / -name passenger -type f

the gem was installed but the executable was not available for bundle exec. This was solved by doing:

 ln -s /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/bin/passenger /var/lib/arvados/lib/ruby/gems/2.5.0/bin

is there any command in the Ruby-world that would do this for us?

#14 Updated by Nico César 12 months ago

after my "fix" ... I discovered that "test lib/controller" in interactive mode mention this:

[z1111] [services/api: bundle] Using themes_for_rails 0.5.1 from https://github.com/arvados/themes_for_rails (at master@ddf6e59)
[z1111] [services/api: bundle] Updating files in vendor/cache
[z1111] [services/api: bundle] Bundle complete! 37 Gemfile dependencies, 114 gems now installed.
[z1111] [services/api: bundle] Bundled gems are installed into `/var/lib/arvados-arvbox/.gem`
[z1111] time="2020-10-16T17:29:44.357842545Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config build-native-support]" dir=services/api
[z1111] [services/api: passenger-config]  [passenger_native_support.so] trying to compile for the current user (arvbox) and Ruby interpreter...
[z1111] [services/api: passenger-config]      (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)
[z1111] [services/api: passenger-config]      Compilation successful. The logs are here:
[z1111] [services/api: passenger-config]      /tmp/arvados-server-boot-231877713/passenger_native_support-9hru1n.log
[z1111] [services/api: passenger-config]  [passenger_native_support.so] successfully loaded.
[z1111] time="2020-10-16T17:29:45.983003706Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config install-standalone-runtime]" dir=services/api
[z1111] [services/api: passenger-config] The Phusion Passenger Standalone runtime is already installed.
[z1111] [services/api: passenger-config] If you want to redownload it, re-run this program with the --force parameter.
[z1111] time="2020-10-16T17:29:46.185192791Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config validate-install]" dir=services/api
[z1111] [services/api: passenger-config]  * Checking whether this Passenger install is in PATH... 
[z1111] [services/api: passenger-config]  * Checking whether there are no other Passenger installations... 
[z1111] [services/api: passenger-config] 
[z1111] [services/api: passenger-config]    You are currently validating against Phusion Passenger 6.0.2, located in:
[z1111] [services/api: passenger-config]    
[z1111] [services/api: passenger-config]      /var/lib/arvados-arvbox/.gem/ruby/2.5.0/gems/passenger-6.0.2/bin/passenger
[z1111] [services/api: passenger-config]    
[z1111] [services/api: passenger-config]    Besides this Passenger installation, the following other
[z1111] [services/api: passenger-config]    Passenger installations have also been detected:
[z1111] [services/api: passenger-config]    
[z1111] [services/api: passenger-config]      /var/lib/arvados/bin/passenger
[z1111] [services/api: passenger-config]      /var/lib/arvados/lib/ruby/gems/2.5.0/bin/passenger
[z1111] [services/api: passenger-config]    
[z1111] [services/api: passenger-config]    Please uninstall these other Passenger installations to avoid
[z1111] [services/api: passenger-config]    confusion or conflicts.
[z1111] [services/api: passenger-config] 
[z1111] [services/api: passenger-config] 
[z1111] [services/api: passenger-config] Detected 0 error(s), 1 warning(s).

maybe is relevant.

Also available in: Atom PDF