Bug #16831
closed
arvbox and bundle errors
Added by Nico César over 4 years ago. Updated about 4 years ago.
Description
as noted https://dev.arvados.org/issues/15888#note-25
I was forced to do:
@@ -558,7 +555,7 @@ setup_ruby_environment() {
fi
done
"$bundle" version | tee /dev/stderr | grep -q 'version 2'
- ) || fatal 'install bundler'
+ ) #|| fatal 'install bundler' ## This is needed for now.
fi
}
to make it possible to run arvbox tests (arvados/tools/arvbox/bin/arvbox restart test)
From Peter in the chat today:
Someone needs to dig into run-tests.sh and arvbox and figure out why, after installing a certain version of bundler, it doesn't get back the version it was expecting.
Also, arvbox was being held back on Ruby 2.3 because of SSO, which has problems with bundler 2.0, so the partial answer may be for arvbox to start using a newer Ruby (in other words bundler 2.0 doesn't actually support ruby 2.3 but earlier versions of bundler didn't check for it, and it worked in practice)
Updated by Nico César over 4 years ago
- Related to Bug #16814: Remove python2 from arvbox added
Updated by Ward Vandewege over 4 years ago
- Target version changed from 2020-10-07 Sprint to 2020-09-23 Sprint
- Assigned To changed from Nico César to Ward Vandewege
Updated by Ward Vandewege over 4 years ago
- Status changed from New to In Progress
Updated by Ward Vandewege over 4 years ago
- Related to Bug #16853: [arvbox] various bugs added
Updated by Ward Vandewege over 4 years ago
Ready for review at c3944344c170a2b1014347cce3ea5388f2b8b79c on branch 16831-fix-arvbox-bundler-errors
This commit also upgrades arvbox to Debian 10. The changes in tools/arvbox/bin/arvbox are whitespace only.
Updated by Nico César over 4 years ago
Ward Vandewege wrote:
Ready for review at c3944344c170a2b1014347cce3ea5388f2b8b79c on branch 16831-fix-arvbox-bundler-errors
This commit also upgrades arvbox to Debian 10. The changes in tools/arvbox/bin/arvbox are whitespace only.
for some reason is not working as expected
I did:
docker rm
docker rmi arvados/arvbox-dev:c3944344c170a2b1014347cce3ea5388f2b8b79c arvados/arvbox-dev:latest docker rmi arvados/arvbox-base:c3944344c170a2b1014347cce3ea5388f2b8b79c arvados/arvbox-base:latest ARVBOX_CONTAINER=16831 arvbox build dev ARVBOX_CONTAINER=16831 arvbox publicdev
That never comes up, and this is the trace that I found:
==> /etc/service/postgres/log/main/current <== 2020-09-21_14:44:31.21291 2020-09-21 14:44:31.212 UTC [75584] arvados@arvados_development FATAL: database "arvados_development" does not exist
Updated by Nico César over 4 years ago
$ arvbox log postgres 2020-09-21_16:43:01.97357 Adding user `arvbox' to group `fuse' ... 2020-09-21_16:43:01.97441 Adding user arvbox to group fuse 2020-09-21_16:43:01.97906 Done. 2020-09-21_16:43:01.99845 Adding user `crunch' to group `fuse' ... 2020-09-21_16:43:01.99937 Adding user crunch to group fuse 2020-09-21_16:43:02.00361 Done. 2020-09-21_16:43:02.26814 + PGVERSION=11 2020-09-21_16:43:02.26817 + test -d /var/lib/postgresql/11/main 2020-09-21_16:43:02.26819 + /usr/lib/postgresql/11/bin/initdb --locale=en_US.UTF-8 -D /var/lib/postgresql/11/main 2020-09-21_16:43:02.28179 The files belonging to this database system will be owned by user "arvbox". 2020-09-21_16:43:02.28180 This user must also own the server process. 2020-09-21_16:43:02.28180 2020-09-21_16:43:02.28193 The database cluster will be initialized with locale "en_US.UTF-8". 2020-09-21_16:43:02.28193 The default database encoding has accordingly been set to "UTF8". 2020-09-21_16:43:02.28194 The default text search configuration will be set to "english". 2020-09-21_16:43:02.28194 2020-09-21_16:43:02.28194 Data page checksums are disabled. 2020-09-21_16:43:02.28195 2020-09-21_16:43:02.28195 creating directory /var/lib/postgresql/11/main ... ok 2020-09-21_16:43:02.28203 creating subdirectories ... ok 2020-09-21_16:43:02.28242 selecting default max_connections ... 100 2020-09-21_16:43:02.29987 selecting default shared_buffers ... 128MB 2020-09-21_16:43:02.31897 selecting default timezone ... Etc/UTC 2020-09-21_16:43:02.33778 selecting dynamic shared memory implementation ... posix 2020-09-21_16:43:02.33781 creating configuration files ... ok 2020-09-21_16:43:02.33836 running bootstrap script ... ok 2020-09-21_16:43:02.42948 performing post-bootstrap initialization ... ok 2020-09-21_16:43:02.82263 syncing data to disk ... ok 2020-09-21_16:43:02.89309 2020-09-21_16:43:02.89309 WARNING: enabling "trust" authentication for local connections 2020-09-21_16:43:02.89309 You can change this by editing pg_hba.conf or using the option -A, or 2020-09-21_16:43:02.89310 --auth-local and --auth-host, the next time you run initdb. 2020-09-21_16:43:02.89310 2020-09-21_16:43:02.89310 Success. You can now start the database server using: 2020-09-21_16:43:02.89310 2020-09-21_16:43:02.89310 /usr/lib/postgresql/11/bin/pg_ctl -D /var/lib/postgresql/11/main -l logfile start 2020-09-21_16:43:02.89311 2020-09-21_16:43:02.89368 + mkdir -p /var/run/postgresql/11-main.pg_stat_tmp 2020-09-21_16:43:02.89371 + sh -c 'while ! (psql postgres -c'\''\du'\'' | grep '\''^ arvbox '\'') >/dev/null ; do createuser -s arvbox ; sleep 1 ; done' 2020-09-21_16:43:02.89439 + rm -f /var/lib/postgresql/11/main/postmaster.pid 2020-09-21_16:43:02.89490 + exec /usr/lib/postgresql/11/bin/postgres -D /var/lib/postgresql/11/main -c config_file=/etc/postgresql/11/main/postgresql.conf 2020-09-21_16:43:02.90472 2020-09-21 16:43:02.904 UTC [93] LOG: listening on IPv4 address "127.0.0.1", port 5432 2020-09-21_16:43:02.90474 2020-09-21 16:43:02.904 UTC [93] LOG: could not bind IPv6 address " ::1": Cannot assign requested address 2020-09-21_16:43:02.90474 2020-09-21 16:43:02.904 UTC [93] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry. 2020-09-21_16:43:02.90664 2020-09-21 16:43:02.906 UTC [93] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2020-09-21_16:43:02.91484 2020-09-21 16:43:02.914 UTC [1576] LOG: database system was shut down at 2020-09-21 16:43:02 UTC 2020-09-21_16:43:02.91805 2020-09-21 16:43:02.918 UTC [93] LOG: database system is ready to accept connections 2020-09-21_16:47:52.73587 2020-09-21 16:47:52.735 UTC [40037] arvados@arvados_development FATAL: database "arvados_development" does not exist 2020-09-21_16:47:53.01654 2020-09-21 16:47:53.016 UTC [40213] arvados@arvados_development FATAL: database "arvados_development" does not exist 2020-09-21_16:48:23.39905 2020-09-21 16:48:23.399 UTC [46688] arvados@arvados_development FATAL: database "arvados_development" does not exist ...
The new directory is: /var/lib/postgresql/11/main
Ad the end this ends up comming up ok... but this came up to my attention:
ARVBOX_CONTAINER=16462 arvbox/bin/arvbox ashell arvbox@e6e6faec6ad3:/usr/src/arvados$ git checkout 16462-forecast-wip M services/api/db/structure.sql Branch '16462-forecast-wip' set up to track remote branch '16462-forecast-wip' from 'origin'. Switched to a new branch '16462-forecast-wip' arvbox@e6e6faec6ad3:/usr/src/arvados$ git pull Already up to date. arvbox@e6e6faec6ad3:/usr/src/arvados$ git diff services/api/db/structure.sql diff --git a/services/api/db/structure.sql b/services/api/db/structure.sql index 83987d051..5a2ebd88f 100644 --- a/services/api/db/structure.sql +++ b/services/api/db/structure.sql @@ -10,20 +10,6 @@ SET check_function_bodies = false; SET xmloption = content; SET client_min_messages = warning; --- --- Name: plpgsql; Type: EXTENSION; Schema: -; Owner: - --- - -CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog; - - --- --- Name: EXTENSION plpgsql; Type: COMMENT; Schema: -; Owner: - --- - --- COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language'; - - -- -- Name: pg_trgm; Type: EXTENSION; Schema: -; Owner: - --
Updated by Peter Amstutz over 4 years ago
- Some gems install scripts in $GEMHOME/bin with ruby2.3 baked in
- Postgres goes from 9.6 to 11 which I think requires a database dump and import
So it won't work with an existing arvbox instance. The easiest solution is "arvbox reset" followed by "rm -rf ~/.arvbox/arvbox/gems".
Longer term, we should start explicitly versioning the arvbox environment in a way that can express what range of arvados versions it should work with, and possibly be able to migrate (or at least provide a useful error) when the arvbox environment itself changes in a non-backwards compatible way.
Updated by Peter Amstutz over 4 years ago
Anyway, aside from those notes, this LGTM.
Updated by Nico César over 4 years ago
I did some research without success about note7 and why did that happened.
LGTM too
Updated by Ward Vandewege over 4 years ago
- % Done changed from 0 to 100
- Status changed from In Progress to Resolved
Applied in changeset arvados|f88624b9408ea31c25e128269305aff4c7dba2ce.
Updated by Nico César about 4 years ago
I don't know if this is related or not. I bumped into the problem that "rails" executable is not available for bundle exec.
I fixed by doing the following:
diff --git a/build/run-tests.sh b/build/run-tests.sh
index 595f72108..97f404a0e 100755
--- a/build/run-tests.sh
+++ b/build/run-tests.sh
@@ -964,7 +964,7 @@ install_services/api() {
set -ex
cd "$WORKSPACE/services/api"
export RAILS_ENV=test
- if "$bundle" exec rails db:environment:set ; then
+ if bin/rails db:environment:set ; then
"$bundle" exec rake db:drop
fi
"$bundle" exec rake db:setup
because /usr/src/arvados/services/api/bin/rails was the only "rails" executable available.
and according to https://stackoverflow.com/questions/23846493/what-is-the-difference-between-rails-s-and-bundle-exec-rails-s seems like the right thing to do
Similar with passenger inside arvbox:
find / -name passenger -type f /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/bin/passenger
the gem was installed but the executable was not available for bundle exec. This was solved by doing:
ln -s /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/bin/passenger /var/lib/arvados/lib/ruby/gems/2.5.0/bin
is there any command in the Ruby-world that would do this for us?
Updated by Nico César about 4 years ago
after my "fix" ... I discovered that "test lib/controller" in interactive mode mention this:
[z1111] [services/api: bundle] Using themes_for_rails 0.5.1 from https://github.com/arvados/themes_for_rails (at master@ddf6e59) [z1111] [services/api: bundle] Updating files in vendor/cache [z1111] [services/api: bundle] Bundle complete! 37 Gemfile dependencies, 114 gems now installed. [z1111] [services/api: bundle] Bundled gems are installed into `/var/lib/arvados-arvbox/.gem` [z1111] time="2020-10-16T17:29:44.357842545Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config build-native-support]" dir=services/api [z1111] [services/api: passenger-config] [passenger_native_support.so] trying to compile for the current user (arvbox) and Ruby interpreter... [z1111] [services/api: passenger-config] (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable) [z1111] [services/api: passenger-config] Compilation successful. The logs are here: [z1111] [services/api: passenger-config] /tmp/arvados-server-boot-231877713/passenger_native_support-9hru1n.log [z1111] [services/api: passenger-config] [passenger_native_support.so] successfully loaded. [z1111] time="2020-10-16T17:29:45.983003706Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config install-standalone-runtime]" dir=services/api [z1111] [services/api: passenger-config] The Phusion Passenger Standalone runtime is already installed. [z1111] [services/api: passenger-config] If you want to redownload it, re-run this program with the --force parameter. [z1111] time="2020-10-16T17:29:46.185192791Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config validate-install]" dir=services/api [z1111] [services/api: passenger-config] * Checking whether this Passenger install is in PATH... ✓ [z1111] [services/api: passenger-config] * Checking whether there are no other Passenger installations... (!) [z1111] [services/api: passenger-config] [z1111] [services/api: passenger-config] You are currently validating against Phusion Passenger 6.0.2, located in: [z1111] [services/api: passenger-config] [z1111] [services/api: passenger-config] /var/lib/arvados-arvbox/.gem/ruby/2.5.0/gems/passenger-6.0.2/bin/passenger [z1111] [services/api: passenger-config] [z1111] [services/api: passenger-config] Besides this Passenger installation, the following other [z1111] [services/api: passenger-config] Passenger installations have also been detected: [z1111] [services/api: passenger-config] [z1111] [services/api: passenger-config] /var/lib/arvados/bin/passenger [z1111] [services/api: passenger-config] /var/lib/arvados/lib/ruby/gems/2.5.0/bin/passenger [z1111] [services/api: passenger-config] [z1111] [services/api: passenger-config] Please uninstall these other Passenger installations to avoid [z1111] [services/api: passenger-config] confusion or conflicts. [z1111] [services/api: passenger-config] [z1111] [services/api: passenger-config] [z1111] [services/api: passenger-config] Detected 0 error(s), 1 warning(s).
maybe is relevant.