Arvados

Ready for review at c3944344c170a2b1014347cce3ea5388f2b8b79c on branch 16831-fix-arvbox-bundler-errors

This commit also upgrades arvbox to Debian 10. The changes in tools/arvbox/bin/arvbox are whitespace only.

Ward Vandewege wrote:

Ready for review at c3944344c170a2b1014347cce3ea5388f2b8b79c on branch 16831-fix-arvbox-bundler-errors

This commit also upgrades arvbox to Debian 10. The changes in tools/arvbox/bin/arvbox are whitespace only.

for some reason is not working as expected

I did:

docker rm

docker rmi arvados/arvbox-dev:c3944344c170a2b1014347cce3ea5388f2b8b79c  arvados/arvbox-dev:latest
docker rmi arvados/arvbox-base:c3944344c170a2b1014347cce3ea5388f2b8b79c  arvados/arvbox-base:latest
ARVBOX_CONTAINER=16831 arvbox build dev
ARVBOX_CONTAINER=16831 arvbox publicdev

That never comes up, and this is the trace that I found:

==> /etc/service/postgres/log/main/current <==
2020-09-21_14:44:31.21291 2020-09-21 14:44:31.212 UTC [75584] arvados@arvados_development FATAL:  database "arvados_development" does not exist

$ arvbox log postgres
2020-09-21_16:43:01.97357 Adding user `arvbox' to group `fuse' ...
2020-09-21_16:43:01.97441 Adding user arvbox to group fuse
2020-09-21_16:43:01.97906 Done.
2020-09-21_16:43:01.99845 Adding user `crunch' to group `fuse' ...
2020-09-21_16:43:01.99937 Adding user crunch to group fuse
2020-09-21_16:43:02.00361 Done.
2020-09-21_16:43:02.26814 + PGVERSION=11
2020-09-21_16:43:02.26817 + test -d /var/lib/postgresql/11/main
2020-09-21_16:43:02.26819 + /usr/lib/postgresql/11/bin/initdb --locale=en_US.UTF-8 -D /var/lib/postgresql/11/main
2020-09-21_16:43:02.28179 The files belonging to this database system will be owned by user "arvbox".
2020-09-21_16:43:02.28180 This user must also own the server process.
2020-09-21_16:43:02.28180 
2020-09-21_16:43:02.28193 The database cluster will be initialized with locale "en_US.UTF-8".
2020-09-21_16:43:02.28193 The default database encoding has accordingly been set to "UTF8".
2020-09-21_16:43:02.28194 The default text search configuration will be set to "english".
2020-09-21_16:43:02.28194 
2020-09-21_16:43:02.28194 Data page checksums are disabled.
2020-09-21_16:43:02.28195 
2020-09-21_16:43:02.28195 creating directory /var/lib/postgresql/11/main ... ok
2020-09-21_16:43:02.28203 creating subdirectories ... ok
2020-09-21_16:43:02.28242 selecting default max_connections ... 100
2020-09-21_16:43:02.29987 selecting default shared_buffers ... 128MB
2020-09-21_16:43:02.31897 selecting default timezone ... Etc/UTC
2020-09-21_16:43:02.33778 selecting dynamic shared memory implementation ... posix
2020-09-21_16:43:02.33781 creating configuration files ... ok
2020-09-21_16:43:02.33836 running bootstrap script ... ok
2020-09-21_16:43:02.42948 performing post-bootstrap initialization ... ok
2020-09-21_16:43:02.82263 syncing data to disk ... ok
2020-09-21_16:43:02.89309 
2020-09-21_16:43:02.89309 WARNING: enabling "trust" authentication for local connections
2020-09-21_16:43:02.89309 You can change this by editing pg_hba.conf or using the option -A, or
2020-09-21_16:43:02.89310 --auth-local and --auth-host, the next time you run initdb.
2020-09-21_16:43:02.89310 
2020-09-21_16:43:02.89310 Success. You can now start the database server using:
2020-09-21_16:43:02.89310 
2020-09-21_16:43:02.89310     /usr/lib/postgresql/11/bin/pg_ctl -D /var/lib/postgresql/11/main -l logfile start
2020-09-21_16:43:02.89311 
2020-09-21_16:43:02.89368 + mkdir -p /var/run/postgresql/11-main.pg_stat_tmp
2020-09-21_16:43:02.89371 + sh -c 'while ! (psql postgres -c'\''\du'\'' | grep '\''^ arvbox '\'') >/dev/null ; do createuser -s arvbox ; sleep 1 ; done'
2020-09-21_16:43:02.89439 + rm -f /var/lib/postgresql/11/main/postmaster.pid
2020-09-21_16:43:02.89490 + exec /usr/lib/postgresql/11/bin/postgres -D /var/lib/postgresql/11/main -c config_file=/etc/postgresql/11/main/postgresql.conf
2020-09-21_16:43:02.90472 2020-09-21 16:43:02.904 UTC [93] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2020-09-21_16:43:02.90474 2020-09-21 16:43:02.904 UTC [93] LOG:  could not bind IPv6 address " 
::1": Cannot assign requested address
2020-09-21_16:43:02.90474 2020-09-21 16:43:02.904 UTC [93] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2020-09-21_16:43:02.90664 2020-09-21 16:43:02.906 UTC [93] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 
2020-09-21_16:43:02.91484 2020-09-21 16:43:02.914 UTC [1576] LOG:  database system was shut down at 2020-09-21 16:43:02 UTC
2020-09-21_16:43:02.91805 2020-09-21 16:43:02.918 UTC [93] LOG:  database system is ready to accept connections
2020-09-21_16:47:52.73587 2020-09-21 16:47:52.735 UTC [40037] arvados@arvados_development FATAL:  database "arvados_development" does not exist
2020-09-21_16:47:53.01654 2020-09-21 16:47:53.016 UTC [40213] arvados@arvados_development FATAL:  database "arvados_development" does not exist
2020-09-21_16:48:23.39905 2020-09-21 16:48:23.399 UTC [46688] arvados@arvados_development FATAL:  database "arvados_development" does not exist
...

The new directory is: /var/lib/postgresql/11/main

Ad the end this ends up comming up ok... but this came up to my attention:

ARVBOX_CONTAINER=16462 arvbox/bin/arvbox ashell 
arvbox@e6e6faec6ad3:/usr/src/arvados$ git checkout 16462-forecast-wip
M    services/api/db/structure.sql
Branch '16462-forecast-wip' set up to track remote branch '16462-forecast-wip' from 'origin'.
Switched to a new branch '16462-forecast-wip'
arvbox@e6e6faec6ad3:/usr/src/arvados$ git pull 
Already up to date.
arvbox@e6e6faec6ad3:/usr/src/arvados$ git diff services/api/db/structure.sql
diff --git a/services/api/db/structure.sql b/services/api/db/structure.sql
index 83987d051..5a2ebd88f 100644
--- a/services/api/db/structure.sql
+++ b/services/api/db/structure.sql
@@ -10,20 +10,6 @@ SET check_function_bodies = false;
 SET xmloption = content;
 SET client_min_messages = warning;

---
--- Name: plpgsql; Type: EXTENSION; Schema: -; Owner: -
---
-
-CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog;
-
-
---
--- Name: EXTENSION plpgsql; Type: COMMENT; Schema: -; Owner: -
---
-
--- COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
-
-
 --
 -- Name: pg_trgm; Type: EXTENSION; Schema: -; Owner: -
 --

• Some gems install scripts in $GEMHOME/bin with ruby2.3 baked in
• Postgres goes from 9.6 to 11 which I think requires a database dump and import

So it won't work with an existing arvbox instance. The easiest solution is "arvbox reset" followed by "rm -rf ~/.arvbox/arvbox/gems".

Longer term, we should start explicitly versioning the arvbox environment in a way that can express what range of arvados versions it should work with, and possibly be able to migrate (or at least provide a useful error) when the arvbox environment itself changes in a non-backwards compatible way.

Anyway, aside from those notes, this LGTM.

I did some research without success about note7 and why did that happened.

LGTM too

I don't know if this is related or not. I bumped into the problem that "rails" executable is not available for bundle exec.

I fixed by doing the following:

diff --git a/build/run-tests.sh b/build/run-tests.sh
index 595f72108..97f404a0e 100755

--- a/build/run-tests.sh
+++ b/build/run-tests.sh
@@ -964,7 +964,7 @@ install_services/api() {
         set -ex
         cd "$WORKSPACE/services/api" 
         export RAILS_ENV=test
-        if "$bundle" exec rails db:environment:set ; then
+        if bin/rails db:environment:set ; then
             "$bundle" exec rake db:drop
         fi
         "$bundle" exec rake db:setup

because /usr/src/arvados/services/api/bin/rails was the only "rails" executable available.

and according to https://stackoverflow.com/questions/23846493/what-is-the-difference-between-rails-s-and-bundle-exec-rails-s seems like the right thing to do

Similar with passenger inside arvbox:

find / -name passenger -type f
/var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/bin/passenger

the gem was installed but the executable was not available for bundle exec. This was solved by doing:

 ln -s /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/bin/passenger /var/lib/arvados/lib/ruby/gems/2.5.0/bin

is there any command in the Ruby-world that would do this for us?

after my "fix" ... I discovered that "test lib/controller" in interactive mode mention this:

[z1111] [services/api: bundle] Using themes_for_rails 0.5.1 from https://github.com/arvados/themes_for_rails (at master@ddf6e59)
[z1111] [services/api: bundle] Updating files in vendor/cache
[z1111] [services/api: bundle] Bundle complete! 37 Gemfile dependencies, 114 gems now installed.
[z1111] [services/api: bundle] Bundled gems are installed into `/var/lib/arvados-arvbox/.gem`
[z1111] time="2020-10-16T17:29:44.357842545Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config build-native-support]" dir=services/api
[z1111] [services/api: passenger-config]  [passenger_native_support.so] trying to compile for the current user (arvbox) and Ruby interpreter...
[z1111] [services/api: passenger-config]      (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)
[z1111] [services/api: passenger-config]      Compilation successful. The logs are here:
[z1111] [services/api: passenger-config]      /tmp/arvados-server-boot-231877713/passenger_native_support-9hru1n.log
[z1111] [services/api: passenger-config]  [passenger_native_support.so] successfully loaded.
[z1111] time="2020-10-16T17:29:45.983003706Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config install-standalone-runtime]" dir=services/api
[z1111] [services/api: passenger-config] The Phusion Passenger Standalone runtime is already installed.
[z1111] [services/api: passenger-config] If you want to redownload it, re-run this program with the --force parameter.
[z1111] time="2020-10-16T17:29:46.185192791Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config validate-install]" dir=services/api
[z1111] [services/api: passenger-config]  * Checking whether this Passenger install is in PATH... 
✓
[z1111] [services/api: passenger-config]  * Checking whether there are no other Passenger installations... 
(!)
[z1111] [services/api: passenger-config] 
[z1111] [services/api: passenger-config]    You are currently validating against Phusion Passenger 6.0.2, located in:
[z1111] [services/api: passenger-config]    
[z1111] [services/api: passenger-config]      /var/lib/arvados-arvbox/.gem/ruby/2.5.0/gems/passenger-6.0.2/bin/passenger
[z1111] [services/api: passenger-config]    
[z1111] [services/api: passenger-config]    Besides this Passenger installation, the following other
[z1111] [services/api: passenger-config]    Passenger installations have also been detected:
[z1111] [services/api: passenger-config]    
[z1111] [services/api: passenger-config]      /var/lib/arvados/bin/passenger
[z1111] [services/api: passenger-config]      /var/lib/arvados/lib/ruby/gems/2.5.0/bin/passenger
[z1111] [services/api: passenger-config]    
[z1111] [services/api: passenger-config]    Please uninstall these other Passenger installations to avoid
[z1111] [services/api: passenger-config]    confusion or conflicts.
[z1111] [services/api: passenger-config] 
[z1111] [services/api: passenger-config] 
[z1111] [services/api: passenger-config] Detected 0 error(s), 1 warning(s).

maybe is relevant.