https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422020-09-10T19:36:59ZArvadosArvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=868992020-09-10T19:36:59ZNico César
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-4 priority-default closed parent" href="/issues/16814">Bug #16814</a>: Remove python2 from arvbox</i> added</li></ul> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=870392020-09-18T14:27:41ZWard Vandewegeward@curii.com
<ul><li><strong>Target version</strong> changed from <i>2020-10-07 Sprint</i> to <i>2020-09-23 Sprint</i></li><li><strong>Assigned To</strong> changed from <i>Nico César</i> to <i>Ward Vandewege</i></li></ul> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=870402020-09-18T14:27:47ZWard Vandewegeward@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=870662020-09-18T21:02:10ZWard Vandewegeward@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-1 priority-4 priority-default" href="/issues/16853">Bug #16853</a>: [arvbox] various bugs</i> added</li></ul> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=870722020-09-19T13:26:02ZWard Vandewegeward@curii.com
<ul></ul><p>Ready for review at <a class="changeset" title="16831: arvbox: bugfix: fix bundler version issue in documentation. Upgrade images to Debia..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/c3944344c170a2b1014347cce3ea5388f2b8b79c">c3944344c170a2b1014347cce3ea5388f2b8b79c</a> on branch 16831-fix-arvbox-bundler-errors</p>
<p>This commit also upgrades arvbox to Debian 10. The changes in <em>tools/arvbox/bin/arvbox</em> are whitespace only.</p> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=870912020-09-21T14:49:20ZNico César
<ul></ul><p>Ward Vandewege wrote:</p>
<blockquote>
<p>Ready for review at <a class="changeset" title="16831: arvbox: bugfix: fix bundler version issue in documentation. Upgrade images to Debia..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/c3944344c170a2b1014347cce3ea5388f2b8b79c">c3944344c170a2b1014347cce3ea5388f2b8b79c</a> on branch 16831-fix-arvbox-bundler-errors</p>
<p>This commit also upgrades arvbox to Debian 10. The changes in <em>tools/arvbox/bin/arvbox</em> are whitespace only.</p>
</blockquote>
<p>for some reason is not working as expected</p>
<p>I did:</p>
<p>docker rm</p>
<pre>
docker rmi arvados/arvbox-dev:c3944344c170a2b1014347cce3ea5388f2b8b79c arvados/arvbox-dev:latest
docker rmi arvados/arvbox-base:c3944344c170a2b1014347cce3ea5388f2b8b79c arvados/arvbox-base:latest
ARVBOX_CONTAINER=16831 arvbox build dev
ARVBOX_CONTAINER=16831 arvbox publicdev
</pre>
<p>That never comes up, and this is the trace that I found:</p>
<pre>
==> /etc/service/postgres/log/main/current <==
2020-09-21_14:44:31.21291 2020-09-21 14:44:31.212 UTC [75584] arvados@arvados_development FATAL: database "arvados_development" does not exist
</pre> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=871132020-09-21T16:51:02ZNico César
<ul></ul><pre>
$ arvbox log postgres
2020-09-21_16:43:01.97357 Adding user `arvbox' to group `fuse' ...
2020-09-21_16:43:01.97441 Adding user arvbox to group fuse
2020-09-21_16:43:01.97906 Done.
2020-09-21_16:43:01.99845 Adding user `crunch' to group `fuse' ...
2020-09-21_16:43:01.99937 Adding user crunch to group fuse
2020-09-21_16:43:02.00361 Done.
2020-09-21_16:43:02.26814 + PGVERSION=11
2020-09-21_16:43:02.26817 + test -d /var/lib/postgresql/11/main
2020-09-21_16:43:02.26819 + /usr/lib/postgresql/11/bin/initdb --locale=en_US.UTF-8 -D /var/lib/postgresql/11/main
2020-09-21_16:43:02.28179 The files belonging to this database system will be owned by user "arvbox".
2020-09-21_16:43:02.28180 This user must also own the server process.
2020-09-21_16:43:02.28180
2020-09-21_16:43:02.28193 The database cluster will be initialized with locale "en_US.UTF-8".
2020-09-21_16:43:02.28193 The default database encoding has accordingly been set to "UTF8".
2020-09-21_16:43:02.28194 The default text search configuration will be set to "english".
2020-09-21_16:43:02.28194
2020-09-21_16:43:02.28194 Data page checksums are disabled.
2020-09-21_16:43:02.28195
2020-09-21_16:43:02.28195 creating directory /var/lib/postgresql/11/main ... ok
2020-09-21_16:43:02.28203 creating subdirectories ... ok
2020-09-21_16:43:02.28242 selecting default max_connections ... 100
2020-09-21_16:43:02.29987 selecting default shared_buffers ... 128MB
2020-09-21_16:43:02.31897 selecting default timezone ... Etc/UTC
2020-09-21_16:43:02.33778 selecting dynamic shared memory implementation ... posix
2020-09-21_16:43:02.33781 creating configuration files ... ok
2020-09-21_16:43:02.33836 running bootstrap script ... ok
2020-09-21_16:43:02.42948 performing post-bootstrap initialization ... ok
2020-09-21_16:43:02.82263 syncing data to disk ... ok
2020-09-21_16:43:02.89309
2020-09-21_16:43:02.89309 WARNING: enabling "trust" authentication for local connections
2020-09-21_16:43:02.89309 You can change this by editing pg_hba.conf or using the option -A, or
2020-09-21_16:43:02.89310 --auth-local and --auth-host, the next time you run initdb.
2020-09-21_16:43:02.89310
2020-09-21_16:43:02.89310 Success. You can now start the database server using:
2020-09-21_16:43:02.89310
2020-09-21_16:43:02.89310 /usr/lib/postgresql/11/bin/pg_ctl -D /var/lib/postgresql/11/main -l logfile start
2020-09-21_16:43:02.89311
2020-09-21_16:43:02.89368 + mkdir -p /var/run/postgresql/11-main.pg_stat_tmp
2020-09-21_16:43:02.89371 + sh -c 'while ! (psql postgres -c'\''\du'\'' | grep '\''^ arvbox '\'') >/dev/null ; do createuser -s arvbox ; sleep 1 ; done'
2020-09-21_16:43:02.89439 + rm -f /var/lib/postgresql/11/main/postmaster.pid
2020-09-21_16:43:02.89490 + exec /usr/lib/postgresql/11/bin/postgres -D /var/lib/postgresql/11/main -c config_file=/etc/postgresql/11/main/postgresql.conf
2020-09-21_16:43:02.90472 2020-09-21 16:43:02.904 UTC [93] LOG: listening on IPv4 address "127.0.0.1", port 5432
2020-09-21_16:43:02.90474 2020-09-21 16:43:02.904 UTC [93] LOG: could not bind IPv6 address "
::1": Cannot assign requested address
2020-09-21_16:43:02.90474 2020-09-21 16:43:02.904 UTC [93] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2020-09-21_16:43:02.90664 2020-09-21 16:43:02.906 UTC [93] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-09-21_16:43:02.91484 2020-09-21 16:43:02.914 UTC [1576] LOG: database system was shut down at 2020-09-21 16:43:02 UTC
2020-09-21_16:43:02.91805 2020-09-21 16:43:02.918 UTC [93] LOG: database system is ready to accept connections
2020-09-21_16:47:52.73587 2020-09-21 16:47:52.735 UTC [40037] arvados@arvados_development FATAL: database "arvados_development" does not exist
2020-09-21_16:47:53.01654 2020-09-21 16:47:53.016 UTC [40213] arvados@arvados_development FATAL: database "arvados_development" does not exist
2020-09-21_16:48:23.39905 2020-09-21 16:48:23.399 UTC [46688] arvados@arvados_development FATAL: database "arvados_development" does not exist
...
</pre>
<p>The new directory is: /var/lib/postgresql/11/main</p>
<p>Ad the end this ends up comming up ok... but this came up to my attention:</p>
<pre>
ARVBOX_CONTAINER=16462 arvbox/bin/arvbox ashell
arvbox@e6e6faec6ad3:/usr/src/arvados$ git checkout 16462-forecast-wip
M services/api/db/structure.sql
Branch '16462-forecast-wip' set up to track remote branch '16462-forecast-wip' from 'origin'.
Switched to a new branch '16462-forecast-wip'
arvbox@e6e6faec6ad3:/usr/src/arvados$ git pull
Already up to date.
arvbox@e6e6faec6ad3:/usr/src/arvados$ git diff services/api/db/structure.sql
diff --git a/services/api/db/structure.sql b/services/api/db/structure.sql
index 83987d051..5a2ebd88f 100644
--- a/services/api/db/structure.sql
+++ b/services/api/db/structure.sql
@@ -10,20 +10,6 @@ SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
---
--- Name: plpgsql; Type: EXTENSION; Schema: -; Owner: -
---
-
-CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog;
-
-
---
--- Name: EXTENSION plpgsql; Type: COMMENT; Schema: -; Owner: -
---
-
--- COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
-
-
--
-- Name: pg_trgm; Type: EXTENSION; Schema: -; Owner: -
--
</pre> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=871142020-09-21T17:38:27ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><ul>
<li>Some gems install scripts in $GEMHOME/bin with ruby2.3 baked in</li>
<li>Postgres goes from 9.6 to 11 which I think requires a database dump and import</li>
</ul>
<p>So it won't work with an existing arvbox instance. The easiest solution is "arvbox reset" followed by "rm -rf ~/.arvbox/arvbox/gems".</p>
<p>Longer term, we should start explicitly versioning the arvbox environment in a way that can express what range of arvados versions it should work with, and possibly be able to migrate (or at least provide a useful error) when the arvbox environment itself changes in a non-backwards compatible way.</p> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=871172020-09-21T18:09:21ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Anyway, aside from those notes, this LGTM.</p> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=871202020-09-21T18:23:50ZNico César
<ul></ul><p>I did some research without success about note7 and why did that happened.</p>
<p>LGTM too</p> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=871242020-09-21T20:35:50ZWard Vandewegeward@curii.com
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '16831-fix-arvbox-bundler-errors' into master closes #16831 Arvados-DCO-1.1-Signed..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f88624b9408ea31c25e128269305aff4c7dba2ce">arvados|f88624b9408ea31c25e128269305aff4c7dba2ce</a>.</p> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=876852020-10-07T02:12:01ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>25</i></li></ul> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=880352020-10-15T19:51:10ZNico César
<ul></ul><p>I don't know if this is related or not. I bumped into the problem that "rails" executable is not available for bundle exec.</p>
<p>I fixed by doing the following:</p>
<p>diff --git a/build/run-tests.sh b/build/run-tests.sh<br />index 595f72108..97f404a0e 100755<br /><pre>
--- a/build/run-tests.sh
+++ b/build/run-tests.sh
@@ -964,7 +964,7 @@ install_services/api() {
set -ex
cd "$WORKSPACE/services/api"
export RAILS_ENV=test
- if "$bundle" exec rails db:environment:set ; then
+ if bin/rails db:environment:set ; then
"$bundle" exec rake db:drop
fi
"$bundle" exec rake db:setup
</pre></p>
<p>because /usr/src/arvados/services/api/bin/rails was the only "rails" executable available.</p>
<p>and according to <a class="external" href="https://stackoverflow.com/questions/23846493/what-is-the-difference-between-rails-s-and-bundle-exec-rails-s">https://stackoverflow.com/questions/23846493/what-is-the-difference-between-rails-s-and-bundle-exec-rails-s</a> seems like the right thing to do</p>
<p>Similar with passenger inside arvbox:</p>
<pre>
find / -name passenger -type f
/var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/bin/passenger
</pre>
<p>the gem was installed but the executable was not available for bundle exec. This was solved by doing:</p>
<pre>
ln -s /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/bin/passenger /var/lib/arvados/lib/ruby/gems/2.5.0/bin
</pre>
<p>is there any command in the Ruby-world that would do this for us?</p> Arvados - Bug #16831: arvbox and bundle errorshttps://dev.arvados.org/issues/16831?journal_id=880632020-10-16T17:33:46ZNico César
<ul></ul><p>after my "fix" ... I discovered that "test lib/controller" in interactive mode mention this:</p>
<pre>
[z1111] [services/api: bundle] Using themes_for_rails 0.5.1 from https://github.com/arvados/themes_for_rails (at master@ddf6e59)
[z1111] [services/api: bundle] Updating files in vendor/cache
[z1111] [services/api: bundle] Bundle complete! 37 Gemfile dependencies, 114 gems now installed.
[z1111] [services/api: bundle] Bundled gems are installed into `/var/lib/arvados-arvbox/.gem`
[z1111] time="2020-10-16T17:29:44.357842545Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config build-native-support]" dir=services/api
[z1111] [services/api: passenger-config] [passenger_native_support.so] trying to compile for the current user (arvbox) and Ruby interpreter...
[z1111] [services/api: passenger-config] (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)
[z1111] [services/api: passenger-config] Compilation successful. The logs are here:
[z1111] [services/api: passenger-config] /tmp/arvados-server-boot-231877713/passenger_native_support-9hru1n.log
[z1111] [services/api: passenger-config] [passenger_native_support.so] successfully loaded.
[z1111] time="2020-10-16T17:29:45.983003706Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config install-standalone-runtime]" dir=services/api
[z1111] [services/api: passenger-config] The Phusion Passenger Standalone runtime is already installed.
[z1111] [services/api: passenger-config] If you want to redownload it, re-run this program with the --force parameter.
[z1111] time="2020-10-16T17:29:46.185192791Z" level=info msg=executing PID=30099 command="[bundle exec passenger-config validate-install]" dir=services/api
[z1111] [services/api: passenger-config] * Checking whether this Passenger install is in PATH...
✓
[z1111] [services/api: passenger-config] * Checking whether there are no other Passenger installations...
(!)
[z1111] [services/api: passenger-config]
[z1111] [services/api: passenger-config] You are currently validating against Phusion Passenger 6.0.2, located in:
[z1111] [services/api: passenger-config]
[z1111] [services/api: passenger-config] /var/lib/arvados-arvbox/.gem/ruby/2.5.0/gems/passenger-6.0.2/bin/passenger
[z1111] [services/api: passenger-config]
[z1111] [services/api: passenger-config] Besides this Passenger installation, the following other
[z1111] [services/api: passenger-config] Passenger installations have also been detected:
[z1111] [services/api: passenger-config]
[z1111] [services/api: passenger-config] /var/lib/arvados/bin/passenger
[z1111] [services/api: passenger-config] /var/lib/arvados/lib/ruby/gems/2.5.0/bin/passenger
[z1111] [services/api: passenger-config]
[z1111] [services/api: passenger-config] Please uninstall these other Passenger installations to avoid
[z1111] [services/api: passenger-config] confusion or conflicts.
[z1111] [services/api: passenger-config]
[z1111] [services/api: passenger-config]
[z1111] [services/api: passenger-config] Detected 0 error(s), 1 warning(s).
</pre>
<p>maybe is relevant.</p>