Bug #15161

Dev clusters are failing: uninitialized constant ArInternalMetadatum (NameError)

Added by Nico César 5 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
04/30/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description


App 51878 stderr: The PGconn, PGresult, and PGError constants are deprecated, and will be
App 51878 stderr: removed as of version 1.0.
App 51878 stderr: 
App 51878 stderr: You should use PG::Connection, PG::Result, and PG::Error instead, respectively.
App 51878 stderr: 
App 51878 stderr: Called from /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
App 51878 stdout: 
[ 2019-04-29 13:28:23.8920 48517/7f99d808c700 age/Cor/App/Implementation.cpp:304 ]: Could not spawn process for application /var/www/c97qk.arvadosapi.com/current: An error occurred while starting up the preloader.
  Error ID: d1d121c0
  Error details saved to: /tmp/passenger-error-CedvzV.html
  Message from application: uninitialized constant ArInternalMetadatum (NameError)
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/inflector/methods.rb:261:in `const_get'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/inflector/methods.rb:261:in `block in constantize'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/inflector/methods.rb:259:in `each'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/inflector/methods.rb:259:in `inject'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/inflector/methods.rb:259:in `constantize'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/core_ext/string/inflections.rb:66:in `constantize'
  /var/www/arvados-api/current/lib/can_be_an_owner.rb:19:in `block in included'
  /var/www/arvados-api/current/lib/can_be_an_owner.rb:15:in `each'
  /var/www/arvados-api/current/lib/can_be_an_owner.rb:15:in `included'
  /var/www/arvados-api/current/app/models/group.rb:12:in `include'
  /var/www/arvados-api/current/app/models/group.rb:12:in `<class:Group>'
  /var/www/arvados-api/current/app/models/group.rb:8:in `<top (required)>'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:472:in `block (2 levels) in eager_load!'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:471:in `each'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:471:in `block in eager_load!'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:469:in `each'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:469:in `eager_load!'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:346:in `eager_load!'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application/finisher.rb:56:in `each'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application/finisher.rb:56:in `block in <module:Finisher>'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `instance_exec'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `run'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:55:in `block in run_initializers'
  /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:228:in `block in tsort_each'
  /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
  /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:431:in `each_strongly_connected_component_from'
  /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:349:in `block in each_strongly_connected_component'
  /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:347:in `each'
  /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:347:in `call'
  /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:347:in `each_strongly_connected_component'
  /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:226:in `tsort_each'
  /usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:205:in `tsort_each'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:54:in `run_initializers'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:352:in `initialize!'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `public_send'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `method_missing'
  /var/www/arvados-api/current/config/environment.rb:10:in `<top (required)>'
  config.ru:7:in `require'
  config.ru:7:in `block in <main>'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.6.11/lib/rack/builder.rb:55:in `instance_eval'
  /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.6.11/lib/rack/builder.rb:55:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:110:in `eval'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:110:in `preload_app'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:156:in `<module:App>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<main>'


Subtasks

Task #15170: Review 15161-config-check-rake-fixResolvedPeter Amstutz


Related issues

Related to Arvados - Bug #15225: [API] installation gives error: Creating git_internal_dir ''...mkdir: cannot create directory ‘’: No such file or directoryResolved

History

#1 Updated by Lucas Di Pentima 5 months ago

Checking c97qk's files, it seems that the structure.sql file is from rails 5 but the rest of the code is from 4.2.

The ar_internal_metadata table causing this error was added on 261543dfc3, and the Gemfile on c97qk asks for rails 4.2. Also, the app/models/arvados_model.rb file is from rails 4.2 because the class is subclassing ActiveRecord::Base instead of ApplicationRecord

#2 Updated by Nico César 5 months ago

  • Project changed from Arvados Private to OPS

Lucas Di Pentima wrote:

Checking c97qk's files, it seems that the structure.sql file is from rails 5 but the rest of the code is from 4.2.

The ar_internal_metadata table causing this error was added on 261543dfc3, and the Gemfile on c97qk asks for rails 4.2. Also, the app/models/arvados_model.rb file is from rails 4.2 because the class is subclassing ActiveRecord::Base instead of ApplicationRecord

Mmmh the problem might be on our side: Dev clusters should be only DEV!

c97qk:~# apt-cache policy arvados-api-server| head
arvados-api-server:
Installed: 1.3.2-1
Candidate: 1.3.2-1
Version table: *** 1.3.2-1 500
500 http://apt.arvados.org xenial/main amd64 Packages
100 /var/lib/dpkg/status
1.3.1.20190422202321-8 500
500 http://apt.arvados.org xenial-dev/main amd64 Packages
1.3.1.20190422184531-8 500

#4 Updated by Nico César 5 months ago

Now that #15163 is out of the way I tried to downgrade the API server ... ended up with another issue:

c97qk:~# apt-get install arvados-api-server=1.3.1.20190422202321-8
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be DOWNGRADED:
  arvados-api-server
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 320 not upgraded.
Need to get 0 B/28.1 MB of archives.
After this operation, 1,262 kB disk space will be freed.
Do you want to continue? [Y/n] 
dpkg: warning: downgrading arvados-api-server from 1.3.2-1 to 1.3.1.20190422202321-8
(Reading database ... 110469 files and directories currently installed.)
Preparing to unpack .../arvados-api-server_1.3.1.20190422202321-8_amd64.deb ...
Unpacking arvados-api-server (1.3.1.20190422202321-8) over (1.3.2-1) ...
Setting up arvados-api-server (1.3.1.20190422202321-8) ...

Assumption: nginx is configured to serve Rails from
            /var/www/arvados-api/current
Assumption: nginx and passenger run as www-data

Creating symlinks to configuration in /etc/arvados/api ...... done.
Running bundle install... done.
Ensuring directory and file permissions ...... done.
Running db:migrate...Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache owner (uid=33) is not me (uid=0)
DEPRECATION WARNING: `config.serve_static_files` is deprecated and will be removed in Rails 5.1.
Please use `config.public_file_server.enabled = false` instead.
 (called from block in <top (required)> at /var/www/arvados-api/current/config/environments/production.rb:12)
 done.
Checking application.yml for completeness...Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache owner (uid=33) is not me (uid=0)
rake aborted!
Don't know how to build task 'config:check' (See the list of available tasks with `rake --tasks`)
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli/exec.rb:75:in `load'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli/exec.rb:75:in `kernel_load'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli.rb:424:in `exec'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli.rb:27:in `dispatch'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli.rb:18:in `start'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/exe/bundle:30:in `block in <top (required)>'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/friendly_errors.rb:122:in `with_friendly_errors'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/exe/bundle:22:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.3.5/bin/bundle:23:in `load'
/usr/local/rvm/gems/ruby-2.3.5/bin/bundle:23:in `<main>'
/usr/local/rvm/gems/ruby-2.3.5/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.3.5/bin/ruby_executable_hooks:15:in `<main>'
(See full trace by running task with --trace)
 failed.
Precompiling assets... skipped.
Initializing git_internal_dir... skipped.

PLEASE NOTE:

The arvados-api-server package was not configured completely because
/etc/arvados/api/application.yml needs some tweaking.
Please refer to the documentation at
<http://doc.arvados.org/install/install-api-server.html#configure> for more details.

When application.yml has been modified,
reconfigure or reinstall this package.

#5 Updated by Nico César 5 months ago

Tomorrow we'll investigate why

Don't know how to build task 'config:check'

#6 Updated by Lucas Di Pentima 5 months ago

The config:check rake task was removed on 2ea729f97166cb5fda4a6ca0818f0a0ab15388ab -- if this is part of a unified rails package building mechanism, maybe we should have made it a no-op instead of deleting it?

#7 Updated by Lucas Di Pentima 5 months ago

Update at 0cd5f7190 - branch: 15161-config-check-rake-fix

Re-adds the config:check task as a no-op, because build/rails-package-scripts/postinst.sh assumes that the task exists on all rails apps.

#8 Updated by Lucas Di Pentima 5 months ago

As requested by Peter, I did a manual test by invalidating the config.yml file and running rake config:check. This is the result:

lucas@veritas:~/arvados/services/api$ rake config:check
rake aborted!
missing Services.SSO.ExternalURL
/media/psf/arvados/services/api/lib/config_loader.rb:93:in `block in coercion_and_check'
/media/psf/arvados/services/api/lib/config_loader.rb:80:in `each'
/media/psf/arvados/services/api/lib/config_loader.rb:80:in `coercion_and_check'
/media/psf/arvados/services/api/config/arvados_config.rb:220:in `<top (required)>'
/media/psf/arvados/services/api/config/application.rb:48:in `require_relative'
/media/psf/arvados/services/api/config/application.rb:48:in `<class:Application>'
/media/psf/arvados/services/api/config/application.rb:44:in `<module:Server>'
/media/psf/arvados/services/api/config/application.rb:43:in `<top (required)>'
/media/psf/arvados/services/api/rakefile:9:in `require'
/media/psf/arvados/services/api/rakefile:9:in `<top (required)>'
/home/lucas/.rvm/gems/ruby-2.4.4@test/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/home/lucas/.rvm/gems/ruby-2.4.4@test/bin/ruby_executable_hooks:24:in `eval'
/home/lucas/.rvm/gems/ruby-2.4.4@test/bin/ruby_executable_hooks:24:in `<main>'
(See full trace by running task with --trace)

#9 Updated by Tom Morris 4 months ago

  • Related to Bug #15225: [API] installation gives error: Creating git_internal_dir ''...mkdir: cannot create directory ‘’: No such file or directory added

#10 Updated by Tom Morris 4 months ago

  • Target version changed from To Be Groomed to 2019-05-22 Sprint
  • Assigned To set to Lucas Di Pentima
  • Status changed from New to In Progress
  • Project changed from OPS to Arvados

Lucas - did your commit fix this? Can it be closed?

#11 Updated by Lucas Di Pentima 4 months ago

Tom: Yes, forgot to close it, doing it now.

#12 Updated by Lucas Di Pentima 4 months ago

  • Status changed from In Progress to Resolved

#13 Updated by Tom Morris 4 months ago

  • Release set to 15

Also available in: Atom PDF