Bug #15161
closed
Dev clusters are failing: uninitialized constant ArInternalMetadatum (NameError)
Added by Nico César almost 5 years ago.
Updated almost 5 years ago.
Release relationship:
Auto
Description
App 51878 stderr: The PGconn, PGresult, and PGError constants are deprecated, and will be
App 51878 stderr: removed as of version 1.0.
App 51878 stderr:
App 51878 stderr: You should use PG::Connection, PG::Result, and PG::Error instead, respectively.
App 51878 stderr:
App 51878 stderr: Called from /var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
App 51878 stdout:
[ 2019-04-29 13:28:23.8920 48517/7f99d808c700 age/Cor/App/Implementation.cpp:304 ]: Could not spawn process for application /var/www/c97qk.arvadosapi.com/current: An error occurred while starting up the preloader.
Error ID: d1d121c0
Error details saved to: /tmp/passenger-error-CedvzV.html
Message from application: uninitialized constant ArInternalMetadatum (NameError)
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/inflector/methods.rb:261:in `const_get'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/inflector/methods.rb:261:in `block in constantize'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/inflector/methods.rb:259:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/inflector/methods.rb:259:in `inject'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/inflector/methods.rb:259:in `constantize'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/core_ext/string/inflections.rb:66:in `constantize'
/var/www/arvados-api/current/lib/can_be_an_owner.rb:19:in `block in included'
/var/www/arvados-api/current/lib/can_be_an_owner.rb:15:in `each'
/var/www/arvados-api/current/lib/can_be_an_owner.rb:15:in `included'
/var/www/arvados-api/current/app/models/group.rb:12:in `include'
/var/www/arvados-api/current/app/models/group.rb:12:in `<class:Group>'
/var/www/arvados-api/current/app/models/group.rb:8:in `<top (required)>'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:472:in `block (2 levels) in eager_load!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:471:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:471:in `block in eager_load!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:469:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:469:in `eager_load!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:346:in `eager_load!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application/finisher.rb:56:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application/finisher.rb:56:in `block in <module:Finisher>'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `instance_exec'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `run'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:55:in `block in run_initializers'
/usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:228:in `block in tsort_each'
/usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:431:in `each_strongly_connected_component_from'
/usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:349:in `block in each_strongly_connected_component'
/usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:347:in `each'
/usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:347:in `call'
/usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:347:in `each_strongly_connected_component'
/usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:226:in `tsort_each'
/usr/local/rvm/rubies/ruby-2.3.5/lib/ruby/2.3.0/tsort.rb:205:in `tsort_each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:54:in `run_initializers'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:352:in `initialize!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `public_send'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `method_missing'
/var/www/arvados-api/current/config/environment.rb:10:in `<top (required)>'
config.ru:7:in `require'
config.ru:7:in `block in <main>'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.6.11/lib/rack/builder.rb:55:in `instance_eval'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.6.11/lib/rack/builder.rb:55:in `initialize'
config.ru:1:in `new'
config.ru:1:in `<main>'
/usr/share/passenger/helper-scripts/rack-preloader.rb:110:in `eval'
/usr/share/passenger/helper-scripts/rack-preloader.rb:110:in `preload_app'
/usr/share/passenger/helper-scripts/rack-preloader.rb:156:in `<module:App>'
/usr/share/passenger/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
/usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<main>'
Checking c97qk's files, it seems that the structure.sql file is from rails 5 but the rest of the code is from 4.2.
The ar_internal_metadata table causing this error was added on 261543dfc3, and the Gemfile on c97qk asks for rails 4.2. Also, the app/models/arvados_model.rb file is from rails 4.2 because the class is subclassing ActiveRecord::Base instead of ApplicationRecord
- Project changed from 35 to 40
Lucas Di Pentima wrote:
Checking c97qk's files, it seems that the structure.sql file is from rails 5 but the rest of the code is from 4.2.
The ar_internal_metadata table causing this error was added on 261543dfc3, and the Gemfile on c97qk asks for rails 4.2. Also, the app/models/arvados_model.rb file is from rails 4.2 because the class is subclassing ActiveRecord::Base instead of ApplicationRecord
Mmmh the problem might be on our side: Dev clusters should be only DEV!
c97qk:~# apt-cache policy arvados-api-server| head
arvados-api-server:
Installed: 1.3.2-1
Candidate: 1.3.2-1
Version table:
*** 1.3.2-1 500
500 http://apt.arvados.org xenial/main amd64 Packages
100 /var/lib/dpkg/status
1.3.1.20190422202321-8 500
500 http://apt.arvados.org xenial-dev/main amd64 Packages
1.3.1.20190422184531-8 500
Now that #15163 is out of the way I tried to downgrade the API server ... ended up with another issue:
c97qk:~# apt-get install arvados-api-server=1.3.1.20190422202321-8
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
arvados-api-server
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 320 not upgraded.
Need to get 0 B/28.1 MB of archives.
After this operation, 1,262 kB disk space will be freed.
Do you want to continue? [Y/n]
dpkg: warning: downgrading arvados-api-server from 1.3.2-1 to 1.3.1.20190422202321-8
(Reading database ... 110469 files and directories currently installed.)
Preparing to unpack .../arvados-api-server_1.3.1.20190422202321-8_amd64.deb ...
Unpacking arvados-api-server (1.3.1.20190422202321-8) over (1.3.2-1) ...
Setting up arvados-api-server (1.3.1.20190422202321-8) ...
Assumption: nginx is configured to serve Rails from
/var/www/arvados-api/current
Assumption: nginx and passenger run as www-data
Creating symlinks to configuration in /etc/arvados/api ...... done.
Running bundle install... done.
Ensuring directory and file permissions ...... done.
Running db:migrate...Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache owner (uid=33) is not me (uid=0)
DEPRECATION WARNING: `config.serve_static_files` is deprecated and will be removed in Rails 5.1.
Please use `config.public_file_server.enabled = false` instead.
(called from block in <top (required)> at /var/www/arvados-api/current/config/environments/production.rb:12)
done.
Checking application.yml for completeness...Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache owner (uid=33) is not me (uid=0)
rake aborted!
Don't know how to build task 'config:check' (See the list of available tasks with `rake --tasks`)
/var/www/arvados-api/shared/vendor_bundle/ruby/2.3.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli/exec.rb:75:in `load'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli/exec.rb:75:in `kernel_load'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli.rb:424:in `exec'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli.rb:27:in `dispatch'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/cli.rb:18:in `start'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/exe/bundle:30:in `block in <top (required)>'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/lib/bundler/friendly_errors.rb:122:in `with_friendly_errors'
/usr/local/rvm/gems/ruby-2.3.5/gems/bundler-1.16.0/exe/bundle:22:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.3.5/bin/bundle:23:in `load'
/usr/local/rvm/gems/ruby-2.3.5/bin/bundle:23:in `<main>'
/usr/local/rvm/gems/ruby-2.3.5/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.3.5/bin/ruby_executable_hooks:15:in `<main>'
(See full trace by running task with --trace)
failed.
Precompiling assets... skipped.
Initializing git_internal_dir... skipped.
PLEASE NOTE:
The arvados-api-server package was not configured completely because
/etc/arvados/api/application.yml needs some tweaking.
Please refer to the documentation at
<http://doc.arvados.org/install/install-api-server.html#configure> for more details.
When application.yml has been modified,
reconfigure or reinstall this package.
Tomorrow we'll investigate why
Don't know how to build task 'config:check'
The config:check rake task was removed on 2ea729f97166cb5fda4a6ca0818f0a0ab15388ab -- if this is part of a unified rails package building mechanism, maybe we should have made it a no-op instead of deleting it?
Update at 0cd5f7190 - branch: 15161-config-check-rake-fix
Re-adds the config:check task as a no-op, because build/rails-package-scripts/postinst.sh assumes that the task exists on all rails apps.
As requested by Peter, I did a manual test by invalidating the config.yml file and running rake config:check. This is the result:
lucas@veritas:~/arvados/services/api$ rake config:check
rake aborted!
missing Services.SSO.ExternalURL
/media/psf/arvados/services/api/lib/config_loader.rb:93:in `block in coercion_and_check'
/media/psf/arvados/services/api/lib/config_loader.rb:80:in `each'
/media/psf/arvados/services/api/lib/config_loader.rb:80:in `coercion_and_check'
/media/psf/arvados/services/api/config/arvados_config.rb:220:in `<top (required)>'
/media/psf/arvados/services/api/config/application.rb:48:in `require_relative'
/media/psf/arvados/services/api/config/application.rb:48:in `<class:Application>'
/media/psf/arvados/services/api/config/application.rb:44:in `<module:Server>'
/media/psf/arvados/services/api/config/application.rb:43:in `<top (required)>'
/media/psf/arvados/services/api/rakefile:9:in `require'
/media/psf/arvados/services/api/rakefile:9:in `<top (required)>'
/home/lucas/.rvm/gems/ruby-2.4.4@test/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/home/lucas/.rvm/gems/ruby-2.4.4@test/bin/ruby_executable_hooks:24:in `eval'
/home/lucas/.rvm/gems/ruby-2.4.4@test/bin/ruby_executable_hooks:24:in `<main>'
(See full trace by running task with --trace)
- Related to Bug #15225: [API] installation gives error: Creating git_internal_dir ''...mkdir: cannot create directory ‘’: No such file or directory added
- Project changed from 40 to Arvados
- Status changed from New to In Progress
- Assigned To set to Lucas Di Pentima
- Target version changed from To Be Groomed to 2019-05-22 Sprint
Lucas - did your commit fix this? Can it be closed?
Tom: Yes, forgot to close it, doing it now.
- Status changed from In Progress to Resolved
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by Tom Morris