Bug #16958

[arvbox] api server startup is unreliable

Added by Ward Vandewege about 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
10/07/2020
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Sometimes, the api server startup script ends up stuck in a loop like this:

2020-10-06_20:15:47.43910 App 78006 output: Error: The application encountered the following error: cannot load such file -- omniauth-oauth2 (LoadError)
2020-10-06_20:15:47.43912 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:34:in `require'
2020-10-06_20:15:47.43913 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:291:in `block in require'
2020-10-06_20:15:47.43913 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:257:in `load_dependency'
2020-10-06_20:15:47.43914 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:291:in `require'
2020-10-06_20:15:47.43914 App 78006 output:     /usr/src/arvados/services/api/lib/josh_id.rb:5:in `<top (required)>'
2020-10-06_20:15:47.43914 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
2020-10-06_20:15:47.43914 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
2020-10-06_20:15:47.43915 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
2020-10-06_20:15:47.43915 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
2020-10-06_20:15:47.43915 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/bootsnap-1.4.7/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
2020-10-06_20:15:47.43916 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:291:in `block in require'
2020-10-06_20:15:47.43916 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:257:in `load_dependency'
2020-10-06_20:15:47.43916 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:291:in `require'
2020-10-06_20:15:47.43916 App 78006 output:     /usr/src/arvados/services/api/config/environment.rb:7:in `<top (required)>'
2020-10-06_20:15:47.43917 App 78006 output:     config.ru:7:in `require'
2020-10-06_20:15:47.43917 App 78006 output:     config.ru:7:in `block in <main>'
2020-10-06_20:15:47.43917 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:125:in `instance_eval'
2020-10-06_20:15:47.43918 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:125:in `initialize'
2020-10-06_20:15:47.43918 App 78006 output:     config.ru:1:in `new'
2020-10-06_20:15:47.43918 App 78006 output:     config.ru:1:in `<main>'
2020-10-06_20:15:47.43918 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/src/helper-scripts/rack-preloader.rb:101:in `eval'
2020-10-06_20:15:47.43919 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/src/helper-scripts/rack-preloader.rb:101:in `preload_app'
2020-10-06_20:15:47.43919 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/src/helper-scripts/rack-preloader.rb:189:in `block in <module:App>'
2020-10-06_20:15:47.43919 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb:380:in `run_block_and_record_step_progress'
2020-10-06_20:15:47.43920 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/src/helper-scripts/rack-preloader.rb:188:in `<module:App>'
2020-10-06_20:15:47.43920 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/src/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
2020-10-06_20:15:47.43920 App 78006 output:     /var/lib/arvados/lib/ruby/gems/2.5.0/gems/passenger-6.0.2/src/helper-scripts/rack-preloader.rb:29:in `<main>'

Related issues

Related to Arvados - Bug #16267: arvbox build uses arvados-server installResolved09/24/2020

Related to Arvados - Bug #16955: Fix jenkins jobs arvados-cwl-conformance-tests and arv-federation-migrate-testResolved10/07/2020

Blocks Arvados - Story #16662: Release Arvados 2.1Resolved09/28/2020

History

#1 Updated by Ward Vandewege about 1 year ago

  • Status changed from New to In Progress

#2 Updated by Ward Vandewege about 1 year ago

  • Related to Bug #16267: arvbox build uses arvados-server install added

#3 Updated by Ward Vandewege about 1 year ago

#4 Updated by Ward Vandewege about 1 year ago

  • Description updated (diff)

#5 Updated by Ward Vandewege about 1 year ago

  • Status changed from In Progress to Closed

The fix for this is part of #16955

#6 Updated by Ward Vandewege about 1 year ago

  • Related to Bug #16955: Fix jenkins jobs arvados-cwl-conformance-tests and arv-federation-migrate-test added

Also available in: Atom PDF