Project

General

Profile

Actions

Bug #22349

closed

RHEL8 Appstream Ruby not useable on 3.0

Added by Ivan Diaz Alvarez 26 days ago. Updated 9 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Story points:
-
Release relationship:
Auto

Description

We did an ARVQA update with 3.0 and the 3.1 Ruby Appstream, as recommended in the upgrade notes, this couldn't progress because of unhandled passenger/nginx dependencies, RHEL8 doesn't handle those gracefully, so in the end we are upgrading with RVM/Ruby 2.7.2 which caused no problems (we did a RVM/3.1 install but required many manual steps).

How can the upgrade notes be followed on a RHEL8 installation (with EPEL)? What versions can be used?


Subtasks 1 (1 open0 closed)

Task #22362: Review 22349-deploy-bundle-passengerIn ProgressBrett Smith12/12/2024Actions

Related issues 3 (1 open2 closed)

Related to Arvados - Bug #22389: Single-host single-hostname installation fails: Rails API server cannot start; /etc/arvados/config.yml "permission deniedResolvedBrett SmithActions
Related to Arvados - Bug #22391: Standardize systemd service handlingNewActions
Precedes Arvados - Bug #22396: Standalone Passenger documentation updatesResolvedBrett SmithActions
Actions #1

Updated by Brett Smith 26 days ago

Confirmed that the very latest Passenger package for RHEL8 is still dutifully built against the original system Ruby, 2.5, and depends on that specific version. Therefore it is not possible to install that package alongside our new API server package.

Our API server package already installs the Passenger gem for the same version of Ruby it's using. I propose we stop running the RailsAPI server through nginx and instead ship a systemd service to run Passenger standalone. That should be safe since controller is the only client with direct access, and this would simplify installation on all platforms: users would no longer need to (explicitly) install Passenger or configure it inside nginx.

Actions #2

Updated by Brett Smith 24 days ago

  • Status changed from New to In Progress

Now that we expect to use distro Ruby arvados-api-server needs to declare its dependencies on: ruby, ruby-devel, zlib-devel (we do this for Debian but not Red Hat).

If we do go ahead with the systemd plan, the postinst script needs to be updated to remove detection of a web server. We'll need to figure out another ownership plan.

Actions #3

Updated by Brett Smith 24 days ago

102 gems installed                                                                                                                                                                                                                                             
 done.                                                                                                                                                                                                                                                         
Running bundle install...Don't run Bundler as root. Installing your bundle as root will break this application for all non-root users on this machine.
 done.                                                                                                                         
Verifying bundle is complete... done.                                                                                          

Assumption: nginx is configured to serve Rails from                                                                            
            /var/www/arvados-api/current           
Assumption: nginx and passenger run as nginx

Creating symlinks to configuration in /etc/arvados/api .../var/tmp/rpm-tmp.wAs9IP: line 204: cmp: command not found              
... done.                                                                                                                      
Ensuring directory and file permissions ...chown: cannot access '/var/www/arvados-api/current/db/schema.rb': No such file or directory
... done.                                                                                                                      
Checking configuration for completeness...time="2024-11-27T10:56:55.100467296-05:00" level=warning msg="deprecated or unknown config entry: Clusters.z2a09.Containers.CloudVMs.DeployRunnerDirectory" 
time="2024-11-27T10:56:55.103973167-05:00" level=warning msg="Clusters.z2a09.Collections.BlobSigningKey: secret token is not set (use 32+ random characters from a-z, A-Z, 0-9)" 
time="2024-11-27T10:56:55.103991844-05:00" level=warning msg="LocalKeepBlobBuffersPerVCPU is 1 but will not be used because at least one volume (SAMPLE) uses AccessViaHosts -- suggest changing to 0" 
Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache does not exist
 done.                                                                                                                         
NOTE: The arvados-api-server package was not configured completely because                                                       
database setup could not be completed.                                                                                         

There are a few minor issues here that should be addressed but I don't understand why it says database setup wasn't completed, there's nothing in here about that.

Actions #4

Updated by Peter Amstutz 24 days ago

Brett Smith wrote in #note-3:

[...]

There are a few minor issues here that should be addressed but I don't understand why it says database setup wasn't completed, there's nothing in here about that.

It may just be that the script says that if there's any error because the job of the postinst script is mainly to set up the database.

Actions #5

Updated by Brett Smith 24 days ago

+ run_and_report 'Setting up database' bin/rake db:schema:load db:seed
+ local 'action_message=Setting up database'
+ shift
+ local retcode=0
+ echo -n 'Setting up database...'
Setting up database...+ bin/rake db:schema:load db:seed
WARN[2024-11-27T11:49:11.771649185-05:00] deprecated or unknown config entry: Clusters.z2a09.Containers.CloudVMs.DeployRunnerDirectory
WARN[2024-11-27T11:49:11.774778259-05:00] LocalKeepBlobBuffersPerVCPU is 1 but will not be used because at least one volume (SAMPLE) uses AccessViaHosts -- suggest changing to 0
Defaulting to memory cache, because /var/www/arvados-api/current/tmp/cache does not exist
rake aborted!
ActionView::Template::Error: both URI are relative
/var/www/arvados-api/current/app/views/admin_notifier/new_user.text.erb:14:in `_app_views_admin_notifier_new_user_text_erb__2952295690788535752_10640'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/base.rb:244:in `public_send'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/base.rb:244:in `_run'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/template.rb:157:in `block in render'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/notifications.rb:208:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/template.rb:361:in `instrument_render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/template.rb:155:in `render'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/template_renderer.rb:65:in `block (2 levels) in render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/notifications.rb:208:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/template_renderer.rb:60:in `block in render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/template_renderer.rb:79:in `render_with_layout'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/template_renderer.rb:59:in `render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/template_renderer.rb:11:in `render'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/rendering.rb:117:in `block in _render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/base.rb:270:in `in_rendering_context'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/rendering.rb:116:in `_render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/rendering.rb:103:in `render_to_body'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionpack-7.0.8.4/lib/abstract_controller/rendering.rb:27:in `render'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:992:in `block in collect_responses_from_templates'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:989:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:989:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:989:in `map'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:989:in `collect_responses_from_templates'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:967:in `collect_responses'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:875:in `mail'
/var/www/arvados-api/current/app/mailers/admin_notifier.rb:21:in `new_user'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionpack-7.0.8.4/lib/abstract_controller/base.rb:215:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionpack-7.0.8.4/lib/abstract_controller/callbacks.rb:234:in `block in process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:99:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionpack-7.0.8.4/lib/abstract_controller/callbacks.rb:233:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionpack-7.0.8.4/lib/abstract_controller/base.rb:151:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/rescuable.rb:27:in `block in process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/rescuable.rb:19:in `handle_exceptions'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/rescuable.rb:26:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/rendering.rb:39:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:646:in `block in process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/notifications.rb:206:in `block in instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/notifications.rb:206:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:645:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/message_delivery.rb:128:in `block in processed_mailer'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/andand-1.3.3/lib/andand.rb:60:in `me'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/message_delivery.rb:127:in `processed_mailer'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/message_delivery.rb:118:in `deliver_now'
/var/www/arvados-api/current/app/models/user.rb:926:in `send_admin_notifications'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:400:in `block in make_lambda'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:236:in `block in halting_and_conditional'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:599:in `block in invoke_after'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:599:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:599:in `invoke_after'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:108:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:929:in `_run_create_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/callbacks.rb:459:in `_create_record'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/timestamp.rb:108:in `_create_record'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/persistence.rb:1069:in `create_or_update'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/callbacks.rb:455:in `block in create_or_update'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/autosave_association.rb:370:in `around_save_collection_association'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:138:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/callbacks.rb:455:in `create_or_update'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/timestamp.rb:126:in `create_or_update'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/persistence.rb:648:in `save!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/validations.rb:53:in `save!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/transactions.rb:302:in `block in save!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/transactions.rb:302:in `save!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/suppressor.rb:54:in `save!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/persistence.rb:55:in `create!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:870:in `_create!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:115:in `block in create!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:881:in `_scoping'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:428:in `scoping'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:115:in `create!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:124:in `first_or_create!'
/var/www/arvados-api/current/lib/current_api_client.rb:61:in `block in system_user'
/var/www/arvados-api/current/lib/current_api_client.rb:234:in `check_cache'
/var/www/arvados-api/current/lib/current_api_client.rb:59:in `system_user'
/var/www/arvados-api/current/app/models/database_seeds.rb:11:in `block in install'
/var/www/arvados-api/current/lib/update_permissions.rb:280:in `batch_update_permissions'
/var/www/arvados-api/current/app/models/database_seeds.rb:10:in `install'
/var/www/arvados-api/current/db/seeds.rb:9:in `<top (required)>'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/railties-7.0.8.4/lib/rails/engine.rb:557:in `load'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/railties-7.0.8.4/lib/rails/engine.rb:557:in `block in load_seed'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/execution_wrapper.rb:92:in `wrap'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/railties-7.0.8.4/lib/rails/engine.rb:626:in `block (2 levels) in <class:Engine>'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:127:in `instance_exec'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:138:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/railties-7.0.8.4/lib/rails/engine.rb:557:in `load_seed'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/tasks/database_tasks.rb:497:in `load_seed'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/railties/databases.rake:397:in `block (2 levels) in <top (required)>'

Caused by:
URI::BadURIError: both URI are relative
/var/www/arvados-api/current/app/views/admin_notifier/new_user.text.erb:14:in `_app_views_admin_notifier_new_user_text_erb__2952295690788535752_10640'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/base.rb:244:in `public_send'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/base.rb:244:in `_run'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/template.rb:157:in `block in render'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/notifications.rb:208:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/template.rb:361:in `instrument_render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/template.rb:155:in `render'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/template_renderer.rb:65:in `block (2 levels) in render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/notifications.rb:208:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/template_renderer.rb:60:in `block in render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/template_renderer.rb:79:in `render_with_layout'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/template_renderer.rb:59:in `render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/template_renderer.rb:11:in `render'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/rendering.rb:117:in `block in _render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/base.rb:270:in `in_rendering_context'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/rendering.rb:116:in `_render_template'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/rendering.rb:103:in `render_to_body'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionpack-7.0.8.4/lib/abstract_controller/rendering.rb:27:in `render'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:992:in `block in collect_responses_from_templates'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:989:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:989:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:989:in `map'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:989:in `collect_responses_from_templates'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:967:in `collect_responses'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:875:in `mail'
/var/www/arvados-api/current/app/mailers/admin_notifier.rb:21:in `new_user'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionpack-7.0.8.4/lib/abstract_controller/base.rb:215:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionpack-7.0.8.4/lib/abstract_controller/callbacks.rb:234:in `block in process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:99:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionpack-7.0.8.4/lib/abstract_controller/callbacks.rb:233:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionpack-7.0.8.4/lib/abstract_controller/base.rb:151:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/rescuable.rb:27:in `block in process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/rescuable.rb:19:in `handle_exceptions'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/rescuable.rb:26:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionview-7.0.8.4/lib/action_view/rendering.rb:39:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:646:in `block in process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/notifications.rb:206:in `block in instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/notifications.rb:206:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/base.rb:645:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/message_delivery.rb:128:in `block in processed_mailer'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/andand-1.3.3/lib/andand.rb:60:in `me'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/message_delivery.rb:127:in `processed_mailer'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/actionmailer-7.0.8.4/lib/action_mailer/message_delivery.rb:118:in `deliver_now'
/var/www/arvados-api/current/app/models/user.rb:926:in `send_admin_notifications'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:400:in `block in make_lambda'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:236:in `block in halting_and_conditional'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:599:in `block in invoke_after'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:599:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:599:in `invoke_after'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:108:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:929:in `_run_create_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/callbacks.rb:459:in `_create_record'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/timestamp.rb:108:in `_create_record'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/persistence.rb:1069:in `create_or_update'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/callbacks.rb:455:in `block in create_or_update'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/autosave_association.rb:370:in `around_save_collection_association'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:138:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/callbacks.rb:455:in `create_or_update'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/timestamp.rb:126:in `create_or_update'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/persistence.rb:648:in `save!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/validations.rb:53:in `save!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/transactions.rb:302:in `block in save!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/transactions.rb:302:in `save!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/suppressor.rb:54:in `save!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/persistence.rb:55:in `create!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:870:in `_create!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:115:in `block in create!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:881:in `_scoping'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:428:in `scoping'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:115:in `create!'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:124:in `first_or_create!'
/var/www/arvados-api/current/lib/current_api_client.rb:61:in `block in system_user'
/var/www/arvados-api/current/lib/current_api_client.rb:234:in `check_cache'
/var/www/arvados-api/current/lib/current_api_client.rb:59:in `system_user'
/var/www/arvados-api/current/app/models/database_seeds.rb:11:in `block in install'
/var/www/arvados-api/current/lib/update_permissions.rb:280:in `batch_update_permissions'
/var/www/arvados-api/current/app/models/database_seeds.rb:10:in `install'
/var/www/arvados-api/current/db/seeds.rb:9:in `<top (required)>'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/railties-7.0.8.4/lib/rails/engine.rb:557:in `load'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/railties-7.0.8.4/lib/rails/engine.rb:557:in `block in load_seed'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/execution_wrapper.rb:92:in `wrap'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/railties-7.0.8.4/lib/rails/engine.rb:626:in `block (2 levels) in <class:Engine>'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:127:in `instance_exec'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activesupport-7.0.8.4/lib/active_support/callbacks.rb:138:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/railties-7.0.8.4/lib/rails/engine.rb:557:in `load_seed'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/tasks/database_tasks.rb:497:in `load_seed'
/var/www/arvados-api/shared/vendor_bundle/ruby/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/railties/databases.rake:397:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:seed
(See full trace by running task with --trace)
+ retcode=1
+ echo ' failed.'
 failed.
+ return 1
+ NOT_READY_REASON='database setup could not be completed'
+ '[' -n nginx ']'
+ chown -R nginx: /var/www/arvados-api/current/tmp
+ chmod -R 2775 /var/www/arvados-api/current/tmp
+ '[' -z 'database setup could not be completed' ']'
+ '[' -n 'database setup could not be completed' ']'
+ cat
NOTE: The arvados-api-server package was not configured completely because
database setup could not be completed.
Actions #6

Updated by Brett Smith 24 days ago

So yes, the real error is:

  • the default new user template refers to <%= Rails.configuration.Services.Workbench1.ExternalURL %>
  • I hadn't set that URL (because I was trying to throw this together as fast as possible, so I hadn't set any external URLs, and especially why would I set a URL for an obsolete service?)
  • and even though we run rake config:check earlier, apparently the first time that template tries to get instantiated is when we start running database tasks
  • so if you don't have the configuration defined, the template can't be instantiated, "database setup" fails, and you get no help from the output
Actions #7

Updated by Brett Smith 24 days ago

Brett Smith wrote in #note-6:

  • so if you don't have the configuration defined, the template can't be instantiated, "database setup" fails, and you get no help from the output

Specifically, we went down the else branch of the prepare_database postinst function, which specifically doesn't show the output of the rake task. But even if it did, there is absolutely no way a regular user could've figured out the problem from the output. The "real" error message is ActionView::Template::Error: both URI are relative, and even if you find it amongst the gigantic stack trace, it gives you zero information about which template has the problem or where this might be under your control. The only reason I could find it is by figuring if we don't see it on "normal" clusters, then the problem is likely somewhere in cluster configuration, and then going through the templates in there and figuring it out.

Actions #8

Updated by Peter Amstutz 24 days ago

Brett Smith wrote in #note-6:

So yes, the real error is:

  • the default new user template refers to <%= Rails.configuration.Services.Workbench1.ExternalURL %>
  • I hadn't set that URL (because I was trying to throw this together as fast as possible, so I hadn't set any external URLs, and especially why would I set a URL for an obsolete service?)

FWIW, and to be fair I don't know if we wrote down the decision anywhere, but I think where we ended up was that Workbench1.ExternalURL and Workbench2.ExternalURL need to be set to the same value, indefinitely, to maintain backwards compatibility for clients that read those configuration values.

  • and even though we run rake config:check earlier, apparently the first time that template tries to get instantiated is when we start running database tasks

Sounds like the place to catch this would be having config:check enforce setting Workbench1.ExternalURL.

Actions #9

Updated by Brett Smith 19 days ago

22349-deploy-bundle-passenger @ e0672ea6373de84b7b839338f6b9f1144cfbd637

  • All agreed upon points are implemented / addressed.
    • This adds a service definition to the Rails API package to run it using a standalone Passenger server that's already included in the bundle. This lets us run Passenger with the same version of Ruby as the Rails API server and eliminates the need for a separate Passenger install.
  • Anything not implemented (discovered or discussed during work) has a follow-up story.
    • Passenger complains about a security update available at startup, I need to file a separate issue about that, as well as the initial config challenges noted above.
  • Code is tested and passing, both automated and manual, what manual testing was done is described
    • I was able to build the package on my system, install it in a fresh Alma 8 VM following the new instructions, and curl the discovery document.
  • Documentation has been updated.
    • Yes
  • Behaves appropriately at the intended scale (describe intended scale).
    • There should be no change in scale, because access to the RailsAPI server is mediated through controller, and none of those settings have changed.
  • Considered backwards and forwards compatibility issues between client and server.
    • This change will conflict with existing setups, as documented in the upgrade note. I hope the migration is straightforward enough to be worth it.
  • Follows our coding standards and GUI style guidelines.
    • N/A (no style guide for any of these languages)
Actions #10

Updated by Brett Smith 15 days ago

  • Release set to 75
Actions #11

Updated by Lucas Di Pentima 15 days ago

I'm doing and e2e test by deploying a Debian11 Arvados 3.0 test cluster with the installer and then:

  1. Build arvados-api-server and arvados-server deb packages from this branch
  2. Install them on the controller node
  3. Apply the installer changes and re-deploy

The issue I observed:

  • The new arvados-railsapi service wasn't able to start
    • Fixed by Brett's suggestion of adding the following to the service file:
      [Service]
      Environment=HOME=/var/www/arvados-api
      
  • RailsAPI errors out with the following message on production.log:
    root@controller:/var/www/arvados-api# tail shared/log/production.log
    [ E 2024-12-06 22:06:43.5013 32308/T8p age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/arvados-api/current: An internal error occurred while preparing to spawn an application process: Cannot get user database entry for user UID 63563; it looks like your system's user database is broken, please fix it.
    
    • I'm not sure from where this 63563 user id comes from, yet.
Actions #12

Updated by Brett Smith 15 days ago

Lucas Di Pentima wrote in #note-11:

I'm not sure from where this 63563 user id comes from, yet.

It's the UID for the dynamic user created by systemd for the service. It does exist, but I'm guessing the prebuilt Passenger binary doesn't use modern enough libraries to find it. I will have to stop using DynamicUser and make the service less secure for everyone in order to support our older distributions.

Actions #13

Updated by Brett Smith 14 days ago

Now at b22b3ff1436294d25a278ce81e0f723632d96826. b1574b59e072a622e0f771853b3291e3405b7bcf is the fix for the various user issues, then there are some other refinements. From here I was able to build and install a bullseye package and then get the discovery document as before.

Actions #14

Updated by Lucas Di Pentima 12 days ago

This LGTM, thanks!

Actions #15

Updated by Brett Smith 10 days ago

I have added some commits to the branch to make the package testable, add a little testing of the systemd configuration, and then fix some small distro-specific bugs. Now at bab651830b8053c8da743088fb18f8c24e39687d; please review the changes after b22b3ff1436294d25a278ce81e0f723632d96826. build-packages-multijob: #4461

Actions #16

Updated by Lucas Di Pentima 10 days ago

LGTM, thanks.

Actions #17

Updated by Brett Smith 10 days ago

  • Related to Bug #22389: Single-host single-hostname installation fails: Rails API server cannot start; /etc/arvados/config.yml "permission denied added
Actions #18

Updated by Brett Smith 10 days ago

  • Related to Bug #22391: Standardize systemd service handling added
Actions #19

Updated by Brett Smith 9 days ago

  • Status changed from In Progress to Resolved
Actions #20

Updated by Brett Smith 8 days ago

  • Precedes Bug #22396: Standalone Passenger documentation updates added
Actions

Also available in: Atom PDF