Arvados

This is on Arvados 2.2.0, on AWS.

I had a cluster with `UsePreemtibleInstances: true`, and there were capacity issues with the availability of spot instances. I changed `UsePreemptibleInstances: false` and restarted a-d-c. The existing, queued containers all had `Preemptible: true` in their definitions, and a-d-c tried to schedule them. The API server responded with `ActiveRecord::RecordInvalid: Validation failed: Scheduling parameters preemptible instances are not allowed`, many times. See example below.

{"method":"PUT","path":"/arvados/v1/containers/2xpu4-dz642-vevpipd27zhqv8k","format":"html","controller":"Arvados::V1::ContainersController","action":"update","status":422,"duration":182.41,"view":0.26,"db":18.34,"request_id":"req-1kzvt393izax7bfbahgu","client_ipaddr":"10.252.103.227","client_auth":"2xpu4-gj3su-000000000000000","exception":"#\u003cActiveRecord::RecordInvalid: Validation failed: Scheduling parameters preemptible instances are not allowed\u003e","exception_backtrace":"/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/validations.rb:80:in `raise_validation_error'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/validations.rb:52:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:315:in `block in save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:212:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:315:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/suppressor.rb:48:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/persistence.rb:441:in `block in update!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:212:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/persistence.rb:439:in `update!'\n/var/www/arvados-api/current/app/models/container.rb:723:in `block (4 levels) in handle_completed'\n/var/www/arvados-api/current/lib/arvados_model_updates.rb:16:in `leave_modified_by_user_alone'\n/var/www/arvados-api/current/app/models/container.rb:722:in `block (3 levels) in handle_completed'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/relation/delegation.rb:71:in `each'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/relation/delegation.rb:71:in `each'\n/var/www/arvados-api/current/app/models/container.rb:721:in `block (2 levels) in handle_completed'\n/var/www/arvados-api/current/lib/current_api_client.rb:138:in `block in act_as_system_user'\n/var/www/arvados-api/current/lib/current_api_client.rb:149:in `act_as_user'\n/var/www/arvados-api/current/lib/current_api_client.rb:137:in `act_as_system_user'\n/var/www/arvados-api/current/app/models/container.rb:675:in `block in handle_completed'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/locking/pessimistic.rb:84:in `block in with_lock'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:212:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:301:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/locking/pessimistic.rb:82:in `with_lock'\n/var/www/arvados-api/current/app/models/container.rb:674:in `handle_completed'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:426:in `block in make_lambda'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:236:in `block in halting_and_conditional'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:517:in `block in invoke_after'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:517:in `each'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:517:in `invoke_after'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:133:in `run_callbacks'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:816:in `_run_save_callbacks'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/callbacks.rb:342:in `create_or_update'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/persistence.rb:308:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/validations.rb:52:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:315:in `block in save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:212:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:315:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/suppressor.rb:48:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/persistence.rb:441:in `block in update!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:212:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/persistence.rb:439:in `update!'\n/var/www/arvados-api/current/app/controllers/application_controller.rb:121:in `update'\n/var/www/arvados-api/current/app/controllers/arvados/v1/containers_controller.rb:33:in `block in update'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/locking/pessimistic.rb:84:in `block in with_lock'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `block in transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'\n/usr/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:212:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/transactions.rb:301:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/locking/pessimistic.rb:82:in `with_lock'\n/var/www/arvados-api/current/app/controllers/arvados/v1/containers_controller.rb:32:in `update'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/abstract_controller/base.rb:194:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/rendering.rb:30:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/abstract_controller/callbacks.rb:42:in `block in process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:109:in `block in run_callbacks'\n/var/www/arvados-api/current/app/controllers/application_controller.rb:430:in `block in set_current_request_id'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/tagged_logging.rb:71:in `block in tagged'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/tagged_logging.rb:28:in `tagged'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/tagged_logging.rb:71:in `tagged'\n/var/www/arvados-api/current/app/controllers/application_controller.rb:429:in `set_current_request_id'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:118:in `block in run_callbacks'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:136:in `run_callbacks'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/abstract_controller/callbacks.rb:41:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/rescue.rb:22:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/notifications.rb:168:in `block in instrument'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/notifications/instrumenter.rb:23:in `instrument'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/notifications.rb:168:in `instrument'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/instrumentation.rb:32:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/railties/controller_runtime.rb:24:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/abstract_controller/base.rb:134:in `process'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/rendering.rb:32:in `process'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal.rb:191:in `dispatch'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal.rb:252:in `dispatch'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/routing/route_set.rb:34:in `serve'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/journey/router.rb:52:in `block in serve'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/journey/router.rb:35:in `each'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/journey/router.rb:35:in `serve'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/routing/route_set.rb:840:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/omniauth-1.4.3/lib/omniauth/strategy.rb:186:in `call!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/omniauth-1.4.3/lib/omniauth/strategy.rb:164:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/omniauth-1.4.3/lib/omniauth/builder.rb:63:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/http/content_security_policy.rb:18:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/cookies.rb:670:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:98:in `run_callbacks'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/callbacks.rb:26:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/lograge-0.10.0/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.5/lib/rails/rack/logger.rb:26:in `block in call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/tagged_logging.rb:71:in `block in tagged'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/tagged_logging.rb:28:in `tagged'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/tagged_logging.rb:71:in `tagged'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.5/lib/rails/rack/logger.rb:26:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/request_store-1.4.1/lib/request_store/middleware.rb:19:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/request_id.rb:27:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/executor.rb:14:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/ssl.rb:74:in `call'\n/var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:67:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.5/lib/rails/engine.rb:524:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.5/lib/rails/railtie.rb:190:in `public_send'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.5/lib/rails/railtie.rb:190:in `method_missing'\n/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'\n/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request'\n/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'\n/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'\n/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'\n","params":{"container":"{\"log\":\"f117e618dadc8378efd4ef3f90103669+2472\",\"state\":\"Cancelled\"}"},"@timestamp":"2021-06-08T18:53:43.993866277Z","@version":"1","message":"[422] PUT /arvados/v1/containers/2xpu4-dz642-vevpipd27zhqv8k (Arvados::V1::ContainersController#update)"}

It would probably be better to automatically cancel containers that are submitted with an invalid setting for the `Preemptible` field.

In the end I had to cancel every container myself, and I was only able to do that after I had worked around bug #17774.