Arvados

• Logged in as activated (admin) user on Tordo. Click "log out" on tordo:

https://tordo.arvadosapi.com/logout?return_to=https%3A%2F%2Fworkbench.tordo.arvadosapi.com%2F

{"errors":["configuration problem: exactly one of Login.Google, Login.OpenIDConnect, Login.SSO, Login.PAM, Login.LDAP, and Login.Test must be enabled"]}

=> created bug #16913

• Click "log in" on tordo, using an account that exists on ce8i5 but has is_active false:

https://workbench.tordo.arvadosapi.com/users#

Oh... fiddlesticks.
An error occurred when Workbench sent a request to the Arvados API server. Try reloading this page. If the problem is temporary, your request might go through next time. If that doesn't work, the information below can help system administrators track down the problem.

API request URL
https://tordo.arvadosapi.com/arvados/v1/users/ce8i5-tpzed-tn806jsy8sjhmpv/activate
API response {
":errors":[
"request failed: https://ce8i5.arvadosapi.com/arvados/v1/users/ce8i5-tpzed-tn806jsy8sjhmpv/activate: 422 Unprocessable Entity: request failed: http://localhost:8004/arvados/v1/users/ce8i5-tpzed-tn806jsy8sjhmpv/activate: 422 Unprocessable Entity: #<ArgumentError: Cannot activate without being invited.> (req-p695alwdpuif0g9mrg09)"
]
}
Report problem or email us if you suspect this is a bug.

=> created bug #16914

Workbench 2 doesn't load on tordo:

https://dev.arvados.org/issues/16915

=> fixed now by making sure the correct packages are loaded on the wb machine.

When loading wb2, this error appears in red at the bottom, briefly:

Getting current user for ce8i5.arvadosapi.com: Network Error

It re-appears on shift-reload.

=> new bug #16927

The API server logs are full of lines like these:

[req-2mm93s76c82r7shm8az5] Warning: received non-boolean parameter 'include_trash' on Arvados::V1::CollectionsController.
[req-2mm93s76c82r7shm8az5] Warning: received non-boolean parameter 'include_old_versions' on Arvados::V1::CollectionsController.

=> new bug #16928

Trying to share a collection as an ordinary user, but I see (on the view collection page, in wb1):

Sharing and permissions
Your API token is not authorized to manage collection sharing links.

1. Why? I see a request like this in the api server logs when loading https://workbench.tordo.arvadosapi.com/collections/tordo-4zz18-aam8gchmw53n426:

[req-7rl47pzbw1vscbqsdcj1] Error 1601328087+69320957: 403
{"method":"GET","path":"/arvados/v1/api_client_authorizations","format":"json","controller":"Arvados::V1::ApiClientAuthorizationsController","action":"index","status":403,"duration":5.63,"view":0.3,"db":1.75,"request_id":"req-7rl47pzbw1vscbqsdcj1","client_ipaddr":"10.253.0.41","client_auth":"ce8i5-gj3su-sqfolnetlyfrzpr","params":{"reader_tokens":"[\"v2/STRIPPED/STRIPPED\"]","_method":"GET","filters":"[[\"scopes\",\"=\",[\"GET /arvados/v1/collections/tordo-4zz18-aam8gchmw53n426\",\"GET /arvados/v1/collections/tordo-4zz18-aam8gchmw53n426/\",\"GET /arvados/v1/keep_services/accessible\"]]]","limit":"9223372036854775807","offset":"0"},"@timestamp":"2020-09-28T21:21:27.270244973Z","@version":"1","message":"[403] GET /arvados/v1/api_client_authorizations (Arvados::V1::ApiClientAuthorizationsController#index)"}

and in the controller logs:

Sep 28 22:06:25 tordo.arvadosapi.com arvados-controller[5343]: {"PID":5343,"RequestID":"req-6pwv2sl9s7y8ujz85v36","level":"info","msg":"response","remoteAddr":"127.0.0.1:35598","reqBytes":123,"reqForwardedFor":"10.253.0.41","reqHost":"tordo.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/collections/tordo-4zz18-aam8gchmw53n426","reqQuery":"","respBody":"{\"errors\":[\"request failed: http://localhost:8004/arvados/v1/collections/tordo-4zz18-aam8gchmw53n426?reader_tokens=%5B%22v2%2FSTRIPPED%2FSTRIPPED%22%5D: 404 Not Found: Path not found (req-6pwv2sl9s7y8ujz85v36)\"]}\n","respBytes":274,"respStatus":"Not Found","respStatusCode":404,"time":"2020-09-28T22:06:25.148905629Z","timeToStatus":0.012382,"timeTotal":0.012397,"timeWriteBody":0.000014}

2. Sharing appears to be undocumented, if this is a config issue, we need to document that better

=> #16923

WB2: trashing a tiny collection with 1 file seems absurdly slow, because there is a sequence of notifications in the bottom right that take multiple seconds, and they only seem to progress when there is some input in the browser. So if you leave the window alone, it will look like it is stuck:

(green) Moving to trash (I've seen it take more than a minute, why is this not instantaneous)?
(red) Project panel is not opened.
(green) Added to trash

And that "Project panel is not opened" seems like another bug

=> new bug #16932

WB2: after trashing a collection, the collection context menu (three dots, upper right) is not refreshed, which means that it still has the "Move to trash" menu item, rather than "Restore".

=> new bug #16933

[login federation] [experiment run from satellite cluster] [WB1]

As an active admin user, tried to re-run a workflow step (re-run, do not reuse, click 'copy and edit inputs', click 'Run >'), got 403 error (!):

#<ArvadosModel::PermissionDeniedError: ArvadosModel::PermissionDeniedError> (req-4pss653l1tdkosg6robc) [API: 403]

Workbench version
2.1.0.dev20200929215353 (5cefb12be01e9df6997f88696048ecb5d77d305e)

API version
2.1.0.dev20200929215353 (5cefb12be01e9df6997f88696048ecb5d77d305e)

After that error, I clicked the 'Run' button again, and then it worked!?! Cf. https://workbench.tordo.arvadosapi.com/container_requests/tordo-xvhdp-6sc6un34k9zub18.

I retried that experiment, got the 403, but no matter how many times I re-click that 'Run' button, I keep getting the 403:

https://workbench.tordo.arvadosapi.com/container_requests/tordo-xvhdp-i6mfusn3ukfc00g

Here's the API log entry from the original failed request:

[req-4pss653l1tdkosg6robc] Inactive user ce8i5-tpzed-xo2k4i24bjzwedw tried to update ContainerRequest tordo-xvhdp-6sc6un34k9zub18
#<ArvadosModel::PermissionDeniedError: ArvadosModel::PermissionDeniedError>
Error 1601487097+e385b38c: 403

{
  "method": "PUT",
  "path": "/arvados/v1/container_requests/tordo-xvhdp-6sc6un34k9zub18",
  "format": "json",
  "controller": "Arvados::V1::ContainerRequestsController",
  "action": "update",
  "status": 403,
  "duration": 43.79,
  "view": 0.28,
  "db": 14.91,
  "request_id": "req-4pss653l1tdkosg6robc",
  "client_ipaddr": "10.253.0.41",
  "client_auth": "ce8i5-gj3su-49qukxrij6udt67",
  "exception": "#<ArvadosModel::PermissionDeniedError: ArvadosModel::PermissionDeniedError>",
  "exception_backtrace": "/var/www/arvados-api/current/app/models/arvados_model.rb:600:in `ensure_permission_to_save'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:426:in `block in make_lambda'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:606:in `block (2 levels) in default_terminator'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:605:in `catch'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:605:in `block in default_terminator'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:199:in `block in halting'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:513:in `block in invoke_before'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:513:in `each'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:513:in `invoke_before'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:131:in `run_callbacks'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:816:in `_run_save_callbacks'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/callbacks.rb:342:in `create_or_update'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/persistence.rb:308:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/validations.rb:52:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/transactions.rb:315:in `block in save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/transactions.rb:212:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/transactions.rb:315:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/suppressor.rb:48:in `save!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/persistence.rb:441:in `block in update!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `block in transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'\n/usr/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/transactions.rb:212:in `transaction'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/persistence.rb:439:in `update!'\n/var/www/arvados-api/current/app/controllers/application_controller.rb:121:in `update'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/abstract_controller/base.rb:194:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/rendering.rb:30:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:109:in `block in run_callbacks'\n/var/www/arvados-api/current/app/controllers/application_controller.rb:430:in `block in set_current_request_id'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/tagged_logging.rb:28:in `tagged'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/tagged_logging.rb:71:in `tagged'\n/var/www/arvados-api/current/app/controllers/application_controller.rb:429:in `set_current_request_id'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:136:in `run_callbacks'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/abstract_controller/callbacks.rb:41:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/rescue.rb:22:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:168:in `block in instrument'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:168:in `instrument'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/railties/controller_runtime.rb:24:in `process_action'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/abstract_controller/base.rb:134:in `process'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.3/lib/action_view/rendering.rb:32:in `process'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal.rb:191:in `dispatch'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal.rb:252:in `dispatch'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:35:in `each'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:35:in `serve'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:840:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/omniauth-1.4.3/lib/omniauth/strategy.rb:186:in `call!'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/omniauth-1.4.3/lib/omniauth/strategy.rb:164:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/omniauth-1.4.3/lib/omniauth/builder.rb:63:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:98:in `run_callbacks'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/lograge-0.10.0/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/rack/logger.rb:26:in `block in call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/tagged_logging.rb:28:in `tagged'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/tagged_logging.rb:71:in `tagged'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/rack/logger.rb:26:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/request_store-1.4.1/lib/request_store/middleware.rb:19:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/executor.rb:14:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/ssl.rb:74:in `call'\n/var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:66:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/engine.rb:524:in `call'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/railtie.rb:190:in `public_send'\n/var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/railtie.rb:190:in `method_missing'\n/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'\n/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request'\n/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'\n/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads'\n/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'\n",
  "params": {
    "reader_tokens": "[\"STRIPPED_v2_TOKEN\"]",
    "container_request": "{\"properties\":{\"template_uuid\":null},\"state\":\"Committed\",\"mounts\":{\"/var/lib/cwl/cwl.input.json\":{\"kind\":\"json\",\"content\":{}},\"stdout\":{\"kind\":\"file\",\"path\":\"/var/spool/cwl/cwl.output.json\"},\"/var/spool/cwl\":{\"kind\":\"collection\",\"writable\":true},\"/var/lib/cwl/workflow.json\":{\"kind\":\"json\",\"content\":{\"$graph\":[{\"class\":\"CommandLineTool\",\"inputs\":[{\"type\":\"Directory\",\"default\":{\"class\":\"Directory\",\"listing\":[{\"class\":\"File\",\"location\":\"keep:ade9d0e032044bd7f58daaecc0d06bc6+51/hello.txt\"}]},\"id\":\"#default-dir3.cwl/inp2\"}],\"outputs\":[],\"arguments\":[\"echo\",\"$(inputs.inp2)\"],\"id\":\"#default-dir3.cwl\"},{\"class\":\"Workflow\",\"inputs\":[],\"outputs\":[],\"steps\":[{\"in\":[],\"out\":[],\"run\":\"#default-dir3.cwl\",\"id\":\"#main/step1\"}],\"id\":\"#main\"}],\"cwlVersion\":\"v1.0\"}}},\"runtime_constraints\":{\"vcpus\":1,\"ram\":1342177280,\"API\":true},\"environment\":{},\"command\":[\"arvados-cwl-runner\",\"--disable-reuse\",\"--local\",\"--api=containers\",\"--no-log-timestamps\",\"--disable-validate\",\"--disable-color\",\"--eval-timeout=20\",\"--thread-count=1\",\"--disable-reuse\",\"--collection-cache-size=256\",\"--on-error=continue\",\"/var/lib/cwl/workflow.json#main\",\"/var/lib/cwl/cwl.input.json\"],\"scheduling_parameters\":{}}",
    "_method": "PUT" 
  },
  "@timestamp": "2020-09-30T17:31:37.871941948Z",
  "@version": "1",
  "message": "[403] PUT /arvados/v1/container_requests/tordo-xvhdp-6sc6un34k9zub18 (Arvados::V1::ContainerRequestsController#update)" 
}

Update: there seems to be some sort of permission flapping going on. I'm seeing errors on wb2 as well, e.g. when trying to create a collection in the home project:

request failed: http://localhost:8004/arvados/v1/collections: 403 Forbidden: #<ArvadosModel::PermissionDeniedError: ArvadosModel::PermissionDeniedError> (req-1dq15i8u20g5nceqenyk)

This was after doing several things without issue. Retrying the things that worked before is now failing with that a permission denied error.

=> fixed in https://dev.arvados.org/issues/16914#note-9

[login federation] [experiment run from satellite cluster] [WB2]

Steps to reproduce:

• give another user can-manage on a project
• log in as the other user
• go to sharing dialog for the project
• change the sharing to "Only you can access / Private"

At this point the other user gets kicked out from viewing the project immediately. This is confusing UX. The "only you" means the user who owns the collection, not the 'other user' who has can-manage permissions on it.

TODO: rephrase to avoid confusion, or make it impossible for non-owners to change the sharing level (public/shared/private) on projects.
TODO: check if the same issue exists with collections, and if so, also fix that.

=> #16969

[login federation] [experiment run from satellite cluster] [WB2]

Steps to reproduce:

• create new project

Bug 1:

• go to the sharing dialog
• click 'Advanced'
• change sharing level from Private to Shared
• click 'SAVE'

At this point the sharing dialog loads again, no error is displayed, and the sharing level is reset to Private. Clearly, WB2 can't make this change (why?) but it is not showing an error message to the user.

TODO: either fix the behavior so that this change is possible, or make it impossible to select 'shared' from the dropdown, or make sure the error message is displayed to the user.

Bug 2:

• go to the sharing dialog
• click 'Advanced'
• change sharing level from Private to Public
• click 'SAVE'
• close sharing dialog, re-open it, verify that 'Public' was indeed saved
• change sharing level from Public to Shared
• click 'SAVE'

At this point the sharing dialog loads again, no error is displayed, and the sharing level is changed to Private.

Clearly, WB2 can't make this change (why?) but it is not showing an error message to the user. It is changing the level from Public to Private.

TODO: make sure the level is not changed to Private when the user didn't ask for that. Also, either fix the behavior so that this change is possible, or make it impossible to select 'shared' from the dropdown, or make sure the error message is displayed to the user.

=> #16970

[login federation] [experiment run from satellite cluster] [WB2]

Issues with Project and Collection name/description not being saved/shown in the edit menu.

Steps to reproduce:

Bug 1

• go to 'Projects'
• create new project, populate Project Name and Description fields
• favorite it
• go to 'my favorites'
• right click the project, edit it
• BUG: the Project Name and Description fields are empty

The exact same bug exists for collections.

Bug 2

• go to 'Projects'
• create new project, populate Project Name and Description field
• go to 'Projects'
• right click on the new project, edit it
• BUG: the Description field is empty

The exact same bug exists for collections.

Bug 3

• go to 'Projects'
• create new project, populate Project Name and Description field
• share the project with another user
• log in as the other user
• go to 'Shared with me'
• right click on the project, click 'edit'
• BUG: the Project Name and Description fields are empty

The exact same bug exists for collections.

=> #16971

[login federation] [experiment run from satellite cluster] [WB1]

When running a workflow, the container request view page never updates the workflow status, it remains stuck at 'Queued', even long after the workflow starts, until the page is reloaded.

=> #16972

[login federation] [experiment run from satellite cluster] [WB1]

No "disable re-use" option when running a newly registered workflow that is indentical to a previously registered workflow.

Steps to reproduce:

• register a workflow, e.g. `arvados-cwl-runner --name "CWL diagnostics hasher" --create-workflow hasher-workflow.cwl hasher-workflow-job.yml`
• click the 'Run this workflow' button, choose project, provide inputs, click 'Run'
• re-register the same workflow, e.g. `arvados-cwl-runner --name "CWL diagnostics hasher" --create-workflow hasher-workflow.cwl hasher-workflow-job.yml`
• click the 'Run this workflow' button, choose project, provide inputs, click 'Run'

Observed behavior: the previous run will be reused (e.g. https://workbench.tordo.arvadosapi.com/container_requests/tordo-xvhdp-viij3s0enut3r35).

With the 'Re-run' button on a container request that has completed, the user has to check a box to 'Enable container reuse'. There is no such checkbox when running a newly registered workflow.

=> #16973

Ward Vandewege wrote:

[login federation] [experiment run from satellite cluster] [WB2]

Steps to reproduce:

• create new project

Bug 1:

• go to the sharing dialog
• click 'Advanced'
• change sharing level from Private to Shared
• click 'SAVE'

At this point the sharing dialog loads again, no error is displayed, and the sharing level is reset to Private. Clearly, WB2 can't make this change (why?) but it is not showing an error message to the user.

This is bad UI.

Selecting "Shared" on its own doesn't do anything. You have to actually add a user or group to share with.

However, if it is actually shared with another user, and you change the state from "Shared" to "Private", then that does has the effect of deleting any sharing links.

Going from "Public" to "Shared" has the effect of deleting sharing with "Anonymous Users" but if you don't add a user to share with, it is equivalent to "Private".

[login federation] [experiment run from satellite cluster] [WB2]

Multisite search not returning results for cluster not part of login federation.

Steps to reproduce:

• go to the Site Manager
• add cluster (I used 9tee4.arvadosapi.com)
• verify that it shows as 'logged in'
• go to multisite search
• search for a term that should produce results from the newly added cluster (I used 'diagnostics')

BUG: no results from the newly added cluster are shown.

=> #16941

[login federation] [experiment run from satellite cluster] [WB2] / [WB1]

Strange multisite search behavior.

Steps to reproduce:

• [WB2] go to the Site Manager. Refresh the page until the login cluster is the second line (not the first).

That's a bug. Why is the 'Logged in' button greyed always out for the first item in the list, not for a real reason (e.g. it's the cluster you are on, and it's the login cluster in a federation)?

• [WB2] click the "Logged in" button for ce8i5 (the login cluster), which will log you out there
• [WB2] verify that it shows as 'logged out'
• switch WB1
• [WB1] go to multisite search
• [WB1] notices that it is not logged in on one of the clusters, and fixes that by redirecting to that cluster, which logs me in there transparently, but that leaves me on that cluster.

Bug: this is very confusing; it's hard to tell the workbenches apart, so now the user is suddenly on a different one than where they started...

search for a term that should produce results from the newly added cluster (I used 'diagnostics')

[peer federation] [WB2]

WB2 v2.1.0.dev20201001205235, which includes the fix from #16927

When loading wb2, this error appears in red at the bottom, briefly:

Getting current user for ce8i5.arvadosapi.com: Request failed with status code 401
It re-appears on shift-reload.

=> #16947

[peer federation] [WB1]

"Copy to project" broken for container requests, tested as an admin user.

Steps to reproduce:

• view a container request (e.g. 9tee4-xvhdp-2koqlwsdxxrzghz)
• click the "Copy to project" button
• choose a destination project (e.g. your Home project)
• click "Copy"

Bug: an error message appears, and the container request is not copied:

Container image can't be blank (req-b4ugb3ujjee7iihovaos); Output path can't be blank (req-b4ugb3ujjee7iihovaos)

Fixed in 02d601eb (see note 25)

Ward Vandewege wrote:

[peer federation] [WB1]

"Copy to project" broken for container requests, tested as an admin user.

Steps to reproduce:

• view a container request (e.g. 9tee4-xvhdp-2koqlwsdxxrzghz)
• click the "Copy to project" button
• choose a destination project (e.g. your Home project)
• click "Copy"

Bug: an error message appears, and the container request is not copied:

Container image can't be blank (req-b4ugb3ujjee7iihovaos); Output path can't be blank (req-b4ugb3ujjee7iihovaos)

I removed the option in workbench 1. You want to be using "Re-run".

This action also exists (and doesn't work) in Workbench 2, but removing/fixing that is a bigger project (=> #16130)

[peer federation] [WB1] [admin interface]

On ce8i5, 9tee4 remote clusters entry:

      9tee4:
        Host: 9tee4.arvadosapi.com
        Proxy: true

On 9tee4, the ce8i5 remote clusters entry:

      ce8i5:
        Host: ce8i5.arvadosapi.com
        Proxy: true

The user list in the admin interface lists entries with foreign UUIDs (their existance is normal). But clicking on 'show' leads to a 'Not found' error page, e.g. https://workbench.9tee4.arvadosapi.com/users/ce8i5-tpzed-o4njwilpp4ov286.

Does it make sense to enumerate the users with foreign UUIDs in that list?

If we don't; what's the implication of ActivateUsers being off for a cluster listed in RemoteClusters?

Update: adding 'ActivateUsers: true' on both sides fixes this issue.

=> #16974

Ward Vandewege wrote:

[login federation] [experiment run from satellite cluster] [WB2] / [WB1]

Strange multisite search behavior.

Steps to reproduce:

• [WB2] go to the Site Manager. Refresh the page until the login cluster is the second line (not the first).

That's a bug. Why is the 'Logged in' button greyed always out for the first item in the list, not for a real reason (e.g. it's the cluster you are on, and it's the login cluster in a federation)?

I couldn't reproduce this. The "active" session is the one that is disabled, and that is always the primary cluster you are communicating with, so I have no idea how you got into a situation where that was wrong.

• [WB2] click the "Logged in" button for ce8i5 (the login cluster), which will log you out there
• [WB2] verify that it shows as 'logged out'
• switch WB1
• [WB1] go to multisite search
• [WB1] notices that it is not logged in on one of the clusters, and fixes that by redirecting to that cluster, which logs me in there transparently, but that leaves me on that cluster.

Bug: this is very confusing; it's hard to tell the workbenches apart, so now the user is suddenly on a different one than where they started...

This seems to be by design. When you are in a peer federation, you want to be on the home cluster to do your searches, so it redirects the browser for you. This is a wb1 specific behavior so I intend to leave it as-is.

search for a term that should produce results from the newly added cluster (I used 'diagnostics')