Bug #17587

Updated by Ward Vandewege about 2 months ago

This is happening on Tordo (AWS test cluster), at version 2.2.0~dev20210426141415-1. Tordo is in a LoginCluster federation with ce8i5 (Azure test cluster) as the login cluster.

Go to https://workbench.tordo.arvadosapi.com/ and log in. The home page is a fiddlesticks with the following backtrace in the logs.

<pre>
#<ActionView::Template::Error: cannot execute federated list query with limit, offset, or order parameter [API: 400]>
/var/www/arvados-workbench/current/app/models/arvados_api_client.rb:180:in `api'
/var/www/arvados-workbench/current/app/models/arvados_resource_list.rb:240:in `each_page'
/var/www/arvados-workbench/current/app/models/arvados_resource_list.rb:151:in `each'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:921:in `recent_collections'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/abstract_controller/helpers.rb:67:in `recent_collections'
/var/www/arvados-workbench/current/app/views/projects/_show_dashboard.html.erb:27:in `_app_views_projects__show_dashboard_html_erb__1029083890495766025_47029307796900'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/template.rb:159:in `block in render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/notifications.rb:170:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/template.rb:354:in `instrument_render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/template.rb:157:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/partial_renderer.rb:344:in `block in render_partial'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/notifications.rb:170:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/partial_renderer.rb:333:in `render_partial'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/partial_renderer.rb:312:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/renderer.rb:49:in `render_partial'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/renderer.rb:23:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/helpers/rendering_helper.rb:33:in `render'
/var/www/arvados-workbench/current/app/views/projects/index.html.erb:10:in `_app_views_projects_index_html_erb__977444551465127700_47029307989520'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/template.rb:159:in `block in render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/notifications.rb:170:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/template.rb:354:in `instrument_render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/template.rb:157:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/template_renderer.rb:54:in `block (2 levels) in render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/abstract_renderer.rb:44:in `block in instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/notifications.rb:170:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/abstract_renderer.rb:43:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/template_renderer.rb:53:in `block in render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/template_renderer.rb:61:in `render_with_layout'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/template_renderer.rb:52:in `render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/template_renderer.rb:16:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/renderer.rb:44:in `render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/renderer/renderer.rb:25:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/rendering.rb:103:in `_render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/streaming.rb:219:in `_render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/rendering.rb:84:in `render_to_body'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/abstract_controller/rendering.rb:25:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/rendering.rb:36:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/instrumentation.rb:46:in `block (2 levels) in render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
/usr/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/core_ext/benchmark.rb:14:in `ms'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/instrumentation.rb:46:in `block in render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/railties/controller_runtime.rb:36:in `cleanup_view_runtime'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/instrumentation.rb:45:in `render'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:197:in `block (2 levels) in render_index'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/mime_responds.rb:203:in `respond_to'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:180:in `render_index'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:231:in `index'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/abstract_controller/base.rb:194:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:681:in `require_thread_api_token'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:663:in `set_thread_api_token'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:588:in `set_current_request_id'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:580:in `thread_clear'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:136:in `run_callbacks'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/notifications.rb:168:in `block in instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/notifications.rb:168:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.2.4.5/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/abstract_controller/base.rb:134:in `process'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionview-5.2.4.5/lib/action_view/rendering.rb:32:in `process'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal.rb:191:in `dispatch'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_controller/metal.rb:252:in `dispatch'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/callbacks.rb:98:in `run_callbacks'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/lograge-0.10.0/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.5/lib/rails/rack/logger.rb:26:in `block in call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/tagged_logging.rb:71:in `block in tagged'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/tagged_logging.rb:28:in `tagged'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/tagged_logging.rb:71:in `tagged'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.5/lib/rails/rack/logger.rb:26:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/request_store-1.4.1/lib/request_store/middleware.rb:19:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.4.5/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/actionpack-5.2.4.5/lib/action_dispatch/middleware/ssl.rb:74:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.5/lib/rails/engine.rb:524:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.5/lib/rails/railtie.rb:190:in `public_send'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.2.4.5/lib/rails/railtie.rb:190:in `method_missing'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
</pre>

I think these are the corresponding log entries from controller:

<pre>
Apr 26 21:40:59 tordo.arvadosapi.com arvados-controller[6632]: {"PID":6632,"RequestID":"req-e072lql42urc35o4j9u9","level":"info","msg":"response","remoteAddr":"127.0.0.1:34102","reqBytes":505,"reqForwardedFor":"10.253.0.41","reqHost":"tordo.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/groups","reqQuery":"","respBody":"{\"errors\":[\"cannot execute federated list query with limit, offset, or order parameter\"]}\n","respBytes":90,"respStatus":"Bad Request","respStatusCode":400,"time":"2021-04-26T21:40:59.280014037Z","timeToStatus":0.000370,"timeTotal":0.000381,"timeWriteBody":0.000010}
Apr 26 21:41:03 tordo.arvadosapi.com arvados-controller[6632]: {"PID":6632,"RequestID":"req-7wav6c701f3awhskq7a8","level":"info","msg":"response","remoteAddr":"127.0.0.1:34134","reqBytes":505,"reqForwardedFor":"10.253.0.41","reqHost":"tordo.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/groups","reqQuery":"","respBody":"{\"errors\":[\"cannot execute federated list query with limit, offset, or order parameter\"]}\n","respBytes":90,"respStatus":"Bad Request","respStatusCode":400,"time":"2021-04-26T21:41:03.813853070Z","timeToStatus":0.000299,"timeTotal":0.000309,"timeWriteBody":0.000009}
Apr 26 21:41:47 tordo.arvadosapi.com arvados-controller[6632]: {"PID":6632,"RequestID":"req-3x8j992o12l1f32dbefm","level":"info","msg":"response","remoteAddr":"127.0.0.1:34184","reqBytes":505,"reqForwardedFor":"10.253.0.41","reqHost":"tordo.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/groups","reqQuery":"","respBody":"{\"errors\":[\"cannot execute federated list query with limit, offset, or order parameter\"]}\n","respBytes":90,"respStatus":"Bad Request","respStatusCode":400,"time":"2021-04-26T21:41:47.244177375Z","timeToStatus":0.000305,"timeTotal":0.000315,"timeWriteBody":0.000010}
</pre>

Back